A stereovision and acoustic fusion type recirculating aquaculture feeding method and system

Abstract

This invention discloses a feeding method and system for recirculating aquaculture systems that integrates stereoscopic vision and acoustics. A depth camera array and a hydrophone array simultaneously acquire three-dimensional depth images, infrared images, and underwater acoustic signals of fish feeding. These are input into a cross-modal attention fusion model to obtain the feeding state characteristics of the fish. Based on feeding biological time-delay priors, a bidirectional reverse verification coupling is implemented between the acoustic and visual channels to suppress acoustic misjudgments caused by mechanical noise from the circulating water pump. The fused features are constructed as a time-series feature sequence and input into an AV-BET model to obtain a recommended feeding amount. Then, a dynamic neural balance solution algorithm is used to solve for the optimal feeding amount based on feeding accuracy, efficiency, and safety cost functions. A feeding robot is driven to perform feeding, and the feeding response data is stored in an experience playback buffer. An elastic weight integration method is used to incrementally train the model.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for aquaculture, specifically to a feeding method and system for recirculating aquaculture that integrates stereoscopic vision and acoustics. Background Technology

[0002] Recirculating aquaculture systems (RAS) utilize water treatment processes such as physical filtration, biological filters, and ultraviolet sterilization to recycle aquaculture water. Compared to traditional flow-through aquaculture, RAS offers advantages such as smaller footprint, lower water consumption, and greater controllability, making it a major development direction for large-scale aquaculture. However, the high-density nature of RAS amplifies the impact of feeding decisions on water quality indicators, growth rate, and feed conversion ratio. Underfeeding significantly reduces growth rate and marketable size, while overfeeding accumulates large amounts of uneaten food and metabolic waste in the water column, rapidly increasing the load on the biological filter and even inducing acute ammonia poisoning. Accurately determining the feeding amount based on the real-time feeding status of the fish population is the core bottleneck restricting the efficiency of RAS.

[0003] Currently, recirculating aquaculture systems (RAS) still largely rely on manual experience for feeding decisions. Operators subjectively estimate feeding timing and amounts by observing water splashes, feeding intensity, and remaining feed, leading to issues such as high subjectivity, poor consistency across shifts, and high labor costs. Most publicly available intelligent feeding solutions are based on single sensors: surface camera-based solutions suffer from a sharp decline in detection accuracy under strong light, water surface reflections, and floating feed interference; underwater camera-based solutions are prone to losing targets in turbid water and when fish obstruct each other's view; while hydrophone-based solutions are unaffected by light, the periodic start-stop of the recirculating pump, aerator blade vibration, and crosstalk from neighboring ponds all generate low-frequency pulses similar to the acoustics of feeding, resulting in an artificially high detected frequency of feeding events.

[0004] In recent years, visual-acoustic multimodal fusion schemes have improved perception robustness to some extent. However, the publicly available schemes generally adopt a one-way strategy of forward weighted fusion. In actual aquaculture scenarios where water pumps are periodically started and stopped, such one-way fusion schemes will incorrectly use water pump noise as a feeding signal and add it to the fusion features. This will cause the downstream time series prediction model to misjudge that the fish are in a state of hunger and push up the feeding amount. In addition, most existing schemes take single feeding as the optimization target and lack a closed-loop correction mechanism based on real-time feedback and adaptive learning ability for fish growth cycle and seasonal changes. The model will experience performance drift after a few weeks of use and requires frequent manual retraining. Summary of the Invention

[0005] This invention proposes a feeding method and system for recirculating aquaculture that integrates stereo vision and acoustics, in order to solve the technical problems of existing recirculating aquaculture feeding schemes, which are caused by the instability of single sensor perception, lack of long-term adaptive learning ability for fish growth cycle and aquaculture environment fluctuations, resulting in distorted judgment of fish feeding status, inflated feeding amount induced by mechanical noise of circulating water pump, and long-term decline in model prediction accuracy requiring frequent manual retraining.

[0006] To solve the above-mentioned technical problems, the present invention provides a feeding method for recirculating aquaculture systems that integrates stereoscopic vision and acoustics, comprising the following steps: Step S1: Acquire three-dimensional depth images, infrared images, and underwater acoustic signals of the fish feeding using a depth camera array and a hydrophone array; Step S2: Input the three-dimensional depth image, infrared image and underwater acoustic signal into the cross-modal attention fusion model to output the feeding state features of the fish school; Step S3: Construct the feeding status characteristics of the fish school into a time-series feature sequence, input it into the time-series prediction model, and output the recommended feeding amount; Step S4: Use the dynamic neural balance solution algorithm to solve for the optimal feeding amount based on the recommended feeding amount; Step S5: Drive the feeding robot to feed the fish according to the optimal feeding amount, and collect the feeding response data of the fish and store it in the experience playback buffer; Step S6: Based on the experience replay buffer, the time series prediction model is incrementally trained using the elastic weight integration method.

[0007] Preferably, the method further includes a step of preprocessing the three-dimensional depth image, infrared image, and underwater acoustic signal: Step S1.1: Use Zhang Zhengyou calibration method to calibrate the intrinsic parameters of the depth camera array, obtain the intrinsic parameter matrix and distortion coefficient of each camera, and obtain the relative extrinsic parameters through multi-camera joint calibration, and align the coordinate system of each camera to the world coordinate system; Step S1.2: Perform median filtering, morphological operations, and histogram equalization on the infrared image in sequence to obtain a denoised infrared image; Step S1.3: Collect background noise samples from the operating environment of the hydrophone array, and perform adaptive filtering and noise reduction on the underwater acoustic signal based on the background noise samples; Step S1.4: Transform the 3D pixel coordinates of the fish detected by the depth camera array to the world coordinate system using the following formula, and perform clustering and merging to eliminate duplicate detections based on the Euclidean distance threshold: ; In the formula, In three-dimensional world coordinates, For the first Rotation matrix of each camera, For the first The intrinsic parameter matrix of each camera, , For pixel coordinates, For depth value, For the first The translation vector of each camera.

[0008] Preferably, the feeding state characteristics of the fish school include the depth distribution entropy of the fish school. Average swimming speed velocity variance Average nearest neighbor distance Correcting the convex hull area Frequency of feeding events and Mel frequency cepstral coefficients MFCC; wherein the fish swarm depth distribution entropy Calculate using the following formula: ; ; In the formula, The fish school is located in the first The probability of a depth interval For the first Number of fish within each depth range The total number of fish. This represents the total number of depth intervals.

[0009] Preferably, the attention of the cross-modal attention fusion model The calculation formula is: ; The acoustic gating factor and the visual gating factor are calculated according to the following formulas: ; ; In the formula, Q, K, and V are the query matrix, key matrix, and value matrix obtained by linear projection of the visual features extracted from the 3D depth image and infrared image by the cross-modal attention fusion model, respectively. For acoustic gating factors; ⊙ represents the visual gating factor; ⊙ represents the Hadamard product. The vector dimension of the query matrix; Use the Sigmoid activation function; Frequency of feeding events; Adjust the convex hull area for the fish population; , These are the acoustic learning weight vector and the visual learning weight vector, respectively. , These are the acoustic gating bias vector and the visual gating bias vector, respectively.

[0010] Preferably, an acoustic-visual bidirectional reverse verification coupling step is further included between step S2 and step S3: Step S2.1: Extract the rate of change of fish swarm aggregation in the visual channel and the rate of change of feeding event frequency in the acoustic channel from the cross-modal attention fusion model, respectively; Step S2.2: According to the preset biological prior time delay The frequency change rate of the feeding event in the acoustic channel is time-delayed and shifted, and the time delay is calculated for a length of [missing information]. The acoustic-visual consistency factor is calculated using the following formula within the sliding window. : ; In the formula, for The acoustic-visual consistency factor at any given time. For covariance operators, This represents the change in fish aggregation intensity output by the visual channel. This represents the frequency variation of the feeding event output by the acoustic channel. For time step, , Sliding windows Inside and Standard deviation; Step S2.3: Set the consistency threshold ,when The original acoustic gating factor of the cross-modal attention fusion model is maintained at the same time. and original visual gating factor Unchanged; when The acoustic gating factor after inverse correction is updated according to the following formula. and the visual gating factor after inverse correction : ; ; In the formula, The acoustic inverse attenuation coefficient, The visual inverse enhancement coefficient; Step S2.4: Use the inversely corrected acoustic gating factor and visual gating factors Replace the original acoustic gating factor and the original visual gating factor of the cross-modal attention fusion model.

[0011] Preferably, the biological prior delay The value range is from 0.3 seconds to 0.8 seconds; the length of the sliding window The value range is from 3 seconds to 10 seconds; the consistency threshold The value range is from 0.3 to 0.5; the acoustic reverse attenuation coefficient The value range is from 0.2 to 0.4; the visual inverse enhancement coefficient The value range is from 0.1 to 0.3.

[0012] Preferably, the temporal prediction model is an AV-BET model, which includes a cross-modal asynchronous temporal feature extraction layer, a feedforward network layer, and an output prediction layer connected in sequence; the feedforward network layer The calculation formula is: ; The output prediction layer maps the hidden state of the last time step of the feedforward network layer to the recommended feed amount according to the following formula: ; In the formula, The cross-modal feature matrix is ​​acoustically gated. , These are the fully connected weight matrices for the first and second layers of the feedforward network, respectively. , These are the bias vectors for the first and second layers of the feedforward network, respectively. This represents the hidden state vector at the last time step of the feedforward network layer. To output the weight vector of the prediction layer, For the output bias term of the output prediction layer, This is the recommended feeding amount.

[0013] Preferably, the solution formula of the dynamic neural balance solution algorithm is: ; ; ; ; In the formula, To achieve the optimal feeding amount, For the recommended feeding amount, For the feeding accuracy cost function, Let the efficiency cost function be... For the safety cost function of feeding, , , These are the weighting coefficients corresponding to the feeding accuracy cost function, efficiency cost function, and feeding safety cost function, respectively. For the target feeding amount, for At any given time, the fish have consumed their food. For the estimated total feeding amount, For feeding time, Let t be the actual feeding amount output by the decision at time t. For maximum safe feeding amount, This is the minimum safe feeding amount.

[0014] The present invention also provides a recirculating aquaculture feeding system integrating stereoscopic vision and acoustics for implementing the above-described method, the system comprising: The sensing module includes the depth camera array and the hydrophone array, and is used to acquire three-dimensional depth images, infrared images and underwater acoustic signals. The feature extraction and fusion module is used to obtain the feeding status features of the fish school based on the three-dimensional depth image, the infrared image and the underwater acoustic signal; The decision-making and optimization module is used to construct the feeding status characteristics of the fish group into a time-series feature sequence and input it into the time-series prediction model to output a recommended feeding amount, and to use a dynamic neural balance solution algorithm to solve for the optimal feeding amount based on the recommended feeding amount; The feeding execution module includes a feeding robot, which is used to drive the feeding robot to feed according to the optimal feeding amount, and to collect the feeding response data of the fish and store it in the experience playback buffer. An incremental learning module is used to incrementally train the time series prediction model based on the experience replay buffer using an elastic weight integration method.

[0015] Preferably, the depth camera array includes at least three depth cameras, wherein at least two of the depth cameras are arranged at the bottom of the aquaculture pond and at least one of the depth cameras is arranged directly above the aquaculture pond.

[0016] The beneficial effects of the present invention include at least the following: (1) By deploying a depth camera array at the bottom and above the aquaculture pond and a hydrophone array that collects data simultaneously, the three-dimensional spatial distribution, movement behavior and feeding acoustic characteristics of the fish can be perceived in all directions. The three-view combination can cover the entire water area of ​​the aquaculture pond. Combined with the infrared channel, the visual blind spots under turbid water are avoided, and the stability of the fish feeding status is improved. (2) This invention transforms the feeding accuracy cost, efficiency cost and safety cost into real-time adjustment gain through a dynamic neural balance solution algorithm, so that the final feeding amount simultaneously meets the triple constraints of being close to the model recommended value, having reasonable feeding efficiency and not exceeding the safety range. It can dynamically adjust the feeding strategy according to the real-time feeding response of the fish, greatly improving the feeding accuracy and adaptability.

[0017] (3) This invention achieves continuous adaptive updating of the model during the aquaculture process by combining the experience replay buffer and the elastic weight integration method. It can adjust the prediction parameters according to the growth stage of the fish population, seasonal changes and fluctuations in the aquaculture environment, avoid the catastrophic forgetting of old task parameters when training new tasks, and maintain long-term prediction accuracy without frequent human intervention. Attached Figure Description

[0018] Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 Examples of depth and infrared images acquired in embodiments of the present invention; Figure 3 This is an example of a comparison between visual image preprocessing and preprocessing in an embodiment of the present invention; Figure 4 This is an example diagram of the waveform and spectrum of the feeding sound in an embodiment of the present invention; Figure 5 This is a schematic diagram of the system structure of the present invention; Figure 6 This is a schematic diagram of hardware deployment according to an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0020] Example 1 like Figure 1 As shown, this embodiment of the invention provides a feeding method for recirculating aquaculture systems that integrates stereoscopic vision and acoustics, comprising the following steps: Step S1: Acquire three-dimensional depth images, infrared images, and underwater acoustic signals of the fish feeding using a depth camera array and a hydrophone array.

[0021] Specifically, in this embodiment of the invention, after the feeding task is initiated, a camera array consisting of three Intel RealSense D435 depth cameras is used. Two cameras are positioned at the bottom of the rearing pond, and one is positioned directly above the pond, ensuring that the camera array can capture all points in the pond. Simultaneously, a hydrophone records the sound signals of the fish feeding process. The three-dimensional depth image generated by the depth camera array directly provides the spatial coordinates of each fish in the water, while the infrared channel can still outline the fish's contours even when the water turbidity increases or the natural light is weak. The hydrophone array collects underwater acoustic signals that simultaneously include high-frequency pulses generated by the fish biting the pellet food, low-frequency water flow sounds generated by the fish swimming, and background mechanical vibrations from equipment such as the circulating water pump and aerator.

[0022] In this embodiment, two tasks need to be completed before the camera array is deployed for the first time: intrinsic parameter calibration and multi-camera extrinsic parameter calibration. The intrinsic and extrinsic parameter calibrations of the camera array are performed, and the intrinsic parameter matrices of each camera are obtained by Zhang Zhengyou's calibration method. The distortion coefficients are obtained through multi-camera joint calibration to obtain relative extrinsic parameters, and the coordinate systems of each camera are aligned with the world coordinate system.

[0023] After each feeding is initiated, the raw signal needs to be preprocessed before entering the feature extraction stage. Infrared images contain noise from scattering by suspended particles in the water and small connected regions formed by floating bait particles. In this embodiment, a 5×5 median filter is first applied to suppress noise, then morphological opening operations (erosion followed by dilation) are used to remove isolated small connected components, closing operations are used to fill in cavities inside the fish's body, and finally histogram equalization is applied to enhance contrast. The original images acquired in this embodiment are shown below. Figure 2 As shown, the preprocessed image is as follows: Figure 3 As shown.

[0024] The preprocessing of the underwater acoustic signal focuses on suppressing low-frequency background noise generated by the circulating water pump and aerator. Since the spectral characteristics of these mechanical noises are relatively stable, this embodiment records 10 minutes of background noise samples continuously during the system debugging phase, prior to the period when the water circulation is normal but fish are not feeding (3:00-3:10 AM). The spectral characteristics are extracted and used as the reference input for the adaptive LMS filter. The underwater acoustic signal measured during feeding is then denoised in real-time by this filter, effectively eliminating the power frequency harmonics of the circulating water pump and the low-frequency vibrations generated by the aerator blades. The waveform after denoising is shown below. Figure 4 As shown.

[0025] The fields of view of the three cameras overlap, meaning the same fish may be detected simultaneously by multiple cameras. This embodiment specifies the three-dimensional pixel coordinates of the fish detected by each camera. Map to the world coordinate system using the following formula: ; In the formula, In three-dimensional world coordinates, and The first The rotation matrix and translation vector of the camera. This is the intrinsic parameter matrix. , For pixel coordinates, This represents the depth value.

[0026] After transformation to the world coordinate system, all candidate coordinates are hierarchically clustered and merged according to the Euclidean distance threshold. This ensures that the same fish body is not incorrectly split into multiple targets and avoids erroneous merging with adjacent individuals. The center position of each cluster is output as the final spatial coordinates of the corresponding fish body in the subsequent feature extraction stage.

[0027] Step S2: Input the 3D depth image, infrared image and underwater acoustic signal into the cross-modal attention fusion model to output the feeding state features of the fish school.

[0028] Specifically, the preprocessed visual and acoustic data enter the feature extraction stage, extracting multidimensional features that reflect the feeding state of the fish school: the visual side includes five items: fish school depth distribution entropy, average swimming speed, speed variance, average nearest neighbor distance, and corrected convex hull area; the acoustic side includes two items: feeding event frequency and Mel frequency cepstral coefficients. The two together support the subsequent cross-modal attention fusion.

[0029] The vertical spatial distribution of fish schools is strongly correlated with their feeding status. When hungry, a large number of individuals rise to the feeding area 0.2m to 0.4m below the surface; after satiating, they sink to the middle and lower layers and distribute evenly. In this embodiment, the water depth is divided into... Each depth interval is used to count the number of fish falling within each interval of the current frame. The total number of all fish Therefore, the probability of each interval is calculated. and depth distribution entropy : ; ; The entropy value is 0 when the fish are completely concentrated in a certain interval, and reaches its maximum value when they are evenly distributed across 5 intervals. In this embodiment K The number of depth intervals is set according to the depth of the aquaculture water. When hungry, fish congregate in the upper layer with lower entropy values; when satiated, they are evenly distributed with higher entropy values.

[0030] Kinematic characteristics are represented by the average swimming speed of the group. and velocity variance Depicting overall activity level. Let the first... Tail fish in The world coordinates of the moment are The instantaneous velocity of adjacent frames is calculated by dividing the Euclidean distance by the inter-frame time Δt: ; The average speed and the speed variance are: ; ; A higher average speed reflects the rapid response of the fish group after the bait is placed. A larger variance indicates that the fish in the front row that are close to the bait swim quickly, while the fish in the outer row are relatively slower. Both of these factors together constitute the characteristics of the group behavior during the intense feeding phase.

[0031] Clustering is determined by the average nearest neighbor distance. and corrected convex hull area Common description. The former calculates the mean distance of each fish to its nearest neighbor: ; During the feeding phase, the fish move closer to each other. Reduced; fish disperse during the satiated stage. Increase.

[0032] The original calculation of the convex hull area used the Graham scan method to construct the minimum convex polygon of the fish school's horizontal projection, and then the Shoelace formula to calculate the geometric area. This approach performs well in most cases, but when there are occasional scattered fish bodies on the pond edge far from the main group, the convex hull vertices are strongly pulled by the outer individuals, resulting in an abnormally large geometric area that is inconsistent with the physical representation of the "overall aggregation degree of the fish school." Therefore, this invention addresses this problem by using a dimensionless correction factor based on depth distribution entropy. Weighted adjustment for Shoelace area: ; in The concentration sensitivity coefficient is determined based on the feeding intensity of the fish population; in this example, the value is 5.60. Acoustic side features based on feeding event frequency With Mel-frequency cepstral coefficients (MFCC) as the core, the acoustic waveform details are further characterized. The transient response of fish biting pellet bait creates short, high-energy pulses on the underwater acoustic waveform. Therefore, a discrimination strategy based on short-term energy exceeding a threshold can identify feeding events. The denoised underwater acoustic signal is divided into segments with a length of... (Sampling points), frame shift is The frame, the Short-time energy of a frame Calculate using the following formula: ; For signal index The sampled value at that location, For length The window function performs Hamming windowing on the current frame signal.

[0033] When the energy exceeds the set threshold for three consecutive frames in a short period of time The segment is then marked as a candidate feeding event, and the time-spectrum graph corresponding to the candidate segment is fed into a pre-trained binary classifier for secondary confirmation. Finally, the feeding frequency per minute is calculated. .

[0034] In this embodiment, the threshold The calibration method is as follows: under normal operating conditions of empty pool water circulation, background acoustic samples are continuously collected for 30 minutes, and the average short-time energy of all frames is calculated. and standard deviation ,Pick The probability of background noise exceeding this threshold is approximately one in a million.

[0035] Mel frequency cepstral coefficients (MFCCs) are more expressive of feeding voiceprint details and are calculated using the following formula: ; In the formula, This is the normalization factor for the cepstral coefficients. The number of Mel filter banks, For the first The energy output of a Mel filter. These are the discrete cosine transform basis functions, used to transform the Mel spectrum to the cepstrum domain.

[0036] Step S3: Construct the feeding status characteristics of the fish school into a time-series feature sequence, input it into the time-series prediction model, and output the recommended feeding amount.

[0037] Visual and acoustic features are coupled via a cross-modal attention fusion model to output fish feeding status features for downstream time-series prediction. In this embodiment, the query matrix is ​​obtained by independent linear projection of the visual features. Key matrix Sum matrix The acoustic feeding event frequency and the visually corrected convex hull area are respectively used to generate acoustic gating factors and visual gating factors through independent Sigmoid gating networks. These two factors serve as the learnable confidence of the corresponding channels to modulate the fused output. ; ; ; In the formula, ⊙ represents the Hadamard product. To query the vector dimension of the matrix, It is the Sigmoid activation function. and These are the acoustic learning weight vector and the visual learning weight vector, respectively. and These are the acoustic gating bias vector and the visual gating bias vector, respectively. Substitute them into the actual measurements. =38、 =0.08、 =-2.0 This indicates that the system injects acoustic event information into the visual feature channel with approximately 73.9% confidence. Visual gating is performed in the same manner by... The driver, in this embodiment, is taken as... =0.5、 =-0.2, therefore Finally, a fusion feature vector reflecting the confidence level of the feeding state at the current moment is obtained.

[0038] Acoustics-guided cross-modal asynchronous temporal feature extraction layer: capturing the long-term dependencies of fish feeding behavior, with input being a multimodal feature sequence of the past T time steps aligned with an elastic time window. Temporal information is injected through biological rhythm location encoding, and then computed via interactive semantic attention: In the formula, This is a visual feature mapping matrix; , is the gating factor. For acoustic learning weights, For acoustic gating bias term, ⊙ represents the Sigmoid activation function; ⊙ represents the Hadamard product. Vector dimension; Modal confidence bias matrix.

[0039] Step S4: Use the dynamic neural balance solution algorithm to solve for the optimal feeding amount based on the recommended feeding amount.

[0040] Specifically, the feeding status characteristics of the fish school in this embodiment are based on past... The time steps are combined to form a temporal feature sequence. After being encoded with biological rhythms and injected with temporal information, the data is fed into the AV-BET temporal prediction model. The main body of the model consists of a cross-modal asynchronous temporal feature extraction layer, a feedforward network layer, and an output prediction layer. The feedforward network layer adopts a two-layer fully connected structure and is stabilized through residual connections and layer normalization. ; The output prediction layer maps the hidden state of the last time step of the feedforward layer to the recommended feed amount: ; in, The cross-modal feature matrix is ​​gated. , This is the feedforward layer weight matrix. , For the corresponding bias vector This is the hidden state vector for the last time step, which is the fusion of the acoustic experiment and the historical feedback deviation. , This is used to output the weight vector and bias term of the prediction layer.

[0041] In this embodiment, since the recommended feeding amount output by AV-BET only reflects the instantaneous judgment of the fish's feeding status and does not yet consider the execution accuracy, cumulative efficiency, and safety margin during the feeding process, this embodiment further uses a dynamic neural balance solution algorithm for a final correction to obtain the optimal feeding amount issued to the robot. : ; Three cost functions measure the rationality of the recommendation volume from different dimensions: ; ; ; The cost to feeding accuracy reflects the difference between the recommended amount and the manually set target amount. Deviation; For the sake of efficiency, The fish have consumed the amount of food at any given time. Normalized to the expected total feed amount To regulate the feeding rhythm; For safety reasons, the maximum safe feeding amount was exceeded. or below the minimum safe feeding amount The corresponding penalty will be triggered at that time. The time-varying weighting coefficients for each cost are used to unify the cost values ​​to the adjacent range of the normalized values. This was confirmed through cross-testing in this embodiment.

[0042] Step S5: Drive the feeding robot to feed the fish according to the optimal feeding amount, and collect the feeding response data of the fish and store it in the experience playback buffer.

[0043] In this embodiment, the optimal feeding amount is distributed to a feeding robot above the aquaculture pond. The robot evenly distributes the feed on the water surface and records the actual feeding amount, feeding start time, and feeding duration in real time for each feeding. During the feeding process, data on the fish's feeding response is continuously collected, including the visual change in the remaining quantity of bait balls over time, detection of food deposited at the bottom of the pond, and continuous monitoring of the frequency of feeding events. All feedback data, along with the input features, recommended amount, and optimal feeding amount for this feeding, are packaged together as a new sample and uploaded to the cloud-based experience playback buffer.

[0044] Step S6: Incrementally train the time series prediction model using the elastic weight integration method based on the experience replay buffer.

[0045] Specifically, offline incremental training of the AV-BET model is triggered when the accumulated number of samples in the cloud-based experience replay buffer reaches a threshold. Fish exhibit different feeding patterns depending on their growth stage, seasonal temperature, and water quality. Simply training with new samples can easily lead to the model catastrophically forgetting key parameters from the old task. This invention employs an elastic weight integration method to apply constraints, superimposing a secondary penalty term for historical key parameters on top of the loss function for new samples. This ensures that the training process fits new samples while maintaining reasonable retention of important parameters from the old task. After each round of incremental training, the new model weights are first subjected to A / B testing on a fixed validation set. Only when their prediction error on the validation set is not inferior to the old model is they officially deployed to the edge inference module for production, thus avoiding the performance degradation risk associated with incremental training.

[0046] Example 2 The aforementioned method is implemented in engineering by five collaborative parts: a perception module, a feature extraction and fusion module, a decision and optimization module, a feeding execution module, and an incremental learning module. The overall system structure is as follows: Figure 5 As shown, the hardware deployment diagram is as follows: Figure 6 As shown.

[0047] The sensing module includes the aforementioned three Intel RealSense D435 depth cameras and four hydrophones. Two underwater cameras in the depth camera array are positioned at the bottom of the aquaculture pond, while the third camera above the water surface is mounted on a top bracket and looks vertically down at the water surface. This three-view combination can cover the entire water area of ​​the aquaculture pond. The four hydrophones are positioned at the four corners of the aquaculture pond at a depth of 0.5 m to simultaneously record the sound signals of the fish feeding.

[0048] The feature extraction and fusion module is responsible for the computation of feature extraction and attention fusion in the method. The decision and optimization module is deployed on a local server and runs the AV-BET time series prediction and dynamic neural balance solution algorithm. The feeding execution module consists of a surface feeding robot with quantitative feeding capability and a surface camera. The incremental learning module is deployed in the cloud, triggering model retraining in batches of 2000 samples, and distributing weights after verification.

[0049] The data flow and control flow between the modules are as follows: the three-dimensional depth image, infrared image and underwater acoustic signal synchronously collected by the perception module are transmitted to the feature extraction and fusion module; the fish feeding status features output by the fusion module are uploaded to the decision and optimization module; the optimal feeding amount output by the decision and optimization module is sent to the feeding execution module; the actual feeding data generated by the feeding execution module and the fish feeding response data collected by the perception module are merged and packaged and uploaded to the cloud incremental learning module.

[0050] Example 3 This embodiment adds an acoustic-visual reverse verification step based on the biological delay of feeding to the basis of embodiment 1.

[0051] In recirculating aquaculture systems, mechanical interference is common, including periodic pump start-stop cycles, aerator blade vibration, and crosstalk from neighboring ponds. These low-frequency pulses, when projected onto hydrophones, closely resemble the sounds of actual feeding and biting, making simple forward-weighted fusion insufficient to distinguish them. If mechanical noise is incorrectly identified as a feeding event, the frequency count of that event becomes artificially high, leading to a higher acoustic weighting through sigmoid gating. This amplifies the noise component in the fused features, transmitting erroneous strong feeding signals from fish to the downstream time-series prediction model. This embodiment introduces an acoustic-visual bidirectional reverse verification coupling mechanism after the cross-modal attention fusion output and before the time-series prediction input to specifically address this interference.

[0052] The mechanism is based on an observation of feeding behavior: after sensing bait entering the water, fish such as largemouth bass undergo a visual response phase of about 0.3 to 0.8 seconds. The fish then quickly swim towards the bait's floating area, approach each other, and adjust their biting posture before initiating a burst of acoustic biting that lasts for several seconds. This means that the rate of change in visual concentration should precede the rate of change in acoustic event frequency, and the covariance of their waveform envelopes should be significantly positive. Conversely, if the acoustic frequency suddenly increases when there is no significant change in visual concentration, the acoustic event is very likely not caused by the fish feeding.

[0053] In this embodiment, this pattern is transformed into the following calculation process in engineering: extract the fish swarm aggregation change rate from the visual channels respectively from the cross-modal attention fusion model. Rate of change of feeding event frequency in acoustic channels Acoustic rate of change according to biological prior time delay After time delay misalignment, in length The normalized covariance of the two is calculated within a sliding window as the audio-visual consistency factor: ; in, For covariance operators, , Each is within the sliding window and The standard deviation. Values ​​range from [-1, 1]: positive values ​​indicate a biologically consistent causal relationship between visual convergence and acoustic bursts, while negative values ​​or values ​​close to zero indicate no significant causal link between the two. A consistency threshold is set. As a discrimination boundary, when When the original gating factor remains unchanged, When the acoustic gating and visual gating are reversed, the following formulas are used: ; ; The acoustic inverse attenuation coefficient, The visual inverse enhancement coefficient. The function ensures that the visual gating correction remains within the upper bound of [0,1]. The corrected gating factor replaces the original value, the attention output is recalculated, and then fed into the subsequent temporal prediction stage.

[0054] The threshold and coefficients in this embodiment are calibrated as follows: biological prior time delay These are core parameters. In this embodiment, 50 sets of synchronized visual and acoustic feeding process videos and audio recordings were collected for each of three typical farmed fish species: California bass, mandarin fish, and tilapia. For each sample, the time difference between the peak visual aggregation and the peak acoustic frequency was calculated, and the statistical median was taken as the mean. Empirical values ​​are used, with approximately 0.5 seconds for California bass, 0.6 seconds for mandarin fish, and 0.4 seconds for tilapia. The overall value range is [insert range here]. Sliding window length A trade-off needs to be struck between response sensitivity and covariance estimation stability. This embodiment addresses this issue. Different values ​​within the range were subjected to 10-fold cross-validation on historical samples, with the false positive rate of feeding amount selected as the evaluation index. The overall value range is Consistency threshold The value was determined by the upper left inflection point of the ROC curve of historical samples. In this embodiment, the calibration result is 0.4, and the overall value range is [missing value]. Reverse correction coefficient , Following the principle of starting with the weak and then moving to the strong, the initial phase of new deployments should focus on... =0.20、 =0.10 to avoid over-correction that could compromise fusion stability; adjust to 0.10 after accumulating approximately 1000 labeled samples. =0.30、 =0.20, maximum not exceeding =0.40、 =0.30.

[0055] Example 4 To demonstrate the reliability of this study's approach to feeding decisions in recirculating aquaculture systems, two groups of experimental fish were selected for comparison in this embodiment. The experiment consisted of 20 California bass in each group, with an average body length of 22 cm. One group used the stereoscopic vision and acoustic fusion feeding decision method proposed in Embodiment 1, while the other group used a single-camera feeding decision method without acoustic information. Both groups of fish were raised in identical culture tanks under the same environmental conditions to ensure the control variables of the experiment. Table 1 shows the actual feeding amounts for the two groups of fish over 20 consecutive days.

[0056] Table 1 Experimental results show that the stereoscopic vision and acoustic information fusion proposed in this invention reduces the feeding amount by 9.7% compared to the single-camera control group without acoustics. Furthermore, to eliminate misjudgments caused by insufficient feeding amount, the amount of remaining feed after each feeding was also recorded, as shown in Table 2. Table 2 Experimental results show that the method proposed in this embodiment reduces the feed waste rate by 84% compared to the control group, proving the reliability of the method.

[0057] Example 5 This invention also provides a recirculating aquaculture feeding system integrating stereoscopic vision and acoustics for implementing the above-described method. The system includes: The sensing module includes a depth camera array and a hydrophone array. The sensing module is used to acquire three-dimensional depth images, infrared images, and underwater acoustic signals. The feature extraction and fusion module is used to obtain the feeding status characteristics of fish based on three-dimensional depth images, infrared images and underwater acoustic signals; The decision-making and optimization module is used to construct the feeding status characteristics of the fish into a time-series feature sequence, input it into the time-series prediction model to output the recommended feeding amount, and use a dynamic neural balance solution algorithm to solve the optimal feeding amount based on the recommended feeding amount. The feeding execution module includes a feeding robot, which is used to drive the feeding robot to feed according to the optimal feeding amount and collect the feeding response data of the fish and store it in the experience playback buffer. The incremental learning module is used to incrementally train the time series prediction model based on the experience replay buffer and the elastic weight integration method.

[0058] The depth camera array includes at least three depth cameras, with at least two depth cameras positioned at the bottom of the aquaculture pond and at least one depth camera positioned directly above the aquaculture pond.

[0059] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; only preferred embodiments of the present invention are illustrated. The descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. As long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0060] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the appended claims.

Claims

1. A stereovision and acoustic fusion method for feeding in recirculating aquaculture, characterized in that, Includes the following steps: Step S1: Acquire three-dimensional depth images, infrared images, and underwater acoustic signals of the fish feeding using a depth camera array and a hydrophone array; Step S2: Input the three-dimensional depth image, infrared image and underwater acoustic signal into the cross-modal attention fusion model to output the feeding state features of the fish school; Step S3: Construct the feeding status characteristics of the fish school into a time-series feature sequence, input it into the time-series prediction model, and output the recommended feeding amount; Step S4: Use the dynamic neural balance solution algorithm to solve for the optimal feeding amount based on the recommended feeding amount; Step S5: Drive the feeding robot to feed the fish according to the optimal feeding amount, and collect the feeding response data of the fish and store it in the experience playback buffer; Step S6: Based on the experience replay buffer, the time series prediction model is incrementally trained using the elastic weight integration method.

2. The stereoscopic vision and acoustic fusion method for feeding in recirculating aquaculture according to claim 1, characterized in that, The method further includes a preprocessing step for the three-dimensional depth image, infrared image, and underwater acoustic signal: Step S1.1: Use Zhang Zhengyou calibration method to calibrate the intrinsic parameters of the depth camera array, obtain the intrinsic parameter matrix and distortion coefficient of each camera, and obtain the relative extrinsic parameters through multi-camera joint calibration, and align the coordinate system of each camera to the world coordinate system; Step S1.2: Perform median filtering, morphological operations, and histogram equalization on the infrared image in sequence to obtain a denoised infrared image; Step S1.3: Collect background noise samples from the operating environment of the hydrophone array, and perform adaptive filtering and noise reduction on the underwater acoustic signal based on the background noise samples; Step S1.4: Transform the 3D pixel coordinates of the fish detected by the depth camera array to the world coordinate system using the following formula, and perform clustering and merging to eliminate duplicate detections based on the Euclidean distance threshold: ； In the formula, In three-dimensional world coordinates, For the first Rotation matrix of each camera, For the first The intrinsic parameter matrix of each camera. , For pixel coordinates, For depth value, For the first The translation vector of each camera.

3. The method for feeding recirculating aquaculture systems that integrates stereoscopic vision and acoustics according to claim 1, characterized in that, the fish school feeding state features include fish school depth distribution entropy , average swimming speed , speed variance , average nearest neighbor distance , modified convex hull area , feeding event frequency and mel frequency cepstral coefficients (MFCCs); wherein the fish depth distribution entropy is calculated according to the following equation: ； ； In the formula, The fish school is located in the first The probability of a depth interval For the first Number of fish within each depth range The total number of fish. This represents the total number of depth intervals.

4. The method for feeding recirculating aquaculture systems that integrates stereoscopic vision and acoustics according to claim 1, characterized in that, The attention of the cross-modal attention fusion model The calculation formula is: ； The acoustic gating factor and the visual gating factor are calculated according to the following formulas: ； ； In the formula, Q, K, and V are the query matrix, key matrix, and value matrix obtained by linear projection of the visual features extracted from the 3D depth image and infrared image by the cross-modal attention fusion model, respectively. For acoustic gating factors; ⊙ represents the visual gating factor; ⊙ represents the Hadamard product. The vector dimension of the query matrix; Use the Sigmoid activation function; Frequency of feeding events; Adjust the convex hull area for the fish population; , These are the acoustic learning weight vector and the visual learning weight vector, respectively. , These are the acoustic gating bias vector and the visual gating bias vector, respectively.

5. The method for feeding recirculating aquaculture systems that integrates stereoscopic vision and acoustics according to claim 4, characterized in that, Between steps S2 and S3, there is also an acoustic-visual bidirectional reverse verification coupling step: Step S2.1: Extract the rate of change of fish swarm aggregation in the visual channel and the rate of change of feeding event frequency in the acoustic channel from the cross-modal attention fusion model, respectively; Step S2.2: According to the preset biological prior time delay The frequency change rate of the feeding event in the acoustic channel is time-delayed and shifted, and the time delay is set at a length of [missing information]. The acoustic-visual consistency factor is calculated using the following formula within the sliding window. : ； In the formula, for Momentary acoustic-visual consistency factor For covariance operators, This represents the change in fish aggregation intensity output by the visual channel. This represents the frequency variation of the feeding event output by the acoustic channel. For time step, , Sliding windows Inside and Standard deviation; Step S2.3: Set the consistency threshold ,when The original acoustic gating factor of the cross-modal attention fusion model is maintained at the same time. and original visual gating factor Unchanged; when The acoustic gating factor after inverse correction is updated according to the following formula. and the visual gating factor after inverse correction : ； ； In the formula, The acoustic inverse attenuation coefficient, The visual inverse enhancement coefficient; Step S2.4: Use the inversely corrected acoustic gating factor and visual gating factors Replace the original acoustic gating factor and the original visual gating factor of the cross-modal attention fusion model.

6. The method for feeding in recirculating aquaculture systems that integrates stereoscopic vision and acoustics according to claim 5, characterized in that, The biological prior delay The value range is from 0.3 seconds to 0.8 seconds; the length of the sliding window The value ranges from 3 seconds to 10 seconds; The consistency threshold The value range is from 0.3 to 0.5; the acoustic reverse attenuation coefficient The value range is from 0.2 to 0.4; The visual inverse enhancement coefficient The value range is from 0.1 to 0.

3.

7. The method for feeding in recirculating aquaculture systems that integrates stereoscopic vision and acoustics according to claim 1, characterized in that, The temporal prediction model is an AV-BET model, which includes a cross-modal asynchronous temporal feature extraction layer, a feedforward network layer, and an output prediction layer connected in sequence; the feedforward network layer... The calculation formula is: ； The output prediction layer maps the hidden state of the last time step of the feedforward network layer to the recommended feed amount according to the following formula: ； In the formula, The cross-modal feature matrix is ​​acoustically gated. , These are the fully connected weight matrices for the first and second layers of the feedforward network, respectively. , These are the bias vectors for the first and second layers of the feedforward network, respectively. This represents the hidden state vector at the last time step of the feedforward network layer. To output the weight vector of the prediction layer, For the output bias term of the output prediction layer, This is the recommended feeding amount.

8. The method for feeding in recirculating aquaculture systems that integrates stereoscopic vision and acoustics according to claim 1, characterized in that, The solution formula for the dynamic neural balance solution algorithm is as follows: ； ； ； ； In the formula, To achieve the optimal feeding amount, For the recommended feeding amount, For the feeding accuracy cost function, Let the efficiency cost function be... For the safety cost function of feeding, , , These are the weighting coefficients corresponding to the feeding accuracy cost function, efficiency cost function, and feeding safety cost function, respectively. For the target feeding amount, for At any given time, the fish have consumed their food. For the estimated total feeding amount, For feeding time, Let t be the actual feeding amount output by the decision at time t. For maximum safe feeding amount, This is the minimum safe feeding amount.

9. A recirculating aquaculture feeding system integrating stereoscopic vision and acoustics, used to implement the method according to any one of claims 1 to 8, characterized in that, The system includes: The sensing module includes the depth camera array and the hydrophone array, and is used to acquire three-dimensional depth images, infrared images and underwater acoustic signals. The feature extraction and fusion module is used to obtain the feeding status features of the fish school based on the three-dimensional depth image, the infrared image and the underwater acoustic signal; The decision-making and optimization module is used to construct the feeding status characteristics of the fish group into a time-series feature sequence and input it into the time-series prediction model to output a recommended feeding amount, and to use a dynamic neural balance solution algorithm to solve for the optimal feeding amount based on the recommended feeding amount; The feeding execution module includes a feeding robot, which is used to drive the feeding robot to feed according to the optimal feeding amount, and to collect the feeding response data of the fish and store it in the experience playback buffer. An incremental learning module is used to incrementally train the time series prediction model based on the experience replay buffer using an elastic weight integration method.

10. The recirculating aquaculture feeding system integrating stereoscopic vision and acoustics according to claim 9, characterized in that, The depth camera array includes at least three depth cameras, wherein at least two of the depth cameras are arranged at the bottom of the aquaculture pond and at least one of the depth cameras is arranged directly above the aquaculture pond.

Images