Multi-modal data acquisition and fusion method and device, electronic equipment and medium

Abstract

The invention relates to the technical field of multi-modal data collection and fusion, and discloses a multi-modal data collection and fusion method and device, electronic equipment and a medium, and the method comprises the steps: guaranteeing the precise alignment of acoustic and visual data in a time domain through timestamp synchronization, laying a foundation for feature-level fusion, and improving the accuracy of feature-level fusion; visual signal quality is dynamically evaluated by using a definition detection model, fusion weights of acoustic and visual features are adaptively allocated according to the visual signal quality, and the most reliable data source in the current environment can be utilized to the greatest extent by using the dynamic weight adjustment strategy to obtain fusion features. The method has the beneficial effects that the robustness and the accuracy of multi-modal feature fusion are remarkably improved, so that a higher-quality data basis is provided for subsequent large yellow croaker behavior recognition, and a self-adaptive weighted fusion mechanism based on visual definition is innovatively introduced; and the inherent defect of unreliable sensing of a single mode in an underwater environment is effectively overcome.

Description

Technical Field

[0001] This invention relates to the field of multimodal data acquisition and fusion technology, and in particular to a method, apparatus, electronic device and medium for multimodal data acquisition and fusion. Background Technology

[0002] As an important economic fish species in my country, the refined aquaculture and behavioral monitoring of large yellow croaker are crucial for improving aquaculture efficiency, early warning of diseases, and reducing losses. Currently, the monitoring of underwater fish behavior mainly relies on single visual or acoustic analysis techniques. Visual analysis can intuitively obtain the posture and movement trajectory of the fish, but in the target water area, factors such as low water visibility, changes in light, and algae obstruction can seriously affect image quality, leading to incomplete or even failed feature extraction. Acoustic analysis judges by collecting the sound waves emitted or reflected by the fish, which is less affected by the optical conditions of the water body, but its signal is easily interfered by environmental noise and overlapping sound waves emitted by fish schools, and its accuracy is limited in the recognition of complex behaviors.

[0003] Existing multimodal technologies attempt to combine visual and acoustic data, but they usually use simple feature splicing or fixed weights for fusion, failing to fully consider the dynamic fluctuations in the quality of the two signals in the underwater environment. This rigid fusion method will drag down the performance of the overall recognition model when the reliability of one signal source drops sharply due to environmental interference, and will not achieve true complementarity of the advantages of the two modalities. Summary of the Invention

[0004] Based on this, it is necessary to propose a method, device, electronic equipment and medium for the acquisition and fusion of multimodal data to address the existing problem of multimodal data acquisition and fusion.

[0005] A method for acquiring and fusing multimodal data, the method comprising: Collect raw acoustic and visual signals of a specified large yellow croaker in the target waters; The original acoustic signal and the original visual signal are time-stamped to obtain the synchronized target acoustic signal and target visual signal; Extract the target acoustic features from the target acoustic signal, and extract the target visual features from the target visual signal; The sharpness of the original visual signal is detected by a preset sharpness detection model, and the signal-to-noise ratio of the original acoustic signal is calculated by a preset calculation method. The visual weights of the target visual features and the acoustic weights of the target acoustic features are dynamically calculated using an attention mechanism; wherein the attention mechanism takes the sharpness and the acoustic signal-to-noise ratio as inputs. The target visual features and target acoustic features are weighted and fused according to the visual weight and the acoustic weight to obtain the fused features of the specified large yellow croaker.

[0006] Further, in the steps of extracting the target acoustic features of the target acoustic signal and extracting the target visual features of the target visual signal, the step of extracting the target visual features of the target visual signal includes: The target visual signal is sent to a preset key point detection model, and the pixel coordinates of the key point in the image are output. Based on the pixel coordinates of the key points, the motion trajectory data of the key points are obtained through a tracking algorithm; The target visual features are constructed based on the motion trajectory data.

[0007] Furthermore, the step of constructing the target visual features based on the motion trajectory data includes: Based on the motion trajectory data of the key points, the velocity vector, acceleration vector, and motion direction angle of the key points are calculated using the optical flow method. Calculate the relative distance and relative angle changes between different key points; The velocity vector, acceleration vector, motion direction angle, relative distance, and relative angle change are combined to construct the target visual features.

[0008] Further, the step of synchronizing the original acoustic signal and the original visual signal with timestamps to obtain the synchronized target acoustic signal and target visual signal includes: Obtain the original timestamp information of the original acoustic signal and the original visual signal; wherein, the original timestamp information is the system time of the original acoustic signal and the original visual signal at the time of acquisition; Calculate the time difference between the original acoustic signal and the original visual signal to determine the time deviation information; Based on the time deviation information, the timestamps of the original acoustic signal and the original visual signal are corrected to obtain the synchronized target acoustic signal and target visual signal; wherein the target acoustic signal and the target visual signal have corresponding time references on the same time axis.

[0009] Further, in the steps of extracting the target acoustic features of the target acoustic signal and extracting the target visual features of the target visual signal, the step of extracting the target acoustic features of the target acoustic signal includes: Extract the sound source location features, spectral features, and energy features from the target acoustic signal; Based on the sound source location characteristics, the spectral characteristics, and the energy characteristics, the comprehensive sound signal in the target acoustic signal is identified; An adaptive filtering algorithm is used to filter out environmental noise signals from the composite sound signal to obtain the pure acoustic signal of the specified large yellow croaker. The target acoustic features are obtained by performing wavelet transform on the pure acoustic signal and extracting time-frequency domain features.

[0010] Furthermore, prior to the step of collecting the raw acoustic and raw visual signals of the designated large yellow croaker in the target waters, the method further includes: Obtain the individual swimming range of each large yellow croaker in the target water area; Based on the individual swimming range of each large yellow croaker, the swimming range of all large yellow croakers is spatially coordinated to determine the overall swimming range information covering the individual swimming range of all large yellow croakers; Based on the overall swimming range and the effective coverage of each sound sensor, the number, installation location, and spacing of the sound sensors are determined to collect the raw acoustic signals of the specified large yellow croaker in the target waters through the sound sensors.

[0011] Further, after the step of weighted fusion of the target visual features and the target acoustic features according to the visual weights and the acoustic weights to obtain the fused features of the specified large yellow croaker, the method further includes: The fused features are input into a preset behavior recognition model, and the behavior type of the specified large yellow croaker is output. Based on the identified behavior type, its corresponding behavior level is determined; wherein, the behavior level includes at least a normal level, a warning level, and an alarm level; Based on the determined behavior level, the system coordinates and controls preset equipment in the aquaculture area to perform intervention operations that match the behavior level.

[0012] A multimodal data acquisition and fusion device, the device comprising: The acquisition module is used to acquire the raw acoustic and raw visual signals of a specified large yellow croaker in the target waters; The synchronization module is used to timestamp the original acoustic signal and the original visual signal to obtain the synchronized target acoustic signal and target visual signal. An extraction module is used to extract the target acoustic features of the target acoustic signal and the target visual features of the target visual signal; The detection module is used to detect the sharpness of the original visual signal through a preset sharpness detection model, and to calculate the acoustic signal-to-noise ratio of the original acoustic signal through a preset calculation method. The configuration module is used to dynamically calculate the visual weights of the target visual features and the acoustic weights of the target acoustic features using an attention mechanism; wherein the attention mechanism takes the sharpness and the acoustic signal-to-noise ratio as inputs; The fusion module is used to perform weighted fusion of the target visual features and the target acoustic features according to the visual weights and the acoustic weights to obtain the fused features of the specified large yellow croaker.

[0013] An electronic device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the following steps: Collect raw acoustic and visual signals of a specified large yellow croaker in the target waters; The original acoustic signal and the original visual signal are time-stamped to obtain the synchronized target acoustic signal and target visual signal; Extract the target acoustic features from the target acoustic signal, and extract the target visual features from the target visual signal; The sharpness of the original visual signal is detected by a preset sharpness detection model, and the signal-to-noise ratio of the original acoustic signal is calculated by a preset calculation method. The visual weights of the target visual features and the acoustic weights of the target acoustic features are dynamically calculated using an attention mechanism; wherein the attention mechanism takes the sharpness and the acoustic signal-to-noise ratio as inputs. The target visual features and target acoustic features are weighted and fused according to the visual weight and the acoustic weight to obtain the fused features of the specified large yellow croaker.

[0014] A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the following steps: Collect raw acoustic and visual signals of a specified large yellow croaker in the target waters; The original acoustic signal and the original visual signal are time-stamped to obtain the synchronized target acoustic signal and target visual signal; Extract the target acoustic features from the target acoustic signal, and extract the target visual features from the target visual signal; The sharpness of the original visual signal is detected by a preset sharpness detection model, and the signal-to-noise ratio of the original acoustic signal is calculated by a preset calculation method. The visual weights of the target visual features and the acoustic weights of the target acoustic features are dynamically calculated using an attention mechanism; wherein the attention mechanism takes the sharpness and the acoustic signal-to-noise ratio as inputs. The target visual features and target acoustic features are weighted and fused according to the visual weight and the acoustic weight to obtain the fused features of the specified large yellow croaker.

[0015] The beneficial effects of this invention are as follows: Time stamp synchronization ensures precise alignment of acoustic and visual data in the temporal domain, laying the foundation for feature-level fusion; a sharpness detection model is used to dynamically evaluate the quality of visual signals, and the fusion weights of acoustic and visual features are adaptively allocated accordingly. This dynamic weight adjustment strategy can maximize the use of the most reliable data sources in the current environment, significantly improving the robustness and accuracy of multimodal feature fusion, thereby providing a higher quality data foundation for subsequent large yellow croaker behavior recognition; and by innovatively introducing an adaptive weighted fusion mechanism based on visual sharpness, the inherent defect of unreliability of single-modal perception in underwater environment is effectively overcome. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] in: Figure 1 This is an application environment diagram of a multimodal data acquisition and fusion method in one embodiment; Figure 2 This is a flowchart of a multimodal data acquisition and fusion method in one embodiment; Figure 3 This is a structural block diagram of a multimodal data acquisition and fusion device in one embodiment; Figure 4 This is a structural block diagram of an electronic device in one embodiment. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Figure 1 This is a diagram illustrating the application environment for multimodal data acquisition and fusion in one embodiment. (Refer to...) Figure 1This multimodal data acquisition and fusion method is applied to a multimodal data acquisition and fusion system. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal; the mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server 120 can be a standalone server or a server cluster consisting of multiple servers. The terminal 110 is used to acquire the raw acoustic and visual signals of a specified large yellow croaker in a target water area, and the server 120 is used to generate the fused features of the specified large yellow croaker.

[0020] like Figure 2 As shown in one embodiment, a method for acquiring and fusing multimodal data is provided. This method can be applied to both terminals and servers; this embodiment uses terminal application as an example. The method for acquiring and fusing multimodal data specifically includes the following steps: S1: Collect raw acoustic and visual signals of the specified large yellow croaker in the target waters; S2: Synchronize the original acoustic signal and the original visual signal with timestamps to obtain the synchronized target acoustic signal and target visual signal; S3: Extract the target acoustic features of the target acoustic signal and extract the target visual features of the target visual signal; S4: Detect the sharpness of the original visual signal using a preset sharpness detection model, and calculate the acoustic signal-to-noise ratio of the original acoustic signal using a preset calculation method; S5: Dynamically calculate the visual weights of the target visual features and the acoustic weights of the target acoustic features using an attention mechanism; wherein the attention mechanism takes the sharpness and the acoustic signal-to-noise ratio as inputs; S6: The target visual features and the target acoustic features are weighted and fused according to the visual weight and the acoustic weight to obtain the fused features of the specified large yellow croaker.

[0021] As described in step S1 above, raw acoustic and visual signals of the designated large yellow croaker in the target waters are acquired. This can be done using pre-deployed hardware: sound sensors and cameras. Sound sensors are typically distributed at different locations in the water to comprehensively capture the fish's vocalizations, including bird calls, near-shore currents, and sounds from other fish, providing rich acoustic data. Simultaneously, visual signal acquisition usually relies on a high-resolution underwater camera capable of operating under varying lighting conditions and accurately capturing the target fish's appearance, behavior, and surrounding environment. In one embodiment, a multispectral camera and infrared-assisted illumination can be used to enhance image quality.

[0022] As described in step S2 above, the original acoustic signal and the original visual signal are time-stamped to obtain synchronized target acoustic and visual signals. Since the acoustic and visual signals are acquired by different sensors at different times, they must be aligned using timestamps to achieve accurate data matching. In this stage, the system first acquires the original timestamp information of the acoustic and visual signals, which records the actual time each signal was captured. Then, by calculating the time difference between the original acoustic and visual signals, the system determines what corrections are needed. For example, some sensors may have a brief delay or a pre-set time lag. Based on the calculated time deviation, the system adjusts the timestamps of the two signals to ensure they correspond on the same time axis. After synchronization, accurate target acoustic and visual signals are obtained.

[0023] As described in step S3 above, target acoustic features of the target acoustic signal and target visual features of the target visual signal are extracted. The extraction of target acoustic features involves analyzing features of different frequencies, amplitudes, and durations in the acoustic signal. Typically, this includes the extraction of sound source location features, spectral features, and energy features. During the analysis, signal processing techniques such as Fast Fourier Transform (FFT) may be used to convert the time-domain signal into a frequency-domain signal for more effective acoustic feature analysis. Visual feature extraction focuses on capturing interesting features in the image, such as color, shape, and motion features. In this process, traditional computer vision algorithms, such as edge detection and feature point extraction, can be used, or deep learning models, such as Convolutional Neural Networks (CNNs), can be used to more accurately identify and classify the behavior and posture of the large yellow croaker. These features will provide important basis for subsequent multimodal data fusion and behavior recognition.

[0024] As described in step S4 above, the sharpness of the original visual signal is detected using a preset sharpness detection model, and the signal-to-noise ratio of the original acoustic signal is calculated using a preset calculation method. Typically, captured visual signals may be affected by various factors such as water quality and lighting conditions, leading to image blurring or distortion. Therefore, using a sharpness detection model can provide necessary basis for subsequent feature weighting. The sharpness detection model can employ image processing algorithms or deep learning methods to automatically analyze image quality. In this step, the model first analyzes the original visual signal, evaluating indicators such as image contrast, sharpness, and blurriness. The sharpness detection model can be a ResNet-50-based convolutional neural network using ImageNet pre-trained weights. Generally, higher sharpness indicates richer image resolution and detail, directly affecting the effectiveness of subsequent feature extraction and fusion. The sharpness detection result will output a numerical index representing the current sharpness level of the visual signal, and this reading will be fed back into the system, providing an important basis for subsequent weight adjustment.

[0025] As described in step S5 above, an attention mechanism is used to dynamically calculate the visual weights of the target visual features and the acoustic weights of the target acoustic features; wherein, the attention mechanism takes the sharpness and signal-to-noise ratio as input. The adaptive weight allocation strategy maximizes the use of data sources with high signal quality, thereby improving the accuracy and effectiveness of multimodal data fusion. If the visual signal has high sharpness, the system will assign higher weights to the target visual features to better reflect the target behavior; conversely, if the visual signal has low sharpness, its weight will be reduced, and the weight of the acoustic features will be increased, allowing the system to rely more on acoustic data for decision-making. By setting a sharpness threshold and weight mapping relationship, the system can automatically determine when to trust which signal source more, thus filtering out the most representative information from massive amounts of data. The calculation method for the visual weights is as follows: ;in, For visual weight, The feature vector corresponding to the visual features of the target. Let C be the weight matrix of the target visual features, and C be the sharpness. The preset weights for sharpness have the same dimensions as the weight matrix for visual features. ;in, The weight matrix for the target acoustic features. The feature vector corresponding to the acoustic features. Indicates the signal-to-noise ratio. The preset weights for the acoustic signal-to-noise ratio. The visual weights are used for calculation. To reflect the relative importance of the visual and acoustic weights, the calculated visual and acoustic weights can be normalized. `softmax` is a normalization function.

[0026] As described in step S6 above, the target visual features and target acoustic features are weighted and fused according to the visual weights and acoustic weights to obtain the fused features of the specified large yellow croaker. The final step involves weighted fusion of the target visual features and acoustic features to generate richer and more reliable fused features. By combining the visual and acoustic weights obtained in the previous step, the machine can intelligently sum or weighted average different features to form a fused feature that combines the advantages of both modalities. This fused feature not only includes the behavioral data of the large yellow croaker but also fully integrates information from different signal sources, making the overall data more complete. The weighted fusion process can use a simple linear weighting model or a more complex deep learning model, depending on the needs. The new fused feature will more accurately reflect the actual state and behavior of the target large yellow croaker, enabling subsequent behavior recognition to perform more accurate classification and analysis. This fused feature will also become key information transmitted by the intelligent behavior monitoring system, thereby realizing real-time monitoring and early warning of the behavior of large yellow croaker in aquaculture management. This provides a scientific basis and data support for subsequent aquaculture decisions, greatly promoting the improvement of aquaculture efficiency.

[0027] In one embodiment, the step S3 of extracting the target acoustic features of the target acoustic signal and extracting the target visual features of the target visual signal includes the step of extracting the target visual features of the target visual signal, which further includes: S301: Send the target visual signal to a preset key point detection model and output the pixel coordinates of the key point in the image; S302: Based on the pixel coordinates of the key points, the motion trajectory data of the key points are obtained through a tracking algorithm; S303: Construct the target visual features based on the motion trajectory data.

[0028] As described in step S301 above, the target visual signal is sent to a preset keypoint detection model, which outputs the pixel coordinates of the keypoints in the image. This detection model is typically based on a deep learning convolutional neural network (CNN) designed to identify key feature points of specific objects, such as key positional information of the head, fins, and tail of a fish. Keypoint detection is crucial for reflecting the shape, posture, and motion characteristics of an object. During processing, the model first parses the input visual signal and calculates the probability of each pixel being identified as a keypoint through a multi-layered network structure including feature extraction layers and classification layers. Once the model completes the calculation, it outputs the pixel coordinates of the keypoints, represented in two-dimensional coordinates, typically including the x and y coordinates of each keypoint in the image. This coordinate information will be used as the basis for subsequent steps. Importantly, the model training process requires labeled sample images, enabling the network to learn to accurately identify keypoints under similar conditions. Therefore, this step not only provides the necessary information for subsequent motion analysis but also ensures the accuracy and effectiveness of the data in subsequent processing.

[0029] As described in step S302 above, motion trajectory data of the key points is obtained through a tracking algorithm based on the pixel coordinates of the key points. The key to this process lies in motion tracking between consecutive image frames, ensuring that the system can capture the dynamic changes of each key point in space in real time. Common tracking algorithms include optical flow, Kalman filtering, and deep learning methods. Optical flow is a classic method that determines the direction and speed of motion by calculating the pixel displacement of key points between adjacent frames. Kalman filtering utilizes a prediction and update mechanism, combining historical data and current observations to reduce noise and obtain a smoother trajectory. When outputting motion trajectory data, information such as velocity, acceleration, and direction of motion can be obtained. This data not only reflects the target's behavioral patterns (e.g., swimming, turning, etc.) but also provides rich information support for subsequent visual feature construction. Furthermore, motion trajectory data can reveal the interaction relationships of fish in the water, laying a solid foundation for subsequent behavioral analysis.

[0030] As described in step S303 above, the target visual features are constructed based on the motion trajectory data. The motion trajectory is extracted into comprehensive data information that reflects the behavioral characteristics of the large yellow croaker, providing a basis for analysis and decision-making. First, the speed, acceleration, and direction of movement of key points can be calculated by statistical analysis of the motion trajectory of key points. For example, speed can help determine the swimming state of the fish, whether it is swimming fast, moving slowly, or stationary. At the same time, by comparing the relative positional changes between different key points, possible behavioral patterns of the fish can be inferred, such as attack, avoidance, and social behavior. Next, the motion trajectory can be transformed into a set of comprehensive feature vectors through temporal analysis methods. These feature vectors can better capture the dynamic performance of the fish in different states. These visual features will be integrated with acoustic features into subsequent multimodal data fusion, thereby supporting higher-level behavioral analysis and recognition. Through this method, the system can not only monitor and analyze the behavioral patterns of large yellow croaker in real time, but also respond flexibly under different environmental conditions, improving the scientificity and accuracy of aquaculture management.

[0031] In one embodiment, step S303, which involves constructing the target visual features based on the motion trajectory data, includes: S3031: Based on the motion trajectory data of the key points, the velocity vector, acceleration vector, and motion direction angle of the key points are calculated using the optical flow method; S3032: Calculate the relative distance and relative angle changes between different key points; S3033: Combine the velocity vector, acceleration vector, motion direction angle, relative distance, and relative angle change to construct the target visual features.

[0032] As described in step S3031 above, based on the motion trajectory data of the key points, the optical flow method is used to calculate the velocity vector, acceleration vector, and motion direction angle of the key points. The optical flow method analyzes the motion trajectory of the key points to obtain their dynamic characteristics, including the velocity vector, acceleration vector, and motion direction angle. The optical flow method calculates the velocity and trajectory of an object by analyzing the pixel motion in an image sequence. First, the system matches identical key points in adjacent frames and estimates the velocity by calculating their displacement in the image. The velocity vector can be represented as a binary tuple containing components in the horizontal direction (x-axis) and the vertical direction (y-axis). The acceleration vector is obtained by taking the time difference of the velocity, representing the rate of change of the key point's velocity over time. The motion direction angle is the angle value that measures the direction of movement of the key point and can be calculated using the arctangent function of the first two vectors. These dynamic features provide rich motion information, such as whether the fish is swimming rapidly, changing direction, or accelerating. By extracting this information using the optical flow method, the system can effectively analyze the behavioral state of the large yellow croaker, thus providing a solid foundation for establishing relationships between subsequent visual features and behavior recognition.

[0033] As described in step S3032 above, the relative distances and relative angle changes between different key points are calculated. By analyzing the geometric relationships between key points, an understanding of the overall fish behavior patterns is enhanced. For example, the relative positions between the head, body, and tail can reveal the fish's swimming posture and dynamic interactions. Relative distance refers to the straight-line distance between any two key points, which can be calculated using the standard Euclidean distance formula and matched pixel coordinates. Relative angle changes describe the directional changes between two key points and can be obtained using the arctangent function. By combining this information, the system can construct a geometric feature map representing the relative positions and movement states of different parts of the fish. Simultaneously, the relative distance and angle changes are also reflected in the fish's behavior, providing important clues for subsequent feature integration and behavior analysis. Calculating the relative relationships between key points allows the system to capture group behaviors, such as fish school interactions, chasing, or escape patterns. Integrating these dynamic and geometric features transforms a single point into dynamic behavior analysis, supporting final behavior recognition and decision-making.

[0034] As described in step S3033 above, the velocity vector, acceleration vector, motion direction angle, relative distance, and relative angle change are combined to construct the target visual features. These features include velocity vector, acceleration vector, motion direction angle, relative distance, and relative angle change. Specifically, the velocity vector v = (vx, vy), where vx and vy are the horizontal and vertical velocities, respectively; the acceleration vector a = (ax, ay), where ax and ay are the horizontal and vertical accelerations, respectively. In this stage, these different types of features are combined into a unified feature vector, thereby simplifying subsequent data processing and model input. To achieve effective feature combination, various methods can be used, such as simple concatenation, weighted integration, or more complex function transformations. Typically, the construction of feature vectors follows certain rules to preserve the importance of each feature. For example, during the construction process, the motion direction angle can be given a higher weight because direction changes are crucial in behavioral decisions, while relative angle changes can be used to capture changes in the fish's posture. The final generated target visual feature set will be a multi-dimensional feature vector containing key information to reflect the motion state and behavioral patterns of the specified large yellow croaker. This feature set can not only be used for subsequent multimodal data fusion, but also provides the necessary data foundation for behavior recognition, analysis and prediction.

[0035] In one embodiment, step S2, which timestamps and synchronizes the original acoustic signal and the original visual signal to obtain synchronized target acoustic signal and target visual signal, includes: S201: Obtain the original timestamp information of the original acoustic signal and the original visual signal; wherein, the original timestamp information is the system time of the original acoustic signal and the original visual signal at the time of acquisition; S202: Calculate the time difference between the original acoustic signal and the original visual signal to determine the time deviation information; S203: Based on the time deviation information, the timestamps of the original acoustic signal and the original visual signal are corrected to obtain the synchronized target acoustic signal and target visual signal; wherein the target acoustic signal and the target visual signal have corresponding time references on the same time axis.

[0036] As described in step S201 above, the original timestamp information of the original acoustic signal and the original visual signal is obtained. Each sensor accurately records the system's time information when it receives a signal, typically using a high-precision clock to ensure the accuracy of the timestamps. For acoustic signal acquisition, the system records the time of each acoustic fluctuation signal; for visual signals, the system records the time when each frame of the image is captured. The correctness of obtaining the original timestamp information directly affects the efficiency and accuracy of subsequent steps. If the timestamp recording is inaccurate or there is a synchronization delay, the synthesized data may be misaligned or mismatched during subsequent time difference calculation and synchronization, thereby reducing the accuracy of subsequent analysis and recognition. Therefore, to improve data reliability, the system needs to store the timestamp information using high-precision equipment and technology.

[0037] As described in step S202 above, the time difference between the original acoustic signal and the original visual signal is calculated to determine the time deviation information. Since the acoustic and visual information are captured by different sensors almost simultaneously, their activation and signal processing may have slight delays due to device or environmental factors. The method for calculating the time difference is relatively simple: the system compares the timestamp value of the acoustic signal with the timestamp of the visual signal to obtain the time deviation information. This time difference can be positive or negative, depending on the order of acquisition. When the timestamp of the acoustic signal is earlier than that of the visual signal, the calculated deviation is negative; otherwise, it is positive. It is important to note that the accurate calculation of the time deviation has a significant impact on the quality of subsequent time synchronization data. Inaccurate calculations will result in inaccurate data alignment, further disrupting subsequent analysis and affecting the final behavior recognition. Therefore, when performing this calculation, the system typically implements a fault-tolerant mechanism, dynamically correcting the calculation process based on the forward and subsequent time delays to ensure the accuracy and validity of the time data.

[0038] As described in step S203 above, based on the time deviation information, the timestamps of the original acoustic signal and the original visual signal are corrected to obtain synchronized target acoustic signal and target visual signal. According to the calculated time deviation, the timestamp of each frame's visual or acoustic signal is adjusted accordingly. For example, if the time deviation is positive, it means the timestamp of the visual signal should be moved forward (minus the time difference) to align with the timestamp of the acoustic signal; if the time deviation is negative, the timestamp of the visual signal needs to be moved backward. This correction operation is achieved by iteratively processing all data timestamps, ensuring that the original signals can be correctly matched under the same time reference. After the correction is completed, the system will obtain synchronized target acoustic signal and target visual signal, which will then have a temporal correspondence. At this point, the user can confidently use this data for subsequent feature extraction and data analysis. The synchronized signal will enable the machine learning model to more accurately capture dynamic behaviors, such as the swimming posture of a fish and related acoustic responses, thereby improving the accuracy and effectiveness of subsequent behavior recognition. Time synchronization lays a solid foundation for subsequent data processing, ensuring high quality and reliability of multimodal feature fusion.

[0039] In one embodiment, step S3, which involves extracting the target acoustic features of the target acoustic signal and the target visual features of the target visual signal, includes the following steps: S311: Extract the sound source location features, spectral features, and energy features from the target acoustic signal; S312: Identify the composite sound signal in the target acoustic signal based on the sound source location characteristics, the spectral characteristics, and the energy characteristics; S313: Use an adaptive filtering algorithm to filter out the environmental noise signal in the integrated sound signal to obtain the pure acoustic signal of the specified large yellow croaker; S314: Perform wavelet transform on the pure acoustic signal to extract time-frequency domain features, thereby obtaining the target acoustic features.

[0040] As described in step S311 above, the sound source location features, spectral features, and energy features of the target acoustic signal are extracted. Fourier transform is used to convert the original acoustic signal from the time domain to the frequency domain to analyze the signal's spectral content. Spectral features are crucial for understanding the frequency distribution and main components of sound waves, providing rich information about the frequency content of acoustic events, including peak frequencies, bandwidth, and harmonic components. Sound source location features are determined by the time difference of arrival of the sound signal and the variation in sound pressure level at different sensors. For example, for multiple sensors installed at different locations in the water, the location of the sound source can be estimated based on the received signal strength and time. This technique is very important for capturing the spatial characteristics of the large yellow croaker's vocalizations in water and its activity range. Energy features reflect the relative intensity of the sound signal and can be obtained by calculating the energy (or power) of the sound signal. Specifically, energy normalization can be used to reflect the loudness of the sound signal for subsequent analysis.

[0041] As described in step S312 above, the comprehensive sound signal in the target acoustic signal is identified based on the sound source location features, the spectrum features, and the energy features. Sound event identification in the scene is achieved through machine learning algorithms or pattern recognition techniques. Specifically, the system inputs the extracted feature vectors into a preset classification model, such as a support vector machine (SVM), neural network, or decision tree. The model is trained based on historical data and labeled samples to identify which sounds are characteristic sounds emitted by the large yellow croaker. By integrating multi-dimensional information such as sound source location, spectrum, and energy, the system can more accurately classify and distinguish the sound content from different sound sources in the audio signal, such as the difference between call sounds, reflected sounds, or environmental noise. Effective comprehensive sound signal recognition not only helps to distinguish the behavior patterns of the large yellow croaker but also makes the process of sound and visual information fusion more accurate.

[0042] As described in step S313 above, an adaptive filtering algorithm is used to filter out environmental noise signals from the composite sound signal, resulting in a pure acoustic signal of the specified large yellow croaker. Since the acoustic environment in water is often interfered with by background noise (such as water flow sounds, sounds of other fish, and boat noise), removing these unnecessary noise signals is crucial for subsequent behavioral analysis. Adaptive filtering algorithms are typically based on mixed signal characteristics, updating their filter parameters by comparing the input signal with the desired signal at each moment. For example, the Least Mean Square (LMS) algorithm or the Recursive Least Squares (RLS) algorithm can be used to iteratively adjust the filter coefficients to achieve optimal noise suppression. This process effectively reduces the impact of environmental noise on signal quality, preserves important sound source information, and ensures the clarity and usability of the audio signal. After this filtering process, the pure acoustic signal obtained by the system more realistically reflects the sound characteristics of the large yellow croaker, helping to improve the accuracy and effectiveness of subsequent analysis. It should be noted that after filtering, the signal-to-noise ratio must be increased to above 20dB to ensure the purity of the acoustic signal.

[0043] As described in step S314 above, wavelet transform is performed on the pure acoustic signal to extract time-frequency domain features, thereby obtaining the target acoustic features. The time-frequency domain features extracted through wavelet transform include energy entropy, spectral centroid, and zero crossings. Wavelet transform is a powerful signal analysis tool that can simultaneously provide time and frequency information of a signal, making it suitable for analyzing instantaneously changing signal features, such as underwater acoustic signals. First, the acoustic signal, after adaptive filtering, is input into the wavelet transform algorithm. The algorithm decomposes the signal into multiple frequency bands. The advantage of wavelet transform is that it can flexibly select appropriate wavelet bases according to different time and frequency requirements, thereby obtaining spectral information at different time scales. This method allows the system to identify high-frequency variations and low-frequency trends in the signal, ensuring comprehensive analysis of complex sound structures. The extracted time-frequency domain features can include energy distribution, frequency components, and their rates of change. These features can reflect the dynamic characteristics of the acoustic signal, such as the duration of the large yellow croaker's vocalization, pitch changes, and changes in the sound source location. By constructing time-frequency domain features, the system can ultimately obtain detailed target acoustic features, providing solid data support for subsequent behavior analysis and recognition models.

[0044] In one embodiment, prior to step S1 of acquiring the raw acoustic and raw visual signals of a specified large yellow croaker in the target waters, the method further includes: S001: Obtain the individual swimming range of each large yellow croaker in the target water area; S002: Based on the individual swimming range of each large yellow croaker, spatially coordinate the swimming range of all large yellow croakers to determine the overall swimming range information covering the individual swimming range of all large yellow croakers; S003: Based on the overall swimming range and the effective coverage of each sound sensor, determine the number, installation location, and spacing of the sound sensors to collect the original acoustic signals of the specified large yellow croaker in the target waters through the sound sensors.

[0045] As described in step S001 above, the individual swimming range of each large yellow croaker in the target water area is obtained. First, the system continuously observes the target water area using a high-resolution underwater camera or other monitoring equipment. During monitoring, the system records the specific location and movement trajectory of the large yellow croakers for later analysis. The key to obtaining the individual swimming range lies in identifying and tracking the behavior of the large yellow croakers. During recording, the system uses algorithms to mark the fish's position and automatically calculates its swimming boundaries. For example, using inter-frame difference or optical flow methods, the system can effectively capture the fish's movement state and estimate their activity area. Thus, through continuous observation, the system establishes a trajectory dataset covering the swimming of each fish at different time points, further estimating its movement range.

[0046] As described in step S002 above, based on the individual swimming range of each large yellow croaker, the swimming range of all large yellow croakers is spatially integrated to determine the overall swimming range information covering the individual swimming ranges of all large yellow croakers. This process not only requires accurately summarizing the swimming range data of each fish, but also requires further analysis of this data to ensure that it covers all individuals in the target water area. The obtained swimming range data of each individual is integrated into a unified model, usually using point cloud algorithms or region partitioning methods. By overlapping and connecting the swimming areas of each individual, the system can identify an envelope region, that is, the minimum boundary of the swimming range of all individuals. This enclosed region is the overall swimming range information, which will cover the activity area of ​​all large yellow croakers during the monitoring period.

[0047] As described in step S003 above, based on the overall swimming range and the effective coverage area of ​​each sound sensor, the number, installation location, and spacing of the sound sensors are determined to collect the raw acoustic signals of the specified large yellow croaker in the target waters. That is, the number, installation location, and spacing of the sound sensors are planned based on the previously obtained overall swimming range and the effective coverage area of ​​each sound sensor. In a preferred embodiment, the swimming range can be recalculated and the sensor parameters adjusted every 24 hours to achieve real-time data updates.

[0048] Specifically, based on the distribution and activity areas of large yellow croaker schools in the target waters, acoustic ranging or pre-observation data is used to delineate the main activity and sound emission areas of the large yellow croaker. Combining the characteristics of the aquatic environment, and adhering to the principle of full coverage and no blind spots, the installation locations and spacing of the acoustic sensors are rationally determined to ensure comprehensive acquisition of acoustic signals. Based on the sound emission characteristics of large yellow croaker at different growth stages, and referring to existing acoustic research data or actual test results, the main frequency distribution range of the large yellow croaker's sound source is determined, and acoustic sensors with frequency response ranges covering the main frequency distribution range are selected. The selected acoustic sensors are fixedly installed at the determined installation locations, maintaining their directional or omnidirectional coverage of the large yellow croaker's activity area. After the acoustic sensors are deployed, simulated acoustic signals generated during the large yellow croaker's activity in the water are continuously collected. The weak simulated acoustic signals output by the acoustic sensors are input to a power amplifier for power amplification to enhance signal strength, suppress the attenuation effects of long-distance or low-amplitude signals, and ensure the signal-to-noise ratio for subsequent processing. The amplified analog acoustic signal is input to the analog-to-digital converter (ADC), which samples and quantizes the signal according to the preset sampling rate and quantization precision, converting the continuous analog acoustic signal into a discrete digital signal, thus obtaining digital acoustic signal data that can be processed and analyzed later.

[0049] Optionally, determining the sensor installation location and spacing based on the swimming range of the large yellow croaker colony in the target water area includes: acquiring the individual swimming range of each large yellow croaker in the target water area; spatially coordinating the swimming ranges of all large yellow croakers based on their individual swimming ranges to determine overall swimming range information covering all individual swimming ranges of large yellow croakers; determining the number of sensors and their installation location and spacing based on the overall swimming range and the effective coverage area of ​​each sensor; acquiring the individual growth stage information of each large yellow croaker in the target water area; determining a comprehensive growth stage index based on the individual growth stage information of all large yellow croakers; and matching the corresponding sensor frequency response range based on the comprehensive growth stage index, the relationship between growth stage and sensor frequency change.

[0050] Specifically, the frequency response range of the sound sensor is dynamically determined based on the growth stages of the large yellow croaker. In other words, the body length, weight, and other individual characteristics of the large yellow croaker change at different growth stages, thus affecting its vocalization mechanism and acoustic characteristics. Generally speaking, the larger the body length of the large yellow croaker, the lower its dominant frequency when emitting sound; conversely, smaller large yellow croakers have relatively higher dominant frequencies, showing an approximately inverse relationship. Therefore, in practical applications, the individual growth stage information of each large yellow croaker in the target water area is first acquired, and then their body length data is extracted. Based on the body length information of all large yellow croakers, a comprehensive growth stage index is constructed. Subsequently, by combining the relationship between body length and vocal frequency, the frequency response range of the sound sensor is selected or adjusted to match the overall growth characteristics of the large yellow croaker colony in the current aquaculture environment, achieving more targeted and better acoustic signal acquisition.

[0051] In practice, the individual swimming ranges of each large yellow croaker in the target water area are obtained. The movement trajectories of each large yellow croaker are extracted using acoustic markers, video tracking, or historical monitoring data. Based on the spatial distribution of these trajectory points, the individual swimming boundaries of each large yellow croaker are determined. All individual swimming boundaries of the large yellow croakers are spatially integrated using minimum bounding rectangle, convex hull algorithms, or spatial clustering methods to determine the overall swimming range information that can cover the activity area of ​​all large yellow croakers. Based on the determined overall swimming range, combined with the effective acoustic coverage radius and sensitivity of each sound sensor, the required number of sound sensors is calculated using coverage optimization algorithms or regular layout methods. The specific installation location and spacing of each sensor are planned to ensure that there are no blind spots in the monitoring area within the target water area. The growth stage information of each large yellow croaker in the target water area is obtained, and the corresponding individual body length data is extracted to form a dataset of the body lengths of all large yellow croakers in the current water area. Statistical analysis is performed on all body length data to calculate the average body length, maximum body length, and minimum body length, determining the comprehensive growth stage index of the large yellow croakers in the current water area. Based on the known relationship between growth stages and vocal frequency variations, and considering the approximately inverse relationship between body length and vocal frequency of the large yellow croaker, a mapping table or calculation model between body length and optimal receiving frequency is constructed. According to comprehensive growth stage indicators, the optimal frequency range is matched from the preset sensor frequency response range, or the sensor's operating frequency is dynamically adjusted to adapt to the current vocal characteristics of the large yellow croaker. Using the determined number of sensors, installation locations, installation spacing, and frequency response range as input parameters, the sound sensors are installed, deployed, and configured to effectively acquire acoustic signals from large yellow croakers at different growth stages.

[0052] In some embodiments, sensors may also be deployed based on historical data statistics and a gridded coverage model.

[0053] In one embodiment, after step S6, which involves weighted fusion of the target visual features and the target acoustic features based on the visual weights and the acoustic weights to obtain the fused features of the specified large yellow croaker, the method further includes: S701: Input the fused features into a preset behavior recognition model and output the behavior type of the specified large yellow croaker; S702: Determine the corresponding behavior level based on the identified behavior type; wherein the behavior level includes at least a normal level, a warning level, and an alarm level; S703: Based on the determined behavior level, the preset equipment in the aquaculture water area is linked to perform intervention operations that match the behavior level.

[0054] As described in step S701 above, the fused features are input into a preset behavior recognition model, which outputs the behavior type of the specified large yellow croaker. This behavior recognition model is typically built on deep learning, such as convolutional neural networks (CNNs) or graph neural networks (GNNs). It aims to extract the complex relationships between features and behavior types by learning from a large amount of labeled data. The model's training process usually uses a training dataset with accurate labels, enabling it to recognize potential behavioral patterns in new data. In practice, the fused features are not simply the sum of visual and acoustic features, but a high-dimensional feature representation that reflects the target's behavioral state after integration by deep learning algorithms. After receiving these fused feature vectors, the model processes them through a multi-layer network structure, outputting one or more possible behavior types. The final output can be a discrete category label (e.g., foraging, swimming, resting), a probability distribution, or more complex decision-making criteria. The accuracy and real-time performance of this process directly affect the decision-making and monitoring effectiveness in aquaculture management, ensuring that the system can provide timely and effective behavior monitoring and management adaptation.

[0055] As described in step S702 above, the corresponding behavior level is determined based on the identified behavior type; wherein the behavior level includes at least a normal level, a warning level, and an alarm level. This process converts the identified behavior type into a level index that can be used for decision-making by setting rules or algorithms. The setting of behavior levels is usually based on the actual needs of biological behavioral research and aquaculture management. Specifically, behavior levels can usually be divided into multiple levels, such as "normal level," "warning level," and "alarm level." "Normal level" indicates that the fish are in good condition, their behavior is within the expected range, and no abnormalities are observed; "warning level" may indicate that the fish are exhibiting some potential abnormal behaviors, such as frequent migration, gathering, or avoidance; "alarm level" indicates that the fish may be facing greater risks, such as significant stress response, escape behavior, or deterioration of health.

[0056] As described in step S703 above, based on the determined behavior level, the system coordinates and controls preset equipment in the aquaculture area to perform intervention operations that match the behavior level. If the behavior level identified by the system is "normal level," no additional intervention measures may be required, and the system will maintain the existing aquaculture environment. If it is "warning level," the system may activate certain preset countermeasures, such as insufficient oxygenation, increasing feed, or adjusting water quality. When the behavior level is "alarm level," the system may take more urgent intervention operations, such as activating an alarm, starting the water circulation system to adjust water quality immediately, or changing the water source, to reduce the stress response and mortality risk of fish. Through this coordinated control, the system can not only respond quickly to potential risks but also provide data-driven decision support for aquaculture management, ensuring that effective measures can be taken in a dynamic underwater environment to protect fish growth and aquaculture benefits. This precise monitoring and response greatly improves the scientific nature and effectiveness of aquaculture management.

[0057] Reference Figure 3 The present invention also provides a multimodal data acquisition and fusion device, the device comprising: Acquisition module 902 is used to acquire the raw acoustic and raw visual signals of a specified large yellow croaker in the target waters; Synchronization module 904 is used to timestamp the original acoustic signal and the original visual signal to obtain synchronized target acoustic signal and target visual signal; Extraction module 906 is used to extract target acoustic features of the target acoustic signal and target visual features of the target visual signal; The detection module 908 is used to detect the sharpness of the original visual signal through a preset sharpness detection model, and to calculate the acoustic signal-to-noise ratio of the original acoustic signal through a preset calculation method. Setting module 910 is used to dynamically calculate the visual weights of the target visual features and the acoustic weights of the target acoustic features using an attention mechanism; wherein the attention mechanism takes the sharpness and the acoustic signal-to-noise ratio as inputs; The fusion module 912 is used to perform weighted fusion of the target visual features and the target acoustic features according to the visual weight and the acoustic weight to obtain the fused features of the specified large yellow croaker.

[0058] In one embodiment, the extraction module 906 includes: The pixel coordinate output submodule is used to send the target visual signal to a preset key point detection model and output the pixel coordinates of the key point in the image; The motion trajectory data acquisition submodule is used to acquire the motion trajectory data of the key points based on the pixel coordinates of the key points through a tracking algorithm; A submodule is constructed to build the target visual features based on the motion trajectory data.

[0059] In one embodiment, constructing a submodule includes: The velocity vector calculation unit is used to calculate the velocity vector, acceleration vector, and motion direction angle of the key points based on the motion trajectory data of the key points using the optical flow method. The relative distance calculation unit is used to calculate the relative distance and relative angle changes between different key points; The target visual feature construction unit is used to combine the velocity vector, acceleration vector, motion direction angle, relative distance, and relative angle change to construct the target visual features.

[0060] In one embodiment, the synchronization module 904 includes: The original timestamp information acquisition submodule is used to acquire the original timestamp information of the original acoustic signal and the original visual signal; wherein, the original timestamp information is the system time of the original acoustic signal and the original visual signal at the time of acquisition; The time difference calculation submodule is used to calculate the time difference between the original acoustic signal and the original visual signal, and to determine the time deviation information. The correction submodule is used to correct the timestamps of the original acoustic signal and the original visual signal based on the time deviation information to obtain synchronized target acoustic signal and target visual signal; wherein the target acoustic signal and the target visual signal have corresponding time references on the same time axis.

[0061] In one embodiment, the extraction module 906 includes: An energy feature extraction submodule is used to extract the sound source location features, spectral features, and energy features from the target acoustic signal; A comprehensive sound signal recognition submodule is used to identify the comprehensive sound signal in the target acoustic signal based on the sound source location features, the spectral features, and the energy features. The pure acoustic signal acquisition submodule is used to filter out the environmental noise signal in the comprehensive sound signal using an adaptive filtering algorithm to obtain the pure acoustic signal of the specified large yellow croaker. The target acoustic feature acquisition submodule is used to perform wavelet transform on the pure acoustic signal and extract time-frequency domain features to obtain the target acoustic features.

[0062] In one embodiment, the multimodal data acquisition and fusion device further includes: The individual swimming range acquisition module is used to acquire the individual swimming range of each large yellow croaker in the target water area; The coverage module is used to spatially coordinate the swimming range of all large yellow croakers based on the individual swimming range of each large yellow croaker, and determine the overall swimming range information that covers the individual swimming range of all large yellow croakers. The raw acoustic signal acquisition module is used to determine the number, installation position, and spacing of the sound sensors based on the overall swimming range and the effective coverage of each sound sensor, so as to acquire the raw acoustic signals of the specified large yellow croaker in the target water area through the sound sensors.

[0063] In one embodiment, the multimodal data acquisition and fusion device further includes: The behavior type output module is used to input the fused features into a preset behavior recognition model and output the behavior type of the specified large yellow croaker; The behavior level determination module is used to determine the corresponding behavior level based on the identified behavior type; wherein, the behavior level includes at least a normal level, a warning level, and an alarm level; The control module is used to coordinate and control preset equipment in the aquaculture water area to perform intervention operations that match the determined behavior level.

[0064] Figure 4 An internal structural diagram of an electronic device in one embodiment is shown. This electronic device can specifically be a terminal or a server, and more specifically, a computer device. Figure 4 As shown, the electronic device includes a processor, a memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program. When executed by the processor, this computer program enables the processor to implement a method for acquiring and fusing multimodal data. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to implement the method for acquiring and fusing multimodal data. Those skilled in the art will understand that… Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0065] In one embodiment, an electronic device is provided, including a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the following steps: Collect raw acoustic and visual signals of a specified large yellow croaker in the target waters; The original acoustic signal and the original visual signal are time-stamped to obtain the synchronized target acoustic signal and target visual signal; Extract the target acoustic features from the target acoustic signal, and extract the target visual features from the target visual signal; The sharpness of the original visual signal is detected by a preset sharpness detection model, and the signal-to-noise ratio of the original acoustic signal is calculated by a preset calculation method. The visual weights of the target visual features and the acoustic weights of the target acoustic features are dynamically calculated using an attention mechanism; wherein the attention mechanism takes the sharpness and the acoustic signal-to-noise ratio as inputs. The target visual features and target acoustic features are weighted and fused according to the visual weight and the acoustic weight to obtain the fused features of the specified large yellow croaker.

[0066] Synchronization with timestamps ensures precise alignment of acoustic and visual data in the temporal domain, laying the foundation for feature-level fusion. A sharpness detection model is used to dynamically evaluate the quality of visual signals and adaptively allocate fusion weights for acoustic and visual features accordingly. This dynamic weight adjustment strategy maximizes the use of the most reliable data sources in the current environment, significantly improving the robustness and accuracy of multimodal feature fusion. This provides a higher-quality data foundation for subsequent large yellow croaker behavior recognition. By innovatively introducing an adaptive weighted fusion mechanism based on visual sharpness, the inherent defect of unreliability of single-modal perception in underwater environments is effectively overcome.

[0067] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, causes the processor to perform the following steps: Collect raw acoustic and visual signals of a specified large yellow croaker in the target waters; The original acoustic signal and the original visual signal are time-stamped to obtain the synchronized target acoustic signal and target visual signal; Extract the target acoustic features from the target acoustic signal, and extract the target visual features from the target visual signal; The sharpness of the original visual signal is detected by a preset sharpness detection model, and the signal-to-noise ratio of the original acoustic signal is calculated by a preset calculation method. The visual weights of the target visual features and the acoustic weights of the target acoustic features are dynamically calculated using an attention mechanism; wherein the attention mechanism takes the sharpness and the acoustic signal-to-noise ratio as inputs. The target visual features and target acoustic features are weighted and fused according to the visual weight and the acoustic weight to obtain the fused features of the specified large yellow croaker.

[0068] Synchronization with timestamps ensures precise alignment of acoustic and visual data in the temporal domain, laying the foundation for feature-level fusion. A sharpness detection model is used to dynamically evaluate the quality of visual signals and adaptively allocate fusion weights for acoustic and visual features accordingly. This dynamic weight adjustment strategy maximizes the use of the most reliable data sources in the current environment, significantly improving the robustness and accuracy of multimodal feature fusion. This provides a higher-quality data foundation for subsequent large yellow croaker behavior recognition. By innovatively introducing an adaptive weighted fusion mechanism based on visual sharpness, the inherent defect of unreliability of single-modal perception in underwater environments is effectively overcome.

[0069] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0070] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0071] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for acquiring and fusing multimodal data, characterized in that, The method includes: Collect raw acoustic and visual signals of a specified large yellow croaker in the target waters; The original acoustic signal and the original visual signal are time-stamped to obtain the synchronized target acoustic signal and target visual signal; Extract the target acoustic features from the target acoustic signal, and extract the target visual features from the target visual signal; The sharpness of the original visual signal is detected by a preset sharpness detection model, and the signal-to-noise ratio of the original acoustic signal is calculated by a preset calculation method. The visual weights of the target visual features and the acoustic weights of the target acoustic features are dynamically calculated using an attention mechanism; wherein the attention mechanism takes the sharpness and the acoustic signal-to-noise ratio as inputs. The target visual features and target acoustic features are weighted and fused according to the visual weight and the acoustic weight to obtain the fused features of the specified large yellow croaker.

2. The method for acquiring and fusing multimodal data according to claim 1, characterized in that, The step of extracting the target acoustic features of the target acoustic signal and the target visual features of the target visual signal, wherein the step of extracting the target visual features of the target visual signal includes: The target visual signal is sent to a preset key point detection model, and the pixel coordinates of the key point in the image are output. Based on the pixel coordinates of the key points, the motion trajectory data of the key points are obtained through a tracking algorithm; The target visual features are constructed based on the motion trajectory data.

3. The method for acquiring and fusing multimodal data according to claim 2, characterized in that, The step of constructing the target visual features based on the motion trajectory data includes: Based on the motion trajectory data of the key points, the velocity vector, acceleration vector, and motion direction angle of the key points are calculated using the optical flow method. Calculate the relative distance and relative angle changes between different key points; The velocity vector, acceleration vector, motion direction angle, relative distance, and relative angle change are combined to construct the target visual features.

4. The method for acquiring and fusing multimodal data according to claim 1, characterized in that, The step of synchronizing the original acoustic signal and the original visual signal with timestamps to obtain the synchronized target acoustic signal and target visual signal includes: Obtain the original timestamp information of the original acoustic signal and the original visual signal; wherein, the original timestamp information is the system time of the original acoustic signal and the original visual signal at the time of acquisition; Calculate the time difference between the original acoustic signal and the original visual signal to determine the time deviation information; Based on the time deviation information, the timestamps of the original acoustic signal and the original visual signal are corrected to obtain the synchronized target acoustic signal and target visual signal; wherein the target acoustic signal and the target visual signal have corresponding time references on the same time axis.

5. The method for acquiring and fusing multimodal data according to claim 1, characterized in that, The steps of extracting the target acoustic features of the target acoustic signal and extracting the target visual features of the target visual signal include: Extract the sound source location features, spectral features, and energy features from the target acoustic signal; Based on the sound source location characteristics, the spectral characteristics, and the energy characteristics, the comprehensive sound signal in the target acoustic signal is identified; An adaptive filtering algorithm is used to filter out environmental noise signals from the composite sound signal to obtain the pure acoustic signal of the specified large yellow croaker. The target acoustic features are obtained by performing wavelet transform on the pure acoustic signal and extracting time-frequency domain features.

6. The method for acquiring and fusing multimodal data according to claim 1, characterized in that, Before the step of collecting the raw acoustic and visual signals of the specified large yellow croaker in the target waters, the method further includes: Obtain the individual swimming range of each large yellow croaker in the target water area; Based on the individual swimming range of each large yellow croaker, the swimming range of all large yellow croakers is spatially coordinated to determine the overall swimming range information covering the individual swimming range of all large yellow croakers; Based on the overall swimming range and the effective coverage of each sound sensor, the number, installation location, and spacing of the sound sensors are determined to collect the raw acoustic signals of the specified large yellow croaker in the target waters through the sound sensors.

7. The method for acquiring and fusing multimodal data according to claim 1, characterized in that, After the step of weighted fusing the target visual features and the target acoustic features according to the visual weight and the acoustic weight to obtain the fused features of the specified large yellow croaker, the method further includes: The fused features are input into a preset behavior recognition model, and the behavior type of the specified large yellow croaker is output. Based on the identified behavior type, its corresponding behavior level is determined; wherein, the behavior level includes at least a normal level, a warning level, and an alarm level; Based on the determined behavior level, the system coordinates and controls preset equipment in the aquaculture area to perform intervention operations that match the behavior level.

8. A multimodal data acquisition and fusion device, characterized in that, The device includes: The acquisition module is used to acquire the raw acoustic and raw visual signals of a specified large yellow croaker in the target waters; The synchronization module is used to timestamp the original acoustic signal and the original visual signal to obtain the synchronized target acoustic signal and target visual signal. An extraction module is used to extract the target acoustic features of the target acoustic signal and the target visual features of the target visual signal; The detection module is used to detect the sharpness of the original visual signal through a preset sharpness detection model, and to calculate the acoustic signal-to-noise ratio of the original acoustic signal through a preset calculation method. The configuration module is used to dynamically calculate the visual weights of the target visual features and the acoustic weights of the target acoustic features using an attention mechanism; wherein the attention mechanism takes the sharpness and the acoustic signal-to-noise ratio as inputs; The fusion module is used to perform weighted fusion of the target visual features and the target acoustic features according to the visual weights and the acoustic weights to obtain the fused features of the specified large yellow croaker.

9. A computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, causes the processor to perform the steps of the multimodal data acquisition and fusion method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the multimodal data acquisition and fusion method as described in any one of claims 1 to 7.

Images