Fish behavior identification method based on big data and artificial intelligence

Abstract

The invention discloses a fish behavior identification method based on big data and artificial intelligence. An identification system comprises a control and data processing unit, a water environment data acquisition unit, a weather data acquisition unit and a video acquisition unit. The fish behavior identification method comprises the following steps: S1, data acquisition: acquiring data through a water environment data acquisition unit, a weather data acquisition unit and a video acquisition unit; s2, data preprocessing: preprocessing a key frame in the video data to obtain a binary image; s3, action recognition: recognizing and obtaining individual fish action types and fish school action types through a hybrid neural network model based on the binary image; and S4, selecting a general large model, performing fine tuning to obtain a fish behavior identification large model, and giving out judgment of fish behaviors based on the data after data fusion. According to the scheme, the accuracy of fish behavior recognition can be effectively improved by integrating various different types of data such as the water environment data, the weather data and the fish behavior data.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a fish behavior recognition method based on big data and artificial intelligence. Background Art

[0002] In fish farming, the role of fish behavior recognition is mainly reflected in the following aspects: Farming management: By analyzing the behavior patterns of fish, farmers can more accurately assess the growth, health and activity level of fish, thereby optimizing feeding conditions and management strategies. Disease prevention: Identifying abnormal behavior helps to detect fish diseases in a timely manner, so that preventive measures can be taken to reduce the spread of diseases. Feed feeding: By identifying the feeding behavior of fish, the amount and frequency of feed can be more reasonably controlled to improve feed utilization.

[0003] At present, fish behavior recognition mainly relies on manual observation, which is inefficient and easily affected by subjective judgment. Some researchers have also tried to use rule-based video or image recognition methods, such as patent CN117576616A, a fish swimming behavior warning method, system and device based on deep learning, which discloses a fish swimming behavior warning method based on deep learning. The patent uses a posture estimation model to obtain fish skeleton information, and judges the fish swimming behavior by the position relationship of the positions of each joint point of the skeleton during swimming behavior. This method only involves the recognition and detection of individual behavior, and does not involve the recognition of the overall behavior of a school of fish including multiple individuals. In the recognition of individual behavior, it also relies on a small number of predefined rules to identify fish behavior. This type of method has poor applicability. In terms of data sources, not only relying on image data, but also affecting the accuracy of the final output results. Summary of the invention

[0004] The purpose of the present invention is to provide a fish behavior recognition method based on big data and artificial intelligence to solve the problems existing in the prior art.

[0005] In order to achieve the above-mentioned purpose, the technical scheme of the present invention provides a fish behavior recognition method based on big data and artificial intelligence, and the fish behavior recognition method is implemented by relying on a recognition system; the recognition system includes a control and data processing unit, a water environment data acquisition unit, a weather data acquisition unit, and a video acquisition unit; the fish behavior recognition method includes the following steps: S1, data acquisition, water environment data, weather data, and video data are collected by the water environment data acquisition unit, the weather data acquisition unit, and the video acquisition unit, and the control and data processing unit obtains seasonal data and fish life habit data; S2, data preprocessing, using the inter-frame difference method to extract key frames in the video data, and preprocessing the image data in the key frame data set to obtain a binary image; S3, action recognition, based on the binary image obtained in step S2, the hybrid neural network model is used to identify the individual fish action type and the fish school action type; S4, data fusion, data fusion is performed on the data obtained in steps S1, S2, and S3, a general large model is selected, and the model is fine-tuned using data to obtain a large model for fish behavior recognition, and the large model for fish behavior recognition is used to use natural language processing technology to give a judgment on the fish behavior based on the data after data fusion.

[0006] Furthermore, the water environment data acquisition unit is used to collect relevant data in the water body, including but not limited to water temperature, dissolved oxygen content, ammonia nitrogen content, nitrite content, turbidity, transparency, and pH value; the weather data acquisition unit is used to collect weather data of the water body area, including but not limited to air temperature, air humidity, air pressure, solar illumination, and wind speed; the video acquisition unit is used to collect image or video data of water bodies, individual fish, and schools of fish, including several above-water cameras and underwater cameras, for collecting surface overhead data and underwater head-on data. The surface overhead data includes but is not limited to overhead fish behavior data and water surface texture data, and the underwater head-on data includes but is not limited to head-on fish behavior data.

[0007] Furthermore, the video data preprocessing process in step 2 includes the following steps: step S2a1, using the inter-frame difference method to extract key frames in the video data to form a top-down fish behavior key frame data set, a water surface texture key frame data set, and an underwater head-on fish behavior key frame data set; correlating the top-down fish behavior key frame data set and the underwater head-on fish behavior key frame data set to form a fish behavior key frame data set; step S2a2, preprocessing the image data in the key frame data set obtained in step S2a1 to obtain a binary image.

[0008] Furthermore, the binarization processing of the images in the fish behavior key frame data set in the step S2a2 includes the following steps: step YS11, obtaining the first several frames of images between each segment of key frames, and averaging the pixel values ​​in the image to obtain a background model; step YS12, using the background difference method to extract the foreground of the image in the key frame, subtracting the current frame from the background model to obtain a difference image; thresholding the difference image to separate the fish body and the background; step YS13, removing noise from the thresholded image, using image morphological operations, dilation and erosion, to remove small noise points and fill small holes in the fish body contour, and adjusting to the same size to obtain a fish behavior binary key frame data set.

[0009] Furthermore, the video data preprocessing process also includes identifying the ripple type of the water surface texture key frame data set, and the specific process is as follows: step YS21, contrast enhancement of the image in the key frame, and noise reduction processing through a filter; step YS22, applying the Canny edge detector to identify the edge of the ripple, and then using morphological operations, including but not limited to opening and closing operations, to remove small noise points and fill the broken parts of the ripples; step YS23, converting the image into a binary image; step YS24, judging the ripple type based on the binary image through Hough transform detection or ripple machine learning model identification.

[0010] Furthermore, the data preprocessing of step S2 also includes the following process: based on the specific values ​​of the water environment data and weather data obtained in step S1, they are divided into a comfortable range, an uncomfortable range, and a harmful range according to the fish living habit data and seasonal data to obtain a quantitative influencing factor data set; based on the fish living habit data obtained in step S1, a fish behavior law data set that is associated with the water environment data, weather data, and seasonal data is established.

[0011] Furthermore, in the action recognition of step S3, the action types of individual fish include but are not limited to swimming in a straight line, floating up and down, turning, accelerating, decelerating, hovering, jumping, rolling, swimming sideways, swimming backwards, shaking, tail slapping, swimming speed, and swinging amplitude; the action types of fish schools include but are not limited to group swimming, circular swimming, close gathering, collective jumping, collective rolling, collective tail slapping, and fighting.

[0012] Furthermore, the step S3 action recognition comprises the following steps: step 31, fish action recognition model construction, using a hybrid neural network model, the model integrates a CNN convolutional neural network and an LSTM long short-term memory network; the model input is a fish behavior binary key frame data set, including an underwater horizontal fish behavior key frame binary image and a top-down fish behavior key frame binary image, the model makes a judgment based on the above input, and outputs an individual fish action type data set and a fish school action type data set; step 32, model training, through a training data set, the training data set includes annotated fish action video data, the annotated data is enhanced by random rotation, scaling and flipping, and a fish action recognition model is obtained after the training is completed; step 33, the fish action recognition model is used to identify the fish actions in the fish behavior binary key frame data set to obtain an individual fish action type data set and a fish school action type data set.

[0013] Furthermore, the hybrid neural network model in step 31 is specifically as follows: a CNN integrated convolutional neural network and an LSTM long short-term memory network are set for the binary images of the key frames of the downward-looking fish behavior and the binary images of the key frames of the downward-looking fish behavior, namely, a downward-looking perspective sub-model and a horizontal perspective sub-model; the model input layer of the downward-looking perspective sub-model and the horizontal perspective sub-model is designed to receive a continuous sequence of key frames; the CNN layer is composed of multiple convolutional layers and pooling layers, aiming to extract key visual features in the key frames; the LSTM layer is set after the CNN layer, for processing the features extracted by the CNN and identifying the temporal pattern of the action; an attention fully connected layer is set after the LSTM long short-term memory network of the downward-looking perspective sub-model and the horizontal perspective sub-model to calculate the importance weights of the feature maps of the two perspectives, and the feature maps of the two perspectives are weightedly fused using the calculated attention weights; a fully connected layer is set after the attention fully connected layer to map the fused spatiotemporal features to specific action categories; through the softmax layer, the output of the fully connected layer is converted into a category probability distribution for multi-classification tasks.

[0014] Furthermore, the data fusion of step S4 includes the following steps: step 41, using time series to align the data in the quantitative influencing factor data set, seasonal data, ripple type data set, individual fish action type data set, and fish school action type data set to obtain a multi-source data set, ensuring the synchronization of different data in time; step 42, natural language large model fine-tuning, selecting a general large model and using fish living habit data and fish behavior law data set to fine-tune the model to obtain a fish behavior recognition large model; step 43, using step 42 to obtain the fish behavior recognition large model, using natural language processing technology, based on the characteristic values ​​in the multi-source data set in step 41, to give a judgment on the fish behavior.

[0015] The technical solution of the present invention has the following technical effects: by integrating various types of data such as water environment data, weather data, and fish behavior data, the solution can effectively improve the accuracy of fish behavior recognition. In specific implementation, through automated data collection and processing, the workload of manual observation and recording is reduced, and real-time monitoring and analysis of fish behavior can be achieved, which helps to detect abnormal behavior in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the identification system in the technical solution of the present invention.

[0017] Figure 2 It is a schematic diagram of a specific implementation scheme of the identification system of the present invention.

[0018] Figure 3 It is a schematic diagram of the unmanned boat component in the specific implementation scheme of the identification system of the present invention.

[0019] Figure 4 Schematic diagram of the behavior recognition method of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] As attached Figure 1-2 As shown, the fish behavior recognition method based on big data and artificial intelligence involved in the present invention is implemented by relying on the recognition system.

[0022] As attached Figure 1 As shown, the identification system includes a control and data processing unit, a water environment data acquisition unit, a weather data acquisition unit, and a video acquisition unit, wherein the water environment data acquisition unit is used to collect relevant data in the water body, including but not limited to water temperature, dissolved oxygen content, ammonia nitrogen content, nitrite content, turbidity, transparency, and pH value; the weather data acquisition unit is used to collect weather data in the water body area, including but not limited to air temperature, air humidity, air pressure, solar illumination, and wind speed.

[0023] The video acquisition unit is used to collect images or video data of water bodies, individual fish, and schools of fish, and includes a number of above-water cameras and underwater cameras for collecting surface bird's-eye view data and underwater head-on view data.

[0024] In terms of hardware, depending on the size of the water body, the water environment data acquisition unit, weather data acquisition unit, and video acquisition unit can select separate equipment, such as specific measuring instruments, above-water cameras, and underwater cameras, to collect relevant data. Multiple measuring instruments and cameras can also be integrated as needed to form a fixed or movable device.

[0025] As attached Figure 2 As shown, in one embodiment of the present application, an unmanned ship and a drone are selected as platforms for the identification system, the water environment data acquisition unit, the weather data acquisition unit, and the underwater camera of the video acquisition unit are set on the unmanned ship, and the above-water camera of the video acquisition unit is set on the drone.

[0026] More specifically, in terms of hardware composition, the identification system includes a control and data processing component 1, an unmanned ship component 2, and an unmanned aerial vehicle component 3.

[0027] The control and data processing component 1 includes an electronic computer, an unmanned ship control module, and an unmanned aircraft control module, and is wirelessly connected to the unmanned ship component 2 and the unmanned aircraft component 3, so as to obtain relevant data of the unmanned ship component 2 and the unmanned aircraft component 3 and control their actions. The electronic computer, the unmanned ship control module, and the unmanned aircraft control module are commercially available products.

[0028] The drone assembly 3 includes a drone, a gimbal, and an underwater camera, all of which are commercially available products.

[0029] The unmanned boat component 2 includes an unmanned boat body and a control component 21 , a water environment data collection subcomponent 22 , a weather data collection subcomponent 23 , and an underwater camera subcomponent 24 .

[0030] The water environment data acquisition subassembly 22 includes a water temperature sensor, a dissolved oxygen sensor, an ammonia nitrogen content sensor, a nitrite content sensor, a turbidity sensor, and a pH sensor. The weather data acquisition subassembly 23 includes a temperature and humidity sensor, an air pressure sensor, a solar illumination sensor, and a portable wind speed meter. The relevant sensors and measuring instruments are commercially available sensors, and commercially available water quality multi-parameter data acquisition equipment and portable meteorological instruments can be directly selected. The water environment data acquisition subassembly 22 and the weather data acquisition subassembly 23 are set at the corresponding positions of the unmanned boat body according to the sensor requirements. The specific selection, installation location, and installation structure can be selected from conventional technical knowledge in the field.

[0031] The unmanned boat component 2 also includes a transparency subcomponent 25, which includes a connector 251, a chassis 252, and a photosensor 253. One end of the connector 251 can be connected to the unmanned boat body, and the other end is connected to the chassis 252. The photosensor 253 and the underwater camera subcomponent 24 are arranged on the chassis 252 or on the side of the connector 251 away from the unmanned boat body. When in use, the connector 251 is located underwater, and the chassis 252 and the photosensor 253 are located at a preset depth, wherein the photosensor 253 can measure the light intensity at the depth, and then judge the transparency of the water body. The control and data processing component 1 can obtain the situation of the chassis 252 in the image according to the above-water camera 32, and realize operations such as calibration of the height and distance of the drone and collection of water transparency data.

[0032] As attached Figure 3 As shown, the present application provides a specific structure of a connecting member 251, and the connecting member 251 includes a limiting tube a1, an upper pulley a2, a winding reel a3, a driving motor a4, a first tube section a5, a second tube section a6, a third tube section a7, a fourth tube section a8, and a counterweight block a9.

[0033] The position limiting tube a1 is arranged on the main body of the unmanned boat and penetrates the boat body. The position limiting tube a1, the first tube section a5, the second tube section a6, the third tube section a7, and the fourth tube section a8 are mutually nested and can slide. The upper part of the fourth tube section a8 at the bottom is connected with a steel rope, which passes through the position limiting tube a1, the first tube section a5, the second tube section a6, the third tube section a7, and the fourth tube section a8 and is connected to the winding reel a3 via the upper pulley a2. The driving motor a4 drives the winding reel a3 to realize the winding and releasing of the steel rope. The counterweight a9 is arranged at the end of the fourth tube section a8. The chassis 252 is connected to the lower part of the fourth tube section a8 through the chassis connecting rod a10. The transparent optical disc a11, the photosensitive sensor 253, and the underwater camera subassembly 24 are arranged on the chassis 252. The transparent optical disc is a pattern including a regular pattern and is pasted on the chassis 252.

[0034] A groove a12 is provided at the lower part of the unmanned boat body for accommodating the chassis connecting rod a10 when the first pipe section a5, the second pipe section a6, the third pipe section a7 and the fourth pipe section a8 are folded. The chassis connecting rod a10 extends out of the hull to avoid being blocked by the hull.

[0035] In order to facilitate observation of underwater conditions when the chassis 252 is released, a downward-looking camera 211 is provided at the lower part of the unmanned boat body.

[0036] As attached Figure 4As shown, the present invention involves a fish behavior recognition method based on big data and artificial intelligence. The applicant of the recognition scheme was developed and verified in a laboratory environment. During the model training process, the model was trained using laboratory images and public data sets (such as the side perspective zebrafish image data set in the laboratory environment of Aalborg University in Denmark, etc.), and preliminary generalization verification was carried out in a low-density breeding environment (such as koi breeding) with relatively clear water quality, and it has good applicability.

[0037] The identification method specifically includes the following steps: S1. Data collection: water environment data, weather data and video data are collected through water environment data collection unit, weather data collection unit and video collection unit. The control and data processing unit obtains seasonal data and fish life habit data.

[0038] The specific water environment data include but are not limited to water temperature, dissolved oxygen content, ammonia nitrogen content, nitrite content, turbidity, transparency, and pH value; the weather data include but are not limited to air temperature, air humidity, air pressure, solar illumination, and wind speed; the video data include but are not limited to water surface overhead data and underwater head-on data. The water surface overhead data include but are not limited to overhead fish behavior data and water surface texture data. The underwater head-on data include but are not limited to head-on fish behavior data; the seasonal data include but are not limited to months, seasons, and solar terms; the fish life habit data include but are not limited to body shape data, activity habits, environmental tolerance, feeding habits, breeding habits, and social behavior during the life cycle.

[0039] The water environment data, weather data, and video data obtain specific values ​​through sensors or cameras, and the seasonal data is directly obtained through the control and data processing unit over the same network; the fish life habit data is obtained by analyzing and studying fish-related literature using a general large model, and can also be manually input.

[0040] S2. Data preprocessing The specific values ​​of the water environment data and weather data obtained in step S1 are divided into a comfortable range, an uncomfortable range, and a harmful range according to the fish life habit data and seasonal data to obtain a quantitative influencing factor data set.

[0041] Taking water temperature as an example, the suitable water temperature for juvenile perch is usually between 24℃ and 28℃. When the water temperature is below 5℃, perch will stop eating or eat only once every few days. If the water temperature exceeds 32℃, perch's feeding will be affected. Therefore, 24℃-28℃ can be set as a comfortable range, 5℃-24℃ and 28℃-32℃ as an uncomfortable range, and below 5℃ and above 32℃ as a harmful range. The above preprocessing of water environment data and weather data effectively reduces the complexity of the data, facilitating subsequent analysis.

[0042] According to the fish life habit data obtained in step S1, a fish behavior law data set is established that associates water environment data, weather data, and seasonal data. Taking sea bass as an example, adult sea bass often exhibit post-breeding behavior after June, often exhibit clustering behavior in low pressure, and often exhibit stress behavior in high temperature.

[0043] The video data preprocessing process obtained in step S1 is as follows: Step S2a1, using the inter-frame difference method to extract key frames in the video data, forming a top-down fish behavior key frame dataset, a water surface texture key frame dataset, and an underwater head-on fish behavior key frame dataset; further, the top-down fish behavior key frame dataset and the underwater head-on fish behavior key frame dataset are correlated to form a fish behavior key frame dataset, specifically obtaining the underwater head-on fish behavior key frames and the top-down fish behavior key frames corresponding to each key frame in the top-down fish behavior key frame dataset and the underwater head-on fish behavior key frame dataset, and the fish behavior key frame dataset includes video data above and below the water at the same time.

[0044] Step S2a2: pre-process the image data in the key frame data set obtained in step S2a1 to obtain a binary image.

[0045] The preprocessing in one embodiment of the present application is to perform binarization processing on the images in the key frame data set.

[0046] The preprocessing process in another embodiment of the present application includes the following steps: The processing process for the fish behavior key frame dataset is as follows: Step YS11, obtain the first several frames (such as 3 frames) between each key frame, average the pixel values ​​in the image to obtain the background model; Step YS12, use the background difference method to extract the foreground of the image in the key frame, subtract the current frame from the background model to obtain the difference image; threshold the difference image to separate the fish body and the background; Step YS13, remove noise from the thresholded image. Use image morphological operations, dilation and erosion to remove small noise points and fill small holes in the fish body contour, and adjust to the same size to obtain the fish behavior binary key frame dataset.

[0047] The processing process for the water surface texture key frame dataset is as follows: Step YS21, enhance the contrast of the image in the key frame and perform noise reduction processing through a filter; Step YS22, apply the Canny edge detector to identify the edge of the ripple, and then use morphological operations, including but not limited to opening and closing operations, to remove small noise points and fill the broken parts of the ripple; Step YS23, convert the image into a binary image; Step YS24, obtain a ripple type data set based on the binary image.

[0048] In step YS24, the ripple type judgment can be performed by using Hough Transform to detect whether the ripple is a circular ripple or a broken line ripple, and further extract the ripple diffusion speed data.

[0049] The ripple type judgment in step YS24 can also be performed through a ripple machine learning model, and the model is trained through training data to distinguish different ripple types. Ripple recognition is roughly equivalent to simple shape recognition, and its training method is consistent with general machine learning. In one embodiment of the present applicant, a support vector machine is selected as a ripple machine learning model, and the model is trained through a training data set, and a ripple machine learning model is obtained through testing and parameter adjustment.

[0050] S3, action recognition, obtaining individual fish action types and fish school action types based on the fish behavior binary key frame data set obtained in step S2.

[0051] Individual fish movement types include, but are not limited to, straight-line swimming, up-and-down floating, turning, accelerating, decelerating, hovering, jumping, rolling, sideways swimming (fish swimming sideways), backward swimming, shaking (quickly shaking the body), tail slapping, swimming speed, and swing amplitude.

[0052] Types of schooling movements include, but are not limited to, group swimming (all or most of the fish in a school swim at a similar speed and direction), circular swimming (the school forms a circular or spiral swimming pattern), tight bunching (individuals in the school are closely grouped together), collective jumping, collective tumbling, collective tail slapping, and fighting.

[0053] This application uses a hybrid neural network model to judge the action type of individual fish and the action type of fish school, which specifically includes the following process: Step 31: fish action recognition model construction. This application adopts a hybrid neural network model that integrates convolutional neural network (CNN) and long short-term memory network (LSTM). The CNN part is responsible for extracting spatial features from video key frames, while the LSTM part is responsible for processing time series data to capture the dynamic changes of actions.

[0054] The model input is a fish behavior binary key frame dataset, including underwater horizontal fish behavior key frame binary images and top-down fish behavior key frame binary images. The model makes judgments based on the above inputs and outputs individual fish action type datasets and fish school action type datasets.

[0055] For the binary images of the key frames of fish behavior when looking horizontally and the binary images of the key frames of fish behavior when looking down, an integrated convolutional neural network (CNN) and a long short-term memory network (LSTM) network are set up respectively, namely, a downward viewing angle sub-model and a horizontal viewing angle sub-model.

[0056] The model input layers of the top-view sub-model and the horizontal-view sub-model are designed to receive continuous keyframe sequences; the CNN layer consists of multiple convolutional layers and pooling layers, aiming to extract key visual features in the keyframes; the LSTM layer is set after the CNN layer to process the features extracted by the CNN and identify the temporal patterns of the actions; then a fully connected layer is set to map the learned spatiotemporal features to specific action categories.

[0057] The specific structure of the model is as follows: The input layer receives a continuous sequence of key frames.

[0058] The convolutional neural network (CNN) includes convolution layer, pooling layer, and normalization layer. The specific structure is as follows: The first convolutional layer is responsible for extracting preliminary spatial features from the input key frame sequence. It uses filters to slide on the image to detect basic features such as edges, colors, and textures.

[0059] Activation function (ReLU): Immediately following the first convolutional layer, nonlinearity is introduced, allowing the model to learn and represent more complex features.

[0060] The first pooling layer: reduces the spatial size of the feature map, reduces the number of parameters and the amount of computation, and extracts more important features.

[0061] The second convolutional layer: further extracts more advanced features based on the feature map extracted by the first convolutional layer.

[0062] Deeper filters can be used to combine low-level features to form more complex patterns.

[0063] The second pooling layer: reduces the size of the feature map again to further extract key features.

[0064] Normalization layer: Batch normalization is used to speed up the training process and improve the generalization ability of the model.

[0065] The long short-term memory network (LSTM) is set after the CNN layer to process the features extracted by the CNN and identify the temporal pattern of the action. The LSTM layer is capable of processing time series data and capturing the dynamic changes of the action. In one embodiment of the applicant, two LSTM layers are provided to increase the depth and learning ability of the model. Specifically, the first layer LSTM receives the feature map sequence extracted by the CNN and processes the time series data. Capture dynamic changes in time, such as the start, progress and end of the action. The second layer LSTM: further processes the output of the first layer LSTM to deepen the model's understanding of the time series. More complex time dependencies can be captured, such as complex patterns and sequences of actions.

[0066] An attention fully connected layer is set after the long short-term memory network (LSTM) of the top-view sub-model and the horizontal view sub-model to calculate the importance weights of the feature maps of the two views, and the calculated attention weights are used to perform weighted fusion of the feature maps of the two views.

[0067] Fully connected layer: A fully connected layer is set after the attention fully connected layer to map the fused spatiotemporal features to specific action categories. This layer usually includes a densely connected neural network layer.

[0068] Output layer: The output of the fully connected layer is converted into a category probability distribution through the softmax layer for multi-classification tasks. The softmax layer converts the real number output of the fully connected layer into a probability distribution. The softmax function normalizes the output of each category so that the sum of the output probabilities of all categories is 1. Specifically, each unit of the softmax layer corresponds to a category, and the output of the unit represents the probability of belonging to that category.

[0069] The calculation formula of the softmax function is as follows, where z is the output vector of the fully connected layer, σ(z) i is the output of the ith category of the softmax layer, e zi is an exponential function, and the denominator is the sum of the results of applying the exponential function to all category outputs. The output of the softmax layer is the model's confidence in each category, and the category with the highest probability is considered the model's prediction result.

[0070]

[0071] Step 32, model training, through the training data set, the training data set includes the fish action video data that has been labeled. In order to enhance the generalization ability of the model, the labeled data is enhanced by random rotation, scaling and flipping. In terms of the loss function, the cross entropy loss function in one embodiment of the applicant selects the Adam optimizer during the training process, which can automatically adjust the learning rate to ensure the efficiency and stability of parameter updates. During the training process, the model is evaluated using the validation set to monitor the model performance and overfitting. According to the performance of the validation set, the hyperparameters of the model are adjusted, including the learning rate, batch size and the number of LSTM units. After the training is completed, a fish action recognition model is obtained.

[0072] Step 33: Use the fish action recognition model to recognize the fish actions in the fish behavior binary key frame data set to obtain an individual fish action type data set and a fish school action type data set.

[0073] S4. Data Fusion Data fusion is performed on the quantitative influencing factor data set, seasonal data, fish life habit data, fish behavior law data set, ripple type data set, individual fish action type data set, and fish school action type data set obtained in steps 1, 2, and 3. The specific process includes the following: Step 41: Use time series to align the data in the quantitative influencing factor dataset, seasonal data, ripple type dataset, individual fish action type dataset, and fish school action type dataset to obtain a multi-source dataset, ensuring the synchronization of different data in time. The data in the quantitative influencing factor dataset and seasonal data are often relatively stable, such as the quarter and month in the seasonal data, and the values ​​corresponding to the time series are directly obtained.

[0074] Step 42: Fine-tune the natural language model. Select a general large model, such as BERT, GPT or Transformer, and use fish living habits data and fish behavior law data sets to fine-tune the model to obtain a large model for fish behavior recognition to adapt to the fish behavior recognition task.

[0075] Step 43, using step 42 to obtain a large model for fish behavior recognition, using natural language processing technology, based on the feature values ​​in the multi-source data set in step 41, gives a judgment on the fish behavior and provides processing suggestions based on the judgment.

[0076] It should be noted that, unless otherwise clearly specified and limited, the terms such as "install", "connect", "connect", "fix", "set" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

Claims

1. A fish behavior recognition method based on big data and artificial intelligence, wherein the fish behavior recognition method is implemented by relying on a recognition system; characterized in that: The identification system includes a control and data processing unit, a water environment data acquisition unit, a weather data acquisition unit, and a video acquisition unit; the fish behavior identification method includes the following steps: S1, data collection, water environment data, weather data, video data are collected through the water environment data collection unit, weather data collection unit, and video collection unit, and the control and data processing unit obtains seasonal data and fish life habit data; S2, data preprocessing, using the inter-frame difference method to extract key frames in the video data, and preprocessing the image data in the key frame data set to obtain a binary image; S3, action recognition, based on the binary image obtained in step S2, the action type of individual fish and the action type of fish school are obtained by using a hybrid neural network model; S4, data fusion, data fusion is performed on the data obtained in steps S1, S2, and S3, a general large model is selected, and the model is fine-tuned using the data to obtain a large model for fish behavior recognition. The large model for fish behavior recognition is used to utilize natural language processing technology to make judgments on fish behavior based on the fused data.

2. The fish behavior identification method according to claim 1, characterized in that: The water environment data acquisition unit is used to collect relevant data in the water body, including but not limited to water temperature, dissolved oxygen content, ammonia nitrogen content, nitrite content, turbidity, transparency, and pH value; the weather data acquisition unit is used to collect weather data in the water area, including but not limited to air temperature, air humidity, air pressure, solar illumination, and wind speed; the video acquisition unit is used to collect image or video data of water bodies, individual fish, and schools of fish, including a number of above-water cameras and underwater cameras, which are used to collect overhead data of the water surface and underwater head-on data. The overhead data of the water surface includes but is not limited to overhead fish behavior data and water surface texture data, and the underwater head-on data includes but is not limited to head-on fish behavior data.

3. The fish behavior identification method according to claim 2, characterized in that: The video data preprocessing process in step 2 includes the following steps: Step S2a1, using the inter-frame difference method to extract key frames in the video data, forming a top-view fish behavior key frame dataset, a water surface texture key frame dataset, and an underwater head-on fish behavior key frame dataset; correlating the top-view fish behavior key frame dataset and the underwater head-on fish behavior key frame dataset to form a fish behavior key frame dataset; Step S2a2: pre-process the image data in the key frame data set obtained in step S2a1 to obtain a binary image.

4. The fish behavior identification method according to claim 3, characterized in that: The binarization processing of the images in the fish behavior key frame data set in the step S2a2 includes the following steps: step YS11, obtaining the first several frames of images between each key frame segment, averaging the pixel values ​​in the image to obtain a background model; step YS12, using the background difference method to extract the foreground of the image in the key frame, subtracting the current frame from the background model to obtain a difference image; thresholding the difference image to separate the fish body and the background; step YS13, removing noise from the thresholded image, using image morphological operations, dilation and erosion, removing small noise points and filling small holes in the fish body contour, and adjusting to the same size to obtain a fish behavior binary key frame data set.

5. The fish behavior identification method according to claim 3, characterized in that: The video data preprocessing process also includes identifying the ripple type of the water surface texture key frame data set, and the specific process is as follows: step YS21, contrast enhancement of the image in the key frame, and noise reduction processing through a filter; step YS22, applying the Canny edge detector to identify the edge of the ripple, and then using morphological operations, including but not limited to opening and closing operations, to remove small noise points and fill the broken parts of the ripples; step YS23, converting the image into a binary image; step YS24, based on the binary image, using Hough transform detection or ripple machine learning model identification to determine the ripple type.

6. The fish behavior identification method according to claim 1, characterized in that: The data preprocessing of step S2 also includes the following process: based on the specific values ​​of the water environment data and weather data obtained in step S1, the specific values ​​are divided into a comfortable range, an uncomfortable range, and a harmful range according to the fish living habit data and seasonal data to obtain a quantitative influencing factor data set; based on the fish living habit data obtained in step S1, a fish behavior law data set associated with the water environment data, weather data, and seasonal data is established.

7. The fish behavior identification method according to claim 1, characterized in that: In the step S3 action recognition, the action types of individual fish include but are not limited to straight-line swimming, up and down floating, turning, acceleration, deceleration, hovering, jumping, rolling, sideways swimming, backward swimming, shaking, tail slapping, swimming speed, and swing amplitude; the action types of fish schools include but are not limited to group swimming, circular swimming, close gathering, collective jumping, collective rolling, collective tail slapping, and fighting.

8. The method for identifying fish behavior according to any one of claims 1 or 7, characterized in that: The step S3 action recognition comprises the following steps: Step 31, constructing a fish action recognition model, using a hybrid neural network model, the model integrates a CNN convolutional neural network and an LSTM long short-term memory network; the model input is a fish behavior binary key frame data set, including an underwater horizontal fish behavior key frame binary image and a top-down fish behavior key frame binary image. The model makes judgments based on the above inputs and outputs an individual fish action type data set and a fish school action type data set; Step 32, model training, using a training data set, the training data set includes annotated fish action video data, and the annotated data is enhanced by random rotation, scaling and flipping, and a fish action recognition model is obtained after the training is completed; Step 33: Use the fish action recognition model to recognize the fish actions in the fish behavior binary key frame data set to obtain an individual fish action type data set and a fish school action type data set.

9. The fish behavior identification method according to claim 8, characterized in that: The hybrid neural network model in step 31 is specifically as follows: a CNN integrated convolutional neural network and an LSTM long short-term memory network are respectively set for the bottom horizontal fish behavior key frame binary image and the top view fish behavior key frame binary image, that is, the top view sub-model and the horizontal view sub-model; the model input layer of the top view sub-model and the horizontal view sub-model is designed to receive a continuous key frame sequence; the CNN layer is composed of multiple convolutional layers and pooling layers, and is intended to extract key visual features in the key frames; The LSTM layer is set after the CNN layer to process the features extracted by the CNN and identify the temporal pattern of the action. The attention fully connected layer is set after the LSTM long short-term memory network of the top-view sub-model and the horizontal view sub-model to calculate the importance weights of the feature maps of the two views, and the calculated attention weights are used to perform weighted fusion on the feature maps of the two views. A fully connected layer is set after the attention fully connected layer to map the fused spatiotemporal features to specific action categories; through the softmax layer, the output of the fully connected layer is converted into category probability distribution for multi-classification tasks.

10. The method for identifying fish behavior according to claim 6, characterized in that: The step S4 data fusion comprises the following steps: Step 41, using time series to align the data in the quantitative influencing factor dataset, the seasonal data, the ripple type dataset, the individual fish action type dataset, and the fish school action type dataset to obtain a multi-source dataset, to ensure the synchronization of different data in time; Step 42, fine-tuning the natural language large model, selecting a general large model and using the fish living habits data and the fish behavior law data set to fine-tune the model to obtain a fish behavior recognition large model; Step 43: adopt step 42 to obtain a large model for fish behavior recognition, and use natural language processing technology to make a judgment on the fish behavior based on the feature values ​​in the multi-source data set in step 41.

Images