Fish biomass estimation method and system based on deep learning multi-modal fusion model
Through deep learning multimodal fusion model, combined with image, audio and water quality data, the problem of difficult and poor accuracy of fish biomass estimation is solved, efficient and accurate fish biomass detection is achieved, and manual detection costs are reduced.
Patent Information
- Application Number
- CN202411033778.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2025-01-07
AI Technical Summary
The prior art is difficult to efficiently estimate fish biomass, which leads to risks in aquaculture insurance, and the cost of manual testing is high and error-prone.
Using a multimodal fusion model based on deep learning, the fusion feature extraction and the use of convolutional neural network, logistic regression model and cross-modal attention mechanism by obtaining image, audio and water quality data in the water, combined with feed feeding data, and using convolutional neural network, logistic regression model and cross-modal attention mechanism, the fusion feature extraction and Dynamic adaptive decision-making to predict fish biomass.
It improves the accuracy and efficiency of fish biomass estimation, reduces the cost of manual detection, and can achieve accurate identification when there are many fish and overlapping occlusions.
Smart Images
Figure CN119271987A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of aquaculture detection technology, and in particular to a fish biomass estimation method and system based on a deep learning multimodal fusion model. Background Art
[0002] As the scale of aquaculture in my country continues to expand, the demand for aquaculture insurance is gradually emerging. Fish biomass is one of the most valuable reference data in the process of fish farming. Fish biomass refers to the total weight of fish in a specific water area. The relevant data of fish biomass can help insurance institutions understand the current growth of fish, and it is of great reference significance for clarifying the breeding density and breeding cycle as well as the growth rate and health status of fish. At the same time, through long-term monitoring and analysis of biomass, insurance personnel can predict breeding trends and assess breeding risks.
[0003] However, most existing studies use fish size, shape, color and other appearance attributes as parameters, and rarely establish a relationship between fish feeding conditions, feeding amount and biomass. However, there is a certain relationship between fish feeding behavior, feeding amount and fish biomass. At present, the inability to efficiently estimate farmed biomass and the difficulty in assessing farming risks have made aquaculture insurance face great risks, making insurance institutions dare not easily get involved in aquaculture-related businesses. At the same time, aquaculture is now mainly counted manually, and manual detection of the number and type of aquatic products is difficult and error-prone, increasing labor costs. Summary of the invention
[0004] In view of the above existing problems, the present invention is proposed.
[0005] Therefore, the present invention provides a fish biomass estimation method and system based on a deep learning multimodal fusion model to solve the current problems of high difficulty, poor accuracy and high labor cost in estimating fish biomass in aquaculture.
[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0007] In a first aspect, the present invention provides a fish biomass estimation method based on a deep learning multimodal fusion model, comprising:
[0008] Obtain images, audio and water quality data in the water area during each feeding period and perform preprocessing;
[0009] The preprocessed data are respectively input into the multimodal fusion model for corresponding feature extraction to obtain the corresponding feature vectors of image, audio and water quality;
[0010] After each round of feeding, obtain the feed feeding amount data;
[0011] The preprocessed feeding amount data is input into the logistic regression model in the multimodal fusion model for feature extraction to obtain the feeding amount feature vector;
[0012] Fusing the image, the audio, and feature vectors corresponding to the water quality, wherein the fusion is spliced through the channel dimension to obtain a first aggregate feature map;
[0013] Predicting the next round of feed delivery based on the first aggregated feature graph;
[0014] Fusing the feature vectors corresponding to the image, audio, water quality, and feeding amount, wherein the fusion is spliced through the channel dimension to obtain a second aggregated feature map;
[0015] The current fish biomass is estimated based on the second aggregate characteristic graph.
[0016] As a preferred solution of the fish biomass estimation method based on the deep learning multimodal fusion model described in the present invention, wherein: the image data includes: the size of the splashes of the fish school and the density of the fish school; the audio data includes: the splashing sound, the collision sound of the fish school and the swimming sound of the fish school;
[0017] The water quality data include: dissolved oxygen value, pH and temperature.
[0018] As a preferred solution of the fish biomass estimation method based on the deep learning multimodal fusion model described in the present invention, the pre-processed data are respectively input into the multimodal fusion model for corresponding feature extraction to obtain the image feature vector, including:
[0019] The preprocessed image data is input into the first convolutional neural network of the parallel processing branch in the multimodal fusion model. The convolution accelerator of the first convolutional neural network extracts the connection and spatial information and expresses the image-related features. Its activation layer introduces nonlinear features into the neural network through the activation function. The pooling layer reduces the dimension of the features and outputs the image feature vector.
[0020] As a preferred solution of the fish biomass estimation method based on the deep learning multimodal fusion model described in the present invention, the pre-processed data are respectively input into the multimodal fusion model for corresponding feature extraction to obtain the audio feature vector, including:
[0021] The preprocessed audio data is input into the second convolutional neural network of the parallel processing branch in the multimodal fusion model. The convolution accelerator of the second convolutional neural network extracts connection and spatial information and expresses audio-related features. Its activation layer introduces nonlinear features into the neural network through the activation function. The pooling layer reduces the dimension of the features and outputs the audio feature vector.
[0022] As a preferred solution of the fish biomass estimation method based on the deep learning multimodal fusion model described in the present invention, the pre-processed data are respectively input into the multimodal fusion model for corresponding feature extraction to obtain the water quality feature vector, including:
[0023] The preprocessed water quality parameter data is input into the text encoder of the parallel processing branch in the multimodal fusion model. The text encoder converts the water quality parameter text into a feature vector using a word embedding model and outputs a water quality feature vector.
[0024] As a preferred solution of the fish biomass estimation method based on the deep learning multimodal fusion model described in the present invention, the pre-processed feeding amount data is input into the logistic regression model in the multimodal fusion model for feature extraction to obtain the feeding amount feature vector, including:
[0025] The preprocessed feeding amount data is input into the logistic regression model of the parallel processing branch in the multimodal fusion model, converted into a feature vector through function calculation, and the feeding amount feature vector is output.
[0026] As a preferred solution of the fish biomass estimation method based on the deep learning multimodal fusion model described in the present invention, wherein: the multimodal fusion model adopts a cross-modal attention mechanism and a deep residual network model to introduce residual features as exogenous variables into the acoustic modality and the visual modality;
[0027] The basic modality features are adjusted frame by frame through the frame-level residual learning function. For each time step in each modality feature, the corresponding residual information is calculated through the attention mechanism.
[0028] Introducing dynamic adaptive decision fusion, which calculates the dynamic weight of features by reintegrating acoustic and visual output features from the attention mechanism and combining the time dimension average;
[0029] Connect the fusion features of the acoustic and visual modes and combine them with dynamic weights to obtain a first fusion result;
[0030] The first fusion result is fused again with the text feature vector through the dynamic adaptive decision to obtain a first aggregated feature map;
[0031] The first aggregate feature graph and the feeding amount feature vector are fused again through the dynamic adaptive decision to obtain a second aggregate feature graph.
[0032] In a second aspect, the present invention provides a fish biomass estimation system based on a deep learning multimodal fusion model, comprising: a first acquisition module, for acquiring images, audio and water quality data in a water area within each feeding period, and performing preprocessing;
[0033] The first control module inputs the preprocessed data into the multimodal fusion model for corresponding feature extraction to obtain feature vectors corresponding to the image, audio and water quality;
[0034] The second acquisition module acquires the feed feeding amount data after each feeding round;
[0035] The second control module inputs the preprocessed feeding amount data into the logistic regression model in the multimodal fusion model for feature extraction to obtain a feeding amount feature vector;
[0036] A first fusion module is used to fuse the image, the audio and the feature vectors corresponding to the water quality, wherein the fusion is spliced through the channel dimension to obtain a first aggregate feature map;
[0037] A first prediction module, used for predicting the next round of feed delivery amount based on the first aggregated feature graph;
[0038] A second fusion module is used to fuse the feature vectors corresponding to the image, audio, water quality and feeding amount, and the fusion is spliced through the channel dimension to obtain a second aggregated feature map;
[0039] The second prediction module is used to estimate the current fish biomass based on the second aggregation characteristic graph.
[0040] In a third aspect, the present invention provides a computing device, comprising:
[0041] Memory and processor;
[0042] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the fish biomass estimation method based on the deep learning multimodal fusion model are implemented.
[0043] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the fish biomass estimation method based on the deep learning multimodal fusion model.
[0044] Compared with the prior art, the present invention has the following beneficial effects: the present invention collects and combines different types of modal data for deep learning, uses convolutional neural networks to recognize images and audios, uses a logistic regression model to recognize feed amounts, and uses a text encoder to recognize water quality information. Compared with traditional recognition methods, the recognition accuracy is higher and the degree of intelligence is higher. Even when there are a large number of fish and occlusion and overlap occur, the recognition is still accurate and the recognition efficiency is higher. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0046] Figure 1 This is a schematic diagram of the overall process logic of a fish biomass estimation method based on a deep learning multimodal fusion model according to an embodiment of the present invention;
[0047] Figure 2 A schematic diagram of the overall process framework structure of a fish biomass estimation method based on a deep learning multimodal fusion model according to an embodiment of the present invention;
[0048] Figure 3 A schematic diagram of a fish pond system in a fish biomass estimation method based on a deep learning multimodal fusion model according to an embodiment of the present invention;
[0049] Figure 4 A schematic diagram of a feeding control module in a fish pond system in a fish biomass estimation method based on a deep learning multimodal fusion model according to an embodiment of the present invention;
[0050] Figure 5 This is a schematic diagram of a biomass estimation module in a fish pond system in a fish biomass estimation method based on a deep learning multimodal fusion model according to an embodiment of the present invention. DETAILED DESCRIPTION
[0051] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.
[0052] Example 1
[0053] Reference Figure 1-Figure 5 , is an embodiment of the present invention, and provides a fish biomass estimation method based on a deep learning multimodal fusion model, comprising:
[0054] S100: Obtain images, audio and water quality data in the water area during each feeding period, and perform preprocessing;
[0055] In the embodiment of the present application, the image data includes: the size of the splashes of the fish school and the density of the fish school;
[0056] Specifically, the size of the splashes caused by a school of fish can be quantified by setting different levels of height and range, and the density of fish gathering can be quantified by the number of fish per unit area.
[0057] Audio data includes: the sound of water splashing, fish colliding, and fish swimming;
[0058] It should be noted that fish produce noticeable sounds when feeding, so collecting audio data can analyze the feeding needs of fish from multiple dimensions.
[0059] Water quality data include: dissolved oxygen value, pH and temperature.
[0060] It should be noted that the feeding behavior of fish will also cause changes in environmental parameters in the water body. Dissolved oxygen value, pH and temperature can better reflect the characteristics of feeding behavior. The behavioral characteristics of fish feeding can further reflect the corresponding biomass, size and species. Therefore, the comprehensive use of multiple information will achieve a more comprehensive and accurate estimation of fish biomass.
[0061] S200: inputting the preprocessed data into the multimodal fusion model to extract corresponding features, so as to obtain feature vectors corresponding to the image, audio and water quality;
[0062] It should be noted that the multimodal fusion model includes an input layer, an encoder, a fusion layer, and a decoder. The input layer inputs the preprocessed image, audio, water quality, and feed amount as raw data; the encoder extracts features from the raw input data; the fusion concatenates and fuses the feature vectors of the image, audio, water quality, and feed amount according to the channel dimension; the decoder uses multimodal features to predict the next feed amount and the current fish biomass.
[0063] In the embodiment of the present application, the preprocessed data are respectively input into the multimodal fusion model for corresponding feature extraction to obtain the image feature vector, including:
[0064] The preprocessed image data is input into the first convolutional neural network of the parallel processing branch in the multimodal fusion model. The convolution accelerator of the first convolutional neural network extracts the connection and spatial information and expresses the image-related features. Its activation layer introduces nonlinear features into the neural network through the activation function. The pooling layer reduces the dimension of the features and outputs the image feature vector.
[0065] In the embodiment of the present application, the preprocessed data are respectively input into the multimodal fusion model for corresponding feature extraction to obtain the audio feature vector, including:
[0066] The preprocessed audio data is input into the second convolutional neural network of the parallel processing branch in the multimodal fusion model. The convolution accelerator of the second convolutional neural network extracts connection and spatial information and expresses audio-related features. Its activation layer introduces nonlinear features into the neural network through the activation function. The pooling layer reduces the dimension of the features and outputs the audio feature vector.
[0067] In the embodiment of the present application, the preprocessed data are respectively input into the multimodal fusion model for corresponding feature extraction to obtain water quality feature vectors, including:
[0068] The preprocessed water quality parameter data is input into the text encoder of the parallel processing branch in the multimodal fusion model. The text encoder converts the water quality parameter text into a feature vector using a word embedding model and outputs a water quality feature vector.
[0069] S300: After each round of feeding, feed feeding amount data is obtained;
[0070] Specifically, the first round of feeding can be determined according to the von Bertalanffy growth equation to determine the first round of feed feeding amount. The subsequent feeding time can be determined according to the actual situation. For example, if the feeding time for each round is 30 minutes, in order to reduce feed waste and water pollution, feed is fed every 5 minutes, for a total of 6 times.
[0071] S400: inputting the preprocessed feeding amount data into a logistic regression model in a multimodal fusion model for feature extraction to obtain a feeding amount feature vector;
[0072] In the embodiment of the present application, the preprocessed feeding amount data is input into the logistic regression model in the multimodal fusion model for feature extraction to obtain a feeding amount feature vector, including:
[0073] The preprocessed feeding amount data is input into the logistic regression model of the parallel processing branch in the multimodal fusion model, converted into a feature vector through function calculation, and the feeding amount feature vector is output.
[0074] S500: fusing the image, the audio, and the feature vectors corresponding to the water quality, and concatenating the fusion through the channel dimension to obtain a first aggregate feature map;
[0075] In the embodiment of the present application, the image, audio and feature vectors corresponding to the water quality are fused and spliced through the channel dimension to obtain a first aggregate feature map, including:
[0076] The multimodal fusion model uses a cross-modal attention mechanism and a deep residual network model to introduce residual features into the acoustic and visual modalities as exogenous variables;
[0077] Specifically, it is expressed as:
[0078] f fuse-a =Attention(f a ,res(f y ))
[0079] f fuse-v =Attention(f y ,res(f a ))
[0080] f fuse-a Capturing the acoustic modes as a fusion of invariant terms, f fuse-v Capturing visual modalities as a fusion of invariants.
[0081] It should be noted that in the specific estimation of fish biomass, the interaction between visual, auditory and textual modes during the training process will have an adverse effect on the fusion features. Therefore, the above-mentioned cross-modal attention mechanism is fused with the deep residual network model. This bifurcation method ensures the retention of the basic features of each modality and has certain advantages in alleviating modal imbalance.
[0082] The basic modality features are adjusted frame by frame through the frame-level residual learning function. For each time step in each modality feature, the corresponding residual information is calculated through the attention mechanism.
[0083] Specifically, it is expressed as:
[0084]
[0085] Where n is the length of the time dimension, β i is the auditory modality feature and visual modality features sofmtax normalized similarity between, attention weight β i Calculated as:
[0086]
[0087] Introducing dynamic adaptive decision fusion, which re-integrates the acoustic and visual output features from the attention mechanism and calculates the dynamic weights of the features by combining the time dimension average;
[0088] Specifically, the normalized weight vector α is expressed as:
[0089] α=softmax(W([M(f fuse-a ); M(f fuse-v )])+bias)
[0090] Where M is the function that calculates the average value of the time dimension.
[0091] It should be noted that the above adaptability optimizes the performance of the model by selectively emphasizing relevant features, thereby improving prediction accuracy.
[0092] Connect the fusion features of the acoustic and visual modes and combine them with dynamic weights to obtain a first fusion result;
[0093] Specifically, it is expressed as:
[0094] Result fuse-av =α×result audio +(1-α)×result video
[0095] Among them, result fuse-av is the first fusion result, result audio is the audio feature in acoustic mode, result video is the image feature in visual mode.
[0096] The first fusion result is fused again with the text feature vector through dynamic adaptive decision to obtain a first aggregated feature map;
[0097] Specifically, it is expressed as:
[0098] Result fuse =α1×result fuse-av +(1-α1)×result text
[0099] Among them, result fuse is the first aggregate feature map, result text is the text feature vector.
[0100] S600: predicting the next round of feed delivery amount based on the first aggregated feature graph;
[0101] Specifically, the decoder receives the first aggregated feature map to predict the next feed delivery amount.
[0102] S700: Fusing the feature vectors corresponding to the image, audio, water quality, and feeding amount, and concatenating the fusion through the channel dimension to obtain a second aggregate feature map;
[0103] In the embodiment of the present application, the feature vectors corresponding to the image, audio, water quality and feeding amount are fused and spliced through the channel dimension to obtain a second aggregate feature map, including:
[0104] The multimodal fusion model uses a cross-modal attention mechanism and a deep residual network model to introduce residual features into the acoustic and visual modalities as exogenous variables;
[0105] The basic modality features are adjusted frame by frame through the frame-level residual learning function. For each time step in each modality feature, the corresponding residual information is calculated through the attention mechanism.
[0106] Introducing dynamic adaptive decision fusion, which re-integrates the acoustic and visual output features from the attention mechanism and calculates the dynamic weights of the features by combining the time dimension average;
[0107] Connect the fusion features of the acoustic and visual modes and combine them with dynamic weights to obtain a first fusion result;
[0108] The first fusion result is fused again with the text feature vector through dynamic adaptive decision to obtain a first aggregated feature map;
[0109] The first aggregate feature map is fused again with the feeding amount feature vector through dynamic adaptive decision making to obtain a second aggregate feature map.
[0110] It should be noted that the specific implementation of the above fusion step can refer to the specific method in S500.
[0111] S800: Estimate the current fish biomass based on the second aggregated characteristic graph.
[0112] Specifically, the decoder receives the second aggregated feature map to predict the current fish biomass.
[0113] The above is a schematic scheme of the fish biomass estimation method based on the deep learning multimodal fusion model of this embodiment. It should be noted that the technical scheme of the system for estimating fish biomass based on the deep learning multimodal fusion model and the technical scheme of the fish biomass estimation method based on the deep learning multimodal fusion model belong to the same concept. The details of the technical scheme of the fish biomass estimation system based on the deep learning multimodal fusion model in this embodiment that are not described in detail can all be referred to the description of the technical scheme of the fish biomass estimation method based on the deep learning multimodal fusion model.
[0114] like Figure 3 As shown, the fish biomass estimation system based on the deep learning multimodal fusion model in this embodiment includes:
[0115] The first acquisition module is used to acquire images, audio and water quality data in the water area during each feeding period and perform preprocessing;
[0116] Specifically, the first acquisition module includes a water quality detection module 3, an image shooting module 1 and an audio monitoring module 2;
[0117] Exemplarily, a water quality detection module 3 and an audio monitoring module 2 are arranged in the fish pond 6, and an image shooting module 1 is arranged on the fish pond;
[0118] The second acquisition module acquires the feed feeding amount data after each feeding round;
[0119] Specifically, the second acquisition module is a feed delivery module 4; the feed delivery module 4 is set on the fish pond;
[0120] The first control module inputs the preprocessed data into the multimodal fusion model for corresponding feature extraction to obtain feature vectors corresponding to the image, audio and water quality;
[0121] A first fusion module is used to fuse the image, audio and feature vectors corresponding to the water quality, and the fusion is spliced through the channel dimension to obtain a first aggregate feature map;
[0122] A first prediction module, used for predicting the next round of feed delivery based on the first aggregated feature graph;
[0123] Specifically, Figure 4 , the first control module, the first fusion module and the first prediction module belong to the feeding control module;
[0124] The second control module inputs the preprocessed feeding amount data into the logistic regression model in the multimodal fusion model for feature extraction to obtain a feeding amount feature vector;
[0125] The second fusion module is used to fuse the feature vectors corresponding to the image, audio, water quality and feeding amount, and the fusion is spliced through the channel dimension to obtain a second aggregated feature map;
[0126] The second prediction module is used to estimate the current fish biomass based on the second aggregated feature map.
[0127] Specifically, Figure 5 , the second control module, the second fusion module and the second prediction module belong to the biomass estimation module 5b;
[0128] The feeding control module 5a and the biomass estimation module 5b belong to the computer control module 5, and the computer control module is connected with the image shooting module data, the audio monitoring module data, the water quality detection module data, and the feed delivery module data;
[0129] The image capture module is used to obtain the fish feeding images in the fish pond and send them to the feeding control module and the biomass estimation module; the audio monitoring module is used to obtain the fish feeding audio in the fish pond and send them to the feeding control module and the biomass estimation module; the water quality detection module is used to obtain the water quality data of the fish pond and send them to the feeding control module and the biomass estimation module; the feed delivery module is used to deliver feed to the fish in the fish pond and send them to the biomass estimation module;
[0130] The feeding control module is used to adjust the feed feeding amount of the feed feeding module for the next time;
[0131] The biomass estimation module is used to calculate and output the current pond fish biomass;
[0132] Accordingly, the above software modules can be implemented by combining hardware, for example:
[0133] The water quality detection module includes an ammonia nitrogen sensor, a temperature sensor, a dissolved oxygen sensor, and a pH sensor; the image capture module is a C8c camera, the audio monitoring module is an HTD-92 hydrophone, the feed delivery module includes a reduction motor and a feed delivery machine electrically connected to each other, and the feeding control module is a deep learning multimodal fusion model, including a water quality text encoder, an image convolutional neural network, and an audio convolutional neural network; the biomass estimation module is a deep learning multimodal fusion model, including a text encoder, an image convolutional neural network, an audio convolutional neural network, and a logistic regression model;
[0134] The model of the ammonia nitrogen sensor is BNH1600, the model of the temperature sensor is DS18B20, the model of the dissolved oxygen sensor is DO-6020, and the model of the pH sensor is PH-3020.
[0135] The feed delivery module includes a reduction motor and a feed delivery machine that are electrically connected to each other. The feed delivery machine uses a box with a cover to form a hopper. The feeding method is mechanical centrifugal throwing, and the speed is adjusted by the reduction motor. The working principle is: when the motor starts working, it drives the eccentric connecting rod to move together, and the feed tray vibrates together. The vibrated feed will fall into the throwing tray. Under the action of centrifugal force, the feed will be evenly thrown onto the water surface in a fan shape. When the motor stops, the feeding tray stops moving, and the throwing tray stops rotating and stops feeding. The start, stop and pause time are set through the electrical control system to achieve the purpose of throwing.
[0136] This embodiment also provides a computing device, which is applicable to the case of estimating fish biomass based on a deep learning multimodal fusion model, including:
[0137] Memory and processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the fish biomass estimation method based on the deep learning multimodal fusion model as proposed in the above embodiment.
[0138] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for estimating fish biomass based on a deep learning multimodal fusion model as proposed in the above embodiment is implemented.
[0139] The storage medium proposed in this embodiment and the method for realizing fish biomass estimation based on deep learning multimodal fusion model proposed in the above embodiment belong to the same inventive concept. The technical details not described in detail in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.
[0140] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ReadOnly, Memory, ROM), random access memory (Random Access Memory, RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform the methods of various embodiments of the present invention.
[0141] Example 2
[0142] Referring to Tables 1 and 2, based on the previous embodiment, this embodiment provides an application comparison case of a fish biomass estimation method based on a deep learning multimodal fusion model to illustrate the feasibility and beneficial effects of our solution.
[0143] Traditional methods generally estimate fish biomass by manual weighing or experience, which is time-consuming and labor-intensive, and has low accuracy. The method of the present invention uses intelligent sensing technologies such as computer vision, combined with a deep learning multimodal fusion model, to achieve better estimation results under the same conditions. The following is a comparison of the present invention and the traditional method, showing the data results.
[0144] Table 1: Estimation parameters and effect comparison
[0145]
[0146] Table 2: Comparison between the convolutional neural network used in the method of the present invention and the traditional manual counting method
[0147]
[0148] From the above data and charts, we can see that compared with previous methods, we can greatly improve the accuracy and efficiency, and the data processing is multidimensional and generalized. Overall, we have more practical advantages in terms of recognition accuracy and coverage, which are the main factors affecting recognition and estimation.
[0149] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.
Claims
1. A fish biomass estimation method based on a deep learning multimodal fusion model, characterized in that: include: Obtain images, audio and water quality data in the water area during each feeding period and perform preprocessing; The preprocessed data are respectively input into the multimodal fusion model for corresponding feature extraction to obtain the corresponding feature vectors of image, audio and water quality; After each round of feeding, obtain the feed feeding amount data; The preprocessed feeding amount data is input into the logistic regression model in the multimodal fusion model for feature extraction to obtain the feeding amount feature vector; Fusing the image, the audio, and feature vectors corresponding to the water quality, wherein the fusion is spliced through the channel dimension to obtain a first aggregate feature map; Predicting the next round of feed delivery based on the first aggregated feature graph; Fusing the feature vectors corresponding to the image, audio, water quality, and feeding amount, wherein the fusion is spliced through the channel dimension to obtain a second aggregated feature map; The current fish biomass is estimated based on the second aggregate characteristic graph.
2. The fish biomass estimation method based on the deep learning multimodal fusion model according to claim 1, characterized in that: The image data includes: the size of the splashes of the fish school and the density of the fish school; The audio data includes: the sound of water splashing, the sound of fish colliding and the sound of fish swimming; The water quality data include: dissolved oxygen value, pH and temperature.
3. The fish biomass estimation method based on the deep learning multimodal fusion model according to claim 2, characterized in that: The preprocessed data are respectively input into the multimodal fusion model for corresponding feature extraction to obtain the image feature vector, including: The preprocessed image data is input into the first convolutional neural network of the parallel processing branch in the multimodal fusion model. The convolution accelerator of the first convolutional neural network extracts the connection and spatial information and expresses the image-related features. Its activation layer introduces nonlinear features into the neural network through the activation function. The pooling layer reduces the dimension of the features and outputs the image feature vector.
4. The fish biomass estimation method based on the deep learning multimodal fusion model according to claim 3, characterized in that: The preprocessed data are respectively input into the multimodal fusion model for corresponding feature extraction to obtain the audio feature vector, including: The preprocessed audio data is input into the second convolutional neural network of the parallel processing branch in the multimodal fusion model. The convolution accelerator of the second convolutional neural network extracts connection and spatial information and expresses audio-related features. Its activation layer introduces nonlinear features into the neural network through the activation function. The pooling layer reduces the dimension of the features and outputs the audio feature vector.
5. The fish biomass estimation method based on the deep learning multimodal fusion model according to claim 4, characterized in that: The preprocessed data are input into the multimodal fusion model for corresponding feature extraction to obtain water quality feature vectors, including: The preprocessed water quality parameter data is input into the text encoder of the parallel processing branch in the multimodal fusion model. The text encoder converts the water quality parameter text into a feature vector using a word embedding model and outputs a water quality feature vector.
6. The fish biomass estimation method based on the deep learning multimodal fusion model according to claim 5, characterized in that: The preprocessed feeding amount data is input into the logistic regression model in the multimodal fusion model for feature extraction to obtain the feeding amount feature vector, including: The preprocessed feeding amount data is input into the logistic regression model of the parallel processing branch in the multimodal fusion model, converted into a feature vector through function calculation, and the feeding amount feature vector is output.
7. The fish biomass estimation method based on the deep learning multimodal fusion model according to claim 1 or 6, characterized in that: The multimodal fusion model adopts a cross-modal attention mechanism and a deep residual network model to introduce residual features as exogenous variables into the acoustic modality and the visual modality; The basic modality features are adjusted frame by frame through the frame-level residual learning function. For each time step in each modality feature, the corresponding residual information is calculated through the attention mechanism. Introducing dynamic adaptive decision fusion, which calculates the dynamic weight of features by reintegrating acoustic and visual output features from the attention mechanism and combining the time dimension average; Connect the fusion features of the acoustic and visual modes and combine them with dynamic weights to obtain a first fusion result; The first fusion result is fused again with the text feature vector through the dynamic adaptive decision to obtain a first aggregated feature map; The first aggregated feature graph and the feeding amount feature vector are fused again through the dynamic adaptive decision to obtain a second aggregated feature graph.
8. A system using the fish biomass estimation method based on the deep learning multimodal fusion model as claimed in claim 1, characterized in that: include: The first acquisition module is used to acquire images, audio and water quality data in the water area during each feeding period and perform preprocessing; The first control module inputs the preprocessed data into the multimodal fusion model for corresponding feature extraction to obtain feature vectors corresponding to the image, audio and water quality; The second acquisition module acquires the feed feeding amount data after each feeding round; The second control module inputs the preprocessed feeding amount data into the logistic regression model in the multimodal fusion model for feature extraction to obtain a feeding amount feature vector; A first fusion module is used to fuse the image, the audio and the feature vectors corresponding to the water quality, wherein the fusion is spliced through the channel dimension to obtain a first aggregate feature map; A first prediction module, used for predicting the next round of feed delivery amount based on the first aggregated feature graph; A second fusion module is used to fuse the feature vectors corresponding to the image, audio, water quality and feeding amount, and the fusion is spliced through the channel dimension to obtain a second aggregated feature map; The second prediction module is used to estimate the current fish biomass based on the second aggregation characteristic graph.
9. Electronic equipment, including: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the fish biomass estimation method based on the deep learning multimodal fusion model described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the fish biomass estimation method based on a deep learning multimodal fusion model as described in any one of claims 1 to 7.