Fish health early warning system based on multiple modes

Abstract

The invention discloses a fish health early warning system based on multiple modes. The system comprises an image and audio data acquisition module, a fish health early warning algorithm control module, a front and back end module, a hardware unit and an EMQX cloud service module. According to the system, video images shot by an underwater camera and audio data collected by an underwater microphone at the same time point are detected in real time through a multi-modal algorithm model, classification is carried out according to the swimming activeness of a fish school and the frequency of sound, and early warning is carried out in a healthy state and a disease state. The method is more stable, can save a large amount of manpower and material resources, is convenient to operate, overcomes the limitation of a single method, detects the health of fish schools according to different videos and audios of each fish pond through an algorithm, achieves the effect of fish health early warning, and has extremely high application value.

Description

Technical Field

[0001] The invention relates to the technical field of fishery health breeding, and in particular to a fish health early warning system based on multi-modality. Background Art

[0002] Traditional fish health monitoring methods mainly rely on manual observation and regular sampling inspection. These methods are not only time-consuming and labor-intensive, but also have the following limitations: Manual observation is to judge the health status of fish by observing their behavior, appearance and swimming posture. This method has problems such as strong subjectivity and limited coverage. Pathological, physiological and biochemical indicators of fish are tested by regular sampling. Although this method is relatively accurate, it is complicated to operate, expensive, causes certain stress reactions to fish, and cannot achieve real-time monitoring. At present, most fish health warning algorithms mainly use single visual or auditory information, which to a certain extent limits the accuracy and comprehensiveness of detection. Joint audio-visual learning of multimodal fusion is the core mechanism of human perception. Through the comprehensive processing and analysis of visual and auditory information, accurate detection of the health status of fish schools is achieved. Therefore, how to provide a system and method that can accurately and real-timely detect the health of fish schools 24 hours a day and obtain fish status information in real time has become an urgent problem to be solved. Summary of the invention

[0003] The purpose of the present invention is to solve the above problems and to design a fish health early warning system based on multimodality.

[0004] The first aspect of the present invention provides a multimodal fish health warning system, which includes an image and audio data acquisition module, a fish health warning algorithm control module, a front-end and back-end module, a hardware unit and a cloud service module, wherein: The image and audio data acquisition module is used to connect the underwater camera and the underwater microphone through the USB interface, save the video captured by the underwater camera and the audio collected by the underwater microphone locally, perform frame processing on the captured video, perform denoising on the audio at the same time, and construct a multimodal audio-video data set for algorithm model training; The fish health algorithm control module is used to analyze the images and audio captured by the camera in real time using a multimodal algorithm to obtain analysis results, determine a fish health warning plan based on the analysis results, and transmit the fish health warning plan to the EMQX cloud service module; The EMQX cloud service module is used to connect to the cloud service using username and password according to the mqttServerUrl address. After the EMQX cloud server receives the fish health warning plan sent by the device number of a specific fish farm, it will forward the fish health warning plan to all devices that subscribe to this topic.

[0005] The front-end and back-end modules are used to receive the fish health warning plan data distributed from the EMQX server by subscribing to the topic of the fish health warning plan, read the fish health warning plan of the specific equipment number under the fish farm, and send the fish health warning plan to the fish farming health experts through the WeChat public account, thereby completing the purpose of fish health warning.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the multimodal algorithm is used to analyze the images captured in real time by the underwater camera and the audio collected by the underwater microphone to obtain analysis results, and a fish health warning plan is determined according to the analysis results, including: The incoming image is resized to 224×224, the low frame rate video frame sequence is input into the slow channel, and the high frame rate video frame sequence is input into the fast channel to obtain the image feature maps of the fast channel and the slow channel respectively; at the same time, the time-synchronized original audio is denoised by wavelet packet transform, and then the time-frequency two-dimensional map is obtained by Fourier transform, Mel filter group weighting, logarithmic transform and other operations, and the Log-mel spectrogram is input into the audio channel to obtain the audio feature map.

[0007] The feature map is subjected to feature extraction. After convolution, activation and pooling operations, the feature map gradually evolves from the initial low-level features to high-level features, and the number of feature channels increases continuously. The network branches all use 3D-Resnet100 as the backbone network to respectively extract the spatial distribution features and dynamic temporal features of the input image sequence; Before the final classifier, the features of the slow channel, fast channel and audio channel are concatenated and fused. The fused features are input into the CBAM attention mechanism and finally into the fully connected layer for classification tasks.

[0008] Optionally, in a second implementation of the first aspect of the present invention, feature extraction is performed on the feature map, and after convolution, activation and pooling operations, the feature map gradually evolves from initial low-level features to high-level features, and the number of feature channels continues to increase, including: The feature map obtained from the low frame rate video frame sequence is input into the slow pathway multi-path residual network, and a deep convolution and activation function are first performed, and then the fastpathway features and the Audio pathway features after the CPCA attention mechanism on the lateral channel are fused for the first time, and then the convolution layers and activation functions with more convolution kernels are passed, and then the fast pathway features and the Audiopathway features after the CPCA attention mechanism on the lateral channel are fused for the second time, and then the feature map is input into the last convolution layer with the maximum depth for convolution to complete the process of slow channel feature extraction, and convolution is performed after obtaining spatiotemporal information, channel information and motion information in different dimensions to obtain more fine-grained features.

[0009] The feature map obtained from the high frame rate video frame sequence is input into the fast pathway multi-path residual network, and a shallow convolution and activation function are first performed, and then the Audio pathway features on the lateral channel are fused for the first time, and then the convolution layers and activation functions of more convolution kernels are passed through, and then the Audio pathway features on the lateral channel are fused for the second time, and then the feature maps are input into the last layer of convolution with the maximum depth for convolution to complete the fast channel feature extraction process.

[0010] The Log-mel spectrogram is input into the Audio pathway multi-path residual network. First, two convolutions and CPCA attention mechanisms are performed successively to capture appropriate spatiotemporal information, strengthen the relationship between space and time, and better understand the changes in some actions in fish activities. Finally, it is input into the last layer of convolution with the maximum depth for convolution to complete the process of audio channel feature extraction.

[0011] Optionally, in a third implementation of the first aspect of the present invention, the fish health warning scheme includes detecting as a disease when the fish school shows high activity or low activity and high frequency or low frequency audio occurs, and detecting as healthy when normal swimming and normal audio occur.

[0012] Optionally, in a fourth implementation of the first aspect of the present invention, the fish health warning scheme includes detecting as a disease when the fish school shows high activity or low activity and the audio shows high frequency (greater than 1000hz) or low frequency (less than 100hz); and detecting as healthy when normal swimming and normal audio (100hz-1000hz) occur.

[0013] Optionally, in the fifth implementation of the first aspect of the present invention, the hardware unit includes at least an underwater camera, an underwater microphone, a ZH-I5-5200U dual serial port host, and an alarm flashing light, wherein the sensor detects the temperature, conductivity, dissolved oxygen, pH and liquid level of the fish pond.

[0014] Optionally, in a sixth implementation of the first aspect of the present invention, the front-end and back-end modules enter the front-end page by accessing the IP address, wherein different users have different permissions, and the administrator has the highest permissions. The front-end functions include fishery staff management, fish health log management, large-screen real-time viewing, facility management, equipment management, and alarm notification management. The fishery staff management part can add, modify, and delete the account information of fishery staff. The fish health log management can view the health status of the 24-hour fish school. In the large-screen real-time viewing, the facility can be selected through the drop-down menu, and then the device can be selected for viewing. The device number is displayed on the right, and the video uploaded by the camera in real time is displayed below. The upper left corner of the video will show the health status of the fish school at the current time. The lower right corner shows the record of the disease and whether the breeding expert has handled it. The record of the disease can be downloaded by clicking Download (Excel table form), and the left half of the page also has the values ​​of temperature, ammonia nitrogen, conductivity, PH, and dissolved oxygen detected by the sensor.

[0015] In the technical solution provided by the present invention, the system adopts a multimodal algorithm to realize real-time detection of fish health, which is integrated into the algorithm according to the suggestions of fish health experts, and then the video images captured by the underwater camera and the audio collected by the underwater microphone are detected in real time through the multimodal algorithm model. The fish in the fish pond are classified according to the activity of the fish and the level of the audio, and the health status is detected by dividing them into healthy and diseased states. If it is healthy, a log record is made; if it is diseased, a log record is made and the fish health expert is notified. The test results are transmitted to the EMQX cloud server, and the back end accepts the distribution results of the EMQX cloud server to decide whether to notify the fish health expert.

[0016] Compared with the existing manual observation and sampling inspection methods, the present invention is more automated, efficient, low-cost, and more stable. It is no longer necessary to hire multiple fish health experts to observe and sample each fish pond, which can save a lot of manpower, material and financial resources. It is easy to operate and highly controllable. Compared with the existing fish health technology, it is more accurate and no longer uses a single modality as the basis for consideration. It will use video and audio modalities as the basis for detection, thereby achieving the effect of improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A logic flow chart of a multi-modal fish health early warning system provided by an embodiment of the present invention; Figure 2A hardware unit structure connection diagram of a multi-modal fish health early warning system provided by an embodiment of the present invention; Figure 3 A logic flow chart of a fish health warning algorithm control module of a multi-modal fish health warning system provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0018] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 A schematic diagram of a multimodal fish health early warning system provided by an embodiment of the present invention, the system includes an image and audio data acquisition module, a fish health algorithm control module, a front-end and back-end module, a hardware unit and a cloud service module, wherein: The image and audio data acquisition module is used to connect the underwater camera and the underwater microphone respectively through the USB interface, and save the video captured by the underwater camera and the audio collected by the underwater microphone at the same time locally. In addition, the captured video is frame-processed, and the processed images and the audio at the same time are made into a multimodal video-audio data set for algorithm model training; The fish health warning algorithm control module is used to analyze the images and audio captured by the camera in real time using a multimodal algorithm to obtain analysis results, determine a fish health warning plan based on the analysis results, and transmit the fish health warning plan to the EMQX cloud service module; EMQX cloud service module is used to connect to the cloud service using username and password according to the mqttServerUrl address. After receiving the fish health warning plan sent by the device number of the specific fish farm, the EMQX cloud server will forward the fish health warning plan to all devices (backends) subscribed to this topic. The backend will send instructions to the ZH-I5-5200U host according to the fish health warning plan. The ZH-I5-5200U host forwards the instructions to the DI / DO module through the USB to RS232 module (adapter), and then the DI / DO module controls the switch of the relay through high and low levels, that is, the start and stop of the alarm flasher.

[0019] The front-end and back-end modules access the front-end page by accessing the IP address. Different users have different permissions, and the administrator has the highest permissions. The front-end functions include fish farm employee management, fish health log management, large screen real-time viewing, facility management, equipment management, and alarm notification management. The fish farm employee management section can add, modify, and delete the account information of fish farm employees. The fish health log management can view the 24-hour health status of the fish. Below the log information, the 24-hour health status of the fish will be displayed in the form of icons or line graphs. In the large screen real-time viewing, you can select the facility through the drop-down menu, and then select the device to view. The device number is displayed on the right, and the video uploaded by the camera in real time is displayed below. The upper left corner of the video will show the current health status of the fish, and the lower right corner will show the disease record and whether the breeding expert has handled it. You can download the disease record (Excel table format) by clicking Download. The left half of the page also has the values ​​of temperature, ammonia nitrogen, conductivity, PH, and dissolved oxygen detected by sensors.

[0020] The fish health warning algorithm control module first performs preprocessing and adjusts the images in the data set to a size of 224×224. In order to extract effective identification information from the lens space and dynamic timing features, different sampling rates are set for the input frame sequence to achieve fast and slow channels. The slow channel captures long-term changes, and the fast channel captures fast dynamic features. The feature map obtained from the low frame rate video frame sequence is input into the slow pathway, and first a deep convolution and activation function is performed, and then the fast pathway features and the Audiopathway features after the CPCA attention mechanism on the lateral channel are fused for the first time, and then the convolution layers and activation functions with more convolution kernels are passed, and then the fast pathway features and the Audio pathway features after the CPCA attention mechanism on the lateral channel are fused for the second time, and then the feature map is input into the last layer of convolution with the maximum depth for convolution to complete the process of slow channel feature extraction.

[0021] The feature map obtained from the high frame rate video frame sequence is input into the fast pathway, and a shallow convolution and activation function is first performed, and then it is fused with the Audio pathway features on the lateral channel for the first time, and then it passes through more convolution layers and activation functions of convolution kernels, and then it is fused with the Audio pathway features on the lateral channel for the second time, and then it is input into the last layer of convolution with the maximum depth for convolution to complete the fast channel feature extraction process.

[0022] The Log-mel spectrogram is input into the Audio pathway, and firstly two convolutions and CPCA attention mechanism are performed successively, and finally it is input into the last convolution layer with the maximum depth for convolution to complete the process of audio channel feature extraction.

[0023] Place the underwater camera and underwater microphone in the middle of the water, and then perform real-time detection on the images captured by the underwater camera and the sounds collected by the underwater microphone. If the fish activity is normal and the audio frequency is normal, only log records are made. If the fish activity is high or low and the audio frequency is high or low at the same time, a notification is sent to the fish health expert and the alarm flashing light comes on. The algorithm model adopts an improved SlowFast network model. First, a CPCA attention mechanism module is introduced on the lateral channel from the fast pathway to the slow pathway. At the same time, a CPCA attention mechanism module is added after the first two convolutions of the Audio pathway. First, a 1×1 convolution is used to extract context information for each channel independently, and then the features of each channel are aggregated to obtain more global context information of the channel. After the gradual aggregation of the context information, the aggregated features are nonlinearly transformed (ReLU activation) to introduce nonlinear expression capabilities. Global MaxPooling (GMP) is used to reduce the dimension of each channel to generate a global representation of each channel, and these global representations are connected through a fully connected layer (Fully Connected, FC) transformation to generate attention weights for each channel; use Sigmoid function to normalize the weights to between 0 and 1 to ensure that the attention weights are within a reasonable range; multiply the original features of each channel by the corresponding attention weights to obtain an enhanced feature map; finally, the output feature map has the same shape as the input feature map, but enhances important features on each channel, while suppressing irrelevant or redundant information, reducing the amount of computation, and increasing the model's attention to important information, thereby improving overall performance; In order to improve SlowFast's feature fusion ability, the CBAM attention mechanism is introduced after feature fusion. The channel fusion strategy of AVSlowfast gives equal weights to audio and video signals. For some behaviors with unclear audio features, the audio and video fusion method may reduce the accuracy of behavior detection compared to the model using single video information. Therefore, the CBAM attention mechanism is introduced to adjust the importance of the audio channel; CBAM combines the two mechanisms of channel attention and spatial attention. By weighting the channel and spatial dimensions respectively, it can more comprehensively enhance the expressiveness of the feature map. The introduced parameters and computational overhead are relatively small, and will not significantly increase the complexity and computational burden of the model. It can effectively enhance the expression of important features and suppress irrelevant features, thereby improving the model's detection ability and robustness for input data, and adaptively adjust the weights of each channel and spatial position of the feature map, so that it can adapt to different data distributions and task requirements.

[0024] In the embodiment of the present invention, the hardware unit includes sensors, DI / DO modules, relays, alarm flashing lights, USB to RS232 modules, ZH-I5-5200U host, underwater microphones and underwater cameras. Figure 2 Connect and fix the ZH-I5-5200U host, DI / DO module, USB to RS232 module and relay into the equipment box, fix the underwater camera and underwater microphone to the middle position underwater to ensure that they can capture the fish swimming horizontally, and fix the alarm flashing light to a suitable position outside the fish pond. First, collect data, annotate the collected data, and make it into a video-audio data set for use by the algorithm model. The video-audio data is mainly divided into two types of data: health and disease. If the fish activity is normal and the audio frequency is normal, it is considered healthy. If the fish activity is high or the fish activity is low and the audio frequency is high or low at the same time, it is considered a disease. Fish health early warning control module, such as Figure 3As shown in the figure, the incoming video and audio are preprocessed, feature extracted, three-branch feature fused, and finally classified through the fully connected layer. The images taken by the underwater camera and the audio collected by the underwater microphone are detected. If the fish activity is normal and the audio frequency is normal, only log records are made. If the fish activity is high or low and the audio frequency is high or low at the same time, a notification is sent to the fish health expert and the alarm flashing light is on. The front-end and back-end modules enter the front-end page by accessing the IP address. Different users have different permissions, and the administrator has the highest permissions. The front-end functions include fish farm employee management, fish health log management, large-screen real-time viewing, facility management, equipment management and alarm notification management. The fish farm employee management section can add, modify and delete the account information of fish farm employees. Among them, employees can be divided into fish farm employees, fish farm tenants and fish farm tenant employees. The fish health log management can view the health status of fish schools for 24 hours, 7 days and 1 month respectively. Below the log information, the health status of fish schools for 24 hours, 7 days and 1 month will be displayed in the form of icons or line graphs, which will display specific fish farm facilities, specific time and specific fish farm water quality or fish health in the fish farm. In the large-screen real-time viewing, you can select facilities through the drop-down menu, and then select equipment to view. The device number is displayed on the right, and the video uploaded by the camera in real time is displayed below. The upper left corner of the video will show the current health status of the fish school, the upper right corner will show the record of the disease and whether the breeding expert has dealt with it, and the lower right corner will show the up, down, left and right operation buttons of the camera to operate the angle you want to shoot. You can download the data record by clicking Download (Excel table format). The left half of the page also has the values ​​of temperature, ammonia nitrogen, conductivity, PH and dissolved oxygen detected by the sensor. Below is the alarm flashing light switch button. When the on button is clicked, the alarm flashing light will light up. The operation time and the three-level name of the fish farm facilities and equipment will be displayed below the button. If the algorithm is abnormal, the fish health warning algorithm can be forcibly controlled by turning on or off the button. EMQX cloud service module uploads the results of multimodal algorithm detection to EMQX cloud service module through mqttServerUrl address and corresponding username and password according to the device number. After receiving the results sent under the device number of the specific fish farm, EMQX cloud server will forward this result to all devices subscribed to this topic. After receiving this result, the back-end device sends the control instruction of the alarm flashing light to the EMQX cloud service module. After receiving this information, the EMQX cloud service module forwards the information to the ZH-I5-5200U dual serial port host. The ZH-I5-5200U dual serial port host transmits the instruction to the DI / DO module through the USB to RS232 module, and then the DI / DO module uses high and low levels to control the switch of the relay, that is, the start and stop of the alarm flashing light.

[0025] In summary, the present embodiment adopts multimodal algorithm detection to realize the detection of fish health warning. According to the multimodal algorithm model, the video image captured by the underwater camera and the audio collected by the underwater microphone are detected in real time. According to the activity of the fish in the fish pond and the frequency of the audio, the fish are classified into two states: health and disease to control the alarm flashing light. If it is healthy, the alarm flashing light is not on, and the back end only records the log. If it is diseased, the alarm flashing light is on. The back end not only records the log, but also sends the specific fish pond number of the specific fish farm to the fish health expert to achieve the notification effect. Compared with the existing manual observation and regular sampling inspection methods, it is more automated, efficient, low-cost, and more stable. There will be no problems such as strong human subjectivity, limited coverage, complex operation, and inability to check the lights in real time. A lot of manpower, material resources and financial resources can be saved. It is easy to operate and highly controllable. Compared with the existing fish health warning technology, it is more intelligent and no longer relies on a single mode as a standard for measuring the health of the fish. The alarm flashing light will be dynamically controlled by the algorithm according to the different fish activity and audio frequency of each fish pond, thereby achieving the effect of fish health warning.

[0026] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-modal fish health early warning system, characterized by: include: An image and audio data acquisition module is used to connect an underwater camera and an underwater microphone, store the video captured by the camera and the audio collected by the microphone locally, perform frame processing on the captured video, and produce a video-audio data set with the processed image and the audio at the same time for algorithm model training; A fish health warning algorithm control module, which is used to use a multimodal algorithm to analyze the images and audio captured in real time by the underwater camera and the underwater microphone to obtain analysis results, and determine a fish health warning plan based on the analysis results, and transmit the fish health warning plan to the front-end and back-end modules; EMQX cloud service module, which uses username and password to connect to the cloud service according to the mqttServerUrl address. After receiving the fish health warning plan sent by the device number of a specific fish farm, the EMQX cloud server will forward the fish health warning plan to all devices subscribed to this topic; The front-end and back-end modules are used to receive the fish health warning plan distributed from the EMQX server by subscribing to the topic of the fish health warning plan, read the fish health warning plan of the specific equipment number under the fish farm, and send the fish health warning plan to the fish farming health experts through the WeChat public account, thereby achieving the purpose of fish health warning.

2. The multimodal fish health early warning system according to claim 1, characterized in that: A multimodal algorithm is used to analyze the images captured by the underwater camera in real time and the audio collected by the underwater microphone to obtain analysis results, and a fish health warning plan is determined based on the analysis results, including: The incoming image is resized to 224×224, and the low frame rate video frame sequence is input into the slow channel, and the high frame rate video frame sequence is input into the fast channel to obtain the image feature maps of the fast channel and the slow channel respectively; at the same time, the time-synchronized original audio is denoised by wavelet packet transform, and then the time-frequency two-dimensional map is obtained by Fourier transform, Mel filter bank weighting, logarithmic transform and other operations, and the Log-mel spectrogram is input into the audio channel to obtain the audio feature map; The feature map is subjected to feature extraction. After convolution, activation and pooling operations, the feature map gradually evolves from the initial low-level features to high-level features, and the number of feature channels increases continuously. The network branches all use 3D-Resnet100 as the backbone network to respectively extract the spatial distribution features and dynamic temporal features of the input image sequence; Before the final classifier, the features of the slow channel, fast channel and audio channel are concatenated and fused. The fused features are input into the CBAM attention mechanism and finally into the fully connected layer for classification tasks.

3. The multimodal fish health early warning system according to claim 2, characterized in that: The feature map is subjected to feature extraction. After convolution, activation and pooling operations, the feature map gradually evolves from the initial low-level features to high-level features, and the number of feature channels continues to increase, including: The feature map obtained from the low frame rate video frame sequence is input into the slow pathway multi-path residual network, and a deep convolution and activation function are first performed, and then the fast pathway features and the Audio pathway features after the CPCA attention mechanism on the lateral channel are first fused, and then the convolution layers and activation functions of more convolution kernels are passed, and then the fast pathway features and the Audio pathway features after the CPCA attention mechanism on the lateral channel are fused for the second time, and then the feature map is input into the last convolution layer with the maximum depth for convolution to complete the process of slow channel feature extraction, and convolution is performed after obtaining spatiotemporal information, channel information and motion information in different dimensions to obtain more fine-grained features; The feature map obtained from the high frame rate video frame sequence is input into the fast pathway residual network, and a shallow convolution and activation function are first performed, and then the Audio pathway features on the lateral channel are first fused, and then more convolution layers and activation functions are passed through, and then the Audio pathway features on the lateral channel are again fused for the second time, and then input into the last convolution layer with the maximum depth for convolution to complete the fast channel feature extraction process; The Log-mel spectrogram is input into the Audio pathway residual network. First, two convolutions and CPCA attention mechanisms are performed successively to capture appropriate spatiotemporal information and strengthen the relationship between space and time. Finally, it is input into the last convolution layer with the maximum depth for convolution to complete the process of audio channel feature extraction.

4. The multimodal fish health early warning system according to claim 1, characterized in that: The fish health warning scheme includes detecting that the fish are sick when they are highly active or less active and the audio frequency is high or low frequency, and detecting that they are healthy when they are swimming normally and have normal audio frequency.

5. The multimodal fish health early warning system according to claim 4, characterized in that: The high frequency is greater than 1000hz, the low frequency is less than 100hz, and the normal audio is 100hz-1000hz.

6. The multimodal fish health early warning system according to claim 1, characterized in that: The system also includes a hardware unit, which includes at least an underwater camera, an underwater microphone, a ZH-I5-5200U dual serial port host, an alarm flashing light, and corresponding sensors for detecting the temperature, conductivity, dissolved oxygen, pH and liquid level of the fish pond.

7. The multimodal fish health early warning system according to claim 1, characterized in that: The front-end and back-end modules enter the front-end page by accessing the IP address.

Images