Fish body state nondestructive testing method and device based on multi-modal data

Abstract

The invention provides a fish body state nondestructive testing method and device based on multi-modal data, and belongs to the technical field of aquaculture fish state monitoring. The device comprises a fixed bracket, a flexible main control unit and an optical detection unit, connection and position constraint are carried out through the flexible carrier plate, the signal line and the flexible ring. The fixing support is fixed to a fish bone on the near side of a fin in a lossless mode in a clamping mode, and the whole device controls data collection, storage and transmission through the flexible main control unit. The invention further discloses a nondestructive testing method. The nondestructive testing method comprises the following steps: collecting signals; collecting a combined optical signal, a kinematics signal and environmental data; performing pretreatment; motion correction is carried out, motion artifacts in optical signals are eliminated through kinematics signals in combination with a specific algorithm, and the signals are reconstructed; finally, state discrimination is achieved, and fish state discrimination is achieved on the basis of multi-modal data and an expert knowledge base.

Description

Technical Field

[0001] The present invention relates to the technical field of fish state monitoring, and in particular to a method and device for non-destructive detection of fish state based on multimodal data. Background Art

[0002] In fish farming and fish physiological research, fish status (including behavior and physiological status) detection and environmental detection are of great significance for production and scientific research. With the development of sensor technology, there are currently implantable devices, flexible wearable devices, etc. used for in-situ detection of fish status. However, implantable devices often require anesthesia of fish and puncture of fish tissues, which has a destructive effect on fish organ tissues, affecting the survival rate and normal life status of fish, such as; flexible wearable sensors can solve the above problems to a certain extent, but due to the non-adhesiveness caused by fish mucus, wearable devices cannot be attached to the fish body for a long time by bonding; bundling and fixing at the fish tail is a solution, but these devices can only be used to detect the kinematic characteristics (acceleration) and environmental parameters of the fish body, and cannot effectively monitor the heart rate and other physiological states of the fish body, such as "marine skin".

[0003] Some researchers have tried to bundle flexible wearable devices onto zebrafish to detect the heart rate of the fish. However, this method does not work for larger fish such as largemouth bass, as the ECG signal obtained based on the chest cavity will cover the heart rate with noise. In addition, this approach cannot be used to bundle and measure flatfish. Summary of the invention

[0004] According to the technical problems mentioned in the above background technology, in order to non-destructively obtain the state of fish (motion information, heart rate information), a method and device for non-destructive detection of fish state based on multimodal data are provided.

[0005] The technical means adopted by the present invention are as follows: A method for nondestructive detection of fish body status based on multimodal data comprises the following steps: Step 1: signal acquisition: the LED light source in the optical detection unit fixed at different parts of the fish fin emits light at a set frequency to the fish fin. After the light passes through the arterial capillaries of the fish fin, the probe on the other side of the corresponding fish fin receives the transmitted light source to obtain a transmitted spectrum combination signal; Step 2: preprocessing the transmission spectrum combined signal; performing denoising and baseline drift processing on the electromyographic noise and baseline drift to obtain an optical signal; Step 3: Perform motion correction; according to the optical signal and the transmission spectrum combination signal, use deep learning and machine learning methods to extract signal features and eliminate motion artifacts in the photoplethysmography signal; Step 4: Perform state discrimination; based on the corrected optical signal, the transmission spectrum combination signal and the environmental data, a feature extraction algorithm is used to obtain the time domain and frequency domain characteristics of the multivariate signal; the time domain and frequency domain characteristics of the multivariate signal are used to discriminate the state of fish.

[0006] Furthermore, in step 1, according to the Lambert-Beer law, at time t, the spectral combination signals obtained from different parts of the fins include: the proximal projection spectral signal of the pectoral fin or skirt and the distal projection spectral signal of the tail fin : (1); (2); in, and Respectively t The intensity of the near and far emitting light sources at the moment; and denote the water film attenuation coefficients of the proximal and distal detection units, respectively; and Represent the optical path lengths of the near-end and far-end detection units, respectively.

[0007] Furthermore, the step 3 comprises the following steps: Step 31: For PPG Signal , And the IMU signal M Using wavelet transform, we can get the optical wavelet coefficients and kinematic wavelet coefficients. , and : (3); (4); (5); in, represents the wavelet basis function; Represents the scale parameter, which affects the details of frequency analysis; Indicates the time shift parameter, and the time effect of the positioning signal; Expressed as complex conjugate; Step 32: Analyze motion wavelet coefficients , identify the main frequency components caused by motion , compare and obtain the optical wavelet coefficients with Wavelet coefficients of energy distributions of similar or same scale and , get the corrected wavelet coefficients after eliminating motion artifacts and The calculation formula is: (6); (7); in, and Respectively represent the artifact elimination factors of the near end and the far end, which are used to control and adjust the degree of motion artifact elimination; Step 33: Reconstruct the pectoral fin or skirt and caudal fin PPG signals using the corrected wavelet coefficients and , and obtain the comprehensive PPG signal for accurate calculation of the fish's heart rate , the formula is:

[0008] in, and The sum is 1, which is used to adjust the signal strength in different parts.

[0009] The present invention also includes a nondestructive detection device for fish body status based on multimodal data, comprising: a fixed bracket, a flexible main control unit and an optical detection unit; the fixed bracket and the flexible main control unit and the optical detection unit are connected through a flexible carrier board and a signal line wrapped by a flexible material; the nondestructive monitoring device realizes the constraint of the device position through a flexible ring; The fixing bracket is a slightly elastic rigid card-type clip structure, which can clamp and fix the fish bones near the gills; the fixing bracket is encapsulated by a flexible material; The device is non-destructively fixed on the fish body by the fixing bracket, and the optical detection unit is controlled by the flexible main control unit to transmit a signal; the transmission signal is transmitted to the LED transmission source port through the optical fiber signal line, and the signal end receiving port corresponding to the LED transmission source port receives the projected light signal and transmits it to the flexible main control unit; the flexible main control unit stores the collected data to complete the fish body status data collection.

[0010] Furthermore, the flexible main control unit includes: a microprocessor MCU, a multi-sensor, a cellular Bluetooth module and an embedded small lithium battery; the microprocessor MCU is connected to the multi-sensor, the cellular Bluetooth module and the embedded small lithium battery; the cellular Bluetooth module is connected to the microprocessor MCU via a serial communication interface to support remote data transmission and wireless communication functions between devices; the embedded small lithium battery provides a stable operating voltage for the flexible main control unit; the flexible main control unit is encapsulated in a flexible material.

[0011] Further, the optical detection unit includes: a distributed magnetic attraction point array, a signal transmitting port and a signal receiving port; The distributed magnetic attraction point array is distributed on both sides of the fin rays of the fish fin to fix the optical detection unit; the signal transmitting port is aligned and fit with the signal receiving port, and are distributed on both sides of the fin membrane of the fish fin; the distributed magnetic attraction point array is connected to the signal transmitting port through a flexible material; the distributed magnetic attraction point array is connected to the signal receiving port through a flexible material.

[0012] Compared with the prior art, the present invention has the following advantages: The scheme of the present invention realizes non-destructive and accurate estimation of the heart rate of fish; for the collection of heart rate, the method of the present invention does not adopt the conventional ECG heart rate detection method, but uses optical methods and kinematic methods to extract heart rate signals through designed devices and methods, and does not require destructive treatment of the fish body, thereby reducing the impact of the equipment on the physiological health status of the fish body.

[0013] The scheme of the present invention adopts a wearable scheme based on the structural characteristics of the fish body; according to the characteristics of the hard fish bones near the gills of the fish body, a supporting structure designed with composite materials is adopted to reduce the interference of the wearable device wrapping the whole fish body with the fish's movement behavior, and avoid the problem that the mucus continuously secreted by the fish body makes the device unable to adhere; at the same time, for the clamping structure of the fish fin part, a flexible material combined with a magnetic clamping structure is adopted to meet the detection of optical signals without interfering with the movement of the fish fin.

[0014] This research proposal utilizes the structural characteristics of fish fins to construct a heart rate acquisition method based on optical signals; this method is based on the physiological structural characteristics of fish fins / skirts with high penetration and sufficient distribution of arterial capillaries, designs an optical signal acquisition structure, and combines a heart rate estimation method that is adapted to the structure to construct an accurate measurement of fish heart rate based on optical PPG signals.

[0015] The scheme of the present invention makes full use of optical information and kinematic information; the method scheme makes full use of and couples the optical information and kinematic information of different parts of the fish fins, eliminates the corresponding motion artifacts in the PPG signal based on the kinematic information, and realizes the accurate measurement of the heart rate signal. Compared with the weak ECG signal which is only applicable to tiny fish, this method can be applied to large fish; multimodal information is used to judge the state of the fish more comprehensively and accurately.

[0016] The scheme of the present invention is applicable to fish with different morphological characteristics; the scheme of the present invention designs two wearable structural transpositions for three types of fish, namely, fusiform body, flattened body and flat body. This is due to the differences in the movement modes of flat body fish, and there are differences in the deployment of detection units on the dorsal fin or the skirt, but different structures can use the same heart rate analysis method and motion analysis method, and the structural method has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 Flow chart of the method of the present invention.

[0019] Figure 2 It is a schematic diagram of the wearable structure of the spindle body and the flat body fish of the present invention.

[0020] Figure 3 This is a schematic diagram of the wearable structure of a flat fish according to the present invention.

[0021] FIG. 4 ( a ) is a schematic diagram of the fixed bracket in the solution of the present invention without clamping.

[0022] FIG. 4( b ) is a schematic diagram of clamping the fixed bracket in the solution of the present invention.

[0023] FIG. 5 ( a ) is a front view of a tail fin of a partially enlarged structure of the magnetic attraction portion of the optical detection unit structure in the solution of the present invention.

[0024] FIG. 5( b ) is a side view of a fish fin of a partially enlarged structure of the magnetic attraction portion of the optical detection unit structure in the solution of the present invention.

[0025] FIG. 5( c ) is a front view of a dorsal fin of a partially enlarged structure of the magnetic attraction portion of the optical detection unit structure in the solution of the present invention.

[0026] Among them, 1 is a fixed bracket; 2 is a flexible main control unit; 3 is an optical detection unit; 4 is a flexible carrier board; 5 is an optical fiber signal line; 6 is a flexible ring; 7 is a carbon fiber card clip; 8 is an LED emission source port; 9 is a signal receiving port; 10 is a distributed magnetic attraction point array; 11 is a fish fin ray; 12 is a fish fin membrane; 13 is a fish fin. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] like Figure 2 , 3 As shown, this example uses fish with spindle and flat body as test objects for explanation. The fish body nondestructive monitoring device and scheme based on multimodal data of this embodiment includes a fixed bracket 1, a flexible main control unit 2, and an optical detection unit 3. The three core parts are connected by a flexible carrier and a signal line including a flexible material, and the entire nondestructive monitoring device is constrained by a flexible ring. Among them, as shown in Figure 4 (a) and Figure 4 (b), the fixed bracket 1 is a card-type clip structure, which is composed of a carbon fiber card-type clip 7 and a flexible material; and the flexible main control unit 2 is encapsulated with a microprocessor, a temperature and pressure sensor tag, a 9-axis IMU, a cellular Bluetooth module and an embedded small lithium battery; as shown in Figure 5 (a), Figure 5 (b), and Figure 5 (c), the optical detection unit 3 is composed of an LED emission source port 8, a signal receiving port 9, and a distributed magnetic absorption dot matrix, and the magnetic absorption dot matrix is ​​connected by a flexible transparent material.

[0030] According to the size of the fish, adjust the size of the flexible ring and put the whole wearable device on the fish. Figure 2 As shown in Figure 4 (a) and Figure 4 (b), the fixing bracket 1 is fixed to the fish bone at the rear part of the gills to ensure that the device is fixed; then, as shown in Figure 5 (a), Figure 5 (b), and Figure 5 (c), the optical detection unit 3 is aligned and fitted from both sides of the dorsal fin (or skirt) and the tail fin. At this time, the distributed magnetic point array 10 is on both sides of the fin ray 11 of the fish fin, and the LED emission source port 8 and the signal receiving port 9 are distributed on the fin membrane 12 of the fish fin; after the above process, the wearing of the non-destructive monitoring device is completed. During the data collection process, the flexible main control unit 2 controls the signal emission from the optical fiber signal line 5 to the LED emission source port 8, and then the corresponding signal receiving port 9 receives the projected light signal and transmits it to the MCU. The MCU stores the synchronously collected 9-axis IMU data in the built-in SD card according to the timestamp, and uploads it to the cloud synchronously to complete the fish body status data collection.

[0031] The process of a method for distinguishing fish state based on a nondestructive detection device for fish state based on multimodal data in this embodiment is as follows: Figure 1 A method for distinguishing fish state based on a fish state nondestructive detection device of multimodal data in this embodiment comprises the following steps: (1) Signal acquisition: The LED light source in the optical detection unit fixed at different parts of the fin emits light to the fin at a set frequency. After the light passes through the arterial capillaries of the fin, it is received by the probe on the other side of the corresponding fin to obtain a transmission spectrum combination signal. According to the Lambert-Beer law, at time t, the spectrum combination signal obtained from different parts of the fin includes the proximal projection spectrum signal representing the pectoral fin or skirt. and the distal projection spectral signal representing the tail fin , the specific formula is as follows: (1) (2) in, and are the near-end and far-end emission light intensity at time t, and are the water film attenuation coefficients of the near-end and far-end detection units, and are the optical path lengths of the near-end and far-end detection units, respectively.

[0032] (2) Preprocessing: Low-pass filtering, morphological filtering and other filtering are used to denoise and process the electromyographic noise and baseline drift caused by respiratory fluctuations, muscle activity and amplification circuits, so as to improve the quality of PPG signals and obtain optical signals. Multiple filtering methods are used to denoise and correct drift of IMU signals. As a preferred processing method, extended Kalman filtering is used to obtain optical signals.

[0033] (3) Motion correction: Based on the preprocessed optical signal and the transmission spectrum combined signal, deep learning and machine learning methods are used to extract signal features and eliminate motion artifacts in the PPG signal; preferably, the wavelet transform method is used to eliminate motion artifacts: A. For PPG signal , And the IMU signal M Using wavelet transform, we can get the optical wavelet coefficients and kinematic wavelet coefficients. , and : (3); (4); (5); in, is the wavelet basis function; is a scale parameter that affects the details of frequency analysis; It is the time shift parameter, which locates the time effect of the signal; Expressed as complex conjugate.

[0034] B. Analyzing Motion Wavelet Coefficients , identify the main frequency components caused by motion , compare and obtain the optical wavelet coefficients with Wavelet coefficients of energy distributions of similar or same scale and , then the corrected wavelet coefficients after eliminating motion artifacts are obtained and The calculation formula is as follows: (6); (7); in, and They are artifact elimination factors for different parts, which are used to control and adjust the degree of motion artifact elimination.

[0035] C. Reconstruct PPG signals at different parts using the corrected wavelet coefficients and , and obtain the comprehensive PPG signal for accurate calculation of the fish's heart rate , the formula is as follows:

[0036] in, and The sum is 1, which is used to adjust the signal strength in different parts.

[0037] (4) State discrimination: Based on the calibrated optical and kinematic signals, as well as environmental data such as temperature and pH, a feature extraction algorithm is used to measure multiple states of the fish, such as heart rate, movement direction, and speed. The health status and behavior knowledge base built based on expert experience is combined to realize fish state discrimination.

[0038] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A nondestructive detection method for fish body status based on multimodal data, characterized in that: The following steps are involved: Step 1: signal acquisition: the LED light source in the optical detection unit fixed at different parts of the fish fin emits light at a set frequency to the fish fin. After the light passes through the arterial capillaries of the fish fin, the probe on the other side of the corresponding fish fin receives the transmitted light source to obtain a transmitted spectrum combination signal; Step 2: preprocessing the transmission spectrum combined signal; performing denoising and baseline drift processing on the electromyographic noise and baseline drift to obtain an optical signal; Step 3: Perform motion correction; according to the optical signal and the transmission spectrum combination signal, use deep learning and machine learning methods to extract signal features and eliminate motion artifacts in the photoplethysmography signal; Step 4: Perform state discrimination; based on the corrected optical signal, the transmission spectrum combination signal and the environmental data, a feature extraction algorithm is used to obtain the time domain and frequency domain features of the multivariate signal; The time domain and frequency domain characteristics of multivariate signals are used to identify the state of fish.

2. According to claim 1, a method for nondestructive detection of fish body status based on multimodal data is characterized in that: In step 1, according to the Lambert-Beer law, at time t, the spectral combination signals obtained from different parts of the fins include: the proximal projection spectral signal of the pectoral fin or skirt and the distal projection spectral signal of the tail fin : （1）； （2）； in, and Respectively t The intensity of the near and far emitting light sources at the moment; and denote the water film attenuation coefficients of the proximal and distal detection units, respectively; and Represent the optical path lengths of the near-end and far-end detection units, respectively.

3. The method for nondestructive detection of fish body status based on multimodal data according to claim 1, characterized in that: The step 3 comprises the following steps: Step 31: For PPG Signal , And the IMU signal M Using wavelet transform, we can get the optical wavelet coefficients and kinematic wavelet coefficients. , and : （3）； （4）； （5）； in, represents the wavelet basis function; Represents the scale parameter, which affects the details of frequency analysis; Indicates the time shift parameter, and the time effect of the positioning signal; Expressed as complex conjugate; Step 32: Analyze motion wavelet coefficients , identify the main frequency components caused by motion , compare and obtain the optical wavelet coefficients with Wavelet coefficients of energy distributions of similar or same scale and , get the corrected wavelet coefficients after eliminating motion artifacts and The calculation formula is: （6）； （7）； in, and Respectively represent the artifact elimination factors of the near end and the far end, which are used to control and adjust the degree of motion artifact elimination; Step 33: Reconstruct the pectoral fin or skirt and caudal fin PPG signals using the corrected wavelet coefficients and , and obtain the comprehensive PPG signal for accurate calculation of the fish's heart rate , the formula is: in, and The sum is 1, which is used to adjust the signal strength in different parts.

4. A non-destructive detection device for fish body status based on multimodal data, applying the detection method according to any one of claims 1 to 3, characterized in that: include: A fixed bracket, a flexible main control unit and an optical detection unit; the fixed bracket, the flexible main control unit and the optical detection unit are connected through a flexible carrier board and a signal line wrapped with a flexible material; the fish body state nondestructive detection device realizes the constraint of the device position through a flexible ring; The fixing bracket is a slightly elastic rigid card-type clip structure, which can clamp and fix the fish bones near the gills; the fixing bracket is encapsulated by a flexible material; The device is fixed to the fish body without loss by the fixing bracket, and the optical detection unit is controlled to transmit a signal by the flexible main control unit; the transmission signal is transmitted to the LED transmission source port through the optical fiber signal line, and the signal end receiving port corresponding to the LED transmission source port receives the projected light signal and transmits it to the flexible main control unit; The flexible main control unit stores the collected data to complete the fish body status data collection.

5. The fish body state nondestructive detection device based on multimodal data according to claim 4 is characterized in that: The flexible main control unit includes: a microprocessor MCU, a multi-sensor, a cellular Bluetooth module and an embedded small lithium battery; the microprocessor MCU is connected to the multi-sensor, the cellular Bluetooth module and the embedded small lithium battery; the cellular Bluetooth module is connected to the microprocessor MCU via a serial communication interface to support remote data transmission and wireless communication functions between devices; the embedded small lithium battery provides a stable operating voltage for the flexible main control unit; the flexible main control unit is encapsulated in a flexible material.

6. The fish body state nondestructive detection device based on multimodal data according to claim 4 is characterized in that: The optical detection unit includes: a distributed magnetic attraction point array, a signal transmitting port and a signal receiving port; The distributed magnetic attraction point array is distributed on both sides of the fin rays of the fish fin to fix the optical detection unit; the signal transmitting port is aligned and fit with the signal receiving port, and are distributed on both sides of the fin membrane of the fish fin; the distributed magnetic attraction point array is connected to the signal transmitting port through a flexible material; the distributed magnetic attraction point array is connected to the signal receiving port through a flexible material.

Images