A non-destructive detection method and device for fish body state based on multi-modal data

Abstract

The present invention provides a non-destructive detection method and device for fish body state based on multi-modal data, belonging to the technical field of fish state monitoring in aquaculture. The device includes: a fixed bracket, a flexible main control unit, and an optical detection unit; they are connected and position-constrained through a flexible carrier board, a signal line, and a flexible ring. The fixed bracket is non-destructively fixed to the fish bone near the fin in a clamping manner, and the overall device controls the acquisition, storage, and transmission of data through the flexible main control unit. The present invention also discloses a non-destructive detection method, including the following steps: signal acquisition; collecting combined optical signals, kinematic signals, and environmental data; performing preprocessing; then performing motion correction, eliminating motion artifacts in the optical signals and reconstructing the signals by combining kinematic signals with a specific algorithm; finally realizing state discrimination, and realizing fish state discrimination based on multi-modal data and integrating an expert knowledge base.

Description

Technical Field

[0001] The present invention relates to the technical field of fish state monitoring. Specifically, it particularly relates to a non-destructive detection method and device for fish body state based on multi-modal data. Background Art

[0002] In fishery aquaculture and fish physiology research, the detection of fish state (including behavior and physiological state) and environmental detection are of great significance for production aquaculture and scientific research. With the research and development of sensor technology, currently, implantable devices, flexible wearable devices, etc. are applied to the in-situ detection of fish body state. However, implantable devices often require anesthetizing fish and puncturing fish body tissues, which has a destructive impact on fish organ tissues, affecting the survival rate and normal living state of fish. For example, flexible wearable sensors solve the above problems to a certain extent, but due to the non-adhesive property caused by fish mucus, wearable devices cannot be attached to the fish body in a fitting manner for a long time. Fixing them to the fish tail by tying is a solution, but these devices can only be used to detect the kinematic characteristics (acceleration) of the fish body and environmental parameters mainly, and cannot effectively monitor physiological states such as the heart rate of the fish body, such as "marine skin".

[0003] Some researchers have tried to tie flexible wearable devices to zebrafish to detect the heart rate of the fish body. However, for larger fish bodies such as largemouth bass, the ECG signals obtained based on the chest cavity will have the situation where noise covers the heart rate. In addition, this method cannot be used to measure the fish bodies of flatfish by tying. Summary of the Invention

[0004] In view of the technical problems mentioned in the above background art, in order to non-destructively obtain fish state (motion information, heart rate information), a non-destructive detection method and device for fish body state based on multi-modal data are provided.

[0005] The technical means adopted by the present invention are as follows:

[0006] A non-destructive detection method for fish body state based on multi-modal data, comprising the following steps:

[0007] Step 1: Signal acquisition; the LED light source in the optical detection unit fixed on different fins of the fish emits light to the fin at a set frequency. After the light passes through the arterial capillaries of the fin, the transmitted light source is received by the probe on the other side of the corresponding fin, and a transmitted spectrum combined signal is obtained.

[0008] Step 2: Preprocess the transmitted spectrum combined signal; perform denoising processing and baseline drift processing on the electromyographic noise and baseline drift to obtain an optical signal.

[0009] Step 3: Perform motion correction; extract signal features using deep learning and machine learning methods based on the optical signal and the transmitted spectrum combined signal, and eliminate motion artifacts in the photoplethysmogram signal.

[0010] Step 4: Perform state discrimination; obtain the time-domain and frequency-domain features of the multi-signal using a feature extraction algorithm based on the corrected optical signal, the transmitted spectrum combined signal, and the environmental data; realize fish state discrimination based on the time-domain and frequency-domain features of the multi-signal.

[0011] Further, in the said Step 1, according to Lambert-Beer's law, at time t, the spectral combined signal obtained from different parts of the fish fin includes: the proximal projection spectral signal of the pectoral fin or the skirt and the distal projection spectral signal of the caudal fin :

[0012] (1);

[0013] (2);

[0014] Wherein, and respectively represent t the proximal and distal emission light source intensities at time and respectively represent the water film attenuation coefficients of the proximal and distal detection units; and respectively represent the optical paths of the proximal and distal detection units.

[0015] Further, the said Step 3 includes the following steps:

[0016] Step 31: For the PPG signal , and the IMU signal M perform wavelet transform to obtain the optical wavelet coefficients and the kinematic wavelet coefficients , and :

[0017] (3);

[0018] (4);

[0019] (5);

[0020] Wherein, represents the wavelet basis function; represents the scale parameter, which affects the frequency analysis details; Represents the time translation parameter, which locates the time effect of the signal; Is represented as the complex conjugate;

[0021] Step 32: Analyze the motion wavelet coefficients , identify the main frequency components caused by motion , compare and obtain the wavelet coefficients of the energy distribution with similar or the same scale in the optical wavelet coefficients , and obtain the corrected wavelet coefficients after removing motion artifacts and , and obtain the corrected wavelet coefficients after removing motion artifacts and The calculation formula is:

[0022] (6);

[0023] (7);

[0024] Among them, and respectively represent the artifact cancellation factors at the proximal and distal ends, which are used to control and adjust the degree of motion artifact cancellation;

[0025] Step 33: Reconstruct the PPG signals of the pectoral fin or skirt and the caudal fin through the corrected wavelet coefficients and , and obtain the comprehensive PPG signal for accurately calculating the heart rate of the fish body , the formula is:

[0026]

[0027] Among them, and sum to 1, which is used to adjust the signal intensity of different parts.

[0028] The present invention also includes a non-destructive detection device for the fish body state based on multi-modal data, including: a fixed bracket, a flexible main control unit, and an optical detection unit; the fixed bracket is connected to the flexible main control unit and the optical detection unit through a flexible carrier plate and a signal line wrapped with a flexible material; the non-destructive monitoring device realizes the constraint of the device position through a flexible ring;

[0029] The fixed bracket is a micro-elastic rigid clip structure, which realizes clamping and fixing on the fish bone near the gill; the fixed bracket is encapsulated with a flexible material;

[0030] The device is fixed to the fish body without damage through the fixing bracket, and the flexible main control unit controls the optical detection unit to emit signals; the emitted signals are transmitted to the LED emission source port through the optical fiber signal line, and the signal end receiving port corresponding to the LED emission source port receives the projected optical signal and transmits it to the flexible main control unit; the flexible main control unit stores the collected data to complete the acquisition of fish body state data.

[0031] Further, the flexible main control unit includes: a microprocessor MCU, a multi-sensor, a cellular Bluetooth module, and an embedded small lithium battery; the microprocessor MCU is connected to the multi-sensor, the cellular Bluetooth module, and the embedded small lithium battery; the cellular Bluetooth module is connected to the microprocessor MCU through a serial communication interface and is used to support remote data transmission and wireless communication functions between devices; the embedded small lithium battery provides a stable working voltage for the flexible main control unit; the flexible main control unit is encapsulated in a flexible material.

[0032] Further, the optical detection unit includes: a distributed magnetic adsorption point array, a signal emission port, and a signal reception port;

[0033] The distributed magnetic adsorption point array is distributed on both sides of the fish fin rays to realize the fixation of the optical detection unit; the signal emission port and the signal reception port are aligned and attached, and are distributed on both sides of the fish fin membrane; the distributed magnetic adsorption point array is connected to the signal emission port through a flexible material; the distributed magnetic adsorption point array is connected to the signal reception port through a flexible material.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] The solution of the present invention realizes non-destructive and accurate estimation of the fish body heart rate; for the acquisition of the heart rate, the method of the present invention does not adopt the conventional ECG heart rate detection method, but extracts the heart rate signal by the designed device and method using the optical method and the kinematic method, and does not need to perform destructive treatment on the fish body, reducing the impact of the device on the physiological health state of the fish body.

[0036] The solution of the present invention adopts a wearable solution based on the structural characteristics of the fish body; according to the characteristics of the hard fish bones near the gills of the fish body, a clamping structure designed with composite materials is adopted to reduce the interference of the wearable device wrapping the whole fish body on the fish movement behavior and avoid the problem that the device cannot be glued due to the mucus continuously secreted by the fish body; at the same time, for the clamping structure of the fish fin part, a flexible material combined with a magnetic adsorption type clamping structure is adopted to meet the detection of optical signals while not interfering with the movement of the fish fin.

[0037] This research plan constructs a heart rate acquisition method based on optical signals using the fin structural characteristics; this method is based on the physiological structural characteristics of high penetrability of the fish fins / skirt of the fish body and the abundant distribution of arterial capillaries, designs an optical signal acquisition structure, and combines a heart rate estimation method adapted to the structure to accurately measure the heart rate of the fish body based on the optical PPG signal.

[0038] The solution of the present invention makes full use of optical information and kinematic information; this method makes full use of and couples the optical information and kinematic information of different parts of the fish fins, eliminates the corresponding motion artifacts in the PPG signal based on kinematic information, and realizes the accurate measurement of the heart rate signal. Compared with the weak ECG signal which is only applicable to tiny fish bodies, this method can be applied to large fish bodies; by adopting multi-modal information, it can more comprehensively and accurately judge the state of the fish body.

[0039] The solution of the present invention is applicable to fish bodies with different morphological characteristics; the present invention designs two wearable structure transpositions for fish bodies of three morphological types: spindle-shaped, laterally compressed, and flat-shaped. This is due to the differences in the movement modes of flat-shaped fish bodies, and there are differences in the deployment of the detection unit on the dorsal fin or skirt. However, the same heart rate analysis method and motion analysis method can be adopted for different structures, and the structural method has strong versatility. Brief Description of the Drawings

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0041] Figure 1 It is a flow chart of the method of the present invention.

[0042] Figure 2 It is a schematic diagram of the wearable structure for spindle-shaped and laterally compressed fish of the present invention.

[0043] Figure 3 It is a schematic diagram of the wearable structure for flat-shaped fish of the present invention.

[0044] Figure 4(a) is a schematic diagram of the non-clamped fixing bracket in the solution of the present invention.

[0045] Figure 4(b) is a schematic diagram of the clamped fixing bracket in the solution of the present invention.

[0046] Figure 5(a) is a front view of the tail fin of the partial enlarged structure of the magnetic attraction part in the optical detection unit structure in the solution of the present invention.

[0047] Figure 5(b) is a side view of the fish fin of the partial enlarged structure of the magnetic attraction part in the optical detection unit structure in the solution of the present invention.

[0048] Figure 5(c) is a front view of the dorsal fin of the partial enlarged structure of the magnetic attraction part in the optical detection unit structure in the solution of the present invention.

[0049] Among them, 1 is a fixed bracket; 2 is a flexible main control unit; 3 is an optical detection unit; 4 is a flexible carrier board; 5 is an optical fiber signal line; 6 is a flexible ring; 7 is a carbon fiber clamping piece; 8 is an LED emission source port; 9 is a signal receiving port; 10 is a distributed magnetic attraction point array; 11 is a fish fin ray; 12 is a fish fin membrane; 13 is a fish fin. Specific embodiments

[0050] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0051] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order different from those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0052] Such as Figure 2 、 3As shown in the figure, in this example, fish in the shape of spindles and flat bodies are used as test objects for illustration. The fish body non-destructive monitoring device and solution based on multi-modal data in this embodiment include a fixed bracket 1, a flexible main control unit 2, and an optical detection unit 3. The three core parts are connected by a flexible carrier board and signal lines included in the flexible material, and the entire non-destructive monitoring device is positionally constrained by a flexible ring. Among them, as shown in Fig. 4(a) and Fig. 4(b), the fixed bracket 1 is a clip structure, composed of a carbon fiber clip 7 and a flexible included material; while the flexible main control unit 2 is encapsulated with a microprocessor, temperature and pressure sensing tags, a 9-axis IMU, a cellular Bluetooth module, and an embedded small lithium battery; as shown in Fig. 5(a), Fig. 5(b), and Fig. 5(c), the optical detection unit 3 is composed of an LED emission source port 8, a signal receiving port 9, and a distributed magnetic attraction dot matrix, and the magnetic attraction dot matrix is connected by a flexible transparent material.

[0053] According to the size of the fish body, adjust the size of the flexible ring, and put the whole set of wearable devices on the fish body, as Figure 2 As shown in Fig. 4(a) and Fig. 4(b), the fixed bracket 1 is fixed to the fish bone at the back of the fish gill to ensure the fixation of the device; then, as shown in Fig. 5(a), Fig. 5(b), and Fig. 5(c), align and attach the optical detection unit 3 from both sides of the fish dorsal fin (or skirt) and the caudal fin. At this time, the distributed magnetic attraction dot array 10 is on both sides of the fish fin rays 11, and the LED emission source port 8 and the signal receiving port 9 are distributed on the fish fin membranes 12; after the above process, the wearing of the non-destructive monitoring device is completed. During the data acquisition process, the flexible main control unit 2 controls the signal to be transmitted from the optical fiber signal line 5 to the LED emission source port 8, and then the corresponding signal receiving port 9 receives the projected light signal and transmits it to the MCU. The MCU stores the synchronously collected 9-axis IMU data in the built-in SD card according to the time stamp and synchronously uploads it to the cloud to complete the acquisition of the fish body state data.

[0054] The flow of the fish state discrimination method of a fish body state non-destructive detection device based on multi-modal data in this embodiment is as Figure 1 shown. The fish state discrimination method of a fish body state non-destructive detection device based on multi-modal data in this embodiment includes the following steps:

[0055] (1) Signal acquisition: The LED light source in the optical detection unit fixed on different parts of the fish fin emits light to the fish fin at a set frequency. The light passes through the arterial capillaries of the fish fin and is received by the probe on the other side of the corresponding fish fin to obtain a transmitted spectrum combined signal; according to the Lambert-Beer law, at time t, the spectrum combined signal obtained from different parts of the fish fin contains a proximal projected spectrum signal representing the pectoral fin or skirt and a distal projected spectrum signal representing the caudal fin , and the specific formula is as follows:

[0056] (1)

[0057] (2)

[0058] Wherein, and are the proximal and distal emission light source intensities at time t, respectively, and are the water film attenuation coefficients of the proximal and distal detection units, respectively, and are the optical paths of the proximal and distal detection units, respectively.

[0059] (2) Preprocessing: Denoise and perform baseline drift processing on the electromyographic noise and baseline drift caused by respiratory fluctuations, muscle activities, and amplifier circuits, etc. by using filters such as low-pass filtering and morphological filtering to improve the quality of the PPG signal and obtain the optical signal; perform denoising and drift correction on the IMU signal by using a variety of filters. As a preferred method, the extended Kalman filter is used as the processing method to obtain the optical signal.

[0060] (3) Motion correction: Based on the preprocessed optical signal and transmission spectrum combined signal, use deep learning and machine learning methods to extract signal features and eliminate motion artifacts in the PPG signal; preferably, use the wavelet transform method to eliminate motion artifacts:

[0061] A. For the PPG signal , and the IMU signal M perform wavelet transform to obtain the optical wavelet coefficients and kinematic wavelet coefficients , and :

[0062] (3);

[0063] (4);

[0064] (5);

[0065] Wherein, is the wavelet basis function; is the scale parameter, which affects the frequency analysis details; is the time translation parameter, which locates the time effect of the signal; represents the complex conjugate.

[0066] B. Analyze the motion wavelet coefficients , identify the main frequency components caused by motion, compare and obtain those in the optical wavelet coefficients Wavelet coefficients of energy distribution at similar or the same scale and then the corrected wavelet coefficients after eliminating motion artifacts are obtained and The calculation formulas are as follows:

[0067] (6);

[0068] (7);

[0069] Wherein, and are respectively the artifact elimination factors for different parts, used to control and adjust the degree of motion artifact elimination.

[0070] C. Reconstruct the PPG signals of different parts using the corrected wavelet coefficients and to obtain the comprehensive PPG signal for accurately calculating the fish heart rate The formula is as follows:

[0071]

[0072] Wherein, and sum to 1, used to adjust the signal intensity of different parts.

[0073] (4) State discrimination: Based on the corrected optical signal and kinematic signal, as well as environmental data such as temperature and PH, a feature extraction algorithm is used to measure multiple states such as fish heart rate, movement direction, and speed, and the fish state discrimination is realized by combining the health state and behavior knowledge base constructed based on expert experience.

[0074] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments. In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways.

[0075] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A nondestructive detection method for fish body status based on multimodal data, characterized in that: The following steps are involved: Step 1: signal acquisition; light is emitted to the fish fins at a set frequency through the LED emission source ports in the optical detection units respectively fixed at different parts of the fish fins, and the light passes through the arterial capillaries of the fish fins and then receives the transmitted light at the signal receiving port on the other side of the corresponding fish fins to obtain a transmission spectrum combination signal; the different parts of the fish fins include: dorsal fin, pectoral fin and caudal fin; Step 2: preprocessing the transmission spectrum combined signal; performing denoising and baseline drift processing on the electromyographic noise and baseline drift to obtain an optical signal; Step 3: Perform motion correction; according to the optical signal, the transmission spectrum combined signal and the inertial measurement unit IMU signal, use deep learning and machine learning methods to extract signal features and eliminate motion artifacts in the optical signal; Step 4: Perform state discrimination; based on the corrected optical signal, the transmission spectrum combination signal and the environmental data, a feature extraction algorithm is used to obtain the time domain and frequency domain characteristics of the multivariate signal; the time domain and frequency domain characteristics of the multivariate signal are used to discriminate the state of the fish body; the environmental data are: temperature, pH value; According to the Lambert-Beer law, at time t, the transmission spectrum combination signal obtained by the optical detection unit of different parts of the fins includes: the proximal transmission spectrum signal of the pectoral fin or dorsal fin And the distal transmission spectrum signal of the tail fin , the two constitute the transmission spectrum combination signal: （1）； （2）； in, and Respectively t The emission light intensity of the near-end and far-end LED emission source ports at the moment; and denote the water film attenuation coefficients of the near-end and far-end optical detection units, respectively; and denote the optical path lengths of the near-end and far-end optical detection units, respectively; The step 3 comprises the following steps: Step 31: Proximal transmission spectrum signal for pectoral or dorsal fins , distal transmission spectrum signal of the tail fin And the inertial measurement unit IMU signal M Using wavelet transform, we can get the optical wavelet coefficients and kinematic wavelet coefficients. , and : （3）； （4）； （5）； in, represents the wavelet basis function; Represents the scale parameter, which affects the details of frequency analysis; Indicates the time shift parameter, and the time effect of the positioning signal; Expressed as complex conjugate; Step 32: Analyze motion wavelet coefficients , identify the main frequency components caused by motion , compare and obtain the optical wavelet coefficients with Wavelet coefficients of energy distributions of similar or same scale and , get the corrected wavelet coefficients after eliminating motion artifacts and The calculation formula is: （6）； （7）； in, and Respectively represent the artifact elimination factors of the near end and the far end, which are used to control and adjust the degree of motion artifact elimination; Step 33: Reconstruct the pectoral fin or skirt and caudal fin PPG signals by correcting the wavelet coefficients and , and obtain the comprehensive PPG signal for accurate calculation of the fish's heart rate , the formula is: in, and The sum is 1, which is used to adjust the signal strength in different parts.

2. A non-destructive detection device for fish body status based on multimodal data, applying the detection method described in claim 1, characterized in that: include: A fixed bracket, a flexible main control unit and an optical detection unit; the fixed bracket, the flexible main control unit and the optical detection unit are connected through a flexible carrier board and a signal line wrapped with a flexible material; the fish body state nondestructive detection device realizes the constraint of the device position through a flexible ring; The fixing bracket is a slightly elastic rigid card-type clip structure, which can clamp and fix the fish bones near the gills; the fixing bracket is encapsulated by a flexible material; The device is fixed to the fish body without loss by the fixing bracket, and the optical detection unit is controlled to transmit a signal by the flexible main control unit; the transmission signal is transmitted to the LED transmission source port through the optical fiber signal line, and the signal end receiving port corresponding to the LED transmission source port receives the transmission spectrum combination signal and transmits it to the flexible main control unit; The flexible main control unit stores the collected data to complete the fish body status data collection.

3. The fish body state nondestructive detection device based on multimodal data according to claim 2 is characterized in that: The flexible main control unit includes: a microprocessor MCU, a multi-sensor, a cellular Bluetooth module and an embedded small lithium battery; the microprocessor MCU is connected to the multi-sensor, the cellular Bluetooth module and the embedded small lithium battery; the cellular Bluetooth module is connected to the microprocessor MCU via a serial communication interface to support remote data transmission and wireless communication functions between devices; the embedded small lithium battery provides a stable operating voltage for the flexible main control unit; the flexible main control unit is encapsulated in a flexible material.

4. The fish body state nondestructive detection device based on multimodal data according to claim 2 is characterized in that: The optical detection unit includes: a distributed magnetic attraction point array, a signal transmitting port and a signal receiving port; The distributed magnetic attraction point array is distributed on both sides of the fin rays of the fish fin to fix the optical detection unit; the signal transmitting port is aligned and fit with the signal receiving port, and are distributed on both sides of the fin membrane of the fish fin; the distributed magnetic attraction point array is connected to the signal transmitting port through a flexible material; the distributed magnetic attraction point array is connected to the signal receiving port through a flexible material.

Images