Fish health discrimination method, system and device based on multi-modal feature fusion

Abstract

The invention provides a fish health discrimination method, system and device based on multi-modal feature fusion, and relates to the technical field of information processing. According to the invention, the online monitoring and intelligent discrimination of fish health are realized by collecting physiological, environmental, behavior, visual and blood glucose indexes, constructing a fusion and reference data set, respectively identifying abnormal parameters and calculating weight factors, obtaining the cumulant of health degree parameters, and outputting the fish health state through a pre-trained network.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and in particular to a method, system and device for fish health assessment based on multimodal feature fusion. Background Technology

[0002] In fish farming, health monitoring methods are often harmful, including sampling of biochemical indicators, observing the mucus on the fish's body surface, color, and respiratory rate to detect the fish's stress status.

[0003] Furthermore, these traditional methods for determining the health level of fish are usually manual and have a very low level of intelligence. Most research indicators for fish health monitoring are relatively singular and cannot comprehensively reflect the cumulative health changes in the organism. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of this invention is to provide a method, system and device for fish health assessment based on multimodal feature fusion, which realizes online monitoring and management of fish health.

[0005] To achieve the above objectives, the present invention provides the following solution: A fish health assessment method based on multimodal feature fusion includes: Collect physiological health indicators, environmental health indicators, behavioral health indicators, visual health indicators, and blood glucose health indicators to construct a multimodal feature fusion network dataset; Based on the multimodal feature fusion network dataset, a fusion dataset and a reference dataset are generated respectively; Based on the fused dataset and the preset fusion function, a parameter outlier range identification partition is constructed and a health correction factor is determined; Based on the reference dataset and the preset reference function, a correlation coefficient matrix is ​​constructed and health weight factors are determined; The cumulative amount of fish health parameters is calculated by combining the health correction factor and the health weighting factor; The accumulated fish health parameters are input into a pre-trained multimodal feature fusion network, which outputs the fish health status judgment result.

[0006] Preferably, the fused dataset includes time-series datasets of all physiological health indicators, environmental health indicators, behavioral health indicators, and visual health indicators related to fish health; the physiological health indicators include electrocardiogram parameters, electromyogram parameters, and body temperature parameters of fish; the environmental health indicators include temperature parameters and humidity parameters of fish; the behavioral health indicators include X-axis acceleration parameters, Y-axis acceleration parameters, and Z-axis acceleration parameters of fish; the visual health indicators include infrared image parameters and RGB image parameters of fish; and the reference dataset includes a blood glucose dataset related to fish health.

[0007] Preferably, the expressions for the fusion function and the reference function are as follows: ; ; Where C1 is the fused dataset, R d C1 represents the parameters of the physiological health indicator, the environmental health indicator, the behavioral health indicator, and the visual health indicator; d represents the dimension of the fused dataset; C2 represents the parameter dataset; R represents the parameter dataset. m Let m be the blood glucose health index parameter, m be the dimension of the reference dataset, and t1 and t2 represent the start and end times of the physiological health and environmental health indicators, respectively. a For health parameters, w( a ) is the health-weighted function.

[0008] Preferably, based on the fused dataset and a preset fusion function, a parameter outlier range identification partition is constructed and a health correction factor is determined, including: The fused dataset is processed based on the fusion function to obtain a fused feature map; Cluster analysis and anomaly detection are performed on the fused feature map to identify partitions containing abnormal health signals, which are used as parameter anomaly range identification partitions; the expression for the parameter anomaly range identification partitions is: ;in, α The expression factor for identifying partitions of the parameter's outlier range; Judge: the process of judgment or evaluation, determining whether the parameter value is within the "comfort zone" by judging the parameter value. The comfort zone refers to a preset range where certain health or status indicators are considered normal if their values ​​are within the preset range; value represents the actual value of the parameter. Extracting the range of outlier values ​​of the parameters identifies the cumulative time of each variable in the partition within a set time window Δt. ;in, This is the cumulative amount of each variable in the merged dataset over the time interval Δt; t1 and t2 represent the start and end times of the time interval Δt, respectively. The health correction factor is calculated based on the expression factors of the partition identified by the accumulated amount and the corresponding parameter outlier range; the formula for calculating the health correction factor is: ;in, Health correction factor for physiological parameters; Health correction factor for environmental parameters; Health correction factor for behavioral parameters; For visual parameter health correction factor; and These are the preprocessed RGB and IR images, respectively.

[0009] Preferably, based on the reference dataset and a preset reference function, a correlation coefficient matrix is ​​constructed and health weight factors are determined, including: The reference dataset is input into the reference function to generate a correlation coefficient matrix; the correlation coefficient matrix records the Pearson correlation coefficients between each parameter signal and the blood glucose concentration signal in the fused dataset. The correlation coefficient matrix was analyzed to obtain the correlation coefficients between each parameter and blood glucose concentration; The absolute value of the correlation coefficient is used as the health weighting factor; the formula for calculating the health weighting factor is: Where r is the correlation coefficient, m and n are the average values ​​of the parameter signal and the blood glucose signal variables, respectively, and i and j represent different indices in the summation process, i is used to sum the m variables, and j is used to sum the n variables.

[0010] Preferably, the expression for the cumulative amount of the fish health parameter is: ; Here, PHL, CHL, BHL, and VHL represent the objective functions or indicators of physical health, chemical health, biological health, and visual health, respectively, i.e., the health indicators that need to be optimized in the model. α is a constant or scaling factor used to adjust the weights of PHL, CHL, BHL, and VHL, representing the importance of each parameter in the final model. P λ C λ B λ V λ Different parameters or score sequences representing environmental health are derived based on a series of time periods or data accumulations; R + This represents the set of all positive real numbers, ensuring that health indicators and parameters are always positive.

[0011] Preferably, the discrimination formula for the fish health status discrimination result is: ; ; Where Z is the cumulative amount of the overall health parameter of the target fish from time t0 to T, SL is the health level of the target fish, and S L0 S L1 S L2 S L3 and S L4 These represent different health levels of the target fish; PHL T CHL T BHL T and VHL T Let α represent the cumulative sequences of physiological health indicators, environmental health parameters, behavioral health parameters, and visual health parameters, respectively. T L represents the set of health factors, where L is the health weighting adjustment factor. , , and These represent the physiological health weight correction factor, environmental health weight correction factor, behavioral health weight correction factor, and visual health weight correction factor, respectively.

[0012] Preferably, the multimodal feature fusion network includes a network input module, a network fusion module, and a network output module; the expressions for the network input module, the network fusion module, and the network output module are respectively: ; ; ; in, , and These represent the network input module, network fusion module, and network output module, respectively. N Indicates the total number of modes; This represents the data feature vector of the i-th mode; This represents the weight of each modality; This represents the fused feature vector; Indicates the output function; This represents the weight matrix of the network output module; This indicates the output bias parameter.

[0013] A fish health assessment system based on multimodal feature fusion includes: The data acquisition module is used to collect physiological health indicators, environmental health indicators, behavioral health indicators, visual health indicators, and blood glucose health indicators in order to construct a multimodal feature fusion network dataset. The data construction module is used to generate a fused dataset and a reference dataset based on the multimodal feature fusion network dataset, respectively; The anomaly zone identification module is used to construct anomaly value range identification partitions and determine health correction factors based on the fused dataset and a preset fusion function; The weighting factor determination module is used to construct a correlation coefficient matrix and determine health weighting factors based on the reference dataset and a preset reference function. The cumulative calculation module is used to calculate the cumulative amount of fish health parameters by combining the health correction factor and the health weight factor. The data fusion and discrimination module is used to input the accumulated amount of fish health parameters into a pre-trained multimodal feature fusion network and output the fish health status discrimination result.

[0014] A fish health assessment device based on multimodal feature fusion includes: The flexible substrate is selected from any one of polydimethylsiloxane, polyimide, polyethylene terephthalate or polyethylene naphthalate; Flexible copper circuit disposed on the flexible substrate; A microprocessor chip is integrated on the flexible copper circuit; the microprocessor chip is used to collect and process physiological parameters, behavioral parameters, environmental parameters, and blood glucose parameters; the microprocessor chip integrates a physiological signal acquisition module, a behavioral signal acquisition module, an environmental signal acquisition module, a blood glucose signal acquisition module, a data processing module, a data storage module, and a power supply module; the data storage module is any one of an SD card driver module or a flash memory driver module; the power supply module is any one of a supercapacitor, a triboelectric nanosheet, a solar electrode, a button cell, or an electrode patch; the microprocessor chip has a pre-stored computer program, which, when executed, implements all the steps of the above-mentioned fish health status determination method.

[0015] The present invention discloses the following technical effects: This invention collects physiological health indicators, environmental health indicators, behavioral health indicators, visual health indicators, and blood glucose health indicators to construct a multimodal feature fusion network dataset. Based on this dataset, a fusion dataset and a reference dataset are constructed. Anomaly range identification partitions are built using the fusion dataset and fusion function to determine health correction factors. A correlation coefficient matrix is ​​constructed using the reference dataset and reference function to determine health weight factors. The cumulative amount of fish health parameters is calculated based on the health correction factors and health weight factors, and this cumulative amount is input into the multimodal feature fusion network to accurately determine the health status of fish. This invention uses the collected and corrected four types of datasets as input to the health discrimination model, significantly improving discrimination accuracy and breaking through the traditional single-indicator evaluation system, providing a new, efficient, and reliable method for fish health monitoring. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart of the method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the technical route provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the multimodal data fusion network structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the fish health assessment system provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the working process of the flexible sensing device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of the flexible sensing device provided in an embodiment of the present invention; Figure 7 This is a schematic diagram comparing the accuracy of the models provided in the embodiments of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The purpose of this invention is to provide a method, system, and device for fish health assessment based on multimodal feature fusion. It uses multimodal sensors and an efficient and accurate multimodal data fusion method to achieve quantitative grading and evaluation of fish health, and ultimately realizes online monitoring and management of fish health.

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Figure 1 The method flowchart provided in the embodiments of the present invention is as follows: Figure 1 As shown, this invention provides a fish health assessment method based on multimodal feature fusion, comprising: Step 100: Collect physiological health indicators, environmental health indicators, behavioral health indicators, visual health indicators, and blood glucose health indicators to construct a multimodal feature fusion network dataset; Step 200: Generate a fusion dataset and a reference dataset based on the multimodal feature fusion network dataset; Step 300: Based on the fused dataset and the preset fusion function, construct a parameter outlier range identification partition and determine the health correction factor; Step 400: Based on the reference dataset and the preset reference function, construct the correlation coefficient matrix and determine the health weight factors; Step 500: Calculate the cumulative amount of fish health parameters by combining the health correction factor and the health weight factor; Step 600: Input the accumulated fish health parameters into the pre-trained multimodal feature fusion network and output the fish health status discrimination result.

[0022] like Figure 2 As shown, the steps of the technical approach in this embodiment include, but are not limited to: Step S1: Based on the physiological health indicators, environmental health indicators, behavioral health indicators, visual health indicators and blood glucose health indicators collected by the flexible sensing device, construct a multimodal feature fusion network dataset. Step S2: Construct a fusion dataset and a reference dataset based on the multimodal feature fusion network dataset; Step S3: Construct a parameter outlier range identification partition based on the fused dataset and fusion function, and determine the health correction factor; Step S4: Construct a correlation coefficient matrix based on the reference dataset and reference function to determine the health weight factor; Step S5: Calculate the cumulative amount of fish health parameters based on health correction factors and health weighting factors; Step S6: Input the accumulated health parameters into the adaptively corrected multimodal feature fusion network to achieve accurate determination of the health status of fish.

[0023] Specifically, during aquaculture, fish are subjected to prolonged abiotic stresses (temperature fluctuations, humidity, and the accumulation of harmful elements), leading to varying degrees of stress responses in individual fish, resulting in decreased health and ultimately significant economic losses. Accurately assessing the health status of fish during aquaculture provides an effective technical solution for intelligent monitoring and control of the aquaculture process.

[0024] In step S1, the method for determining the health status of fish provided by the present invention continuously collects the health characteristic parameters of the target fish using flexible sensing devices during the aquaculture process.

[0025] The health characteristic parameters include physiological parameters, environmental parameters, behavioral parameters, visual parameters, and reference parameters (i.e., various health indicators); physiological parameters include electrocardiogram parameters, electromyogram parameters, and body temperature parameters of fish; environmental parameters include temperature parameters and humidity parameters of fish; behavioral parameters include X-axis angular velocity parameters, Y-axis angular velocity parameters, and Z-axis angular velocity parameters of fish; visual parameters include infrared image parameters and RGB image parameters of fish; reference parameters include blood glucose parameters, cortisol parameters, and lactate parameters, etc., which are not specifically limited in this embodiment of the invention.

[0026] In step S2, the present invention provides a fusion dataset and a reference dataset based on the multimodal feature fusion network dataset; for example, if the collected fusion dataset is divided into four categories, then the dimension of the fusion dataset is 4, and each dimension includes multiple sub-datasets in the time series.

[0027] Furthermore, each subset of the dataset includes time series of feature parameters related to fish health. The convolution function of the fused dataset constructed from the feature parameter time series is called the fusion function, and the convolution function of the fused dataset constructed from the reference parameter time series is called the reference function.

[0028] Specifically, the fusion function design takes into account the correlation between different data types and feature representation methods. Deep network functions are employed, utilizing network mapping layers and network pooling layers for feature extraction and dimensionality reduction.

[0029] The expression for the fusion function is: The expression for the reference function is: Where C1 is the fused dataset, R dHere, d represents the parameters of the physiological health indicators, environmental health indicators, behavioral health indicators, and visual health indicators; C2 represents the dimension of the fused dataset; and R represents the parameter dataset. m Let m be the blood glucose health indicator parameter, and m be the dimension of the reference dataset.

[0030] In step S3, according to the fish health assessment method provided by the present invention, a parameter outlier range identification partition is constructed based on the fused dataset and the fusion function, and a health correction factor is determined; Specifically, when constructing a fused dataset, preprocessing and cleaning are performed on each data source, including handling missing values, outliers, and data format conversion.

[0031] Furthermore, when establishing partitions for identifying parameter outlier ranges, the feature map output by the fusion function is used to identify parameter outlier ranges by setting a threshold. In the clustering method, the dataset is first divided into multiple clusters using a clustering algorithm to ensure that data points within each cluster have similar characteristics.

[0032] ; Among them, D k This represents the data point in the k-th cluster.

[0033] Next, for each cluster C k Calculate the mean and standard deviation of the data points within the cluster: ; in, Indicates cluster C k The number of data points in the data. Indicates cluster C k The i-th data point in the dataset.

[0034] ; in, Indicates cluster C k The standard deviation of the data points.

[0035] For each data point d in cluster C i Calculate the standard deviation multiple of its distance from the mean and compare it with the threshold 'a' set in this invention: ; if If it is greater than a, then d is considered to be greater than a. i It is cluster C k Outliers.

[0036] By combining the outlier detection results from all clusters, we can obtain the outlier set A in the entire dataset: Furthermore, by combining the results of the outlier range identification partitioning, the specific values ​​and calculation methods of the fish health correction factor are determined. Real datasets of various fish species (such as sturgeon, bass, and turbot) are used for model training and testing to enhance the generalization ability of the outlier range identification partitioning.

[0037] The expression for the outlier range identification partition range is: ; The formula for calculating the cumulative amount of each variable in the fused dataset over the time period Δt is as follows: ; The formula for calculating the health correction factor is: ; in, Identify partition expression factors for outlier ranges; This is the cumulative amount of each variable in the merged dataset over the time interval Δt; t1 and t2 represent the start and end times of the time interval Δt, respectively. Health correction factor for physiological parameters; Health correction factor for environmental parameters; Health correction factor for behavioral parameters; For visual parameter health correction factor; and These are the preprocessed RGB and IR images, respectively.

[0038] In step S4, according to the fish health discrimination method provided by the present invention, a correlation coefficient matrix is ​​constructed based on the reference dataset and the second convolution function to determine the health weight factor; Specifically, when determining health weighting factors, the correlation coefficient matrix is ​​used to analyze the correlation between various parameters in the reference dataset; The formula for calculating the health weight correction factor is: ; in, The value represents the weight between each parameter in the fused dataset and the blood glucose dataset. Its value is determined by the correlation coefficient r between each parameter and the health standard value (blood glucose concentration), and its absolute value is directly used as the weight value. m and n represent the average values ​​of the parameter signal and the blood glucose signal variable, respectively. The value of r ranges from [-1, 1]. The closer the value of |r| is to 1, the stronger the correlation.

[0039] Furthermore, based on the correlation between each parameter in the correlation coefficient matrix and the health status, different weights are assigned to different parameters, i.e., feature weighting, to reflect their degree of influence on fish health.

[0040] Furthermore, based on the selected features and their corresponding weighting factors, and according to the absolute value of the correlation coefficient, a linear weighting algorithm is used to assign different weights to different parameters: ; Among them, W i It is the weight of the i-th feature, r i Let be the correlation coefficient between the i-th feature and the health status, and n be the number of features. For example, if a physiological parameter has a high correlation with the health status, then the physiological parameter is given a larger weight.

[0041] In step S5, the present invention provides a method for judging the health of fish, which calculates the cumulative amount of fish health parameters based on health correction factors and health weighting factors. The formula for calculating the cumulative amount of fish health parameters is as follows: ; Wherein, PHL, CHL, BHL and VHL represent the cumulative sequence of physiological health indicators, cumulative sequence of environmental health parameters, cumulative sequence of behavioral health parameters and cumulative sequence of visual health parameters of the target fish, respectively. These represent the normalized physiological health weight correction factor, environmental health weight correction factor, behavioral health weight correction factor, and visual health weight correction factor, respectively.

[0042] In step S6, the fish health assessment method provided by this invention inputs the accumulated amount of health parameters into a multimodal feature fusion network to accurately assess the health status of fish. The formula for assessing fish health status is as follows: ; ; Where Z is the cumulative amount of the overall health parameter of the target fish from time t0 to T, SL is the health level of the target fish, and S L0 S L1 S L2 S L3 and S L4 These represent the five health levels of the target fish.

[0043] The criteria for classifying health level SL are as follows: L0: The normalized cumulative health value Z is in the interval [0, 0.2), indicating that the fish is in excellent health. L1: The normalized cumulative health value Z is in the interval [0.2, 0.4), indicating that the fish are in good health. L2: The normalized cumulative health value Z is in the interval [0.4, 0.6), indicating that the fish's health status is average; L3: The normalized cumulative health value Z is in the interval [0.6, 0.8), indicating that the fish are in poor health. L4: The normalized cumulative health value Z is in the interval [0.8, 1], indicating that the fish's health status is very poor; Figure 3 This is a schematic diagram of the multimodal data fusion network structure provided by the present invention. (See diagram below.) Figure 3 As shown, the present invention provides a multimodal feature fusion network, characterized by comprising three modules: network input, network fusion, and network fusion. The expressions for the three modules are as follows: ; ; ; Where Network1, Network2, and Network3 represent the network input, network fusion, and network output modules, respectively; N represents the total number of modes; xi represents the data feature vector of the i-th mode; and wi represents the weight of each mode. Let f(x) represent the fused feature vector; f(x) represents the output function. This represents the weight matrix of the output layer; This indicates the output bias parameter.

[0044] Specifically, in the network input module, each dataset contains physiological health indicators, environmental health indicators, behavioral health indicators, visual health indicators, and blood glucose health indicators collected by flexible electronic devices. The fused dataset includes physiological health indicators, environmental health indicators, behavioral health indicators, and visual health indicators, while the reference dataset includes blood glucose health information; the labels of the fused dataset and the reference dataset are all one-to-one.

[0045] In data preprocessing, an enhancement module is used to resize each group of images (visual health information) to 224×224 to match the non-image layers (physiological health information, environmental health information, and behavioral health information) in the fused dataset. The input weights of the non-image layers are processed into concatenated layers, which are matched with the image input layers through their input weights.

[0046] For visual image data, a deep fusion module is used for feature extraction. Assuming i is the input image data, Conv_i represents the i-th convolutional layer, and Pool_j represents the j-th pooling layer, the input of the visual image can be represented as: ; For non-image data (physiological health data, environmental health data, and behavioral health data), a weighted fusion module is used for multimodal feature extraction: Assuming j is a non-image data time series and func_m is the feature extraction function, it can be expressed as: ; The comprehensive representation of the network input module can be obtained by concatenating or adding the feature representations of each modality: ; Concat can be a multiplication concatenation operation, an addition operation, or a combination operation of other methods, depending on the input program of the network dataset. This invention does not impose any specific limitations.

[0047] Specifically, the network fusion module includes various modules to enhance fusion performance, including a connection module that fuses physiological health parameters, behavioral health parameters, and environmental health parameters, a main network module that processes the fused data, and a feedback module before output. In the network structure, image and non-image features are divided into two channels. Image features pass through channel 1, where a weighted fusion module extracts deep features. Non-image features pass through channel 2, where a linearly weighted fusion module extracts deep features. The composition of each module will be described in detail below.

[0048] The network structure for channel 1, which processes image data, is as follows: ; Where I is the input image, F image These are image features.

[0049] The network structure for channel 2, which processes non-image data, is as follows: ; Where N represents the non-image features of the input, W is the weight matrix, b is the bias term, and F... non-image These are non-image features.

[0050] The fused non-image features are arranged into a column of color blocks, and their weights are calculated based on the color path color. The size of the color blocks is then arranged according to the weights.

[0051] ; Among them, C j For the color characteristics of the color path channel, w j These are the weighting coefficients.

[0052] Specifically, image and non-image features are deeply fused using a weighted fusion module. The fusion method is a weighted summation, combined with domain constraint methods and trajectory computation: ; in, and The fusion weights represent the importance of different feature modalities.

[0053] A region constraint method is used to spatially constrain the features, limiting their range and orientation. The maximum and minimum value ranges of the features are set, and the region constraint is performed using the following function: ; Among them, F max and F min These are the maximum and minimum values ​​of the feature.

[0054] In time series data, trajectory operations are used to analyze the temporal dimension of features. The trajectory function is expressed as: ; Where T(t) is the trajectory value at time point t, and F i (t) represents the value of the i-th feature at time t. For trajectory weights.

[0055] Specifically, in the network output module, the first and second normalization functions are used to output the regression model equation. The expressions for the N1 and N2 functions are as follows: ; Where N represents the value of the i-th element in the input vector after passing through the N1 function, i.e., the probability prediction of the i-th class; x i This represents the raw score of the i-th element in the input vector; x j Nj represents the linear output of the j-th element in the input vector; N2 represents the value of the k-th element in the input vector after passing through the N2 function, x. k This represents the linear output of the k-th element in the input vector.

[0056] ; In this function, N1 transforms each output vector into a probability distribution, ensuring that each element ranges from 0 to 1, and the sum of all elements is 1. This helps the model's output layer select the class with the highest probability as the prediction result. The N2 function smooths the N1 function, representing the activation level of neurons and converting the output vector values ​​from 0 to 1. The classification results from the N1 and N2 functions are averaged to achieve the classification function.

[0057] Figure 4 This is a schematic diagram of the fish health assessment system provided by the present invention; including: The data acquisition module M1 is used to collect physiological health indicators, environmental health indicators, behavioral health indicators, visual health indicators, and blood glucose health indicators in order to construct a multimodal feature fusion network dataset. Data construction module M2 is used to generate a fused dataset and a reference dataset based on the multimodal feature fusion network dataset; The abnormal region identification module M3 is used to construct an abnormal value range identification partition and determine the health correction factor based on the fused dataset and the preset fusion function; The weighting factor determination module M4 is used to construct a correlation coefficient matrix and determine health weighting factors based on the reference dataset and a preset reference function. The cumulative calculation module M5 is used to calculate the cumulative amount of fish health parameters by combining the health correction factor and the health weight factor. The data fusion and discrimination module M6 is used to input the accumulated amount of the fish health parameters into a pre-trained multimodal feature fusion network and output the fish health status discrimination result.

[0058] Figure 5 This is a schematic diagram of the flow structure of the flexible sensing device provided by the present invention. The present invention also provides a flexible sensing device, including a flexible substrate (such as...). Figure 5 As shown in C1), flexible circuit (such as...) Figure 5 As shown in C2), a microprocessor chip and a computer program stored on and operable on the microprocessor chip, wherein the microprocessor chip includes a physiological sensing module, a behavioral sensing module, an environmental sensing module, and a visual sensing module (such as...). Figure 5 The system comprises a blood glucose sensing module, a data processing module, a data storage module, and a power supply module (as shown in C3). The data storage module includes either an SD card module or a flash memory drive module. The power supply module includes either a supercapacitor, a triboelectric nanosheet, a solar electrode, a button cell, or an electrode patch. When the data processing module executes the computer program, it implements the steps of any of the above-described fish health status determination methods.

[0059] Specifically, the flexible substrate is one of the flexible substrates such as PDMS / PI / PET / PEN; the flexible circuit is a Cu circuit.

[0060] Figure 6 This is a three-dimensional structural diagram of the flexible sensing device provided by the present invention. It includes a three-layer structure: a chip layer, a circuit layer, and a substrate layer, as detailed below: Chip Layer L1: This layer is the core of the flexible sensing device, carrying the sensors and data acquisition elements. Typically, the chip layer includes various sensors, such as physiological health, environmental health, behavioral health, visual health, and blood glucose health chips. These sensors are responsible for collecting information on the fish's physiological health, environmental health, behavioral health, visual health, and blood glucose health.

[0061] Flexible Circuit Layer L2: This circuit layer sits above the chip layer and is responsible for processing and transmitting sensor data. This layer includes components such as data processing units and signal conditioners, used to convert and process the data collected by the sensor and transmit it to subsequent processing units.

[0062] L3 Substrate Layer: The substrate layer is the supporting structure of the entire device and the foundation of the circuit layers. It is usually made of flexible materials to ensure that the device has sufficient flexibility and adaptability to fit onto the fish or its environment, enabling accurate data acquisition and transmission.

[0063] Figure 7 The accuracy of fish health assessment using the model provided in this invention is shown in the figure. It can be seen that the accuracy of this invention is significantly higher than that of image processing algorithms and non-image processing algorithms.

[0064] These three layers together constitute the flexible sensing device, enabling comprehensive monitoring and collection of information on all aspects of fish health.

[0065] The present invention also provides a non-transient storable data processing medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the fish health status determination methods described above. Specifically, the medium may include a multi-layer network model consisting of multiple neuron layers (including an input layer, multiple hidden layers, and an output layer). Each hidden layer contains a non-linear activation function to enhance the non-linear representation capability of the network. The activation function of the output layer depends on the specific health status determination task and can be used for classification or regression problems.

[0066] During implementation, the computer program within this medium is executed by a processor to determine the health status of the fish. These steps may include preprocessing data for input into a neural network, forward propagating data to generate predictions, and taking appropriate actions based on the predictions. Furthermore, this medium is not merely a data storage medium; it is also a key tool for complex data processing and decision-making, providing an efficient and highly automated solution for fish health monitoring.

[0067] Therefore, the technological innovation of this invention lies not only in the algorithm itself, but also in how to effectively implement and deploy it in real-world data processing platforms to improve the accuracy and efficiency of fish health monitoring.

[0068] The innovation of this invention lies not only in providing a method, system, and device for fish health assessment, but also in its complexity and diversity. Data collected through a flexible sensing system, encompassing physiological health, environmental health, behavioral health, visual health, and blood glucose health indicators, forms a multimodal feature fusion network dataset, reflecting a comprehensive focus on and monitoring of fish health. Furthermore, by combining the fused dataset with a reference dataset, a fusion function is used to identify outlier ranges and construct a correlation coefficient matrix, thereby determining health correction factors and health weighting factors. This step emphasizes in-depth analysis and trade-offs of multimodal data, ensuring the accuracy and reliability of the assessment model. The depth of this invention lies in its comprehensive utilization of multiple data sources and its multimodal feature fusion strategy, as well as its refined processing in model construction and parameter optimization. Through this comprehensive approach, this invention not only improves the accuracy of fish health monitoring but also provides new ideas and methods for research and practice in related fields.

[0069] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0070] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for fish health assessment based on multimodal feature fusion, characterized in that, include: Collect physiological health indicators, environmental health indicators, behavioral health indicators, visual health indicators, and blood glucose health indicators to construct a multimodal feature fusion network dataset; Based on the multimodal feature fusion network dataset, a fusion dataset and a reference dataset are generated respectively; Based on the fused dataset and the preset fusion function, a parameter outlier range identification partition is constructed and a health correction factor is determined; Based on the reference dataset and the preset reference function, a correlation coefficient matrix is ​​constructed and health weight factors are determined; The cumulative amount of fish health parameters is calculated by combining the health correction factor and the health weighting factor; The accumulated fish health parameters are input into a pre-trained multimodal feature fusion network, which outputs the fish health status judgment result.

2. The fish health assessment method based on multimodal feature fusion according to claim 1, characterized in that, The fused dataset includes time-series datasets of all physiological health indicators, environmental health indicators, behavioral health indicators, and visual health indicators related to fish health; the physiological health indicators include electrocardiogram parameters, electromyogram parameters, and body temperature parameters of fish; the environmental health indicators include temperature parameters and humidity parameters of fish; the behavioral health indicators include X-axis acceleration parameters, Y-axis acceleration parameters, and Z-axis acceleration parameters of fish; the visual health indicators include infrared image parameters and RGB image parameters of fish; and the reference dataset includes a blood glucose dataset related to fish health.

3. The fish health assessment method based on multimodal feature fusion according to claim 1, characterized in that, The expressions for the fusion function and the reference function are as follows: ； ； Where C1 is the fused dataset, R d C1 represents the parameters of the physiological health indicator, the environmental health indicator, the behavioral health indicator, and the visual health indicator; d represents the dimension of the fused dataset; C2 represents the parameter dataset; R represents the parameter dataset. m Here, m represents the blood glucose health index parameter, m is the dimension of the reference dataset, and t1 and t2 represent the start and end times of the physiological health and environmental health indicators, respectively. a For health parameters, w( a ) is the health-weighted function.

4. The fish health assessment method based on multimodal feature fusion according to claim 1, characterized in that, Based on the fused dataset and the preset fusion function, a parameter outlier range identification partition is constructed and a health correction factor is determined, including: The fused dataset is processed based on the fusion function to obtain a fused feature map; Cluster analysis and anomaly detection are performed on the fused feature map to identify partitions containing abnormal health signals, which are used as parameter anomaly range identification partitions; the expression for the parameter anomaly range identification partitions is: ;in, α The expression factor for identifying partitions of the parameter's outlier range; Judge: the process of judgment or evaluation, determining whether the numerical value is within the "comfort zone" by judging the parameter value. The comfort zone refers to a preset range where certain health or status indicators are considered normal if their values ​​are within the preset range; value represents the actual value of the parameter. Extracting the range of outlier values ​​of the parameters identifies the cumulative time of each variable in the partition within a set time window Δt. ;in, This is the cumulative amount of each variable in the merged dataset over the time interval Δt; t1 and t2 represent the start and end times of the time interval Δt, respectively. The health correction factor is calculated based on the expression factors of the partition identified by the accumulated amount and the corresponding parameter outlier range; the formula for calculating the health correction factor is: ;in, Health correction factor for physiological parameters; Health correction factor for environmental parameters; Health correction factor for behavioral parameters; For visual parameter health correction factor; and These are the preprocessed RGB and IR images, respectively.

5. The fish health assessment method based on multimodal feature fusion according to claim 1, characterized in that, Based on the reference dataset and the preset reference function, a correlation coefficient matrix is ​​constructed and health weight factors are determined, including: The reference dataset is input into the reference function to generate a correlation coefficient matrix; the correlation coefficient matrix records the Pearson correlation coefficients between each parameter signal and the blood glucose concentration signal in the fused dataset. The correlation coefficient matrix was analyzed to obtain the correlation coefficients between each parameter and blood glucose concentration; The absolute value of the correlation coefficient is used as the health weighting factor; the formula for calculating the health weighting factor is: Where r is the correlation coefficient, m and n are the average values ​​of the parameter signal and the blood glucose signal variables, respectively, and i and j represent different indices in the summation process, i is used to sum the m variables, and j is used to sum the n variables.

6. The fish health assessment method based on multimodal feature fusion according to claim 1, characterized in that, The expression for the cumulative amount of the fish health parameter is: ； Where PHL, CHL, BHL, and VHL represent the objective functions or indicators of physical health, chemical health, biological health, and visual health, respectively, i.e., the health indicators that need to be optimized in the model; α is a constant or scaling factor used to adjust the weights of PHL, CHL, BHL, and VHL, representing the importance of each parameter in the final model; P λ C λ B λ and V λ These represent different parameters or score sequences for environmental health, derived from a series of time periods or accumulated data; R + This represents the set of all positive real numbers, ensuring that health indicators and parameters are always positive.

7. The fish health assessment method based on multimodal feature fusion according to claim 1, characterized in that, The formula for determining the health status of fish is as follows: ； ； Where Z is the cumulative amount of the overall health parameter of the target fish from time t0 to T, SL is the health level of the target fish, and S L0 S L1 S L2 S L3 and S L4 These represent different health levels of the target fish; PHL T CHL T BHL T and VHL T Let α represent the cumulative sequences of physiological health indicators, environmental health parameters, behavioral health parameters, and visual health parameters, respectively. T L represents the set of health factors, where L is the health weighting adjustment factor. , , and These represent the physiological health weight correction factor, environmental health weight correction factor, behavioral health weight correction factor, and visual health weight correction factor, respectively.

8. The fish health assessment method based on multimodal feature fusion according to claim 1, characterized in that, The multimodal feature fusion network includes a network input module, a network fusion module, and a network output module; the expressions for the network input module, the network fusion module, and the network output module are respectively: ； ； ； in, , and These represent the network input module, network fusion module, and network output module, respectively. N Indicates the total number of modes; This represents the data feature vector of the i-th mode; This represents the weight of each modality; This represents the fused feature vector; Indicates the output function; This represents the weight matrix of the network output module; This indicates the output bias parameter.

9. A fish health assessment system based on multimodal feature fusion, characterized in that, include: The data acquisition module is used to collect physiological health indicators, environmental health indicators, behavioral health indicators, visual health indicators, and blood glucose health indicators in order to construct a multimodal feature fusion network dataset. The data construction module is used to generate a fused dataset and a reference dataset based on the multimodal feature fusion network dataset, respectively; The anomaly zone identification module is used to construct anomaly value range identification partitions and determine health correction factors based on the fused dataset and a preset fusion function; The weighting factor determination module is used to construct a correlation coefficient matrix and determine health weighting factors based on the reference dataset and a preset reference function. The cumulative calculation module is used to calculate the cumulative amount of fish health parameters by combining the health correction factor and the health weight factor. The data fusion and discrimination module is used to input the accumulated amount of fish health parameters into a pre-trained multimodal feature fusion network and output the fish health status discrimination result.

10. A fish health assessment device based on multimodal feature fusion, characterized in that, include: The flexible substrate is selected from any one of polydimethylsiloxane, polyimide, polyethylene terephthalate or polyethylene naphthalate; Flexible copper circuit disposed on the flexible substrate; A microprocessor chip integrated on the flexible copper circuit; the microprocessor chip is used to collect and process physiological parameters, behavioral parameters, environmental parameters, and blood glucose parameters; the microprocessor chip integrates a physiological signal acquisition module, a behavioral signal acquisition module, an environmental signal acquisition module, a blood glucose signal acquisition module, a data processing module, a data storage module, and a power supply module; the data storage module is any one of an SD card driver module or a flash memory driver module; the power supply module is any one of a supercapacitor, a triboelectric nanosheet, a solar electrode, a button cell, or an electrode patch; the microprocessor chip has a pre-stored computer program, which, when executed, implements all the steps of the fish health status determination method as described in any one of claims 1 to 8.

Images