Seawater fish freshness detection method and device based on machine vision and FPS

Abstract

The invention provides a seawater fish freshness detection method and device based on machine vision and FPS, and relates to the technical field of prefabricated aquatic product nondestructive detection.The device comprises a mounting box, an automatic lifting clamp device is arranged at the inner bottom of the mounting box, and a fixed-distance pressing device is arranged at the position, corresponding to the automatic lifting clamp device, of the top of the mounting box; a top image shooting device is arranged on the fixed-distance pressing device, an infrared sensing positioning device corresponding to the fixed-distance pressing device is arranged in the mounting box body, a side industrial camera is arranged on one side of the interior of the mounting box body through a first support, a plurality of lighting devices are arranged on the upper portion of the side wall of the mounting box body, and a control unit is arranged at the bottom of the mounting box body; the top image shooting device, the infrared sensing positioning device, the side industrial camera, the automatic lifting clamp device, the lighting device and the fixed-distance pressing device are all electrically connected with the control unit. According to the invention, high-efficiency and automatic seawater fish freshness detection can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of non-destructive testing of prefabricated aquatic products, and particularly to a method and device for detecting the freshness of seawater fish based on machine vision and FPS. Background Art

[0002] As a food with high nutritional value, seawater fish is rich in protein, vitamins, minerals and unsaturated fatty acids, which are crucial for maintaining human health. However, the key to prefabricated aquatic products lies in the selection of raw materials. Fresh fish meat can not only ensure the excellent taste and nutritional value of dishes, but also reduce the risk of spoilage, thus ensuring food safety. Therefore, how to quickly and accurately detect the freshness of raw seawater fish has become one of the core issues in the current industrial development.

[0003] Existing non-destructive freshness detection technologies for fish include machine vision, electronic nose, infrared spectroscopy, Raman spectroscopy, hyperspectral, nuclear magnetic resonance, multi-source perception information fusion, etc. Although these technologies have their own advantages, they also have limitations. Single machine vision mainly relies on the analysis of the surface color and texture of fish meat, making it difficult to detect the internal freshness or hidden defects of meat, and the detection results are accidental; Spectral technologies (infrared, Raman, hyperspectral, nuclear magnetic resonance, etc.) have high detection accuracy, but the equipment is expensive and the operation is complex, making it difficult to be popularized and applied on the production line of prefabricated foods; Electronic nose technology detects volatile substances through sensors, which is easily affected by environmental conditions such as temperature and humidity, resulting in signal drift, and frequent calibration is required to maintain the stability and accuracy of measurement.

[0004] A flexible pressure sensor is a new type of force-sensitive sensor, which is usually made of flexible materials, can adapt to various complex shapes, and still maintain stable performance under bending, stretching and other conditions.

[0005] In view of these problems, in order to achieve efficient and high-precision detection of the freshness of raw seawater fish for prefabricated aquatic products. It is very necessary to design a method and device for detecting the freshness of seawater fish based on machine vision and FPS. Summary of the Invention

[0006] In order to overcome the deficiencies of the prior art, the purpose of the present invention is to provide a method and device for detecting the freshness of seawater fish based on machine vision and FPS.

[0007] To achieve the above purpose, the present invention provides the following solutions:

[0008] The present invention provides a method for detecting the freshness of seawater fish based on machine vision and FPS, including:

[0009] S1: Determine the range of fish species supported by the system. Then, collect pictures of different species of seawater fish, extract the outermost edge shape and scale characteristic scores of the seawater fish, determine the score range for each species of seawater fish, collect the classification information of each fish species and the uses of fish meat at different freshness levels, and construct a knowledge base for different fish species, dynamic grading criteria, and use recommendations.

[0010] S2: Based on the seawater fish freshness detection device, collect multi-directional image information and pressure curve information of the seawater fish. Among them, the multi-directional image information includes the top view and side view of the seawater fish. Measure the total volatile basic nitrogen content of the seawater fish by the Kjeldahl method to calibrate the freshness of the seawater fish.

[0011] S3: According to the obtained top view of the seawater fish to be detected, perform edge detection, identify the species through the outermost edge shape and scale characteristics of the seawater fish, and judge the freshness classification standard and the number of levels τ2 according to its score and the knowledge base.

[0012] S4: According to the obtained top view of the seawater fish to be detected, segment the pupil and iris regions of the seawater fish. According to the obtained side view of the seawater fish to be detected, segment the fish belly region of the seawater fish. Based on the preset feature extraction model for extracting the features of the target region, extract the features of the segmented target region.

[0013] S5: According to the obtained pressure information of the seawater fish to be detected, establish a curve feature extraction method for processing the pressure curve and extract the pressure curve features.

[0014] S6: Fuse the processed image features and pressure curve features and perform dynamic adjustment of importance.

[0015] S7: Combine the judged freshness classification standard, dynamically adjust the number of output layer neurons, construct a prediction model with the seawater fish image and pressure curve features as input values and the freshness grading of the seawater fish as output to predict the freshness of the seawater fish, and output the corresponding recommended uses of the fish meat.

[0016] Preferably, in S3, identifying the species through the outermost edge shape and scale characteristics of the seawater fish specifically includes:

[0017] Combining the smoothness of the edge curve and the density of the cusp distribution of the seawater fish to judge the complexity C of its outermost edge shape f is:

[0018]

[0019] In the formula, k i is the curvature of the i-th point of the edge curve, is the average value of the edge curvatures, N is the number of sampling points on the edge curve, M pis the number of local maximum curvature points, and L is the total length of the fish body edge curve;

[0020] Extract the scale structure characteristics S of seawater fish f as follows:

[0021]

[0022] In the formula, A i is the area of the i-th scale, P i is the perimeter of the i-th scale, N is the number of scales in the sampling area, D is the number of scales per unit area, is the gradient change of scale density;

[0023] Combined with the complexity C of the outermost edge shape f and the scale structure characteristics S f , judge the score score of the detected seawater fish, and classify the seawater fish according to the score, as follows:

[0024]

[0025] In the formula, is the weight coefficient, determined by fitting experimental data, and R is the closure degree characteristic of the fish body edge contour, which is:

[0026]

[0027] In the formula, P s is the perimeter of the actual contour, and P n is the perimeter of the fitted ellipse.

[0028] Preferably, according to the obtained top view of the seawater fish to be detected, the pupil and iris areas of the seawater fish are segmented, and according to the obtained side view of the seawater fish to be detected, the fish belly area of the seawater fish is segmented. The specific steps are as follows:

[0029] S401: Convert the original RGB image of the top view of the seawater fish to be detected into a grayscale image, as follows:

[0030]

[0031] In the formula, the value of ω is 0.2989, the value of is 0.5870, and the value of τ is 0.1140. The obtained grayscale image is blurred;

[0032] S402: Use the iris recognition algorithm to detect the pupil and iris areas in the image and segment the fish eye area, and separate the circular iris and the circular pupil, specifically as follows:

[0033] Locate the boundary radii of the pupil and iris by the consistency score in the gradient direction, as follows:

[0034]

[0035] In the formula, (x c , y c ) is the coordinate of the candidate center of the circle, R is the radius of the circle, G(x c , y c , R) is the combination of pixels of the circle with (x c , y c ) as the center and R as the radius, θ(x, y) represents the gradient direction of the point (x, y), and θ R (x, y) represents the ideal gradient direction of the point (x, y), which points to the center of the circle (x c , y c );

[0036] Determine the pupil radius R c , y c , R) by finding two local maxima of G(x p and the outer boundary radius R i of the iris. According to the detected center coordinate (x c , y c ), pupil radius R p and the outer boundary radius R i of the iris, construct the partition mask of the iris and pupil, as follows:

[0037]

[0038] In the formula, M(x, y) is the value of the partition mask, 1 represents the iris of the fish, 2 represents the pupil of the fish, and 0 represents other regions;

[0039] After the segmentation is completed, verify the segmentation quality of the iris and pupil, as follows:

[0040]

[0041] In the formula, μ p , μ i are the average brightness values of the pupil and iris region values, σ p , σ i are the brightness variances of the pupil and iris region values, ε is a smoothing factor to ensure the numerical stability of the formula, θ represents the weight of , represents the gradient intensity of (x, y), and N p is the size of the pupil region;

[0042] S403: Scale the segmented image to convert it to the same pixel size;

[0043] S404: Convert the original RGB image of the side view of the seawater fish to be detected into a grayscale image, as follows:

[0044]

[0045] where the value of ω is 0.2989, the value of is 0.5870, and the value of τ is 0.1140;

[0046] S405: Convert the grayscale image to a binary image, determine the threshold through color features, and segment the entire fish belly area:

[0047]

[0048] where I ij represents the pixel value at position (i, j) in the original image I, represents the pixel value at position (i, j) in the binary image I * and colorrange represents the preset color range (180, 230);

[0049] S406: Establish a coordinate system at the axis of symmetry of the picture, and further segment the entire fish belly area through the position function of the colored fixture teeth, specifically:

[0050] Establish a coordinate system with the axis of symmetry of the side view, the length direction of the fish as the X-axis, and the thickness direction of the fish as the Y-axis. Determine the function expression of the colored fixture teeth in the coordinate system, and further segment the entire fish belly area through the position function of the colored fixture teeth, as follows:

[0051]

[0052] where y1 and y2 respectively represent the position functions of the two colored fixture teeth on the fish belly side, and a, b, c, and d are their parameters.

[0053] Preferably, in S4, based on the preset feature extraction model for extracting the features of the target area, extract the features of the segmented target area, specifically:

[0054] First, construct the convolutional layer, as follows:

[0055]

[0056] where represents the output of the i-th convolutional kernel in the l-th layer at position j, σ is the ReLU activation function, is the bias term of the i-th convolutional kernel in the l-th layer, is the weight matrix of the i-th convolutional kernel in the l-th layer corresponding to the k-th input feature map, is the k-th feature map of the (l - 1)-th layer;

[0057] A pooling operation is performed. The pooling layer is used to reduce the spatial dimension of the feature map and enhance the generalization ability of the model, which is:

[0058] P(x, y) = max (a,b) ∈R (x,y) F(a, b);

[0059] In the formula, P(x, y) is the output at the position (x, y) after the pooling operation, R(x, y) is the pooling window area on the input feature map centered at (x, y), F(a, b) is the feature value at the position (a, b) in the pooling window, and |R(x, y)| is the number of elements in the pooling window;

[0060] After stacking multiple such convolutional and pooling layers, a residual connection is further introduced to improve the efficiency and effect of training a deep network. The expression of the residual block is:

[0061]

[0062] In the formula, x is the input feature, and θ(x, {W i}) represents the residual mapping with weights {W i}, and y is the output of the residual block.

[0063] Preferably, in S5, according to the pressure information of the seawater fish to be detected, a curve feature extraction method for processing the pressure curve is established to extract the pressure curve features, specifically:

[0064] When the pressure sensor presses the fish meat at a fixed depth, the curve of the pressure changing with the pressing depth is expressed as P(l) = {P(l1), P(l2), P(l3), …, P(l n ),}, where l i is the change in the pressing depth, and P(l i ) is the change in the pressure generated with the change in the pressing depth;

[0065] A curve feature extraction method for processing the pressure curve is established to extract the pressure curve features, where the local response ability of the pressure changing with the pressing depth is expressed as:

[0066]

[0067] In the formula, R(l) is the amplitude of the pressure change response at the depth l, P(l + Δl) and P(l - Δl) are the pressure values near the depth l, and Δl is the depth interval of the local window;

[0068] The local curvature of the pressure curve is expressed as:

[0069]

[0070] Wherein, is the local curvature at depth l, is the rate of change of pressure with pressing depth, is the acceleration of change of pressure with pressing depth;

[0071] The recovery rate E of pressure during the release phase V is expressed as:

[0072]

[0073] Wherein, P max is the maximum pressure value during pressing, P release is the pressure value when the pressure is released, and Δt is the time difference between P max and P release .

[0074] Preferably, in S6, the processed image features and pressure curve features are fused and dynamically adjusted for importance, specifically:

[0075] Fuse the features extracted from the eyes and belly regions of seawater fish with the elastic features to form a unified feature vector X = [F 瞳孔 , F 虹膜 , F 鱼腹 , P 弹 , where

[0076] Based on dynamic importance adjustment, further optimize the feature combination. Take the feature vector X = [F 瞳孔 , F 虹膜 , F 鱼腹 , P 弹 as the input, perform a linear transformation on the comprehensive feature matrix X to obtain the query matrix Q, the key matrix K, and the value matrix V, as:

[0077]

[0078] Wherein, W Q is the weight of the query matrix, W K is the weight of the key matrix, W V is the weight of the value matrix;

[0079] Calculate the similarity between the query matrix and the key matrix through dot product, and perform scaling and normalization to obtain the attention weight matrix, as:

[0080]

[0081] Wherein, d kis the dimension of the key matrix;

[0082] Multiply the weight matrix A by the value matrix V to obtain a weighted feature representation as:

[0083] O = AV.

[0084] Preferably, in S7, in combination with the determined freshness classification criteria, dynamically adjust the number of output layer neurons, construct a prediction model with the seawater fish image and pressure curve features as input values and the freshness classification of seawater fish as output for predicting the freshness of seawater fish, and output corresponding recommended uses of the fish meat, specifically:

[0085] In combination with the determined freshness classification criteria, dynamically adjust the number of output layer neurons, construct a prediction model with the seawater fish image and pressure curve features as input values and the freshness classification of seawater fish as output, and input the weighted feature representation O into the prediction model for feature fusion and freshness prediction;

[0086] Among them, the prediction model includes an input layer, a hidden layer and an output layer. The input layer is O, and each neuron in the input layer is fully connected to each neuron in the hidden layer 1 to form a weight matrix W m1 , the number of neurons in the hidden layer 1 is τ1, the dropout rate of the Dropout layer is 50%, the activation function is ReLU, and the input feature O is linearly combined with the weight matrix of the hidden layer 1 through the fully connected layer and added with a bias vector b m1 , that is, Z m1 = W m1 ·O + b m1 , and further after being processed by the activation function, O is obtained m1 = ReLU(Z m1 ), output to the Dropout layer, and the Dropout layer randomly discards the outputs of some neurons, and passes the outputs O m1 * of the remaining neurons to the hidden layer 2. The number of neurons in the hidden layer 2 is The hidden layer 2 receives the output O from the hidden layer 1 m1 * , and each neuron in the hidden layer 1 is fully connected to each neuron in the hidden layer 2 to form a weight matrix W m2 , and through the fully connected layer for linear transformation to obtain Z m2 = W m2 ·O m1 * + b m2 , and through the ReLU activation function, introduce non-linearity to obtain O m2 = ReLU(Z m2 ), and the output layer receives the output O from the hidden layer 2 m2, the third layer of the prediction model is the output layer. According to different types of seawater fish, different freshness classification criteria and the number of levels τ2 are required. There are τ2 neurons in the third layer corresponding to τ2 freshness classification levels, and the activation function is softmax.

[0087] The present invention also provides a seawater fish freshness detection device based on machine vision and FPS, including: an installation box body, a top image capturing device, an infrared sensing and positioning device, a side industrial camera, an automatic lifting jig device, a lighting device, a fixed-distance pressing device and a control unit. The automatic lifting jig device is arranged at the inner bottom of the installation box body. The fixed-distance pressing device is arranged corresponding to the automatic lifting jig device at the top of the installation box body, and the top image capturing device is arranged on the fixed-distance pressing device. The infrared sensing and positioning device is arranged corresponding to the fixed-distance pressing device inside the installation box body. The side industrial camera is arranged on one side inside the installation box body through a first bracket. A plurality of lighting devices are arranged on the upper part of the side wall of the installation box body. The control unit is arranged at the bottom of the installation box body. The top image capturing device, the infrared sensing and positioning device, the side industrial camera, the automatic lifting jig device, the lighting device and the fixed-distance pressing device are all electrically connected to the control unit.

[0088] Preferably, the top image capturing device includes a top industrial camera, a second bracket, a support rod, a slider, a first slideway, a first coupling, and a first stepping motor. The fixed-distance pressing device includes a third stepping motor, a second ball screw, a fixed baffle, and a slide table. One end of the first slideway is arranged at the back of the installation box body, and the other end is arranged on the upper side of the top of the installation box body. A first coupling is arranged at the top of this end. The first stepping motor is drivingly connected to the first coupling. The slider is slidably arranged on the first slideway. The first stepping motor drives the slider to slide along the first slideway through the first coupling. A support rod is arranged at the rear side of the bottom of the slider. The support rod is arranged at the bottom of the second bracket, and the top industrial camera is arranged on the second bracket. A base is arranged at the front side of the bottom of the slider. A chute is arranged inside the base. The third stepping motor is arranged at the top of the chute. The output end of the third stepping motor is arranged with the second ball screw. The slide table is connected to the second ball screw. The fixed baffle is arranged at the top of the slide table. A pressing block is arranged at the front side of the bottom of the slide table. A flexible pressure sensor is arranged on the pressing block, and the flexible pressure sensor is connected to the control unit.

[0089] Preferably, the automatic lifting fixture device includes colored fixture teeth, a second slideway, a second stepping motor, a first ball screw, a support frame, a lifting table, a cylinder and a solenoid valve. The lifting table, the second stepping motor and the cylinder are arranged on the inner bottom of the installation box body. The second stepping motor is connected to the first ball screw through a second coupling. The support frame is connected to the first ball screw. The support frame is arranged at the slidable end of the lifting table. The second stepping motor drives the first ball screw to rotate through the second coupling, drives the support frame to move back and forth, and further drives the lifting table to realize lifting adjustment. Six second slideways are arranged on the peripheral side of the top of the lifting table. The colored fixture teeth are slidably arranged inside the second slideways. The cylinder is drivingly connected to the colored fixture teeth and is used to drive the colored fixture teeth to slide along the second slideways. The cylinder is connected to a gas source through the solenoid valve. A flexible pressure sensor is arranged on the colored fixture teeth. The flexible pressure sensor is connected to the control unit.

[0090] According to the specific embodiments provided by the present invention, the following technical effects are disclosed by the present invention:

[0091] The present invention provides a method and device for detecting the freshness of seawater fish based on machine vision and FPS. The fixing and position adjustment of the seawater fish are realized through an automatic lifting fixture device and an infrared sensing positioning device. The measurement of the elastic data of the seawater fish is completed through a fixed-distance pressing device. A stable light source is provided by an illumination device so that an image capturing device can complete image acquisition. The control unit controls the detection device to work through preset instructions, and sends the captured images and pressure information to a computer through a connection line. After the data acquisition is completed, according to the obtained top view of the seawater fish to be detected, edge detection is performed, species identification is carried out based on the outermost edge shape and scale characteristics of the seawater fish, and the freshness division standard and level number τ2 are judged according to its species. The pupil, iris and fish belly area of the seawater fish are segmented, a feature extraction model for extracting the characteristics of the target area is constructed, the features of the picture are extracted, then a curve feature extraction model for processing the pressure curve is used to extract the pressure curve features, these features are fused and dynamically adjusted in importance, and finally a freshness prediction model is used to predict the freshness of the seawater fish. The present invention can realize high-efficiency and automated detection of the freshness of seawater fish, so as to facilitate the subsequent production of prefabricated aquatic products. Description of the Drawings

[0092] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0093] Figure 1 Schematic flow diagram of a method for detecting the freshness of seawater fish based on machine vision and FPS;

[0094] Figure 2 Schematic diagram of the segmentation of the target area of the picture data provided by the embodiment of the present invention;

[0095] Figure 3 Schematic diagram of the classical pressure data at all levels provided by the embodiment of the present invention;

[0096] Figure 4 Schematic diagram of the module of a device for detecting the freshness of seawater fish based on machine vision and FPS;

[0097] Figure 5 Structural diagram of the freshness detection model of seawater fish;

[0098] Figure 6 Bar chart of the accuracy rate of the results of judging freshness visually, elastically, and comprehensively visually + elastically provided by the embodiment of the present invention;

[0099] Figure 7 Partial enlarged schematic diagram of the fixed-distance pressing device;

[0100] Figure 8 Partial enlarged schematic diagram of the automatic lifting fixture device;

[0101] Figure 9 Schematic flow diagram of the operation of a device for detecting the freshness of seawater fish based on machine vision and FPS;

[0102] Figure 10 Schematic structural diagram of a device for detecting the freshness of seawater fish based on machine vision and FPS provided by the embodiment of the present invention.

[0103] Reference numerals: 1, top image capturing device; 101, top industrial camera; 102, second bracket; 103, support rod; 104, slider; 105, first slideway; 106, first coupling; 107, first stepping motor; 2, infrared sensing positioning device; 3, side industrial camera; 4, first bracket; 5, automatic lifting fixture device; 501, colored fixture teeth; 502, second slideway; 503, second stepping motor; 504, second coupling; 505, first ball screw; 506, support frame; 507, lifting platform; 508, cylinder; 509, solenoid valve; 6, lighting device; 7, fixed-distance pressing device; 701, third stepping motor; 702, second ball screw; 703, fixed baffle; 704, slide table; 8, control unit; 9, installation box. Detailed implementation manners

[0104] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0105] The purpose of the present invention is to provide a method and device for detecting the freshness of seawater fish based on machine vision and FPS, which can achieve high-efficiency and automated detection of the freshness of seawater fish, so as to facilitate the production of subsequent prefabricated aquatic products.

[0106] In order to make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0107] Figure 4 It is a schematic diagram of the module composition provided by the embodiment of the present invention. Figure 10 It is the device structure diagram provided by the embodiment of the present invention. As Figure 4 and Figure 10 shown, the present invention provides a device for detecting the freshness of seawater fish based on machine vision and FPS, including: an installation box body 9, a top image capturing device 1, an infrared sensing positioning device 2, a side industrial camera 3, an automatic lifting fixture device 5, a lighting device 6, a fixed-distance pressing device 7, and a control unit 8. The automatic lifting fixture device 5 is arranged at the inner bottom of the installation box body 9, the fixed-distance pressing device 7 is arranged corresponding to the automatic lifting fixture device 5 at the top of the installation box body 9, and the top image capturing device 1 is arranged on the fixed-distance pressing device 7. The infrared sensing positioning device 2 is arranged corresponding to the fixed-distance pressing device 7 inside the installation box body 9. The side industrial camera 3 is arranged on the inner side of the installation box body 9 through a first bracket 4. A plurality of lighting devices 6 are arranged on the upper part of the side wall of the installation box body 9. The control unit 8 is arranged at the bottom of the installation box body 9. The top image capturing device 1, the infrared sensing positioning device 2, the side industrial camera 3, the automatic lifting fixture device 5, the lighting device 6, and the fixed-distance pressing device 7 are all electrically connected to the control unit 8.

[0108] The top image capturing device 1 includes a top industrial camera 101, a second bracket 102, a support rod 103, a slider 104, a first slideway 105, a first coupling 106, and a first stepping motor 107. As Figure 7As shown, the fixed-distance pressing device includes a third stepping motor 701, a second ball screw 702, a fixed baffle 703, and a sliding table 704. One end of the first slideway 105 is provided at the back of the installation box body 9, and the other end is provided on the upper side of the top of the installation box body 9. The first coupling 106 is provided at the top of this end. The first stepping motor 107 is drivingly connected to the first coupling 106. The slider 104 is slidably arranged on the first slideway 105. The first stepping motor 107 drives the slider to slide along the first slideway 105 through the first coupling 106. The support rod 103 is provided at the rear side of the bottom of the slider 104. The second support 102 is provided at the bottom of the support rod 103. The top industrial camera 101 is provided on the second support 102. A base is provided at the front side of the bottom of the slider 104. A chute is provided inside the base. The third stepping motor 701 is provided at the top of the chute. The output end of the third stepping motor 701 is provided with the second ball screw 702. The sliding table 704 is connected to the second ball screw 702. The fixed baffle 703 is provided at the top of the sliding table 704. A pressing block is provided at the front side of the bottom of the sliding table 704. A flexible pressure sensor is provided on the pressing block. The flexible pressure sensor is connected to the control unit 8.

[0109] As Figure 8 shown, the automatic lifting jig device 5 includes a colored jig tooth 501, a second slideway 502, a second stepping motor 503, a first ball screw 505, a support frame 506, a lifting table 507, a cylinder 508, and a solenoid valve 509. The lifting table 507, the second stepping motor 503, and the cylinder 508 are provided on the inner bottom of the installation box body 9. The second stepping motor 503 is connected to the first ball screw 505 through a second coupling 504. The support frame 506 is connected to the first ball screw 505. The support frame 506 is arranged at the slidable end of the lifting table 507. The second stepping motor 503 drives the first ball screw 505 to rotate through the second coupling 504, driving the support frame 506 to move back and forth, and further driving the lifting table 507 to achieve lifting adjustment. Six second slideways 502 are provided on the peripheral side of the top of the lifting table 507. The colored jig tooth 501 is slidably arranged inside the second slideway 502. The cylinder 508 is drivingly connected to the colored jig tooth 501 for driving the colored jig tooth 501 to slide along the second slideway 502. The cylinder 508 is connected to the air source through the solenoid valve 509. A flexible pressure sensor is provided on the colored jig tooth 501. The flexible pressure sensor is connected to the control unit 8;

[0110] For its detailed introduction, the automatic lifting jig device 5 is as Figure 8As shown, it includes a second stepping motor 503, a first ball screw 505, a lifting platform 507, a solenoid valve 509, two cylinders 508 ( stroke 50mm), and three groups of colored fixture teeth 501. One end of the ball screw 505 is connected to the stepping motor 503 through a coupling 504, and the other end is connected to a threaded support frame 506. The stepping motor 503 is used as a power element to control the lifting of the lifting platform 507, and the lifting speed is 2mm / s. The number of steps of the stepping motor operation is. The cylinder 508 moves the fixture teeth along the slideway 502 through the solenoid valve 509, so as to adjust the clamping force of the fixture teeth 501. A small flexible pressure sensor is attached to the colored fixture teeth 501 to feedback the movement of the pneumatic device;

[0111] The automatic lifting fixture device 5 can realize the fixation of seawater fish and the adjustment of the height of the seawater fish.

[0112] The infrared sensing and positioning device 2 includes a group of infrared laser pair sensors distributed on both sides of the box body, which is used to provide feedback information to the second stepping motor 503 to ensure that the detected seawater fish reaches the required height.

[0113] The height H that the lifting platform rises s varies with the change of the thickness of the measured fish and can be calculated according to the formula:

[0114] H s = 0.025·N1;

[0115] The fixed-distance pressing device 7 is as Figure 7 shown, including a second ball screw 702 (pitch p2 is 5mm), one end of which is connected to a third stepping motor 701 (rated current I1A, step angle θ1°), and the other end is connected to a threaded slide 704 (the length of the slide L2 is 40mm, and the distance L3 between the bottom of the slide and the infrared laser pair sensor in the initial state is 40mm). There is a fixed baffle 703 above the slide 704. A flexible pressure sensor is installed on the convex block (i.e., the pressing block) at the bottom of the slide 704. The fixed-distance pressing device 7 can realize the downward pressing of the fixed stroke, so as to realize the detection of the meat elasticity of seawater fish. The standard thickness C of seawater fish is 40mm, and the corresponding standard height H that the lifting platform rises b is 90mm. The number of steps N2 of the stepping motor operation in the fixed-distance pressing device can be expressed as:

[0116] C1 = C - H s + H b ;

[0117]

[0118] where C1 is the actual thickness of the detected seawater fish;

[0119] The lighting device 6 includes four circumferentially arranged LED lights with a power of 0.5W to 1W. The LED lights can rotate axially and can provide stable brightness (about 100 lumens) for image acquisition of the seawater fish to be measured. The side industrial camera is an industrial camera 3 located on the side (resolution 1920×1080), which is connected to the installation box 9 through a bracket 4. In addition, the top industrial camera 101 is clamped by a bracket 102 and is connected to the X-axis displacement platform through a support rod 103. The X-axis displacement platform includes a slider 104 located on a slideway 105 and a stepping motor 107 located at one end of the device. The slider 104 is connected to an image capturing device 1 and a fixed-distance pressing device 7. The stepping motor 107 transmits torque through a coupling 106, so that the slider 104 moves to realize image acquisition of the fish's eyes. The industrial camera 3 located on the side can realize the acquisition of the lateral picture features of the seawater fish;

[0120] The control unit is arranged at the bottom of the device shell, controls the detection device to work through preset instructions, and sends the collected image and pressure information to the computer through a connecting wire.

[0121] The present invention adopts an Interlink Electronics FSR 400 flexible pressure sensor. The pressure range of the flexible pressure sensor is 0.2N to 20N, the thickness is 0.45mm, the response time is less than 5ms, and the diameter is 18.28mm.

[0122] As Figure 9 shown, the specific operation process of the device is introduced. After the device is initialized, the control unit 8 starts, controls the automatic lifting fixture device 5 to start, supplies power to the coil of the solenoid valve 509 to make it energized (the power supply is 12V or 24VDC). At this time, the internal spool of the solenoid valve 509 switches, connecting the air source (pressure range 0.1 - 0.5MPa) to the air inlet of the cylinder 508. The cylinder 508 is a double-acting cylinder with a diameter of a stroke length of 50mm. The air source completes the clamping action through the control of the solenoid valve 509. At the same time, the flexible pressure sensor (response time < 5ms) on the colored fixture teeth 501 ensures that the pressure will not be too large so as not to affect subsequent measurements. The coil of the solenoid valve 509 remains energized, and the piston rod of the cylinder 508 remains extended to ensure that the longitudinal fixture tooth group and the transverse fixture tooth group maintain the clamping state;

[0123] After the clamping is completed, the automatic lifting module starts to operate. The control unit activates the second stepper motor 503 of the lifting table 507 (rated current I1 is 0.5A, step angle θ1 is 1.8°). The second stepper motor 503 drives the lifting table 507 to slowly rise at a speed of 2mm / s through the ball screw 505 (pitch 5mm, diameter 12mm) and the support frame 506. When the infrared sensing and positioning device detects a seawater fish (detection range 0.5m - 1m, response time <20ms), the stepper motor 503 stops moving to ensure that the seawater fish is located at the predetermined shooting position;

[0124] Next, the first stepper motor 107 that controls the X-axis displacement device is activated (with the same parameters as the lifting module). Through the slideway 105 (length 300mm), the top camera 101 is moved directly above the seawater fish. Two industrial cameras in the image capturing device (resolution 1920×1080, shutter speed 1 / 30s - 1 / 1000s) take pictures to collect seawater fish images from different orientations. At the same time, the fixed-distance pressing device 7 is activated. The second ball screw 702 (pitch 5mm) drives the slide 704 to descend (pressing speed 2mm / s) to complete the collection of pressure data. The collected images and pressure information are sent to the computer (data transfer rate 115200bps, image format JPEG or PNG, pressure data in CSV format) through a USB cable (data transfer rate 115200bps, image format JPEG or PNG, pressure data in CSV format).

[0125] As Figure 1 shown, the present invention also provides a method for detecting the freshness of seawater fish based on machine vision and FPS, which is applied to the above-mentioned device, including:

[0126] S1: Determine the range of fish species supported by the system, and then collect pictures of different types of seawater fish. Extract the outermost edge shape and scale characteristic scores of the seawater fish, determine the scoring interval for each type of seawater fish, collect the classification information of each type of fish and the uses of fish meat at different freshness levels, and construct a knowledge base for different fish species, dynamic grading criteria, and use recommendations;

[0127] S2: Based on the seawater fish freshness detection device, collect multi-directional image information and pressure curve information of the seawater fish. Among them, the multi-directional image information includes the top view and side view of the seawater fish. Measure the total volatile basic nitrogen content of the seawater fish by a Kjeldahl nitrogen analyzer to calibrate the freshness of the seawater fish;

[0128] S3: According to the top view of the seawater fish to be detected obtained, perform edge detection, identify the species through the outermost edge shape and scale characteristics of the seawater fish, and judge the freshness classification standard and the number of levels τ2 according to its score and the knowledge base;

[0129] S4: According to the top view of the seawater fish to be detected obtained, segment the pupil and iris regions of the seawater fish. According to the side view of the seawater fish to be detected obtained, segment the belly region of the seawater fish. Based on a preset feature extraction model for extracting the features of the target region, extract the features of the segmented target region;

[0130] S5: According to the pressure information of the seawater fish to be detected obtained, establish a curve feature extraction method for processing the pressure curve and extract the pressure curve features;

[0131] S6: Fuse the processed image features and pressure curve features and perform dynamic adjustment of importance;

[0132] S7: Combine the judged freshness classification criteria, dynamically adjust the number of neurons in the output layer, construct a prediction model with the seawater fish image and pressure curve features as input values and the freshness grading of the seawater fish as output to predict the freshness of the seawater fish, and output the corresponding recommended uses of the fish meat.

[0133] In S1, determine the range of fish species supported by the system, and then collect pictures of different types of seawater fish, extract the outermost edge shape and scale characteristic scores of the seawater fish, determine the scoring interval for each type of seawater fish, collect the classification information of each type of fish and the uses of fish meat at different freshness levels, and construct a knowledge base for different fish species, dynamic grading criteria and use recommendations, and store it in a CSV file;

[0134] Specifically, the number of freshness classifications for yellow croaker is 4 (fresh: suitable for making soup; less fresh: suitable for steaming or braising; not fresh: suitable for frying or making fish cakes; spoiled: suitable for making feed), the number of freshness classifications for salmon is 3 (fresh: suitable for making sashimi and eating raw; less fresh: suitable for steaming or light processing; not fresh: suitable for deep processing, such as fish balls or smoked fish), the number of freshness classifications for tuna is 4 (fresh: suitable for high-end sashimi; less fresh: suitable for ordinary sashimi or light processing; not very fresh: suitable for cooked food, such as frying and grilling; not fresh: suitable for deep processing, such as smoking or canning), the number of freshness classifications for sea bream is 3 (fresh: suitable for eating raw or making sashimi; not fresh: suitable for steaming or braising; spoiled: suitable for making feed), and the number of freshness classifications for cod is 3 (fresh: suitable for steaming or frying; less fresh: suitable for making soup or stewing; not fresh: suitable for processing into fish cakes or fish balls).

[0135] In S2, based on the seawater fish freshness detection device, collect the multi-directional image information and pressure curve information of the seawater fish. Among them, the multi-directional image information includes the top view and side view of the seawater fish. Calibrate the freshness of the seawater fish by measuring the content of total volatile basic nitrogen in the seawater fish with a Kjeldahl nitrogen analyzer. Specifically:

[0136] Obtain seawater fish samples with different volatile basic nitrogen contents, measure the volatile basic nitrogen content of the seawater fish samples, and conduct the measurement according to the method specified in the national standard for the determination of volatile basic nitrogen in aquatic products. In this embodiment, yellow croaker is selected as the experimental object, and according to the characteristics of yellow croaker, the following classification is carried out: Yellow croaker is divided into four categories: fresh, sub-fresh, not fresh, and spoiled according to the content of volatile basic nitrogen (fresh, TVB-N content ≤ 10mg / 100g; sub-fresh, TVB-N content: 10 - 20mg / 100g; not fresh, TVB-N content: 20 - 30mg / 100g; spoiled, TVB-N content ≥ 30mg / 100g).

[0137] Purchase 201 yellow croakers. The average weight and average length of the selected yellow croakers are 320 grams and 22 centimeters respectively. Randomly select one yellow croaker from the 201 yellow croakers as the sample fish, and randomly divide the remaining 200 yellow croakers into 10 groups, which are respectively labeled as A, B, C, D, E, F, G, H, I, J, with 20 fish in each group. Keep all the yellow croakers in an incubator at a temperature of 4°C. Every day, take 2 fish from each of the 10 groups of yellow croakers for image acquisition, elasticity measurement, and measurement of the volatile basic nitrogen content. Continuously carry out the above method for 10 days to obtain a total of 200 samples, among which 160 samples are used as the training set and 40 samples are used as the test set.

[0138] In S3, according to the obtained top view of the seawater fish to be detected, perform edge detection, identify the species through the outermost edge shape and scale characteristics of the seawater fish, and determine the freshness classification standard and the number of levels τ2 according to its score and the knowledge base. Specifically:

[0139] Combine the smoothness of the edge curve and the density of cusp distributions of the seawater fish (yellow croaker in this embodiment, and yellow croaker will be used subsequently) to judge the complexity C of its outermost edge shape f as follows:

[0140]

[0141] In the formula, k i is the curvature of the i-th point of the edge curve, is the average value of the edge curvatures, N is the number of sampling points on the edge curve, and M p is the number of local maximum curvature points, and L is the total length of the fish body edge curve;

[0142] Extract the scale structure characteristics S of the yellow croaker f as follows:

[0143]

[0144] In the formula, A i is the area of the i-th scale, and Pi is the perimeter of the i-th scale, N is the number of scales in the sampling area, D is the number of scales per unit area, is the gradient change of scale density;

[0145] Combined with the complexity C of the outermost edge shape f and the scale structure characteristic S f , judge the score score of the detected yellow croaker, and classify the yellow croaker according to the score, as:

[0146]

[0147] In the formula, is the weight coefficient, determined by fitting experimental data, and R is the closure characteristic of the fish body edge contour, which is:

[0148]

[0149] In the formula, P s is the perimeter of the actual contour, and P m is the perimeter of the fitted ellipse;

[0150] Classify the yellow croaker according to the score score. Different score intervals correspond to different types of seawater fish, and also correspond to different freshness grading standards and the number of levels τ2;

[0151] In this embodiment, the score score interval of the yellow croaker is [5, 7], the number of levels τ2 is 4, the complexity C of the outermost edge shape f is about 1.5, the scale structure characteristic S f is about 0.8, the closure characteristic R of the fish body edge contour is about 0.9, and the weight

[0152] In S4, according to the top view of the seawater fish to be detected obtained, the pupil and iris regions of the seawater fish are segmented. According to the side view of the seawater fish to be detected obtained, the fish belly region of the seawater fish is segmented. Based on a preset feature extraction model for extracting target region features, feature extraction is performed on the segmented target regions, which specifically includes the following steps:

[0153] Simply put, first, read the yellow croaker image information collected by the industrial camera and preprocess it. The image preprocessing includes the segmentation and noise reduction of the eye part and the fish belly part. The schematic diagram of the target region segmentation is as Figure 2 shown, and the overall data processing process is as Figure 1 shown. The specific steps are:

[0154] S401: For the eye part, convert the original RGB image of the top view of the yellow croaker into a grayscale image, which is:

[0155]

[0156] In the formula, the value of ω is 0.2989, the value of is 0.5870, the value of τ is 0.1140, the obtained grayscale image is blurred, and then the area where the fish eye is located is segmented;

[0157] S402: The segmentation uses the rainbow pupil identification algorithm to detect the pupil and iris regions in the image and segment the fish eye region, separating the circular iris and the circular pupil. Specifically:

[0158] By the consistency score of the gradient direction, the boundary radii of the pupil and the iris are located, which are:

[0159]

[0160] In the formula, (x c , y c ) is the candidate center coordinate, R is the circumference radius, G(x c , y c , R) is the pixel combination of the circumference with (x c , y c ) as the center and R as the radius, θ(x, y) represents the gradient direction of the point (x, y), and θ R (x, y) represents the ideal gradient direction of the point (x, y), which points to the center (s c , y c );

[0161] By finding two local maxima of G(x c , y c , R), the pupil radius R p and the outer boundary radius Ri of the iris are determined. According to the detected center coordinate (x c , y c ), the pupil radius R p and the outer boundary radius R i of the iris, a partition mask for the iris and the pupil is constructed, which is:

[0162]

[0163] In the formula, M(x, y) is the partition mask value, 1 represents the iris of the fish, 2 represents the pupil of the fish, and 0 represents other regions;

[0164] In this embodiment, the value range of the pupil radius R p is 2 - 3 mm, and the value range of the outer boundary radius Ri of the iris is 5 - 7 mm;

[0165] After the segmentation is completed, the segmentation quality of the iris and the pupil is verified, which is:

[0166]

[0167] Wherein, μ p , μ i are the average brightness values of the pupil and iris region values, σ p , σ i are the brightness variances of the pupil and iris region values, ε is a smoothing factor to ensure the numerical stability of the formula, θ represents 's weight, represents the gradient intensity of (x, y), N p the size of the pupil region;

[0168] S403: Scale the segmented image so that it is converted to the same pixel size, which is converted to the same pixel size h×h. According to the formula:

[0169]

[0170] m i = m p ·q p ;

[0171] n i = n p ·q i ;

[0172] Wherein, m p is the original pixel size of the pupil, m i is the converted pupil pixel size, n p is the original pixel size of the iris, n i is the converted iris pixel size, q p is the scaling ratio of the pupil, and q2 is the scaling ratio of the iris;

[0173] In this embodiment, the pupil region pixel size is approximately 50×50 pixels, and the iris region pixel size is approximately 200×200 pixels. Convert them to the same 100×100 pixels, that is:

[0174]

[0175] m i = 50×2 = 100;

[0176] n i = 200×0.5 = 100;

[0177] S404: For the belly part, convert the original RGB image of the side view of the yellow croaker into a grayscale image, which is:

[0178]

[0179] Wherein, the value of ω is 0.2989, the value of is 0.5870, and the value of τ is 0.1140;

[0180] S405: Convert the grayscale image into a binary image, determine the threshold through color features, and segment the entire fish belly area:

[0181]

[0182] Wherein, I ij represents the pixel value at the position (i, j) in the original image I, represents the pixel value at the position (i, j) in the binary image I * and colorrange represents the preset color range (180, 230);

[0183] S406: Establish a coordinate system at the axis of symmetry of the picture, and further segment the entire fish belly area through the position function of the colored fixture teeth, specifically:

[0184] Establish a coordinate system with the axis of symmetry of the side view. The length direction of the fish is the X-axis, and the thickness direction of the fish is the Y-axis. Determine the function expression of the colored fixture teeth in the coordinate system, and further segment the entire fish belly area through the position function of the colored fixture teeth, which is:

[0185]

[0186] Wherein, y1 and y2 respectively represent the position functions of the two colored fixture teeth on the fish belly side, and a, b, c, and d are their parameters;

[0187] In this embodiment, the central position of the yellow croaker is used as the coordinate origin. Generally, x1 = 8 and x2 = -8.

[0188] In S4, based on the preset feature extraction model for extracting the features of the target area, extract the features of the segmented target area, specifically:

[0189] First, construct the convolutional layer, which is:

[0190]

[0191] Wherein, represents the output of the i-th convolutional kernel in the l-th layer at the position j, σ is the ReLU activation function, is the bias term of the i-th convolutional kernel in the l-th layer, is the weight matrix of the i-th convolutional kernel in the l-th layer corresponding to the k-th input feature map, is the k-th feature map of the (l - 1)-th layer;

[0192] In this embodiment, the convolution kernel size is 3x3, the number of convolution kernels in the first layer is set to 32, in the second layer is set to 64, and in the third layer is set to 128. The bias term is initialized to 0;

[0193] Pooling operation is performed. The pooling layer is used to reduce the spatial dimension of the feature map and enhance the generalization ability of the model, which is:

[0194] P(x,y) = max (a,b) ∈R (x,y) F(a,b);

[0195] In the formula, P(x,y) is the output at position (x,y) after the pooling operation, R(x,y) is the pooling window area on the input feature map centered at (x,y), F(a,b) is the feature value at position (a,b) in the pooling window, and |R(x,y)| is the number of elements in the pooling window;

[0196] In this embodiment, the pooling window size is 2x2, and the corresponding stride is 2;

[0197] After stacking multiple such convolutional and pooling layers, residual connections are further introduced to improve the efficiency and effect of training deep networks. The expression of the residual block is:

[0198]

[0199] In the formula, x is the input feature, θ(x,{W i ) represents the residual mapping with weights {W i}, and y is the output of the residual block;

[0200] In this embodiment, the initial value of the input feature is between [-1, 1].

[0201] In S5, according to the obtained pressure information of the seawater fish to be detected, a curve feature extraction method for processing the pressure curve is established to extract the pressure curve features, specifically:

[0202] According to the pressure data obtained by the flexible pressure sensor, the elasticity of the yellow croaker meat is deduced. When the pressure sensor presses the fish meat at a fixed depth, the curve of the generated pressure changing with the pressing depth is expressed as P(l) = {P(l1), P(l2), P(l3),…, P(l n ),}, where l i is the change in the pressing depth, P(l i ) is the change in pressure generated with the change in the pressing depth. The collected pressure change curve is as Figure 3 shown, Figure 3 in which are the pressure curve graphs corresponding to 4 different freshness levels;

[0203] In this embodiment, the pressure measured by the flexible pressure sensor is between 0.2 N and 15 N;

[0204] Establish a curve feature extraction method for processing the pressure curve to extract the pressure curve features. Among them, the local response ability of the pressure changing with the pressing depth is expressed as:

[0205]

[0206] In the formula, R(l) is the amplitude of the pressure change response at depth l, P(l + Δl) and P(l - Δl) are the pressure values near depth l, and Δl is the depth interval of the local window;

[0207] In this embodiment, Δl is 2 mm. When the maximum pressure is 15 N, R(l) = [1.65, 2.125, 1.8, 1.625];

[0208] The local curvature of the pressure curve is expressed as:

[0209]

[0210] In the formula, is the local curvature at depth l, is the rate of change of the pressure with the pressing depth, is the acceleration of the change of the pressure with the pressing depth;

[0211] In this embodiment, the local curvature of the pressure curve changes with the pressing depth and freshness;

[0212] The recovery rate E of the pressure during the release stage V is expressed as:

[0213]

[0214] In the formula, P max is the maximum pressure value during the pressing process, P release is the pressure value when the pressure is released, and Δt is the time difference between P max and P release ;

[0215] In this embodiment, when P max is 15 N and P release is 7.8 N, Δt is 4 s, and E V is 1.8 N / s. The recovery rate E V is relatively low, indicating that the meat elasticity gradually weakens.

[0216] In S6, the processed image features and pressure curve features are fused and dynamically adjusted for importance, specifically:

[0217] After the feature extraction of the image data and pressure data is completed, the data fusion and freshness prediction are then started. The specific process is as Figure 5 shown;

[0218] Fuse the features extracted from the eyes and belly regions of the yellow croaker with the elastic features to form a unified feature vector X = [F 瞳孔 , F 虹膜 , F 鱼腹 , P 弹 , where

[0219] Based on the dynamic adjustment of importance to further optimize the feature combination, the dynamic adjustment of importance can help the model focus on the most critical features, improve the accuracy of prediction, and dynamically adjust the contributions of different features, so as to optimize the model performance for the freshness judgment of yellow croaker. Take the feature vector X = [F 瞳孔 , F 虹膜 , F 鱼腹 , P 弹 as the input, perform a linear transformation on the comprehensive feature matrix X to obtain the query matrix Q, the key matrix K, and the value matrix V, which are:

[0220]

[0221] In the formula, W Q is the weight of the query matrix, W K is the weight of the key matrix, and W V is the weight of the value matrix;

[0222] Calculate the similarity between the query matrix and the key matrix through the dot product, and perform scaling and normalization to obtain the attention weight matrix, which is:

[0223]

[0224] In the formula, d k is the dimension of the key matrix;

[0225] Multiply the weight matrix A by the value matrix V to obtain the weighted feature representation as:

[0226] O = AV;

[0227] In S7, combined with the judged freshness classification criteria, dynamically adjust the number of output layer neurons, construct a prediction model with the seawater fish image and pressure curve features as the input values and the freshness grading of seawater fish as the output to predict the freshness of seawater fish, and output the corresponding recommended uses of the fish meat, specifically:

[0228] Combined with the judged freshness classification criteria, dynamically adjust the number of neurons in the output layer, and construct a prediction model with the seawater fish image and pressure curve features as input values and the freshness classification of seawater fish as output. Input the weighted feature representation O into the prediction model for feature fusion and freshness prediction;

[0229] Among them, the prediction model includes an input layer, a hidden layer, and an output layer. The input layer is O, and each neuron in the input layer is fully connected to each neuron in the hidden layer 1 to form a weight matrix W m1 , the number of neurons in the hidden layer 1 is τ1, the dropout rate of the Dropout layer is 50%, the activation function is ReLU, and the input feature O is linearly combined with the weight matrix of the hidden layer 1 through the fully connected layer and added with a bias vector b m1 , that is, Z m1 =W m1 ·O + b m1 , and further processed by the activation function to obtain O m1 =ReLU(Z m1 ), output to the Dropout layer, and the Dropout layer randomly discards the output of some neurons. The output O m1 * of the remaining neurons is passed to the hidden layer 2, and the number of neurons in the hidden layer 2 is The hidden layer 2 receives the output O from the hidden layer 1 m1 * , each neuron in the hidden layer 1 is fully connected to each neuron in the hidden layer 2 to form a weight matrix W m2 , and a linear transformation is performed through the fully connected layer to obtain Z m2 =W m2 ·O m1 * +b m2 , and the ReLU activation function is used to introduce non-linearity to obtain O m2 =ReLU(Z m2 ), and the output layer receives the output O from the hidden layer 2 m2 , the third layer of the prediction model is the output layer. According to the different types of seawater fish, different freshness classification criteria and the number of levels τ2 are required. τ2 freshness classification levels correspond to τ2 neurons in the third layer, and the activation function is softmax.

[0230] According to the knowledge base, the yellow croaker in this embodiment adopts four classifications, that is, it is set to 4 neurons;

[0231] Combined with the detected freshness labels, perform feature fusion and freshness prediction, which is:

[0232]

[0233] Where Y 新鲜度 is the classification result of the freshness of yellow croaker; is the activation function that converts the result of the linear transformation into a probability distribution; W is the weight matrix b is the bias vector [b1, b2, …, b j ;

[0234] In this embodiment, the initial value range of the weight matrix W is between [-0.1, 0.1], and the bias vector b is usually initialized to 0 or 0.1, and here it is taken as 0;

[0235] The cross-entropy loss of the prediction model is expressed as:

[0236]

[0237] Where Y Δi represents the value of the true label vector in the i-th class;

[0238] In this embodiment, the length of the true label is 4, representing the true category of the classification, and the predicted value is the probability distribution of a 4-dimensional vector, ranging from [0, 1];

[0239] According to the predicted freshness level, different usage recommendations are made. The number of yellow croaker freshness classifications is 4 (fresh: suitable for stewing; sub-fresh: suitable for steaming or braising; not fresh: suitable for frying or making fish cakes; spoiled: suitable for making feed);

[0240] Finally, 40 samples of the test set are verified using three different methods respectively, and the correct rates obtained are as Figure 6 shown. The detection method provided by the embodiment of the present invention performs optimally under various yellow croaker freshness levels.

Claims

1. A method for detecting the freshness of seawater fish based on machine vision and FPS, characterized in that, Including: S1: Determine the range of fish species supported by the system, then collect pictures of different species of seawater fish, extract the outermost edge shape and scale characteristic scores of the seawater fish, determine the score range for each type of seawater fish, collect the classification information of each fish species and the uses of fish meat at different freshness levels, and construct a knowledge base for different fish species, dynamic grading criteria, and use recommendations. S2: Based on the seawater fish freshness detection device, collect multi-directional image information and pressure curve information of the seawater fish. Among them, the multi-directional image information includes the top view and side view of the seawater fish, and measure the content of total volatile basic nitrogen in the seawater fish by the Kjeldahl method to calibrate the freshness of the seawater fish. S3: According to the obtained top view of the seawater fish to be detected, perform edge detection, identify the species through the outermost edge shape and scale characteristics of the seawater fish, and judge the freshness division criteria and the number of levels τ2 according to its score and the knowledge base. S4: According to the obtained top view of the seawater fish to be detected, segment the pupil and iris regions of the seawater fish. According to the obtained side view of the seawater fish to be detected, segment the belly region of the seawater fish. Based on the preset feature extraction model for extracting target region features, extract features from the segmented target regions. S5: According to the obtained pressure information of the seawater fish to be detected, establish a curve feature extraction method for processing the pressure curve and extract the pressure curve features. S6: Fuse the processed image features and pressure curve features and perform dynamic adjustment of importance. S7: Combining the judged freshness division criteria, dynamically adjust the number of output layer neurons, construct a prediction model with the seawater fish image and pressure curve features as input values and the freshness grading of the seawater fish as output to predict the freshness of the seawater fish, and output the corresponding recommended uses of the fish meat.

2. The method according to claim 1, characterized in that, In S3, the species is identified through the outermost edge shape and scale characteristics of the seawater fish, specifically: Judging the complexity C of the outermost edge shape of marine fish by combining the edge curve smoothness and cusp distribution density of marine fish f It is as follows: where k i is the curvature of the i-th point of the edge curve, is the average value of the edge curvature, N is the number of sampling points on the edge curve, M p is the number of local maximum curvature points, and L is the total length of the fish body edge curve; Extract the scale structure characteristics S of marine fish f for extraction: where A i is the area of the i-th scale, P i is the perimeter of the i-th scale, N is the number of scales in the sampling area, D is the number of scales per unit area, is the gradient change of scale density; Combined with the complexity C of the outermost edge shape f and the scale structure characteristics S f , determine the score score of the detected seawater fish, and classify the seawater fish according to the score as follows: In the formula, is the weight coefficient, which is determined by fitting experimental data. R is the closure feature of the edge contour of the fish body and is: where P s is the perimeter of the actual contour, and P n is the perimeter of the fitted ellipse.

3. The method according to claim 2, wherein According to the obtained top view of the seawater fish to be detected, segment the pupil and iris regions of the seawater fish. According to the obtained side view of the seawater fish to be detected, segment the belly region of the seawater fish, specifically including the following steps: S401: Convert the original RGB image of the top view of the seawater fish to be detected into a grayscale image, as: where ω takes the value of 0.2989, takes the value of 0.5870, τ takes the value of 0.1140, and the obtained grayscale image is blurred; S402: Use the iris and pupil identification algorithm to detect the pupil and iris regions in the image and segment the fish eye region, separating the circular iris and the round pupil, specifically: Locate the boundary radius of the pupil and iris through the consistency score of the gradient direction, as: Where, (x c , y c ) are the candidate center coordinates, R is the circumferential radius, and G(x c , y c , R) is the combination of circumferential pixels with (x c , y c ) as the center and R as the radius. θ(x, y) represents the gradient direction of the point (x, y), and θ R (x, y) represents the ideal gradient direction of the point (x, y), which points to the center (x c , y c ); By finding two local maxima of G(x c ,y c ,R), the pupil radius R p and the outer iris boundary radius R i are determined. According to the detected center coordinates (x c ,y c ), the pupil radius R p and the outer iris boundary radius R i , a partition mask of the iris and pupil is constructed as follows: where M(x, y) is the partition mask value, 1 represents the iris of the fish, 2 represents the pupil of the fish, and 0 represents other regions; After the segmentation is completed, verify the segmentation quality of the iris and pupil, as: where μ p , μ i are the average brightness values of the pupil and iris region values, σ p , σ i are the brightness variances of the pupil and iris region values, ε is a smoothing factor to ensure the numerical stability of the formula, represents 's weight, represents the gradient intensity of (x, y), N p is the size of the pupil region; S403: Scale the segmented image so that it is converted to the same pixel size. S404: Convert the original RGB image of the side view of the seawater fish to be detected into a grayscale image, as: where ω takes a value of 0.2989, takes a value of 0.5870, and τ takes a value of 0.1140; S405: Convert the grayscale image into a binary image, determine the threshold through color features, and segment the entire belly region of the fish. Where, I ij represents the pixel value at the position (i, j) in the original image I, represents the binary image I * the pixel value at the position (i, j) in, and colorrange represents the preset color range (180, 230); S406: Establish a coordinate system at the axis of symmetry of the picture, and further segment the entire fish belly area through the position function of the colored fixture teeth, specifically: Establish a coordinate system with the axis of symmetry of the side view. The length direction of the fish is the X-axis, and the thickness direction of the fish is the Y-axis. Determine the function expression of the colored fixture teeth in the coordinate system, and further segment the entire fish belly area through the position function of the colored fixture teeth, which is: In the formula, y1 and y2 respectively represent the position functions of the two colored fixture teeth on the fish belly side, and a, b, c, and d are their parameters.

4. The method according to claim 3, wherein In S4, based on the preset feature extraction model for extracting the features of the target area, extract the features of the segmented target area, specifically: First, construct the convolutional layer, which is: In the formula, represents the output of the \(i\)-th convolutional kernel in the \(l\)-th layer at position \(j\), \(\sigma\) is the ReLU activation function, is the bias term of the \(i\)-th convolutional kernel in the \(l\)-th layer, is the weight matrix of the \(i\)-th convolutional kernel in the \(l\)-th layer corresponding to the \(k\)-th input feature map, is the \(k\)-th feature map of the \((l - 1)\)-th layer; Perform the pooling operation. The pooling layer is used to reduce the spatial dimension of the feature map and enhance the generalization ability of the model, which is: P(x,y) = max (a,b) ∈R (x,y) F(a,b); In the formula, P(x, y) is the output at the position (x, y) after the pooling operation, R(x, y) is the pooling window area centered on (x, y) on the input feature map, F(a, b) is the feature value at the position (a, b) in the pooling window, and |R(x, y)| is the number of elements in the pooling window; After completing the stacking of multiple such convolutional and pooling layers, further introduce residual connections to improve the efficiency and effect of training deep neural networks. The expression of the residual block is: where x is the input feature, represents the residual mapping with weights {W i}, and y is the output of the residual block.

5. The method according to claim 4, characterized in that In S5, according to the obtained pressure information of the seawater fish to be detected, establish a curve feature extraction method for processing the pressure curve and extract the pressure curve features, specifically: When the pressure sensor presses the fish meat at a fixed depth, the curve showing the change of pressure with the pressing depth is expressed as P(l) = {P(l1), P(l2), P(l3), …, P(l n ),}, where l i is the change in the pressing depth, and P(l i ) is the change in pressure generated with the change in the pressing depth; Establish a curve feature extraction method for processing the pressure curve and extract the pressure curve features. Among them, the local response ability of the pressure changing with the pressing depth is expressed as: In the formula, R(l) is the amplitude of the pressure change response at the depth l, P(l + Δl) and P(l - Δl) are the pressure values near the depth l, and Δl is the depth interval of the local window; The local curvature of the pressure curve is expressed as: wherein, is the local curvature at depth l, is the rate of change of pressure with respect to the pressing depth, is the acceleration of change of pressure with respect to the pressing depth; Recovery rate E of pressure during the release phase V Expressed as: Wherein, P max is the maximum pressure value during the pressing process, and P release is the pressure value when the pressure is released. Δt is the time difference between P max and P release .

6. The method according to claim 5, wherein In S6, fuse the processed image features and pressure curve features and perform dynamic adjustment of importance, specifically: Fuse the features extracted from the eyes and belly regions of seawater fish with the elastic features to form a unified feature vector X = [F 瞳孔 , F 虹膜 , F 鱼腹 , P 弹 , where Further optimize the feature combination based on dynamic importance adjustment. Take the feature vector X = [F 瞳孔 , F 虹膜 , F 鱼腹 , P 弹 as the input, perform a linear transformation on the comprehensive feature matrix X to obtain the query matrix Q, the key matrix K, and the value matrix V, as follows: where, W Q is the weight of the query matrix, W K is the weight of the key matrix, and W V is the weight of the value matrix; Calculate the similarity between the query matrix and the key matrix through dot product, and perform scaling and normalization to obtain the attention weight matrix, which is: where d k is the dimension of the key matrix; Multiply the weight matrix A by the value matrix V to obtain the weighted feature representation as: O = AV.

7. The method according to claim 6, wherein In S7, in combination with the judged freshness division standard, dynamically adjust the number of neurons in the output layer, construct a prediction model with the seawater fish image and pressure curve features as input values and the freshness grading of the seawater fish as output to predict the freshness of the seawater fish, and output the corresponding recommended uses of the fish meat, specifically: In combination with the judged freshness division standard, dynamically adjust the number of neurons in the output layer, construct a prediction model with the seawater fish image and pressure curve features as input values and the freshness grading of the seawater fish as output, and input the weighted feature representation O into the prediction model for feature fusion and freshness prediction; Among them, the prediction model includes an input layer, a hidden layer, and an output layer. The input layer is O, and each neuron in the input layer is fully connected to each neuron in the hidden layer 1 to form a weight matrix W m1 , the number of neurons in the hidden layer 1 is τ1, the dropout rate of the Dropout layer is 50%, the activation function is ReLU, and the input feature O is linearly combined with the weight matrix of the hidden layer 1 through a fully connected layer, and a bias vector b is added m1 , that is, Z m1 = W m1 ·O + b m1 , after further processing by the activation function, O m1 = ReLU(Z m1 ), and it is output to the Dropout layer. The Dropout layer randomly discards the outputs of some neurons, and the outputs O m1 * of the remaining neurons are passed to the hidden layer 2. The number of neurons in the hidden layer 2 is The hidden layer 2 receives the output O from the hidden layer 1 m1 * , and each neuron in the hidden layer 1 is fully connected to each neuron in the hidden layer 2 to form a weight matrix W m2 , and Z is obtained through a linear transformation by the fully connected layer m2 = W m2 ·O m1 * + b m2 , and through the ReLU activation function, the nonlinear characteristic is introduced to obtain O m2 = ReLU(Z m2 ), and the output layer receives the output O from the hidden layer 2 m2 , the third layer of the prediction model is the output layer. According to the different types of seawater fish, different freshness classification criteria and the number of levels τ2 are required. τ2 freshness classification levels correspond to τ2 neurons in the third layer, and the activation function is softmax.

8. A freshness detection device for seawater fish based on machine vision and FPS, characterized in that, Including: An installation box body, a top image capturing device, an infrared sensing and positioning device, a side industrial camera, an automatic lifting fixture device, a lighting device, a fixed-distance pressing device and a control unit. The automatic lifting fixture device is arranged at the inner bottom of the installation box body, the fixed-distance pressing device is arranged corresponding to the automatic lifting fixture device at the top of the installation box body, and the top image capturing device is arranged on the fixed-distance pressing device. The infrared sensing and positioning device is arranged corresponding to the fixed-distance pressing device inside the installation box body. The side industrial camera is arranged on one side inside the installation box body through a first bracket. A plurality of lighting devices are arranged on the upper part of the side wall of the installation box body. The control unit is arranged at the bottom of the installation box body. The top image capturing device, the infrared sensing and positioning device, the side industrial camera, the automatic lifting fixture device, the lighting device and the fixed-distance pressing device are all electrically connected to the control unit.

9. The device according to claim 8, characterized in that, The top image capturing device includes a top industrial camera, a second bracket, a support rod, a slider, a first slideway, a first coupling and a first stepping motor. The fixed-distance pressing device includes a third stepping motor, a second ball screw, a fixed baffle and a slide table. One end of the first slideway is arranged at the back of the installation box body, and the other end is arranged on the upper side of the top of the installation box body, and the first coupling is arranged at the top of this end. The first stepping motor is drivingly connected to the first coupling. The slider is slidably arranged on the first slideway. The first stepping motor drives the slider to slide along the first slideway through the first coupling. The support rod is arranged at the rear side of the bottom of the slider, and the second bracket is arranged at the bottom of the support rod. The top industrial camera is arranged on the second bracket. A base is arranged at the front side of the bottom of the slider, a chute is arranged inside the base, the third stepping motor is arranged at the top of the chute, the second ball screw is arranged at the output end of the third stepping motor, the slide table is connected to the second ball screw, the fixed baffle is arranged at the top of the slide table, a pressing block is arranged at the front side of the bottom of the slide table, a flexible pressure sensor is arranged on the pressing block, and the flexible pressure sensor is connected to the control unit.

10. The device according to claim 9, characterized in that The automatic lifting fixture device includes colored fixture teeth, a second slideway, a second stepping motor, a first ball screw, a support frame, a lifting table, a cylinder and a solenoid valve. The lifting table, the second stepping motor and the cylinder are arranged on the inner bottom of the installation box body. The second stepping motor is connected to the first ball screw through a second coupling. The first ball screw is connected to the support frame. The support frame is arranged at the slidable end of the lifting table. The second stepping motor drives the first ball screw to rotate through the second coupling, drives the support frame to move back and forth, and further drives the lifting table to realize lifting adjustment. Six second slideways are arranged on the peripheral side of the top of the lifting table. The colored fixture teeth are slidably arranged inside the second slideways. The cylinder is drivingly connected to the colored fixture teeth for driving the colored fixture teeth to slide along the second slideways. The cylinder is connected to a gas source through the solenoid valve. A flexible pressure sensor is arranged on the colored fixture teeth. The flexible pressure sensor is connected to the control unit.

Images