Intelligent fish recognition grading method combining visual detection and text semantic recognition

Abstract

The present application relates to the technical field of image recognition, and particularly relates to a fish intelligent recognition grading method fusing visual detection and text semantic recognition. According to the morphological features and the edge gradient features of each candidate highlight area, the water droplet tendency coefficient of each candidate highlight area is obtained, and the water droplet interference intensity of the fish eye image is obtained. According to the area and the gradient features of different candidate highlight areas, the effective reflective light area is screened out. According to the brightness distribution features of the pixel points in the effective reflective light area, the reflective light sharpness of the effective reflective light area is obtained. The rays along the different directions from the center of gravity position in the effective reflective light area are obtained, the visual interference dominant factor is obtained according to the brightness distribution of the pixel points on each ray and the water droplet tendency coefficient of different candidate highlight areas, and the final freshness of the fish sample is obtained to intelligently recognize and grade the fish. The present application can adaptively distinguish the deviation sources, obtain the final freshness, and improve the accuracy of fish recognition grading.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and specifically to a method for intelligent identification and classification of fish that integrates visual detection and text semantic recognition. Background Technology

[0002] As one of the core supporting technologies for aquaculture, the development level of fish intelligent identification and grading technology directly restricts the intelligentization process of the entire industry. Image-based automation technology can quickly and effectively separate fish species, improve operational efficiency, and reduce spoilage. Computer vision has been widely used in automated fish species classification to minimize human labor and facilitate faster decision-making. With the development of multimodal technologies, visual-text fusion methods have been applied in the field of food inspection, which improve the comprehensiveness of assessment by combining visual features with industry-standard text semantics.

[0003] In existing technologies, visual-text fusion methods based on simple weighted or contrastive learning strategies are used for intelligent identification and grading of fish. However, in aquatic industry standards, text descriptions themselves are ambiguous and subject to batch fluctuations. Directly using absolute values ​​for cross-modal hard alignment can lead to severe measurement distortion. Furthermore, real aquatic processing scenarios involve a large number of uncontrollable environmental disturbances such as water droplets, making it impossible for existing technologies to adaptively distinguish the source of deviation. This can easily lead to misjudging systematic visual deviations caused by environmental disturbances as a downgrade of the true quality of the fish, resulting in insufficient accuracy of grading results. Summary of the Invention

[0004] To address the technical problem that existing technologies cannot adaptively distinguish the sources of deviation, easily misjudging systematic visual deviations caused by environmental interference as a downgrade of the true quality of fish, thus leading to insufficient accuracy in grading results, the present invention aims to provide a fish intelligent identification and grading method that integrates visual detection and text semantic recognition. The specific technical solution adopted is as follows: This invention proposes an intelligent fish identification and classification method that integrates visual detection and text semantic recognition. The method includes: Obtain fish eye images of fish samples, along with textual descriptions and quantified values; Candidate highlight regions are selected based on the brightness characteristics of different pixels in the fisheye image. The water droplet tendency coefficient of each candidate highlight region is obtained based on its morphological and edge gradient characteristics. The water droplet interference intensity of the fisheye image is obtained based on the distribution of water droplet tendency coefficients across all candidate highlight regions and the number of candidate highlight regions. Effective reflective regions are selected based on the area and gradient characteristics of different candidate highlight regions. The reflective sharpness of the effective reflective regions is obtained based on the brightness distribution characteristics of pixels within the effective reflective regions. Rays along the centroid position in different directions within the effective reflective area are obtained. Based on the brightness distribution of pixels on each ray and the water droplet tendency coefficient of different candidate highlight areas, the pupil boundary contrast of the effective reflective area is obtained. Based on the deviation between the pupil boundary contrast of different fish samples and the text description quantization value, as well as the water droplet interference intensity, the visual interference dominant factor is obtained. The final freshness of fish samples is obtained based on the reflectivity of the fish sample, the contrast of the pupil boundary, the quantification value of the text description, and the dominant factor of visual interference, and the fish are intelligently identified and graded.

[0005] Furthermore, the method for obtaining the candidate highlight region includes: The brightness value of each pixel in the fisheye image in HSV space is obtained. The adaptive brightness threshold is obtained using the Otsu algorithm. If the brightness value of a pixel is greater than the adaptive brightness threshold, the corresponding pixel is taken as a candidate highlight point, and the region formed by adjacent candidate highlight points is taken as a candidate highlight region.

[0006] Furthermore, the method for obtaining the water droplet tendency coefficient includes: Obtain the gradient magnitude of each edge pixel in the candidate highlight region, and obtain the ratio of the cumulative gradient magnitude of all edge pixels to the total number of edge pixels, which is used as the average edge gradient value of the candidate highlight region. The minimum relative distance between different pixels of the candidate highlight region and other candidate highlight regions is obtained as the reference distance between the candidate highlight region and other candidate highlight regions; the minimum reference distance between the candidate highlight region and all other candidate highlight regions is selected as the nearest neighbor distance. The product of the number of pixels in the candidate highlight region, the nearest neighbor distance, and the average gradient value at the edge is obtained and negatively correlated and normalized to form the water droplet tendency coefficient.

[0007] Furthermore, the method for obtaining the water droplet interference intensity includes: The average water droplet tendency coefficient of all candidate highlight regions is obtained as the overall water droplet tendency level. If the number of all candidate highlight regions is greater than or equal to the preset maximum expected number, the water droplet interference coefficient is set to 1; if the number of all candidate highlight regions is less than the preset maximum expected number, the ratio of the number of all candidate highlight regions to the preset maximum expected number is obtained as the water droplet interference coefficient. The product of the overall water droplet tendency level and the water droplet interference coefficient is obtained as the water droplet interference intensity.

[0008] Furthermore, the method for obtaining the effective reflective area includes: Select the median number of pixels in all candidate highlight regions and the median average gradient value at the edges; If both the number of pixels and the average edge gradient value within a candidate highlight region are greater than the median, the corresponding candidate highlight region is selected as a strong candidate region. If neither the number of pixels nor the average edge gradient value within a candidate highlight region are greater than the median, the candidate highlight region with the largest number of pixels is selected as a strong candidate region. Expanding outward from strong candidate regions as seed points, if an adjacent pixel belongs to a candidate highlight region, it is incorporated into the expanded region; until none of the adjacent pixels belong to a candidate highlight region, the expansion stops, and the connected region where the expansion ends is taken as the candidate reflective region; the candidate reflective region with the largest number of pixels among all candidate reflective regions is selected as the effective reflective region.

[0009] Furthermore, the method for obtaining the reflective sharpness includes: Obtain the gradient magnitude of each edge pixel in the effective reflective region, obtain the ratio of the cumulative gradient magnitude of all edge pixels to the total number of edge pixels, and normalize it to serve as the edge gradient feature of the effective reflective region. The ratio of the average brightness to the maximum brightness of all pixels in the effective reflective area is obtained as the brightness ratio; the difference between the constant 1 and half of the water droplet interference intensity is obtained as the reflectivity. The product of the edge gradient features of the effective reflective area, the brightness ratio, and the reflectivity is obtained as the reflectivity sharpness.

[0010] Furthermore, the method for obtaining the pupil boundary contrast includes: Based on the brightness distribution of pixels on each ray, the boundary pixels on each ray are obtained, and the brightness difference between the inner and outer sides of the boundary pixels is obtained. The sum of the water droplet tendency coefficients of all pixels in the candidate specular region within the neighborhood of the upper boundary pixel of each ray is obtained as the local interference weight of each ray. Obtain the cumulative value of the product between the local interference weights of all rays and the brightness difference, and calculate the ratio of the cumulative product to the cumulative sum of the local interference weights of all rays as the pupil boundary contrast.

[0011] Further, obtaining the boundary pixels on each ray and the brightness difference between the inner and outer sides of the boundary pixels includes: Obtain the brightness sequence of pixels along each ray direction, obtain the difference between the next element and the previous element in the brightness sequence to form a brightness difference sequence; obtain the element with the largest maximum value in the brightness difference sequence and take the corresponding pixel as the boundary pixel. The difference in the average brightness of each boundary pixel on its inner and outer sides, within a predetermined number of pixels, is used as the first difference. The ratio between the first difference and the average brightness of a preset number of pixels outside the boundary pixel is obtained as the brightness difference degree.

[0012] Furthermore, the method for obtaining the dominant factor of visual interference includes: The difference between the pupil boundary contrast and the text description quantization value for each fish sample is obtained as the semantic bias. The slope of the fitted line is obtained by linearly fitting the data points composed of semantic bias and water droplet interference intensity of different fish samples. Obtain the product between the absolute value of the slope of the fitted line and the intensity of the water droplet interference, and normalize it; obtain the product between the sign value of the slope of the fitted line and the normalization result, and use it as the dominant factor of visual interference.

[0013] Furthermore, the method for obtaining the final freshness includes: The product of reflective sharpness and pupil boundary contrast is obtained as visual acuity. The mean of visual clarity and text description quantification is obtained as the initial fusion score; the difference between constant 1 and the dominant factor of visual interference is obtained, the product between the difference and the initial fusion score is calculated, and the minimum value between the product and the preset freshness upper limit is selected as the final freshness.

[0014] The present invention has the following beneficial effects: This invention obtains the water droplet tendency coefficient for each candidate highlight region based on its morphological features and edge gradient features, reflecting the physical similarity between the morphological features of a single highlight region and a water droplet; it obtains the water droplet interference intensity of the fisheye image based on the distribution of the water droplet tendency coefficients of all candidate highlight regions and the number of candidate highlight regions, reflecting the severity of objective physical interference in the current image; it filters out effective reflective regions based on the area and gradient features of different candidate highlight regions; and it obtains the reflective sharpness of the effective reflective regions based on the brightness distribution features of pixels in the effective reflective regions, reflecting the transparency and smoothness of the cornea; This invention obtains rays radiating from the centroid of the effective reflective area in different directions. Based on the brightness distribution of pixels on each ray and the water droplet tendency coefficient of different candidate highlight areas, the pupil boundary contrast of the effective reflective area is obtained, reflecting the clarity between the pupil and iris and the turbidity of the ocular fluid. Based on the deviation between the pupil boundary contrast and the text description quantification value of different fish samples, as well as the water droplet interference intensity, a visual interference dominance factor is obtained, quantifying the degree of dominance of visual interference. Based on reflective sharpness, pupil boundary contrast, text description quantification value, and visual interference dominance factor, the final freshness of the fish sample is obtained, enabling intelligent identification and grading of fish. This invention improves the accuracy of fish identification and grading by adaptively distinguishing the sources of deviation to obtain the final freshness. Attached Figure Description

[0015] Figure 1 A flowchart illustrating a method for intelligent fish identification and grading that integrates visual detection and text semantic recognition, provided in one embodiment of the present invention; Figure 2 A flowchart illustrating a method for obtaining a water droplet tendency coefficient according to an embodiment of the present invention; Figure 3 This is a schematic diagram of an effective reflective area provided in one embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the distribution of a normal pupil boundary according to an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the comparison of fish freshness grading performance under different methods according to an embodiment of the present invention. Detailed Implementation

[0016] The following description, in conjunction with the accompanying drawings, details a specific scheme for an intelligent fish identification and grading method that integrates visual detection and text semantic recognition provided by the present invention.

[0017] Please see Figure 1 The diagram illustrates a flowchart of a fish intelligent identification and classification method integrating visual detection and text semantic recognition, provided by an embodiment of the present invention. The specific method includes: Step S1: Obtain the fish eye image of the fish sample, as well as the text description quantization value.

[0018] In an embodiment of the present invention, on a factory-scale chilled fish grading production line, after the fish have been deheaded and gutted, and before they are rinsed, a high-resolution RGB camera is set up, and a standard light source is configured to illuminate the fish eye area at a 45° angle. For each fish to be graded, the fish eyes are uniformly facing upwards through physical constraints or visual positioning on the production line, and a single high-resolution close-up image of the fish eye is captured. At the same time, quality inspectors independently record the freshness judgment of the fish sample and the observed interference phenomena based on industry grading standards, such as the qualitative descriptions of clear and bright fish eyes and clear pupils in seafood grading standards, forming a natural language text description.

[0019] It should be noted that a keyword mapping table containing feature keywords and initial quantification values ​​is pre-constructed based on seafood grading standards. After obtaining the text description of the fish sample, natural language processing techniques, such as TF-IDF or dictionary-based maximum positive matching, are used to extract feature keywords from the text. The corresponding initial quantification value and correction coefficient are then found in the keyword mapping table, and multiplied to obtain the initial quantification value of the fish sample's text description. The initial quantification values ​​of all fish within a pre-defined historical time range are sorted in ascending order to obtain the number of fish samples whose quantification values ​​are less than or equal to the initial quantification value of the analyzed fish sample. This number is then divided by the total number of fish within the historical time range to obtain the normalized percentile ranking, which is the text description quantification value of the analyzed fish sample.

[0020] It should be noted that in the keyword mapping table of this embodiment, the qualitative classification terms of cornea and pupil are divided into different levels and assigned initial quantification values: clear and bright fish eyes and completely transparent corneas correspond to a quantification value of 0.9; slightly cloudy fish eyes and slightly blurred pupil boundaries correspond to a quantification value of 0.6; cloudy and opaque fish eyes and pupils that are difficult to identify or completely blurred correspond to a quantification value of 0.2. At the same time, correction coefficients are assigned to interference descriptions: no interference corresponds to a coefficient of 1.0, local water droplets / mucus correspond to a coefficient of 0.8, and a large amount of blood / water droplets correspond to a coefficient of 0.5.

[0021] It should be noted that, in the embodiments of the present invention, the historical time range of obtaining a preset number of fish samples is determined in advance based on the average flow rate of the production line. For example, the preset number can be set to 50, that is, the 50 fish samples most recently passed by the production line are analyzed.

[0022] Step S2: Based on the brightness characteristics of different pixels in the fisheye image, candidate highlight regions are selected; based on the morphological characteristics and edge gradient characteristics of each candidate highlight region, the water droplet tendency coefficient of each candidate highlight region is obtained; based on the distribution of water droplet tendency coefficients of all candidate highlight regions and the number of candidate highlight regions, the water droplet interference intensity is obtained; based on the area and gradient characteristics of different candidate highlight regions, effective reflective regions are selected; based on the brightness distribution characteristics of pixels in the effective reflective regions, the reflective sharpness of the effective reflective regions is obtained.

[0023] In actual visual acquisition, fisheye and surrounding areas are often covered with condensation droplets and body mucus, resulting in multiple tiny specular reflection spots on the corneal surface, which are mixed with the real corneal specular reflection points. Therefore, candidate highlight areas are selected based on the brightness characteristics of different pixels in the fisheye image.

[0024] Preferably, in one embodiment of the present invention, the method for obtaining candidate highlight regions includes: The brightness value of each pixel in the fisheye image in HSV space is obtained. The adaptive brightness threshold is obtained using the Otsu algorithm. If the brightness value of a pixel is greater than the adaptive brightness threshold, the corresponding pixel is taken as a candidate highlight point, and the region formed by adjacent candidate highlight points is taken as a candidate highlight region.

[0025] It should be noted that the Otsu algorithm is a well-known technique in the field, and will not be elaborated upon here.

[0026] When there are tiny water droplets or a layer of mucus on the surface of a fish eye, multiple tiny highlight spots will be generated under illumination. The reflection points formed by water droplets are usually small in area, numerous, and have relatively gentle edge gradients because water droplets are approximately spherical, and the center of the reflected light spot is bright but the edges are blurred. In contrast, the specular reflection points of a fresh cornea are larger in area, fewer in number, and have steep edge gradients. Based on the morphological characteristics and edge gradient characteristics of each candidate highlight region, the water droplet tendency coefficient of each candidate highlight region is obtained.

[0027] Preferably, in one embodiment of the present invention, the method for obtaining the water droplet tendency coefficient is described in [reference needed]. Figure 2 It shows a flowchart of a method for obtaining the water droplet tendency coefficient, including: Step S201: Obtain the gradient magnitude of each edge pixel in the candidate highlight region, and obtain the ratio of the cumulative gradient magnitude of all edge pixels to the total number of edge pixels, as the average gradient value of the edge of the candidate highlight region.

[0028] It should be noted that in some embodiments of the present invention, the gradient magnitude of edge pixels is obtained based on the Sobel operator or the Scharr operator, highlighting the sensitivity to the edge capture of tiny water droplets; the specific means are well known to those skilled in the art and will not be described in detail here.

[0029] Step S202: Obtain the minimum relative distance between different pixels of the candidate highlight region and other candidate highlight regions, and use it as the reference distance between the candidate highlight region and other candidate highlight regions; select the minimum reference distance between the candidate highlight region and all other candidate highlight regions as the nearest neighbor distance.

[0030] It should be noted that, in the embodiments of the present invention, the relative distance between the coordinates of different pixels is obtained by existing distance calculation methods such as Euclidean distance or Manhattan distance; the smaller the relative distance, the closer the candidate highlight regions are; in particular, if the number of candidate highlight regions selected is only one, it is more likely to be a real corneal specular reflection, so the preset maximum distance constant is directly used as the nearest neighbor distance, and the preset maximum distance constant is specifically defined as the diagonal length of the fisheye image; the specific means are well known to those skilled in the art and will not be described in detail here.

[0031] Step S203: Obtain the product of the number of pixels in the candidate highlight region, the nearest neighbor distance, and the average gradient value of the edge, and perform negative correlation normalization mapping as the water droplet tendency coefficient.

[0032] It should be noted that the fewer the number of pixels, the smaller the area of ​​the candidate highlight region, and the smaller the average gradient value at the edge. The edge gradient is relatively gentle, and it is more likely to be a reflection point formed by water droplets. The smaller the nearest neighbor distance, the closer the candidate highlight regions are to each other. There are more candidate highlight regions, and it is more likely to be a reflection point formed by water droplets.

[0033] It should be noted that, in the embodiments of the present invention, the reciprocal of the product is used for negative correlation mapping. Considering that the average gradient at the edge may be 0, a very small positive number with consistent dimensions is added to the denominator, and its value is specifically set according to the range of the denominator. After negative correlation mapping, maximum and minimum value normalization is adopted, that is, all candidate highlight regions are analyzed, and the maximum and minimum values ​​of the negative correlation mapping results under the dimension to be normalized are selected and normalized to the range of 0-1. If there is only one candidate highlight region, its water droplet tendency coefficient is directly set to a preset constant value of 0. The specific means are well known to those skilled in the art and will not be described in detail here.

[0034] The water droplet tendency coefficient reflects the morphological characteristics of a single highlight region and its physical similarity to a water droplet. The larger the water droplet tendency coefficient, the greater the physical similarity to a water droplet. The number of candidate highlight regions reflects the overall coverage density and distribution breadth of water droplets on the fisheye surface. The more regions there are, the greater the coverage density and distribution breadth of water droplets, and the greater the intensity of water droplet interference. The water droplet interference intensity is obtained based on the distribution of water droplet tendency coefficients of all candidate highlight regions and the number of candidate highlight regions.

[0035] Preferably, in one embodiment of the present invention, the method for obtaining the water droplet interference intensity includes: The average water droplet tendency coefficient of all candidate highlight regions is obtained as the overall water droplet tendency level. If the number of all candidate highlight regions is greater than or equal to the preset maximum expected number, the water droplet interference coefficient is set to 1; if the number of all candidate highlight regions is less than the preset maximum expected number, the ratio of the number of all candidate highlight regions to the preset maximum expected number is obtained as the water droplet interference coefficient. The product of the overall water droplet tendency level and the water droplet interference coefficient is obtained as the water droplet interference intensity.

[0036] It should be noted that, in the embodiments of the present invention, the preset maximum expected number is based on the physical statistical experience boundary of the real reflective points and water droplet interference points that may appear in the fisheye area. If the preset maximum expected number is exceeded, the water droplet interference coefficient reaches a maximum of 1. Based on relevant historical experience, the preset maximum expected number is set to 20. In other embodiments of the present invention, the size of the preset maximum expected number can be set according to the specific situation, and will not be limited or elaborated here.

[0037] False reflective points generated by water droplets can be mixed in, causing an overestimation of the presence of multiple small reflective points on the corneal surface while ignoring the true corneal reflection. Considering that the specular reflective points of a fresh cornea have a large area, a small number, and a steep edge gradient, strong candidate regions and effective reflective regions are selected based on the area and gradient characteristics of different candidate highlight regions.

[0038] Preferably, in one embodiment of the present invention, the method for obtaining the effective reflective area includes: Select the median number of pixels in all candidate highlight regions and the median average gradient value at the edges; If both the number of pixels and the average edge gradient value within a candidate highlight region are greater than the median, the corresponding candidate highlight region is selected as a strong candidate region. If neither the number of pixels nor the average edge gradient value within a candidate highlight region are greater than the median, the candidate highlight region with the largest number of pixels is selected as a strong candidate region. Expanding outward from strong candidate regions as seed points, if an adjacent pixel belongs to a candidate highlight region, it is incorporated into the expanded region; until none of the adjacent pixels belong to a candidate highlight region, the expansion stops, and the connected region where the expansion ends is taken as the candidate reflective region; the candidate reflective region with the largest number of pixels among all candidate reflective regions is selected as the effective reflective region.

[0039] Therefore, the more pixels a candidate highlight region has, and the greater its average edge gradient, the steeper the edge gradient, and the clearer the edge, the greater the likelihood that it is corneal reflection. Thus, it is selected as a strong candidate region and expanded to obtain an effective reflective region, such as... Figure 3 It shows a schematic diagram of an effective reflective area.

[0040] Brightness distribution reflects the brightness of reflective points; a fresh cornea is like a mirror, with extremely high brightness of reflective points, and the boundary between the highlight area and the surrounding dark area is very clear, with a greater gradient at the boundary; based on the brightness distribution characteristics of pixels in the effective reflective area, the reflective sharpness of the effective reflective area is obtained.

[0041] Preferably, in one embodiment of the present invention, the method for obtaining reflective sharpness includes: Obtain the gradient magnitude of each edge pixel in the effective reflective region, obtain the ratio of the cumulative gradient magnitude of all edge pixels to the total number of edge pixels, and normalize it to serve as the edge gradient feature of the effective reflective region. It should be noted that, in the embodiments of the present invention, all candidate highlight regions are analyzed, the maximum value of the ratio is selected, and the ratio of the ratio result to the maximum value is calculated, i.e., normalized, and used as the edge gradient feature of the effective reflective region.

[0042] The ratio of the average brightness to the maximum brightness of all pixels in the effective reflective area is obtained as the brightness ratio; the difference between the constant 1 and half of the water droplet interference intensity is obtained as the reflectivity. The product of the edge gradient features of the effective reflective area, the brightness ratio, and the reflectivity is obtained as the reflectivity sharpness.

[0043] Based on this, the larger the gradient value, the steeper the edge gradient, and the more reflective the corneal surface. The intensity of water droplet interference reflects the degree of influence of water droplet interference. The greater the degree of influence of water droplet interference, the lower the reliability of the reflection, the overestimation of the reflection on the corneal surface, and the lower the sharpness of the reflection.

[0044] The formula for reflective sharpness is expressed as: ;in, Indicates the reflective sharpness of the effective reflective area; The edge gradient characteristics representing the effective reflective area; This represents the maximum brightness value of a pixel within the effective reflective area; This represents the average brightness of pixels within the effective reflective area; This indicates the intensity of water droplet interference.

[0045] Step S3: Obtain rays from the centroid position in different directions within the effective reflective area. Based on the brightness distribution of pixels on each ray and the water droplet tendency coefficient of different candidate highlight areas, obtain the pupil boundary contrast of the effective reflective area. Based on the deviation between the pupil boundary contrast of different fish samples and the text description quantization value, as well as the water droplet interference intensity, obtain the dominant factor of visual interference.

[0046] Since the pupil is a nearly circular area, the center of gravity of the effective reflective area is usually located at the edge of the pupil or very close to the center. Radiation is emitted from this center outwards, essentially performing a full-range radar scan from the inside out. Then, the search proceeds radially outwards to find the location of the largest brightness jump, i.e., the pupil boundary. Figure 4It shows a schematic diagram of the distribution of a normal pupil boundary; multi-directional ray scanning is used to avoid positioning failure caused by damage or water droplet interference when unidirectional scanning is used; rays in different directions along the center of gravity within the effective reflective area are obtained; it should be noted that in the embodiments of the present invention, a ray is divided into 15° intervals starting from the center of gravity for analysis.

[0047] In a fish's eye, the pupil area is darker in color, while the reflective points of the iris or cornea are usually located near the edge of the pupil. Due to optical principles, the specular reflective points are located at the point of maximum corneal curvature, which is closer to the edge of the pupil. In a fresh fish eye, there is a significant brightness change between the pupil and the iris. When the eye is not fresh, the brightness change is reduced due to the diffusion of iris pigment or blurred boundaries. Therefore, based on the brightness distribution of pixels on each ray and the water droplet tendency coefficient of different candidate highlight areas, the pupil boundary contrast of the effective reflective area can be obtained.

[0048] Preferably, in one embodiment of the present invention, the method for obtaining pupil boundary contrast includes: The first step is to obtain the boundary pixels on each ray based on the brightness distribution of the pixels on each ray, and to obtain the brightness difference between the inner and outer sides of the boundary pixels. Preferably, in one embodiment of the present invention, obtaining the boundary pixels on each ray and obtaining the brightness difference between the inner and outer sides of the boundary pixels includes: Obtain the brightness sequence of pixels along each ray direction, obtain the difference between the next element and the previous element in the brightness sequence to form a brightness difference sequence; obtain the element with the largest maximum value in the brightness difference sequence and take the corresponding pixel as the boundary pixel. The difference in the average brightness of each boundary pixel on its inner and outer sides, within a predetermined number of pixels, is used as the first difference. The ratio between the first difference and the average brightness of a preset number of pixels outside the boundary pixel is obtained as the brightness difference degree.

[0049] It should be noted that the difference represents the absolute value of the difference. The inner side of the boundary pixel is the side closest to the centroid, and the outer side of the boundary pixel is the side along the direction of the ray extension. The boundary pixel is the position where the brightness changes the most from dark to bright on the ray. Therefore, the brightness on the outer side of the boundary pixel is greater than the brightness on the inner side. The closer the brightness difference is to 1, the greater the brightness difference between the inner and outer sides. The closer the brightness difference is to 0, the smaller the brightness difference between the inner and outer sides. In one embodiment of the present invention, considering that there is a certain gradual transition zone at the junction of the pupil and iris of a real fisheye, a sufficient number of pixels need to be obtained for brightness difference analysis. Based on relevant historical experience, the preset number is set to 5. In other embodiments of the present invention, the preset number can be adaptively set according to the spatial resolution of the actual image and the actual width of the transition zone of the fisheye pupil. It is not limited or elaborated here.

[0050] The second step is to obtain the sum of the water droplet tendency coefficients of all pixels in the candidate specular region within the neighborhood of the upper boundary pixel of each ray, and use it as the local interference weight of each ray. The third step is to obtain the cumulative value of the product between the local interference weights of all rays and the brightness difference, and calculate the ratio of the cumulative product to the cumulative sum of the local interference weights of all rays as the pupil boundary contrast.

[0051] It should be noted that, in the embodiments of the present invention, the neighborhood range is the range formed by selecting 8 neighboring pixels based on the boundary pixel. In other embodiments of the present invention, the size of the neighborhood range can be set according to specific circumstances, and will not be limited or elaborated here.

[0052] The formula for pupil boundary contrast is expressed as: ,in, Indicates the contrast of the pupil boundary; Indicates the first The sum of the water droplet tendency coefficients of all candidate specular regions belonging to the neighborhood of the pixel at the upper boundary of the ray; Indicates the degree of brightness difference; Indicates the number of rays.

[0053] It should be noted that if the sum of the local interference weights of all rays is equal to 0, then the mean of the brightness differences of all rays is used as the pupil boundary contrast.

[0054] The text description quantification value reflects the subjective freshness expectation judged by quality inspectors based on industry standards; the pupil boundary contrast reflects the objective freshness assessment calculated by machine vision equipment from physical images; the difference between the two reflects how much the machine vision deviates from the quality inspector's judgment due to external optical interference; the water droplet interference intensity reflects the severity of objective physical interference, thus quantifying the degree to which water droplet interference dominates; based on the deviation between the pupil boundary contrast and the text description quantification value of different fish samples, as well as the water droplet interference intensity, the visual interference dominance factor is obtained.

[0055] Preferably, in one embodiment of the present invention, the method for obtaining the dominant factor of visual interference is as follows: The difference between the pupil boundary contrast and the text description quantization value for each fish sample is obtained as the semantic bias. The slope of the fitted line is obtained by linearly fitting the data points composed of semantic bias and water droplet interference intensity of different fish samples. It should be noted that the larger the difference, the greater the semantic deviation, and the higher the expected evaluation of visual extraction compared to text description; the smaller the difference, the smaller the semantic deviation, and the lower the expected evaluation of visual extraction compared to text description. If the intensity of water droplet interference decreases as semantic deviation increases and the slope is negative, it means that as interference increases, the deviation shifts in the negative direction, and the interference causes the contrast to be underestimated. Conversely, when the slope is positive, the interference causes the contrast to be artificially high, and the impact of the interference is greater.

[0056] Obtain the product between the absolute value of the slope of the fitted line and the intensity of the water droplet interference, and normalize it; obtain the product between the sign value of the slope of the fitted line and the normalization result, and use it as the dominant factor of visual interference.

[0057] It should be noted that, in the embodiments of the present invention, the product between the absolute value of the slope of the fitted straight line and the intensity of the water droplet interference is obtained as the first product; the ratio of the first product to the sum of the first product and the smallest positive number is calculated, i.e., normalization is performed; so that the original value of any size in the range [0, +∞) is compressed and mapped to the fixed interval [0, 1); wherein, in order to avoid the denominator being 0, a smallest positive number with the same dimensions is added, and its value is specifically set according to the range of the denominator, which will not be elaborated here.

[0058] Based on this, the sign value of the slope of the fitted line is 1. The closer the normalization result is to 1, the closer the dominant factor of visual interference is to 1, the more significant the interference, and the more the visual contrast is artificially inflated, requiring a reduction in visual evaluation. The sign value of the slope of the fitted line is -1. The closer the normalization result is to 1, the smaller the dominant factor of visual interference, the more significant the interference, and the more the visual contrast is underestimated, requiring a compensation for visual evaluation. The closer the normalization result is to 0, the smaller the dominant factor of visual interference, the less the interference, and the more likely it is to reflect the true quality difference, maintaining the original evaluation unchanged.

[0059] Step S4: Based on the reflectivity of the fish sample, the contrast of the pupil boundary, the quantification value of the text description, and the dominant factor of visual interference, the final freshness of the fish sample is obtained, and the fish are intelligently identified and graded.

[0060] Reflective sharpness reflects the transparency and smoothness of the cornea; the more transparent and smooth the cornea, the stronger the specular reflection and the sharper the edges. Pupil boundary contrast reflects the clarity between the pupil and iris and the turbidity of the ocular fluid; fresh fish eyes have deep, dark pupils with a clear boundary with the iris, while decomposition results in cloudy fluid, blurring the boundary and decreasing contrast. The textual description quantification value reflects the quality inspector's subjective expectation of freshness based on industry standards. The visual interference dominance factor reflects the degree of visual interference; the larger the visual interference dominance factor, the higher the overestimation of visual interference, potentially leading to water droplets being identified as corneal reflection, and the freshness rating should be lowered. Combining multi-feature analysis helps to more accurately and comprehensively quantify freshness.

[0061] Preferably, in one embodiment of the present invention, the method for obtaining the final freshness includes: The product of reflective sharpness and pupil boundary contrast is obtained as visual acuity. The mean of visual clarity and text description quantification is obtained as the initial fusion score; the difference between constant 1 and the dominant factor of visual interference is obtained, the product between the difference and the initial fusion score is calculated, and the minimum value between the product and the preset freshness upper limit is selected as the final freshness.

[0062] Based on this, the greater the reflective sharpness, the greater the pupil boundary contrast, and the greater the text description quantification value, resulting in a higher final freshness. Conversely, the smaller the reflective sharpness, the smaller the pupil boundary contrast, the more likely the cornea is cloudy and the less fresh it is. After obtaining the initial fusion score using a simple averaging method, the bias caused by systematic visual interference is corrected using the dominant visual interference factor. The larger the positive value of the dominant visual interference factor, the more the interference artificially inflates the visual contrast, requiring a reduction in visual evaluation and thus a lower initial fusion score. Conversely, the larger the negative value of the dominant visual interference factor, the more the interference underestimates the visual contrast, requiring compensation for visual evaluation and thus a higher initial fusion score.

[0063] It should be noted that, in the embodiments of the present invention, the initial fusion score ranges from 0 to 1, and the difference between the constant 1 and the dominant visual interference factor ranges from 0 to 2. Therefore, the product ranges from 0 to 2. In the physical logic, the product exceeding 1 occurs under specific conditions where the fish sample itself has a high initial fusion score and encounters severe visual underestimation. To ensure that the output results meet the standard quantization system, the preset freshness upper limit is set to 1. When the product exceeds the upper limit, a threshold truncation is performed to limit the final freshness to the range of 0 to 1, thus ensuring the scale invariance of the multimodal fusion score.

[0064] It should be noted that, in another embodiment of the present invention, the intelligent identification and grading of fish based on the final freshness includes: if the final freshness is greater than or equal to a preset first threshold, the corresponding fish is classified as a first freshness level; If the final freshness is greater than or equal to the preset second threshold and less than the preset first threshold, the corresponding fish will be classified as the second freshness level. If the final freshness is less than the preset second threshold, the corresponding fish will be classified as the third freshness level.

[0065] It should be noted that, in the embodiments of the present invention, the greater the final freshness, the greater the freshness level of the fish classification, that is, the preset first threshold is greater than the preset second threshold, the first freshness level is greater than the second freshness level, and the second freshness level is greater than the third freshness level; the preset first threshold is set to 0.8 based on relevant historical experience; the preset second threshold is set to 0.5; in other embodiments of the present invention, the preset first threshold and the preset second threshold can be specifically set according to the specific situation, and are not limited or elaborated here.

[0066] Based on this, considering the systematic bias of visual measurements caused by interference factors in real processing scenarios, as well as the ambiguity and batch fluctuations of text descriptions themselves, the accuracy of grading results should be improved; for example... Figure 5 It shows a schematic diagram comparing the performance of fish freshness grading under different methods.

[0067] In summary, this invention obtains the water droplet tendency coefficient of each candidate highlight region based on its morphological features and edge gradient features, and thus obtains the water droplet interference intensity of the fisheye image. Based on the area and gradient features of different candidate highlight regions, effective reflective regions are selected. Based on the brightness distribution features of pixels within the effective reflective regions, the reflective sharpness of the effective reflective regions is obtained. Rays extending in different directions along the centroid position within the effective reflective regions are obtained, and based on the brightness distribution of pixels on each ray and the water droplet tendency coefficient of different candidate highlight regions, the dominant factor of visual interference is obtained. Finally, the freshness of the fish sample is obtained, enabling intelligent identification and grading of fish. This invention improves the accuracy of fish identification and grading by adaptively distinguishing the sources of deviation to obtain the final freshness.

[0068] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not reflect the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0069] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A method for intelligent fish identification and classification that integrates visual detection and text semantic recognition, characterized in that, The method includes: The process involves acquiring fish eye images and text description quantization values ​​for fish samples. The text description quantization values ​​are obtained as follows: after acquiring the text description of the fish sample, feature keywords are extracted from the text description using natural language processing techniques; the initial quantization value and correction coefficient corresponding to the feature keywords are found in a pre-constructed keyword mapping table, and multiplied to obtain the initial quantization value of the fish sample's text description; the initial quantization values ​​of all fish samples within a pre-defined historical time range are sorted in ascending order, and the number of fish samples whose initial quantization values ​​are less than or equal to the initial quantization value of the fish sample is obtained. This number is then divided by the total number of fish samples within the historical time range to obtain the text description quantization value of the fish sample. Candidate highlight regions are selected based on the brightness characteristics of different pixels in the fisheye image. The water droplet tendency coefficient of each candidate highlight region is obtained based on its morphological and edge gradient characteristics. The water droplet interference intensity of the fisheye image is obtained based on the distribution of water droplet tendency coefficients across all candidate highlight regions and the number of candidate highlight regions. Effective reflective regions are selected based on the area and gradient characteristics of different candidate highlight regions. The reflective sharpness of the effective reflective regions is obtained based on the brightness distribution characteristics of pixels within the effective reflective regions. Rays along the centroid position in different directions within the effective reflective area are obtained. Based on the brightness distribution of pixels on each ray and the water droplet tendency coefficient of different candidate highlight areas, the pupil boundary contrast of the effective reflective area is obtained. Based on the deviation between the pupil boundary contrast of different fish samples and the text description quantization value, as well as the water droplet interference intensity, the visual interference dominant factor is obtained. The final freshness of fish samples is obtained based on the reflectivity of the fish sample, the contrast of the pupil boundary, the quantification value of the text description, and the dominant factor of visual interference, and the fish are intelligently identified and graded.

2. The intelligent fish identification and grading method integrating visual detection and text semantic recognition according to claim 1, characterized in that, The method for obtaining the candidate highlight region includes: The brightness value of each pixel in the fisheye image in HSV space is obtained. The adaptive brightness threshold is obtained using the Otsu algorithm. If the brightness value of a pixel is greater than the adaptive brightness threshold, the corresponding pixel is taken as a candidate highlight point, and the region formed by adjacent candidate highlight points is taken as a candidate highlight region.

3. The intelligent fish identification and grading method integrating visual detection and text semantic recognition according to claim 1, characterized in that, The method for obtaining the water droplet tendency coefficient includes: Obtain the gradient magnitude of each edge pixel in the candidate highlight region, and obtain the ratio of the cumulative gradient magnitude of all edge pixels to the total number of edge pixels, which is used as the average edge gradient value of the candidate highlight region. The minimum relative distance between different pixels of the candidate highlight region and other candidate highlight regions is obtained as the reference distance between the candidate highlight region and other candidate highlight regions; the minimum reference distance between the candidate highlight region and all other candidate highlight regions is selected as the nearest neighbor distance. The product of the number of pixels in the candidate highlight region, the nearest neighbor distance, and the average gradient value at the edge is obtained and negatively correlated and normalized to form the water droplet tendency coefficient.

4. The intelligent fish identification and grading method integrating visual detection and text semantic recognition according to claim 1, characterized in that, The method for obtaining the water droplet interference intensity includes: The average water droplet tendency coefficient of all candidate highlight regions is obtained as the overall water droplet tendency level. If the number of all candidate highlight regions is greater than or equal to the preset maximum expected number, the water droplet interference coefficient is set to 1; if the number of all candidate highlight regions is less than the preset maximum expected number, the ratio of the number of all candidate highlight regions to the preset maximum expected number is obtained as the water droplet interference coefficient. The product of the overall water droplet tendency level and the water droplet interference coefficient is obtained as the water droplet interference intensity.

5. The intelligent fish identification and grading method integrating visual detection and text semantic recognition according to claim 3, characterized in that, The method for obtaining the effective reflective area includes: Select the median number of pixels in all candidate highlight regions and the median average gradient value at the edges; If both the number of pixels and the average edge gradient value within a candidate highlight region are greater than the median, the corresponding candidate highlight region is selected as a strong candidate region. If neither the number of pixels nor the average edge gradient value within a candidate highlight region are greater than the median, the candidate highlight region with the largest number of pixels is selected as a strong candidate region. Expanding outward from strong candidate regions as seed points, if an adjacent pixel belongs to a candidate highlight region, it is incorporated into the expanded region; until none of the adjacent pixels belong to a candidate highlight region, the expansion stops, and the connected region where the expansion ends is taken as the candidate reflective region; the candidate reflective region with the largest number of pixels among all candidate reflective regions is selected as the effective reflective region.

6. The intelligent fish identification and grading method integrating visual detection and text semantic recognition according to claim 1, characterized in that, The method for obtaining the reflective sharpness includes: Obtain the gradient magnitude of each edge pixel in the effective reflective region, obtain the ratio of the cumulative gradient magnitude of all edge pixels to the total number of edge pixels, and normalize it to serve as the edge gradient feature of the effective reflective region. The ratio of the average brightness to the maximum brightness of all pixels in the effective reflective area is obtained as the brightness ratio; the difference between the constant 1 and half of the water droplet interference intensity is obtained as the reflectivity. The product of the edge gradient features of the effective reflective area, the brightness ratio, and the reflectivity is obtained as the reflectivity sharpness.

7. The intelligent fish identification and grading method integrating visual detection and text semantic recognition according to claim 1, characterized in that, The method for obtaining the pupil boundary contrast includes: Based on the brightness distribution of pixels on each ray, the boundary pixels on each ray are obtained, and the brightness difference between the inner and outer sides of the boundary pixels is obtained. The sum of the water droplet tendency coefficients of all pixels in the candidate specular region within the neighborhood of the upper boundary pixel of each ray is obtained as the local interference weight of each ray. Obtain the cumulative value of the product between the local interference weights of all rays and the brightness difference, and calculate the ratio of the cumulative product to the cumulative sum of the local interference weights of all rays as the pupil boundary contrast.

8. The intelligent fish identification and classification method integrating visual detection and text semantic recognition according to claim 7, characterized in that, The process of obtaining the boundary pixels on each ray and the brightness difference between the inner and outer sides of the boundary pixels includes: Obtain the brightness sequence of pixels along each ray direction, obtain the difference between the next element and the previous element in the brightness sequence to form a brightness difference sequence; obtain the element with the largest maximum value in the brightness difference sequence and take the corresponding pixel as the boundary pixel. The difference in the average brightness of each boundary pixel on its inner and outer sides, which is a preset number of pixels, is used as the first difference. The ratio between the first difference and the average brightness of a preset number of pixels outside the boundary pixel is obtained as the brightness difference degree.

9. The intelligent fish identification and classification method integrating visual detection and text semantic recognition according to claim 1, characterized in that, The method for obtaining the dominant factor of visual interference includes: The difference between the pupil boundary contrast and the text description quantization value for each fish sample is obtained as the semantic bias. The slope of the fitted line is obtained by linearly fitting the data points composed of semantic bias and water droplet interference intensity of different fish samples. Obtain the product between the absolute value of the slope of the fitted line and the intensity of the water droplet interference, and normalize it; obtain the product between the sign value of the slope of the fitted line and the normalization result, and use it as the dominant factor of visual interference.

10. The intelligent fish identification and classification method integrating visual detection and text semantic recognition according to claim 1, characterized in that, The method for obtaining the final freshness of the fish sample includes: The product of reflective sharpness and pupil boundary contrast is obtained as visual acuity. The mean of visual clarity and text description quantification is obtained as the initial fusion score; the difference between constant 1 and the dominant factor of visual interference is obtained, the product between the difference and the initial fusion score is calculated, and the minimum value between the product and the preset freshness upper limit is selected as the final freshness.

Images