Automatic fish sorting system based on image classification

Abstract

The present application relates to the field of machine vision and intelligent sorting, in particular to an automatic fish sorting system based on image classification, comprising data acquisition, reference construction, simulation generation, difference extraction, classification decision, sorting execution and feedback update modules; synchronously collecting fish body RGB-D images and dynamic light source parameters, generating a reference fish body model based on a preset category template, injecting light disturbance, morphological deformation and texture disturbance to generate simulation reference data; after registering the measured data and the simulation reference data with the reference RGB-D reference data respectively, differences are extracted to generate measured residual matrix and simulation residual matrix, and based on similarity, fish variety labels and / or fish body defect labels are output, and the fish body is controlled to enter the corresponding sorting channel; according to residual deviation, the disturbance parameter range is updated, so that the simulation space is close to the real disturbance distribution, and the misjudgment and omission in the high-speed production line are reduced.

Description

Technical Field

[0001] This invention relates to the field of machine vision and intelligent sorting technology, specifically to an automatic fish sorting system based on image classification. Background Technology

[0002] In the current high-speed automated sorting environment of aquatic product processing plants, fish targets on the conveyor line will exhibit various species characteristics and appearance abnormalities, and the image acquisition process is easily affected by dynamic lighting, surface water film reflection and changes in the spatial posture of the fish.

[0003] To identify and sort fish images, existing solutions generally use a single color vision sensor to collect features, or directly extract global features from the original target image and perform classification calculations. Although this solution has a certain recognition capability under ideal static imaging conditions, it is highly dependent on a single vision channel and does not accurately align the non-rigid deformation and spatial position of the target. Real surface defects are often mixed with physical interferences such as high gloss reflection, scale drift, misalignment, and bending and twisting during transportation. This makes the image feature matching process easily dominated by positional offset and optical noise, resulting in frequent misjudgments and omissions in high-speed production lines, making it difficult to support the accurate decoupling and judgment of the true category features of the target under complex working conditions.

[0004] Therefore, how to eliminate the interference of dynamic lighting and non-rigid deformation on image feature extraction and improve the accuracy and robustness of image classification and recognition has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide an automatic fish sorting system based on image classification, and to solve the following technical problems:

[0006] To avoid mistaking changes in the imaging environment for changes in the fish itself, and by directly classifying images into residual space comparison, decoupled recognition of dynamic lighting, non-rigid deformation and true category features is achieved, thereby effectively reducing misjudgments and missed judgments in high-speed production lines.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] An image-based automatic fish sorting system includes:

[0009] The data acquisition module is used to acquire RGB-D image data of the fish and dynamic light source parameters corresponding to each frame of the RGB-D image data. The dynamic light source parameters include at least illumination direction parameters and illumination intensity parameters.

[0010] The benchmark construction module is used to generate benchmark fish models based on preset category templates;

[0011] The simulation generation module is used to perform simulation transformation on the reference fish model based on the dynamic light source parameters, preset illumination perturbation parameters, preset morphological deformation parameters and preset texture perturbation parameters, and generate simulation reference data.

[0012] The differential extraction module is used to render the benchmark fish model into benchmark RGB-D reference data corresponding to the acquisition angle and resolution of the RGB-D image data, and to perform differential extraction on the RGB-D image data and the simulation reference data after registering them with the benchmark RGB-D reference data to generate the measured residual matrix and the simulation residual matrix.

[0013] The classification decision module is used to output classification labels based on the similarity between the measured residual matrix and the simulated residual matrix. The classification labels include fish species labels and / or fish body defect labels.

[0014] The sorting execution module is used to generate sorting control instructions based on the classification labels and control the sorting mechanism to transport the fish to the corresponding sorting channel.

[0015] The feedback update module is used to calculate the residual deviation between the measured residual matrix and the simulation residual matrix, update the parameter range of the preset illumination perturbation parameter, the preset morphological deformation parameter, or the preset texture perturbation parameter based on the residual deviation, and send it to the simulation generation module.

[0016] Preferably, the data acquisition module includes a color image acquisition unit, a depth image acquisition unit, and a light source parameter acquisition unit;

[0017] The color image acquisition unit is used to acquire a color image of the fish, and the depth image acquisition unit is used to acquire a depth image of the fish. The RGB-D image data is composed of the color image and the depth image.

[0018] The light source parameter acquisition unit is used to read the illumination direction parameter and the illumination intensity parameter from the light source controller, or to determine the illumination direction parameter and the illumination intensity parameter based on the illumination sensor and calibration parameters.

[0019] Preferably, the baseline construction module includes a template storage unit and a model generation unit;

[0020] The template storage unit is used to store preset category templates, which include fish species templates or fish body defect templates.

[0021] The model generation unit is used to generate a standard pose contour, surface reflection parameters and standard texture distribution based on the preset category template. The reference fish body model is composed of the standard pose contour, the surface reflection parameters and the standard texture distribution.

[0022] Preferably, the surface reflection parameters include diffuse reflection parameters based on the Lambert diffuse reflection model, or specular reflection parameters used to describe the high-gloss reflection of the fish surface.

[0023] Preferably, the simulation generation module includes a lighting simulation unit, a deformation simulation unit, and a texture simulation unit;

[0024] The illumination simulation unit is used to perform specular reflection simulation on the benchmark fish model based on the dynamic light source parameters and the preset illumination disturbance parameters.

[0025] The deformation simulation unit is used to apply bending, torsion, or local deformation to the reference fish model according to the preset morphological deformation parameters.

[0026] The texture simulation unit is used to change the texture color, texture contrast or texture continuity in a preset area according to the preset texture perturbation parameters, so as to simulate surface damage, color difference abnormalities or patch defects of fish.

[0027] Preferably, the differential extraction module includes a reference rendering unit, a registration unit, a measured differential unit, and a simulation differential unit;

[0028] The reference rendering unit is used to render the reference fish model into a reference RGB image and a reference depth image, and the reference RGB-D reference data is composed of the reference RGB image and the reference depth image;

[0029] The registration unit is used to perform scale normalization and spatial registration on the RGB-D image data, the simulation reference data, and the baseline RGB-D reference data;

[0030] The measured difference unit is used to calculate the pixel-level color difference and depth difference between the RGB-D image data and the reference RGB-D data to generate the measured residual matrix;

[0031] The simulation differential unit is used to calculate the pixel-level color difference and depth difference between the simulation reference data and the baseline RGB-D reference data to generate the simulation residual matrix.

[0032] Preferably, both the measured residual matrix and the simulated residual matrix include a color residual submatrix and a depth residual submatrix;

[0033] The color residual submatrix is ​​used to characterize the residuals of the RGB channels, and the depth residual submatrix is ​​used to characterize the residuals of the depth channels.

[0034] Preferably, the classification decision module includes a similarity calculation unit and a result output unit;

[0035] The similarity calculation unit is used to expand the measured residual matrix and the simulated residual matrix into feature vectors and calculate the cosine similarity, and calculate the structural similarity index based on the residual submatrices corresponding to the measured residual matrix and the simulated residual matrix;

[0036] The result output unit is used to output the classification label based on the cosine similarity and the structural similarity index.

[0037] Preferably, the classification decision module is further configured to perform threshold interval determination, wherein the thresholds involved in the threshold interval determination include a first matching threshold and a first exclusion threshold for the cosine similarity, and a second matching threshold and a second exclusion threshold for the structural similarity index, wherein the first matching threshold is greater than the first exclusion threshold, and the second matching threshold is greater than the second exclusion threshold.

[0038] When the cosine similarity is greater than or equal to the first matching threshold, and the structural similarity index is greater than or equal to the second matching threshold, the classification result is confirmed.

[0039] When the cosine similarity is less than or equal to the first exclusion threshold, and the structural similarity index is less than or equal to the second exclusion threshold, the rejection result is output.

[0040] In other cases, the verification results will be output.

[0041] Preferably, the feedback update module is used to record the corresponding measured residual matrix, the simulated residual matrix, the cosine similarity, and the structural similarity index when outputting the confirmed classification result, and to use the sum of the absolute values ​​of the differences between the corresponding elements of the measured residual matrix and the simulated residual matrix as the residual deviation, and to determine the corresponding parameters that cause the deviation based on the spatial distribution characteristics of the residual deviation in the color residual submatrix and the depth residual submatrix;

[0042] Specifically, when the residual deviation is concentrated in the highlight area and the depth deviation is small, the corresponding parameter is determined to be the preset illumination perturbation parameter; when the residual deviation manifests as a continuous local contour shift and the depth deviation accounts for a higher proportion, the corresponding parameter is determined to be the preset morphological deformation parameter; when the residual deviation is concentrated in a fixed surface area and the depth change is not significant, the corresponding parameter is determined to be the preset texture perturbation parameter; the specific feedback update rule is as follows:

[0043] If the residual deviation is greater than or equal to the preset upper limit of deviation, then the parameter range of the corresponding parameter is expanded;

[0044] If the residual deviation is less than or equal to the preset lower limit of deviation, then the parameter range of the corresponding parameter is narrowed.

[0045] In other cases, the center value of the corresponding parameter is adjusted, wherein the upper limit of the preset deviation is greater than the lower limit of the preset deviation, and the corresponding parameter includes at least one of the preset illumination disturbance parameter, the preset shape deformation parameter, and the preset texture disturbance parameter.

[0046] Preferably, the sorting execution module includes a control unit and the sorting mechanism.

[0047] The control unit is used to determine the target sorting channel based on the classification label and generate the sorting control command.

[0048] The sorting mechanism includes a conveying device and at least one of a lever, a jet device, or a robotic arm, and is used to guide fish into the target sorting channel according to the sorting control command.

[0049] The beneficial effects of this invention are:

[0050] 1) This invention decouples multiple imaging perturbations, significantly reducing the false and false judgment rates. This system no longer directly classifies images, but instead constructs a benchmark model and generates simulation reference data by combining illumination, morphology and texture parameters, and completes the matching decision in the residual space. This effectively decouples perturbations such as dynamic illumination, highlights and fish body bending from real defects, solves the problem of mixed environmental changes and body differences, and improves sorting reliability.

[0051] 2) This invention provides synchronous optical condition observation, which enhances the ability to resist light fluctuations. The system synchronously acquires color images, depth images and dynamic light source parameters, so that the acquired data is completely correlated with real-time optical conditions. Combined with the benchmark constructed by surface diffuse reflection and specular reflection, it can accurately separate and predict surface reflection interference, avoid mistaking sudden changes in ambient light as abnormalities on the fish surface, and effectively improve robustness under complex lighting conditions.

[0052] 3) This invention eliminates spatial pose differences and enhances dynamic recognition stability. After scaling and spatial registration of measured and simulated data, color and depth residuals are extracted separately, effectively removing the interference of position offset and size difference. The independent expression mechanism of residuals in separate channels realizes flexible adjustment of weights under multiple working conditions, overcoming the feature extraction failure problem caused by continuous posture fluctuations on high-speed conveyor belts.

[0053] 4) The present invention uses hierarchical judgment and parameter adaptation to overcome the dependence on static samples and introduce a threshold interval judgment mechanism to realize hierarchical output of confirmation, rejection and verification, and establish a reliable fault-tolerant branch for samples with ambiguous boundaries; at the same time, the residual deviation is calculated through the feedback update mechanism to adaptively correct the simulation parameter range, so that the model can stably track the drift of the actual working conditions of the production line and reduce the dependence on environmental stability and sample completeness. Attached Figure Description

[0054] The invention will now be further described with reference to the accompanying drawings.

[0055] Figure 1 This is a schematic diagram of the modules of the automatic fish sorting system based on image classification provided in the embodiments of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Please see Figure 1 An image-based automatic fish sorting system includes:

[0058] The data acquisition module is used to acquire RGB-D image data of the fish and dynamic light source parameters corresponding to each frame of RGB-D image data. The dynamic light source parameters include at least the illumination direction parameter and the illumination intensity parameter.

[0059] The benchmark construction module is used to generate benchmark fish models based on preset category templates;

[0060] The simulation generation module is used to simulate and transform the benchmark fish model based on dynamic light source parameters, preset lighting perturbation parameters, preset morphological deformation parameters, and preset texture perturbation parameters, and generate simulation reference data.

[0061] The differential extraction module is used to render the benchmark fish model into benchmark RGB-D reference data corresponding to the acquisition viewpoint and resolution of the RGB-D image data, and to perform differential extraction on the RGB-D image data and simulation reference data after registering them with the benchmark RGB-D reference data to generate the measured residual matrix and the simulation residual matrix.

[0062] The classification decision module is used to output classification labels based on the similarity between the measured residual matrix and the simulated residual matrix. The classification labels include fish species labels and / or fish body defect labels.

[0063] The sorting execution module is used to generate sorting control instructions based on the classification labels and control the sorting mechanism to transport the fish to the corresponding sorting channel.

[0064] The feedback update module is used to calculate the residual deviation between the measured residual matrix and the simulation residual matrix, update the parameter range of the preset illumination perturbation parameters, preset morphological deformation parameters or preset texture perturbation parameters based on the residual deviation, and send them to the simulation generation module.

[0065] This embodiment provides an automatic fish sorting mechanism based on image classification, which is deployed in a high-speed sorting production line of an aquatic product processing plant. The front section of the production line is the fish conveying section, the middle section is the imaging recognition section, and the rear section is the pneumatic sorting section and the verification and return section. The fish to be sorted can be different species such as perch, yellow croaker, and crucian carp, or they can be further distinguished within the same species by appearance defects such as abrasions, lesions, and localized descaling.

[0066] The system does not directly output the final classification result based on the original fish image. Instead, it first establishes an ideal benchmark, then generates possible perturbation patterns based on parameters, and completes matching and decision-making in the residual space.

[0067] The data acquisition module simultaneously acquires color and depth images as the fish passes through the imaging area and records the dynamic light source parameters at that moment. The dynamic light source parameters include at least the illumination direction and illumination intensity, and preferably also carry the light source number, trigger time, and exposure compensation value. The benchmark construction module establishes a benchmark fish model for each target category. The model includes at least the standard contour, surface reflection properties, and standard texture distribution.

[0068] The simulation generation module injects lighting perturbation, morphological deformation and texture perturbation into the benchmark fish model. The lighting perturbation is used to simulate water stains and scale highlights, the morphological deformation is used to simulate bending, twisting and local tail lifting during the transportation process, and the texture perturbation is used to simulate surface changes such as lesions, abrasions and abnormal color difference.

[0069] The differential extraction module renders the benchmark fish model to the same viewpoint and resolution as the actual acquisition, and obtains the benchmark RGB-D reference data. Then, the RGB-D image data, simulation reference data and benchmark RGB-D reference data are registered and differentially analyzed to obtain the measured residual matrix representing the deviation from reality and the simulation residual matrix representing the deviation from theoretical perturbation.

[0070] The classification decision module compares the similarity between two types of residuals. If a certain simulated residual is highly consistent with the measured residual, it outputs the corresponding fish species label or fish body defect label. The sorting execution module generates channel control instructions based on the classification label and sends the fish to the qualified product, substandard product, disease verification product, and other channels.

[0071] The feedback update module continuously calculates the residual deviation between the measured residual matrix and the simulation residual matrix, and corrects the range of subsequent simulation parameters accordingly, so that the simulation space gradually approaches the actual disturbance distribution of the production line.

[0072] For example, the acquired fish images can be reduced to... Color difference matrix and The local extrema in the depth difference matrix and color difference matrix that exceed the first preset deviation threshold can represent pixel overexposure caused by highlights, and the local extrema in the depth difference matrix that exceed the second preset deviation threshold can represent depth deviation caused by the fishtail lifting.

[0073] The system can generate simulation references such as normal fish with highlights within a preset first intensity range and bending with deformation less than a preset threshold, and diseased fish with highlights within a preset first intensity range and bending with deformation less than a preset threshold, and compare their closeness to the measured residuals in the color and depth channels, thereby outputting normal labels or diseased labels.

[0074] Regarding handling abnormal working conditions, if the acquired frame cannot be stably registered due to severe occlusion by the fish, camera defocus, or missing depth map, the frame is marked as an invalid frame and will not be included in the classification decision. Instead, it will trigger a reshoot or be sent to the manual review channel. If multiple frames of data are acquired continuously for the same fish, the classification result of a single fish can be output by majority voting or weighted average.

[0075] If the classification results of multiple frames are inconsistent, the fish will enter the review channel as a conflicting sample, and the confidence record of each candidate category will be retained for subsequent manual verification or threshold backtracking, instead of being forcibly sorted according to a certain low confidence category. If the parameter range expands abnormally after the feedback update and exceeds the safety limit, the update will be frozen and rolled back to the previous round of valid parameter set.

[0076] The purpose of this step is to transform the direct classification of images into residual space comparison, thereby achieving decoupled recognition of dynamic lighting, non-rigid deformation and true category features, and reducing misjudgments and missed judgments in high-speed production lines.

[0077] In a preferred embodiment of the present invention, the data acquisition module includes a color image acquisition unit, a depth image acquisition unit, and a light source parameter acquisition unit; the color image acquisition unit is used to acquire color images of the fish, the depth image acquisition unit is used to acquire depth images of the fish, and the RGB-D image data is composed of color images and depth images; the light source parameter acquisition unit is used to read illumination direction parameters and illumination intensity parameters from the light source controller, or to determine illumination direction parameters and illumination intensity parameters based on the illumination sensor and calibration parameters.

[0078] This embodiment provides a data acquisition mechanism. In the sorting production line, relying solely on a single color image can easily lead to problems such as reflective glare obscuring texture, unstable scale, and inaccurate judgment of fish posture. Therefore, a color image acquisition unit, a depth image acquisition unit, and a light source parameter acquisition unit are set in the imaging area to form a multi-source acquisition structure that is synchronized with the actual production line.

[0079] The color image acquisition unit can use an industrial area scan camera and be arranged above the conveyor belt; the depth image acquisition unit can use a structured light camera, a binocular depth camera or a time-of-flight camera, and complete the external parameter calibration with the color camera; the light source parameter acquisition unit communicates with the light source controller and directly reads the direction, intensity and trigger time of each flash.

[0080] If multiple fixed light sources are used on site, the reflection intensity can be detected by the light sensor around the imaging area, and the current light direction and intensity can be deduced by combining the pre-stored light source geometry calibration table. Thus, each frame of fish image is no longer just pixel data, but together with the optical conditions corresponding to that frame, it forms a complete observation.

[0081] For example, the same fish at the first collection time The average brightness of the fish's back was 180, and the depth distance was 420 mm. The corresponding direction of the main light source can be represented as a vector. The intensity was 80; at the second acquisition time Due to the slight turning of the fish's body, the average brightness of the fish's back increased to 220 and the depth distance changed to 415 mm. At the same time, the auxiliary light source was turned on and the total intensity became 110.

[0082] The system will collect data at the first moment. Second acquisition time Instead of simply attributing the brightness change to differences in fish texture, the data was processed as two frames of data under different observation conditions.

[0083] In terms of handling abnormal operating conditions, if the color image exists but the depth map is lost, the single-frame degradation processing mode is entered, and only the color residual is used for prediction and the confidence level is reduced; if the depth map is available but the color image is severely overexposed, the depth channel is retained and the color channel is blocked; if the light source controller communication is interrupted, the parameters are back-derived from the light sensor first.

[0084] If neither is available, the illumination parameters of the most recent consecutive valid frames will be used for short-term interpolation. If the interpolation duration exceeds the set upper limit, an alarm will be automatically triggered and the batch of fish will be imported into the review channel.

[0085] The purpose of this step is to establish a basis for synchronous observation of image data and optical conditions, so that subsequent simulations and actual measurements can correspond one-to-one, avoiding mistaking changes in the imaging environment as changes in the fish itself.

[0086] In a preferred embodiment of the present invention, the benchmark construction module includes a template storage unit and a model generation unit; the template storage unit is used to store preset category templates, which include fish species templates or fish body defect templates; the model generation unit is used to generate standard posture contours, surface reflection parameters and standard texture distributions based on the preset category templates, and the benchmark fish body model is composed of standard posture contours, surface reflection parameters and standard texture distributions.

[0087] This embodiment provides a benchmark construction mechanism; without a unified, stable and reusable ideal benchmark, simulations of different fish species and different defects will lack a common reference, leading to residual definition drift; therefore, the system sets up a template storage unit and a model generation unit to transform expert experience into a standardized benchmark model;

[0088] The preset category templates are generated from a large number of fish body sample images collected in advance and labeled by aquatic experts, through statistical averaging or feature clustering extraction algorithms; the template storage unit stores fish variety templates and fish body defect templates; the fish variety template is used to describe the standard body length-to-height ratio, fin position layout, head-to-tail transition, and typical texture zones of a certain variety under normal conditions; the fish body defect template is used to describe the spatial distribution of abnormalities such as lesions, abrasions, missing scales, and local discoloration;

[0089] After reading the corresponding template, the model generation unit first generates a standard posture profile under conditions of no bending and no twisting, then configures the surface reflection parameters to describe the response of the fish skin and scales to light, and superimposes the standard texture distribution to form a complete reference fish model. This model can be a combination of 3D mesh and surface mapping, or a combination of point cloud skeleton and texture mapping, as long as it can be called by subsequent rendering and simulation.

[0090] For example, the template library contains templates for normal yellow croaker and templates for diseased yellow croaker; the normal template can simplify the fish body outline to a standard shape with a length of 100 and a maximum height of 28, and the texture distribution is that the back of the fish is darker and the belly is lighter; the diseased template defines an abnormal area located on the back of the fish in addition to the standard shape.

[0091] The model generation unit outputs two baseline models accordingly. The first model does not contain abnormal regions, while the second model adds abnormal texture regions on the same contour and basic reflection properties. In subsequent simulations, whether specular highlights or curvature are added, both models start from their respective baseline models, thus ensuring the consistency of category interpretation.

[0092] In terms of handling abnormal operating conditions, if a certain sub-category template is missing in the template storage unit, the system can go back to the previous level category template to complete the coarse classification, and then add the sample to the template library after manual review and confirmation.

[0093] If there are conflicts between templates, such as two standard profiles of different varieties overlapping too much at the same scale, the model generation unit can be subject to constraint rules to prioritize the preservation of more stable skeletal proportion differences and head geometric features in order to reduce template confusion.

[0094] The purpose of this step is to provide a unified ideal reference for different varieties and different defects, so as to achieve the standardization of subsequent residual definitions and the traceability of the simulation process;

[0095] In a preferred embodiment of the present invention, the surface reflection parameters include diffuse reflection parameters based on the Lambertian diffuse reflection model, or specular reflection parameters used to describe the high-gloss reflection of the fish surface.

[0096] This embodiment provides a surface reflection modeling mechanism. If the baseline model only includes geometric contours and texture distribution without distinguishing between diffuse and specular reflection, the true imaging characteristics of the fish skin surface cannot be fully expressed. On the surface of a wet fish, water film and scales will produce obvious highlights. Using only a single brightness parameter can easily misclassify the highlights as texture anomalies. Therefore, this embodiment further introduces diffuse reflection parameters and specular reflection parameters.

[0097] Diffuse reflection parameters are used to describe the basic reflectivity of the fish body in all directions, and can be set separately for areas such as the back, belly, and fins; specular reflection parameters are used to describe the intensity, range, and concentration of bright spots appearing in a specific area under specific incident and observation directions.

[0098] When rendering the baseline model and generating the simulation reference, the system calls both types of reflection parameters simultaneously, so that the same piece of fish skin presents different brightness distributions under low light and strong side light, and regards this difference as an interpretable imaging result, rather than uncontrollable noise.

[0099] For example, a certain yellow croaker model is simplified into a dorsal region and a ventral region. The diffuse reflectance coefficient of the dorsal region is set to 0.5 and the specular reflectance coefficient is set to 0.7, while the diffuse reflectance coefficient of the ventral region is set to 0.8 and the specular reflectance coefficient is set to 0.2. When the light source shines from the upper left and the viewing direction is close to the specular reflection direction, the brightness of the dorsal region may increase from the base value of 120 to 210, while the brightness of the ventral region may only increase from 140 to 160.

[0100] If the composition of reflection is not recorded, it is easy to mistake the sudden increase in brightness in the fish back area for abnormal texture; after introducing two types of reflection parameters, the rendering module can predict the highlight phenomenon and retain the corresponding features in the simulation residual.

[0101] Regarding handling abnormal working conditions, if a certain variety lacks fine mirror parameters, the default values ​​for the region can be used first, such as high on the back, medium on the abdomen, and low on the fins. After enough samples are accumulated on site, the system can be gradually corrected through feedback updates. If the mirror parameters are set too high, resulting in large-area saturation in the rendering results, the system will limit the upper limit of brightness and mark the overexposed areas as low-confidence areas to avoid them occupying too much weight in similarity calculations.

[0102] The purpose of this step is to express the imaging mechanism of the fish surface in a more granular way, so as to separate the difference between highlights and real texture.

[0103] In a preferred embodiment of the present invention, the simulation generation module includes a lighting simulation unit, a deformation simulation unit, and a texture simulation unit; the lighting simulation unit is used to perform specular reflection simulation on the reference fish model according to dynamic light source parameters and preset lighting disturbance parameters; the deformation simulation unit is used to apply bending, torsion, or local deformation to the reference fish model according to preset morphological deformation parameters.

[0104] The texture simulation unit is used to change the texture color, texture contrast or texture continuity in a preset area according to preset texture perturbation parameters in order to simulate surface damage, color difference abnormalities or patch defects of fish.

[0105] This embodiment provides a simulation generation mechanism; if only a single benchmark is used to subtract from the measured image, all kinds of real disturbances will be mixed in the same residual, making it difficult to distinguish between the highlights and bends that may occur and the real defects; therefore, this embodiment subdivides the simulation generation module into a lighting simulation unit, a deformation simulation unit and a texture simulation unit, which are used to actively construct theoretical deviation paths.

[0106] The illumination simulation unit receives real-time light source direction, intensity and disturbance range, and generates highlight distributions of different intensities and positions on the reference model; the deformation simulation unit receives parameters such as bending angle, twisting angle, local head-up or tail-up amplitude, and performs geometric transformations on the fish skeleton or mesh.

[0107] The texture simulation unit changes the color, contrast and continuity within a specified area according to preset texture perturbation parameters to simulate scaling, scratches or lesions; the three can be superimposed in series or generated in a predetermined combination to generate multiple sets of simulation references, so that the system obtains a theoretical sample family that is extended within a reasonable physical range.

[0108] For example, the ideal fish body back texture value is originally 100. The lighting simulation unit adds a highlight on the right side of the back to increase the local brightness to 160. The deformation simulation unit applies an upturn to the tail to increase the tail depth value by 2 units. The texture simulation unit adds a color difference area in the middle of the back to reduce the brightness to 70.

[0109] The system can obtain simulation references such as highlights only, highlights plus tail upturn, highlights plus tail upturn and then superimposed blemishes; when the measured residual is closer to the third type, it indicates that the deviation is not caused by a single highlight or attitude disturbance, but by a physical defect.

[0110] In terms of handling abnormal working conditions, if the number of simulation combinations is too large and affects real-time performance, impossible combinations can be filtered out based on fish species, size and current lighting conditions. For example, small short-tailed fish will not generate large twists, and extremely strong lateral highlights will not be generated under weak top lighting conditions. If the texture simulation area and the highlight area completely overlap, making it difficult to distinguish between the two, the system can set spatial priority for texture simulation and prioritize the retention of more stable defect features such as structural continuity disruption.

[0111] The purpose of this step is to parametrically incorporate optical perturbations, attitude perturbations, and texture anomalies that frequently coexist in reality into the theoretical space, thereby achieving interpretive matching for complex operating conditions.

[0112] In a preferred embodiment of the present invention, the differential extraction module includes a reference rendering unit, a registration unit, a measured differential unit, and a simulation differential unit; the reference rendering unit is used to render the reference fish body model into a reference RGB image and a reference depth image, and the reference RGB-D reference data is composed of the reference RGB image and the reference depth image;

[0113] The registration unit is used to perform scale normalization and spatial registration on RGB-D image data, simulation reference data and reference RGB-D reference data; the measured difference unit is used to calculate the pixel-level color difference and depth difference between RGB-D image data and reference RGB-D reference data to generate the measured residual matrix.

[0114] The simulation differential unit is used to calculate the pixel-level color and depth differences between the simulation reference data and the baseline RGB-D reference data to generate the simulation residual matrix.

[0115] This embodiment provides a differential extraction mechanism. Even if there are existing benchmark models and simulation models, without unified rendering and strict registration, the actual fish's position offset, orientation change, and size difference on the conveyor belt will still dominate the subsequent comparison results, causing misalignment to be mistakenly treated as inconsistent categories. Therefore, this embodiment extracts truly discriminative residuals through reference rendering, registration, and dual-path differential.

[0116] The reference rendering unit renders the baseline fish model into a baseline RGB image and a baseline depth image that are consistent with the parameters of the real-time camera; the registration unit performs scale normalization and spatial registration on the RGB-D image data, simulation reference data, and baseline RGB-D reference data in a unified manner.

[0117] Scale normalization can be performed based on the length of the main axis of the fish body, the distance between the head and tail ends, or the size of the bounding box; spatial registration can be rigidly registered based on key points such as the head, caudal peduncle, and dorsal fin origin, or elastic fine-tuning can be introduced in local areas; after registration is completed, the measured differential unit calculates the difference between the RGB-D image data and the reference RGB-D data pixel by pixel to generate the measured residual matrix.

[0118] The simulation differential unit calculates the difference between the simulation reference data and the baseline RGB-D reference data pixel by pixel to generate the simulation residual matrix. Since both are defined relative to the same baseline RGB-D reference data, their meanings remain consistent, so they can be directly compared.

[0119] To avoid the registration process from eliminating the real defects or real deformations that should be retained, elastic fine-tuning is preferred only as a restricted correction step after rigid registration, rather than as unlimited free deformation.

[0120] Specifically, the system can first complete global rigid registration with the head center, tail shank center and main axis direction, and then perform elastic correction within the constraint range only in locations that are easily affected by acquisition noise, such as the fin edge, tail end thin plate area or deep cavity boundary; the small elastic correction is based on thin plate spline interpolation or free deformation algorithm, and by establishing a set of control points in the target area, the local mesh or pixel set is subject to restricted deformation constraints.

[0121] For areas with high defect incidence, areas with abrupt changes in texture continuity, and areas with abrupt changes in depth, the original observations are maintained first, and the reference is not forcibly fitted by elastic deformation; this ensures that the subsequent difference still reflects the differences in the fish body itself, rather than the artificial compensation result of the registration algorithm.

[0122] Furthermore, the registration unit can set upper limits for displacement and region masking rules for elastic fine-tuning; for example, in the two-dimensional image plane, the elastic displacement of a single point does not exceed a preset proportion of the width of the local fish body, or does not exceed a preset pixel threshold; in the depth direction, only the obvious hole patching area is allowed to be continuously corrected, without rewriting the abnormal protrusions in the complete effective depth area;

[0123] If a local area requires a large displacement to complete the fitting, the system will treat it as a suspected real anomaly area and retain it in the residual matrix, instead of continuing to expand the registration degrees of freedom. This restriction can prevent tail-up tilting, local pressure marks, or lesion boundaries from being mistakenly eliminated as ordinary attitude offsets.

[0124] To ensure consistency, scale normalization and spatial registration preferably use the same set of key point definitions, coordinate references, and masking rules for RGB-D image data, simulation reference data, and baseline RGB-D reference data, so as to avoid introducing additional residuals due to the use of different registration references for the three data paths;

[0125] For example, in a reference RGB image, a pixel has a value of 120 and a depth value of 50. After registration in the actual image, the pixel value is 150 and the depth value is 53. Then the measured residual at that location is the color difference. Depth difference If a set of simulation reference data, after rendering at the same location, has a pixel value of 148 and a depth value of 52, then the simulation residual at that location is the color difference. Depth difference The two are very close at this position; after the system integrates the features of multiple positions, it can calculate and determine the simulation residual that has the highest overall matching degree with the measured residual.

[0126] In terms of handling abnormal working conditions, if key points cannot be extracted stably during the registration process, such as when the fish head is obscured or the tail is outside the field of view, it can be degraded to coarse registration based on the bounding box center and the main axis direction, and the decision confidence level can be reduced.

[0127] If the registration error exceeds the preset limit, such as a head alignment deviation exceeding 5 pixels or an average depth deviation exceeding 2 millimeters, the frame will not be included in the similarity calculation, but will request compensation from adjacent frames; if there are many holes in the depth map, a mask can be set for the hole area, and the area will be skipped in the difference calculation.

[0128] The purpose of this step is to eliminate the interference of location, scale, and perspective differences on classification and achieve a consistent representation of the residual space.

[0129] In a preferred embodiment of the present invention, both the measured residual matrix and the simulated residual matrix include a color residual submatrix and a depth residual submatrix; the color residual submatrix is ​​used to characterize the residuals of the RGB channels, and the depth residual submatrix is ​​used to characterize the residuals of the depth channels.

[0130] This embodiment provides a residual channel expression mechanism; although a single mixed residual matrix can also express deviations, color changes and geometric changes have different causes in the production line: highlights and blemishes often manifest as color changes, while bending and dents often manifest as depth changes;

[0131] If the two are simply superimposed, one change may mask the other; therefore, in this embodiment, the residual is explicitly decomposed into a color residual submatrix and a depth residual submatrix.

[0132] The color residual submatrix can be composed of pixel differences in the R, G, and B channels, or it can be further converted to a more stable color space and the difference values ​​can be preserved.

[0133] The depth residual submatrix stores the offset of each pixel relative to the reference in the depth direction; when calculating similarity in the subsequent process, the system can assign weights to the two types of submatrices separately, or it can calculate the similarity separately and then make a fusion decision.

[0134] This multi-channel structure is beneficial for adjusting the recognition focus under different working conditions. For example, in a strong reflective environment, the weight of the color channel can be appropriately reduced, while in a scene where the fish body bends frequently and the texture is stable, the weight of the depth channel can be increased.

[0135] For example, if the color residual submatrix of a certain fish increases significantly locally while the depth residual is small, it indicates that there are local bright areas in the color channel while the geometric shape only undergoes minor deformation; if another fish has a small color residual but a continuously increasing depth residual, it indicates that the color is basically normal but the shape distortion is more obvious.

[0136] The system can allocate weights accordingly, so that the former focuses on the simulated reference of highlights or blemishes in similarity calculation, while the latter focuses on the simulated reference of bending or crushing. If they are mixed into a single matrix, this discriminative guidance will be weakened.

[0137] In terms of handling abnormal working conditions, if the color channel is severely contaminated, such as water droplets obscuring the lens and causing abnormal fluctuations in color information, the weight of the color residual in that frame can be automatically reduced; if there are large holes in the depth channel, the color residual will participate normally, and the depth residual will be skipped by the mask.

[0138] If there is a conflict in the similarity calculation results based on the two sub-matrices, for example, the color residual feature is closer to the normal fish while the depth residual feature is closer to the damaged fish, the system outputs the verification result or enters the multi-frame fusion judgment, instead of directly sorting according to a single high-confidence category. The purpose of this step is to model color changes and geometric changes in layers to achieve more robust difference analysis and more flexible similarity fusion.

[0139] In a preferred embodiment of the present invention, the classification decision module includes a similarity calculation unit and a result output unit; the similarity calculation unit is used to calculate the cosine similarity after expanding the measured residual matrix and the simulated residual matrix into feature vectors, and to calculate the structural similarity index based on the residual submatrices corresponding to the measured residual matrix and the simulated residual matrix; the result output unit is used to output classification labels based on the cosine similarity and the structural similarity index.

[0140] This embodiment provides a classification decision mechanism. If only the sum of pixel-by-pixel errors is considered, the overall direction and spatial structure of the residuals can be easily overlooked. For example, the total difference between two residual submatrices may be close, but one is concentrated at the back and the other is scattered at the tail. Therefore, this embodiment introduces both cosine similarity and structural similarity index to take into account both numerical trend consistency and spatial structure consistency.

[0141] The similarity calculation unit expands the color residual and depth residual into feature vectors in a predetermined order, for example, first expanding the color channels by row, and then adding the depth channels;

[0142] The cosine similarity between the measured vector and each simulated vector is calculated to reflect whether their directions are consistent. At the same time, the structural similarity index is calculated for the measured residual submatrix and the simulated residual submatrix by channel. The structural similarity index focuses more on the brightness, contrast and structural distribution relationship between local blocks.

[0143] Specifically, the structural similarity index is obtained by calculating the mean, variance, and covariance of the local sliding windows of the measured residual submatrix and the simulated residual submatrix, respectively, and then weighting them together using the brightness contrast function, contrast contrast function, and structural contrast function.

[0144] The result output unit integrates two types of indicators and selects the category that best conforms to the preset rules as the output label. To avoid ambiguity in the integration process, the result output unit preferably performs the following steps: first, remove candidate categories whose cosine similarity or structural similarity index is lower than the basic lower limit.

[0145] For the remaining candidate categories, compare the ranking results of the two indicators respectively; when the same candidate category ranks best in both indicators, directly output the category label corresponding to the candidate category;

[0146] When the optimal categories of the two indicators are inconsistent, the candidate categories with higher structural similarity indices and cosine similarity not lower than the preset lower limit are retained first to enter the subsequent threshold determination or review process; through this order, the classification output can have a definite data flow rule, rather than just giving results based on a general comprehensive comparison.

[0147] To avoid the color residual and depth residual having different dimensions, causing a certain channel to occupy too much weight after vector expansion, the similarity calculation unit preferably performs intra-channel normalization or range compression on each residual submatrix before expansion.

[0148] Among them, the channel normalization adopts the maximum and minimum value normalization method to linearly map the residual value to the interval [0,1] or [-1,1]; the range compression processing adopts the logarithmic function or the Sigmoid function to nonlinearly suppress abnormal residual values ​​that exceed the preset normal deviation interval.

[0149] Color residuals can be scaled to a uniform range according to the maximum absolute difference among the valid pixels in the current frame, the quantile threshold, or the preset brightness upper limit.

[0150] The depth residual can be scaled according to the current fish thickness range, depth camera calibration range, or the maximum absolute offset within the effective area. After scaling, the color vector and depth vector are then spliced ​​together according to preset weights to ensure that the cosine similarity reflects the consistency of the multi-channel residual direction, rather than the similarity result being unreasonably dominated by the channel with larger numerical dimensions.

[0151] Furthermore, the structural similarity index can also be calculated separately for the color residual submatrix and the depth residual submatrix and then fused, instead of directly mixing the two types of residuals into one image for unified calculation; preferably, the system first obtains the color structural similarity index and the depth structural similarity index, and then assigns weights to the two based on the imaging quality: when the color image is overexposed or contaminated by water droplets, the weight of the color structural similarity index is reduced.

[0152] When the depth map has many holes or obvious edge jitter, the weight of the depth structure similarity index is reduced. This maintains the stability of the structure comparison and is consistent with the aforementioned residual channel expression mechanism, avoiding the drag on the overall decision result by a single low-quality channel.

[0153] For example, the measured residual vector of a certain fish is The simulated vector A is The simulated vector B is If the measured vector direction is closer to that of the simulated vector, the cosine similarity is higher.

[0154] If both the actual test and simulation A show continuous abnormal blocks on the back of the fish in the residual submatrix structure, while the abnormality of simulation B is mainly located in the belly of the fish, then the structural similarity index of simulation A will be higher, and the system outputs the classification label corresponding to simulation A.

[0155] If a certain category has a high cosine similarity but a low structural similarity index, it indicates that the overall numerical trends are similar but the spatial locations are inconsistent, and it is not appropriate to confirm directly.

[0156] In terms of anomaly handling, if all expanded vectors are zero, that is, the measured residuals are almost non-existent, the cosine calculation will result in a denominator that is too small. In this case, the system can directly regard the frame as a near-baseline sample and prioritize comparison with the normal category. If the structural similarity index of the normal category is also high, the normal label will be output.

[0157] If both indicators for all candidate categories are below the baseline, the output should be "reject" or "review," rather than "forced classification." If the scores for multiple categories are close, multi-frame fusion, historical batch distribution, or detection results from adjacent workstations can be used for auxiliary judgment.

[0158] The purpose of this step is to utilize both global trends and local structure information simultaneously to achieve more reliable classification decisions.

[0159] In a preferred embodiment of the present invention, the classification decision module is further configured to perform threshold interval determination, wherein the thresholds involved in the threshold interval determination include a first matching threshold and a first exclusion threshold for cosine similarity, and a second matching threshold and a second exclusion threshold for structural similarity index, wherein the first matching threshold is greater than the first exclusion threshold, and the second matching threshold is greater than the second exclusion threshold.

[0160] When the cosine similarity is greater than or equal to the first matching threshold and the structural similarity index is greater than or equal to the second matching threshold, the confirmed classification result is output; when the cosine similarity is less than or equal to the first exclusion threshold and the structural similarity index is less than or equal to the second exclusion threshold, the rejection result is output; otherwise, the verification result is output.

[0161] This embodiment provides an interval threshold determination mechanism; simply using the highest similarity classification in production line boundary samples is prone to misclassification. For example, when a fish has both minor scratches and strong highlights, the scores of different categories may not be low.

[0162] If the highest score label is forcibly output, it is easy to mistakenly send ambiguous samples into the qualified channel; therefore, this embodiment sets matching thresholds and exclusion thresholds to clearly divide the results into three categories: confirmation, rejection, and review.

[0163] The system sets a first matching threshold and a first exclusion threshold for cosine similarity, and sets a second matching threshold and a second exclusion threshold for structural similarity, with the matching threshold being higher than the exclusion threshold.

[0164] During runtime, if a candidate category simultaneously satisfies that the cosine similarity is not lower than the first matching threshold and the structural similarity is not lower than the second matching threshold, then the confirmed classification result is output.

[0165] If all candidate categories fall within the range where the cosine is not higher than the first exclusion threshold and the structure is not higher than the second exclusion threshold, then the rejection result is output, indicating that the current sample does not belong to the known template range; for other cases in the middle range, the verification result is output to reduce the risk of misclassification.

[0166] For example, the first matching threshold is set to 0.90, the first exclusion threshold is set to 0.60, the second matching threshold is set to 0.88, and the second exclusion threshold is set to 0.55; if a fish has a cosine similarity of 0.93 and a structural similarity of 0.90 to other normal yellow croaker fish, then the confirmation condition is met, and normal yellow croaker fish is directly output.

[0167] If the cosine similarity of another fish to all categories is below 0.55 and the structural similarity is below 0.50, then the rejection output is output; if a fish has a cosine similarity of 0.91 to a slightly abraded bass but a structural similarity of only 0.72, then it is more prudent to proceed to the review.

[0168] In terms of handling abnormal operating conditions, if multiple categories meet the confirmation conditions at the same time, the one with the highest comprehensive score can be selected first, or the output order can be determined according to the rule of defects taking precedence over normal and diseases taking precedence over general flaws. If the manual review channel is temporarily congested, the review results can be re-scheduled, with high-risk categories being retained for review first, and the rest being imported into the pending cache channel.

[0169] If the threshold consistently leads to an excessively high review rate, the threshold can be adjusted appropriately based on feedback updates and historical statistics. However, version records should be retained after adjustment to avoid inconsistencies between standards in previous and subsequent batches. The purpose of this step is to establish clear handling branches for boundary samples, achieve hierarchical output of confirmation, rejection, and review, and reduce the cost of misclassification.

[0170] In a preferred embodiment of the present invention, the feedback update module is used to record the corresponding measured residual matrix, simulated residual matrix, cosine similarity and structural similarity index when outputting the confirmed classification result, and to take the sum of the absolute values ​​of the differences between the corresponding elements of the measured residual matrix and the simulated residual matrix as the residual deviation, and to determine the corresponding parameters that cause the deviation based on the spatial distribution characteristics of the residual deviation in the color residual submatrix and the depth residual submatrix.

[0171] Specifically, when the residual deviation is concentrated in the highlight area and the depth deviation is small, the corresponding parameter is determined to be the preset lighting perturbation parameter; when the residual deviation manifests as a continuous local contour shift and the depth deviation accounts for a higher proportion, the corresponding parameter is determined to be the preset morphological deformation parameter; when the residual deviation is concentrated in a fixed surface area and the depth change is not significant, the corresponding parameter is determined to be the preset texture perturbation parameter; the specific feedback update rules are as follows:

[0172] If the residual deviation is greater than or equal to the preset upper limit of deviation, the parameter range of the corresponding parameter will be expanded.

[0173] If the residual deviation is less than or equal to the preset lower limit of deviation, the parameter range of the corresponding parameter is shrunk; otherwise, the center value of the corresponding parameter is adjusted. The preset upper limit of deviation is greater than the preset lower limit of deviation, and the corresponding parameter includes at least one of the preset illumination disturbance parameter, preset morphological deformation parameter, and preset texture disturbance parameter.

[0174] This embodiment provides a feedback update mechanism; the simulation parameters can be set by expert experience at the beginning, but the production line environment is not static for a long time. For example, seasonal changes will change the thickness of the water film on the fish after cleaning, and equipment wear will change the vibration amplitude of the conveyor.

[0175] If the simulation parameters are not updated, the theoretical samples will gradually deviate from the actual samples; therefore, this embodiment uses high-confidence samples to adaptively correct the range of simulation parameters after outputting the confirmed classification results.

[0176] When the feedback update module outputs a confirmation result, it records the measured residual matrix, the best-matched simulated residual matrix, the corresponding cosine similarity, and the structural similarity index of the sample; it sums the absolute values ​​of the differences between corresponding elements of the two types of residual matrices to obtain the residual bias.

[0177] If the residual deviation is higher than the preset upper limit of deviation, it indicates that the current simulation parameter setting range has not fully covered the actual physical disturbance. In this case, the corresponding parameter range should be expanded. If the residual deviation is lower than the preset lower limit of deviation, it indicates that the simulation parameter setting range is too large and introduces redundant non-physical artifacts. In this case, the parameter range should be narrowed to improve the discrimination sharpness.

[0178] If the residual deviation is between the two, keep the parameter range width basically unchanged, and instead fine-tune the center value to make the simulation distribution closer to the current mainstream state of the production line.

[0179] To make the process of determining the corresponding parameters executable, the feedback update module prefers to attribute the source of the residual deviation rather than arbitrarily choosing among the three types of parameters: illumination, deformation, and texture.

[0180] Specifically, the deviation distribution can be statistically analyzed according to the color residual submatrix and the depth residual submatrix, and then the main source can be determined by combining the spatial region where the deviation is located: if the deviation is mainly concentrated in the bright area and the depth deviation is small, it is primarily attributed to the preset lighting disturbance parameter; if the deviation is mainly manifested as continuous shift of the tail, the main axis of the fish body or the local contour, and the depth deviation accounts for a higher proportion, it is primarily attributed to the preset morphological deformation parameter.

[0181] If the deviation is concentrated in a fixed surface area and manifests as a disruption of color continuity, abnormal local contrast, and no significant change in depth, it is primarily attributed to the preset texture perturbation parameters. This attribution rule ensures that parameter updates are consistent with the causes of residuals, avoiding situations where a change in highlights actually expands the range of texture defects.

[0182] Furthermore, when the same confirmed sample presents multiple sources of deviation, the system can adopt a primary and secondary update strategy; that is, first identify the source of deviation that contributes the most as the primary update parameter, and then perform a smaller-scale linkage adjustment on the secondary sources.

[0183] If the contributions from the three sources are similar, then in this round only the center value is adjusted without expanding the range, and the parameter boundary is decided after more confirmed samples are accumulated; this satisfies real-time adaptation and prevents overfitting of the parameter space caused by a single abnormal sample.

[0184] For example, the current range of light disturbance parameters for a certain type of yellow croaker is... The deformation parameter range is In a certain confirmed sample, the sum of the absolute differences between the elements of the measured residual and the optimal simulation residual was 42. The upper limit of the deviation was set to 35 and the lower limit to 15.

[0185] If further analysis reveals that most of these 42 points originate from the highlighted area on the back, while the depth deviation between the tail and the outline is relatively small, the system will prioritize determining that the lighting disturbance coverage is insufficient, and the lighting disturbance range can be expanded to... While keeping the deformation parameter range constant; if the deviation mainly comes from continuous depth shift at the tail and bending of the main axis of the fish body, then the deformation range should be expanded first, for example from... Expand to ;

[0186] If the residual deviation of another batch of samples remains stable between 8 and 10 for a long period of time, it indicates that the simulation range is set too large. In this case, the system can shrink the range of illumination perturbation parameters to eliminate invalid search space. If the deviation is stable at around 20, only the center value can be adjusted to make the simulated samples more closely match the actual distribution on site.

[0187] Regarding abnormal operating conditions, feedback updates only apply to confirmed results. Rejected and re-verified samples are not included in the update to prevent low-reliability data from polluting the parameter space. If the number of a certain type of sample is too small, for example, if the number of confirmed samples in a day is less than the preset number, then the parameters of that type will not be updated and the previous version will continue to be used.

[0188] If the recognition accuracy drops abnormally after an update, the system supports rolling back to a historical parameter version. If the range of a parameter expands continuously for multiple rounds and approaches the safety limit, it will indicate that the template for that category may not be sufficient to describe the real situation, and the template needs to be recalibrated or a new category needs to be split. The purpose of this step is to use highly reliable production line data to continuously correct the simulation space, so that the model can stably track the drift of the working conditions and avoid the long-term rigidity of the benchmark.

[0189] In a preferred embodiment of the present invention, the sorting execution module includes a control unit and a sorting mechanism. The control unit is used to determine the target sorting channel according to the classification label and generate sorting control instructions. The sorting mechanism includes a conveying device and at least one of a lever, an air jet device or a robotic arm. The sorting mechanism is used to guide the fish into the target sorting channel according to the sorting control instructions.

[0190] This embodiment provides a sorting execution mechanism; if the identification result cannot be implemented in a timely and accurate manner in the physical execution stage, a closed-loop sorting cannot be formed; therefore, after the identification output, the control unit is set to work in coordination with the sorting mechanism to convert the classification label into executable timing control instructions;

[0191] The control unit receives the tag, confidence state, current position of the fish and transmission speed output by the classification decision module. The confidence state is calculated by the weighted sum of the cosine similarity and structural similarity index of the target category in the classification decision module, or by mapping the difference between the highest similarity of the target category and the second best similarity of the target category.

[0192] Calculate the time window for the fish to reach the sorting execution point and generate control commands accordingly; the sorting mechanism can be selected in different forms according to the production line cycle: for small and medium-sized fish, an air jet device can be used for rapid lateral sorting;

[0193] For larger fish or fish requiring precise posture control, levers or robotic arms can be used; the conveying device is responsible for maintaining the fish at a predetermined pace through each execution point; the control unit can encode the target channel as channel 1 for qualified products, channel 2 for slightly defective products, channel 3 for disease-reviewed products, and channel 4 for rejected products, and trigger the corresponding mechanism action at the corresponding time;

[0194] For example, three fish bodies that pass through the identification zone and the execution zone consecutively are denoted as fish bodies. Fish body and fish body Fish body If a fish is identified as a normal yellow croaker in the identification area and is currently 300 mm away from the vent, and the conveyor belt speed is 600 mm per second, it is expected to reach the vent in 0.5 seconds.

[0195] If a normal yellow croaker corresponds to a straight passage, the control unit will not trigger the jet spray; the fish body The fish marked as having abrasions and corresponding to the left diversion channel is given a jet pulse 20 milliseconds before it reaches the jet nozzle, causing it to enter channel 2.

[0196] Fish body The labels are marked as verification results and correspond to the robotic arm's grasping channel. When the labels arrive at the grasping window, the system outputs the grasping pose and grasping time. In this way, the identification labels are converted into physical sorting actions.

[0197] In terms of handling abnormal operating conditions, if the control unit detects that the transmission speed fluctuation exceeds the allowable range, it will recalculate the arrival time, and if necessary, suspend execution and import into the default buffer channel.

[0198] If the sorting mechanism fails, such as insufficient jet pressure, the lever not returning to zero, or the robotic arm being occupied for too long, the system records the failure event and prioritizes guiding the fish into the safe verification channel to avoid it being mistakenly entered into the qualified channel. If the distance between multiple fish in the same time window is less than the preset safe distance, and the sorting accuracy cannot be guaranteed, the control unit can send a deceleration or spacing adjustment command to the front-end feeding cycle.

[0199] The purpose of this step is to stably map the classification results into executable mechanical control actions, thereby achieving closed-loop automated sorting from image recognition to entity separation.

[0200] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. An automatic fish sorting system based on image classification, characterized in that, include: The data acquisition module is used to acquire RGB-D image data of the fish and dynamic light source parameters corresponding to each frame of the RGB-D image data. The dynamic light source parameters include at least illumination direction parameters and illumination intensity parameters. The benchmark construction module is used to generate benchmark fish models based on preset category templates; The simulation generation module is used to perform simulation transformation on the reference fish model based on the dynamic light source parameters, preset illumination perturbation parameters, preset morphological deformation parameters and preset texture perturbation parameters, and generate simulation reference data. The differential extraction module is used to render the benchmark fish model into benchmark RGB-D reference data corresponding to the acquisition angle and resolution of the RGB-D image data, and to perform differential extraction on the RGB-D image data and the simulation reference data after registering them with the benchmark RGB-D reference data to generate the measured residual matrix and the simulation residual matrix. The classification decision module is used to output classification labels based on the similarity between the measured residual matrix and the simulated residual matrix. The classification labels include fish species labels and / or fish body defect labels. The sorting execution module is used to generate sorting control instructions based on the classification labels and control the sorting mechanism to transport the fish to the corresponding sorting channel. The feedback update module is used to calculate the residual deviation between the measured residual matrix and the simulation residual matrix, update the parameter range of the preset illumination perturbation parameter, the preset morphological deformation parameter, or the preset texture perturbation parameter based on the residual deviation, and send it to the simulation generation module.

2. The fish automatic sorting system based on image classification according to claim 1, characterized in that, The data acquisition module includes a color image acquisition unit, a depth image acquisition unit, and a light source parameter acquisition unit; The color image acquisition unit is used to acquire a color image of the fish, and the depth image acquisition unit is used to acquire a depth image of the fish. The RGB-D image data is composed of the color image and the depth image. The light source parameter acquisition unit is used to read the illumination direction parameter and the illumination intensity parameter from the light source controller, or to determine the illumination direction parameter and the illumination intensity parameter based on the illumination sensor and calibration parameters.

3. The fish automatic sorting system based on image classification according to claim 1, characterized in that, The benchmark construction module includes a template storage unit and a model generation unit; The template storage unit is used to store preset category templates, which include fish species templates or fish body defect templates. The model generation unit is used to generate a standard pose contour, surface reflection parameters and standard texture distribution based on the preset category template. The reference fish body model is composed of the standard pose contour, the surface reflection parameters and the standard texture distribution.

4. The fish automatic sorting system based on image classification according to claim 3, characterized in that, The surface reflection parameters include diffuse reflection parameters based on the Lambert diffuse reflection model, or specular reflection parameters used to describe the high-gloss reflection of the fish surface.

5. The automatic fish sorting system based on image classification according to claim 1, characterized in that, The simulation generation module includes a lighting simulation unit, a deformation simulation unit, and a texture simulation unit; The illumination simulation unit is used to perform specular reflection simulation on the benchmark fish model based on the dynamic light source parameters and the preset illumination disturbance parameters. The deformation simulation unit is used to apply bending, torsion, or local deformation to the reference fish model according to the preset morphological deformation parameters. The texture simulation unit is used to change the texture color, texture contrast or texture continuity in a preset area according to the preset texture perturbation parameters, so as to simulate surface damage, color difference abnormalities or patch defects of fish.

6. The fish automatic sorting system based on image classification according to claim 1, characterized in that, The differential extraction module includes a reference rendering unit, a registration unit, a measured differential unit, and a simulation differential unit; The reference rendering unit is used to render the reference fish model into a reference RGB image and a reference depth image, and the reference RGB-D reference data is composed of the reference RGB image and the reference depth image; The registration unit is used to perform scale normalization and spatial registration on the RGB-D image data, the simulation reference data, and the baseline RGB-D reference data; The measured difference unit is used to calculate the pixel-level color difference and depth difference between the RGB-D image data and the reference RGB-D data to generate the measured residual matrix; The simulation differential unit is used to calculate the pixel-level color difference and depth difference between the simulation reference data and the baseline RGB-D reference data to generate the simulation residual matrix.

7. The fish automatic sorting system based on image classification according to claim 6, characterized in that, Both the measured residual matrix and the simulated residual matrix include a color residual submatrix and a depth residual submatrix; The color residual submatrix is ​​used to characterize the residuals of the RGB channels, and the depth residual submatrix is ​​used to characterize the residuals of the depth channels.

8. The fish automatic sorting system based on image classification according to claim 1, characterized in that, The classification decision module includes a similarity calculation unit and a result output unit; The similarity calculation unit is used to expand the measured residual matrix and the simulated residual matrix into feature vectors and calculate the cosine similarity, and calculate the structural similarity index based on the residual submatrices corresponding to the measured residual matrix and the simulated residual matrix; The result output unit is used to output the classification label based on the cosine similarity and the structural similarity index.

9. The fish automatic sorting system based on image classification according to claim 8, characterized in that, The classification decision module is also used to perform threshold interval determination, wherein the threshold interval determination involves thresholds including a first matching threshold and a first exclusion threshold for the cosine similarity, and a second matching threshold and a second exclusion threshold for the structural similarity index, and the first matching threshold is greater than the first exclusion threshold, and the second matching threshold is greater than the second exclusion threshold. When the cosine similarity is greater than or equal to the first matching threshold, and the structural similarity index is greater than or equal to the second matching threshold, the classification result is confirmed. When the cosine similarity is less than or equal to the first exclusion threshold, and the structural similarity index is less than or equal to the second exclusion threshold, the rejection result is output. In other cases, the verification results will be output.

10. The fish automatic sorting system based on image classification according to claim 9, characterized in that, The feedback update module is used to record the corresponding measured residual matrix, the simulated residual matrix, the cosine similarity, and the structural similarity index when outputting the confirmed classification result, and to use the sum of the absolute values ​​of the differences between the corresponding elements of the measured residual matrix and the simulated residual matrix as the residual deviation, and to determine the corresponding parameters that cause the deviation based on the spatial distribution characteristics of the residual deviation in the color residual submatrix and the depth residual submatrix. Specifically, when the residual deviation is concentrated in the highlight area and the depth deviation is small, the corresponding parameter is determined to be the preset illumination perturbation parameter; when the residual deviation manifests as a continuous local contour shift and the depth deviation accounts for a higher proportion, the corresponding parameter is determined to be the preset morphological deformation parameter; when the residual deviation is concentrated in a fixed surface area and the depth change is not significant, the corresponding parameter is determined to be the preset texture perturbation parameter; the specific feedback update rules are as follows: If the residual deviation is greater than or equal to the preset upper limit of deviation, then the parameter range of the corresponding parameter is expanded; If the residual deviation is less than or equal to the preset lower limit of deviation, then the parameter range of the corresponding parameter is narrowed. In other cases, the center value of the corresponding parameter is adjusted, wherein the upper limit of the preset deviation is greater than the lower limit of the preset deviation, and the corresponding parameter includes at least one of the preset illumination perturbation parameter, the preset shape deformation parameter, and the preset texture perturbation parameter; The sorting execution module includes a control unit and the sorting mechanism. The control unit is used to determine the target sorting channel based on the classification label and generate the sorting control command. The sorting mechanism includes a conveying device and at least one of a lever, a jet device, or a robotic arm, and is used to guide fish into the target sorting channel according to the sorting control command.

Images