Fish fry sorting method, system, electronic equipment and storage medium
Through the zoom method and convolutional neural network model, the automatic fry sorting is achieved, which solves the problem of untimely sorting timing and low accuracy, improves the uniformity and efficiency of fry sorting, and reduces manual intervention and seedlings. Excited.
Patent Information
- Application Number
- CN202210199206.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-02
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-03-02
AI Technical Summary
In the prior art, the timing of fish fry sorting is not timely and inappropriate, the accuracy is low and the uniformity is poor, resulting in phenotypic degeneration of varieties, the elimination of excellent genes and the loss of germplasm resources, and the artificial sorting is highly subjective and low efficiency. High labor intensity can easily cause stress and death in seedlings.
The fry image is obtained by the zoom distance method, image feature information is generated through preprocessing, multi-dimensional statistical analysis is carried out to monitor the sorting timing, determine the sorting standards, and use an automated sorting actuator to sort the fry individually, combined with the convolutional neural network model Identify and classify to realize automated sorting.
It improves the uniformity, consistency and quality of fry sorting, reduces manual intervention, reduces seedling stress response, and improves sorting efficiency and accuracy.
Smart Images

Figure CN114612397B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of protection of excellent aquatic germplasm resources and automation of breeding operations, and in particular to a fry sorting method, system, electronic equipment and storage medium. Background Art
[0002] In the process of selecting and breeding ornamental fish such as koi and special fish with obvious differential development, such as grouper, and breeding of high-quality seedlings, it is usually carried out manually in the early stage for multiple irregular and timely sorting to eliminate deformed and underdeveloped young fish, reduce cannibalism and color quality differences caused by large body size differences, so as to ensure, maintain and optimize the stable category characteristics of ornamental fish seedlings such as koi, and ensure the quality of seedlings of different specifications to improve the quality of adult fish and breeding efficiency. At the same time, it is necessary to select undamaged individuals with excellent characteristics such as texture multiple times in the early and subsequent stages of development. In order to cope with this high-intensity selection demand, the speed, uniformity and accuracy of sorting must be improved. Especially in the practice of selecting seedlings for facility aquaculture, in order to improve the uncertain periodic sorting based on one or more characteristics, it is particularly important to accurately monitor or effectively determine the timing of sorting.
[0003] Since artificial sorting is usually carried out when the species is fully developed and the characteristics are very obvious, there is a certain lag. Especially in large-scale breeding under facility conditions, due to the large number of hatching and breeding, high density and deep breeding water, the artificial selection of seedlings with good appearance faces tremendous pressure. Phenomena such as untimely and inappropriate sorting, low accuracy and poor uniformity often occur, which can easily cause phenotypic degradation of varieties, erroneous elimination of excellent genes and loss of germplasm resources. In addition, artificial sorting has problems such as strong subjectivity, low efficiency, high labor intensity and insufficient sorting, and multiple manual interventions can easily cause stress and death of seedlings. Therefore, it is urgent to develop a more integrated, automated and efficient device and method for sorting high-quality seedlings of special aquatic products such as ornamental fish to solve the problems existing in the prior art. Summary of the invention
[0004] The present invention provides a fish fry sorting method, system, electronic device and storage medium, aiming to overcome the following problems in the prior art solutions: untimely and inappropriate sorting timing, low sorting accuracy and poor uniformity, and easy phenotypic degeneration of varieties, erroneous elimination of excellent genes and loss of germplasm resources. The fish fry sorting method, system, electronic device and storage medium provided by the present invention realize the automatic sorting of ornamental fish such as koi and special aquatic seedlings by assisting or replacing manual work, improving the uniformity, consistency and quality of seedlings of different specifications in the same batch, while reducing manual intervention in the sorting process of aquatic seedlings, thereby effectively reducing adverse stress reactions and improving sorting quality and efficiency.
[0005] Specifically, the embodiment of the present invention provides the following technical solutions:
[0006] In a first aspect, an embodiment of the present invention provides a method for sorting fry, comprising:
[0007] Acquiring multiple images of multiple fry by zooming;
[0008] Preprocessing the multiple images to generate multiple image feature information;
[0009] Performing statistical analysis on the plurality of image feature information generated by the preprocessing from multiple dimensions, and monitoring sorting timing based on the results of the statistical analysis;
[0010] Determining sorting criteria based on sorting opportunities; and
[0011] The individual information of the fry is identified according to the sorting standard, and the individual fry are sorted by using an automated sorting actuator.
[0012] Furthermore, the fry sorting method also includes:
[0013] The preprocessing of the plurality of images to generate a plurality of image feature information comprises:
[0014] Edge recognition is performed on the target fry in the multiple images obtained, and then the images are binarized based on a double threshold method.
[0015] Furthermore, the fry sorting method also includes:
[0016] The plurality of dimensions include a first dimension, a second dimension, and a third dimension, wherein the first dimension includes a color component of the fry, the second dimension includes a ratio of spots of the fry, and the third dimension includes a length-to-width ratio of the fry; and
[0017] The plurality of image feature information includes first image feature information corresponding to the first dimension, second image feature information corresponding to the second dimension, and third image feature information corresponding to the third dimension.
[0018] Furthermore, the fry sorting method also includes:
[0019] Determining a threshold range of a plurality of image feature information; and
[0020] The sorting timing is monitored according to whether the feature values of the plurality of image feature information satisfy the threshold range.
[0021] Furthermore, the fry sorting method also includes:
[0022] Based on the fry group image corresponding to the sorting time, the group characteristics and individual characteristics of the fry are analyzed, and the sorting standard is determined according to the overall distribution law of the fry characteristics.
[0023] Furthermore, the fry sorting method also includes:
[0024] Based on the recommended sorting criteria and multiple image information, a convolutional neural network model is trained to generate sorting parameters of the model;
[0025] Retraining the convolutional neural network model through transfer learning; and
[0026] Based on the sorting parameters, a cascade convolutional neural network model for sorting fry is established.
[0027] In a second aspect, an embodiment of the present invention further provides a fry sorting system, comprising:
[0028] An image acquisition module, used for acquiring multiple images of multiple fry by zooming method;
[0029] A preprocessing module, used for preprocessing the multiple images to generate multiple image feature information;
[0030] A sorting timing monitoring module, used to perform statistical analysis on the plurality of image feature information generated by the preprocessing from multiple dimensions, and monitor the sorting timing based on the statistical analysis;
[0031] A sorting standard determination module, used to determine the sorting standard based on the sorting opportunity; and
[0032] The sorting execution module is used to identify the individual information of seedlings according to the sorting criteria, and use an automated sorting execution mechanism to sort the individual seedlings.
[0033] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned fry sorting method when executing the program.
[0034] In a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above-mentioned fry sorting method when executed by a processor.
[0035] It can be seen from the above technical scheme that the fish fry sorting method, system, electronic device and storage medium provided in the embodiments of the present invention realize the automated sorting of ornamental fish such as koi and special aquatic seedlings by assisting or replacing manual work, thereby improving the uniformity, consistency and quality of seedlings of different specifications in the same batch, and reducing manual intervention in the sorting process of aquatic seedlings, thereby effectively reducing adverse stress reactions and improving sorting quality and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0037] Figure 1 A flow chart of a method for sorting fry provided by one embodiment of the present invention;
[0038] Figure 2 A schematic diagram of the structure of a fry sorting system provided by an embodiment of the present invention; and
[0039] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0040] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] In this embodiment, various terms or phrases used in the present invention have the general meanings known to those of ordinary skill in the art. Even so, the present invention still hopes to provide a more detailed description and explanation of these terms or phrases. If the terms and phrases involved in this article are inconsistent with the commonly known meanings, the meanings expressed in the present invention shall prevail; and if not defined in this application, they have the meanings generally understood by those of ordinary skill in the art.
[0042] In the existing technology, there are problems such as untimely and inappropriate sorting, low accuracy and poor uniformity, which can easily lead to phenotypic degradation of varieties, erroneous elimination of excellent genes and loss of germplasm resources. In addition, manual sorting has problems such as strong subjectivity, low efficiency, high labor intensity and insufficient sorting, and multiple manual interventions can easily cause stress and death of seedlings.
[0043] To this end, in the first aspect, an embodiment of the present invention proposes a fish fry sorting method, which aims to assist or replace manual automated sorting of ornamental fish such as koi and specialty aquatic seedlings, improve the uniformity, consistency and quality of seedlings of different specifications in the same batch, and reduce manual intervention in the sorting process of aquatic seedlings, thereby effectively reducing adverse stress reactions and improving sorting quality and efficiency.
[0044] Combine the following Figure 1 The fry sorting method of the present invention is described.
[0045] Figure 1 This is a flow chart of a fry sorting method provided in one embodiment of the present invention.
[0046] In this embodiment, it should be noted that the fry sorting method includes:
[0047] 101: Acquire multiple images of multiple fry by zooming method;
[0048] 102: Preprocessing the multiple images to generate multiple image feature information;
[0049] 103: performing statistical analysis on multiple image feature information generated by preprocessing from multiple dimensions, and monitoring sorting timing based on the results of the statistical analysis;
[0050] 104: Determine a sorting criterion based on the sorting opportunity; and
[0051] 105: Identify individual information of fry according to sorting criteria, and use automated sorting actuators to sort individual fry.
[0052] In one embodiment of the present invention, 101 and 102 can be combined and described as the following first step S1: obtaining image information of the back of a group of young fish such as koi and other ornamental fish and special aquatic products in a large range under facility breeding conditions, performing corresponding preprocessing, and realizing online monitoring of status information such as the growth and development stage of koi ornamental fish;
[0053] In one embodiment of the present invention, 103 can be partially described as the following second step S2: performing statistical feature analysis of color, texture, shape, etc. of the color information of the seedling population at the current moment, combining the temporal characteristics of development, and setting a suitable threshold value for monitoring (or determining) the timing of sorting;
[0054] In one embodiment of the present invention, the rest of 103 and 104 can be combined and described as the following step 3: repeat the first step S1 to step S2 until a suitable sorting opportunity is monitored (or determined), and a suitable sorting standard that meets the requirements is determined;
[0055] In one embodiment of the present invention, a fourth step S4 may be provided: combining the sorting recommendation criteria and the image information, retraining the model by transfer learning or other methods, and establishing a method model for sorting fry of ornamental fish such as koi and special aquatic products according to the parameters;
[0056] In one embodiment of the present invention, 105 can be described as the following fifth step S5: executing the above steps on the memory through the processing unit, identifying the individual information of the seedlings according to the sorting criteria, operating the automated sorting actuator, and completing the sorting of individual seedlings.
[0057] Specifically, for the first step S1: obtaining image information of the back of a group of koi fry in a large range under facility breeding conditions, performing corresponding preprocessing to realize online monitoring of status information such as the growth and development stage of koi ornamental fish, specifically including: step S11, realizing online monitoring and acquisition of specific indicator characteristic image information of a group of koi fry in a specific range at a fixed time interval; step S12, performing edge recognition on the fish body target in the acquired image, and then performing binarization and other processing on the image to eliminate the background and cover up the information difference within the species, so as to realize filling and sorting of the fish body target; step S13, using the target recognition method to locate the position of the fry fish body, and restoring the corresponding fish body color texture and other information based on its original information; through the recognition method, narrowing the extraction range (increasing the effectiveness of feature processing such as texture), for the overlapping part combined with the swimming speed v of the fish body, formulating a reasonable frame time difference t≥n / v, wherein n is the acceptable number of pixels, and v is the number of pixels moving per second, and obtaining as much information as possible by inter-frame subtraction;
[0058] Specifically, for the second step S2: perform statistical feature analysis on the color, texture, shape, etc. (i.e., multiple dimensions) of the color information of the seedling group at the current moment, combine the temporal characteristics of development, and set appropriate thresholds to determine the timing of sorting, wherein the multiple dimensions include a first dimension, a second dimension, and a third dimension, the first dimension includes the color component of the fry, the second dimension includes the proportion of the fry's spots, and the third dimension includes the length-to-width ratio of the fry.
[0059] Specifically, for the third step S3: repeat the first step S1 to the second step S2 until a suitable sorting time is monitored (or determined), and determine a suitable sorting standard that meets the requirements, specifically including: constructing statistics based on image information to formulate a suitable threshold range, using a multi-feature vector to form a sorting threshold, and determining the sorting time within a reasonable threshold range; at the same time, using group photos at that moment as sorting standards, and conducting a comprehensive evaluation based on the calculated values of individual features of the fish image, combining the values of the matrix of changing features such as color and relative area in each image, and analyzing the various components of the group characteristics in combination with the above-mentioned group characteristics, performing statistical analysis by comparing individual characteristics, and determining the value of the standard number S according to distribution rules such as histograms, to complete the determination of the standard number S and other corresponding indicators.
[0060] Specifically, for the fourth step S4: combining the recommended sorting standards and image information, retraining the model through transfer learning and other methods, and establishing a method model for sorting fry of ornamental fish such as koi and specialty aquatic products based on the parameters, specifically including: combining the S value and the pre-selection provided by the convolutional neural network, classifying these individual features, and then having experts use the open source image annotation software labelImg for manual annotation, generating an XML file indicating the true grade of the fish body, and forming a training set for the efficient classification model; based on these training sets, adjusting the model through transfer learning and other methods to form a sorting method for the sorting target at the current stage.
[0061] Specifically, for the fifth step S5: execute the above-mentioned step codes on the memory through the processing unit, identify the individual information of the seedlings according to the sorting standards, operate each automated sorting actuator in the sorting device, and complete the sorting of individual seedlings within the sorting opportunity range, specifically including: execute the above-mentioned first step S1 to the fourth step S4, wherein the first step S1 and the second step S2 are conditional steps, and when the TR and standard equal values of the current stage are generated, execute the fourth step S4 to generate the corresponding model and action execution method; obtain the single image information in combination with the camera device on the sorting mechanism, generate the grade classification according to the convolutional neural network method and use it for the action of the sorting actuator, enter the designated area through the designated sorting unit channel, and complete the sorting.
[0062] Specifically, the embodiment provided by the present invention continuously monitors the seedlings of ornamental fish such as koi and special aquatic products in a facility breeding environment, collects videos before and after the seedlings of ornamental fish such as koi are sorted, and obtains multiple videos of the same batch of koi seedlings in the same period by setting multiple collection points as the basic material for experimental research. Based on the multiple monitoring data sets in the same period, the accuracy and real-time performance of the image information are analyzed by analyzing the quality and completeness of the image set, evaluating the quality of the automatically acquired image set, and formulating the specifications and storage specifications of the images in the image set.
[0063] In a further embodiment, the koi (e.g., common koi) seedlings (0-1 years old) and the stocking density (200-1000 tails / m3) before and after the critical period of body color development are continuously monitored. Specifically, the monitoring is mainly carried out during the day, and the effect of continuous monitoring on individual development is ignored. Specifically, considering that fish gather in groups during the feeding stage, it is easy to cause the acquired images to overlap, so the image information is mainly in the non-feeding stage.
[0064] In one embodiment of the present invention, image information of the back side of a group of koi seedlings in the facility is acquired at intervals of no more than 30 minutes.
[0065] Specifically, the present invention optimizes the position and number of image acquisition devices, and there are multiple image acquisition positions, and their distribution uses a relative position to cover the water surface in an equilateral triangle with a side length of L, where the actual image radius of the middle position of the multi-layer depth image is R, that is, L = √3R, to achieve the acquisition of image information of larger-scale breeding groups.
[0066] Specifically, the image acquisition device used is at least one digital camera (obviously, the image acquisition device of the present invention is not limited thereto), wherein the focal length change is controlled in a digitally programmed manner to adjust when the water depth or the development stage of the fry changes.
[0067] Specifically, the embodiment provided by the present invention preliminarily estimates the key points of artificial intelligence method-assisted sorting operation monitoring and control by analyzing the characteristics of individual and group developmental changes of seedlings of ornamental fish such as koi in the video. More specifically, key points such as the accuracy of koi seedling fish body recognition, the quality of individual (group) image feature analysis calculation, and the effectiveness of sorting recognition, as well as key parameters such as sorting timing and sorting criteria are determined. At the same time, the pertinence and effectiveness of the key points and key parameters determined by video image analysis are combined, and adjustments and optimizations are made to guide the formulation and correction of the basic logic of online sorting.
[0068] Although considering that koi fry have a certain body width-to-height ratio and the main information of the back texture of the body is generally obtained from the top view angle, the embodiment provided by the present invention is not limited to this. Without departing from the spirit and scope of the present invention, ordinary technicians in this field can obtain image information from more perspectives according to actual needs.
[0069] In a further embodiment, the plurality of images includes image information of a side view of a school of fish.
[0070] For example, it is also possible to obtain image information of koi seedlings of different ages in two states (free swimming and swimming channel design) by arranging more camera angles (underwater top, bottom, side). Specifically, under the condition of free swimming, the image acquisition angles include two angles: the top camera, the distance between the lens and the water surface is -5±2mm; the camera is placed horizontally. More specifically, more than 400 videos with a duration of more than 15 minutes can be obtained, and the acquired video frame rate is 60fps to further improve the video data set.
[0071] Specifically, the entire process is carried out underwater, and the free-swimming fish are monitored and sorted. According to the water flow tendency and other characteristics of the fish, the sorting process is carried out entirely underwater, which reduces the process stress of the fry.
[0072] Specifically, grading and sorting are all based on image information. On the basis of judging the health level for sorting, the size and appearance are graded (improving timeliness, rapidity and responsiveness); the monitoring of sorting operations is mainly for images in the non-feeding stage (due to the overlap of feeding stages, it will be aggravated). The intra-species sorting is classified by the neural network method to classify implicit information such as information that is not recognizable or difficult to identify by the human eye, so as to achieve the purpose of mining phenotypic classification. More specifically, all images are acquired when the water surface is relatively still, that is, the monitoring of the sorting timing is mainly in relatively still stages such as the non-feeding stage to eliminate the influence of water surface fluctuations.
[0073] Specifically, the monitoring (or determination) of the sorting timing should be placed before the sorting operation, that is, through continuous monitoring of the breeding status, the information can be comprehensively acquired by zooming in accordance with a fixed focal length change method, and a specific zoom method and image processing method can be combined to search for more information on the entire fish school to obtain corresponding developmental information, thereby monitoring the appropriate sorting time, and then reminding manual work or triggering the corresponding actuator to perform timely sorting operations.
[0074] In the embodiments of the present invention, specifically, the zoom method is based on obtaining fixed multiple levels of deep images, with the purpose of achieving a focal length for clear imaging, wherein the change of the focal length is controlled in a digitally programmed manner to adjust when the water depth or the development stage of the fry changes.
[0075] Through the zoom method, the image plane with a single depth used to determine the sorting timing can be effectively optimized into information of multiple depth planes to provide multi-layer depth image information, which is conducive to the generation of clear image information and also saves the hardware cost of the monitoring system.
[0076] Specifically, the layer with the largest amount of clear information in the multi-layer depth image is obtained. In other words, there is a comparison link between several layers. Specifically, the interval between layers can be changed, for example, the corresponding layer is determined according to the height of the fish body, and it is changed according to the age of the fish body or evenly distributed according to the main depth range of the fish distribution in the water depth. More specifically, let the number of layers be N, and the interval between each layer n=L / N, where L is the depth range where the fish is mainly distributed, and N is an odd number such as 3, 5, 7, etc.; in addition, the acquisition of the layer can also have certain temporal characteristics, that is, it has a sequence of precedence - first the middle, then the shallow layer, and finally the deep layer.
[0077] Specifically, the population characteristics used for sorting timing monitoring refer to the secondary statistics of individual statistical characteristics.
[0078] In this embodiment, it should be noted that the fry sorting method includes:
[0079] Acquiring multiple images of multiple fry and preprocessing the multiple images to generate multiple image feature information includes: performing edge recognition on target fish bodies in the acquired multiple images and then performing binarization processing on the images.
[0080] Specifically, the target recognition method is used to locate the position of the young fish, and the corresponding fish body color and texture information is restored based on its original information. Based on this, rich information is provided for the analysis of group development characteristics, the uniformity of high-quality seedlings is improved, and conditions are provided for refined management. Continuous sorting is required in the future to provide high-quality seedlings of different specifications.
[0081] Specifically, based on the multiple images of the multiple fry acquired, the target fish bodies in the multiple images acquired are edge identified and binarized using an adaptive threshold method and a P parameter threshold method to eliminate the background and mask the information differences within the species, thereby achieving the filling and sorting of the fish body targets, and the fish bodies are identified and located based on target recognition methods such as Mask R-CNN, and restoration processing is performed on the multiple binarized images to generate multiple image feature information including first image feature information and second image feature information based on time conditions and spatial conditions.
[0082] More specifically, the time interval obtained may include, but is not limited to, any time less than or equal to 30 minutes, and without departing from the spirit and scope of the present invention, a person of ordinary skill in the art may select other time intervals as needed according to actual needs. For example, the time interval is specifically adjusted according to the growth rate and growth stage.
[0083] More specifically, the spatial condition can be to narrow the extraction scope through recognition methods to increase the effectiveness of feature processing such as texture, and the temporal condition can be to formulate a reasonable frame time difference for the overlapping parts combined with the swimming speed of the fish body to obtain as much information as possible by inter-frame subtraction.
[0084] In this embodiment, it should be noted that the fry sorting method includes:
[0085] The multiple dimensions include a first dimension, a second dimension, and a third dimension, wherein the first dimension includes the color component of the fry, the second dimension includes the pattern ratio of the fry, and the third dimension includes the aspect ratio of the fry; and the multiple image feature information includes first image feature information corresponding to the first dimension, second image feature information corresponding to the second dimension, and third image feature information corresponding to the third dimension.
[0086] In this embodiment, it should be noted that the fry sorting method includes:
[0087] Determine a threshold range of multiple image feature information; and monitor sorting timing according to whether feature values of the multiple image feature information meet the threshold range.
[0088] Specifically, the present invention pre-places a monitoring method for sorting timing before the sorting operation, which can improve the monitoring accuracy of the sorting timing, and the sorting timing is monitored by the sorting timing threshold. In addition, the monitoring method for sorting timing can also be used as a separate method to monitor and determine the timing of the sorting operation to remind humans or take relevant intervention measures. The steps of the monitoring method for sorting timing are iterative, and various indicators are floating and adjustable to enhance the adaptability of sorting special seedlings such as koi ornamental fish at different specifications and different developmental stages; by setting sorting thresholds at multiple different stages, identifying key points of sorting and multi-stage characteristics of development, multiple periodic sorting can be achieved; to achieve the optimization of koi seedlings at multiple developmental stages of koi seedlings of different specifications, thereby improving the reuse rate and applicability of the sorting device.
[0089] Specifically, based on the image feature information of the seedling population in the above multiple dimensions at the current moment, combined with the temporal characteristics of development, a suitable threshold range T for determining the sorting timing T is determined. R .
[0090] Specifically, the timing of the sorting operation refers to the best time to complete the current sorting operation, which is an interval quantity determined based on indicators such as development stage and quantity.
[0091] Specifically, the sorting time T is an interval quantity, which satisfies the sorting threshold range T R The corresponding time t plus the compensation of sorting timing, T = t + P·Q·T stage,n(P and Q are both less than 1), where P is a correction coefficient related to the sorting quantity scale, Q is the stage adjustment coefficient (this coefficient is related to different specifications and varieties), T stage,n The duration of the stage.
[0092] Specifically, the sorting threshold T R It can be a multidimensional vector, that is, when sorting around multiple features, such as color component, pattern proportion and aspect ratio, then T R =[T Ra , T Rb , T Rc ], and [α, β, γ] are weight values, then the sorting threshold T' R =[T Ra , T Rb , T Rc ]·[α, β, γ], when T' R ≥T value,n When , it indicates the trigger sorting moment. Among them, α, β, γ are constants of 0-1, and α>β>γ, indicating that the three feature components have certain priorities; T value,n is the trigger threshold, which is greater than zero, and n represents different stages.
[0093] More specifically, if the feature value of the image feature information meets the reasonable threshold range T R It is determined that it is at the time of sorting. If the characteristic value of the image feature information does not meet the reasonable threshold range T R It is determined that it is not at the time of sorting.
[0094] For example, key points of population development dynamics information can be combined with T R A component of T Ra As an example, the threshold range T is determined based on the feature matrix values of multiple images. Ra , and other more accurate indicator statistics can be further refined. The following shows how to determine the threshold range T by calculating the area ratio of body color change in the sorting threshold vector. Ra Example to determine the timing of sorting:
[0095]
[0096]
[0097]
[0098]
[0099] Among them, T Rarepresents the sorting opportunity range. When its value is 1, the sorting operation of this feature can be performed. Specifically, T1 represents the sorting value of this component, which is a percentage determined according to the development stage. A represents the area where the color development quality is greater than a certain value, and B represents the total area of the fish body, that is, the sum of all valid pixels. For the binarization formula (1) based on the double threshold method, f′(x,y), f(x,y) represent the pixel values, a represents the lower limit of the pixel value, and b represents the upper limit.
[0100] The triggering of sorting time, taking sorting in three dimensions as an example, T' R =[T Ra , T Rb , T Rc ]·[α, β, γ]; take T value,n In the above equation, n is 1, which means stage 1, and its value is 1;
[0101] For example, refer to T R , when T Ra , T Rb , T Rc When all are 1, and α, β, γ are 0.5, 0.3, and 0.2 respectively, T' R =1≥T value,1 =1, indicating triggering sorting; when T Ra , T Rb , T Rc When α, β, and γ are 0.5, 0.3, and 0.2 respectively, T' R =0.8≤T value,1 =1, indicating that sorting is not triggered.
[0102] Each sorting stage corresponds to a set of vector values. If the sorting threshold contains three components, it is determined that the appropriate threshold range T' is met. R , that is, it is at the right time for sorting. Then there are S = M*N*L sorting criteria, that is, S represents the number of sorting criteria of the indicator vector, M, N, L correspond to the number of groups of a component respectively, which is used to group the components in the indicator group, S, M, N, L are all positive integers greater than or equal to 1 and are not 1 at the same time, so that the sorting grouping is more subtle.
[0103] In other words, in the embodiment of the present invention, the fish schools at the time of sorting are sorted according to the determined number of classification criteria S. Specifically, images are acquired based on the individual individuals entering the sorting device or structure to analyze their classification, and the fish bodies (or fry) that meet the classification criteria are sorted to complete the sorting.
[0104] Specifically, the monitoring information within the time period is analyzed individually, and the identified individual fish bodies are marked and grouped. The existing classification model is trained through the transfer learning method. The existing classification model is a cascade neural network. Specifically, the tracking and recognition method and the classification and recognition method are cascaded, and the model and the automatic sorting execution process are optimized through the above-mentioned parameters such as the sorting timing, the number of sorting standards S (or the number of standards S), etc.
[0105] In this embodiment, it should be noted that the fry sorting method includes:
[0106] Based on the fry group image corresponding to the sorting time, the group characteristics and individual characteristics of the fry are analyzed, and the sorting criteria are determined according to the overall distribution law of the fry characteristics.
[0107] Specifically, based on the fry group image corresponding to the sorting time, the group characteristics and individual characteristics of the fry are analyzed, and the sorting standard is determined according to the overall distribution law of the fry characteristics. Specifically, the group photos at that moment are used as the sorting standard, and a comprehensive evaluation is performed based on the calculated values of the individual characteristics of the fish body image. Combined with the change feature matrix values such as color and relative area in each image, the various components of the group characteristics are analyzed in combination with the above-mentioned group characteristics, and statistical analysis is performed by comparing the individual characteristics. The value of S is determined according to the distribution law such as histogram, and the standard number S and other corresponding indicators are determined. Furthermore, these individual characteristics are classified in combination with the S value and convolutional neural network, and then manually annotated using the open source image annotation software labelImg to generate an XML file indicating the true grade of the fish body, forming a training set for an efficient classification model.
[0108] In this embodiment, it should be noted that the fry sorting method includes:
[0109] Based on the recommended sorting criteria and multiple image information, a convolutional neural network model is trained to generate sorting parameters of the model; the convolutional neural network model is retrained through transfer learning; and a cascaded convolutional neural network model is established based on the sorting parameters.
[0110] Specifically, different convolutional neural network entities are used in series and matched with the sorting threshold range T R And the number of sorting criteria S are two important sorting parameters to form a new cascade convolutional neural network.
[0111] More specifically, based on the serial processing of two convolutional neural networks with different functions, according to the cascade model logic of “fish body recognition-efficient classification”, the Mask R-CNN series methods and the 3D-CNN series methods were initially used in cascade.
[0112] For another example, in order to make the established model achieve a certain recognition accuracy, the present invention applies specific indicators to control the algorithm training accuracy, and trains and tests the two parts of the cascaded convolutional neural network respectively to analyze the key influencing parameters in the selection of koi seedlings.
[0113] Specifically, the koi fish body was extracted manually, and then the manual extraction and algorithm extraction were compared. The algorithm's extraction quality for koi seedling fish bodies was evaluated based on indicators such as similarity S (similarity), accuracy A (Accuracy), over extraction rate OER (Over Extraction Rate), and missed extraction rate MER (Missed Extraction Rate). When the algorithm's extraction area is larger than the actual extraction area, or an area is divided into two or more, it is called over-extraction; conversely, when the algorithm's extraction area is smaller than the actual extraction area, it is called missed extraction.
[0114] According to the following formula, the evaluation index of the koi seedling extraction result is obtained:
[0115]
[0116]
[0117]
[0118]
[0119] Among them, I g-t For manual extraction standards, define I g For actual extraction, I s In order to utilize the extraction criteria extracted by the algorithm, subjective evaluation is used in the evaluation to assist in judging the errors existing in manual extraction.
[0120] According to the following formula, the koi seedling classification and sorting algorithm index is obtained:
[0121]
[0122]
[0123]
[0124]
[0125] Among them, the mean average detection accuracy M is selected as the sorting performance evaluation index. AP 、Detection accuracy of each level of sorting standards corresponding to the category A P , accuracy P, detection time T r , error rate E r, recall rate R, intersection-over-intersection ratio I OU ; Error rate E r =100%-P; intersection ratio I OU is the ratio of the intersection and union of the predicted box and the true box, which represents the accuracy of the koi fry fish feature recognition; M AP It is to calculate the A of each item in each sorting standard of Koi seedlings P The average value of A P is the integral of precision P over recall R; and where T P is the number of samples that are actually positive and detected as positive, F P is the number of samples that are actually negative and detected as positive, F N is the number of samples that are actually positive but detected as negative samples, c is the total number of categories, k is the category number (k=1, 2, ...); J is the average precision function, that is, the area of the PR curve composed of the precision P and recall R of category number k.
[0126] The beneficial technical effects of the method and system proposed in the present invention are as follows: 1) relying on the special growth and development characteristics and tendencies of koi and other species and assisted by mechanical devices for sorting and processing, the process stress and damage of the fish body can be effectively reduced, and the efficiency and effect of the sorting operation can be improved; 2) the method effectively adapts to the multi-stage characteristics of koi and the characteristics of automated sorting operations, and provides better and more specific technical parameters for the automatic sorting of high-quality seedlings of ornamental fish such as koi to achieve precise control of the automatic sorting process; 3) the method adopts a technical method combining statistical analysis of group characteristics and individual characteristics for sorting, which improves the quality of seedlings of ornamental fish such as koi, ensures the timeliness, sufficiency and advance amount of sorting operations, and improves the consistency and uniformity of sorting high-quality seedlings.
[0127] Based on the same inventive concept, on the other hand, an embodiment of the present invention provides a fry sorting system.
[0128] Combine the following Figure 2 The fry sorting system provided by the present invention is described. The fry sorting system described below and the fry sorting method described above can correspond to each other.
[0129] Figure 2 A schematic diagram of the structure of a fry sorting system provided in one embodiment of the present invention.
[0130] In the present embodiment, it should be noted that a fry sorting system 1 includes: an image acquisition module 10, used to acquire multiple images of multiple fry; a preprocessing module 20, used to preprocess the multiple images to generate multiple image feature information; a sorting timing monitoring module 30, used to perform statistical analysis on the multiple image feature information generated by preprocessing from multiple dimensions, and monitor the sorting timing based on the statistical analysis; a sorting standard determination module 40, used to determine the sorting standard based on the sorting timing; and a sorting execution module 50, used to identify the individual information of the fry according to the sorting standard, operate the automated sorting execution mechanism, and complete the sorting of the individual fry.
[0131] Since the fry sorting system provided in the embodiment of the present invention can be used to execute the fry sorting method and sorting timing determination method described in the above embodiment, its working principle and beneficial effects are similar, so they will not be described in detail here. For specific contents, please refer to the introduction of the above embodiment.
[0132] The fry sorting system provided in one embodiment of the present invention has a certain suspension and support structure, and can be flexibly and directly arranged in a facility breeding environment without complex adjustments; in addition, the fry sorting system also has a counting function, and the counting is strictly related to the sorting grade, and can be used to count the number of sortings for each developmental characteristic, and can be used for artificial breeding and phenotypic statistics and other occasions; the sorting result is koi fry with relatively consistent uniformity, similar color and texture, and consistent appearance grade, and can achieve sorting of a variety of different specifications, providing good fry for refined breeding.
[0133] In this embodiment, it should be noted that the various modules in the system of the embodiment of the present invention can be integrated into one or deployed separately. The above modules can be combined into one module or further divided into multiple sub-modules.
[0134] On the other hand, based on the same inventive concept, another embodiment of the present invention provides an electronic device.
[0135] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention.
[0136] In this embodiment, it should be noted that the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the above-mentioned fry sorting method and the method for determining the sorting timing, and the fry sorting method includes: obtaining multiple images of multiple fry by zooming method; preprocessing the multiple images to generate multiple image feature information; statistically analyzing the multiple image feature information generated by preprocessing from multiple dimensions, and monitoring the sorting timing based on the results of the statistical analysis; determining the sorting criteria based on the sorting timing; identifying the individual information of the fry according to the sorting criteria and sorting the individual fry using an automated sorting actuator.
[0137] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0138] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the fry sorting method provided in the above-mentioned embodiments, the method comprising: acquiring multiple images of multiple fry by a zoom method; preprocessing the multiple images to generate multiple image feature information; performing statistical analysis on the multiple image feature information generated by the preprocessing from multiple dimensions, and monitoring the sorting timing based on the results of the statistical analysis; determining the sorting criteria based on the sorting timing; identifying the individual information of the fry according to the sorting criteria and using an automated sorting actuator to sort the individual fry.
[0139] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.
[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0141] In addition, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0142] In addition, in the present invention, the description with reference to the terms "embodiment", "present embodiment", "another embodiment", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.
[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for sorting fry, It is characterized in that include: By using the zoom method, multiple images of multiple fry are obtained; Preprocessing the multiple images to generate multiple image feature information; Statistical analysis is performed on the plurality of image feature information generated by the preprocessing from multiple dimensions, and sorting timing is monitored based on the result of the statistical analysis, the plurality of dimensions comprising a first dimension, a second dimension, and a third dimension, wherein the first dimension comprises a color component of the fry, the second dimension comprises a proportion of spots of the fry, and the third dimension comprises an aspect ratio of the fry, the plurality of image feature information comprises first image feature information corresponding to the first dimension, second image feature information corresponding to the second dimension, and third image feature information corresponding to the third dimension, including: Determining a threshold range of the plurality of image feature information; and monitoring the sorting timing according to whether the feature values of the plurality of image feature information satisfy the threshold range; Determine the sorting criteria based on the sorting timing, including: Analyzing the group characteristics and individual characteristics of the fry based on the fry group image corresponding to the sorting time, and determining the sorting standard according to the overall distribution law of the fry characteristics; and The individual information of the fry is identified according to the sorting standard, and the individual fry are sorted by using an automated sorting actuator.
2. The method for sorting fry according to claim 1, It is characterized in that The preprocessing of the plurality of images to generate a plurality of image feature information comprises: Edge recognition is performed on the target fry in the multiple images obtained, and then the images are binarized based on a double threshold method.
3. The method for sorting fry according to claim 1, It is characterized in that The method further comprises: training a fry sorting model, wherein the training fry sorting model comprises: Based on the recommended sorting criteria and multiple image information, a convolutional neural network model is trained to generate sorting parameters of the model; Retraining the convolutional neural network model through transfer learning; and Based on the sorting parameters, a cascade convolutional neural network model for sorting fry is established.
4. A fish fry sorting system, It is characterized in that The fry sorting system comprises an image acquisition module, a preprocessing module, a sorting timing monitoring module, a sorting standard determination module and a sorting execution module, and the fry sorting system is used to execute the fry sorting method according to claim 1, The image acquisition module is used to acquire multiple images of multiple fry by zooming method. The preprocessing module is used to preprocess the multiple images to generate multiple image feature information, The sorting timing monitoring module is used to perform statistical analysis on the multiple image feature information generated by the preprocessing from multiple dimensions, and monitor the sorting timing based on the result of the statistical analysis, the multiple dimensions include a first dimension, a second dimension and a third dimension, wherein the first dimension includes the color component of the fry, the second dimension includes the proportion of the stripes of the fry, and the third dimension includes the aspect ratio of the fry, the multiple image feature information includes first image feature information corresponding to the first dimension, second image feature information corresponding to the second dimension, and third image feature information corresponding to the third dimension, including: Determining a threshold range of the plurality of image feature information; and monitoring the sorting timing according to whether the feature values of the plurality of image feature information satisfy the threshold range; The sorting standard determination module is used to determine the sorting standard based on the sorting opportunity, including: Analyzing the group characteristics and individual characteristics of the fry based on the fry group image corresponding to the sorting time, and determining the sorting standard according to the overall distribution law of the fry characteristics; and The sorting execution module is used to identify the individual information of the fry according to the sorting standard, and use the automatic sorting execution mechanism to sort the individual fry.
5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the steps of the fry sorting method according to any one of claims 1 to 3 are implemented.
6. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the fry sorting method according to any one of claims 1 to 3 are implemented.
