A data augmentation method for fish identification and a fish identification method and system
Through the augmentation method of fish identification data, the problem of no public data set and insufficient recognition accuracy is solved, efficient fish recognition in different scenarios is achieved, the deployment process is simplified, and the recognition accuracy and flexibility are improved.
Patent Information
- Application Number
- CN202210192884.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-01
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-03-01
AI Technical Summary
In the prior art, fish identification has problems such as no public data set, insufficient recognition accuracy, poor flexibility and cumbersome deployment, especially when model training is difficult to achieve rapid deployment and efficient identification.
Using the data augmentation method of fish recognition, the environmental images and fish images of different underwater scenes are obtained, and the fused fish image dataset is generated and divided into training Set and test sets are used for convolutional neural network model training to realize automatic annotation and recognition.
Rapidly establish public data sets covering different scenarios, improve the accuracy of fish identification models, simplify implementation methods, improve flexibility, and realize rapid automatic labeling to reduce the number of model iterations.
Smart Images

Figure CN114663721B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data augmentation method for fish identification and a fish identification method and system. Background Art
[0002] Scientific development and protection of fishery resources and the construction of modern fishery demonstration parks are the inevitable requirements for the transformation and upgrading of fisheries and the realization of greater development. The development of modern fisheries involves many new facilities, new technologies, and new mechanisms, but the most important part is fishery monitoring. Automated detection methods based on computer vision technology have been widely used in the field of aquaculture, and many systematic and effective fish identification methods have been established.
[0003] However, there are the following defects in the current identification of underwater fish:
[0004] First, there is no public dataset;
[0005] Second, if the same batch of data is used for model training in different scenarios (natural environment, water clarity, light changes, etc.), the recognition accuracy is not enough;
[0006] Third, it cannot be deployed quickly, and it is necessary to repeatedly collect images at different time periods and then perform data annotation.
[0007] As a result, the current use of deep learning to identify fish is cumbersome and inflexible. Summary of the invention
[0008] The present invention provides a data augmentation method for fish identification and a fish identification method and system to solve the problem that the implementation methods of fish identification in the prior art are relatively cumbersome and inflexible. The data augmentation method in the present invention can quickly establish a public data set covering different scenes, improve the accuracy of the fish identification model, and has a simple implementation method and high flexibility. At the same time, the fish identification method and system in the present invention can realize fast and automatic labeling, reducing the number of model iterations in engineering deployment.
[0009] The technical solution adopted by the present invention is:
[0010] A data augmentation method for fish identification, the method comprising the following steps:
[0011] Step S1, establishing different underwater scenes, and obtaining at least one underwater environment image for each underwater scene; the different underwater scenes are generated by the natural environment, water clarity and / or light changes, and fish do not appear in the underwater scenes; the pixel size of each underwater environment image is the same;
[0012] Step S2, setting a clear water underwater scene, and placing fish in the clear water underwater scene; obtaining a plurality of first fish images of the clear water underwater scene; the pixel size of each first fish image is the same, and the pixel size of the first fish image is the same as the pixel size of the underwater environment image;
[0013] Step S3, selecting a first fish image, determining and marking the identifiable fish in the first fish image one by one, and recording them in the first record file; the marking content includes the fish category and location information, and the marking format is: {(class), (x1, y1), (x2, y2)},
[0014] Among them, class is the fish category;
[0015] (x1, y1) is the coordinate point of the upper left corner of the largest circumscribed rectangular box of a single fish;
[0016] (x2, y2) is the coordinate point of the lower right corner of the largest circumscribed rectangular box of the same single fish;
[0017] Step S4, after all identifiable fish in the first fish image are marked, a first record file is saved, and the save name of the first record file is the same as the save name of the corresponding first fish image;
[0018] Step S5, repeating step S3 and step S4 until all first fish images are labeled and corresponding first record files are saved;
[0019] Step S6, selecting a first annotated fish image and performing grayscale processing and image segmentation processing to obtain a second fish image without background; the pixel size of the second fish image is the same as the pixel size of the underwater environment image, and the annotation content and annotation format of the second fish image are the same as the annotation content and annotation format of the first fish image;
[0020] Step S7, repeating step S6 until all first fish images are converted into corresponding second fish images without background;
[0021] Step S8, selecting a second fish image as a foreground, selecting an underwater environment image as a background, superimposing the second fish image and the underwater environment image by pixels to generate a third fish image, and recording the annotation content of the second fish image in a second record file; the second record file is saved with the same name as the corresponding third fish image;
[0022] Step S9, repeat step S8, complete pixel superposition of each second fish image and all underwater environment images, obtain a data set of the third fish image and a data set of the corresponding second record file, and complete data augmentation for fish identification.
[0023] Furthermore, in step S3, the specific process of determining the identifiable fish in the first fish image one by one includes:
[0024] Step S31, selecting any fish in the first fish image;
[0025] Step S32, if the fish does not overlap with the existing images of other fish, observe whether the head, body and tail morphological features of the fish are complete and clearly visible; if the observation result is complete, determine that it is identifiable and mark it;
[0026] Step S33, if the fish overlaps with the existing images of other fish, observe whether the overlapping area of the fish and other fish affects the display of the morphological characteristics of the fish; if the observation result shows that the morphological characteristics are relatively complete and clearly visible, determine that the fish is identifiable and mark it;
[0027] Step S34, repeating steps S31 to S33 until all identifiable fish in the first fish image are determined and marked.
[0028] Furthermore, in step S6, a specific process of selecting a first labeled fish image and performing grayscale processing and image segmentation processing to obtain a second fish image without background includes:
[0029] Step S61, selecting a first labeled fish image and performing grayscale processing;
[0030] Step S62, selecting the {(x1, y1), (x2, y2)} region where a certain identifiable fish is located on the first fish image after grayscale processing;
[0031] Step S63, performing image segmentation on the {(x1, y1), (x2, y2)} region by using threshold segmentation, binarization or watershed segmentation, removing the background inside and outside the {(x1, y1), (x2, y2)} region except the image of the fish, retaining only the identifiable fish image, and obtaining a second fish image;
[0032] Step S64, repeating steps S62 and S63, performing image segmentation processing on the remaining areas where the identifiable fish are located on the first fish image, to obtain a corresponding second fish image;
[0033] Alternatively, in step S6, the specific process of selecting a first labeled fish image and performing grayscale processing and image segmentation processing to obtain a second fish image without background includes:
[0034] Step S65, selecting a first labeled fish image and performing grayscale processing;
[0035] Step S66, selecting the {(x1, y1), (x2, y2)} region where a certain identifiable fish is located on the first fish image after grayscale processing;
[0036] Step S67, performing image segmentation on the {(x1, y1), (x2, y2)} region by using threshold segmentation, binarization or watershed segmentation, removing the background except the image of the fish in and outside the {(x1, y1), (x2, y2)} region, and retaining only the identifiable fish image;
[0037] Step S68, repeat steps S66 and S67, perform image segmentation processing on the remaining areas where the identifiable fish on the first fish image are located, and wait until all the areas where the identifiable fish on the first fish image are located have completed the image separation processing, only retain all the identifiable fish images, and the first fish image is converted as a whole into a second fish image without background.
[0038] Furthermore, in step S8, the specific process of performing pixel superposition of the second fish image and the underwater environment image to generate the third fish image is as follows:
[0039] The whole second fish image is pixel-superimposed with the whole underwater environment image to generate a new underwater environment image with fish species and position information, and the new underwater environment image is the third fish image;
[0040] A fish identification method, the method comprising the following steps:
[0041] Step SP1, using the aforementioned data augmentation method for fish identification to obtain a data set of a third fish image and a corresponding data set of a second record file;
[0042] Step SP2, dividing the data set of the third fish image and the data set of the corresponding second record file into a training set and a test set according to a data volume ratio of 8:2, and sending them to a convolutional neural network for fish recognition model training, retaining the fish recognition model with the highest recognition performance, and using the fish recognition model with the highest recognition performance as the final fish recognition model;
[0043] Step SP3, obtain real-time images in the fishway, input the image frames into the final fish recognition model, and automatically mark the fish.
[0044] Furthermore, the method further comprises the following steps:
[0045] Step SP4, performing fish size recognition, fish swimming speed recognition, fish swimming direction recognition and fish counting on the fish species that have been recognized;
[0046] Step SP5, the generated fish data is stored in a database, and the fish data includes fish species, fish size, fish swimming speed, fish swimming direction and fish quantity.
[0047] Furthermore, in step SP4, the specific process of fish size identification is as follows:
[0048] Step SP41, using Kalman filtering and Hungarian algorithm to locate the same fish in real time and obtain a real-time fish image of the fish;
[0049] Step SP42, the final fish recognition model identifies the head center coordinates (xhead, yhead), tail center coordinates (xtail, ytail) and fish position information {(x1, y1), (x2, y2)} in the real-time fish image of the fish; connect the fish head center and tail center coordinates, and record the angle θ between the fish head and tail connection line and the horizontal direction, and And 0≤θ<90; the fish size is calculated as:
[0050] Furthermore, in step SP4, the specific process of fish swimming speed identification is as follows:
[0051] Step SP43, setting the front boundary and the rear boundary in the real-time picture in the fishway, and the water flow direction is from the front boundary to the rear boundary;
[0052] Step SP44, obtaining the distance L between the center points of the maximum circumscribed rectangular frames of the same fish when the center point of the maximum circumscribed rectangular frame crosses the front boundary or the rear boundary between the two picture frames;
[0053] Step SP44, calculate the fish swimming speed v = distance between two points L / two frame time T.
[0054] Furthermore, in step SP4, the specific process of fish swimming direction identification and fish counting is as follows:
[0055] Step SP45, taking the line connecting the center points of the largest circumscribed rectangular frame of the same fish in two adjacent picture frames and the rear side boundary line as the analysis line;
[0056] Step SP46, the rear boundary line vector is The line connecting the two fish detection frames is
[0057] like but exist Clockwise, away from the camera screen, the fish swim downstream;
[0058] like but exist In the counterclockwise direction, the fish enters the camera screen and swims against the current;
[0059] like but and co-linear, fish not swimming;
[0060] If the same fish crosses both the rear boundary line and the front boundary line, it is counted once;
[0061] Alternatively, step SP47, taking the line connecting the center points of the largest circumscribed rectangular frame of the same fish in the two picture frames and the front side boundary line as the analysis line;
[0062] Step SP48, the rear boundary line vector is The line connecting the two fish detection frames is
[0063] like but exist In the counterclockwise direction, away from the camera screen, the fish swims upstream;
[0064] like but exist Clockwise, the fish enter the camera screen and swim downstream;
[0065] like but and co-linear, fish not swimming;
[0066] If the same fish crosses both the front and rear boundary lines, it is counted once.
[0067] A fish identification system for implementing the aforementioned fish identification method, the system comprising:
[0068] An image acquisition module, the image acquisition module comprising an underwater camera and a compensation light source, for acquiring a first fish image, an underwater environment image, and a real-time screen image;
[0069] A training data augmentation module, which receives the fish data and underwater environment data collected by the image acquisition module and performs a data augmentation operation;
[0070] An interaction and training module, which is used for fish species labeling, fish recognition model training and fish information display during data augmentation; the fish information includes fish species, fish size, fish swimming speed, fish swimming direction and fish quantity;
[0071] The data storage module is used for data storage.
[0072] The beneficial effects of the present invention are:
[0073] The present invention is to solve the problem that the implementation method of fish identification in the prior art is relatively cumbersome and the flexibility is poor, and provides a data augmentation method for fish identification and a fish identification method and system. In the data augmentation method for fish identification, an underwater environment image is first obtained, and then a first fish image in a clear water underwater scene is obtained and annotated, and then the first fish image is grayed and segmented to obtain a second fish image, and finally the second fish image is fused with the underwater environment image to obtain a third fish image. In the fish identification method and system, the data set of the third fish image obtained by the data augmentation method and the data set of the corresponding second record file are divided into a training set test set, and sent to a convolutional neural network for fish identification model training, and the fish identification model with the highest recognition performance is retained, and the fish identification model with the highest recognition performance is used as the final fish identification model, and finally the fish is automatically annotated by the final fish identification model. The data augmentation method in the present invention can quickly establish a public data set covering different scenes, improve the accuracy of the fish identification model, and has a simple implementation method and high flexibility. At the same time, the fish identification method and system in the present invention can realize rapid automatic annotation and reduce the number of model iterations in engineering deployment. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0075] Figure 1 This is the image obtained during data augmentation in Example 1.
[0076] Figure 2 In Example 1, it is the first fish image.
[0077] Figure 3 This is a flow chart of the fish identification method in Example 2.
[0078] Figure 4 This is the real-time picture frame in Example 2.
[0079] Figure 5 This is a schematic diagram of θ when calculating the fish size in Example 2. DETAILED DESCRIPTION
[0080] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.
[0081] In the description of the present invention, it needs to be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.
[0082] The disclosure below provides many different embodiments or examples to realize different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention.
[0083] The embodiments of the invention are described in detail below with reference to the accompanying drawings.
[0084] Example 1
[0085] At present, there are the following defects in identifying underwater fish:
[0086] First, there is no public dataset;
[0087] Second, if the same batch of data is used for model training in different scenarios (background, water clarity, etc.), the recognition accuracy is not enough;
[0088] Third, it cannot be deployed quickly, and it is necessary to repeatedly collect images at different time periods and then perform data annotation.
[0089] As a result, the current use of deep learning to identify fish is cumbersome and inflexible.
[0090] In order to solve the problem that the implementation methods of fish identification in the prior art are relatively complicated and inflexible due to defects in the data set, a data augmentation method for fish identification is provided in this embodiment.
[0091] Specifically, the data augmentation method for fish identification includes the following steps:
[0092] Step S1, establishing different underwater scenes, and obtaining at least one underwater environment image for each underwater scene; different underwater scenes are generated by natural environment, water clarity and / or light changes, and fish do not appear in the underwater scenes; the pixel size of each underwater environment image is the same.
[0093] Step S2, setting a clear water underwater scene, and placing fish in the clear water underwater scene; obtaining a plurality of first fish images of the clear water underwater scene; the pixel size of each first fish image is the same, and the pixel size of the first fish image is the same as the pixel size of the underwater environment image.
[0094] For example, a simulated on-site fishway is built according to the focal length of the on-site camera and the size of the fishway. The netting method and the blocking method are used to obtain the actual fish species that need to be tested. The captured fish are placed in the simulated fishway and clean water is added to facilitate the acquisition of fish characteristics and the collection and processing of fish data.
[0095] Step S3, selecting a first fish image, determining and marking the identifiable fish in the first fish image one by one, and recording them in the first record file; the marking content includes the fish category and location information, and the marking format is: {(class), (x1, y1), (x2, y2)},
[0096] Among them, class is the fish category;
[0097] (x1, y1) is the coordinate point of the upper left corner of the largest circumscribed rectangular box of a single fish.
[0098] (x2, y2) is the coordinate point of the lower right corner of the largest circumscribed rectangular box of the same single fish;
[0099] Step S4, after all identifiable fish in the first fish image are marked, save the first record file, and the save name of the first record file is the same as the save name of the corresponding first fish image.
[0100] Step S5, repeating step S3 and step S4 until all first fish images are labeled and corresponding first record files are saved.
[0101] Step S6, select a labeled first fish image and perform grayscale processing and image segmentation processing to obtain a second fish image without background; the pixel size of the second fish image is the same as the pixel size of the underwater environment image, and the annotation content and annotation format of the second fish image are the same as the annotation content and annotation format of the first fish image.
[0102] Step S7, repeating step S6 until all first fish images are converted into corresponding second fish images without background.
[0103] Step S8, select a second fish image as the foreground, select an underwater environment image as the background, superimpose pixels of the second fish image and the underwater environment image to generate a third fish image, and record the annotation content of the second fish image in the second record file; the save name of the second record file is the same as the save name of the corresponding third fish image.
[0104] Step S9, repeat step S8, complete pixel superposition of each second fish image and all underwater environment images, obtain a data set of the third fish image and a data set of the corresponding second record file, and complete data augmentation for fish identification.
[0105] For example, Figure 1 As shown in , an underwater environment image G1 and a first fish image G2 are first captured, and then G2 is processed to obtain a second fish image G3. Finally, pixels of G1 and G3 are superimposed to obtain a third fish image G4.
[0106] In this embodiment, the data augmentation method of the invention obtains an underwater environment image and a second fish image without background respectively, and then fuses the underwater environment image and the second fish image to obtain a third fish image. This can quickly establish a public data set covering different scenes, eliminating the need to repeatedly collect images of different time periods, facilitating rapid deployment and improving the accuracy of the fish recognition model. The implementation method is simple and highly flexible.
[0107] Furthermore, in step S3, the specific process of determining the identifiable fish in the first fish image one by one includes:
[0108] Step S31, selecting any fish in the first fish image.
[0109] Step S32, if the fish does not overlap with the existing images of other fish, observe whether the head, body and tail morphological features of the fish are complete and clearly visible; if the observation result is complete, it is determined to be identifiable and marked.
[0110] for example, Figure 2 At point A in the figure, the fish is not obstructed by other fish, and its morphological features are complete and clearly visible, confirming that it can be identified.
[0111] Step S33, if the images of the fish overlap with those of other fish, observe whether the overlapping area between the fish and other fish affects the display of the morphological characteristics of the fish; if the observation result is that the morphological characteristics are relatively complete and clearly visible, it is determined to be identifiable and marked.
[0112] for example, Figure 2 In the middle B, the head of the fish is blocked by other fish, and its morphological features are incomplete and cannot be identified.
[0113] Step S34, repeating steps S31 to S33 until all identifiable fish in the first fish image are determined and marked.
[0114] The technical effect of the above technical scheme is: through the above method for determining identifiable fish, the rules are confirmed to be the same, and different personnel can refer to this method to determine all the identifiable fish in the first fish image, reduce the interference of human factors, and improve the utilization rate of the first fish image, so as to utilize subsequent fish labeling and improve the recognition accuracy of the fish recognition model.
[0115] Furthermore, in step S6, a specific process of selecting a first labeled fish image and performing grayscale processing and image segmentation processing to obtain a second fish image without background includes:
[0116] Step S61, selecting a first labeled fish image and performing grayscale processing.
[0117] Step S62, selecting the {(x1, y1), (x2, y2)} region where a certain identifiable fish is located on the first fish image after grayscale processing.
[0118] Step S63, perform image segmentation on the {(x1, y1), (x2, y2)} region by using threshold segmentation, binarization or watershed segmentation, remove the background inside and outside the {(x1, y1), (x2, y2)} region except the image of the fish, and only retain the recognizable fish image to obtain a second fish image.
[0119] Step S64, repeating steps S62 and S63, performing image segmentation processing on the remaining areas of the first fish image where the identifiable fish are located, to obtain a corresponding second fish image.
[0120] For example, there are 10 identifiable fish in the first fish image, and after grayscale processing and segmentation processing, ten second fish images can be obtained. Each second fish image only shows one corresponding identifiable fish.
[0121] Alternatively, in step S6, the specific process of selecting a first labeled fish image and performing grayscale processing and image segmentation processing to obtain a second fish image without background includes:
[0122] Step S65, selecting a labeled first fish image and performing grayscale processing.
[0123] Step S66, selecting the {(x1, y1), (x2, y2)} region where a certain identifiable fish is located on the first fish image after grayscale processing.
[0124] Step S67, perform image segmentation on the {(x1, y1), (x2, y2)} region by using threshold segmentation, binarization or watershed segmentation, remove the background inside and outside the {(x1, y1), (x2, y2)} region except the image of the fish, and only retain the recognizable fish image.
[0125] Step S68, repeat steps S66 and S67, perform image segmentation processing on the remaining areas where the identifiable fish on the first fish image are located, and wait until all the areas where the identifiable fish on the first fish image are located have completed the image separation processing, only retain all the identifiable fish images, and the first fish image is converted as a whole into a second fish image without background.
[0126] For example, there are 10 identifiable fish in the first fish image, and after grayscale processing and segmentation processing, a second fish image can be obtained. The second fish image shows ten identifiable fish.
[0127] The above technical solution has the beneficial effect that after grayscale processing and image segmentation processing, the background in the first fish image can be removed, and only the image that can fully display the morphological characteristics of the fish is retained, and this is used as the foreground of the data augmentation image. Therefore, it is not necessary to obtain images in the fishway under different scenes, and a large amount of data required for training the fish recognition model can be obtained.
[0128] Furthermore, in step S8, the specific process of performing pixel superposition of the second fish image and the underwater environment image to generate the third fish image is as follows:
[0129] The entire second fish image is pixel-superimposed with the entire underwater environment image to generate a new underwater environment image with fish species and position information, and the new underwater environment image is the third fish image.
[0130] The beneficial effects of the above technical solution are: through image fusion technology, a large amount of data required for training fish recognition models in different scenarios can be obtained in a short time, and rapid deployment can be carried out, which is conducive to improving the recognition accuracy of the fish recognition model, and the implementation method is simple and highly flexible. The pixel size of the third fish image, the pixel size of the underwater environment image, the pixel size of the first fish image, and the pixel size of the second fish image obtained by the overall fusion of the picture are all the same, which is conducive to the establishment of subsequent fish recognition models and real-time fish recognition.
[0131] Example 2
[0132] Fish identification method, the process is as follows Figure 3 as shown in .
[0133] The fish identification method comprises the following steps:
[0134] Step SP1, using the aforementioned data augmentation method for fish identification, to obtain a data set of a third fish image and a corresponding data set of a second record file.
[0135] Step SP2, divide the data set of the third fish image and the corresponding data set of the second record file into a training set and a test set according to a data volume ratio of 8:2, and send them into a convolutional neural network for fish recognition model training, retain the fish recognition model with the highest recognition performance, and use the fish recognition model with the highest recognition performance as the final fish recognition model.
[0136] Step SP3, obtain the real-time picture in the fishway, input the picture frame into the final fish recognition model, and automatically mark the fish. Figure 4 as shown in .
[0137] In this embodiment, augmented images are used as training sets and test sets of the fish recognition model. The obtained fish recognition model has high recognition accuracy and can realize fast and automatic labeling of fish.
[0138] Furthermore, the method further comprises the following steps:
[0139] Step SP4, performing fish size recognition, fish swimming speed recognition, fish swimming direction recognition and fish counting on the fish species that have been recognized;
[0140] Step SP5, the generated fish data is stored in a database, and the fish data includes fish species, fish size, fish swimming speed, fish swimming direction and fish quantity.
[0141] Furthermore, in step SP4, the specific process of fish size identification is as follows:
[0142] Step SP41, using Kalman filtering and Hungarian algorithm to locate the same fish in real time and obtain a real-time fish image of the fish;
[0143] Step SP42, the final fish recognition model identifies the head center coordinates (xhead, yhead), tail center coordinates (xtail, ytail) and fish position information ((x1, y1), (x2, y2)} in the real-time fish image of the fish; connect the fish head center and tail center coordinate points, and record the angle θ between the fish head and tail connection line and the horizontal direction, and And 0≤θ<90, such as Figure 5 As shown in ; the final calculated fish size is:
[0144] Furthermore, in step SP4, the specific process of fish swimming speed identification is as follows:
[0145] Step SP43, the front boundary and the rear boundary are set in the real-time picture in the fishway, and the water flow direction is from the front boundary to the rear boundary, such as Figure 4 As shown in;
[0146] Step SP44, obtaining the distance L between the center points of the maximum circumscribed rectangular frames of the same fish when the center point of the maximum circumscribed rectangular frame crosses the front boundary or the rear boundary between the two picture frames;
[0147] Step SP44, calculate the fish swimming speed v = distance between two points L / two frame time T.
[0148] Furthermore, in step SP4, the specific process of fish swimming direction identification and fish counting is as follows:
[0149] Step SP45, taking the center point connection line of the largest circumscribed rectangular frame of the same fish in two adjacent picture frames and the rear side boundary line as the analysis line; if frame extraction is performed to cater to the hardware device, then two adjacent picture frames after processing are selected;
[0150] Step SP46, the rear boundary line vector is The line connecting the two fish detection frames is
[0151] like but exist Clockwise, away from the camera screen, the fish swim downstream;
[0152] like but exist In the counterclockwise direction, the fish enters the camera screen and swims against the current;
[0153] like but and co-linear, fish not swimming;
[0154] If the same fish crosses both the rear boundary line and the front boundary line, it is counted once;
[0155] Alternatively, step SP47, taking the line connecting the center points of the largest circumscribed rectangular frame of the same fish in the two picture frames and the front side boundary line as the analysis line;
[0156] Step SP48, the rear boundary line vector is The line connecting the two fish detection frames is
[0157] like but exist In the counterclockwise direction, away from the camera screen, the fish swims upstream;
[0158] like but exist Clockwise, the fish enter the camera screen and swim downstream;
[0159] like but and co-linear, fish not swimming;
[0160] If the same fish crosses both the front and rear boundary lines, it is counted once.
[0161] Example 3
[0162] A fish identification system, the system comprising:
[0163] An image acquisition module, which includes an underwater camera and a compensation light source, and is used to acquire a first fish image, an underwater environment image, and a real-time screen image, etc.;
[0164] A training data augmentation module, which receives the fish data and underwater environment data collected by the image acquisition module and performs a data augmentation operation;
[0165] An interaction and training module, which is used for fish species labeling, fish recognition model training and fish information display during data augmentation; the fish information includes fish species, fish size, fish swimming speed, fish swimming direction and fish quantity;
[0166] The data storage module is used for data storage.
Claims
1. Data augmentation methods for fish identification, It is characterized in that The method comprises the following steps: Step S1, establishing different underwater scenes, and obtaining at least one underwater environment image for each underwater scene; the different underwater scenes are generated by the natural environment, water clarity and / or light changes, and fish do not appear in the underwater scenes; the pixel size of each underwater environment image is the same; Step S2, setting a clear water underwater scene, and placing fish in the clear water underwater scene; obtaining a plurality of first fish images of the clear water underwater scene; the pixel size of each first fish image is the same, and the pixel size of the first fish image is the same as the pixel size of the underwater environment image; Step S3, selecting a first fish image, determining and marking the identifiable fish in the first fish image one by one, and recording them in a first record file; the marking content includes the fish category and location information, and the marking format is: {(class), (x1, y1), (x2, y2)}, where class is the fish category; (x1, y1) is the coordinate point of the upper left corner of the largest circumscribed rectangular box of a single fish; (x2, y2) is the coordinate point of the lower right corner of the largest circumscribed rectangular box of the same single fish; Step S4, after all identifiable fish in the first fish image are marked, a first record file is saved, and the save name of the first record file is the same as the save name of the corresponding first fish image; Step S5, repeating step S3 and step S4 until all first fish images are labeled and corresponding first record files are saved; Step S6, selecting a first annotated fish image, and performing grayscale processing and image segmentation processing to obtain a second fish image without background; the pixel size of the second fish image is the same as the pixel size of the underwater environment image, and the annotation content and annotation format of the second fish image are the same as the annotation content and annotation format of the first fish image; Step S7, repeating step S6 until all first fish images are converted into corresponding second fish images without background; Step S8, selecting a second fish image as a foreground, selecting an underwater environment image as a background, superimposing the second fish image and the underwater environment image by pixels to generate a third fish image, and recording the annotation content of the second fish image in a second record file; the second record file is saved with the same name as the corresponding third fish image; Step S9, repeat step S8, complete pixel superposition of each second fish image and all underwater environment images, obtain a data set of the third fish image and a data set of the corresponding second record file, and complete data augmentation for fish identification.
2. The data augmentation method for fish identification according to claim 1, It is characterized in that In step S3, the specific process of determining the identifiable fish in the first fish image one by one includes: Step S31, selecting any fish in the first fish image; Step S32, if the fish does not overlap with the existing images of other fish, observe whether the head, body and tail morphological features of the fish are complete and clearly visible; if the observation result is complete, determine that it is identifiable and mark it; Step S33, if the fish overlaps with the existing images of other fish, observe whether the overlapping area of the fish and other fish affects the display of the morphological characteristics of the fish; if the observation result shows that the morphological characteristics are relatively complete and clearly visible, determine that the fish is identifiable and mark it; Step S34, repeating steps S31 to S33 until all identifiable fish in the first fish image are determined and marked.
3. The data augmentation method for fish identification according to claim 1, It is characterized in that In step S6, a specific process of selecting a first labeled fish image and performing grayscale processing and image segmentation processing to obtain a second fish image without background includes: Step S61, selecting a first labeled fish image and performing grayscale processing; Step S62, selecting the {(x1, y1), (x2, y2)} region where a certain identifiable fish is located on the first fish image after grayscale processing; Step S63, performing image segmentation on the {(x1, y1), (x2, y2)} region by using threshold segmentation, binarization or watershed segmentation, removing the background inside and outside the {(x1, y1), (x2, y2)} region except the image of the fish, retaining only the identifiable fish image, and obtaining a second fish image; Step S64, repeating steps S62 and S63, performing image segmentation processing on the remaining areas where the identifiable fish are located on the first fish image, to obtain a corresponding second fish image; Alternatively, in step S6, a specific process of selecting a first labeled fish image and performing grayscale processing and image segmentation processing to obtain a second fish image without background includes: Step S65, selecting a first fish image after labeling and performing grayscale processing; Step S66, selecting the {(x1, y1), (x2, y2)} region where a certain identifiable fish is located on the first fish image after grayscale processing; Step S67, performing image segmentation on the {(x1, y1), (x2, y2)} region by using threshold segmentation, binarization or watershed segmentation, removing the background in and around the {(x1, y1), (x2, y2)} region except for the image of the fish, and retaining only the identifiable fish image; Step S68, repeat steps S66 and S67, perform image segmentation processing on the remaining areas where the identifiable fish on the first fish image are located, and wait until all the areas where the identifiable fish on the first fish image are located have completed the image separation processing, only retain all the identifiable fish images, and the first fish image is converted as a whole into a second fish image without background.
4. The data augmentation method for fish identification according to claim 3, It is characterized in that In step S8, the specific process of performing pixel superposition of the second fish image and the underwater environment image to generate the third fish image is as follows: The entire second fish image is pixel-superimposed with the entire underwater environment image to generate a new underwater environment image with fish species and position information, and the new underwater environment image is the third fish image.
5. Fish identification methods, It is characterized in that The method comprises the following steps: Step SP1, using the data augmentation method for fish identification as described in any one of claims 1 to 4 to obtain a data set of a third fish image and a data set of a corresponding second record file; Step SP2, dividing the data set of the third fish image and the data set of the corresponding second record file into a training set and a test set according to a data volume ratio of 8:2, and sending them to a convolutional neural network for fish recognition model training, retaining the fish recognition model with the highest recognition performance, and using the fish recognition model with the highest recognition performance as the final fish recognition model; Step SP3, obtain real-time images in the fishway, input the image frames into the final fish recognition model, and automatically mark the fish.
6. The fish identification method according to claim 5, It is characterized in that The method further comprises the following steps: Step SP4, performing fish size recognition, fish swimming speed recognition, fish swimming direction recognition and fish counting on the fish species that have been recognized; Step SP5, storing the generated fish data into a database, the fish data including fish species, fish size, fish swimming speed, fish swimming direction and fish quantity.
7. The fish identification method according to claim 6, It is characterized in that In step SP4, the specific process of fish size identification is as follows: Step SP41, using Kalman filtering and Hungarian algorithm to locate the same fish in real time and obtain a real-time fish image of the fish; Step SP42, the final fish recognition model identifies the head center coordinates (xhead, yhead), tail center coordinates (xtail, ytail) and fish position information {(x1, y1), (x2, y2)} in the real-time fish image of the fish; connect the fish head center and tail center coordinates, and record the angle θ between the fish head and tail connection line and the horizontal direction, and And 0≤θ<90; the fish size is calculated as:
8. The fish identification method according to claim 7, It is characterized in that In step SP4, the specific process of fish swimming speed identification is as follows: Step SP43, setting the front boundary and the rear boundary in the real-time picture in the fishway, and the water flow direction is from the front boundary to the rear boundary; Step SP44, obtaining the distance L between the center points of the maximum circumscribed rectangular frames of the same fish when the center point of the maximum circumscribed rectangular frame crosses the front boundary or the rear boundary between the two picture frames; Step SP44, calculate the fish swimming speed v = distance between two points L / two frame time T.
9. The fish identification method according to claim 8, It is characterized in that In step SP4, the specific process of fish swimming direction identification and fish counting is as follows: Step SP45, taking the line connecting the center points of the largest circumscribed rectangular frame of the same fish in two adjacent picture frames and the rear side boundary line as the analysis line; Step SP46, the rear boundary line vector is The line connecting the two fish detection frames is like but exist Clockwise, away from the camera screen, the fish swim downstream; like but exist In the counterclockwise direction, the fish enters the camera screen and swims against the current; like but and co-linear, fish not swimming; If the same fish crosses both the rear boundary line and the front boundary line, it is counted once; Alternatively, in step SP47, a line connecting the center points of the largest circumscribed rectangular frame of the same fish in two picture frames and a front boundary line are used as analysis lines; Step SP48, the rear boundary line vector is The line connecting the two fish detection frames is like but exist In the counterclockwise direction, away from the camera screen, the fish swims upstream; like but exist Clockwise, the fish enter the camera screen and swim downstream; like but and co-linear, fish not swimming; If the same fish crosses both the front and rear boundary lines, it is counted once.
10. A fish identification system implementing the fish identification method according to any one of claims 5 to 9, It is characterized in that The system comprises: An image acquisition module, the image acquisition module comprising an underwater camera and a compensation light source, for acquiring a first fish image, an underwater environment image, and a real-time screen image; A training data augmentation module, which receives the fish data and underwater environment data collected by the image acquisition module and performs a data augmentation operation; An interaction and training module, which is used for fish species labeling, fish recognition model training and fish information display during data augmentation; the fish information includes fish species, fish size, fish swimming speed, fish swimming direction and fish quantity; The data storage module is used for data storage.
