School state recognition method and system, electronic device and storage medium

Abstract

The present application provides a kind of fish population state identification method and system, electronic equipment and storage medium, method includes: based on the video to be identified, determine the image set to be identified;Wherein, the video to be identified includes the image information of the fish population to be identified;The image set to be identified includes: RGB image and optical flow image;RGB image is the image of space information characteristic;Optical flow image is the image of time information characteristic;The image set to be identified is input into fish population state identification double-flow neural network, determines target fish population state identification result;Wherein, fish population state identification double-flow neural network is based on the labeled sample image set determined according to sample video training obtains;When fish population state identification double-flow neural network is trained, target loss function is used to transfer the time flow and space flow prediction probability modified according to preset hyperparameter. It can improve the precision of fish population state identification, reduce the waste of human resources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, and in particular to a fish school state recognition method and system, an electronic device and a storage medium. BACKGROUND

[0002] At present, aquaculture is changing from extensive to intensive, and the production structure is constantly upgrading. Land-based factory farming is one of the important farming methods. Land-based factory farming is supported by high scientific and technological level, and uses technologies such as Internet of Things, big data and intelligent equipment.

[0003] In land-based factory farming, the fish school has a large density, and most fish have a clustering behavior, and are distributed and moved in a very orderly state. Dynamic analysis of the position distribution and activity of the fish school as a whole is beneficial to realize water quality regulation, water purification, intelligent feeding, automation and precision of land-based factory circulating water aquaculture.

[0004] At present, the breeding of fish in land-based factory workshops basically depends on manual inspection and the experience of breeding personnel to judge the survival state of the fish school, which consumes a large amount of manpower and material resources, causing waste of human resources. Moreover, the state recognition result is greatly affected by the subjective factors of the breeding personnel, and its accuracy depends on the experience of the breeding personnel, so it is difficult to objectively and accurately determine the state of the fish school.

[0005] Therefore, how to provide a fish school state recognition method and system, an electronic device and a storage medium, which can realize the recognition of the state of the fish school without contact and at low cost, without the participation of manual work, and the recognition result is less affected by subjective factors, can improve the accuracy of the recognition of the state of the fish school, and reduce the waste of human resources, has become a problem to be solved. SUMMARY

[0006] The fish school state recognition method and system, the electronic device and the storage medium provided by the present application can solve the above problems in the prior art, can realize the recognition of the state of the fish school without contact and at low cost, without the participation of manual work, and the recognition result is less affected by subjective factors, can improve the accuracy of the recognition of the state of the fish school, and reduce the waste of human resources.

[0007] The fish school state recognition method provided by the present application comprises:

[0008] Based on the to-be-recognized video, a to-be-recognized image set is determined; wherein the to-be-recognized video comprises image information of a to-be-recognized fish school; the to-be-recognized image set comprises an RGB image and an optical flow image; the RGB image is an image representing spatial information; and the optical flow image is an image representing temporal information;

[0009] input the image set to be identified into a fish school state identification double-flow neural network to determine a target fish school state identification result;

[0010] The fish school state identification double-flow neural network is trained based on a sample image set with labels determined according to a sample video; and the target loss function is used to deliver time flow and space flow prediction probabilities modified according to preset hyperparameters during training of the fish school state identification double-flow neural network.

[0011] According to the fish school state identification method provided in the application, the fish school state identification double-flow neural network comprises a space information network layer, a time information network layer and an identification result fusion layer.

[0012] The space information network layer is configured to determine a space fish school state identification result of the video to be identified according to the RGB image.

[0013] The time information network layer is configured to determine a time fish school state identification result of the video to be identified according to the optical flow image.

[0014] The identification result fusion layer is configured to determine a target fish school state identification result by weighting fusion according to preset weights based on the space fish school state identification result and the time fish school state identification result.

[0015] According to the fish school state identification method provided in the application, the space information network layer comprises a space feature extraction layer and a space feature identification layer; and the time information network layer comprises a time feature extraction layer and a time feature identification layer.

[0016] The space information network layer is configured to obtain a space flow feature of the video to be identified according to the RGB image.

[0017] The space feature identification layer is configured to determine the space fish school state identification result according to the space flow feature of the video to be identified.

[0018] The time feature extraction layer is configured to obtain a time flow feature of the video to be identified according to the optical flow image.

[0019] The time feature identification layer is configured to determine the time fish school state identification result according to the time flow feature of the video to be identified.

[0020] According to the fish school state identification method provided in the application, before the step of inputting the image set to be identified into the fish school state identification double-flow neural network to determine the target fish school state identification result, the method further comprises determining a sample image set based on a sample video.

[0021] The step of determining the sample image set based on the sample video specifically comprises:

[0022] processing the sample video to obtain a continuous video frame atlas;

[0023] determining an RGB image set based on the continuous video frame atlas;

[0024] adding an optical flow displacement field between every few consecutive video frames in the continuous video frame atlas to obtain an optical flow image;

[0025] determining an optical flow image set based on the optical flow image;

[0026] labeling sample fish school state labels according to a certain proportion of the training set and the test set based on the RGB image set and the optical flow image set, and determining a sample image set.

[0027] According to the fish school state recognition method provided by the application, before the step of inputting the to-be-recognized image set into the fish school state recognition dual-flow neural network and determining the target fish school state recognition result, the fish school state recognition dual-flow neural network is trained.

[0028] The training of the fish school state recognition dual-flow neural network specifically includes:

[0029] determining the sample image set based on the sample video; wherein the sample video includes image information of a sample fish school; and the sample image set includes an RGB image and an optical flow image;

[0030] training the fish school state recognition dual-flow neural network by using the sample image set;

[0031] updating the network parameters of the fish school state recognition dual-flow neural network by using a gradient descent method based on the target loss function; wherein the target loss function is determined according to the preset hyperparameters, the preset loss function weight, the relative entropy loss function and the cross entropy loss function; the hyperparameters include a time hyperparameter and a space hyperparameter; the time hyperparameter is used to correct the time flow category information in the fish school state recognition dual-flow neural network; and the space hyperparameter is used to correct the space flow category information in the fish school state recognition dual-flow neural network.

[0032] According to the fish school state recognition method provided by the application, the target loss function Loss is:

[0033] Loss=KL(Ps,Pt′)*α*T*T+CE(Ps,G)*(1-2α)*0.5+CE(Pt,G)*(1-2α)*0.5+KL(Ps′,Pt)*α*T*T

[0034] Wherein, KL is a relative entropy loss function; CE is a cross-entropy loss function; Ps is time flow class information; Ps' is modified time flow class information; Pt is spatial flow class information; Pt' is modified spatial flow class information; a is a weight of the relative entropy loss function; (1-2a) is a weight of the cross-entropy loss function; G is sample fish swarm real class information; and T is a distillation parameter.

[0035] According to the fish swarm state recognition method provided in the application, the network parameters of the fish swarm state recognition double-flow neural network are updated using the gradient descent method based on the target loss function, and the fish swarm state recognition double-flow neural network is iteratively trained based on the updated network parameters until the fish swarm state recognition double-flow neural network converges, and the fish swarm state recognition method specifically comprises the following steps:

[0036] The network parameters of the fish swarm state recognition double-flow neural network are updated using the small-batch gradient descent method based on the target loss function, the learning rate is adjusted according to the cosine annealing strategy, and the fish swarm state recognition double-flow neural network is iteratively trained based on the updated network parameters until the fish swarm state recognition double-flow neural network converges.

[0037] The application further provides a fish swarm state recognition system, which comprises an input determination unit and a state recognition unit.

[0038] The input determination unit is configured to determine a set of to-be-recognized images based on a to-be-recognized video, wherein the to-be-recognized video comprises image information of a to-be-recognized fish swarm, the set of to-be-recognized images comprises an RGB image and an optical flow image, the RGB image is an image representing spatial information, and the optical flow image is an image representing temporal information.

[0039] The state recognition unit is configured to input the set of to-be-recognized images into a fish swarm state recognition double-flow neural network to determine a target fish swarm state recognition result.

[0040] The fish swarm state recognition double-flow neural network is trained based on a set of labeled sample images determined according to a sample video; and the target loss function is used to transfer the time flow and spatial flow prediction probabilities modified according to preset hyperparameters when the fish swarm state recognition double-flow neural network is trained.

[0041] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the fish swarm state recognition method according to any one of the above-described methods when executing the program.

[0042] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the steps of the fish swarm state recognition method according to any one of the above-described methods.

[0043] The fish school state recognition method and system, the electronic device and the storage medium provided by the application recognize the fish school state by extracting the space-time information from the to-be-recognized video containing the fish school state information. The motion and distribution information of the fish school in space and time are comprehensively utilized to obtain better state recognition effect, accurate recognition of the fish school state can be realized without contact and at low cost, and problems such as low efficiency and non-objective recognition result caused by manual observation and recording of the fish school state are solved. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0045] Figure 1 is a flow chart of the fish school state recognition method provided by the application;

[0046] Figure 2 is a schematic diagram of the dual-stream neural network structure of the fish school state recognition provided by the application;

[0047] Figure 3 is a flow chart of the fish school state recognition method provided by the application;

[0048] Figure 4 is a structural schematic diagram of the fish school state recognition system provided by the application;

[0049] Figure 5 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the application more clear, the technical solutions in the application will be described clearly and completely in combination with the drawings in the application. Obviously, the described embodiments are some embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0051] In the prior art, the study on fish behavior is mainly based on RGB image to analyze the abnormal behavior and feeding behavior of fish individuals. Due to the characteristics of non-closedness, incomplete contour, non-uniform and changing spatial distribution of fish school, and rich visual information in real breeding environment, when the actual algorithm is used to recognize the fish school, it is difficult to effectively extract the features of the fish school in the analysis of fish behavior.

[0052] Figure 1 is a flow chart of a fish population state recognition method provided by the present application, as shown in the figure, the fish population state recognition method provided by the present application comprises: Figure 1

[0053] Step S1, based on the to-be-recognized video, determining a to-be-recognized image set; wherein the to-be-recognized video comprises image information of a to-be-recognized fish population; the to-be-recognized image set comprises an RGB image and an optical flow image; the RGB image is an image representing spatial information; and the optical flow image is an image representing temporal information;

[0054] Step S2, inputting the to-be-recognized image set into a fish population state recognition dual-stream neural network to determine a target fish population state recognition result;

[0055] Wherein, the fish population state recognition dual-stream neural network is trained based on a labeled sample image set determined according to a sample video; and when the fish population state recognition dual-stream neural network is trained, a target loss function is used to transfer temporal flow and spatial flow prediction probabilities modified according to preset hyperparameters.

[0056] It should be noted that the execution subject of the above method can be a computer device.

[0057] Specifically, taking the identification of different states of fish populations in a video segment obtained from a land-based factory farming workshop as an example, the present application is described. First, a to-be-recognized video containing image information of a to-be-recognized fish population is obtained, which is shot in a land-based factory farming workshop.

[0058] It can be understood that, in order to more accurately recognize the state of the fish population, when obtaining the video of the fish population, the image information of the distribution of the fish population on the surface of the water (horizontal direction distribution) needs to be included.

[0059] It should be noted that, when obtaining the to-be-recognized video, it can be obtained based on a camera pre-set in the land-based factory, it can also be shot by a staff, or it can be obtained based on other methods, and the present application does not limit the source of the to-be-recognized video. In addition, the present application can adaptively change the sample data used for training the network and the input recognition data, and is adaptively used for state recognition of other fish populations or other groups with clustering behavior, and the present application does not limit this.

[0060] ​In step S1, based on the obtained to-be-identified video including image information of a fish school to be identified, the to-be-measured video is preprocessed to obtain a to-be-identified image set. The to-be-identified image set includes an RGB image and an optical flow image; the RGB image is an image representing spatial information; the optical flow image is an image representing time information and can capture motion information, so that the network can focus on moving objects in the video.

[0061] It should be noted that the method of pre-processing the to-be-identified video to obtain the RGB image and the optical flow image can be that the video segment is pre-processed into consecutive RGB video frames by using a code or a video capture tool, and an optical flow displacement field is added between consecutive video frames to obtain optical flow information and an optical flow image. The to-be-identified image set is composed of the obtained RGB video frame (RGB image) and the optical flow image. This method is only used as a specific example to explain and illustrate the present application, and the method of determining the to-be-identified image set is not limited.

[0062] The to-be-identified image set can include all the processed RGB video frames, or a certain number of video frames can be selected according to a preset extraction rule (random extraction, selection of a fixed number of video frames, further extraction of key video frames based on frame difference method, etc.). The proportional relationship between the spatial color image and the optical flow image in the to-be-identified image set can be selected according to actual conditions, and the present application is not limited in this regard.

[0063] In step S2, the to-be-identified image set is input into the fish school state recognition dual-flow neural network. The dual-flow neural network includes a time flow and a space flow. The time flow processes the optical flow image, and the space flow processes the RGB image. According to the extracted time and space features in the to-be-identified video, the target fish school state recognition result is determined.

[0064] It should be noted that before using the trained fish school state recognition dual-flow neural network, the fish school state recognition dual-flow neural network needs to be trained using a sample image set with labels determined according to a sample video.

[0065] Before training the network based on the sample image set, the state of the fish school in the sample image set needs to be labeled. For example, the state of the fish school is divided into: normal state of the fish school, early abnormal state, late abnormal state, weak feeding degree, general feeding degree, and severe feeding degree. The sample data with the same fish school state are divided together and labeled with the corresponding state category label.

[0066] It should be noted that the state division of the fish school can be adjusted according to actual needs, and secondly, the method of adding labels can be selected according to actual conditions, and the present application is not limited in this regard.

[0067] The fish school state recognition double-flow neural network can be regarded as a time flow network and a space flow network, and when training, according to the knowledge distillation idea, a hyperparameter is introduced, the prediction probability of the time flow and the space flow is respectively corrected according to the preset hyperparameter, and the information contained in the time flow and the space flow prediction probability distribution corrected according to the preset hyperparameter is transmitted to each other by using a target loss function, and the joint training mode can effectively improve the recognition rate of the network.

[0068] It can be understood that with the gradual deepening or widening of the network structure, the network performance is greatly improved, but at the same time, the calculation amount is greatly increased, the network parameters and the forward calculation time are increased, for example, VGGNet, ResNet and DenseNet, and it is difficult to achieve real-time operation effect by relying on these networks. Therefore, the application can use a lighter neural network SqueezeNet as a backbone network, which can effectively reduce the parameters participating in operation and has faster detection speed, and can better meet the needs of actual industrial applications.

[0069] The fish school state recognition system provided by the application extracts space-time information from the to-be-recognized video containing fish school state information and performs fish school state recognition. The motion and distribution information of fish groups in space and time are comprehensively utilized to obtain better state recognition effect, and accurate recognition of fish school state can be realized without contact and at low cost. According to the knowledge distillation idea, a hyperparameter is introduced, the prediction probability of the time flow and the space flow is respectively corrected according to the preset hyperparameter, and the information contained in the time flow and the space flow prediction probability distribution corrected according to the preset hyperparameter is transmitted to each other by using a target loss function, and the joint training mode can effectively improve the recognition rate of the network. The problems of low efficiency and non-objective recognition results caused by relying on manual observation and recording of fish school state are solved.

[0070] Optionally, according to the fish school state recognition method provided by the application, the fish school state recognition double-flow neural network comprises a space information network layer, a time information network layer and an identification result fusion layer.

[0071] The space information network layer is used for determining the space fish school state recognition result of the to-be-recognized video according to the RGB image.

[0072] The time information network layer is used for determining the time fish school state recognition result of the to-be-recognized video according to the optical flow image.

[0073] The identification result fusion layer is used for determining a target fish school state recognition result by weighting fusion based on the space fish school state recognition result and the time fish school state recognition result according to a preset weight.

[0074] Specifically,Figure 2 is a schematic diagram of a fish population state recognition double-flow neural network structure provided by the present application, as shown in Figure 2 The fish population state recognition double-flow neural network includes a spatial information network layer, a time information network layer, and a recognition result fusion layer.

[0075] The image set to be recognized is input into the fish population state recognition double-flow neural network, and the image set to be recognized is divided into RGB images and optical flow images, which are respectively input into different layers for processing. The time information network layer is used for processing the optical flow images.

[0076] The RGB images are input into the spatial information network layer, which is used to determine the spatial fish population state recognition result of the video to be recognized according to the RGB images.

[0077] The optical flow images are input into the time information network layer, which is used to determine the time fish population state recognition result of the video to be recognized according to the optical flow images.

[0078] The recognition result fusion layer is used to determine the target fish population state recognition result by weighting fusion according to the pre-set weight based on the spatial fish population state recognition result and the time fish population state recognition result.

[0079] It can be understood that the recognition result determined by the network is the score (probability) of different categories. When performing weighted fusion, the same category scores in the spatial fish population state recognition result and the time fish population state recognition result are weighted and fused according to the pre-set weight.

[0080] For example, the weight of the spatial network is set to 0.4, the weight of the time network is set to 0.6, the probability of determining that the fish population state to be measured is the initial abnormal state in the spatial fish population state recognition result is 0.7, the probability of determining that the fish population state to be measured is the initial abnormal state in the time fish population state recognition result is 0.8, and the target fish population state probability determined after the recognition result fusion layer is 0.4 x 0.7 + 0.6 x 0.8 = 0.76.

[0081] It should be noted that the pre-set weight setting described above is only used as a specific example to illustrate the present application. In the actual application process of the present application, the specific value of the pre-set weight can be set according to the actual situation, and the present application does not limit it.

[0082] Further, it can be understood that the final state recognition result output mode can be to output only the predicted fish population state with the highest probability, or to select multiple fish population states with higher probabilities, and other modes, which are not limited by the present application.

[0083] The fish school state recognition method provided by the application extracts space-time information from a to-be-recognized video containing fish school state information, and recognizes the fish school state. The output results of the time flow and space flow networks are weighted and fused, and the motion and distribution information of the fish school in space and time is comprehensively utilized to obtain better state recognition effect, so that the fish school state can be accurately recognized without contact and at low cost. The problems of low efficiency and non-objective recognition result caused by manual observation and recording of the fish school state of the cultured fish in the current aquaculture industry can be effectively solved.

[0084] Optionally, according to the fish school state recognition method provided by the application, the space information network layer comprises a space feature extraction layer and a space feature recognition layer; and the time information network layer comprises a time feature extraction layer and a time feature recognition layer.

[0085] The space information network layer is used to obtain the space flow feature of the to-be-recognized video according to the RGB image.

[0086] The space feature recognition layer is used to determine the space fish school state recognition result according to the space flow feature of the to-be-recognized video.

[0087] The time feature extraction layer is used to obtain the time flow feature of the to-be-recognized video according to the optical flow image.

[0088] The time feature recognition layer is used to determine the time fish school state recognition result according to the time flow feature of the to-be-recognized video.

[0089] Specifically, the space information network layer comprises a space feature extraction layer and a space feature recognition layer; and the time information network layer comprises a time feature extraction layer and a time feature recognition layer.

[0090] The space information network layer is used to extract the feature of the input RGB image, and obtain the space flow feature of the to-be-recognized video. The space feature recognition layer is used to perform feature recognition according to the obtained space flow feature of the to-be-recognized video, and determine the space fish school state recognition result.

[0091] The time feature extraction layer is used to extract the feature of the input optical flow image, and obtain the time flow feature of the to-be-recognized video. The time feature recognition layer is used to perform feature recognition according to the obtained time flow feature of the to-be-recognized video, and determine the time fish school state recognition result.

[0092] The fish school state recognition method provided by the application determines an RGB image and an optical flow image from a to-be-recognized video containing fish school state information, extracts spatio-temporal information according to a spatial feature extraction layer and a temporal feature extraction layer, and uses a spatial feature recognition layer and a temporal feature recognition layer respectively to recognize the fish school state. The motion and distribution information of the fish school in space and time are comprehensively utilized to obtain better state recognition effect, the accurate recognition of the fish school state can be realized without contact and at low cost, and the problems of low efficiency and non-objective recognition result caused by manual observation and recording of the fish school state are solved.

[0093] According to the fish school state recognition method provided by the application, before the step of inputting the to-be-recognized image set into the fish school state recognition double-flow neural network and determining the target fish school state recognition result, the method further includes: determining a sample image set based on a sample video.

[0094] The step of determining the sample image set based on the sample video specifically includes:

[0095] The sample video is processed to obtain a continuous video frame image set.

[0096] Based on the continuous video frame image set, an RGB image set is determined.

[0097] An optical flow displacement field is added between every several continuous video frames in the continuous video frame image set to obtain an optical flow image.

[0098] Based on the optical flow image, an optical flow image set is determined.

[0099] Based on the RGB image set and the optical flow image set, a sample fish school state label is labeled, and a sample image set is determined.

[0100] Specifically, Figure 3 is a flow diagram of the fish school state recognition method provided by the application, as Figure 3 shown, before the step of inputting the to-be-recognized image set into the fish school state recognition double-flow neural network and determining the target fish school state recognition result, the method further includes: determining a sample image set based on a sample video, and the sample image set is used for training and testing of the fish school state recognition double-flow neural network.

[0101] The step of determining the sample image set based on the sample video specifically includes:

[0102] A large number of sample videos containing different fish school states are obtained, and the sample videos are processed to obtain a continuous video frame image set.

[0103] Based on the continuous video frame image set, an RGB image set is determined, which is used to represent spatial information.

[0104] It should be noted that the video frame set contains continuous video frames, and when determining the RGB image set, a certain number of video frames can be selected according to a preset extraction rule (random extraction, selection of a fixed number of video frames, and further extraction of video key frames based on inter-frame difference method) to form the RGB image set. The present application does not limit this.

[0105] The optical flow displacement field is added between every few continuous video frames in the continuous video frame set to obtain an optical flow image, and an optical flow image set is determined based on the obtained optical flow image. The optical flow feature map is used to represent the time information.

[0106] It should be noted that when the optical flow image is determined based on the continuous video frames, the number of continuous video frames used to generate each optical flow image can be set according to actual conditions, and the present application does not limit this.

[0107] It can be understood that when the RGB image set and the optical flow image set are determined according to the selected RGB image and the optical flow image, the color feature map and the optical flow image can be preprocessed. For example: adjusting the image size, rotation transformation, brightness contrast transformation, image enhancement processing and denoising, etc. The determined sample image set meets the network pre-input size requirement, and the diversity of sample data is increased.

[0108] Similarly, when the size of the sample data of the network is determined, the image to be recognized also needs to be processed to the same size when using the trained network.

[0109] Taking the obtained color feature map as an example, the preprocessing step is illustrated. For example: adjusting the size of the RGB image and the optical flow image, adjusting the size of the RGB image to 224*224*3, and adjusting the size of the optical flow image to 224*224*2. Rotate the adjusted image, randomly rotate the original image by a certain angle, and save the image before and after rotation. The number of data sets can be expanded to twice that before rotation. Perform brightness contrast transformation on the image after rotation transformation, randomly select one-half of the images of each type (fish school state category) for brightness contrast transformation, and save the images before and after transformation to further expand the sample data set.

[0110] It can be understood that since the number of sample images of different fish school states in the obtained image may be too large, part of the sample data can also be expanded to ensure the balance of the data. The specific expansion method can be selected according to actual needs, and the present application does not limit this.

[0111] It should be noted that the above method of preprocessing the obtained RGB image and the optical flow image is only used as a specific example to illustrate the present application. In actual application of the present application, the specific preprocessing method can be adjusted according to the actual situation, and the present application does not limit this.

[0112] Based on the RGB image set and the optical flow image set, real labels corresponding to the fish school state (for example, storing different categories of pictures in corresponding folders) are added to the samples to determine the sample image set.

[0113] It should be noted that the specific corresponding proportional relationship between the number of images in the RGB image set and the optical flow image set can be selected according to the actual situation, and the present embodiment does not limit this.

[0114] In addition, in order to further enrich the time flow features obtained by the network and better capture the motion information of the target in the video segment, improve the accuracy of behavior recognition, one RGB image can correspond to multiple optical flow images in each part of the video, and the specific corresponding proportional relationship can be selected according to the actual situation, and the present embodiment does not limit this.

[0115] It can be understood that when training the neural network, the sample image set can be divided into a training set and a test set (for example, the sample image set is divided into a training set and a test set according to a ratio of 8:2), after the network is trained using the training set, the recognition ability of the trained network is verified using the test set. The test set is input into the trained fish school state recognition dual-flow neural network, and the test error is calculated according to the output of the fish school state recognition dual-flow neural network. If the test error is within the error tolerance range, it indicates that the fish school state recognition dual-flow neural network is successfully trained.

[0116] The fish school state recognition method provided by the present application obtains a sample image set from a sample video, encodes and labels the fish school state categories corresponding to the sample image set using a one-hot encoding method, and trains a fish school state recognition dual-flow neural network using the labeled sample image set. The to-be-identified image set is input into the trained fish school state recognition dual-flow neural network, the spatio-temporal information of the fish school state is extracted, and the fish school state recognition is performed. The motion and distribution information of the fish school in space and time are comprehensively utilized to obtain better state recognition effect, which can realize accurate recognition of the fish school state without contact and at low cost, and solve the problems of low efficiency and non-objective recognition results caused by manual observation and recording of the fish school state.

[0117] Optionally, according to the fish school state recognition method provided by the present application, before the step of inputting the to-be-identified image set into the fish school state recognition dual-flow neural network and determining the target fish school state recognition result, the fish school state recognition dual-flow neural network is trained.

[0118] The training of the fish population state recognition double-flow neural network specifically comprises:

[0119] Based on the sample video, a sample image set is determined; wherein the sample video comprises image information of a sample fish population; and the sample image set comprises an RGB image and an optical flow image.

[0120] The fish population state recognition double-flow neural network is trained by using the sample image set.

[0121] Based on the target loss function, the network parameters of the fish population state recognition double-flow neural network are updated by using a gradient descent method; wherein the target loss function is determined according to the preset hyperparameters, preset loss function weights, a relative entropy loss function and a cross-entropy loss function; the hyperparameters comprise a time hyperparameter and a space hyperparameter; the time hyperparameter is used to correct the time flow class information in the fish population state recognition double-flow neural network; and the space hyperparameter is used to correct the space flow class information in the fish population state recognition double-flow neural network.

[0122] Specifically, as shown in Figure 3 Before the step of inputting the image set to be recognized into the fish population state recognition double-flow neural network and determining the target fish population state recognition result, the network needs to be trained.

[0123] Based on a sample video comprising image information of a sample fish population, a sample image set comprising an RGB image and an optical flow image is determined. The fish population state recognition double-flow neural network is trained by using the sample image set.

[0124] Based on the idea of knowledge distillation, during the training of the neural network, the hyperparameters (time hyperparameters and space hyperparameters) are set in advance. The time hyperparameters are used to correct the time flow class information in the fish population state recognition double-flow neural network, and the space hyperparameters are used to correct the space flow class information in the fish population state recognition double-flow neural network. During the recognition process, the one-hot encoding of the real label corresponding to the recognition result (for example, the normal state of the fish population is marked as 1000, the initial abnormal state is marked as 0100, the late abnormal state is marked as 0010, and the weak feeding degree is marked as 0001) and the probability distribution predicted by the two networks are weighted and fused, which to some extent corrects the class information obtained by the time flow network and the space flow network during the training process by using the real label, and improves the recognition accuracy.

[0125] Based on the preset hyperparameters, preset loss function weights, a relative entropy loss function (i.e., a Kullback Leibler Divergence Loss) and a cross-entropy loss function, a target loss function is determined.

[0126] The target loss function is used for knowledge transmission during training, and class information learned by the time flow network (time domain) is transmitted to the spatial flow network, and class information learned by the spatial flow network (space domain) is transmitted to the time flow network, so as to strengthen the extraction of feature information, realize joint training of the time domain network and the space domain network, improve the recognition accuracy of the survival state of the fish group, and lay a foundation for timely detection of the abnormal state of the fish and the state of poor feeding in the breeding process, and subsequent management and decision-making.

[0127] It should be noted that the specific formula of the target loss function can be set according to actual conditions, and the present application does not limit this.

[0128] Based on the set target loss function, the network parameters of the fish group state recognition double-flow neural network are updated using the gradient descent method, and the fish group state recognition double-flow neural network is iteratively trained based on the updated network parameters until the fish group state recognition double-flow neural network converges.

[0129] It should be noted that the gradient descent method includes: batch gradient descent (Batch Gradient Descent, BGD for short), stochastic gradient descent (Stochastic Gradient Descent, SGD for short) and mini-batch gradient descent (Mini-Batch Gradient Descent, MBGD for short). In the actual application process of the present application, the specific type of gradient descent method used can be adjusted according to actual conditions, and the present application does not limit this.

[0130] Optionally, according to the fish group state recognition method provided by the present application, the target loss function Loss is:

[0131] Loss=KL(Ps,Pt′)*α*T*T+CE(Ps,G)*(1-2α)*0.5+CE(Pt,G)*(1-2α)*0.5+KL(Ps′,Pt)*α*T*T

[0132] Wherein, KL is the relative entropy loss function; CE is the cross-entropy loss function; Ps is the time flow class information; Ps' is the corrected time flow class information; Pt is the spatial flow class information; Pt' is the corrected spatial flow class information; α is the weight of the relative entropy loss function; (1-2α) is the weight of the cross-entropy loss function; G is the real class information of the sample fish group; T is the distillation parameter.

[0133] Specifically, the method for correcting the inter-class information learned by the time flow and spatial flow networks during training, the information of the real label is represented by One-Hot encoding (One-Hot encoding), and is represented as G.

[0134] The adjustment degree of the time flow and the space flow network obtaining inter-class information is adjusted by setting the hyperparameters a and b respectively. The information of the real label represented by one-hot encoding is multiplied by a and b respectively, and then the two vectors obtained are added to the prediction probability distribution of the time flow and the space flow network respectively, so that the modified class information is obtained.

[0135] The target loss function Loss for transmitting the class information between two networks is designed as:

[0136] Loss=KL(Ps, Pt') * a * T * T + CE(Ps, G) * (1-2a) * 0.5 + CE(Pt, G) * (1-2a) * 0.5 + KL(Ps', Pt) * a * T * T

[0137] Wherein, KL is the relative entropy loss function; CE is the cross-entropy loss function; Ps is the time flow class information; Ps' is the modified time flow class information; Ps' = G * a + Ps; Pt is the space flow class information; Pt' is the modified space flow class information; Pt' = G * b + Pt; a is the weight of the relative entropy loss function; (1-2a) is the weight of the cross-entropy loss function; G is the sample fish swarm real class information; T is the distillation parameter.

[0138] The relative entropy loss function is

[0139] Wherein, i represents the label class, n represents the total number of label classes, P represents the real distribution, Q represents the fitting distribution of P, and pi and qi are two discrete probability distributions when the label class is i.

[0140] The cross-entropy loss function is

[0141] Wherein, i represents the label class, n represents the total number of label classes. y is the label, when the label class is i, y i = 1, otherwise y i = 0. p i is the neural network output when the label class is i.

[0142] It should be noted that in the present application, the specific values of the hyperparameters and the distillation parameters can be adjusted according to actual conditions, and the present application does not limit this.

[0143] The fish population state recognition system provided by the application recognizes fish population state by extracting space-time information from a to-be-recognized video containing fish population state information. The motion and distribution information of fish population in space and time is comprehensively utilized to obtain better state recognition effect, and accurate recognition of fish population state can be realized without contact and at low cost. According to the knowledge distillation idea, hyperparameters are introduced, the prediction probability of the time stream and the space stream is respectively corrected according to the preset hyperparameters, and the information contained in the time stream and the space stream prediction probability distribution corrected according to the preset hyperparameters is transmitted to the time stream network and the space stream network by using a target loss function, so that the network recognition rate can be effectively improved. The problems of low efficiency and non-objective recognition result caused by manual observation and recording of fish population state are solved.

[0144] Optionally, according to the fish population state recognition method provided by the application, the network parameters of the fish population state recognition double-flow neural network are updated using the gradient descent method based on the target loss function, and the fish population state recognition double-flow neural network is iteratively trained based on the updated network parameters until the fish population state recognition double-flow neural network converges, and specifically includes:

[0145] The network parameters of the fish population state recognition double-flow neural network are updated using the small-batch gradient descent method based on the target loss function, the learning rate is adjusted according to the cosine annealing strategy, and the fish population state recognition double-flow neural network is iteratively trained based on the updated network parameters until the fish population state recognition double-flow neural network converges.

[0146] Specifically, the network parameters of the fish population state recognition double-flow neural network are updated using the gradient descent method based on the target loss function, and the fish population state recognition double-flow neural network is iteratively trained based on the updated network parameters until the fish population state recognition double-flow neural network converges, and specifically includes:

[0147] The RGB images and the optical flow image sequences in the training set are respectively trained by the improved space network and the time network of the double-flow convolutional neural network, and the small-batch gradient descent method is used in the training of the two networks. The neural network parameters are updated according to the target loss function.

[0148] During training, the learning rate (Learning Rate) is adjusted according to the cosine annealing strategy, and the coefficient of the partial derivative in the gradient descent method is optimized through the learning rate to speed up the convergence speed of the neural network.

[0149] When the batch stochastic gradient descent algorithm is used, the neural network should be closer and closer to the global minimum value of the loss value, and the learning rate should be reduced, so that the model can be as close as possible to the global minimum value without overshooting.

[0150] Cosine annealing uses a cosine function to reduce the learning rate. The cosine annealing learning rate is different from the traditional learning rate. With the increase of the epoch, the learning rate first rapidly decreases, then sharply increases, and then repeatedly repeats this process. The purpose of such violent fluctuations is to escape the current local optimum point and ensure that the final determined optimum point is the global optimum point.

[0151] For example, during training, 64 samples are taken as a small batch block, the initial learning rate is set to 0.0001, the learning rate is adjusted by cosine annealing, the learning rate decay period is set to 14 epochs, the minimum value of the learning rate decay is 0.000005, the training is ended when the loss of the training no longer decreases, and the weight file of the network model with the highest recognition accuracy in the training process is saved.

[0152] It should be noted that the above training method is only used as a specific example to illustrate the present application, and in actual application, the specific values of the parameters can be adjusted according to actual needs, and the present application does not limit this.

[0153] The fish school state recognition method provided by the present application obtains a sample image set from a sample video, encodes and labels the fish school state category corresponding to the sample image set with a one-hot code, and uses the labeled sample image set to train a fish school state recognition double-flow neural network. During training, the network parameters are updated according to the small batch gradient descent method, and the learning rate is adjusted using the cosine annealing strategy. The learning rate is used to optimize the coefficient of the partial derivative in the gradient descent method, and the convergence speed of the neural network is accelerated. The to-be-recognized image set is input into the trained fish school state recognition double-flow neural network, the spatio-temporal information of the fish school state is extracted, and the fish school state recognition is performed. The motion and distribution information of the fish school in space and time are comprehensively utilized to obtain better state recognition effect, which can realize accurate recognition of the fish school state without contact and at low cost, and solve the problems of low efficiency and non-objective recognition result caused by manual observation and recording of the fish school state.

[0154] Figure 4 is a structural schematic diagram of the fish school state recognition system provided by the present application, as Figure 4 shown, the present application also provides a fish school state recognition system, which comprises an input determination unit 410 and a state recognition unit 420.

[0155] The input determination unit 410 is used for determining a to-be-recognized image set based on a to-be-recognized video; wherein the to-be-recognized video comprises image information of a to-be-recognized fish school; the to-be-recognized image set comprises an RGB image and an optical flow image; the RGB image is an image representing spatial information; and the optical flow image is an image representing temporal information.

[0156] The state recognition unit 420 is configured to input the to-be-recognized image set into a fish school state recognition double-flow neural network to determine a target fish school state recognition result.

[0157] The fish school state recognition double-flow neural network is trained based on a labeled sample image set determined according to a sample video; and the target loss function is used to deliver time flow and space flow prediction probabilities that are corrected according to preset hyperparameters.

[0158] Specifically, the present application is described by taking the identification of different states of a fish school in a video clip obtained from a land-based factory farming workshop as an example. First, a to-be-recognized video containing image information of a to-be-recognized fish school is obtained, which is shot in a land-based factory farming workshop.

[0159] It can be understood that, in order to more accurately identify the state of the fish school, the video of the fish school needs to contain image information of the distribution of the fish school on the surface of the water (horizontal direction distribution) when the video is obtained.

[0160] It should be noted that, the to-be-recognized video can be obtained based on a camera pre-installed in the land-based factory, can be shot by a staff, or can be obtained based on other methods, and the present application does not limit the source of the to-be-recognized video. In addition, the present application can be adapted to the state recognition of fish schools farmed in land-based factories, and can also be adapted to changes in sample data used for training the network and input recognition data, so that the present application can be adapted to the state recognition of other fish schools or other groups with clustering behavior, and the present application does not limit this.

[0161] The input determination unit 410 is configured to pre-process the to-be-recognized video based on the to-be-recognized video containing image information of the to-be-recognized fish school to obtain a to-be-recognized image set. The to-be-recognized image set includes an RGB image and an optical flow image; the RGB image is an image representing space information; the optical flow image is an image representing time information and can capture motion information, so that the network can focus on moving objects in the video. It should be noted that, the method of pre-processing the to-be-recognized video to obtain the RGB image and the optical flow image can be to pre-process the video clip into continuous RGB video frames by using a code or a video capture tool, and add an optical flow displacement field between consecutive video frames to obtain optical flow information and an optical flow image. The to-be-recognized image set is composed of the obtained RGB video frames (RGB images) and the optical flow image. This method is only a specific example for explaining the present application, and the method of determining the to-be-recognized image set is not limited by the present application.

[0162] The to-be-identified image set can include all the processed RGB video frames, or a certain number of video frames can be selected according to preset extraction rules (random extraction, selection of a fixed number of video frames, and further extraction of key video frames based on inter-frame difference method). The proportional relationship between the space color image and the optical flow image in the to-be-identified image set can be selected according to actual conditions, and the application does not limit this.

[0163] The state recognition unit 420 is configured to input the to-be-identified image set into the fish school state recognition dual-flow neural network. The dual-flow neural network includes a time flow and a space flow. The time flow processes the optical flow image, and the space flow processes the RGB image. According to the extracted time and space features in the to-be-identified video, the target fish school state recognition result is determined.

[0164] It should be noted that before using the trained fish school state recognition dual-flow neural network, the fish school state recognition dual-flow neural network needs to be trained using the sample image set with labels determined according to the sample video.

[0165] Before training the network based on the sample image set, the state of the fish school in the sample image set needs to be labeled. For example, the state of the fish school is divided into: normal state of the fish school, early abnormal state, late abnormal state, weak feeding degree, general feeding degree and severe feeding degree. The sample data with the same fish school state are divided together, and the corresponding state category label is marked.

[0166] It should be noted that the state division of the fish school can be adjusted according to actual needs, and secondly, the method of adding labels can be selected according to actual conditions, and the application does not limit this.

[0167] The fish school state recognition dual-flow neural network can be regarded as a time flow network and a space flow network. During training, according to the knowledge distillation idea, a hyperparameter is introduced, the prediction probability of the time flow and the space flow is respectively corrected according to the preset hyperparameter, and the information contained in the time flow and the space flow prediction probability distribution corrected according to the preset hyperparameter is transmitted to each other by using a target loss function, so that the joint training mode can effectively improve the recognition rate of the network.

[0168] It can be understood that with the gradual deepening or widening of the network structure, the network performance has been greatly improved, but at the same time, the calculation amount is greatly increased, the network parameters and the forward calculation time are increased, for example, VGGNet, ResNet and DenseNet, etc. It is difficult to achieve real-time operation effect by relying on these networks. Therefore, the application can use a lighter neural network SqueezeNet as a backbone network, which can effectively reduce the parameters participating in operation, has a faster detection speed, and can better meet the needs of actual industrial applications.

[0169] The fish school state recognition system provided by the application recognizes fish school state by extracting space-time information from a to-be-recognized video containing fish school state information. The motion and distribution information of fish school in space and time are comprehensively utilized to obtain better state recognition effect, and the accurate recognition of fish school state can be realized without contact and at low cost. According to the knowledge distillation idea, the hyperparameters are introduced, the prediction probability of the time stream and the space stream is respectively corrected according to the preset hyperparameters, and the information contained in the time stream and the space stream prediction probability distribution corrected according to the preset hyperparameters is transmitted to the time stream network and the space stream network by using the target loss function, so that the joint training mode can effectively improve the recognition rate of the network. The problems of low efficiency and non-objective recognition result caused by manual observation and recording of fish school state are solved.

[0170] It should be noted that the fish school state recognition system provided by the application is used to execute the fish school state recognition method described above, and the specific implementation manner is consistent with the method implementation manner, which will not be described here.

[0171] Figure 5 is a schematic diagram of an entity structure of an electronic device provided by the application, as shown in Figure 5 The electronic device can include a processor 510, a communication interface 511, a memory 512 and a bus 513, wherein the processor 510, the communication interface 511 and the memory 512 can communicate with each other through the bus 513. The processor 510 can call the logical instructions in the memory 512 to execute the following method: based on a to-be-recognized video, a to-be-recognized image set is determined; wherein the to-be-recognized video includes image information of a to-be-recognized fish school; the to-be-recognized image set includes an RGB image and an optical flow image; the RGB image is an image representing space information; the optical flow image is an image representing time information; the to-be-recognized image set is input into a fish school state recognition double-flow neural network to determine a target fish school state recognition result; wherein the fish school state recognition double-flow neural network is trained based on a labeled sample image set determined according to a sample video; and when the fish school state recognition double-flow neural network is trained, a target loss function is used to transmit the time stream and the space stream prediction probability corrected according to the preset hyperparameters.

[0172] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer power supply screen (which can be a personal computer, a server, or a network power supply screen) to execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0173] Further, the present application discloses a computer program product, the computer program product comprises a computer program stored on a non-transitory computer readable storage medium, the computer program comprises program instructions, when the program instructions are executed by a computer, the computer can execute the fish school state recognition method provided by the above-mentioned method embodiments, for example, comprising: determining a to-be-recognized image set based on a to-be-recognized video; wherein the to-be-recognized video comprises image information of a to-be-recognized fish school; the to-be-recognized image set comprises: an RGB image and an optical flow image; the RGB image is an image representing spatial information; the optical flow image is an image representing time information; inputting the to-be-recognized image set into a fish school state recognition double-flow neural network to determine a target fish school state recognition result; wherein the fish school state recognition double-flow neural network is trained based on a labeled sample image set determined according to a sample video; when the fish school state recognition double-flow neural network is trained, a target loss function is used to transmit time flow and spatial flow prediction probabilities corrected according to preset hyperparameters.

[0174] In another aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a fish school state recognition method provided by any of the above embodiments, for example comprising: determining a to-be-recognized image set based on a to-be-recognized video; wherein the to-be-recognized video comprises image information of a to-be-recognized fish school; the to-be-recognized image set comprises an RGB image and an optical flow image; the RGB image is an image representing spatial information; the optical flow image is an image representing temporal information; inputting the to-be-recognized image set into a fish school state recognition dual-flow neural network to determine a target fish school state recognition result; wherein the fish school state recognition dual-flow neural network is trained based on a labeled sample image set determined according to a sample video; and when training the fish school state recognition dual-flow neural network, a target loss function is used to transmit temporal flow and spatial flow prediction probabilities modified according to preset hyperparameters.

[0175] The system embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0176] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer power supply screen (which can be a personal computer, server, or network power supply screen, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0177] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A fish school state recognition method characterized by comprising: The method comprises the following steps: determining a set of to-be-identified images based on a to-be-identified video, wherein the to-be-identified video comprises image information of a to-be-identified fish school; the set of to-be-identified images comprises an RGB image and an optical flow image; the RGB image is an image representing spatial information; the optical flow image is an image representing temporal information; inputting the set of to-be-identified images into a fish school state identification dual-flow neural network to determine a target fish school state identification result; wherein the fish school state identification dual-flow neural network is trained based on a set of labeled sample images determined according to a sample video; when the fish school state identification dual-flow neural network is trained, a target loss function is used to transfer temporal flow and spatial flow prediction probabilities that are corrected according to preset hyperparameters; before the step of inputting the set of to-be-identified images into the fish school state identification dual-flow neural network to determine the target fish school state identification result, the method further comprises training the fish school state identification dual-flow neural network; the training of the fish school state identification dual-flow neural network specifically comprises the following steps: determining the set of sample images based on the sample video, wherein the sample video comprises image information of a sample fish school; the set of sample images comprises an RGB image and an optical flow image; training the fish school state identification dual-flow neural network using the set of sample images; updating network parameters of the fish school state identification dual-flow neural network using a gradient descent method based on the target loss function, and iteratively training the fish school state identification dual-flow neural network based on the updated network parameters until the fish school state identification dual-flow neural network converges; wherein the target loss function is determined according to the preset hyperparameters, preset loss function weights, a relative entropy loss function, and a cross-entropy loss function; the hyperparameters comprise a temporal hyperparameter and a spatial hyperparameter; the temporal hyperparameter is used to correct temporal flow class information in the fish school state identification dual-flow neural network; the spatial hyperparameter is used to correct spatial flow class information in the fish school state identification dual-flow neural network. The target loss function is: ； wherein, is a cross-entropy loss function; is a cross-entropy loss function; is temporal flow class information; is modified temporal flow class information; is spatial flow class information; is modified spatial flow class information; is a weight for the relative entropy loss function; is a weight for the cross-entropy loss function; is sample fish swarm real class information; is a distillation parameter.

2. The fish school state identification method according to claim 1, wherein the fish school state identification dual-flow neural network comprises a spatial information network layer, a temporal information network layer, and an identification result fusion layer; the spatial information network layer is used to determine a spatial fish school state identification result of the to-be-identified video based on the RGB image; the temporal information network layer is used to determine a temporal fish school state identification result of the to-be-identified video based on the optical flow image; the identification result fusion layer is used to determine the target fish school state identification result based on the spatial fish school state identification result and the temporal fish school state identification result by weighting fusion according to preset weights.

3. The fish school state identification method according to claim 2, wherein the spatial information network layer comprises a spatial feature extraction layer and a spatial feature identification layer; and the temporal information network layer comprises a temporal feature extraction layer and a temporal feature identification layer; the spatial information network layer is used to obtain spatial flow features of the to-be-identified video based on the RGB image; The spatial feature recognition layer is configured to determine the spatial school state recognition result according to the spatial flow feature of the to-be-recognized video. The time feature extraction layer is configured to obtain the time flow feature of the to-be-recognized video according to the optical flow image. The time feature recognition layer is configured to determine the time school state recognition result according to the time flow feature of the to-be-recognized video.

4. The fish school state recognition method according to claim 1, characterized by, Before the step of inputting the to-be-recognized image set into the school state recognition double-flow neural network and determining the target school state recognition result, the method further includes: determining a sample image set based on a sample video. The method of determining the sample image set based on the sample video specifically includes: processing the sample video to obtain a set of continuous video frame images; determining an RGB image set based on the set of continuous video frame images; adding an optical flow displacement field between every few continuous video frames in the set of continuous video frame images to obtain an optical flow image; determining an optical flow image set based on the optical flow image; annotating sample school state labels in the training set and the test set according to a certain proportion based on the RGB image set and the optical flow image set, and determining the sample image set.

5. The fish school state recognition method according to claim 1, characterized by, The method of updating the network parameters of the school state recognition double-flow neural network based on the target loss function using the gradient descent method and iteratively training the school state recognition double-flow neural network based on the updated network parameters until the school state recognition double-flow neural network converges specifically includes: The method of updating the network parameters of the school state recognition double-flow neural network based on the target loss function using the mini-batch gradient descent method, adjusting the learning rate according to the cosine annealing strategy, and iteratively training the school state recognition double-flow neural network based on the updated network parameters until the school state recognition double-flow neural network converges.

6. A fish school state recognition system characterized by comprising: The method includes: an input determination unit and a state recognition unit; The input determination unit is configured to determine a to-be-recognized image set based on a to-be-recognized video; the to-be-recognized video includes image information of a to-be-recognized school of fish; the to-be-recognized image set includes an RGB image and an optical flow image; the RGB image is an image representing spatial information; and the optical flow image is an image representing temporal information. The state recognition unit is configured to input the to-be-recognized image set into a school state recognition double-flow neural network to determine a target school state recognition result. The school state recognition double-flow neural network is trained based on a labeled sample image set determined according to a sample video; and a target loss function is used to transfer a time flow and a spatial flow prediction probability that is corrected according to a preset hyperparameter during training of the school state recognition double-flow neural network. Before the step of inputting the to-be-recognized image set into the school state recognition double-flow neural network and determining the target school state recognition result, the state recognition unit is further configured to train the school state recognition double-flow neural network. The method of training the school state recognition double-flow neural network specifically includes: determining a sample image set based on the sample video; the sample video includes image information of a sample school of fish; and the sample image set includes an RGB image and an optical flow image. training the fish school state recognition double-flow neural network by using the sample image set; updating network parameters of the fish school state recognition double-flow neural network by using a gradient descent method based on the target loss function, and iteratively training the fish school state recognition double-flow neural network based on the updated network parameters until the fish school state recognition double-flow neural network converges; wherein the target loss function is determined according to the preset hyperparameters, preset loss function weights, a relative entropy loss function, and a cross-entropy loss function; the hyperparameters include a time hyperparameter and a space hyperparameter; the time hyperparameter is used to correct time flow category information in the fish school state recognition double-flow neural network; and the space hyperparameter is used to correct space flow category information in the fish school state recognition double-flow neural network; The target loss function is: ； wherein, is a relative entropy loss function; is a cross-entropy loss function; is time flow class information; is modified time flow class information; is spatial flow class information; is modified spatial flow class information; is a weight of the relative entropy loss function; is a weight of the cross-entropy loss function; is sample fish swarm real class information; is a distillation parameter.

7. An electronic device, comprising: The fish school state recognition method comprises a memory and a processor, the processor and the memory are in communication with each other through a bus; the memory stores program instructions executable by the processor, and the processor calling the program instructions can execute the fish school state recognition method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the fish school state recognition method according to any one of claims 1 to 5.

Images