Fish abnormal behavior diagnosis method and device
Through video image processing and autoencoder technology, abnormal behavior of fish is identified and analyzed, and the problems of traditional manual observation are solved, and efficient and accurate monitoring of abnormal behavior of fish is achieved.
Patent Information
- Application Number
- CN202411351754.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2024-12-27
AI Technical Summary
Traditional artificial on-site observation of abnormal behavior of fish has problems with low diagnostic efficiency and poor recognition accuracy. It is greatly affected by human subjective factors and has limited observation range and time.
By collecting the original video images of fish schools, using the object detection model for fish identification and prediction box annotation, calculate the correlation between prediction boxes, determine the fish movement trajectory, and perform feature transformation through the autoencoder to identify abnormal behavior.
It improves the efficiency and accuracy of fish abnormal behavior diagnosis, reduces manual intervention, expands the observation range and time, and improves the level of diagnosis automation.
Smart Images
Figure CN119206583A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of biological behavior monitoring, and in particular to a method and device for diagnosing abnormal behavior of fish. Background Art
[0002] In aquaculture, fish are very sensitive to changes and stimuli in the external environment, such as changes in water salinity, changes in dissolved oxygen content, changes in water temperature, and changes in environmental light intensity and different light colors. An inappropriate water environment will affect the normal growth and development of fish, and will be accompanied by some stress behavior reactions. If these abnormal behaviors are not discovered in time and effective measures are not taken, it will often lead to fish diseases and even large-scale deaths.
[0003] The current traditional approach is manual on-site observation, but manual methods are time-consuming and labor-intensive, not only greatly affected by human subjective factors, but also subject to many restrictions on the scope and time of observation, resulting in low diagnostic efficiency and poor recognition accuracy.
[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention
[0005] The main purpose of the present invention is to provide a method and device for diagnosing abnormal behavior of fish, aiming to solve the problem that the current traditional practice is manual on-site observation, but the manual method is time-consuming and labor-intensive, not only greatly affected by human subjective factors, but also the observation range and observation time are subject to many restrictions, resulting in low diagnostic efficiency and poor recognition accuracy.
[0006] To achieve the above object, the present invention provides a method for diagnosing abnormal behavior of fish, the method comprising the following steps:
[0007] Collecting original video images of the fish school to be identified;
[0008] Performing fish identification on each frame of the original video image through the target detection model, and marking a corresponding prediction box for each frame of the image based on the identified fish;
[0009] Determine the correlation between each prediction box in the plurality of prediction boxes;
[0010] Determine the movement trajectory of each fish according to the correlation degree;
[0011] Performing feature transformation on the motion trajectory of each of the fishes through an autoencoder;
[0012] Identify abnormal behaviors of individual fish species based on the transformed features.
[0013] In some embodiments, determining the degree of association between each prediction box in the plurality of prediction boxes includes:
[0014] Filter out overlapping prediction boxes from multiple prediction boxes;
[0015] Randomly select two prediction boxes from the overlapping prediction boxes, and determine the area of the overlapping area and the area of the covered area of the two prediction boxes respectively;
[0016] The intersection-over-union ratio of the two prediction frames is calculated according to the area of the overlapping area and the area of the covered area. The calculation formula is: Where x and y are any two selected prediction boxes, x∩y is the area of the overlapping area of the two prediction boxes, and x∪y is the area of the covered area of the two prediction boxes;
[0017] The correlation between the overlapping prediction frames is determined according to the IoU ratio, and the IoU ratio is positively correlated with the correlation.
[0018] In some embodiments, the method further comprises:
[0019] Randomly select two prediction boxes from the remaining non-overlapping prediction boxes, and determine the center point distance between the two prediction boxes and the diagonal distance of one of the prediction boxes;
[0020] The correlation between the two prediction frames is calculated based on the center point distance and the diagonal distance in combination with the preset coefficient. The calculation formula is: Among them, d is the center point distance, l is the diagonal distance, and α is the preset coefficient.
[0021] In some embodiments, determining the movement trajectory of each fish according to the correlation degree includes:
[0022] Constructing a correlation matrix according to the correlation, wherein the size of the correlation matrix corresponds to the number of prediction frames of two consecutive frames of images;
[0023] Extracting the maximum element from the correlation matrix, and excluding the row and column where the maximum element is located, and the elements in the excluded row and column do not participate in the selection of the maximum element;
[0024] Extract the maximum element again from the remaining elements, and exclude the row and column where the maximum element is extracted again;
[0025] Return to the step of extracting the maximum element from the remaining elements again and excluding the row and column where the maximum element extracted again is located, until there are no selectable elements in the association matrix;
[0026] The corresponding prediction box is determined according to all the extracted elements to obtain the movement trajectory of each fish.
[0027] In some embodiments, the autoencoder includes an encoder and a decoder, and the feature transformation of the motion trajectory of each fish by the autoencoder includes:
[0028] Input the movement trajectory of each fish into the autoencoder;
[0029] Mapping the running trajectory to a low-dimensional space through an encoder network corresponding to the encoder;
[0030] Obtaining a low-dimensional feature representation of the motion trajectory in the low-dimensional space through a bottleneck layer of the autoencoder, wherein the bottleneck layer connects the output of the encoder and the input of the decoder;
[0031] A decoder network corresponding to the decoder is used to extract multiple trajectory features corresponding to any fish trajectory from the low-dimensional feature representation.
[0032] In some embodiments, the encoder is used to encode the original representation of the data to obtain a hidden layer representation, and the decoder is used to extract the feature representation of the original representation based on the hidden layer representation. The expression is B=Encoder(A), A'=Decoder(B), where A is the original representation, B is the encoded hidden layer representation, and A' is the feature representation of the original representation. Both the encoder and the decoder adopt a GRU network structure.
[0033] In some embodiments, the identifying abnormal behaviors of each fish based on the transformed features includes:
[0034] Determine the distances between a plurality of trajectory features corresponding to any fish trajectory and preset sample features;
[0035] Determine a plurality of trajectory features corresponding to the distance being less than or equal to a preset distance, and determine the number of corresponding features to be a first number, and determine a plurality of trajectory features corresponding to the distance being greater than the preset distance, and determine the number of corresponding features to be a second number;
[0036] Calculating a trajectory abnormality ratio according to the first number and the second number;
[0037] If the trajectory abnormality ratio exceeds a preset ratio, it is determined that the corresponding fish has abnormal behavior.
[0038] In addition, to achieve the above-mentioned purpose, the present invention also proposes a fish abnormal behavior diagnosis device, the fish abnormal behavior diagnosis device comprising:
[0039] An acquisition module, used for acquiring original video images of the fish school to be identified;
[0040] A recognition module, used to identify fish for each frame of the original video image through a target detection model, and to mark a corresponding prediction box for each frame of the image based on the identified fish;
[0041] A calculation module, used to determine the correlation between each prediction box in the plurality of prediction boxes;
[0042] A construction module is used to determine the movement trajectory of each fish according to the correlation degree;
[0043] A transformation module, used for performing feature transformation on the motion trajectory of each fish through an autoencoder;
[0044] The diagnosis module is used to identify abnormal behaviors of individual fish species based on the transformed features.
[0045] In some embodiments, the calculation module is used to filter out overlapping prediction boxes from multiple prediction boxes;
[0046] Randomly select two prediction boxes from the overlapping prediction boxes, and determine the area of the overlapping area and the area of the covered area of the two prediction boxes respectively;
[0047] The intersection-over-union ratio of the two prediction frames is calculated according to the area of the overlapping area and the area of the covered area. The calculation formula is: Where x and y are any two selected prediction boxes, x∩y is the area of the overlapping area of the two prediction boxes, and x∪y is the area of the covered area of the two prediction boxes;
[0048] The correlation between the overlapping prediction frames is determined according to the IoU ratio, and the IoU ratio is positively correlated with the correlation.
[0049] In some embodiments, the calculation module is used to select any two prediction boxes from the remaining prediction boxes that do not overlap, and respectively determine the center point distance between the two prediction boxes and the diagonal distance of one of the prediction boxes;
[0050] The correlation between the two prediction frames is calculated based on the center point distance and the diagonal distance in combination with the preset coefficient. The calculation formula is: Among them, d is the center point distance, l is the diagonal distance, and α is the preset coefficient.
[0051] The present invention collects original video images of fish schools to be identified; identifies fish for each frame of the original video images through a target detection model, and labels each frame with a corresponding prediction frame based on the identified fish; determines the correlation between each prediction frame among a plurality of prediction frames; determines the motion trajectory of each fish according to the correlation; performs feature transformation on the motion trajectory of each fish through an autoencoder; and identifies abnormal behaviors of each fish based on the transformed features. By replacing the manual method with this method, not only the diagnosis efficiency but also the diagnosis accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 It is a flow chart of the first embodiment of the method for diagnosing abnormal behavior of fishes of the present invention;
[0053] Figure 2 This is a structural block diagram of the first embodiment of the device for diagnosing abnormal behavior of fishes according to the present invention.
[0054] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0055] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0056] The embodiment of the present invention provides a method for diagnosing abnormal behavior of fish. Figure 1 , Figure 1 The figure is a flow chart of a first embodiment of a method for diagnosing abnormal behavior of fishes according to the present invention.
[0057] In this embodiment, the method for diagnosing abnormal fish behavior comprises the following steps:
[0058] Step S10: collecting original video images of the school of fish to be identified;
[0059] In this embodiment, the executor of this embodiment is a fish abnormal behavior diagnostic device, wherein the fish abnormal behavior diagnostic device has functions such as data processing, data communication and program running. The fish abnormal behavior diagnostic device can be a computer terminal device or other network device, and of course it can also be other devices with similar functions, and this embodiment does not limit this.
[0060] It should be noted that the current traditional approach is manual on-site observation, but manual methods are time-consuming and labor-intensive. Not only are they greatly affected by human subjective factors, but the observation range and time are also subject to many restrictions, resulting in low diagnostic efficiency and poor recognition accuracy.
[0061] In order to solve the above technical problems, in this embodiment, the original video images of the fish school to be identified are collected; the fish are identified for each frame of the original video image through the target detection model, and the corresponding prediction box is labeled for each frame based on the identified fish; the correlation between each prediction box in a plurality of prediction boxes is determined; the movement trajectory of each fish is determined according to the correlation; the movement trajectory of each fish is transformed by the autoencoder; the abnormal behavior of each fish is identified based on the transformed features. This method replaces the manual method, which not only improves the diagnosis efficiency but also improves the diagnosis accuracy.
[0062] In a specific implementation, in this embodiment, the original video image of the fish school to be identified is first collected. The original video image can be extracted frame by frame from the original video, and the original video image is a collection of multiple frame images.
[0063] Step S20: fish identification is performed on each frame of the original video image through the target detection model, and a corresponding prediction box is marked for each frame based on the identified fish.
[0064] In this embodiment, based on the original video images collected above, the fish in each frame of the image are further identified through the target detection model, wherein the target model can adopt the target detection model of the YOLO series, which is not described in detail in this embodiment. When fish and the like are identified, in this embodiment, corresponding prediction boxes are marked for identifying fish. It can be understood that there are multiple types of fish in each frame of the image, so each frame of the image has several prediction boxes.
[0065] Step S30: Determine the correlation between the prediction boxes in the plurality of prediction boxes.
[0066] It should be noted that determining the correlation between each prediction box in multiple prediction boxes is to determine the correlation between the previous and next two frame images. For example, the current frame image is A, the current frame image A has m prediction boxes, the next frame image of the current frame image A is B, and the next frame image B has n prediction boxes. In this embodiment, what is determined is the correlation between the m prediction boxes and the n prediction boxes. In actual situations, the number of m is the same as that of n.
[0067] The specific process of determining the association degree is to select overlapping prediction boxes from multiple prediction boxes, and then select any two prediction boxes from the overlapping prediction boxes, for example, select prediction box x from m prediction boxes and select prediction box y from n prediction boxes, and then determine the area of the overlapping area and the area of the covered area of the two prediction boxes, and then calculate the intersection and union ratio of the two prediction boxes according to the area of the overlapping area and the area of the covered area. The calculation formula is: Among them, x and y are any two selected prediction boxes, x∩y is the area of the overlapping area of the two prediction boxes, x∪y is the area of the covered area of the two prediction boxes, the intersection and union ratio corresponds to the correlation degree, and the intersection and union ratio is positively correlated with the correlation degree. The larger the intersection and union ratio, the greater the corresponding correlation degree.
[0068] Furthermore, for the remaining prediction frames that do not overlap, two prediction frames are randomly selected in the same manner as described above in this embodiment, and then the center point distance between the two prediction frames and the diagonal distance of one of the prediction frames are determined based on the center points of the two prediction frames. The prediction frame is the prediction frame in the previous frame image, for example, the current frame image is A, and the next frame image is B. The diagonal distance is the diagonal distance of the prediction frame in the current frame image A. Finally, the correlation between the two prediction frames can be calculated based on the center point distance and the diagonal distance combined with the preset coefficient. The calculation formula is: Wherein, d is the center point distance, l is the diagonal distance, and α is a preset coefficient. The preset coefficient can be set to 1.5. Of course, it can also be set according to actual conditions, which is not limited in this embodiment.
[0069] Step S40: determining the movement trajectory of each fish according to the correlation degree.
[0070] After determining the correlation, the corresponding correlation matrix can be constructed in this embodiment. The size of the constructed correlation matrix corresponds to the number of prediction frames of two consecutive frames. For example, the current frame image A has m prediction frames, the next frame image of the current frame image A is B, and the next frame image B has n prediction frames, then the size of the correlation matrix is m×n. After obtaining the correlation matrix, in this embodiment, the maximum element is first extracted from the correlation matrix, and then the row and column where the maximum element is located are excluded. The elements in the excluded rows and columns do not participate in the subsequent selection of the maximum element. After completing the first selection of the maximum element, the maximum element is then extracted again from the remaining elements, that is, the second selection of the maximum element. Similarly, after selecting the maximum element, the row and column where the maximum element is located are excluded again. The excluded rows and columns also do not participate in the subsequent selection of the maximum element. This process is repeated until there are no selectable elements in the association matrix. Finally, multiple maximum elements can be obtained. Each element of the association matrix corresponds to the association degree, and each association degree corresponds to a set of prediction frames. Through these extracted maximum elements, multiple correlated prediction frames can be obtained. The prediction frames correspond to the real-time positions of the fish. Therefore, after obtaining the above multiple prediction frames, the movement trajectory of each fish can be determined.
[0071] Step S50: performing feature transformation on the motion trajectory of each fish through an autoencoder.
[0072] After obtaining the motion trajectory, the present embodiment further uses an autoencoder to perform feature transformation on the motion trajectory of each fish. The specific process is to input the motion trajectory of each fish into the autoencoder; map the motion trajectory to a low-dimensional space through the encoder network corresponding to the encoder; obtain the low-dimensional feature representation of the motion trajectory in the low-dimensional space through the bottleneck layer of the autoencoder, and the bottleneck layer connects the output of the encoder and the input of the decoder; and extract multiple trajectory features corresponding to any fish trajectory from the low-dimensional feature representation using the decoder network corresponding to the decoder. Among them, the motion trajectory is mapped to a lower-dimensional space with a lower dimension. For example, the motion trajectory of a fish is composed of a series of position coordinates (x, y) at a time point. On a two-dimensional plane, this trajectory will be a straight line. Although this trajectory is two-dimensional, it needs to be further reduced to a one-dimensional space for simplification. The linear dimensionality reduction method adopted in this embodiment, such as principal component analysis (PCA), is used to map the two-dimensional trajectory to a one-dimensional space. For example, if the motion trajectory starts from the origin (0,0) and moves upward along the straight line y=3x, each two-dimensional coordinate point (x,y) can be mapped to a one-dimensional coordinate point z, and the corresponding low-dimensional feature representation is z=x+3y. Finally, the feature vector constructed by the low-dimensional feature representation can be used to obtain the trajectory feature.
[0073] It should be noted that the encoder in this embodiment is used to encode the original representation of the data to obtain a hidden layer representation, and the decoder is used to extract the feature representation of the original representation based on the hidden layer representation. The expression is B=Encoder(A), A'=Decoder(B), wherein A is the original representation, B is the encoded hidden layer representation, and A' is the feature representation of the original representation. Both the encoder and the decoder adopt a GRU network structure.
[0074] Step S60: Identify abnormal behaviors of individual fish based on the transformed features.
[0075] It should be noted that the transformed features are also trajectory features. In this embodiment, it is necessary to determine the distances between multiple trajectory features corresponding to any fish trajectory and preset sample features, wherein the preset sample features are constructed by collecting a large amount of data, and represent sample features corresponding to fish abnormalities. The above distance can be calculated using Euclidean distance, and then multiple trajectory features corresponding to distances less than or equal to the preset distance are determined, and the corresponding number of features is determined to be a first number, and multiple trajectory features corresponding to distances greater than the preset distance are determined, and the corresponding number of features is determined to be a second number. The trajectory abnormality ratio can be calculated based on the first number and the second number, for example, p=TP / FP, wherein TP represents the first number and FP represents the second number. If the trajectory abnormality ratio exceeds the preset ratio, it can be determined that the corresponding fish has abnormal behavior. The preset ratio can be set according to actual needs, and this is not limited in this embodiment.
[0076] This embodiment collects original video images of fish schools to be identified; identifies fish for each frame of the original video images through a target detection model, and labels each frame with a corresponding prediction box based on the identified fish; determines the correlation between each prediction box among a plurality of prediction boxes; determines the movement trajectory of each fish according to the correlation; performs feature transformation on the movement trajectory of each fish through an autoencoder; and identifies the abnormal behavior of each fish based on the transformed features. This method replaces the manual method, which not only improves the diagnosis efficiency but also improves the diagnosis accuracy.
[0077] Reference Figure 2 , Figure 2 This is a structural block diagram of the first embodiment of the device for diagnosing abnormal behavior of fishes according to the present invention.
[0078] like Figure 2 As shown, the abnormal behavior diagnosis device for fish proposed in the embodiment of the present invention comprises:
[0079] The acquisition module 10 is used to acquire the original video image of the school of fish to be identified.
[0080] The recognition module 20 is used to identify fish in each frame of the original video image through a target detection model, and to mark a corresponding prediction box for each frame based on the identified fish.
[0081] The calculation module 30 is used to determine the correlation between each prediction box in the plurality of prediction boxes.
[0082] The construction module 40 is used to determine the movement trajectory of each fish according to the correlation degree.
[0083] The transformation module 50 is used to perform feature transformation on the motion trajectory of each fish through an autoencoder.
[0084] The diagnosis module 60 is used to identify abnormal behaviors of individual fish based on the transformed features.
[0085] This embodiment collects original video images of fish schools to be identified; identifies fish for each frame of the original video images through a target detection model, and labels each frame with a corresponding prediction box based on the identified fish; determines the correlation between each prediction box among a plurality of prediction boxes; determines the movement trajectory of each fish according to the correlation; performs feature transformation on the movement trajectory of each fish through an autoencoder; and identifies the abnormal behavior of each fish based on the transformed features. This method replaces the manual method, which not only improves the diagnosis efficiency but also improves the diagnosis accuracy.
[0086] In some embodiments, the calculation module 30 is used to filter out overlapping prediction boxes from multiple prediction boxes;
[0087] Randomly select two prediction boxes from the overlapping prediction boxes, and determine the area of the overlapping area and the area of the covered area of the two prediction boxes respectively;
[0088] The intersection-over-union ratio of the two prediction frames is calculated according to the area of the overlapping area and the area of the covered area. The calculation formula is: Where x and y are any two selected prediction boxes, x∩y is the area of the overlapping area of the two prediction boxes, and x∪y is the area of the covered area of the two prediction boxes;
[0089] The correlation between the overlapping prediction frames is determined according to the IoU ratio, and the IoU ratio is positively correlated with the correlation.
[0090] In some embodiments, the calculation module 30 is used to select any two prediction boxes from the remaining prediction boxes that do not overlap, and respectively determine the center point distance between the two prediction boxes and the diagonal distance of one of the prediction boxes;
[0091] The correlation between the two prediction frames is calculated based on the center point distance and the diagonal distance in combination with the preset coefficient. The calculation formula is: Among them, d is the center point distance, l is the diagonal distance, and α is the preset coefficient.
[0092] In some embodiments, the construction module 40 is used to construct a correlation matrix according to the correlation, and the size of the correlation matrix corresponds to the number of prediction boxes of two consecutive frames of images;
[0093] Extracting the maximum element from the correlation matrix, and excluding the row and column where the maximum element is located, and the elements in the excluded row and column do not participate in the selection of the maximum element;
[0094] Extract the maximum element again from the remaining elements, and exclude the row and column where the maximum element is extracted again;
[0095] Return to the step of extracting the maximum element from the remaining elements again and excluding the row and column where the maximum element extracted again is located, until there are no selectable elements in the association matrix;
[0096] The corresponding prediction box is determined according to all the extracted elements to obtain the movement trajectory of each fish.
[0097] In some embodiments, the transformation module 50 is used to input the motion trajectory of each fish into the autoencoder;
[0098] Mapping the running trajectory to a low-dimensional space through an encoder network corresponding to the encoder;
[0099] Obtaining a low-dimensional feature representation of the motion trajectory in the low-dimensional space through a bottleneck layer of the autoencoder, wherein the bottleneck layer connects the output of the encoder and the input of the decoder;
[0100] A decoder network corresponding to the decoder is used to extract multiple trajectory features corresponding to any fish trajectory from the low-dimensional feature representation.
[0101] In some embodiments, the encoder is used to encode the original representation of the data to obtain a hidden layer representation, and the decoder is used to extract the feature representation of the original representation based on the hidden layer representation. The expression is B=Encoder(A), A'=Decoder(B), where A is the original representation, B is the encoded hidden layer representation, and A' is the feature representation of the original representation. Both the encoder and the decoder adopt a GRU network structure.
[0102] In some embodiments, the diagnostic module 60 is used to
[0103] Determine the distances between a plurality of trajectory features corresponding to any fish trajectory and preset sample features;
[0104] Determine a plurality of trajectory features corresponding to the distance being less than or equal to a preset distance, and determine the number of corresponding features to be a first number, and determine a plurality of trajectory features corresponding to the distance being greater than the preset distance, and determine the number of corresponding features to be a second number;
[0105] Calculating a trajectory abnormality ratio according to the first number and the second number;
[0106] If the trajectory abnormality ratio exceeds a preset ratio, it is determined that the corresponding fish has abnormal behavior.
[0107] An embodiment of the present application also provides a device for diagnosing abnormal behavior of fish, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus, and the memory is used to store a program for diagnosing abnormal behavior of fish; and the processor is used to implement the above-mentioned method for diagnosing abnormal behavior of fish when executing the program stored in the memory.
[0108] The communication bus mentioned in the above-mentioned abnormal behavior diagnosis device of fish can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc.
[0109] The communication interface is used for communication between the above-mentioned fish abnormal behavior diagnostic equipment and other equipment.
[0110] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0111] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.
[0112] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0113] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0114] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0115] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
[0116] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.
[0117] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.
[0118] In addition, for technical details not fully described in this embodiment, reference can be made to the abnormal behavior diagnosis method for fish provided in any embodiment of the present invention, and will not be repeated here.
[0119] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0120] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0122] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
[0123] It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above method.
Claims
1. A method for diagnosing abnormal behavior of fish, characterized in that: The method for diagnosing abnormal behavior of fish comprises: Collecting original video images of the fish school to be identified; Performing fish identification on each frame of the original video image through the target detection model, and marking a corresponding prediction box for each frame of the image based on the identified fish; Determine the correlation between each prediction box in the plurality of prediction boxes; Determine the movement trajectory of each fish according to the correlation degree; Performing feature transformation on the motion trajectory of each of the fishes through an autoencoder; Identify abnormal behaviors of individual fish species based on the transformed features.
2. The method for diagnosing abnormal behavior of fish according to claim 1, characterized in that: Determining the correlation between the prediction boxes in the plurality of prediction boxes includes: Filter out overlapping prediction boxes from multiple prediction boxes; Randomly select two prediction boxes from the overlapping prediction boxes, and determine the area of the overlapping area and the area of the covered area of the two prediction boxes respectively; The intersection-over-union ratio of the two prediction frames is calculated according to the area of the overlapping area and the area of the covered area. The calculation formula is: Where x and y are any two selected prediction boxes, x∩y is the area of the overlapping area of the two prediction boxes, and x∪y is the area of the covered area of the two prediction boxes; The correlation between the overlapping prediction frames is determined according to the IoU ratio, and the IoU ratio is positively correlated with the correlation.
3. The method for diagnosing abnormal behavior of fish according to claim 2, characterized in that: The method further comprises: Randomly select two prediction boxes from the remaining non-overlapping prediction boxes, and determine the center point distance between the two prediction boxes and the diagonal distance of one of the prediction boxes; The correlation between the two prediction frames is calculated based on the center point distance and the diagonal distance in combination with the preset coefficient. The calculation formula is: Among them, d is the center point distance, l is the diagonal distance, and α is the preset coefficient.
4. The method for diagnosing abnormal behavior of fish according to claim 1, characterized in that: Determining the movement trajectory of each fish according to the correlation degree includes: Constructing a correlation matrix according to the correlation, wherein the size of the correlation matrix corresponds to the number of prediction frames of two consecutive frames of images; Extracting the maximum element from the correlation matrix, and excluding the row and column where the maximum element is located, and the elements in the excluded row and column do not participate in the selection of the maximum element; Extract the maximum element again from the remaining elements, and exclude the row and column where the maximum element is extracted again; Return to the step of extracting the maximum element from the remaining elements again and excluding the row and column where the maximum element extracted again is located, until there are no selectable elements in the association matrix; The corresponding prediction box is determined according to all the extracted elements to obtain the movement trajectory of each fish.
5. The method for diagnosing abnormal behavior of fish according to claim 1, characterized in that: The autoencoder includes an encoder and a decoder, and the feature transformation of the motion trajectory of each fish by the autoencoder includes: Input the movement trajectory of each fish into the autoencoder; Mapping the running trajectory to a low-dimensional space through an encoder network corresponding to the encoder; Obtaining a low-dimensional feature representation of the motion trajectory in the low-dimensional space through a bottleneck layer of the autoencoder, wherein the bottleneck layer connects the output of the encoder and the input of the decoder; A decoder network corresponding to the decoder is used to extract multiple trajectory features corresponding to any fish trajectory from the low-dimensional feature representation.
6. The method for diagnosing abnormal behavior of fish according to claim 5, characterized in that: The encoder is used to encode the original representation of the data to obtain a hidden layer representation, and the decoder is used to extract the feature representation of the original representation based on the hidden layer representation. The expression is B=Encoder(A), A'=Decoder(B), where A is the original representation, B is the encoded hidden layer representation, and A' is the feature representation of the original representation. Both the encoder and the decoder adopt a GRU network structure.
7. The method for diagnosing abnormal behavior of fishes according to claim 5, characterized in that: The method of identifying abnormal behaviors of various fish species based on the transformed features includes: Determine the distances between a plurality of trajectory features corresponding to any fish trajectory and preset sample features; Determine a plurality of trajectory features corresponding to the distance being less than or equal to a preset distance, and determine the number of corresponding features to be a first number, and determine a plurality of trajectory features corresponding to the distance being greater than the preset distance, and determine the number of corresponding features to be a second number; Calculating a trajectory abnormality ratio according to the first number and the second number; If the trajectory abnormality ratio exceeds a preset ratio, it is determined that the corresponding fish has abnormal behavior.
8. A device for diagnosing abnormal behavior of fish, characterized in that: The fish abnormal behavior diagnostic device comprises: An acquisition module, used for acquiring original video images of the fish school to be identified; A recognition module, used to identify fish for each frame of the original video image through a target detection model, and to mark a corresponding prediction box for each frame of the image based on the identified fish; A calculation module, used to determine the correlation between each prediction box in the plurality of prediction boxes; A construction module is used to determine the movement trajectory of each fish according to the correlation degree; A transformation module, used for performing feature transformation on the motion trajectory of each fish through an autoencoder; The diagnostic module is used to identify abnormal behaviors of individual fish species based on the transformed features.
9. The abnormal behavior diagnosis device for fishes according to claim 8, characterized in that: The calculation module is used to filter out overlapping prediction frames from multiple prediction frames; Randomly select two prediction boxes from the overlapping prediction boxes, and determine the area of the overlapping area and the area of the covered area of the two prediction boxes respectively; The intersection-over-union ratio of the two prediction frames is calculated according to the area of the overlapping area and the area of the covered area. The calculation formula is: Where x and y are any two selected prediction boxes, x∩y is the area of the overlapping area of the two prediction boxes, and x∪y is the area of the covered area of the two prediction boxes; The correlation between the overlapping prediction frames is determined according to the IoU ratio, and the IoU ratio is positively correlated with the correlation.
10. The abnormal behavior diagnosis device for fishes according to claim 9, characterized in that: The calculation module is used to select any two prediction boxes from the remaining prediction boxes without overlap, and respectively determine the center point distance between the two prediction boxes and the diagonal distance of one of the prediction boxes; The correlation between the two prediction frames is calculated based on the center point distance and the diagonal distance in combination with the preset coefficient. The calculation formula is: Among them, d is the center point distance, l is the diagonal distance, and α is the preset coefficient.