Fishway fish observation method based on mixed reality and image recognition

Abstract

The application discloses a fishway fish observation method based on mixed reality and image recognition, carries out fish species and quantity identification training on fish images based on a YOLOv3 algorithm, and constructs a fish species and quantity identification model; a three-dimensional modeling software is used to construct a fishway three-dimensional model, and a conversion relationship between the fishway three-dimensional model coordinates and fishway coordinates is established; the fishway three-dimensional model is imported into three-dimensional engine software to write a mixed reality application, and is imported into a mixed reality device; underwater cameras are installed at certain intervals in the fishway to collect fishway images; fish species, quantity, migration direction in the fishway images are identified through the fish species and quantity identification model, and the positions of fishes between adjacent underwater cameras are estimated; the positions are sent to the mixed reality device; and the fishway fish situation is presented in real time through the mixed reality device, so that fishway fish observation research is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of fishway fish observation, and in particular to a fishway fish observation method based on mixed reality and image recognition. BACKGROUND

[0002] Currently, fish migration observation is mostly carried out by establishing an observation room at a fixed point in the fishway, which lacks a systematic observation method for fish migration and does not provide a macroscopic observation of fish in the fishway as a whole.

[0003] Mixed reality (English full name: Mixed Reality, English abbreviation: MR) is a further development of virtual reality technology. This technology presents virtual scene information in a real scene, and provides an information loop for users to interact and feedback between the real world and the virtual world, so as to enhance the sense of reality of the user experience.

[0004] Through training of a deep learning model, the image recognition technology can accurately identify fish species, quantity and other data. If mixed reality technology and image recognition technology are combined, the migration of fish in the fishway can be observed in a holographic image, and the migration of fish can be studied, so as to better protect the fish ecology at the water conservancy hub. SUMMARY

[0005] The present application aims to provide a fishway fish observation method based on mixed reality and image recognition.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] The fishway fish observation method based on mixed reality and image recognition provided by the present application comprises the following steps:

[0008] S1, fish species and quantity recognition training of fish images based on YOLOv3 algorithm, to construct a fish species and quantity recognition model;

[0009] S2, constructing a fishway three-dimensional model by using a three-dimensional modeling software, and establishing a conversion relationship between the fishway three-dimensional model coordinates and the fishway coordinates;

[0010] S3, importing the fishway three-dimensional model into a three-dimensional engine software to write a mixed reality application, and importing the mixed reality device;

[0011] S4, installing underwater cameras in the fishway at certain intervals to collect fishway images;

[0012] S5, identifying the fish species, quantity, migration direction in the fishway images by the fish species and quantity recognition model, and estimating the position of fish between adjacent underwater cameras;

[0013] S6, send the fish species, quantity, migration direction and position information identified in S5 to the mixed reality device;

[0014] S7, present the fishway fish situation in real time through the mixed reality device to realize fishway fish observation and research.

[0015] The method of the present application automatically identifies fish information in the fishway through artificial intelligence image recognition method, and then realizes observation of fish situation in the fishway through mixed display device in holographic effect.

[0016] Further, the distance in S4 is 50m; the underwater camera installation requirement is that the lens axis of the underwater camera is perpendicular to the side wall of the fishway; and the fishway image is an image taken when the fish passes through the lens axis of the underwater camera.

[0017] Further, in S5, the position of the fish between the adjacent underwater cameras is estimated, including interpolating and calculating the specific position of the fish in the fishway three-dimensional model according to the migration speed of the fish, and converting the fishway three-dimensional model coordinates into fishway coordinates.

[0018] The present application has the advantages of using image recognition and mixed reality technology to observe fish in the fishway in real time, obtaining the position, quantity, migration habit of migratory fish in the fishway in real time, providing a new holographic, full-line and real-time fish observation method, which is very helpful for studying fish migration and can better protect the fish ecology at the water conservancy hub. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flow chart of the method of the present application.

[0020] Figure 2 is a schematic diagram of the underwater camera in the method of the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0022] As shown in Figure 1 The fishway fish observation method based on mixed reality and image recognition of the present application adopts Baidu PaddleDetetection target detection development kit, and uses the YOLOv3 target detection algorithm provided by PaddleDetection to train and verify the model.

[0023] The inventor collects 10 common migratory freshwater fish species, a total of 1000 pictures, of which 850 are used as a training set and 150 are used as a verification set. After the pictures are collected, the pictures are labeled, then trained to obtain a fish species and quantity recognition model.

[0024] The fishway is modeled in three dimensions by using modeling software such as Microstation, the built fishway three-dimensional model is imported into 3dmax for material processing, and the common 10 kinds of migratory freshwater fish are modeled in 3dmax. Then the fishway three-dimensional model exported by 3dmax is imported into the game engine Unity3d, the fishway three-dimensional model is processed in Unity3d, the MRTK development package of Microsoft is introduced, and the mixed reality application is developed based on the OpenXR interface of Unity3d, realizing the services of space, UI interaction, information acquisition, etc. of the mixed reality application. The migratory fish position in the mixed reality application is updated every 3 seconds and displayed. Then the mixed reality application is exported and installed in the mixed reality device HoloLens2, and the application can be used by opening it in HoloLens2.

[0025] As shown in Figure 2 An underwater camera 1 is arranged every 50m in the actual fishway, and the lens axis of the underwater camera 1 is perpendicular to the side wall of the fishway 2. Taking the lens axis of the underwater camera 1 as the dividing line, when the migratory fish passes through the dividing line, it indicates that the migratory fish passes through the underwater camera 1, and the underwater camera 1 shoots the image information in the fishway at this time. The fish species and quantity recognition model is used to identify the species, quantity, migration direction, etc. of the fish in the fishway image. At the same time, by the migration speed of a certain fish, the position of the fish between the adjacent two underwater cameras 1 is calculated, and the coordinate is converted into the coordinate in the fishway three-dimensional model. The current recognized fishway situation, such as the species, quantity, migration direction, position, etc. of the fish, is sent to the mixed display device HoloLens2, and the holographic image of the migratory fish in the mixed reality application is refreshed. The observer can use HoloLens2 to directly observe and study the fish, quantity, migration direction, etc. in the fishway through the holographic image, realizing holographic, full-line, real-time observation, which is of great help to the study of fish migration, and can better protect the fish ecology at the water conservancy hub.

Claims

1. A method for observing fish in fishways based on mixed reality and image recognition, characterized in that: Includes the following steps: S1, Based on the YOLOv3 algorithm, fish images are trained to identify fish species and quantities, and a fish species and quantity identification model is constructed. S2, Construct a three-dimensional model of the fishway using three-dimensional modeling software, and establish the transformation relationship between the coordinates of the three-dimensional model of the fishway and the coordinates of the fishway; S3: Import the 3D model of the fishway into the 3D engine software to write a mixed reality application, and then import it into the mixed reality device; S4, Install underwater cameras at regular intervals in the fishway to capture images of the fishway; the interval is 50m; the installation requirements for the underwater cameras are: the lens axis of the underwater camera is perpendicular to the side wall of the fishway; the fishway images are images taken when fish pass through the lens axis of the underwater camera. S5, using the fish species and quantity recognition model, identify the fish species, quantity, and migration direction in the fishway image, and estimate the position of the fish between adjacent underwater cameras; estimating the position of the fish between adjacent underwater cameras includes interpolating the fish's migration speed to calculate the specific position of the fish in the fishway 3D model, and converting the fishway 3D model coordinates into fishway coordinates; S6, send the fish species, quantity, migration direction, and location information identified in step S5 to the mixed reality device; The S7 uses mixed reality devices to present the fish situation in the fishway in real time, enabling fish observation and research in the fishway.

Images