Interactive virtual reality device for animal group observation experiment

Abstract

In order to facilitate animal behavior induction and observation experiments, the invention provides a device for laboratory animal group behavior observation. After four industrial cameras (1) are connected to an upper computer by referring to a figure 1 in the specification, a related target tracking algorithm is started, captured target poses are transmitted to a virtual scene simulator, and a virtual scene is generated according to a panoramic view construction method. A virtual scene in the upper computer is projected through the virtual image generation device (3) and is reflected to the group container (4) through the reflection mirror surface (6), and an immersive virtual scene for animals is generated. The infrared light source array (5) is annularly arranged below the group container (4) and is mainly used for providing a light source for the industrial camera (1), eliminating interference of visible light and enhancing image quality. All the devices are fixed through trusses, and the device comprises a supporting truss (2), a fixing truss (5) and a reinforcing truss (7).

Description

Technical Field

[0001] The present invention is an interactive virtual reality device for animal group observation experiments, which is mainly used to identify the schooling and avoiding behaviors of fish and generate corresponding virtual reality scene mapping, and belongs to the field of engineering application. Background Art

[0002] The current animal group observation platform basically passively observes the group behavior of animals in natural scenes. Observation in natural conditions has very large limitations and requires a huge amount of manpower and material resources. In order to facilitate laboratory researchers to observe animal groups in artificial situations, this new practical device has been developed based on inducing and observing the group behavior of animal groups. Compared with the VR engine FreeMoveVR developed by the Couzin team at the Max Planck Institute in Germany, this system has the following advantages: (1) The model adds a skeletal system to establish a kinematic model between different joints and skeletons of the animal group, which is used to coordinate the muscles and fins of the virtual animal group to produce natural movement animation effects, and can be covered with real textures to make the model's animation effects closer to reality; (2) In addition to tracking the three-dimensional Cartesian coordinate system position (x, y, z) of the animal group, it can also track the pitch angle θ and yaw angle ψ; (3) It can generate a first-person perspective of the animal group in the virtual environment as a basis for judging interactive experiments; (4) The tracked coordinate and posture information is sent to communicate with UE4 via the UDP protocol, overcoming the shortcoming that FreeMoveVR only supports the Melodic version of ROS and can achieve multi-platform compatibility.

[0003] In summary, the present invention has developed a naked-eye immersive interactive virtual reality device for animal behavior induction and observation experiments, which can realize the generation of naked-eye immersive and interactive virtual reality environments for various animal groups, especially animal groups, and can be used for animal behavior induction and observation experimental research. Summary of the invention

[0004] In order to facilitate animal behavior induction and observation experiments, the present invention provides a device for observing group behavior of laboratory animals. Figure 1 After connecting the four industrial cameras ① to the host computer, the relevant target tracking algorithm is started, and the captured target posture is transmitted to the virtual scene simulator, and a virtual scene is generated according to the panoramic view construction method. The virtual scene in the host computer is projected through the virtual image generation device ③, and reflected to the group container ④ through the reflective mirror ⑥, generating an immersive virtual scene for animals. The infrared light source array ⑤ is arranged in a ring below the group container ④. Its main function is to provide light source for the industrial camera ① and eliminate the interference of visible light to enhance the image quality. All devices are fixed by trusses. This device includes a supporting truss ②, a fixed truss ⑤ and a reinforcing truss ⑦.

[0005] (1) Body coordinate system Assume that the center of mass of the animal group is O; the x-axis is in the positive direction pointing to the front of the animal group along the symmetry plane of the animal group, and the rotation angle around the x-axis is the roll angle. The y-axis is perpendicular to the symmetry plane of the animal group, and is in the positive direction pointing to the right side of the animal group, and the rotation angle of the y-axis is the pitch angle. The z-axis is perpendicular to the xOy plane, and is in the positive direction pointing upward, and the rotation angle around the z-axis is the yaw angle ψ. The schematic diagram of the body coordinate system is shown in the attached figure. Figure 2 shown.

[0006] The novel practical method uses a binocular camera to identify and track the three-dimensional translation position (x, y, z) of the test animal group; and uses an algorithm to solve the rotation posture (θ, φ) of the animal group except rolling; finally, the three-dimensional translation position (x, y, z) and rotation posture (θ, φ) are transmitted to a virtual reality scene generation module.

[0007] (2) Tracking algorithm The tracking algorithm of the present invention can automatically track animal groups, obtain the three-dimensional coordinates of the animal group and save them according to the current frame. The algorithm first processes the image read by the industrial camera, and converts the three-channel grayscale image into a single-channel grayscale image for subsequent processing. The first frame of the image captured by the camera is used as the background, and the background frame is subtracted from each subsequent frame. What remains after subtraction is the part of the extra feature object (the object to be detected). However, the subtracted part will also affect the grayscale value of the feature object, and generally a relevant threshold is set and judged. After obtaining the thresholded image, it is necessary to detect the contour of the image, and the maximum value point of the (x, y) coordinate of the contour is used as the minimum circumscribed rectangular frame of the contour, and the center point of the rectangular frame is used as the coordinate point of the target.

[0008] In this device, the hemispherical fish tank is made of highly transparent acrylic material, and the outer spherical surface uses a rear projection cloth to facilitate the imaging of the projector and ensure the authenticity and clarity of the imaging. The video data collected by the camera is input into the computer through the communication interface. When examining the behavior of the target, it is only necessary to regard the single target coordinate as a point and calculate the three-dimensional coordinates of each target separately. Since the calculation of three-dimensional coordinates cannot be obtained through a monocular camera, it is necessary to first calculate the two-dimensional pixel coordinates of the target in the image in each camera, and match the serial number (ID) belonging to the same target in each camera according to the minimum distance. The depth of the target in the water is calculated according to the two-dimensional pixel coordinates of the same target in each camera, thereby obtaining the three-dimensional coordinates of the target. The specific calculation process of the three-dimensional coordinates is as follows.

[0009] First, create the following Figure 3The coordinate origin of the spatial coordinate system is selected as the point on the horizontal plane passing through the center line of the fish tank. The X direction and the Y direction correspond to two directions on the horizontal plane, and the Z direction is the direction perpendicular to the horizontal plane. The industrial camera is fixedly installed on the bracket, and the coordinates in the spatial coordinate system are According to the principle of image intersection measurement, the world coordinates (X, Y, Z) of the object point can be calculated by formula 1-1:

[0011] Let the parameter matrix M be:

[0013] Combined to get:

[0015] The actual algorithm is analyzed. The thresholding of the image uses the Threshold method, and a fixed threshold is selected to obtain a binary image of background subtraction, which can basically maintain the image acquisition and processing rate of 25 frames per second. The contour extraction of the animal group uses the findContours method. After extracting the connected domain contour array, the second and third columns of the array, i.e. the corresponding (x, y) coordinates, are searched, and their maximum values ​​are used as the vertex coordinates of the minimum circumscribed rectangle. Since there are few contour pixels, the operation time is very short.

[0016] The virtual reality scene generation system of this patent is implemented using Unreal Engine and projected onto the surface of a hemispherical animal group tank. First, a hemispherical virtual animal group tank scene is built in Unreal Engine 4. The position of the animal group in the virtual scene is the translation coordinates input by the three-dimensional tracking system, and the rotation posture is the posture angle input by the tracking system. The six environmental maps of the front, back, left, right, top and bottom at the translation coordinates are obtained frame by frame to form a panoramic cube, as shown in the attached figure. Figure 4 As shown: Then, the pixels of the panoramic cube are projected onto the spherical surface. The projection relationship is: let the center of the projected sphere coincide with the center of the panoramic cube, and emit a ray from the center of the panoramic cube, intersecting the cube at point , explaining the projection sphere at point , then The RGBA value of the pixel at that location is assigned to the position on the sphere The panoramic image can be constructed with pixels.

[0017] Finally, the lower hemisphere of the sphere is projected onto the wall of the experimental device, and the deployment of the virtual reality scene is completed. The flowchart of the entire virtual reality scene generation is shown in the attached figure. Figure 5 shown.

[0018] In addition, the virtual reality scene generation system can also generate a first-person perspective image of the animal group in the virtual scene based on the received translation coordinates (x, y, z) and rotation angles (θ, φ) of the animal group, so as to visualize the field of view of the virtual character corresponding to the animal group in the virtual scene. UDP communication is used between this new practical virtual reality image generation system and the tracking system. The virtual reality image generation system has an embedded UDP / TCP socket deployment module, which can establish a UDP / TCP receiving end and accept UDP / TCP data packets sent from the outside. The tracking system also has an embedded UDP / TCP socket deployment module, which can establish a UDP / TCP sending end and send the solution results to the virtual reality image generation system. The overall communication flow chart is as shown in the attached figure. Figure 6 shown. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 It is a schematic diagram of an interactive virtual reality device used in animal group observation experiments; Figure 2 is the coordinate system diagram of the observed animal body; Figure 3 It is the spatial coordinate system diagram of the experimental container; Figure 4 It is the flow chart of the device operation; Figure 5 It is a schematic diagram of the principle of generating panoramic virtual reality images; Figure 6 It is a virtual scene generation flow chart; Figure 7 It is the communication flow chart between the sub-modules of the system; In the figure: 1. Industrial camera; 2. Support truss; 3. Virtual image generation device; 4. Group container; 5. Fixed truss; 6. Reflective mirror; 7. Strengthening truss; 8. Graphic workstation; 9. Router. (3) Specific implementation methods

[0020] The present invention provides a novel device for observing group behavior of laboratory animals, and its main implementation steps are as follows.

[0021] Step 1: This new utility model requires the observation of the animal group in the experimental container ④ by an industrial camera array ① Step 2: Start the relevant target detection and tracking algorithm to track the animal group in the experimental container ④, model the group behavior mechanism, and transfer the relevant group behavior parameters to the host computer. The host computer can be a Linux or Windows computer that needs to be able to run Unreal Engine 4.

[0022] Step 3: Generate the corresponding virtual reality scene in the host computer. The scene is formed by image segmentation and fusion.

[0023] Step 4: Use the virtual image generation device ③ to project two virtual guide bodies into the group container ④ and move them back and forth in parallel, providing two guide and follow targets for the real group. Since animal groups are social, they tend to follow these virtual bodies at a fixed relative distance.

[0024] Step 5: While generating the virtual guide body, the observed group behavior data is exported and saved. When there are two virtual bodies in the scene, it can be observed that as the distance L between the two increases, there will be an obvious bifurcation in the decision-making of the animal group. In this bifurcation, the animal group facing two choices will suddenly make a choice from the two targets, that is, before the bifurcation, there is no obvious bias in its following swimming, and after the bifurcation, it will switch to closely following other animal individuals. This implementation includes two sets of tests: (1) By setting different implementation scenarios, find a better following scenario. (2) By changing the distance before the follower, explore the characteristics of the animal group's decision-making process. According to the implementation results, by providing the animal group with a discrete binary choice companion implementation, it is found that as the distance between the virtual options increases, the individual will have two critical states, which will greatly change the movement pattern of the organism, verifying the effectiveness of the present invention.

Claims

1. A virtual reality immersive interactive experimental system for animal group observation, comprising (1) an infrared light single-pass camera array (2) a camera support mechanism (3) an image projection device (4) and a light source converger (5), characterized in that an imaging position adjustment system consisting of an active adjuster (6) and a support member (7) is provided, wherein the active adjuster is fixedly connected to the image projection device and is mainly used to adjust the imaging position of the generated image, and the support member is arranged on the bottom side of the mechanism and is provided with an anti-slip device.

2. The virtual reality immersive interactive experiment system as described in claim 1 is characterized in that it can adaptively adjust the imaging position of the virtual reality image source according to the position of the animal group, generating a high-fidelity simulated natural environment for visual animal group observation.

Images