A method and system for target recognition and ranging of a manta ray robot with isometric binoculars

Abstract

The application discloses a target recognition and distance measurement method and system for a heterogeneous binocular manta ray robot, and belongs to the field of intelligent perception of underwater robots. The method comprises the following steps: encoding original event stream data of a dynamic vision sensor into images; calibrating camera parameters of multiple images generated by encoding of the dynamic vision sensor and multiple images of an industrial camera, so as to obtain the focal lengths of the two cameras and the positional relationship between the two cameras; performing target recognition on the images generated by event encoding of the dynamic vision sensor and the images of the industrial camera through a deep learning algorithm, marking the image area of the target, and obtaining the image coordinates of the target area center; and calculating the distance between the target manta ray robot and the heterogeneous binocular camera by using a binocular distance estimation algorithm according to the target feature center coordinates, the internal parameters and the external parameters of the two cameras. The application can obtain more effective data underwater, and improves the information transmission efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of intelligent perception of underwater robots, and particularly relates to a target recognition and distance measurement method and system for a heterogeneous binocular manta ray robot. BACKGROUND

[0002] In the cluster motion of underwater bionic manta ray robots, a robot needs to identify other robots in the surrounding through a visual perception system and measure the distance between itself and other robots, so as to provide front-end perception information for the cluster motion control and cooperative action among multiple robots. The existing underwater visual target recognition and distance measurement method mainly relies on underwater binocular same-industry cameras. The same-industry cameras acquire images based on frames, and have problems of motion blur, low dynamic range and large data redundancy. In the underwater environment, there is large motion disturbance, weak light and redundant background interference, and it is difficult to perceive the target by relying on the same-industry cameras alone.

[0003] The binocular system composed of the same-industry cameras in the prior art has unstable data acquired underwater and low efficiency of information transmission. At the same time, the effect of target recognition and distance measurement by the same-industry cameras in the underwater environment is unstable, and the reliability of underwater visual target perception is low.

[0004] In view of this, the application discloses a target recognition and distance measurement method for an underwater manta ray robot with a heterogeneous binocular system. The heterogeneous binocular system is composed of a dynamic vision sensor on the left and an industry camera on the right. The dynamic vision sensor is introduced on the basis of the industry camera, which improves the effect of target recognition and distance measurement by the same-industry cameras in the underwater environment and improves the reliability of underwater visual target perception. SUMMARY

[0005] Technical problems to be solved:

[0006] In order to avoid the shortcomings of the prior art, the application provides a target recognition and distance measurement method based on a heterogeneous binocular system composed of a dynamic vision sensor and an industry camera, which is used for visual perception of other robot targets in the cluster motion of underwater bionic manta ray robots. The purpose of the application is to enable the manta ray robot to effectively recognize other manta ray robots in the underwater low-light, motion disturbance and data redundancy environment when performing cluster cooperative tasks underwater, and to calculate the relative distance, so as to provide perception information for subsequent robot motion control. The dynamic vision sensor is introduced into the binocular camera system, which can acquire more effective data underwater compared with the binocular system composed of the same-industry cameras, and improves the efficiency of information transmission.

[0007] The technical scheme of the application is a target recognition and distance measurement method for a heterogeneous binocular manta ray robot, characterized by the following specific steps:

[0008] Step 1: encode the dynamic vision sensor raw event stream data into images for subsequent calibration and image recognition;

[0009] Step 2: calibrate the camera parameters by encoding the generated multiple images of the dynamic vision sensor and the multiple images of the industrial camera, to obtain the focal length of the two cameras and the positional relationship between the two cameras;

[0010] Step 3: perform target recognition on the images generated by the dynamic vision sensor event encoding and the industrial camera images through deep learning algorithms, and mark the image area of the target to obtain the image coordinates of the target area center;

[0011] Step 4: using the binocular distance estimation algorithm, calculate the distance between the target manta ray robot and the heterogeneous binocular camera based on the target feature center coordinates obtained from the binocular images in step 3 and the internal and external parameters of the two cameras obtained in step 2.

[0012] A further technical solution of the present application is that in step 1, the dynamic vision sensor raw event stream data is encoded into images using a frequency-based encoding method.

[0013] A further technical solution of the present application is that in step 1, each event of the dynamic vision sensor event stream data consists of four parameters (t, x, y, p), including a microsecond-level timestamp t, position coordinates (x, y) in the image, and event change polarity p; the data is encoded into images using an event frequency-based method, which divides the event stream data at a fixed millisecond-level time interval, i.e. 30 milliseconds, accumulates all events within the time interval, and maps and normalizes the event frequency value to obtain each pixel value of the image; the more the target in the background moves, the higher the event frequency, and the higher the frequency, the higher the pixel brightness at the corresponding position in the image; the mapping and normalization calculation formula is:

[0014]

[0015] Where x is the total number of events occurring at a certain pixel point within a fixed time interval, and sigma(x) is the pixel value of a certain pixel point in the image between 0 and 255;

[0016] After calculating the pixel brightness value, different colors are used to represent different polarity events, with green dots representing positive polarity events with increased brightness, and red dots representing negative polarity events with decreased brightness; the pixel value of each point in the image is obtained from the event frequency through formula (1), and the color of different polarity events is marked, so that the event stream is encoded into a color image.

[0017] A further technical solution of the present application is that in step 2, the internal and external parameter calibration method of the industrial camera and the dynamic vision sensor is:

[0018] For the industrial camera, a checkerboard calibration board is used to take dozens of pictures of the calibration board at different angles and sizes, and the Zhang calibration method is used to obtain the internal focal length parameters of the industrial camera;

[0019] For the dynamic vision sensor, since it can only detect moving targets, the calibration board needs to be constantly moved during shooting. During the movement, the event stream is accumulated continuously using the encoding method in the previous step until dozens of frames of images are generated, and then the Zhang calibration method is used to calibrate the frame images, remove the images with large recognition deviation of the calibration points, and reduce the re-projection error to within an acceptable range of 1, to obtain the internal focal length parameters of the dynamic vision sensor;

[0020] After obtaining the internal parameters, the external parameters are calculated, and according to the binocular stereo calibration principle, the position translation relationship between the left and right images generated at the same time is obtained, which is represented by a translation matrix; the camera internal parameters and external parameters obtained by calibration are used for binocular distance measurement in step 4.

[0021] A further technical solution of the present application is that in step 3, an isomorphic binocular vision system is used to shoot the manta ray robot movement form, and industrial camera image sets and DVS image sets are obtained respectively; the robot target of each image is labeled to generate a corresponding label file, record the minimum and maximum values of the target pixel coordinates in the X and Y directions of the image, the name category label and the recognition difficulty, and use the image set and the labeling information to obtain the industrial camera data set and the DVS data set about the manta ray robot respectively, and divide the data set into a training set and a test set according to a 9:1 ratio for deep learning, and the category of the training target is the manta ray robot;

[0022] The data set is sent to neural network training, and the network model is YOLOv3, which is divided into Darknet-53 structure, feature layer fusion structure and classification detection structure; after the image is input into Darkenet-53, three branches are obtained, up-sampling and down-sampling are performed to make the feature maps of the same size, then stacking, fusion and corresponding convolution operations are performed to obtain three final feature maps, which are 1 / 32, 1 / 16 and 1 / 8 of the original image, and then the three feature maps are fused;

[0023] The DVS data set and the industrial camera data set are respectively iteratively trained for 30 rounds through the neural network to obtain two trained models; the two models are used to predict the manta ray robot target position area in the DVS and industrial camera images respectively and marked with a rectangular frame;

[0024] The predicted target region has been marked with a rectangular frame, and the X-direction and Y-direction coordinates of the upper left corner point and the lower right corner point of the frame are used to calculate the median as the feature center coordinates of the target, and the feature center coordinates of the two images are P l and P r respectively. l and P r The X-direction values of the coordinates are used as the basis for the next step of binocular lateral disparity calculation.

[0025] A further technical solution of the application is that the spatial model of the heterogeneous binoculars is that the left DVS coordinate system and the world coordinate system three axes coincide, and the right is an industrial camera coordinate system, the imaging surface is to the right as the X-axis direction, the imaging surface is downward as the Y-axis direction, and the forward is the Z-axis direction, and the spatial coordinate system is constructed.

[0026] A further technical solution of the application is that in step 4, the imaging planes of the two cameras are parallel to the XY plane of the world coordinate system, the imaging planes of the industrial cameras and the DVS imaging planes are coplanar, the X, Y and Z axes of the two camera coordinate systems are parallel and the principal axes are parallel, and the origins of the two cameras are different; the feature point P(X p , Y p , Z p ) of the target in space has differences in X coordinates and Y coordinates of the left and right imaging planes of the two cameras, while the Z coordinates are basically consistent in space.

[0027] According to the difference X l -X r between the X coordinates of the two feature centers P l and P r of the binocular recognition target, which is the disparity, and the focal lengths f1 and f2 obtained by step 2 calibration, and the baseline distance B between the principal axes of the centers of the two cameras, the front-back distance Z between the target manta ray robot and the heterogeneous binocular camera can be calculated by the following formula:

[0028]

[0029] This distance Z is the spatial distance relationship between the head heterogeneous binocular camera of the rear manta ray robot and the front target manta ray robot, that is, the ranging result is obtained.

[0030] A heterogeneous binocular vision system, characterized in that it comprises an underwater industrial camera, a dynamic vision sensor DVS, a waterproof camera shell and a waterproof data line; the dynamic vision sensor DVS is placed in the shell and connected and fixed with the underwater industrial camera by using a rigid support;

[0031] The rigid support is horizontally placed in water to make the two cameras level, the waterproof data lines of the two cameras are connected with a computer to transmit data, and the data of the dynamic visual sensor DVS is encoded to generate an image by using an event frequency-based method.

[0032] A further technical solution of the present application is that the fixed distance between the dynamic visual sensor DVS and the underwater industrial camera is 250 mm.

[0033] A further technical solution of the present application is that the dynamic visual sensor DVS is located on the left side, and the underwater industrial camera is located on the right side.

[0034] Beneficial effects

[0035] The present application has the beneficial effects that the present application uses a dynamic visual sensor and an industrial camera to form a heterogeneous binocular system, compared with a common homogeneous binocular system composed of two industrial cameras, since the dynamic visual sensor has the characteristics of capturing moving targets, high dynamic range and low data redundancy at one end, it can automatically remove static environmental information in the environment that is irrelevant to the moving robot target, reduce underwater motion blur of the industrial camera image, eliminate the influence of redundant static background of the industrial camera image, improve the clarity of the moving robot target in the image, reduce the target recognition difficulty of the image algorithm, make the target recognition accuracy higher, and improve the accuracy of subsequent binocular positioning.

[0036] At the same time, the system retains one industrial camera on the other side, so that the heterogeneous binocular system can not only identify the moving target in the dynamic visual sensor, but also identify the static target in the industrial camera. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 Heterogeneous binocular prediction target area position and confidence;

[0038] Figure 2 Heterogeneous binocular space target imaging and ranging principle;

[0039] Figure 3 Target distance calculation and measurement result curve. DETAILED DESCRIPTION

[0040] The embodiments described below with reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0041] Based on the binocular system composed of the prior art industrial camera, the data obtained underwater is unstable, the information transmission efficiency is low, the effect of target recognition and distance measurement of a single industrial camera in underwater environment is unstable, and the reliability of underwater visual target perception is low; a heterogeneous binocular underwater manta ray robot target recognition and distance measurement method and system is invented, the heterogeneous binocular system is composed of a dynamic vision sensor on the left and an industrial camera on the right, the advantages of introducing a dynamic vision sensor on the basis of an industrial camera are introduced, the effect of target recognition and distance measurement of a single industrial camera in underwater environment is improved, and the reliability of underwater visual target perception is improved.

[0042] The technical scheme of the embodiment of the heterogeneous binocular manta ray robot target recognition and distance measurement method is:

[0043] Step 1: encode the raw event stream data of the dynamic vision sensor into images using a frequency-based encoding method for subsequent calibration and image recognition.

[0044] Each event of the dynamic vision sensor event stream data is composed of four parameters (t, x, y, p), including a microsecond-level timestamp t, position coordinates (x, y) in the image, and event change positive and negative polarity p. The data is encoded into an image using an event frequency-based method, the event stream data is divided at a fixed millisecond-level time interval (30 milliseconds), all events in the time interval are accumulated, the pixel value of each image is obtained by mapping and normalizing the high and low values of the event frequency value. The more the target moves in the background, the higher the event frequency, and the higher the pixel brightness in the corresponding position in the image. The mapping and normalization calculation formula is:

[0045]

[0046] Wherein, x is the total number of events occurring at a certain pixel point in a fixed time interval, and sigma (x) is the pixel value of a certain pixel point between 0 and 255 in the image.

[0047] After calculating the pixel brightness value, different colors are used to represent different polarity events, green dots represent positive polarity events with increasing brightness, and red dots represent negative polarity events with decreasing brightness. The pixel value of each point in the image is obtained from the event frequency by formula (1), and the color of different polarity events is marked, so that the event stream is encoded into a color image.

[0048] Step 2: calibrate the camera parameters for the multiple images generated by the dynamic vision sensor and the multiple images of the industrial camera, obtain the focal length of the two cameras and the positional relationship between the two cameras, and use it for subsequent distance measurement calculation in step 4.

[0049] According to different characteristics of industrial camera and dynamic vision sensor, different methods are used to calibrate internal parameters and then external parameters. For industrial camera, a chessboard calibration board is used to take dozens of pictures at different angles and different sizes. Zhang's calibration method is used to get the internal focal length parameters of the industrial camera. For dynamic vision sensor, since it can only detect moving targets, the calibration board needs to be constantly moved during shooting. First, the event stream is accumulated until dozens of frames of images are generated during the movement process. Then, Zhang's calibration method is used to calibrate the frame images, remove the images with large recognition deviation of calibration points, and make the re-projection error drop within an acceptable range of 1, to get the internal focal length parameters of the dynamic vision sensor.

[0050] After obtaining the internal parameters, the external parameters are calculated. According to the principle of binocular stereo calibration, the position translation relationship between the left and right cameras is obtained from the multiple images generated at the same time. This relationship is represented by a translation matrix. The internal and external parameters obtained by calibration are used for binocular distance measurement in the fourth step.

[0051] Step 3: The images generated by the event encoding of the dynamic vision sensor and the images of the industrial camera are respectively subjected to target recognition through deep learning algorithm, and the image area of the target center is marked to obtain the image coordinates of the target area center.

[0052] The industrial camera image set and the DVS image set are obtained by shooting the manta ray robot motion form using the heterogeneous binocular vision system. The robot target of each image is labeled to generate the corresponding label file, which records the minimum and maximum values of the target pixel coordinates in the X and Y directions of the image, the name category label and the recognition difficulty. The image set and the labeling information are combined to obtain the industrial camera dataset and the DVS dataset about the manta ray robot, respectively. The dataset is divided into training set and test set in the ratio of 9:1, which is used for deep learning. The target category is manta ray robot.

[0053] The dataset is sent to the neural network training, and the network model is YOLOv3. The framework is divided into Darknet-53 structure, feature layer fusion structure and classification detection structure. After the image is input into Darkenet-53, three branches are obtained. Up-sampling and down-sampling are performed to make the feature maps of the same size. Then, after stacking, fusing and corresponding convolution operations, three final feature maps are obtained, which are 1 / 32, 1 / 16 and 1 / 8 of the original image, respectively. Then, the three feature maps are fused.

[0054] The DVS dataset and the industrial camera dataset are respectively trained through the neural network for 30 iterations to obtain two trained models. These two models are used to predict the position area of the manta ray robot target in the DVS and industrial camera images and mark them with rectangular frames. The prediction of target area position and confidence of heterogeneous binocular is shown in the followingFigure 1 The left image is a dynamic vision sensor event image, and the right image is an industrial camera color image.

[0055] The predicted target region has been marked with a rectangular box. The X and Y coordinates of the upper-left corner and the lower-right corner of the box are used to calculate the median as the feature center coordinates of the target. The feature center coordinates of the two images are P l and P r respectively. The X and Y coordinates of P l and P r are used as the basis for the next step of binocular lateral disparity calculation.

[0056] Step 4: Using the feature center coordinates of the target obtained in step 3, and the internal and external parameters of the two cameras obtained in step 2, the binocular distance estimation algorithm is used to calculate the distance between the target manta ray robot and the heterogeneous binocular camera.

[0057] The heterogeneous binocular space model is shown in the figure, where the left DVS coordinate system and the world coordinate system are coincident in three axes, and the right is the industrial camera coordinate system. The imaging plane is parallel to the XY plane of the world coordinate system, and the imaging planes of the two cameras are coplanar. The X, Y, and Z axes of the two camera coordinate systems are parallel, and the principal axes are parallel. The origin positions of the two cameras are different. The feature point P(X p , Y p , Z p ) of the target in space has different X and Y coordinates in the left and right imaging planes, while the Z coordinate is basically consistent in space. The principle of heterogeneous binocular space target imaging and distance measurement is shown in the figure. Figure 2 According to the difference X l -X r between the X coordinates of the two feature centers P l and P r of the recognized target, which is the disparity, and the focal lengths f1 and f2 of the two cameras obtained in step 2, as well as the baseline distance B between the principal axes of the two camera centers, the front-to-back distance Z between the target manta ray robot and the heterogeneous binocular camera can be calculated by the following formula:

[0058]

[0059] This distance Z is the spatial distance relationship between the head heterogeneous binocular camera of the rear manta ray robot and the front target manta ray robot, i.e. the distance measurement result is obtained.

[0060] The embodiment of the isometric binocular vision system includes an underwater industrial camera, a dynamic vision sensor (DVS), a waterproof camera shell, and a waterproof data line; the DVS is placed in the shell and fixed with the underwater industrial camera by using a rigid support; the rigid support is horizontally placed in water to make the two cameras level, and the waterproof data lines of the two cameras are connected with a computer to transmit data; and the data of the DVS is encoded to generate an image by using an event frequency-based method.

[0061] In the embodiment, the DVS is located on the left side, and the underwater industrial camera is located on the right side.

[0062] Compared with a common isometric binocular system composed of two industrial cameras, the embodiment of the isometric binocular system composed of a DVS and an industrial camera has the characteristics of capturing a moving target, a high dynamic range, and low data redundancy at one end of the DVS, reduces underwater motion blur of the image of the industrial camera, eliminates the influence of a redundant static background of the image of the industrial camera, improves the clarity of a moving target robot in the image, reduces the difficulty of target recognition of an image algorithm, makes the accuracy of target recognition higher, and improves the accuracy of subsequent binocular positioning. Meanwhile, the structure retains an industrial camera on the other side, so that the isometric binocular system can not only recognize a moving target in the DVS, but also recognize a static target in the industrial camera.

[0063] Embodiment 1

[0064] The embodiment of the isometric binocular manta robot target recognition and ranging method has the following steps:

[0065] (1) An underwater isometric binocular experimental platform is made. A waterproof camera shell and a waterproof data line are made, the DVS is placed in the shell and fixed with the underwater industrial camera by using a rigid support, the fixed distance is 250 mm, the support is horizontally placed in water to make the two cameras level, the waterproof data lines of the two cameras are connected with a computer to transmit data, and the data of the DVS is encoded to generate an image by using an event frequency-based method.

[0066] (2) The internal and external parameters of the isometric binocular system are calibrated. The isometric binocular camera continuously captures a chessboard calibration plate, and the internal parameter matrix of the industrial camera is obtained by using the calibration method described above.

[0067]

[0068] The internal parameter matrix of the DVS is obtained.

[0069]

[0070] The main results show that the focal length of the industrial camera is f1 of 565 and the focal length of the DVS is f2 of 276. The baseline distance B of the binocular external parameters is 254mm.

[0071] (3) Using heterogeneous binoculars to identify the manta ray robot target. The manta ray robot was controlled by a host computer to perform flapping and turning movements within the binocular field of view, first moving from near to far, then from far to near, with a movement distance of approximately 1 meter. Images and event stream data from both cameras were continuously collected by the computer to create a heterogeneous binocular dataset of the target robot. Using the target recognition method described above, the position of the target region in both types of images was predicted, and the X and Y coordinates of the target feature center point in the images were extracted.

[0072] A target robot distance measurement experiment was conducted. The disparity was calculated using the X-direction coordinates of the binocular feature points obtained from the previous target recognition step. Combined with the focal lengths f1 and f2 of the two cameras obtained from the above calibration and the binocular baseline distance B, the front-to-back distance Z between the manta ray robot and the heterogeneous binocular platform was obtained using the distance calculation formula (2). Simultaneously, the actual distance between the manta ray robot and the binocular platform was measured with a measuring tape for comparison. The distance calculation and measurement results curves over 40 seconds are shown in the attached figure. Figure 3 As shown, the top curve represents the distance calculated by the method of this invention, the middle curve represents the measured distance, and the bottom curve represents the error value. It can be seen that the calculated distance matches the characteristics of the manta ray robot swimming from near to far and then from far to near, with a distance range of approximately 1 meter. The error between the calculated and measured values ​​is small. Therefore, the heterogeneous binocular manta ray robot target recognition and ranging method of this invention is effective.

[0073] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A method for target recognition and ranging of a robotic manta ray with hetero-binocular vision, characterized in that The specific steps are as follows: Step 1: encode the dynamic visual sensor raw event stream data into images for subsequent calibration and image recognition; Step 2: calibrate the camera parameters of the multiple images generated by the dynamic visual sensor encoding and the multiple images of the industrial camera, to obtain the focal length of the two cameras and the positional relationship between the two cameras; Step 3: perform target recognition on the images generated by the dynamic visual sensor event encoding and the industrial camera images through deep learning algorithms, and mark the image area of the target to obtain the image coordinates of the target area center; Step 4: using the binocular distance estimation algorithm, calculate the distance between the target manta ray robot and the heterogeneous binocular camera based on the target feature center coordinates obtained from the binocular images in step 3, and the internal and external parameters of the two cameras obtained in step 2.

2. The method according to claim 1, wherein the method is used for target recognition and ranging of a robotic manta ray with heteronymous binocular vision. In step 1, the frequency-based encoding method is used to encode the dynamic visual sensor raw event stream data into images.

3. The method of claim 2, wherein the method further comprises: In step 1, each event of the dynamic visual sensor event stream data consists of four parameters, namely t, x, y, and p, including a microsecond-level timestamp t, position coordinates (x, y) in the image, and event change polarity p; using the event frequency-based method, the data is encoded into images by dividing the event stream data at a fixed millisecond-level time interval, i.e. 30 milliseconds, accumulating all events within the time interval, mapping and normalizing the event frequency value to obtain the pixel value of each image; the more the target moves in the background, the higher the event frequency, and the higher the frequency, the higher the brightness of the corresponding pixel point in the image; the mapping and normalization calculation formula is: （1） Where x is the total number of events occurring at a certain pixel point within a fixed time interval, and σ(x) is the pixel value of a certain pixel point between 0 and 255 in the image; After calculating the pixel brightness value, different colors are used to represent different polarity events, with green dots representing positive polarity events with increasing brightness, and red dots representing negative polarity events with decreasing brightness; by formula (1), the pixel value of each point in the image is obtained from the event frequency, and the color of different polarity events is marked, so that the event stream is encoded into a color image.

4. The method according to claim 3, wherein the method is used for target recognition and ranging of a robotic manta ray with heteronymous binocular vision. In step 2, the internal and external parameter calibration method of the industrial camera and the dynamic visual sensor is as follows: For the industrial camera, use a checkerboard calibration board to take dozens of calibration board pictures at different angles and different sizes, and use Zhang's calibration method to obtain the internal focal length parameter of the industrial camera; For the dynamic visual sensor, since it can only detect moving targets, the calibration board needs to be constantly moved during shooting. First, use the encoding method in the previous step to continuously accumulate event streams until dozens of images are generated, then use Zhang's calibration method on the frame images to remove images with large target recognition deviation, so that the re-projection error is reduced to within an acceptable range of 1, and the internal focal length parameter of the dynamic visual sensor is obtained; After the internal parameters are obtained, the external parameters are calculated, and the position translation relationship between the binocular cameras is obtained from the left and right images generated at the same time according to the binocular stereo calibration principle, and the relationship is represented by a translation matrix; the obtained camera internal parameters and external parameters are used for binocular distance measurement in step 4.

5. The method of claim 4, wherein the method further comprises: In step 3, the manta ray robot motion form is photographed using the heterogeneous binocular vision system to obtain an industrial camera image set and a DVS image set; each image of the robot target is labeled to generate a corresponding label file, record the minimum and maximum values of the target in the X and Y direction pixel coordinates of the image, the name category label and the recognition difficulty, and use the image set and the labeling information to obtain the industrial camera data set and the DVS data set about the manta ray robot, respectively; the data set is divided into a training set and a test set according to a 9:1 ratio for deep learning, and the category of the training target is the manta ray robot; The data set is input into the neural network for training, and the network model is YOLOv3, which has a framework including a Darknet-53 structure, a feature layer fusion structure and a classification detection structure; after the image is input into Darkenet-53, three branches are obtained, up-sampling and down-sampling are performed to make the feature maps have the same size, then stacking, fusion and corresponding convolution operations are performed to obtain three final feature maps, which are 1 / 32, 1 / 16 and 1 / 8 of the original image, respectively, and then the three feature maps are fused; The DVS data set and the industrial camera data set are iteratively trained for 30 rounds through the neural network to obtain two trained models; the two models are used to predict the position area of the manta ray robot target in the DVS and industrial camera images and mark the position area with a rectangular frame. The predicted target region has been marked with a rectangular frame, and the X and Y coordinates of the upper left corner and the lower right corner of the frame are used to calculate the median as the feature center coordinates of the target. The feature center coordinates of the two images are P l and P r respectively, and the X and Y coordinate values of P l and P r are used as the basis for the next step of binocular horizontal disparity calculation.

6. The method of claim 5, wherein the method further comprises: The spatial model of the heterogeneous binocular is that the left DVS coordinate system and the world coordinate system are coincided with the three axes, and the right is the industrial camera coordinate system, the imaging surface is the X axis direction, the imaging surface is the Y axis direction, and the forward direction is the Z axis direction, and the spatial coordinate system is constructed.

7. The method of claim 6, wherein the method further comprises: In the step 4, the imaging planes of the two cameras are parallel to the XY plane of the world coordinate system, the imaging planes of the industrial camera and the DVS are coplanar, the X, Y and Z axes of the two camera coordinate systems are parallel, the main optical axes are parallel, and the origins of the two cameras are different; the feature point P (X p , Y p , Z p ) of the target in space has different X and Y coordinates in the left and right imaging planes of the two cameras, and the Z coordinate is basically consistent in space; According to the two feature centers P of the target of binocular recognition l and P r The difference X of the X coordinate of l -X r That is, the parallax, and step 2 calibrates the focal length f1 and f2 of the two cameras, and the baseline distance B between the centers of the two cameras, that is, the front and back distance Z between the target manta ray robot and the heterogeneous binocular camera can be calculated by the following formula: （2） The distance Z is the spatial distance relationship between the rear manta ray robot head heterogeneous binocular camera and the front target manta ray robot, that is, the distance measurement result is obtained.

8. A vision system of anisoholographic vision for performing the method of target recognition and ranging of anisoholographic manta ray robot according to any one of claims 1-7, characterized in that it comprises: The underwater industrial camera, the dynamic vision sensor DVS, the waterproof camera shell and the waterproof data line are included; the dynamic vision sensor DVS is placed in the shell and connected and fixed with the underwater industrial camera using a rigid support; The rigid support is horizontally placed in water to make the two cameras level, the waterproof data lines of the two cameras are connected with the computer to transmit data; the data of the dynamic vision sensor DVS is encoded to generate an image using an event frequency-based method.

9. The anisallopia visual system of claim 8, wherein: The fixed distance between the dynamic vision sensor DVS and the underwater industrial camera is 250 mm.

10. The anisallopia visual system of claim 8, wherein: The dynamic vision sensor DVS is located on the left side, and the underwater industrial camera is located on the right side.

Images