A mobile explosive disposal device and method based on immersive perception and bionic control

Abstract

The present invention relates to a mobile bomb disposal device and method based on immersive perception and bionic control, which belongs to the field of virtual reality technology and solves the problems of poor safety and low efficiency of bomb disposal methods in the prior art. The device includes a bionic control device, wherein a camera device is used to collect facial images of operators; a processor is used to process the facial images to obtain binocular eyeball information, and send it to a controller mounted on a mobile bomb disposal robot through a wireless communication module; the controller adjusts the pan / tilt and variable baseline binocular camera on the mobile bomb disposal robot according to the binocular eyeball information, and then collects a first field of view stereoscopic image of the bomb disposal scene that is the same as the operator's perspective; the operator controls the mobile bomb disposal robot to perform bomb disposal based on the first field of view stereoscopic image of the bomb disposal scene. This method can effectively enhance the operator's immersive sense of reality of the bomb disposal scene, thereby improving the operator's bomb disposal efficiency and safety.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a mobile bomb disposal device and method based on immersive perception and bionic control. Background Art

[0002] In the prior art, bomb disposal is mainly carried out in the following ways: first, the operator wears explosion-proof clothing to manually dispose of bombs, or the operator wears explosion-proof clothing to dispose of bombs by manipulators; second, based on virtual reality technology, the RGB-D camera on the bomb disposal robot collects images of the bomb disposal site, and the operator performs bomb disposal operations based on the images of the bomb disposal site.

[0003] There are at least the following defects in the prior art. First, the operators wear explosion-proof clothing to manually defuse bombs or the operators wear explosion-proof clothing to defuse bombs through manipulators, which is unsafe. Second, based on virtual reality technology, the RGB-D camera on the bomb disposal robot end is used to collect images of the bomb disposal site. The baseline of the RGB-D camera is fixed. Wearing virtual reality equipment for a long time to observe the bomb disposal site through the RGB-D camera can easily cause visual fatigue and poor immersive experience. It is also easy to reduce the operator's operating accuracy, thereby reducing the efficiency and safety of bomb disposal. Summary of the invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide a mobile bomb disposal device and method based on immersive perception and bionic control, so as to solve the problems of poor safety and low efficiency of existing bomb disposal methods.

[0005] On the one hand, the present invention provides a mobile bomb disposal device based on immersive perception and bionic control, including a mobile bomb disposal robot, a bionic control device and a wireless communication module;

[0006] The bionic control device comprises:

[0007] Camera equipment, used to capture facial images of operators;

[0008] A processor, configured to process the operator's facial image to obtain binocular eyeball information, and send the binocular eyeball information to a controller mounted on the mobile bomb disposal robot through the wireless communication module;

[0009] The controller adjusts the pan / tilt and the variable baseline binocular camera on the mobile bomb disposal robot according to the binocular eye information, thereby acquiring a first field of view stereo image of the bomb disposal scene that is the same as the operator's perspective;

[0010] The operator controls the mobile bomb disposal robot to perform bomb disposal based on the first field of view stereoscopic image of the bomb disposal site.

[0011] Furthermore, the binocular eyeball information includes binocular eyeball posture information, binocular eyeball distance and binocular eyeball pupil size, and the binocular eyeball posture information includes the real-time center coordinates of the eyeballs and the rotation azimuth.

[0012] Furthermore, the processor obtains eyeball position information of both eyes in the following manner:

[0013] Processing the operator's facial image using an image detection algorithm to determine the positions of both eyes in the operator's facial image;

[0014] The regional image corresponding to the positions of the two eyes in the operator's facial image is converted into two grayscale images corresponding to the two eyes, and the gradient of each pixel point in each grayscale image is calculated, and the position of the point where the straight lines where the gradient direction intersects the most is taken as the pixel coordinate position of the eyeball center;

[0015] According to the conversion relationship between the calibrated pixel coordinate system and the world coordinate system, the pixel coordinates of the eyeball center are converted into the real-time center coordinates of the eyeball;

[0016] The eyeball rotation azimuth is calculated based on the calibrated eyeball center coordinates when the operator is looking straight ahead and the converted eyeball real-time center coordinates.

[0017] Furthermore, the pixel coordinates of the center of the eyeball are calculated by the following formula using an optimization method:

[0018]

[0019]

[0020]

[0021] Among them, X i =[u i ,v i ] T is the pixel coordinate of any point in the grayscale image corresponding to the eye, c = [u c ,v c ] T is the pixel coordinate of the eyeball center, d i is the normalized displacement vector, g i For X i The gradient vector at the point, I(u i ,v i ) is the grayscale function, and N represents the number of pixels in the grayscale image.

[0022] Furthermore, the processor obtains the pupil size of both eyes in the following manner:

[0023] Clustering the grayscale image according to the grayscale values ​​of the grayscale image using a Gaussian mixture model and an EM algorithm, and extracting pupil contour pixel coordinates according to the clustered grayscale values;

[0024] The pupil outline pixel coordinates are converted into world coordinates and fitted into a circle, and the diameter of the circle is calculated to be the pupil size of the eyeball.

[0025] Furthermore, the conversion relationship between the calibrated pixel coordinate system and the world coordinate system is:

[0026]

[0027] Wherein, [u, v] is the coordinate of any point in the pixel coordinate system, [x w ,y w ,z w ] is the coordinate of any point in the world coordinate system, γ, f x 、f y , u0, v0, are the internal parameters of the variable baseline binocular camera, [u0, v0] is the pixel coordinate of the eye center when the operator is looking straight ahead, R 3×3 , T 3×3 They represent the rotation matrix and translation matrix from the variable baseline stereo camera coordinate system to the world coordinate system respectively.

[0028] Furthermore, the mobile bomb disposal robot includes a mobile chassis, a mechanical arm, a gripper, a pan / tilt platform and a variable baseline binocular camera;

[0029] The controller performs closed-loop control on the angle of the pan / tilt platform according to the eyeball posture information of the binocular eyes;

[0030] The controller performs closed-loop control on the baseline of the variable baseline binocular camera according to the binocular eye distance;

[0031] The controller performs closed-loop control on the focal length of the variable baseline binocular camera according to the pupil size of the binocular eyeballs;

[0032] The binocular eye distance and the baseline distance of the variable baseline binocular camera are in a first preset ratio, the pupil size and the focal length of the variable baseline binocular camera are in a second preset ratio, and the first preset ratio is the same as the second preset ratio.

[0033] Furthermore, the bionic control device also includes smart 3D glasses and an electromyographic sensing module;

[0034] The variable baseline binocular camera is used to collect a first field of view stereoscopic image of the bomb disposal site, the first field of view stereoscopic image includes a left eye image and a right eye image, and the left eye image and the right eye image are respectively transmitted to the left eye display screen and the right eye display screen of the smart 3D glasses through the wireless communication module;

[0035] The operator performs a grabbing operation based on the first field of view stereoscopic image of the bomb disposal site. The electromyography sensing module is used to sense the operator's grabbing operation and generate a grabbing control instruction, which is then transmitted to the controller through the wireless communication module. The controller controls the robotic arm and the gripper to perform a grabbing operation to dispose of the bomb according to the grabbing control instruction.

[0036] Furthermore, the camera device is installed at the top center of the smart 3D glasses.

[0037] On the other hand, the present invention provides a mobile bomb disposal method based on immersive perception and bionic control, using the aforementioned mobile bomb disposal device;

[0038] Using a camera to capture the operator's facial image;

[0039] Processing the operator's facial image to obtain eyeball information of both eyes, and sending the eyeball information of both eyes to a controller mounted on the mobile bomb disposal robot through a wireless communication module;

[0040] Using a controller to adjust the pan / tilt and variable baseline binocular camera on the mobile bomb disposal robot according to the binocular eye information, thereby acquiring a first field of view stereoscopic image of the bomb disposal scene that is the same as the operator's perspective;

[0041] The mobile bomb disposal robot is controlled to perform bomb disposal based on the first field of view stereoscopic image of the bomb disposal site.

[0042] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0043] 1. The mobile bomb disposal device and method based on immersive perception and bionic control proposed in the present invention collects the posture information, distance information and pupil size of the operator's eyeballs through a camera, and performs closed-loop control on the pan / tilt of the mobile bomb disposal robot and the focal length and baseline of the variable baseline binocular camera based on the information, so that the first view of the bomb disposal scene taken by the variable baseline binocular camera is consistent with the perspective of the operator's eyes. On the one hand, the immersive experience of the operator is improved. In addition, the first view of the bomb disposal scene collected by the binocular camera includes a left eye image and a right eye image, which are presented to the operator in the form of a stereo image through the left and right display screens of the smart 3D glasses, respectively, to enhance the sense of reality. On the other hand, the baseline of the variable baseline binocular camera can be adjusted according to the eye dynamics of the operator, which can largely avoid the operator's eye fatigue and improve the operator's bomb disposal accuracy and efficiency.

[0044] 2. The mobile bomb disposal device and method based on immersive perception and bionic control proposed in the present invention can realize the operation of the operator from the rear through immersive bionic control, that is, the control of the mobile bomb disposal robot to perform tasks such as grasping and bomb disposal, thereby improving the safety of the operator's bomb disposal, and effectively solving the problem of difficult collaboration between the bomb disposal robot and the operator by synchronizing the perspective of the variable baseline binocular camera with the perspective of the operator.

[0045] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0047] Figure 1 A schematic diagram of a mobile explosive disposal device based on immersive perception and bionic control according to an embodiment of the present invention;

[0048] Figure 2 Schematic diagram of the eyeball center gradient vector field according to an embodiment of the present invention;

[0049] Figure 3 The present invention is a flowchart of closed-loop control of a pan / tilt and a variable baseline binocular camera according to binocular eye information according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0051] A specific embodiment of the present invention discloses a mobile bomb disposal device based on immersive perception and bionic control. Figure 1 As shown, the device includes a mobile bomb disposal robot, a bionic control device and a wireless communication module.

[0052] Preferably, the bionic control device includes a camera device, a processor and smart 3D glasses. The camera device is used to collect facial images of the operator; preferably, the camera device is installed at the top center of the smart 3D glasses. In addition, two cameras can be installed on the smart 3D glasses to collect images of the operator's left eye and right eye respectively. Specifically, the camera device can be a miniature camera, etc. The processor is used to process the facial image of the operator to obtain eyeball information of both eyes, and send it to the controller mounted on the mobile bomb disposal robot through the wireless communication module.

[0053] The controller adjusts the pan / tilt and variable baseline binocular camera on the mobile bomb disposal robot according to the information from the binocular eyes, so that the view of the bomb disposal site captured by the binocular camera is consistent with the perspective of the operator's eyes, thereby enhancing the operator's sense of reality, that is, acquiring a first-field stereo image of the bomb disposal site that is the same as the operator's perspective.

[0054] The controller transmits the collected first field of view stereoscopic image to the smart 3D glasses worn by the operator through the wireless communication module. The operator issues corresponding grabbing and bomb disposal instructions based on the first field of view stereoscopic image of the bomb disposal site, and then controls the mobile bomb disposal robot to perform bomb disposal.

[0055] Preferably, the wireless communication module may be a cellular mobile network module, a Wifi module, or a Bluetooth module. Specifically, the operator may select one according to the distance or the environment.

[0056] Preferably, the binocular eyeball information includes binocular eyeball posture information, binocular eyeball distance and binocular eyeball pupil size, and the binocular eyeball posture information includes the real-time center coordinates of the eyeballs and the rotation azimuth.

[0057] Preferably, the pixel coordinate system and the world coordinate system are calibrated. Specifically, the pixel coordinates [u0, v0] of the center of the operator's eyeball when looking straight ahead are determined, and the coordinate transformation relationship from the pixel coordinate system to the world coordinate system is established based on the internal and external parameters of the variable baseline binocular camera. Specifically, as follows:

[0058]

[0059] Wherein, [u, v] is the coordinate of any point in the pixel coordinate system, [x w ,y w ,z w ] is the coordinate of any point in the world coordinate system, γ, f x 、f y , u0, v0, are the internal parameters of the variable baseline binocular camera, which have been calibrated before leaving the factory. [u0, v0] is the pixel coordinate of the center of the eyeball when the operator is looking straight ahead, R 3×3 , T3×3 are the external parameters of the variable baseline stereo camera, representing the rotation matrix and translation matrix from the variable baseline stereo camera coordinate system to the world coordinate system respectively.

[0060] Preferably, the processor obtains the eyeball position information of both eyes in the following manner:

[0061] The operator's facial image is processed using an image detection algorithm to determine the positions of both eyes in the operator's facial image.

[0062] The regional image corresponding to the position of both eyes in the operator's facial image is converted into two grayscale images corresponding to both eyes, and the gradient of each pixel point in each grayscale image is calculated. The position of the point where the straight line where the gradient direction intersects the most is taken as the pixel coordinate position of the eyeball center. Specifically, the eyeball center gradient vector field is as follows: Figure 2 shown.

[0063] According to the conversion relationship between the calibrated pixel coordinate system and the world coordinate system, the pixel coordinates of the eye center are converted into the real-time center coordinates of the eye.

[0064] The eyeball rotation azimuth is calculated based on the eyeball center coordinates calibrated when the operator is looking straight ahead and the converted eyeball real-time center coordinates. Specifically, a direction vector is formed based on the eyeball center coordinates and the converted eyeball real-time center coordinates, and the eyeball rotation azimuth is determined based on the direction vector.

[0065] Preferably, the pixel coordinates of the center of the eyeball are calculated by the following formula using an optimization method:

[0066]

[0067]

[0068]

[0069] Among them, X i =[u i ,v i ] T is the pixel coordinate of any point in the grayscale image corresponding to the eye, c = [u c ,v c ] T is the pixel coordinate of the eyeball center, d i is the normalized displacement vector, g i For X i The gradient vector at the point, I(u i ,v i ) is the grayscale function.

[0070] Preferably, the processor obtains the pupil size of both eyes by the following method:

[0071] The Gaussian mixture model and EM algorithm are used to cluster the grayscale image according to its grayscale value, and the pupil outline pixel coordinates are extracted according to the clustered grayscale value. Since the grayscale value of the pupil in the eyeball is quite different from the grayscale value of the part outside the pupil, the boundary between the pupil and the part outside the pupil can be determined according to the grayscale value, and the pupil outline pixel coordinates can be determined according to the boundary.

[0072] The pupil outline pixel coordinates are converted into world coordinates and fitted into a circle. The diameter of the circle is calculated as the pupil size of the eyeball.

[0073] Preferably, the distance b between the operator's eyeballs is calculated according to the real-time center coordinates of the operator's eyeballs by the following formula:

[0074] b=||(X 1w -X 2w )||2,

[0075] Among them, X 1w =[x 1w ,y 1w ,z 1w ] T , X 2w =[x 2w ,y 2w ,z 2w ] T They are respectively the real-time center coordinates of the operator’s left eye and right eye, which are world coordinates.

[0076] Preferably, the mobile bomb disposal robot includes a mobile chassis, a mechanical arm, a gripper, a pan-tilt head and a variable baseline binocular camera. The controller controls the pan-tilt head and the variable baseline binocular camera in a closed loop according to the binocular eye information. Figure 3 As shown, specifically:

[0077] The controller performs closed-loop control on the angle of the pan-tilt according to the eyeball position information of both eyes. The pan-tilt has a servo mechanism, which tracks the pan-tilt after receiving the eyeball rotation angle instruction, so as to realize the servo tracking of the rotation of the two eyeballs of the human eye behind by the two binocular lenses of the front robot.

[0078] The controller performs closed-loop control of the baseline of the variable baseline binocular camera according to the binocular eye distance. The two cameras of the binocular camera on the front mobile bomb disposal robot are controlled by the servo motor so that they can move on the roller screw. The servo motor tracks the binocular camera after receiving the binocular eye distance command, realizing the servo tracking of the baseline of the binocular camera of the front robot.

[0079] The controller performs closed-loop control on the focal length of the variable baseline binocular camera according to the pupil size of the binocular eyeballs. Specifically, after the camera receives the corresponding instruction of the pupil size, it can automatically complete the closed-loop control of the focal length.

[0080] Preferably, the distance between the eyeballs of both eyes and the baseline distance of the variable baseline binocular camera are in a first preset ratio, the pupil size and the focal length of the variable baseline binocular camera are in a second preset ratio, and the first preset ratio is the same as the second preset ratio. According to the above settings, the operator's binocular viewing angle can be ensured to be consistent with the viewing angle of the variable baseline binocular camera, thereby enhancing the operator's immersive experience.

[0081] Preferably, the bionic control device also includes an electromyographic sensing module. Specifically, the electromyographic sensing module may be an electromyographic sensor.

[0082] Preferably, the variable baseline binocular camera is used to collect a first field of view stereoscopic image of the bomb disposal site, the first field of view stereoscopic image includes a left eye image and a right eye image, the left eye image and the right eye image are respectively transmitted to the left eye display screen and the right eye display screen of the smart 3D glasses through the wireless communication module, and the left eye image and the right eye image on the left eye display screen and the right eye display screen are merged into the first field of view stereoscopic image.

[0083] The operator performs grabbing operations based on the first field of view stereoscopic image of the bomb disposal site. The electromyography sensing module is used to sense the operator's grabbing operations and generate grabbing control instructions, which are then transmitted to the controller through the wireless communication module. The controller controls the robotic arm and gripper to perform grabbing operations to dispose of bombs based on the grabbing control instructions.

[0084] Another embodiment of the present invention discloses a mobile bomb disposal method based on immersive perception and bionic control, which utilizes the aforementioned mobile bomb disposal device.

[0085] Since this embodiment and the above-mentioned device embodiment are based on the same working principle, the repeated parts can be referred to the above-mentioned device embodiment and will not be described again here.

[0086] Specifically, the method includes:

[0087] The facial image of the operator is collected by using a camera.

[0088] The operator's facial image is processed to obtain eyeball information of both eyes, and is sent to a controller mounted on the mobile bomb disposal robot through a wireless communication module.

[0089] The controller is used to adjust the pan / tilt and variable baseline binocular camera on the mobile bomb disposal robot according to the information of the binocular eyeballs, so as to acquire the first field of view stereo image of the bomb disposal scene with the same perspective as the operator.

[0090] The mobile bomb disposal robot is controlled to carry out bomb disposal based on the first field of view stereoscopic image of the bomb disposal scene.

[0091] The mobile bomb disposal device and method based on immersive perception and bionic control disclosed in the embodiment of the present invention first collects the posture information, distance information and pupil size of the operator's eyeballs through a camera, and performs closed-loop control on the pan / tilt of the mobile bomb disposal robot section and the focal length and baseline of the variable baseline binocular camera based on the information, so that the first view of the bomb disposal scene taken by the variable baseline binocular camera is consistent with the perspective of the operator's eyes. On the one hand, the immersive experience of the operator is improved. In addition, the first view of the bomb disposal scene collected by the binocular camera includes a left eye image and a right eye image, which are presented to the operator in the form of a stereo image through the left and right display screens of the smart 3D glasses, respectively, to enhance the sense of reality. On the other hand, the baseline of the variable baseline binocular camera can be adjusted according to the eye dynamics of the operator, which can largely avoid the operator's eye fatigue and improve the operator's bomb disposal accuracy and efficiency. Secondly, the mobile bomb disposal device and method based on immersive perception and bionic control proposed in the present invention can realize the operation of the operator from the rear through immersive bionic control, that is, the control of the mobile bomb disposal robot to perform tasks such as grasping and bomb disposal, thereby improving the safety of the operator's bomb disposal, and effectively solving the problem of difficulty in collaboration between the bomb disposal robot and the operator by synchronizing the perspective of the variable baseline binocular camera with the perspective of the operator.

[0092] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0093] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A mobile explosive disposal device based on immersive perception and bionic control, characterized in that: Including mobile bomb disposal robots, bionic control equipment and wireless communication modules; The bionic control device comprises: Camera equipment, used to capture facial images of operators; A processor, configured to process the operator's facial image to obtain binocular eyeball information, and send the binocular eyeball information to a controller mounted on the mobile bomb disposal robot through the wireless communication module; The binocular eyeball information includes binocular eyeball posture information, binocular eyeball distance and binocular eyeball pupil size, and the binocular eyeball posture information includes the real-time center coordinates of the eyeballs and the rotation azimuth; The controller adjusts the pan / tilt and the variable baseline binocular camera on the mobile bomb disposal robot according to the binocular eye information, thereby acquiring a first field of view stereo image of the bomb disposal scene that is the same as the operator's perspective; An operator controls the mobile bomb disposal robot to perform bomb disposal based on the first field of view stereoscopic image of the bomb disposal site; The mobile bomb disposal robot comprises a mobile chassis, a mechanical arm, a gripper, a pan / tilt platform and a variable baseline binocular camera; The controller performs closed-loop control on the angle of the pan-tilt according to the eyeball posture information of the two eyes, wherein the pan-tilt is provided with a servo mechanism, and tracks the pan-tilt after receiving the eyeball rotation angle instruction, so as to realize the servo tracking of the rotation of the two eyeballs of the human eye behind by the two binocular lenses of the front robot; The controller performs closed-loop control on the baseline of the variable baseline binocular camera according to the binocular eyeball distance, specifically including controlling two cameras of the binocular camera on the front mobile bomb disposal robot through a servo motor so that they can move on the roller screw, and the servo motor tracks it after receiving the binocular eyeball distance instruction; The controller performs closed-loop control on the focal length of the variable baseline binocular camera according to the pupil size of the binocular eyeballs, specifically comprising automatically completing the closed-loop control of the focal length after the camera receives a corresponding instruction on the pupil size; The binocular eye distance and the baseline distance of the variable baseline binocular camera are in a first preset ratio, the pupil size and the focal length of the variable baseline binocular camera are in a second preset ratio, and the first preset ratio is the same as the second preset ratio.

2. The mobile explosive disposal device according to claim 1, characterized in that: The processor obtains binocular eyeball position information in the following manner: Processing the operator's facial image using an image detection algorithm to determine the positions of both eyes in the operator's facial image; The regional image corresponding to the positions of the two eyes in the operator's facial image is converted into two grayscale images corresponding to the two eyes, and the gradient of each pixel point in each grayscale image is calculated, and the position of the point where the straight lines where the gradient direction intersects the most is taken as the pixel coordinate position of the eyeball center; According to the conversion relationship between the calibrated pixel coordinate system and the world coordinate system, the pixel coordinates of the eyeball center are converted into the real-time center coordinates of the eyeball; The eyeball rotation azimuth is calculated based on the calibrated eyeball center coordinates when the operator is looking straight ahead and the converted eyeball real-time center coordinates.

3. The mobile explosive disposal device according to claim 2, characterized in that: The optimization method is used to calculate the pixel coordinates of the eyeball center using the following formula: Among them, X i =[u i ,v i ] T is the pixel coordinate of any point in the grayscale image corresponding to the eye, c = [u c ,v c ] T is the pixel coordinate of the eyeball center, d i is the normalized displacement vector, g i For X i The gradient vector at the point, I(u i ,v i ) is the grayscale function, and N represents the number of pixels in the grayscale image.

4. The mobile explosive disposal device according to claim 2, characterized in that: The processor obtains the pupil size of both eyes in the following manner: Clustering the grayscale image according to the grayscale values ​​of the grayscale image using a Gaussian mixture model and an EM algorithm, and extracting pupil contour pixel coordinates according to the clustered grayscale values; The pupil outline pixel coordinates are converted into world coordinates and fitted into a circle, and the diameter of the circle is calculated to be the pupil size of the eyeball.

5. The mobile explosive disposal device according to claim 2, characterized in that: The conversion relationship between the calibrated pixel coordinate system and the world coordinate system is: Among them, [u,v] is the coordinate of any point in the pixel coordinate system, [x w ,y w ,z w ] is the coordinate of any point in the world coordinate system, γ, f x 、f y , u0, v0, are the internal parameters of the variable baseline binocular camera, [u0, v0] is the pixel coordinate of the eye center when the operator is looking straight ahead, R 3×3 、T 3×3 They represent the rotation matrix and translation matrix from the variable baseline stereo camera coordinate system to the world coordinate system respectively.

6. The mobile explosive disposal device according to claim 1, characterized in that: The bionic control device also includes smart 3D glasses and an electromyographic sensing module; The variable baseline binocular camera is used to collect a first field of view stereoscopic image of the bomb disposal site, the first field of view stereoscopic image includes a left eye image and a right eye image, and the left eye image and the right eye image are respectively transmitted to the left eye display screen and the right eye display screen of the smart 3D glasses through the wireless communication module; The operator performs a grabbing operation based on the first field of view stereoscopic image of the bomb disposal site. The electromyography sensing module is used to sense the operator's grabbing operation and generate a grabbing control instruction, which is then transmitted to the controller through the wireless communication module. The controller controls the robotic arm and the gripper to perform a grabbing operation to dispose of the bomb according to the grabbing control instruction.

7. The mobile explosive disposal device according to claim 6, characterized in that: The camera device is installed at the top center of the smart 3D glasses.

8. A mobile bomb disposal method based on immersive perception and bionic control, characterized in that: Utilize the mobile explosive disposal device described in any one of claims 1 to 7; Using a camera to capture the operator's facial image; Processing the operator's facial image to obtain eyeball information of both eyes, and sending the eyeball information of both eyes to a controller mounted on the mobile bomb disposal robot through a wireless communication module; Using a controller to adjust the pan / tilt and variable baseline binocular camera on the mobile bomb disposal robot according to the binocular eye information, thereby acquiring a first field of view stereoscopic image of the bomb disposal scene that is the same as the operator's perspective; The mobile bomb disposal robot is controlled to perform bomb disposal based on the first field of view stereoscopic image of the bomb disposal site.

Images