Teleoperation visual auxiliary system based on multi-sensor fusion and operation method

Abstract

The invention discloses a teleoperation visual auxiliary system based on multi-sensor fusion and an operation method, and belongs to the technical field of attitude detection. The teleoperation visual auxiliary system comprises a remote augmented reality operation terminal, a multi-degree-of-freedom binocular holder and a remote communication subsystem; the multi-degree-of-freedom binocular holder transmits image and sound information to the remote augmented reality operation terminal through the remote communication subsystem, and the remote augmented reality operation terminal transmits head posture information to the multi-degree-of-freedom binocular holder through the remote communication subsystem. According to the invention, the improved multi-sensor fusion attitude positioning algorithm is adopted to obtain the head attitude information, the accumulative drift phenomenon is greatly eliminated, the obtained attitude information is more accurate and stable, and the robustness of the overall operation of the system is improved. According to the invention, a driver can obtain the audio and video images of the visual angle in the cab of the far-end engineering machinery indoors in real time and remotely control the far-end holder and the engineering machinery, so that the working efficiency is effectively improved and the risk of injury is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of posture detection, and in particular to a remote operation visual assistance system and an operation method based on multi-sensor fusion. Background Art

[0002] With the rapid development of my country's manufacturing and industry, more construction tasks in dangerous environments have been generated, which poses a challenge to the safety of operators. With the introduction of remote operation technology, more and more construction machinery, such as gantry cranes, excavators, loaders, etc., have begun to apply this technology. This requires our construction machinery to be more intelligent, efficient, stable, convenient and safe.

[0003] When construction machinery is working under remote control, the operator operates it in a safe place far away from the construction site. This places higher requirements on the convenience of the remote operation system. To meet the operational needs, the operator should also obtain the external images of the construction site in real time. To address this problem, a high real-time and mobile remote pan-tilt system is particularly critical. When facing difficult working conditions, it can provide stable real-time video and sound information with a sense of presence, making the operator feel as if he is inside the construction machinery, saving training costs. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a remote-operated visual assistance system and an operation method based on multi-sensor fusion, so that operators can remotely operate engineering machinery indoors, effectively improve work efficiency and reduce the risk of injury.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A remote operation visual assistance system based on multi-sensor fusion, including a remote augmented reality operation terminal, a multi-degree-of-freedom binocular gimbal and a remote communication subsystem;

[0007] The remote augmented reality operation terminal includes a movable remote operation console, an operator head posture acquisition module, a head-mounted augmented reality display, a panoramic audio system and a core processor;

[0008] The multi-DOF binocular gimbal includes an external data acquisition module, an embedded processor, a power supply voltage stabilization device, a motion control module and a portable loading platform;

[0009] The remote communication subsystem uses network communication, including wired network port communication, local area network wireless communication, and 5G wireless communication;

[0010] The multi-degree-of-freedom binocular gimbal transmits image and sound information to the remote augmented reality operation terminal through the remote communication subsystem, and the remote augmented reality operation terminal transmits head posture information to the multi-degree-of-freedom binocular gimbal through the remote communication subsystem.

[0011] A further improvement of the technical solution of the present invention is that the structure of the movable remote operating platform is the same as the structure of the operating platform; the movable remote operating platform communicates with the outdoor engineering machinery through a remote communication subsystem, remotely controls the outdoor engineering machinery, and at the same time collects status information of the engineering machinery, including posture, mechanical rotation angle, slope, and transmits it to the head-mounted augmented reality display.

[0012] A further improvement of the technical solution of the present invention is that: the operator head posture acquisition module includes a binocular depth camera, an inertial sensor and an industrial computer; the operator head posture acquisition module acquires the operator's head posture in real time and transmits it to the multi-degree-of-freedom binocular gimbal through a remote communication subsystem;

[0013] The binocular depth camera acquires depth information and video images and transmits them to the industrial computer;

[0014] The inertial sensor collects information from the magnetometer, accelerometer and gyroscope and transmits it to the industrial computer;

[0015] The industrial computer obtains real-time head posture data by using information collected by binocular depth cameras and inertial sensors; the industrial computer obtains the head posture by running a posture positioning algorithm based on multi-sensor fusion.

[0016] The further improvement of the technical solution of the present invention is that: the posture positioning algorithm based on multi-sensor fusion adopts a method of fusing a binocular depth camera and an inertial sensor with a visual sensor on the basis of the ORB-SLAM3 framework, and then introduces an improved point-line fusion module and a semantic segmentation and dynamic interference removal module to reduce the interference of texture information and remove dynamic interference; the posture estimation is improved by intelligent integration of point and line features; semantic and geometric filters are applied to dynamic points and lines in RGB images, a method based on deep learning is used to perform initial bounding box detection of dynamic objects, and an optical flow method is used to manage dynamic point and line elements inside and outside the box, and the original SLAM framework is input to obtain posture data, which specifically includes the following steps:

[0017] S1. Use the point-line fusion module to fully utilize the structure and texture information in the scene to extract a large number of feature points and lines, improve positioning accuracy, enhance robustness and reduce interference caused by dynamic noise;

[0018] S2. For the semantic segmentation and dynamic interference removal module, semantic segmentation detection is first used to identify dynamic objects in the scene; secondly, dynamic feature point line recognition is performed, and the Lucas-Kanade optical flow algorithm is applied to execute the extreme outer constraints based on the basic matrix; finally, a dynamic feature point removal mechanism based on the proportion of dynamic feature points in the bounding box of the detected dynamic target is combined, and the dynamic feature points that introduce erroneous estimation to the SLAM system are effectively filtered out by setting the proportion threshold.

[0019] A further improvement of the technical solution of the present invention is that a system client and an augmented reality processing module are run in the core processor, and the augmented reality processing module performs visual reality enhancement processing on the remote signal cockpit perspective video image transmitted from the multi-degree-of-freedom binocular gimbal, and transmits it to the head-mounted augmented reality display.

[0020] A further improvement of the technical solution of the present invention is that the augmented reality processing module performs visual reality augmentation processing on the remote signal cockpit perspective video image transmitted from the multi-degree-of-freedom binocular gimbal; the augmented processing process includes providing the operator with a prediction of the landing point of the engineering machinery arm and providing real-time distance information between the current position and the engineering target.

[0021] A further improvement of the technical solution of the present invention is that the external data acquisition module includes a sound acquisition module and a zoom camera module; the external data acquisition module transmits the collected remote cab perspective image and sound information to the remote augmented reality operation terminal through coding and compression from the remote communication subsystem.

[0022] A further improvement of the technical solution of the present invention is that the zoom camera module includes multiple zoom cameras, and the zoom camera module collects images from the perspective of the remote cab and transmits the images to the remote augmented reality operation terminal through a remote communication subsystem.

[0023] A further improvement of the technical solution of the present invention is that: the motion control module includes a motor and an embedded control board, wherein the main control module of the embedded control board is connected to the motor and controls the motor movement; the motion control module controls the multi-degree-of-freedom binocular gimbal to follow the operator's head posture in real time according to the received head posture information.

[0024] An operating method of a remote-operated visual assistance system based on multi-sensor fusion comprises the following steps:

[0025] Step 1: The operator uses the mobile remote control console to turn on the remote control mode of the construction machinery on the remote augmented reality operation terminal, and wears the head-mounted augmented reality display at the same time, and the system enters the initialization. After the system initialization is completed, the client appears in the head-mounted augmented reality display to perform initial posture calibration. The driver adjusts the posture according to the prompt of the client to complete the posture calibration. After the calibration is completed, the multi-degree-of-freedom binocular gimbal starts to start, and the gimbal initialization is turned on. After the initialization is completed, it enters the scanning mode, and the gimbal performs a dual-degree-of-freedom 360° rotation detection. After the detection is correct, it automatically switches to the follow-up mode.

[0026] Step 2: After entering the follow-up mode, the operator's head posture acquisition module of the remote augmented reality operation terminal calculates the real-time head posture and transmits it to the multi-degree-of-freedom binocular gimbal through the remote communication subsystem. At the same time, the head-mounted augmented reality display and panoramic audio system play the remote cockpit perspective image and sound information transmitted by the multi-degree-of-freedom binocular gimbal to the operator in real time;

[0027] After receiving the follow-up mode signal from the remote augmented reality operation terminal, the multi-degree-of-freedom binocular gimbal transmits the remote cockpit view image and sound information collected by the external data acquisition module to the remote augmented reality operation terminal in real time through the remote communication subsystem; at the same time, after receiving the posture signal, the motion control module controls the multi-degree-of-freedom binocular gimbal to follow the operator's head posture in real time, and the entire engineering machinery remote operation visual assistance system enters the normal operation mode;

[0028] Step 3: If the operator chooses to change the operation mode, the manual mode is selected through the client in the head-mounted augmented reality display. At this time, the real-time head posture calculated by the operator's head posture acquisition module will no longer be transmitted to the multi-degree-of-freedom binocular gimbal. Instead, the gimbal is adjusted to rotate in multiple degrees of freedom by using the joystick of the movable remote control console. At the same time, the image can be zoomed and magnified. The manual mode is suitable for long-term continuous work in a fixed position;

[0029] If the operator chooses to switch back to the follow-up mode, the follow-up mode is selected through the client in the head-mounted augmented reality display, and the operation method is the same as step 2;

[0030] Step 4: If the operator finishes using the remote-operated visual assistance system for construction machinery, the movable remote control console selection system is turned off. After the multi-degree-of-freedom binocular gimbal receives the shutdown signal through the remote communication subsystem, it enters the scanning mode again. The gimbal performs a dual-degree-of-freedom 360° rotation detection. After the detection is correct, the embedded processor controls the power supply stabilization device to cut off the power supply and the system is shut down.

[0031] Due to the adoption of the above technical solution, the technical progress achieved by the present invention is:

[0032] 1. The present invention adopts an improved multi-sensor fusion posture positioning algorithm to obtain head posture information. Compared with only using inertial sensors or camera visual information to obtain position and posture information, the drift phenomenon is greatly eliminated, and the posture information obtained is more accurate and stable, which improves the robustness of the overall operation of the system.

[0033] 2. The present invention adopts multi-mode network transmission, which can adapt to a variety of network transmission methods, including wired network port communication, LAN wireless communication, 5G wireless communication, etc. It can adapt to various construction plans and has better versatility and expansibility.

[0034] 3. The present invention adopts an ergonomic design, and uses a stereo matching algorithm and binocular parallax principle to generate a more immersive image in the VR eye. The operator feels as if he is in the cab, and at the same time provides augmented reality information, including providing the operator with a prediction of the landing point of the engineering machinery arm, real-time distance information between the current position and the engineering target, etc. The system is easy to operate, reduces operator training costs, and improves work efficiency.

[0035] 4. The present invention uses a sound collection module and a zoom camera module to collect information, utilizes ergonomic technology, and uses panoramic audio and a head-mounted augmented reality display for playback, so that the system can provide the operator with an extremely high sense of three-dimensional presence. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a system structure block diagram of the present invention;

[0037] Figure 2 It is a structural block diagram of an improved multi-sensor fusion attitude positioning algorithm in the present invention;

[0038] Figure 3 is a flow chart of the system operation method of the present invention;

[0039] Among them, 1. Remote augmented reality operation terminal, 2. Multi-degree-of-freedom binocular gimbal, 3. Remote communication subsystem, 4. Movable remote operation console, 5. Operator head posture acquisition module, 6. Head-mounted augmented reality display, 7. Panoramic audio, 8. Core processor, 9. External data acquisition module, 10. Embedded processor, 11. Power supply voltage stabilization device, 12. Motion control module, 13. Portable loading platform, 14. Binocular depth camera, 15. Inertial sensor, 16. Industrial computer, 17. Sound acquisition module, 18. Variable-focus camera module. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below with reference to the accompanying drawings and embodiments:

[0041] like Figure 1As shown, a remote operation visual assistance system based on multi-sensor fusion includes a remote augmented reality operation terminal 1, a multi-degree-of-freedom binocular gimbal 2 and a remote communication subsystem 3.

[0042] The remote augmented reality operation terminal 1 includes a movable remote operation console 4, an operator head posture acquisition module 5, a head-mounted augmented reality display 6, a panoramic audio system 7 and a core processor 8; the remote augmented reality operation terminal 1 has the function of enabling the operator to remotely obtain a cockpit perspective image that conforms to human visual characteristics and remotely operate outdoor engineering machinery in real time;

[0043] The structure of the movable remote control platform 4 is strictly modeled after the internal structure of a real engineering machine, and is a 1:1 realistically restored operating platform; the movable remote control platform 4 has the function of enabling the operator to remotely control the outdoor engineering machine, and at the same time collects the state information of the engineering machine, including the position, rotation angle of the machine, slope, etc., and transmits it to the head-mounted augmented reality display 6;

[0044] The operator head posture acquisition module 5 integrates a binocular depth camera 14, an inertial sensor 15 and an industrial computer 16; the operator head posture acquisition module 5 has the function of obtaining the operator's head posture in real time and transmitting it to the multi-degree-of-freedom binocular gimbal 2 through the remote communication subsystem 3; the binocular depth camera 14 has the function of obtaining depth information and video images and transmitting them to the industrial computer 16; the inertial sensor 15 has the function of collecting magnetometer, accelerometer and gyroscope information and transmitting them to the industrial computer 16; the industrial computer 16 has the function of obtaining real-time head posture data using the information collected by the binocular depth camera 14 and the inertial sensor 15; the industrial computer 16 obtains the head posture by running a posture positioning algorithm based on multi-sensor fusion inside; a posture positioning algorithm based on multi-sensor fusion adopts the fusion of vision and inertial sensors 15, and introduces a point-line fusion method and a semantic segmentation method to reduce the interference of texture information and remove dynamic interference. It makes full use of the structure, texture and semantic information existing in the scene, eliminates dynamic noise interference, and improves the positioning accuracy and robustness of the system;

[0045] The head-mounted augmented reality display 6 has the function of playing the video image from the perspective of the remote cockpit in real time and making the image conform to the binocular vision characteristics of the human body, providing the driver with a sense of presence, and at the same time displaying the current state of the construction machinery on both sides of the image screen, including posture, mechanical rotation angle, slope, etc.; the augmented reality effect is that information such as the predicted landing point of the machine arm is added to the image, reducing the difficulty of operation and improving work efficiency;

[0046] The core processor 8 runs a system client and an augmented reality processing module. The client includes multiple functions such as system initialization, posture correction, start operation, shut down the system, single-eye and binocular switching, scanning mode, follow-up mode, manual mode, etc. The augmented reality processing module is capable of performing visual reality augmentation processing on the remote signal cockpit perspective video image transmitted from the multi-degree-of-freedom binocular gimbal 2, and transmitting it to the head-mounted augmented reality display 6; the enhancement effect includes providing the operator with a prediction of the landing point of the engineering machinery arm, and real-time distance information between the current position and the engineering target, etc.

[0047] The panoramic audio system 7 has the function of playing the sound information collected by the remote cockpit in real time, further improving the sense of presence.

[0048] The multi-DOF binocular gimbal 2 includes an external data acquisition module 9, an embedded processor 10, a power supply and voltage stabilization device 11, a motion control module 12 and a portable loading platform 13;

[0049] The external data acquisition module 9 includes a sound acquisition module 17 and a variable-focus camera module 18; the external data acquisition module 9 has the function of transmitting the collected remote cab perspective image and sound information to the remote augmented reality operation terminal 1 through the remote communication subsystem 3 in a coded and compressed manner; the variable-focus camera module 18 is composed of a plurality of variable-focus cameras, and the variable-focus camera module 18 has the function of collecting the remote cab perspective image and transmitting the image to the remote augmented reality operation terminal 1 through the remote communication subsystem 3;

[0050] The motion control module 12 includes a motor and an embedded control board; the motion control module 12 has the function of controlling the multi-degree-of-freedom binocular gimbal 2 to follow the operator's head posture in real time according to the received head posture information;

[0051] The portable loading platform 13 is used to load the two-degree-of-freedom pan-tilt head in the remote operation mode and is connected to the seat in the engineering machinery cab. It will be folded when switching to manual mode. This position is selected to facilitate disassembly and installation. Secondly, the platform height is consistent with the actual operator's field of vision height in the excavator, further improving the sense of presence and improving the operator's operating efficiency.

[0052] The remote communication subsystem 3 uses network communication and adapts to various network transmission methods, including wired network port communication, LAN wireless communication, and 5G wireless communication; the remote communication subsystem 3 has the function of enabling two-way transmission of control interaction instructions and audio and video data between the remote augmented reality operation terminal 1 and the multi-degree-of-freedom binocular gimbal 2 subsystems.

[0053] The multi-sensor fusion-based attitude positioning algorithm adopts a method of fusing visual sensors and inertial sensors 15 on the basis of the ORB-SLAM3 framework, and then introduces an improved point-line fusion module and a semantic segmentation and dynamic interference removal module to reduce the interference of texture information and remove dynamic interference; the attitude estimation is improved by intelligent integration of point and line features; semantic and geometric filters are applied to dynamic points and lines in RGB images, a deep learning-based method is used for initial bounding box detection, and an optical flow method is used to manage dynamic elements.

[0054] like Figure 2 As shown, the multi-sensor fusion-based attitude positioning algorithm improves attitude estimation through intelligent integration of point and line features. Semantic and geometric filters are applied to dynamic points and lines in RGB images, a deep learning-based method is used for initial bounding box detection, and an optical flow method is used to manage dynamic elements, specifically including the following steps:

[0055] S1. Use the point-line fusion module to make full use of the structure and texture information in the scene to reduce the interference caused by dynamic noise and improve the robustness of the system;

[0056] The point-line fusion module includes three parts: line feature matching algorithm, line feature attribute update algorithm and line feature optimization algorithm:

[0057] The line feature matching algorithm described in S11 adopts a comprehensive line feature matching method of a stereo camera configuration, identifies line features of left and right images from a stereo camera, which have the same grid ID at the start and end points, and then compares the line descriptor of the left image with the candidate line segment of the right image, and selects the descriptor with the smallest distance for matching;

[0058] The line feature attribute update algorithm described in S12 initializes the depth value of the line endpoint, the three-dimensional vector representation of the line segment, and the disparity to achieve the purpose of eliminating redundant line features and maintaining stable tracking; for the matching segment pairs that meet the preliminary filtering conditions, the matching pairs with the same start and endpoint IDs are matched by bidirectional cosine similarity, and the line segments with low similarity are omitted; the overlap between the left and right line segments is estimated, and the line segments with overlap greater than 0.75 are retained; the disparity threshold is applied to filter the matching line segments, and the line segments with disparity less than the threshold are deleted;

[0059] The line feature optimization algorithm described in S13 uses an optimized reference keyframe attitude correction method to improve the accuracy and consistency of map line features: reference keyframe selection and attitude correction; accurate coordinate transformation is achieved, and the camera attitude accuracy is improved; this method effectively reduces positioning errors and dynamic environmental interference, improves the accuracy and consistency of map line features, and provides more reliable 3D visual SLAM results;

[0060] S131 reference keyframe selection, selects a suitable reference keyframe as a benchmark by considering a wide range of viewing angles and minimal reprojection error; for each map line feature, performs posture correction based on its corresponding reference keyframe; if the reference keyframe matches the current keyframe, uses its corrected reference keyframe for posture correction; otherwise, uses its original reference keyframe;

[0061] S132 For posture correction, let R wr and t wr They represent the rotation matrix and translation vector from the reference keyframe to the world coordinate system, respectively. Assume that the world coordinate of the map line feature to be corrected is P 3Dw ,in and Respectively represent the starting coordinates and ending coordinates of the line feature; use Sim3 transformation matrix S rw and corSwr, transforming the pose of the map line feature from the reference keyframe of the current frame to the corrected reference keyframe, see the following equation:

[0062]

[0063]

[0064] Where × represents matrix multiplication, and are the start and end coordinates of the calibrated map line feature in the calibrated reference keyframe coordinate system.

[0065] S2. For the semantic segmentation and dynamic interference removal module, semantic segmentation detection is first used to identify dynamic objects in the scene; secondly, dynamic feature point line recognition is performed, and the Lucas-Kanade (LK) optical flow algorithm is applied to perform extreme outer constraints based on the basic matrix, ensuring that the feature point pairs comply with the geometric constraints imposed by the camera motion; finally, a dynamic feature point removal mechanism based on the proportion of dynamic feature points in the bounding box of the detected dynamic target is combined, and by setting a proportion threshold, the dynamic feature points that may introduce erroneous estimates to the SLAM system are effectively filtered out.

[0066] Its functionality lies in that the real-time detection process ensures accurate and efficient recognition of various objects, improving the system's robustness in dealing with dynamic object interference.

[0067] S21 semantic segmentation detection uses CUDA-optimized Yolov8, a cutting-edge single-stage object detection algorithm that can extract semantic information of 80 different object categories in the environment;

[0068] S22 dynamic feature point and line recognition, first enhances Harris corner matching through LK optical flow pyramid. Unlike traditional Harris corner points, the algorithm discards matches near pixel edges and excludes matches with high parallax in the central pixel blocks, thereby refining the candidate point and line pool; secondly, the distance between the remaining points and their corresponding epipolar lines is calculated, and points exceeding the predefined distance threshold are classified as outliers. This process is also applicable to line features. After that, the RANSAC algorithm is combined to identify the basic matrix that produces the largest inner line:

[0069] Assume that the matching points in the previous frame and the current frame are p 1 and p 2 , whose homogeneous coordinate form is P 1 and P 2 . Then, we get:

[0070] P 1 =[u 1 ,v 1 ,1]P 2 =[u 2 ,v 2 ,1]

[0071] p 1 =[u 1 ,v 1 ]p 2 =[u 2 ,v 2 ]

[0072] Where u, v are the coordinate values ​​of the image, and then calculate the corresponding epipolar line L 1 for

[0073] L 1 =[X,Y,Z]=FP 1 =F[u 1 ,v 1 ,1]

[0074] Where X, Y, Z are line vectors and F is the fundamental matrix.

[0075] The distance between the matching point and its corresponding epipolar line is then calculated as follows:

[0076]

[0077] Where D represents the distance. If D exceeds the preset threshold, the feature point line is regarded as an outlier. Such an outlier will be defined as a dynamic feature point and removed.

[0078] S23 dynamic feature point removal mechanism, after detecting the dynamic feature point line, further processes the features within the target frame by integrating semantic information. In order to maximize the retention of key data while minimizing the error of key feature matching, a local feature filtering algorithm is proposed:

[0079] For dynamic feature points and lines within the semantic box, the key features within a 15-pixel radius centered on the dynamic features are completely deleted. If the dynamic features within the object box account for more than 40% of all features in the frame, all features in the object box will not be passed to the backend; for feature points and lines outside the semantic box, the dynamic points detected in the image background are directly removed, the unstable data association is eliminated, and the recognition ability of unknown objects is enhanced. Its function is to combine the extreme outer constraints of optical flow with dense semantic information, effectively eliminate the dynamic features of semantic objects, and process the features of unknown moving objects.

[0080] The function of the multi-sensor fusion-based attitude positioning algorithm is to make full use of the structure, texture and semantic information in the scene, eliminate dynamic noise interference, and improve the positioning accuracy and robustness of the system.

[0081] like Figure 3 As shown, an operation method of a remote operation visual assistance system based on multi-sensor fusion includes the following steps:

[0082] Step 1: The operator uses the movable remote operating console 4 to turn on the remote control mode of the construction machinery on the remote augmented reality operation terminal 1, and wears the head-mounted augmented reality display 6 at the same time, and the system enters the initialization; after the system initialization is completed, the client will appear in the head-mounted augmented reality display 6 to perform initial posture calibration, and the driver completes the posture calibration after adjusting the posture according to the prompt of the client; after the calibration is completed, the multi-degree-of-freedom binocular gimbal 2 starts to start, and the gimbal initialization is turned on. After the initialization is completed, it enters the scanning mode, and the gimbal performs a dual-degree-of-freedom 360° rotation detection. After the detection is correct, it automatically switches to the follow-up mode;

[0083] Step 2: After entering the follow-up mode, the operator head posture acquisition module 5 of the remote augmented reality operation terminal 1 calculates the real-time head posture and transmits it to the multi-degree-of-freedom binocular gimbal 2 through the remote communication subsystem 3. At the same time, the head-mounted augmented reality display 6 and the panoramic audio 7 play the remote cockpit perspective image and sound information transmitted by the multi-degree-of-freedom binocular gimbal 2 to the operator in real time;

[0084] After receiving the follow-up mode signal from the remote augmented reality operation terminal 1, the multi-degree-of-freedom binocular gimbal 2 transmits the remote cockpit view image and sound information collected by the external data acquisition module 9 to the remote augmented reality operation terminal 1 in real time through the remote communication subsystem 3; at the same time, after receiving the posture signal, the motion control module 12 controls the multi-degree-of-freedom binocular gimbal 2 to follow the operator's head posture movement in real time, and the entire engineering machinery remote operation visual assistance system enters the normal operation mode;

[0085] Step 3: If the operator chooses to change the operation mode, the manual mode is selected through the client in the head-mounted augmented reality display 6. At this time, the real-time head posture calculated by the operator's head posture acquisition module 5 will no longer be transmitted to the multi-degree-of-freedom binocular gimbal 2, but the gimbal is adjusted to rotate in multiple degrees of freedom by using the joystick of the movable remote control console 4. At the same time, the image can be zoomed and magnified. The manual mode is suitable for long-term continuous work in a fixed position;

[0086] If the operator chooses to switch back to the follow-up mode, the client in the head-mounted augmented reality display 6 selects the follow-up mode, and the operation method is the same as step 2.

[0087] Step 4: If the operator finishes using the remote-operated visual assistance system for construction machinery, the movable remote operating console 4 used selects system shutdown. After the multi-degree-of-freedom binocular gimbal 2 receives the shutdown signal through the remote communication subsystem 3, it enters the scanning mode again. The gimbal performs a dual-degree-of-freedom 360° rotation detection. After the detection is correct, the embedded processor 10 controls the power supply stabilizing device 11 to cut off the power supply and the system is shut down.

[0088] In summary, the present invention enables an operator to remotely operate engineering machinery indoors, thereby effectively improving work efficiency and reducing the risk of injury.

Claims

1. A teleoperation visual assistance system based on multi-sensor fusion, characterized by: It comprises a remote augmented reality operation terminal (1), a multi-degree-of-freedom binocular gimbal (2) and a remote communication subsystem (3); The remote augmented reality operation terminal (1) comprises a movable remote operation console (4), an operator head posture acquisition module (5), a head-mounted augmented reality display (6), a panoramic audio system (7) and a core processor (8); The multi-degree-of-freedom binocular pan / tilt platform (2) comprises an external data acquisition module (9), an embedded processor (10), a power supply voltage stabilization device (11), a motion control module (12) and a portable loading platform (13); The remote communication subsystem (3) uses network communication, including wired network communication, local area network wireless communication, and 5G wireless communication; The multi-degree-of-freedom binocular pan-tilt platform (2) transmits image and sound information to a remote augmented reality operation terminal (1) via a remote communication subsystem (3), and the remote augmented reality operation terminal (1) transmits head posture information to the multi-degree-of-freedom binocular pan-tilt platform (2) via the remote communication subsystem (3).

2. The teleoperation visual assistance system based on multi-sensor fusion according to claim 1, characterized in that: The structure of the movable remote control platform (4) is the same as that of the operating platform; the movable remote control platform (4) communicates with the outdoor engineering machinery through the remote communication subsystem (3), remotely controls the outdoor engineering machinery, and collects state information of the engineering machinery, including position, mechanical rotation angle, and slope, and transmits the information to the head-mounted augmented reality display (6).

3. The teleoperation visual assistance system based on multi-sensor fusion according to claim 1, characterized in that: The operator head posture acquisition module (5) comprises a binocular depth camera (14), an inertial sensor (15) and an industrial computer (16); the operator head posture acquisition module (5) acquires the operator's head posture in real time and transmits it to the multi-degree-of-freedom binocular gimbal (2) through a remote communication subsystem (3); The binocular depth camera (14) acquires depth information and video images and transmits them to the industrial control computer (16); The inertial sensor (15) collects information from the magnetometer, accelerometer and gyroscope and transmits it to the industrial computer (16); The industrial computer (16) obtains real-time head posture data by using information collected by the binocular depth camera (14) and the inertial sensor (15); the industrial computer (16) obtains the head posture by running a posture positioning algorithm based on multi-sensor fusion.

4. The teleoperation visual assistance system based on multi-sensor fusion according to claim 3 is characterized in that: The multi-sensor fusion-based attitude positioning algorithm adopts a method of fusing a visual sensor with a binocular depth camera (14) and an inertial sensor (15) based on the ORB-SLAM3 framework, and then introduces an improved point-line fusion module and a semantic segmentation and dynamic interference removal module to reduce the interference of texture information and remove dynamic interference; The pose estimation is improved through the intelligent integration of point and line features; semantic and geometric filters are applied to dynamic points and lines in RGB images, a deep learning-based method is used to detect the initial bounding box of dynamic objects, and the optical flow method is used to manage the dynamic point and line elements inside and outside the box, and the pose data is input into the original SLAM framework. The specific steps include: S1. Use the point-line fusion module to fully utilize the structure and texture information in the scene to extract a large number of feature points and lines, improve positioning accuracy, enhance robustness and reduce interference caused by dynamic noise; S2. For the semantic segmentation and dynamic interference removal module, semantic segmentation detection is first used to identify dynamic objects in the scene; secondly, dynamic feature point line recognition is performed, and the Lucas-Kanade optical flow algorithm is applied to execute the extreme outer constraints based on the basic matrix; finally, a dynamic feature point removal mechanism based on the proportion of dynamic feature points in the bounding box of the detected dynamic target is combined, and the dynamic feature points that introduce erroneous estimation to the SLAM system are effectively filtered out by setting the proportion threshold.

5. The teleoperation visual assistance system based on multi-sensor fusion according to claim 1, characterized in that: The core processor (8) runs a system client and an augmented reality processing module, which performs visual reality augmentation processing on a remote signal cockpit perspective video image transmitted from the multi-degree-of-freedom binocular gimbal (2), and transmits it to a head-mounted augmented reality display (6).

6. The teleoperation visual assistance system based on multi-sensor fusion according to claim 1, characterized in that: The augmented reality processing module performs visual reality augmentation processing on the remote signal cockpit perspective video image transmitted from the multi-degree-of-freedom binocular gimbal (2); The enhanced processing includes providing the operator with a prediction of the landing point of the construction machinery arm and providing real-time distance information between the current position and the construction target.

7. The teleoperation visual assistance system based on multi-sensor fusion according to claim 1, characterized in that: The external data acquisition module (9) comprises a sound acquisition module (17) and a variable-focus camera module (18); the external data acquisition module (9) transmits the acquired remote cab perspective image and sound information to the remote augmented reality operation terminal (1) via the remote communication subsystem (3) in a coded and compressed manner.

8. The teleoperation visual assistance system based on multi-sensor fusion according to claim 7, characterized in that: The variable-focus camera module (18) comprises a plurality of variable-focus cameras, and the variable-focus camera module collects images from the perspective of the remote cab and transmits the images to the remote augmented reality operation terminal (1) via the remote communication subsystem (3).

9. The teleoperation visual assistance system based on multi-sensor fusion according to claim 1, characterized in that: The motion control module (12) comprises a motor and an embedded control board, wherein a main control module of the embedded control board is connected to the motor and controls the motor movement; the motion control module (12) controls the multi-degree-of-freedom binocular pan-tilt platform (2) to rotate in real time following the operator's head posture according to the received head posture information.

10. An operating method of a remote-operated visual assistance system based on multi-sensor fusion, characterized in that: Using the remote operation visual assistance system based on multi-sensor fusion as described in any one of claims 1 to 9, comprising the following steps: Step 1: The operator uses the movable remote control console (4) to turn on the remote control mode of the construction machinery on the remote augmented reality operation terminal (1), and wears the head-mounted augmented reality display (6) at the same time, and the system enters the initialization stage; after the system initialization is completed, the client appears in the head-mounted augmented reality display (6) to perform initial posture calibration, and the driver adjusts the posture according to the prompt of the client to complete the posture calibration; after the calibration is completed, the multi-degree-of-freedom binocular gimbal (2) starts to start, and the gimbal initialization is turned on. After the initialization is completed, the scanning mode is entered, and the gimbal performs a dual-degree-of-freedom 360° rotation detection. After the detection is correct, it automatically switches to the follow-up mode; Step 2: After entering the follow-up mode, the operator's head posture acquisition module (5) of the remote augmented reality operation terminal (1) calculates the real-time head posture and transmits it to the multi-degree-of-freedom binocular gimbal (2) through the remote communication subsystem (3). At the same time, the head-mounted augmented reality display (6) and the panoramic audio (7) play the remote cockpit perspective image and sound information transmitted by the multi-degree-of-freedom binocular gimbal (2) to the operator in real time; After receiving the follow-up mode signal from the remote augmented reality operation terminal (1), the multi-degree-of-freedom binocular gimbal (2) transmits the remote cockpit visual image and sound information collected by the external data acquisition module (9) to the remote augmented reality operation terminal (1) in real time through the remote communication subsystem (3); at the same time, after receiving the posture signal, the motion control module (12) controls the multi-degree-of-freedom binocular gimbal (2) to follow the operator's head posture movement in real time, and the entire engineering machinery remote operation visual assistance system enters the normal operation mode; Step 3: If the operator chooses to change the operation mode, the manual mode is selected through the client in the head-mounted augmented reality display (6). At this time, the real-time head posture calculated by the operator's head posture acquisition module (5) will no longer be transmitted to the multi-degree-of-freedom binocular gimbal (2). Instead, the gimbal is adjusted to rotate in multiple degrees of freedom by using the joystick of the movable remote control console (4). At the same time, the image can be zoomed and magnified. The manual mode is suitable for long-term continuous work in a fixed position. If the operator chooses to switch back to the follow-up mode, the client in the head-mounted augmented reality display (6) selects the follow-up mode, and the operation method is the same as step 2; Step 4: If the operator finishes using the engineering machinery remote operation visual assistance system, the movable remote operation console (4) is turned off to select the system. After the multi-degree-of-freedom binocular gimbal (2) receives the shutdown signal through the remote communication subsystem (3), it enters the scanning mode again. The gimbal performs a dual-degree-of-freedom 360° rotation detection. After the detection is correct, the embedded processor (10) controls the power supply stabilizing device (11) to cut off the power supply, and the system is turned off.

Images