Intelligent piloting system and method based on humanoid robot and attitude mapping technology

Abstract

The invention discloses an intelligent piloting system and method based on a humanoid robot and an attitude mapping technology. The system comprises a shore end control center, a 5G communication subsystem and a ship end execution system. The shore end captures the whole body attitude information of the pilot through the multi-view attitude capture subsystem, and the attitude mapping and control subsystem converts the whole body attitude information into a robot control instruction by adopting a layered mapping strategy; a robot control instruction is transmitted to the ship end through a 5G network in a low-delay mode, and piloting operation is executed by the humanoid robot; the first visual angle video of the robot is transmitted back to the shore end VR display subsystem, and hand-eye coordinated closed-loop control is formed. Therefore, by means of the scheme provided by the embodiment of the invention, the safety risk that the pilot climbs and leaves the wheel is thoroughly eliminated, all-weather and uninterrupted piloting operation is achieved, the natural operation experience is kept by means of the layered mapping strategy, and the safety, continuity and efficiency of port piloting are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent shipping technology, and in particular to an intelligent navigation system and method based on humanoid robots and posture mapping technology. Background Technology

[0002] Ship pilotage is a crucial link in ensuring port safety and operational efficiency. Currently, it mainly relies on pilots personally boarding ships for operations, but this traditional model has many drawbacks: First, pilots climb between pilot boats and ships using rope ladders, which poses a high risk of falls and drowning, especially in severe weather. Second, extreme weather such as typhoons and giant waves can force pilotage operations to be interrupted, leading to delays, port congestion, and significant economic losses. Third, pilots rely solely on their naked eyes and limited onboard equipment for situational awareness, making it difficult to fully grasp environmental information in low visibility or at night, increasing pilotage risks. Finally, the training period for senior pilots is long and their numbers are limited, and the traditional model is labor-intensive and cannot meet the ever-increasing port throughput demands.

[0003] Currently, research on intelligent pilotage focuses primarily on pilotage information systems and auxiliary decision-making systems, but there is still no technical solution that uses humanoid robots to replace pilots on board and achieves actual ship operations through remote attitude synchronization control. Existing technologies cannot fundamentally solve the safety risks of boarding and disembarking and the challenges of operational continuity under adverse weather conditions.

[0004] Therefore, how to maintain the same level of accuracy and efficiency as traditional operations while ensuring safety and continuous operation is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] This invention provides a smart navigation system and method based on humanoid robots and posture mapping technology, which can improve the safety, continuity and efficiency of port navigation.

[0006] The first aspect of the present invention provides an intelligent navigation system based on humanoid robots and posture mapping technology, including: a shore-based control center, a 5G communication subsystem, and a ship-based execution system; the shore-based control center includes a multi-view posture capture subsystem, a VR display subsystem, and a posture mapping and control subsystem, and the ship-based execution system includes a humanoid robot, a perception system integrated into the humanoid robot, and a control and computing system; The multi-view attitude capture subsystem is used to simultaneously capture the pilot's full-body attitude information from different perspectives; The posture mapping and control subsystem is used to convert whole-body posture information into robot control commands using a hierarchical mapping strategy. The hierarchical mapping strategy includes at least precise mapping of the head, inverse kinematic mapping of the upper limbs, and recognition and discrete command mapping of the walking intention of the lower limbs. The 5G communication subsystem is used to establish a low-latency bidirectional data transmission link between the shore-based control center and the ship-based execution system, and to transmit robot control commands uplink to the ship-based execution system. The control and computing system is used to receive and parse robot control commands and drive the humanoid robot's joints to perform corresponding actions. Humanoid robots are used to perform pilotage operations under the drive of control and computing systems; A perception system is used to collect first-person perspective videos of the humanoid robot. The 5G communication subsystem is used to transmit first-person view video downlink to the shore control center; The VR display subsystem is used to display first-person perspective video in real time.

[0007] Optionally, the hierarchical mapping strategy includes a head posture mapping module, an upper limb posture mapping module, and a lower limb walking intention recognition and mapping module; The head pose mapping module is used to map the pose data of the navigator's head to the joint control commands of the humanoid robot's head. The upper limb posture mapping module is used to generate joint control commands for the humanoid robot arm by solving inverse kinematics based on the three-dimensional coordinates of the pilot's upper limb joints. The lower limb walking intention recognition and mapping module is used to identify gait intentions by analyzing the temporal trajectory of the pilot's lower limb joints and generate corresponding discrete movement commands.

[0008] Optionally, the lower limb walking intention recognition and mapping module includes a feature extraction unit, a classification unit, and an instruction generation unit; The feature extraction unit is used to extract gait features from the temporal trajectory of the pilot's lower limb joints. The gait features include stride frequency, stride length, and centroid displacement. The classification unit is used to analyze gait features using a long short-term memory network to identify gait intentions of moving forward, backward, sideways, or stopping. The instruction generation unit is used to convert the identified gait intention into corresponding discrete motion instructions.

[0009] Optionally, the upper limb posture mapping module is specifically used for: Calculate the target vector of the pilot's wrist relative to the shoulder joint; The target vector is transformed from the human coordinate system to the robot coordinate system and scaled according to a preset arm length scaling factor. Based on the scaled target vector, the target angles of the shoulder and elbow joints of the humanoid robot arm are calculated using an inverse kinematics algorithm.

[0010] Optionally, the control and computing system is also used for: After receiving discrete motion commands generated by the lower limb walking intention recognition and mapping module, local path planning and gait generation are performed. Drive the robot to execute the planned gait and perform real-time obstacle avoidance control during the execution process.

[0011] Optionally, the control and computing system includes a robot controller, a motion planner, and a real-time perception and obstacle avoidance module; A robot controller is used to drive the various joints of a robot to perform movements. The motion planner integrates path planning and gait generation algorithms to convert discrete motion commands into a sequence of motion trajectories for the robot's feet. The real-time perception and obstacle avoidance module is used to process the environmental data collected by the perception system and make real-time adjustments based on the trajectory generated by the motion planner.

[0012] Optionally, the multi-view pose capture subsystem is specifically used for: Images of the pilot were simultaneously captured from different perspectives using multiple cameras. For each viewpoint image, run a human pose estimation algorithm to extract two-dimensional human joints; Based on the calibration parameters of multiple cameras, multiple two-dimensional observations of the same joint point are fused using triangulation to calculate its coordinates in three-dimensional space.

[0013] A second aspect of the present invention provides an intelligent navigation method based on humanoid robots and posture mapping technology, comprising: The multi-view attitude capture subsystem simultaneously captures the pilot's full-body attitude information from different perspectives; The posture mapping and control subsystem employs a hierarchical mapping strategy to convert whole-body posture information into robot control commands. The hierarchical mapping strategy includes at least precise mapping of the head, inverse kinematic mapping of the upper limbs, and recognition and discrete command mapping of the lower limbs' walking intentions. A low-latency two-way data transmission link is established between the shore-based control center and the ship-based execution system through the 5G communication subsystem, so that robot control commands can be transmitted uplink to the ship-based execution system. The control and computing system receives and parses robot control commands and drives the humanoid robot's joints to perform corresponding actions. Pilotage operations are performed by a humanoid robot driven by a control and computing system; First-person perspective video of a humanoid robot is collected through a sensing system; The first-person perspective video is transmitted downlink to the shore control center via the 5G communication subsystem; The VR display subsystem displays first-person perspective video in real time.

[0014] A third aspect of the present invention provides an apparatus comprising: One or more processors; A memory on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors enable the one or more processors to perform the functions of the intelligent navigation system based on humanoid robots and posture mapping technology as described above.

[0015] A fourth aspect of the present invention provides a computer storage medium for storing a program, which, when executed, performs the functions of an intelligent navigation system based on humanoid robot and posture mapping technology as described in any of the preceding claims.

[0016] Beneficial Effects: This invention discloses a smart pilotage system and method based on humanoid robots and posture mapping technology. The system includes a shore-based control center, a 5G communication subsystem, and a ship-based execution system. At the shore, a multi-view posture capture subsystem captures the pilot's full-body posture information, and the posture mapping and control subsystem uses a hierarchical mapping strategy to convert this information into robot control commands. These commands are transmitted to the ship via a low-latency 5G network, where the humanoid robot performs the pilotage operation. The robot's first-view video is then transmitted back to the shore-based VR display subsystem, forming a closed-loop control system with hand-eye coordination. Therefore, the solution provided by this invention completely eliminates the safety risks associated with pilots boarding and disembarking, enabling 24 / 7 uninterrupted pilotage operations. Furthermore, the hierarchical mapping strategy maintains a natural operating experience, significantly improving the safety, continuity, and efficiency of port pilotage. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of the structure of an intelligent navigation system based on humanoid robot and posture mapping technology provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the spatial layout for multi-view attitude capture provided in an embodiment of the present invention; Figure 3 This is a flowchart of a multi-view, multi-pose fusion algorithm provided in an embodiment of the present invention; Figure 4 A schematic diagram of a hierarchical attitude mapping strategy provided in an embodiment of the present invention; Figure 5This is a diagram of OpenPose skeletal joint points provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the upper limb inverse kinematics solution provided in an embodiment of the present invention; Figure 7 This is a flowchart of lower limb walking intention recognition provided in an embodiment of the present invention; Figure 8 This invention provides a VR first-person perspective synchronization system framework for embodiments of the invention. Figure 9 The 5G network framework and bandwidth allocation diagram provided in the embodiments of the present invention; Figure 10 A flowchart illustrating an intelligent navigation method based on humanoid robot and posture mapping technology provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0019] This invention provides a smart pilotage system and method based on humanoid robots and posture mapping technology, which is particularly suitable for remote control of port ship pilotage operations and can improve the safety, continuity and efficiency of port pilotage.

[0020] To facilitate understanding, the application scenarios of the embodiments of the present invention will be introduced first.

[0021] Ship pilotage refers to the professional service provided by experienced pilots familiar with local waters, who board a vessel to assist the captain in safely navigating the ship into and out of ports and through complex waters. Pilotage operations are a crucial link in ensuring port safety and improving port operational efficiency.

[0022] Traditional manual navigation methods have the following significant problems: 1. Safety hazards for pilots boarding and disembarking vessels: Pilots need to climb a rope ladder between the pilot boat and the vessel to board and disembark. Especially in high winds, waves, and severe weather conditions, the operation is extremely risky and has resulted in numerous personal injury accidents. According to statistics, the pilot boarding and disembarking process has the highest accident rate in pilotage operations.

[0023] 2. Severe weather restricts operations: When encountering severe weather such as typhoons, strong winds, and giant waves, pilots are unable to board ships safely, forcing pilotage operations to be interrupted. This causes delays in the overall port operation plan, backlog of shipping schedules, and obstruction of cargo flow, which has a serious adverse impact on port operation safety and economic benefits.

[0024] 3. Limited information perception capabilities: Under traditional pilotage methods, pilots rely solely on visual observation and limited onboard equipment for situational awareness. In situations such as reduced visibility or nighttime operations, it is difficult to obtain comprehensive and accurate information about the surrounding environment of the ship, which increases pilotage risks.

[0025] 4. Shortage of pilot resources: Training a mature senior pilot takes more than 15 years, and the number of pilots is limited. Under the traditional pilotage model, pilots need to work on the ship for long periods of time, which is labor-intensive and difficult to meet the growing port pilotage demand.

[0026] Currently, domestic and international research on intelligent pilotage mainly focuses on pilotage informatization and pilotage decision support, but there are no technical solutions that use humanoid robots to replace pilots on boarding ships or achieve pilotage operations through remote attitude synchronization control. Existing technologies cannot effectively solve the safety risks of pilots boarding and disembarking ships, nor can they continue pilotage operations under extreme weather conditions.

[0027] Therefore, this invention proposes an intelligent navigation system based on humanoid robots and posture mapping technology, which can ensure the safety and continuity of navigation operations, operate normally under adverse weather conditions, and maintain operational accuracy and efficiency comparable to traditional navigation.

[0028] See Figure 1 This figure is a schematic diagram of the structure of a smart navigation system based on humanoid robots and attitude mapping technology provided in an embodiment of the present invention. The smart navigation system based on humanoid robots and attitude mapping technology provided in this embodiment of the present invention includes: a shore-based control center, a 5G communication subsystem, and a ship-based execution system.

[0029] The shore-based control center includes a multi-view attitude capture subsystem, a VR display subsystem, and an attitude mapping and control subsystem.

[0030] The multi-view attitude capture subsystem is used to simultaneously capture the navigator's full-body attitude information from different perspectives. The attitude mapping and control subsystem is used to convert the full-body attitude information into robot control commands using a hierarchical mapping strategy. The hierarchical mapping strategy includes at least precise mapping of the head, inverse kinematic mapping of the upper limbs, and recognition and discrete command mapping of the lower limbs' walking intentions.

[0031] Specifically, the shore-based control center includes: Multi-view pose capture subsystem: such as Figure 2 As shown, Figure 2The schematic diagram of the multi-view attitude capture spatial layout provided in this embodiment of the invention includes four RGB cameras, which are respectively installed in the four corners of the control center to form a 6m×6m capture space for capturing the pilot's full-body attitude information from different perspectives. For example, four industrial-grade RGB cameras (such as Hikvision DS-2CD2T46WD or equivalent products) with a resolution of 1920×1080, a frame rate of 30fps, an installation height of 2.5m, and a pitch angle of 30° are installed in the four corners of the 6m×6m control room to form omnidirectional coverage. The cameras are connected to the attitude processing workstation via gigabit Ethernet.

[0032] The VR display subsystem includes a VR headset with 6DoF (6 Degrees of Freedom) tracking capabilities. This headset displays the first-person view captured by the ship's robotic head camera in real time and tracks the pilot's head position and posture. For example, it can use a VR headset supporting 6DoF tracking, such as the HTC Vive Pro 2 or Meta Quest Pro, with a resolution of 2448×2448 per eye, a refresh rate of 90-120Hz, and a 120° field of view. The headset connects to the VR rendering workstation wirelessly, avoiding the limitations imposed by cables on the pilot's movements.

[0033] The posture mapping and control subsystem converts captured human posture information into robot control commands using a hierarchical mapping strategy. It includes a 1:1 head-to-head precise mapping module, an upper limb target posture mapping module, and a lower limb intention recognition mapping module. The posture mapping and control subsystem includes a high-performance computing server configured with an Intel Xeon or AMD EPYC processor and at least 32GB of memory to run the OpenPose posture estimation algorithm, multi-view fusion algorithm, and hierarchical mapping algorithm. The system adopts a modular design, allowing each mapping module to be independently optimized and updated.

[0034] In one implementation of this invention, the multi-view attitude capture subsystem is specifically used to simultaneously acquire images containing the navigator from different perspectives using multiple cameras; for each perspective image, a human attitude estimation algorithm is run to extract two-dimensional human joint points; based on the calibration parameters of multiple cameras, multiple two-dimensional observations of the same joint point are fused using triangulation to calculate its coordinates in three-dimensional space.

[0035] Specifically, the multi-view pose capture subsystem uses the following methods for pose fusion: Camera calibration: The Zhang Zhengyou calibration method was used to calibrate the four RGB cameras, obtain the intrinsic and extrinsic parameter matrices of each camera, and establish the transformation relationship between the camera coordinate system and the world coordinate system. Pose estimation: Each camera independently runs the OpenPose algorithm to extract 2D human skeletal joints, with no fewer than 17 joints, including head, neck, shoulders, elbows, wrists, hips, knees, and ankles; Triangulation: For the same joint point, the three-dimensional world coordinates of the joint point are calculated using the direct linear transformation algorithm based on the epipolar geometric constraints. Confidence-weighted fusion: When the estimation results of the same key point from the four perspectives are inconsistent, a weighted fusion is performed based on the confidence scores output by OpenPose. The formula is as follows: ,in For the first Confidence level from each perspective For the first Three-dimensional coordinates estimated from a single viewpoint; Timing smoothing: Kalman filtering is used to smooth the trajectory of the joint points in time, eliminate single-frame jitter, and predict the attitude of the next frame to compensate for transmission delay.

[0036] Multi-view pose fusion algorithm implementation as follows Figure 3 As shown, Figure 3 The flowchart below illustrates the multi-view, multi-pose fusion algorithm provided in this embodiment of the invention. The multi-view pose fusion algorithm fuses the 2D joint coordinates from four perspectives into 3D joint coordinates. The specific implementation steps are as follows: Step 31: Camera Calibration The Zhang Zhengyou calibration method was used to calibrate four cameras. Specific steps included: ① Printing a 10×7 checkerboard calibration board (grid size 30mm×30mm); ② Taking 20-30 images of the calibration board from different angles; ③ Using OpenCV's `calibrateCamera` function to calculate the camera intrinsic parameter matrix K and distortion coefficients D; ④ Using the `stereoCalibrate` function to calculate the relative poses between the cameras (rotation matrix R and translation vector t).

[0037] Calibration result example (camera 1): Intrinsic parameter matrix ,in (pixels) , Distortion coefficient ,in , , , The relative poses of the cameras are unified using a world coordinate system.

[0038] Step 32: Polar geometric constraints For the 2D projection points of the same joint point from different viewpoints, epipolar geometric constraints are used for matching. The epipolar constraint formula is: ; in, , Let be the homogeneous coordinates of the key points in the two viewpoints. The base matrix is ​​used. By calculating the epipolar distance between all candidate point pairs, the point pair with the smallest distance is selected as the matching point. The matching threshold can be set to 5 pixels; point pairs exceeding the threshold are considered mismatches and are discarded.

[0039] Step 33: Triangulation Triangulation is performed using the Direct Linear Transform (DLT) algorithm to calculate the 3D coordinates of the joints. Let the projection of the joint onto camera i be... The camera projection matrix is Then the triangulation problem can be expressed as: ,in Let P be the homogeneous world coordinates of the joint. By constructing a system of equations AP=0 and solving it using SVD decomposition, the least squares solution of P is obtained.

[0040] Step 34: Confidence-weighted fusion When all four viewpoints detect the same keypoint, a weighted fusion is performed based on the confidence level output by OpenPose. Let the first... The estimated 3D coordinates for each viewpoint are Confidence level is The fused coordinates are: ; Confidence weight The calculation considers two factors: ① the joint confidence score (between 0 and 1) of the OpenPose output; ② the viewpoint angle factor, with the frontal viewpoint having a higher weight than the side and rear views. The final weight is the product of the two.

[0041] Step 35: Kalman filtering timing smoothing Kalman filtering is applied to the 3D trajectory of each joint for temporal smoothing. The state vector is... This includes position and velocity. The state transition matrix is:

[0042] ,in The sampling interval is 1 / 30 second. It is a 3×3 identity matrix. It is a 3×3 zero matrix. The observation matrix is: The process noise covariance Q and the observation noise covariance R are set empirically, with the diagonal elements of Q being [0.01, 0.01, 0.01, 0.1, 0.1, 0.1] and the diagonal elements of R being [0.1, 0.1, 0.1].

[0043] Using the aforementioned fusion algorithm, the system can output smooth and accurate 3D joint trajectories, providing reliable input for subsequent pose mapping. Experiments show that the 3D joint positioning error after multi-view fusion is less than 2cm (within a 6m×6m spatial range).

[0044] The OpenPose attitude estimation algorithm is implemented as follows: Figure 5 As shown, Figure 5 This is an OpenPose skeletal joint definition diagram provided in this embodiment of the invention. The OpenPose algorithm is the core algorithm for multi-view pose capture, used to extract human skeletal joints from 2D images. Combined with... Figure 5 This embodiment uses OpenPose version 1.7.0, and the specific implementation steps are as follows: Step 51: Image Preprocessing The RGB images captured by the four cameras were preprocessed, including: ① scaling the images to 368×368 pixels (the standard input size of OpenPose); ② normalizing the pixel values ​​from [0,255] to [-1,1]; ③ subtracting the mean of the ImageNet dataset (R:104,G:117,B:123).

[0045] Step 52: Feature Extraction from Convolutional Neural Networks The VGG-19 network is used as the backbone to extract high-level semantic features from the image. The network consists of 19 layers, including 16 convolutional layers and 3 fully connected layers. After the input image passes through 5 convolutional blocks, the output feature map size is 46×46×512.

[0046] Step 53: Multi-stage component affinity field (PAF) prediction Part affinity fields (PAFs) are predicted using a 6-stage convolutional network. A PAF is a 2D vector field representing the position and orientation of a limb. Each stage contains 5 convolutional layers, and the output PAF feature map is 46×46×38 pixels (19 limb pairs × 2 orientation components). The loss function for each stage is L1 loss, and the total loss is a weighted sum of the losses from each stage.

[0047] Step 54: Joint Confidence Map Prediction Building upon PAF prediction, confidence maps for 18 key points (17 body joints + 1 background) are predicted through an additional 6 stages. Each key point corresponds to a 46×46 heatmap, with the peak position of the heatmap representing the key point's 2D coordinates. Each stage is optimized using the L2 loss function.

[0048] Step 55: Joint Location and Skeleton Assembly ① Extract candidate joint locations from the confidence map using the non-maximum suppression (NMS) algorithm; ② Calculate the connection confidence between any two candidate joints using the PAF vector field and line integral; ③ Perform joint matching using the Hungarian algorithm or a greedy algorithm to assemble the complete human skeleton; ④ Output the 2D coordinates and confidence of 17 joints, with coordinate accuracy reaching sub-pixel level (optimized by Gaussian fitting).

[0049] Step 56: Coordinate system transformation Convert the image coordinate system (pixel coordinates) output by OpenPose to the camera coordinate system (physical coordinates). The conversion formula is: , ,in For pixel coordinates, As the camera's main point, , The camera focal length is obtained through calibration.

[0050] In this embodiment, the OpenPose algorithm achieves an inference speed of approximately 15fps (single-person detection) on an NVIDIA GTX 1080 Ti GPU. By processing images from four cameras in parallel using multi-threading, the overall system processing latency is approximately 50-60ms.

[0051] In one implementation of this invention, the hierarchical mapping strategy includes a head posture mapping module, an upper limb posture mapping module, and a lower limb walking intention recognition and mapping module. The head posture mapping module is used to map the pose data of the navigator's head to the joint control commands of the humanoid robot's head. The upper limb posture mapping module is used to generate the joint control commands of the humanoid robot's arm by solving inverse kinematics based on the three-dimensional coordinates of the navigator's upper limb joints. The lower limb walking intention recognition and mapping module is used to identify the gait intention by analyzing the temporal trajectory of the navigator's lower limb joints and generate corresponding discrete motion commands.

[0052] The lower limb walking intention recognition and mapping module includes a feature extraction unit, a classification unit, and a command generation unit. The feature extraction unit is used to extract gait features from the temporal trajectory of the pilot's lower limb joints. Gait features include stride frequency, stride length, and center of mass displacement. The classification unit is used to analyze the gait features using a long short-term memory network to identify gait intentions of walking forward, walking backward, moving laterally, or stopping. The command generation unit is used to convert the identified gait intentions into corresponding discrete motion commands.

[0053] The upper limb posture mapping module is specifically used for: calculating the target vector of the pilot's wrist relative to the shoulder joint; transforming the target vector from the human coordinate system to the robot coordinate system and scaling it according to the preset arm length scaling factor; and calculating the target angles of the shoulder and elbow joints of the humanoid robot arm based on the scaled target vector using an inverse kinematics algorithm.

[0054] Specifically, the posture mapping and control subsystem adopts a hierarchical mapping strategy, using different mapping logic for different body parts: (1) Head pose mapping (1:1 precise mapping), the input is the 6DoF tracking data of the VR headset, including position Euler angles The output is the robot's head joint angles, including... (Turn left and right) and (Nodding up and down); The expression for the mapping relationship is as follows: ; ; in The value range is [-90°, +90°]. The value range is [-30°, +30°], and the angular velocity limit is ≤60° / s.

[0055] (2) Upper limb pose mapping (target pose mapping): The input is the 3D coordinates of the shoulder, elbow, and wrist joints extracted by OpenPose; calculate the target position, the vector of the wrist relative to the shoulder. The system obtains direction and distance; performs coordinate system transformation and scaling, transforming the vectors from the human coordinate system to the robot coordinate system and scaling them according to the arm length ratio (human arm length / robot arm length), with a scaling factor typically between 0.8 and 1.2; and solves inverse kinematics using the TRAC-IK algorithm to calculate the angles of the robot's shoulder and elbow joints based on the target wrist position. (3) Trunk posture mapping (orientation mapping): The input is the coordinates of the midpoints of the shoulders and hips extracted by OpenPose; Orientation calculation: Calculate the vector from the left shoulder to the right shoulder, and obtain the human body orientation angle through the atan2 function. The mapping relationship is as follows: The execution method is that the robot rotates in place to reach the target orientation, and the robot can complete the turning action in multiple steps.

[0056] (4) Lower limb walking intention recognition and mapping (discrete command mapping): The input is the time series data of lower limb joint trajectories (hip, knee, ankle) extracted by OpenPose; Gait feature extraction: The motion pattern of the lower limb joints is analyzed by LSTM neural network to identify gait features (alternating foot forward and backward, forward shift of center of mass, step frequency, etc.); Intention classification: The recognition results are classified into discrete movement intentions such as walking forward, walking backward, moving laterally, and stopping; Command generation: High-level motion commands are generated, such as WALK_FORWARD + default speed 0.5m / s, instead of transmitting specific foot positions; Action filtering: Abnormal fast actions (angular velocity > 180° / s) and non-standard actions are filtered to maintain the previous effective posture.

[0057] The specific implementation of the hierarchical pose mapping algorithm is as follows: Figure 4 As shown, Figure 4 This is a schematic diagram of a hierarchical pose mapping strategy provided in an embodiment of the present invention. The hierarchical pose mapping algorithm employs different mapping strategies for different body parts. The specific implementation is as follows: Implementation of head pose mapping: Head pose mapping adopts a 1:1 precise mapping strategy. The input data comes from the 6DoF tracking system of the VR headset, and the output is the yaw (left and right rotation) and pitch (up and down nodding) joint angles of the robot's head.

[0058] The mapping formula is as follows: ; ; in, , Angular velocity , .

[0059] Implementation details: ① The VR headset outputs head posture quaternions at a frequency of 90Hz. ② Convert quaternions to Euler angles ; ③ To and ④ Perform constraint checks and angular velocity filtering; ⑤ Send the angle command to the ship-end robot via the 5G network; ⑥ After receiving the command, the robot drives the head servo motor to the target angle through the PID controller, with a control cycle of 10ms and a response time of <50ms.

[0060] Implementation of upper limb posture mapping: Upper limb posture mapping adopts a target posture mapping strategy, including coordinate system transformation, arm length scaling, and inverse kinematics solution. For example... Figure 6 As shown, Figure 6 This is a schematic diagram of the upper limb inverse kinematics solution provided in an embodiment of the present invention. Step 61: Coordinate system transformation First, calculate the target position of the wrist. Let the coordinates of the human shoulder joint be... ; The wrist coordinates are: ; The relative vector is calculated as follows: .

[0061] Then, a coordinate system transformation is performed. The transformation matrix from the human coordinate system to the robot coordinate system is: , in, , To bypass , The rotation matrix of the axis.

[0062] Step 61: Length scaling Arm length scaling factor: , in, The length of the robot's upper arm plus forearm is approximately 0.6m. The length of the upper arm plus the forearm is approximately 0.7m. .

[0063] Target vector in robot coordinate system: .

[0064] Step 63: IK Solving. The TRAC-IK algorithm is used to solve for the angles of the robot's shoulder and elbow joints. TRAC-IK is an IK solver based on nonlinear optimization, combining the advantages of numerical and analytical IK. The input is the target wrist position. The output consists of 7 joint angles: This corresponds to 3 degrees of freedom in the shoulder, 1 degree of freedom in the elbow, and 3 degrees of freedom in the wrist.

[0065] Step 64: Workspace Inspection Check if the target location is within the reachable workspace of the robotic arm. If it is outside the workspace, project the target location onto the workspace boundary using the nearest point projection method. The workspace is defined as a spherical region with a radius of 0.6m centered on the shoulder joint, excluding the portion obscured by the body.

[0066] Finally, the pointing motion is recognized, and the detection conditions include: ① the distance from the wrist to the shoulder is >0.5m (arm extended); ② the wrist speed is <0.1m / s (stationary); ③ the angle between the arm direction and the head direction is <30° (pointing forward). When the conditions are met, it is determined to be a pointing motion, and the robot maintains the current arm posture and extends the index finger.

[0067] The trunk pose mapping is implemented using an orientation mapping strategy.

[0068] Calculate body orientation: Let the coordinate of the left shoulder be... The coordinates of the right shoulder are The shoulder vector is .

[0069] Human body orientation angle: ,in This is the projection of the shoulder vector onto the horizontal plane.

[0070] Robot target orientation: .

[0071] The robot achieves the target orientation by rotating in place at a speed of 30° / s, and completes the task in multiple steps (each step involves a rotation angle of ≤30°).

[0072] Implementation details: ① Calculate the deviation between the current orientation and the target orientation. ; ②If ① Generate rotation commands; ② The robot gait controller plans the rotation gait, alternately lifting the left and right feet to complete the turn in place; ③ After each rotation, the deviation is recalculated until... .

[0073] The implementation of lower limb walking intention recognition and mapping employs an LSTM-based gait classification method for lower limb walking intention recognition, such as... Figure 7 As shown, Figure 7 A flowchart for recognizing lower limb walking intentions provided in an embodiment of the present invention.

[0074] Step 71: Input Data The input is time-series trajectory data of the lower limb joints (both hips, both knees, and both ankles), and the output is discrete motion intention.

[0075] Step 72: Feature extraction.

[0076] Gait features are extracted from the joint trajectories of the past 1 second (30 frames): ① Step frequency: detects the frequency of alternating lifting of the left and right feet; ② Step length: calculates the forward and backward displacement of the ankle within one gait cycle; ③ Center of mass displacement: calculates the displacement direction and velocity of the midpoint of the hip joint; ④ Phase difference: detects the phase difference between the left and right feet (approximately 180° in normal gait). The feature vector dimension is 12 (6 joints × 2 coordinate components) × 30 frames = 360 dimensions.

[0077] Step 73: LSTM Network Classification LSTM network structure: 360-dimensional input layer → LSTM layer 1 (128 units) → Dropout (0.3) → LSTM layer 2 (64 units) → Dropout (0.3) → fully connected layer → Softmax output layer (4 classes: walking forward, walking backward, moving laterally, stopping). Training data: 100 hours of pilot walking data, including various gait samples. Training parameters: Adam optimizer, learning rate 0.001, batch size 32, training for 50 epochs, classification accuracy on the validation set >95%.

[0078] Step 74: Motion Command Generation Based on the classification results, discrete motion commands are generated: ① Move forward → WALK_FORWARD + speed 0.5m / s; ② Move backward → WALK_BACKWARD + speed 0.3m / s; ③ Move laterally → WALK_SIDEWAY + direction (left / right) + speed 0.3m / s; ④ Stop → STOP. Commands are continuously sent until a new motion intention is detected.

[0079] Step 75: Action Filtering Filtering abnormally rapid movements: Calculate the angular velocity of any joint. If the angular velocity of any joint exceeds 180° / s, it is considered an abnormal movement, and the previous valid command is retained. This filtering mechanism prevents the robot from performing dangerous operations when the pilot accidentally falls or makes an improper movement.

[0080] The VR display subsystem is used to display first-person perspective video in real time.

[0081] Specifically, such as Figure 8 As shown, Figure 8 The VR first-view synchronization system framework provided in this embodiment of the invention includes a VR display subsystem that achieves first-view synchronization through the following methods: Video capture: The robot's head camera captures first-person perspective video at a frame rate of 25-30fps, with a resolution of 1280×720; Video encoding: H.265 / HEVC encoding format, bitrate 3-5Mbps, to reduce bandwidth usage while ensuring image quality; Low-latency transmission: The video stream is transmitted via the downlink of the 5G network, using the UDP protocol and fragmented transmission technology to ensure end-to-end latency of <100ms; VR rendering: The VR headset receives the decoded video stream and renders it at a 90Hz refresh rate, combining the headset's 6DoF tracking data to achieve an immersive first-person perspective experience. Hand-eye coordination optimization: Through a timestamp synchronization mechanism, the total delay from navigator action to robot execution to visual feedback is ensured to be less than 300ms, meeting the hand-eye coordination requirements of human perception.

[0082] The 5G communication subsystem is used to establish a low-latency bidirectional data transmission link between the shore-based control center and the ship-based execution system, and to transmit robot control commands uplink to the ship-based execution system. And transmit the first-person perspective video downlink to the shore control center; Specifically, such as Figure 9 As shown, Figure 9 This is a diagram illustrating the 5G network framework and bandwidth allocation provided in this embodiment of the invention. The 5G communication subsystem provides bidirectional data transmission between the shore and ship. The uplink transmits attitude data, control commands, and voice commands with a bandwidth of not less than 50 Mbps; the downlink transmits video streams, sensor data, and status feedback information with a bandwidth of not less than 500 Mbps; and the system end-to-end latency is not greater than 20 ms.

[0083] The 5G communication subsystem includes: Shore-side 5G base station: Supports SA standalone networking, operating frequency band n78 (3.5GHz) or n79 (4.9GHz), output power 40W, coverage radius 3-5km, installed at the highest point in the port to ensure network coverage for ships in the pilotage area.

[0084] Shipboard 5G CPE: Utilizes Huawei 5G CPE Pro 2 or equivalent products, supporting the Sub-6GHz band, with uplink peak rates of no less than 230Mbps and downlink peak rates of no less than 1.65Gbps. The CPE is installed on top of the ship's bridge and equipped with an omnidirectional antenna to ensure signal stability.

[0085] Network Slice Configuration 230: Configure a dedicated network slice for the pilotage service to guarantee end-to-end QoS. Uplink bandwidth is guaranteed at 50Mbps, downlink bandwidth at 500Mbps, end-to-end latency is ≤20ms, and packet loss rate is ≤0.1%.

[0086] The shipboard execution system includes a humanoid robot, a perception system integrated into the humanoid robot, and a control and computing system. The control and computing system receives and parses robot control commands and drives the humanoid robot's joints to perform corresponding actions. The system includes a robot controller, a motion planner, and a real-time perception and obstacle avoidance module. The robot controller drives the robot's joints to perform movements. The motion planner integrates path planning and gait generation algorithms to convert discrete motion commands into a sequence of motion trajectories for the robot's feet. The real-time perception and obstacle avoidance module processes environmental data collected by the perception system and makes real-time adjustments based on the trajectories generated by the motion planner.

[0087] The control and computing system is also used to: perform local path planning and gait generation after receiving discrete motion commands generated by the lower limb walking intention recognition and mapping module; drive the robot to execute the planned gait; and perform real-time obstacle avoidance control during execution.

[0088] Humanoid robots are used to perform pilotage operations under the drive of control and computing systems; A perception system is used to collect first-person perspective videos of the humanoid robot. Specifically, the shipboard execution system includes: Humanoid robots: These robots possess bipedal walking ability, autonomous balance, and motion execution capabilities, enabling them to move and perform pilotage operations within the ship's bridge area. For example, they utilize mature commercial humanoid robot platforms (such as the Unitree H1, Fourier GR-1, or equivalent products), are 1.6-1.8m tall, weigh 40-60kg, and are bipedal with a walking speed of 0.3-1.0m / s. They also possess autonomous balance and obstacle avoidance capabilities. The robots are equipped with multiple servo joints: 2 degrees of freedom (yaw, pitch) in the neck, 7 degrees of freedom in a single arm (3 degrees of freedom in the shoulder + 1 degree of freedom in the elbow + 3 degrees of freedom in the wrist), 3 degrees of freedom in the hip joint, 1 degree of freedom in the knee joint, and 2 degrees of freedom in the ankle joint.

[0089] The perception system includes cameras mounted on the robot's head to capture first-person view video and encode it using H.265 encoding. The video resolution should be no less than 1280×720, and the frame rate should be 25-30fps. For example, an industrial camera (such as the Hikvision MV-CA series) can be mounted on the head, with a resolution of 1280×720, a frame rate of 30fps, and a horizontal field of view of 90°. The camera is connected to the robot's control board for real-time H.265 encoding at a bitrate of 3-5Mbps. Multiple cameras can be optionally equipped to provide front, left, and right multi-view images.

[0090] Control and Computing System: This includes NVIDIA computing boards used to receive and parse attitude control commands transmitted from the shore-based system, perform path planning, obstacle avoidance control, and motion planning, and generate and execute control commands for each joint of the robot. For example, it uses an NVIDIA Jetson AGX Orin or Xavier NX computing board with an AI computing power of at least 100 TOPS and at least 16GB of memory. The board runs the ROS robot operating system and integrates attitude parsing, path planning, obstacle avoidance control, and motion planning modules.

[0091] Compared with the prior art, the present invention has the following significant advantages: Eliminating safety risks during boarding and disembarking: Pilots no longer need to board the ship, completely avoiding safety accidents such as falls and slips during the boarding and disembarking process, fundamentally solving the most dangerous aspect of pilotage operations.

[0092] Achieving all-weather operation capability: Even under extreme weather conditions such as strong winds, waves, and typhoons, the robot can still work normally, ensuring the continuity of pilotage operations and avoiding port operation interruptions and economic losses due to weather.

[0093] Maintaining a natural operating experience: Through multi-view posture fusion and hierarchical mapping strategies, the robot achieves precise conversion from human posture to robot motion, making the navigator's operating experience close to that of traditional navigation, with low learning costs.

[0094] Achieving low-latency hand-eye coordination: The system's end-to-end latency is <300ms, meeting the hand-eye coordination requirements of human perception, and the navigator will not experience obvious operational delays or incoordination.

[0095] Improve pilotage efficiency: A single pilot can simultaneously monitor the pilotage operations of multiple vessels on shore, or provide pilotage services to other vessels during pilotage breaks, significantly improving the utilization rate of pilotage resources.

[0096] Highly innovative in technology: For the first time in China and abroad, multi-view attitude fusion technology, hierarchical attitude mapping strategy and humanoid robot technology are applied to the field of ship pilotage, which has significant technological innovation and application value.

[0097] See Figure 10 This figure is a flowchart illustrating a smart navigation method based on a humanoid robot and posture mapping technology provided by an embodiment of the present invention. The complete navigation operation process based on the smart navigation system of the present invention is as follows: S1. System Initialization: The pilot enters the shore control center, and the system starts the attitude capture subsystem, VR display subsystem, and attitude mapping subsystem.

[0098] S2. Camera Calibration Verification: The system automatically checks whether the camera calibration parameters are valid. If the calibration parameters are expired (>30 days) or a change in camera position is detected, a recalibration prompt will be given.

[0099] S3. Deployment of the robot to the ship's bridge: The staff transports the robot to the ship's bridge and places it in the center of the bridge. The robot self-checks its joints and sensors to confirm that it is in normal condition.

[0100] S4. Establish 5G communication link: The ship-side 5G CPE establishes a connection with the shore-side 5G base station, the system tests the network quality (bandwidth, latency, packet loss rate), and after confirming that the requirements are met, a dedicated communication link is established.

[0101] S5. Initiate Attitude Capture: The multi-view attitude capture subsystem simultaneously captures the pilot's full-body attitude information from different perspectives. The pilot wears a VR headset and enters a 6m×6m attitude capture area. The system activates the OpenPose algorithm to capture the pilot's attitude in real time and displays an overlay of skeletal joint points for the pilot's confirmation.

[0102] S6. Posture Mapping and Command Generation: The posture mapping and control subsystem employs a hierarchical mapping strategy to convert full-body posture information into robot control commands. This hierarchical mapping strategy includes at least precise mapping of the head, inverse kinematic mapping of the upper limbs, and recognition and discrete command mapping of the lower limbs' walking intentions. The posture mapping subsystem performs hierarchical mapping processing on the captured posture data to generate control commands for the robot's head, upper limbs, torso, and lower limbs.

[0103] S7. Control Command Transmission: A low-latency bidirectional data transmission link is established between the shore-based control center and the ship-based execution system via the 5G communication subsystem to transmit robot control commands uplink to the ship-based execution system. Control commands are transmitted to the ship-based robot via the 5G network uplink, using the UDP+FEC (Forward Error Correction) protocol to ensure low latency and reliability.

[0104] S8. Robot Action Execution: The control and computing system receives and parses robot control commands, driving the humanoid robot's joints to perform corresponding actions. The humanoid robot performs pilotage operations under the control and computing system's guidance. The ship's control system receives commands, performs path planning and obstacle avoidance, and drives the robot's joints to perform corresponding actions. The robot moves within the bridge area, interacts with the crew via voice, and conveys pilotage commands.

[0105] S9. Video Acquisition and Transmission: The humanoid robot's first-person perspective video is acquired through the sensing system and transmitted to the shore control center via the 5G communication subsystem. The robot's head camera acquires first-person perspective video in real time, which is then encoded in H.265 and transmitted to the shore via the 5G network downlink.

[0106] S10. VR Display and Hand-Eye Loop Control: The VR display subsystem displays first-person perspective video in real time. The navigator watches the robot's first-person view through a VR headset and adjusts their posture based on visual feedback, forming a closed-loop control system for hand-eye coordination. The system monitors end-to-end latency in real time, ensuring it is <300ms.

[0107] S11. Pilotage Operation Completed: After the vessel successfully berths or departs, the pilot issues a completion command, the robot returns to its initial position, and the system saves all data (attitude, video, commands, etc.) of the pilotage process for post-operation analysis.

[0108] Throughout the pilotage process, the network status, robot status, and pilot status are continuously monitored. If an anomaly is detected (such as network interruption or robot malfunction), an emergency mechanism is immediately triggered: ① The robot automatically returns to its default safe position; ② Remote control is suspended; ③ The crew is notified to conduct manual pilotage; ④ The system attempts to re-establish the connection, and if the connection is restored, the pilotage operation continues.

[0109] Since the intelligent navigation method based on humanoid robots and posture mapping technology is the same as the intelligent navigation system based on humanoid robots and posture mapping technology provided in the above system embodiments, and the specific implementation of each step of the intelligent navigation method based on humanoid robots and posture mapping technology is based on the same concept as the above system embodiments, the specific implementation of each step of the intelligent navigation method based on humanoid robots and posture mapping technology can be referred to the description of the intelligent navigation system based on humanoid robots and posture mapping technology in the above system embodiments, and will not be repeated here.

[0110] This invention also provides a device comprising: a processor and a memory; The memory is used to store instructions; The processor is used to execute the instructions in the memory to perform the functions of the intelligent navigation system based on humanoid robot and posture mapping technology mentioned in the above embodiments.

[0111] It should be noted that the hardware structure of the devices provided in the embodiments of the present invention can all be as follows. Figure 11 The structure shown, Figure 11 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention.

[0112] Please see Figure 11 As shown, device 1100 includes: a processor 1110, a communication interface 1120, and a memory 1130. The number of processors 1110 in device 1100 can be one or more. Figure 11 Taking a processor as an example, in this embodiment of the invention, the processor 1110, communication interface 1120, and memory 1130 can be connected via a bus system or other means. Figure 11 Taking the connection between China and Israel via bus system 1140 as an example.

[0113] Processor 1110 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. Processor 1110 may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0114] The memory 1130 may include volatile memory, such as random-access memory (RAM); the memory 1130 may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 1130 may also include a combination of the above types of memory.

[0115] Optionally, the memory 1130 stores an operating system and programs, executable modules, or data structures, or subsets thereof, or extended sets thereof. The programs may include various operation instructions for implementing various operations. The operating system may include various system programs for implementing various basic business processes and handling hardware-based tasks. The processor 1110 can read the programs in the memory 1130 to implement the functions of the intelligent navigation system based on humanoid robots and posture mapping technology provided in this embodiment of the invention.

[0116] The bus system 1140 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus system 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0117] This invention also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the functions of the intelligent navigation system based on humanoid robot and posture mapping technology mentioned in the above embodiments.

[0118] This invention also provides a computer program product containing instructions that, when run on a computer, causes the computer to perform the functions of the intelligent navigation system based on humanoid robots and posture mapping technology mentioned in the above embodiments.

[0119] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A smart navigation system based on humanoid robots and posture mapping technology, characterized in that, include: The system comprises a shore-based control center, a 5G communication subsystem, and a ship-based execution system. The shore-based control center includes a multi-view attitude capture subsystem, a VR display subsystem, and an attitude mapping and control subsystem. The ship-based execution system includes a humanoid robot, a perception system integrated into the humanoid robot, and a control and computing system. The multi-view attitude capture subsystem is used to simultaneously capture the pilot's full-body attitude information from different perspectives; The posture mapping and control subsystem is used to convert the whole-body posture information into robot control commands using a hierarchical mapping strategy. The hierarchical mapping strategy includes at least precise mapping of the head, inverse kinematic mapping of the upper limbs, and recognition and discrete command mapping of the walking intention of the lower limbs. The 5G communication subsystem is used to establish a low-latency bidirectional data transmission link between the shore-based control center and the ship-based execution system, and to transmit the robot control commands uplink to the ship-based execution system. The control and computing system is used to receive and parse the robot control commands and drive the joints of the humanoid robot to perform corresponding actions. The humanoid robot is used to perform piloting operations under the drive of the control and computing system; The sensing system is used to collect first-person perspective video of the humanoid robot; The 5G communication subsystem is used to transmit the first-view video downlink to the shore control center; The VR display subsystem is used to display the video from the first perspective in real time.

2. The system according to claim 1, characterized in that, The hierarchical mapping strategy includes a head posture mapping module, an upper limb posture mapping module, and a lower limb walking intention recognition and mapping module. The head posture mapping module is used to map the pose data of the pilot's head to the joint control commands of the humanoid robot's head. The upper limb posture mapping module is used to generate joint control commands for the humanoid robot arm by solving inverse kinematics based on the three-dimensional coordinates of the pilot's upper limb joints. The lower limb walking intention recognition and mapping module is used to identify gait intentions by analyzing the temporal trajectory of the pilot's lower limb joints and generate corresponding discrete movement commands.

3. The system according to claim 2, characterized in that, The lower limb walking intention recognition and mapping module includes a feature extraction unit, a classification unit, and an instruction generation unit; The feature extraction unit is used to extract gait features from the temporal trajectory of the pilot's lower limb joints, the gait features including stride frequency, stride length and centroid displacement; The classification unit is used to analyze the gait features using a long short-term memory network to identify gait intentions of walking forward, walking backward, moving laterally, or stopping. The instruction generation unit is used to convert the identified gait intention into corresponding discrete motion instructions.

4. The system according to claim 2, characterized in that, The upper limb posture mapping module is specifically used for: Calculate the target vector of the pilot's wrist relative to the shoulder joint; The target vector is transformed from the human coordinate system to the robot coordinate system and scaled according to a preset arm length scaling factor; Based on the scaled target vector, the target angles of the shoulder and elbow joints of the humanoid robot arm are calculated using an inverse kinematics algorithm.

5. The system according to claim 2, characterized in that, The control and computing system is also used for: After receiving the discrete motion command generated by the lower limb walking intention recognition and mapping module, local path planning and gait generation are performed; Drive the robot to execute the planned gait and perform real-time obstacle avoidance control during the execution process.

6. The system according to claim 5, characterized in that, The control and computing system includes a robot controller, a motion planner, and a real-time perception and obstacle avoidance module; The robot controller is used to drive the robot's joints to perform movements; The motion planner integrates a path planning algorithm and a gait generation algorithm to convert the discrete motion commands into a sequence of motion trajectories of the robot's feet. The real-time perception and obstacle avoidance module is used to process the environmental data collected by the perception system and make real-time adjustments based on the trajectory generated by the motion planner.

7. The system according to claim 1, characterized in that, The multi-view attitude capture subsystem is specifically used for: Images of the pilot were simultaneously captured from different perspectives using multiple cameras. For each viewpoint image, run a human pose estimation algorithm to extract two-dimensional human joints; Based on the calibration parameters of multiple cameras, multiple two-dimensional observations of the same joint point are fused using triangulation to calculate its coordinates in three-dimensional space.

8. A smart navigation method based on humanoid robots and posture mapping technology, characterized in that, The method includes: The multi-view attitude capture subsystem simultaneously captures the pilot's full-body attitude information from different perspectives; The posture mapping and control subsystem employs a hierarchical mapping strategy to convert the whole-body posture information into robot control commands. The hierarchical mapping strategy includes at least precise mapping of the head, inverse kinematic mapping of the upper limbs, and recognition and discrete command mapping of the lower limbs' walking intentions. A low-latency bidirectional data transmission link is established between the shore-based control center and the ship-based execution system via a 5G communication subsystem to transmit the robot control commands uplink to the ship-based execution system. The control and computing system receives and parses the robot control commands, and drives the joints of the humanoid robot to perform corresponding actions. Pilotage operations are performed by a humanoid robot driven by the control and computing system. The humanoid robot's first-person perspective video is collected through a sensing system; The first-view video is transmitted downlink to the shore control center via the 5G communication subsystem. The first-view video is displayed in real time through the VR display subsystem.

9. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions; The processor is configured to execute the instructions in the memory and perform the functions of the system according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Including instructions that, when executed on a computer, cause the computer to perform the functions of the system described in any one of claims 1-7 above.

Images