AR-based wearable upper limb assistance exoskeleton control method and system

Abstract

The invention discloses a wearable upper limb assisting exoskeleton control method and system based on AR, and relates to the technical field of exoskeleton application. Three-dimensional space data and image data of an automobile chassis working area are obtained; inputting the depth image obtained after three-dimensional space data conversion and image data into a neural network CNN model, and extracting a feature information target component image; comparing the target component image with a reference image in a normal state, and judging the working state of the target component; converting the three-dimensional space data of the target component into an equipment display coordinate system, and superposing a preset standard virtual information base above the target component based on a visual SLAM technology; an EMG signal and a shoulder joint angle are obtained, and the muscle fatigue degree is calculated; operation steps are executed according to the standard virtual information base in combination with the upper limb assisting exoskeleton, and the operation sequence and the action posture are adjusted in real time according to the virtual information and the muscle fatigue degree. The AR technology is combined with the exoskeleton, so that the problems of excessive fatigue, sports injury and the like can be effectively avoided.

Description

Technical Field

[0001] The present invention relates to the field of exoskeleton application technology, and in particular to an AR-based wearable upper limb power-assisted exoskeleton control method and system. Background Art

[0002] When assembling, inspecting and repairing automobile chassis parts, workers are often required to maintain an uncomfortable posture for a long time to lift heavy objects. Since the human body has a limited time to lift heavy objects, workers need to suspend work many times, resulting in low work efficiency. In addition, the long-term lifting process can easily cause muscle strain and joint pain to workers, and there is a certain risk of work-related accidents.

[0003] In order to solve the above-mentioned problems existing in the process of mechanical maintenance, there are some solutions combining AR technology in the prior art, for example: Chinese invention patent CN112085223A discloses an induction system and method for mechanical maintenance, including AR glasses, an external input controller and a server, the AR glasses are used to collect image information, the server performs deep training according to the image information to obtain a training model, and the external input controller is used to send control information to the AR glasses; Chinese invention patent CN113570732A discloses a shield maintenance auxiliary method based on AR technology, including a three-dimensional tracking registration unit, a virtual-reality fusion display unit, a human-computer interaction unit, and a maintenance auxiliary management unit, which combines AR technology with shield maintenance equipment. After the maintenance personnel enter the maintenance scene, they do not need to blindly observe the fault information and consult various maintenance manuals and maintenance information.

[0004] Although the above scheme improves the maintenance efficiency and accuracy to a certain extent, it does not solve the problem of muscle fatigue caused by the direct operation of the operator. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide an AR-based wearable upper limb assisted exoskeleton control method and system. By combining AR technology with the exoskeleton, the physical condition of the operator (such as muscle fatigue, joint angle, etc.) can be monitored in real time, and the operator can be prompted to adjust his posture or movement in time, thereby effectively avoiding problems such as excessive fatigue and sports injuries.

[0006] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides an AR-based wearable upper limb power-assisted exoskeleton control method, comprising:

[0008] Acquire the three-dimensional spatial data and image data of the working area of ​​the automobile chassis;

[0009] The depth image and image data after the conversion of the three-dimensional space data are input into the neural network CNN model to extract the feature information target component image; the target component image is compared with the reference image in the normal state to determine the working state of the target component; when the working state of the component is abnormal, the precise position of the target component in the image is obtained based on the neural network CNN model;

[0010] The three-dimensional spatial data of the target component is converted into the device display coordinate system, and the preset standard virtual information library is superimposed on the target component based on the visual SLAM technology;

[0011] Obtain EMG signals and shoulder joint angles to calculate muscle fatigue levels;

[0012] Combined with the upper limb assisted exoskeleton, the operation steps are executed according to the standard virtual information library, and the operation sequence and action posture are adjusted in real time according to the virtual information and muscle fatigue level.

[0013] As a further implementation, the three-dimensional space data is obtained in the following manner:

[0014] The LiDAR depth sensor is used to scan the working area of ​​the automobile chassis, obtain the three-dimensional spatial data of the target components in real time, and generate three-dimensional point cloud data.

[0015] As a further implementation, the image data is obtained in the following manner:

[0016] Obtained based on HMD head mounted display device.

[0017] As a further implementation method, the three-dimensional spatial data of the target component is converted into a device display coordinate system based on a coordinate conversion algorithm.

[0018] As a further implementation, the EMG signal is acquired by an electromyographic sensor, and the shoulder joint angle is acquired by a rotary encoder;

[0019] The EMG signal and shoulder joint angle are transmitted to the MCU microcontroller after signal processing. The MCU microcontroller calculates the degree of muscle fatigue and transmits the degree of muscle fatigue to the AR system interface via wireless transmission.

[0020] As a further implementation, the degree of muscle fatigue is estimated by calculating the frequency spectrum of the EMG signal, wherein the average frequency is used as a fatigue indicator.

[0021] As a further implementation, the muscle activation intensity is represented by calculating the absolute value of the electromyographic signal and performing time averaging on the absolute value.

[0022] As a further implementation method, the supporting force of the upper limb assisted exoskeleton is a dynamic function based on muscle load and working posture; the supporting force of the upper limb assisted exoskeleton is adjusted according to the degree of muscle fatigue.

[0023] As a further implementation method, real-time feedback is provided to the virtual environment in the HMD head mounted display device according to the muscle state, and the feedback includes: posture correction and feedback, and real-time action guidance.

[0024] In a second aspect, an embodiment of the present invention further provides an AR-based wearable upper limb assist exoskeleton system, comprising:

[0025] An upper limb assist exoskeleton, which is equipped with an electromyographic sensor, a rotary encoder and an MCU microcontroller. The MCU microcontroller is used to calculate the degree of muscle fatigue based on the EMG signal obtained by the electromyographic sensor and the shoulder joint angle obtained by the rotary encoder, and transmit the degree of muscle fatigue to the HMD head-mounted display device;

[0026] LiDAR depth sensor, used to scan the working area of ​​the vehicle chassis to obtain three-dimensional spatial data;

[0027] The HMD head-mounted display device is used to acquire image data, combine the image data with three-dimensional space data and input them into the neural network model, and generate operation instructions after processing.

[0028] The beneficial effects of the present invention are as follows:

[0029] The present invention obtains real-time data from electromyographic sensors, LiDAR depth sensors, and rotary encoders, processes EMG signals, and calculates muscle activation, fatigue level, etc.; adjusts the support force of the exoskeleton according to real-time feedback, and provides posture and movement adjustment suggestions through the AR system; the exoskeleton adjusts the support force according to the electromyographic signals and fatigue level, and provides instant movement guidance through the AR interface; therefore, health monitoring and movement evaluation functions can be realized by combining AR with the exoskeleton, and the body status (such as muscle fatigue, joint angle, etc.) is displayed in real time, and prompts are given in time to adjust posture or movement, thereby effectively avoiding problems such as excessive fatigue and sports injuries. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0031] Figure 1 is a flow chart of the present invention according to one or more embodiments;

[0032] Figure 2 is a system block diagram of the present invention according to one or more embodiments. DETAILED DESCRIPTION

[0033] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0034] Embodiment 1:

[0035] This embodiment provides an AR-based wearable upper limb assisted exoskeleton control method, which combines an HMD head-mounted display device (AR system) with an upper limb assisted exoskeleton, and can accurately position virtual elements in the three-dimensional space of the real world, thereby providing intuitive guidance for operators; in this way, operators can obtain auxiliary information at the virtual level without leaving the actual working environment.

[0036] Specifically, Figure 1 As shown, the following steps are included:

[0037] S1: The worker wears an upper limb power-assist exoskeleton.

[0038] S2: The LiDAR depth sensor emits laser pulses and receives reflected signals, scans the working area of ​​the vehicle chassis, obtains the three-dimensional spatial data of each component in the working area of ​​the vehicle chassis in real time, and generates accurate three-dimensional point cloud data; at the same time, the camera of the HMD head-mounted display device captures the image of the working area of ​​the vehicle chassis.

[0039] The three-dimensional point cloud data is converted into a two-dimensional depth image. The conversion process is:

[0040] (1) Alignment of the camera coordinate system with the LiDAR coordinate system:

[0041] The three-dimensional point cloud coordinates of LiDAR are converted into coordinates in the camera coordinate system through the external parameter matrix composed of the rotation matrix R and the translation vector t.

[0042] The formula for converting the point cloud data in the LiDAR coordinate system to the camera coordinate system is:

[0043] P Camera =R·P LiDAR +t (1)

[0044] In formula (1), P LiDAR =[x LiDAR ,y LiDAR , z LiDAR ] T Represents the coordinates of a three-dimensional point in the LiDAR coordinate system; P Camera =[x Camera ,y Camera , z Camera ] TRepresents the coordinates of a 3D point in the camera coordinate system; the rotation matrix R represents the rotation relationship from the LiDAR coordinate system to the camera coordinate system. The rotation matrix is ​​a 3×3 orthogonal matrix in which each column is the direction of the coordinate axis after rotation; during the calibration process, the rotation matrix is ​​obtained by aligning the camera and LiDAR and using a checkerboard calibration plate. The translation vector t represents the translation offset from the origin of the LiDAR coordinate system to the origin of the camera coordinate system, which is obtained through the camera calibration process. Assuming that the positions of the camera and LiDAR are known, the translation vector can be calculated from the spatial distance between the two.

[0045] (2) Projection to the image plane:

[0046] The aligned 3D point cloud is projected onto the image plane of the camera using the camera intrinsic parameter matrix. The x and y coordinates of each 3D point will be mapped to the pixel position of the image, and the z coordinate will determine the depth value.

[0047] (3) Generate depth image:

[0048] The z value (depth) of each pixel will be stored in a depth image, which is a two-dimensional matrix in which each element represents the depth information of the corresponding pixel.

[0049] S3: The above two-dimensional depth image is combined with the image data obtained by the HMD head-mounted display device, and is sent to the convolutional neural network CNN model as an input image (composite image). The convolutional neural network CNN model extracts local features (such as the edge, texture, shape, etc. of the object) and depth information (such as the spatial structure and distance of the object) in the composite image; the CNN model is used to detect the target component on the depth map or RGB-D image, and the target component in the image is compared with the reference image to determine whether the target component is in normal working condition. If the target component is abnormal, CNN can locate the precise position of the target component and connect it with the three-dimensional point cloud data to further analyze its spatial position and state. The three-dimensional spatial data of the target component is converted into the device display coordinate system, and the preset standard virtual information library is superimposed on the target component based on the visual SLAM technology.

[0050] Furthermore, the depth image is combined with the RGB image obtained from the camera by direct superposition (feature-level fusion) to form a composite image containing RGB and depth information - RGB-D image: that is, each pixel contains not only color information (R, G, B) but also depth information (D), and the depth image is usually added to the RGB image as the fourth channel. Each pixel of this image contains the color and depth information of the object, allowing CNN to learn the color features and spatial depth features of the object at the same time.

[0051] This embodiment takes screws as the target component for detailed description:

[0052] The two-dimensional depth image of the screw is combined with the RGB image and input into the convolutional neural network CNN model to extract the local features and depth information of the composite image of the screw. The CNN model is used to detect the screw on the depth map or RGB-D image, and the screw in the image is compared with the screw in the normal state of the reference image to determine whether the screw is in normal working condition. If the position of the screw relative to other parts changes or the thread is exposed more, it is judged that the screw may be loose; at this time, the neural network CNN model will generate a bounding box to mark the exact position of the screw in the image; in the AR system, the coordinate conversion algorithm is used to convert the three-dimensional space data of the screw into the display coordinate system of the device, and the preset standard virtual information library is accurately and stably superimposed on the screw through the visual SLAM technology. For screws, the preset standard virtual information library refers to the loosening repair steps, including: loosening repair steps "please use a cross screwdriver to tighten the screw", repair tool recommendation "use a cross screwdriver of model X", operation prompt "insert the screwdriver into the screw and rotate clockwise" and virtual guidance "tighten the screw to the standard of 10N·m".

[0053] In this embodiment, the AR system uses a coordinate conversion algorithm to convert the three-dimensional spatial data of the target component into the display coordinate system of the device, and uses visual SLAM technology to accurately and stably superimpose a preset standard virtual information library on top of the target component. Then, the upper limb assisted exoskeleton transmits the fatigue status to the AR system interface through wireless transmission. In the above process, in order to ensure that the different information provided by the AR system and the upper limb assisted exoskeleton can be displayed simultaneously and accurately in the HMD, these two types of information must be coordinated through information fusion and a unified display mechanism.

[0054] Specifically, the information hierarchy management method is adopted to display different data sources in layers. The virtual targets provided by the AR system (such as screw images, tool suggestions, etc.) can be displayed on the main layer of the HMD screen, while the sensor feedback of the exoskeleton (such as fatigue prompts or posture suggestions of electromyographic signals) can be displayed on the secondary information layer. This information can be displayed through transparent, semi-transparent graphics or dynamic pop-up windows to avoid interfering with the staff's vision, while ensuring that necessary information can be displayed at any time.

[0055] S4: The electromyographic sensor on the upper limb assisted exoskeleton obtains the EMG signal, and the rotary encoder obtains the shoulder joint angle. The EMG signal and the shoulder joint angle are transmitted to the MCU microcontroller after signal processing. The MCU microcontroller calculates the degree of muscle fatigue in real time and transmits the degree of muscle fatigue to the AR system interface via wireless transmission (such as Bluetooth).

[0056] Furthermore, the EMG signals need to be processed in order to calculate muscle load and fatigue; the EMG signals are converted into muscle activity levels or force output.

[0057] Among them, muscle activation is expressed as:

[0058]

[0059] In formula (2), A(t) is the muscle activation at time t, EMG(τ) is the EMG signal in the time period [tT, t], and T is the time length of the integration window (for example, a short time window of 50 ms to 200 ms), which determines the smoothness of the calculation.

[0060] Formula (2) expresses the activation intensity of the muscle by calculating the absolute value of the electromyographic signal and performing time averaging on it; a higher value indicates that the muscle is in a higher activation state, and vice versa, it indicates that the muscle is in a lower activity state.

[0061] As the working time increases, muscles will fatigue, causing the frequency characteristics of the EMG signal to change. Usually, the low-frequency components of the electromyographic signal (such as 20-50Hz) will increase, while the high-frequency components (such as 100-500Hz) will decrease. The degree of muscle fatigue can be estimated by calculating the spectrum of the EMG signal. The common fatigue indicator is the average frequency, which is calculated as follows:

[0062]

[0063] In formula (3), P(f) is the power spectral density at frequency f (calculated by Fourier transform), f max It is the highest frequency of the EMG signal.

[0064] The decrease in MNF indicates an increase in muscle fatigue. If the MNF value drops too quickly, the upper limb assist exoskeleton can issue a rest reminder or increase support strength based on this signal.

[0065] Furthermore, for a continuous-time signal x(t), its Fourier transform is defined as:

[0066]

[0067] In formula (4), X(f) is the complex representation of the signal at frequency f (including amplitude and phase information), and x(t) is the signal in the time domain; -j2πft Is a complex exponential function, which is used to decompose the frequency components of the signal.

[0068] The power spectral density P(f) is the square of the energy of the frequency domain signal obtained by Fourier transform:

[0069] P(f)=|X(f)|2 (5)

[0070] In formula (5), |X(f)| is the amplitude of X(f).

[0071] S5: The operation sequence and action posture are adjusted in real time according to the virtual information of the HMD head-mounted display device and the degree of muscle fatigue. The upper limb assist exoskeleton provides mechanical support and posture monitoring. At the same time, the HMD head-mounted display device adjusts the display position and angle of the virtual graphics in real time to ensure that the position and status of the virtual target parts can be clearly seen, avoiding fatigue or errors caused by inappropriate postures, and improving the accuracy and efficiency of operations.

[0072] The upper limb power-assisted exoskeleton of this embodiment is used to reduce the physical burden of the operator, so its supporting force needs to be adjusted in real time according to the muscle load, posture and movement, and the control strategy adopted includes dynamic supporting force adjustment based on electromyographic signals.

[0073] The support force of the exoskeleton is a dynamic function based on muscle load and working posture, expressed as:

[0074]

[0075] In formula (6), F exo is the support force provided by the exoskeleton, k is the gain coefficient of the exoskeleton, which determines the degree of response of the exoskeleton to muscle load; θ is the angle between the operator and the exoskeleton support point (shoulder angle).

[0076] As muscle activation increases, the exoskeleton needs to provide more support. In addition, the angle parameter takes the operator's posture into account, ensuring that the support force can be adjusted according to the working angle.

[0077] As workers work longer hours, they may become fatigued. The exoskeleton can adjust its response based on the EMG signals of fatigue. For example, when fatigue is detected, the support force can be adjusted using the following formula:

[0078]

[0079] In formula (7), F exo_adjusted is the exoskeleton support force after fatigue adjustment, MNF is the current average frequency (indicating the degree of fatigue), MNF0 is the initial average frequency (indicating the state without fatigue); α is the adjustment coefficient, which controls the influence of fatigue on the support force.

[0080] Formula (7) shows that as fatigue increases, the exoskeleton will provide more support to help workers reduce muscle load and extend working time.

[0081] In AR systems, EMG signals can be used to provide real-time feedback on the operator's muscle activity, helping them adjust their posture and movements. The virtual environment in the AR system can provide real-time feedback based on the operator's muscle status, guiding them to adjust their behavior.

[0082] 1) Posture correction and feedback:

[0083] The AR system uses a posture correction algorithm to ensure that the operator's posture meets the requirements. The following calculation methods can be used:

[0084] Error(t)=||θ ideal -θ real (t)|| (8)

[0085] In formula (8), Error(t) is the error between the operator’s current posture and the ideal posture, θ ideal is the ideal target posture (angle of the shoulder), θ real (t) is the actual posture angle of the operator at a certain moment.

[0086] This error can be used to drive the virtual feedback of the AR system, such as displaying virtual alignment lines or guidance prompts to prompt the operator to adjust their posture.

[0087] 2) Real-time action guidance

[0088] By combining EMG signals with the AR system, real-time action guidance can be achieved. For example, when the operator's posture is detected to deviate from the ideal state, the AR system can make real-time corrections through virtual task markers or guide lines:

[0089] Guide_adjustment(t) = β·(θ ideal -θ real (t)) (9)

[0090] In formula (9), Guide_adjustment(t) is the real-time action adjustment prompt of the AR system; β is the adjustment factor, which determines the strength of the prompt.

[0091] This embodiment is based on AR technology, which helps operators locate maintenance parts or installation positions through visual guidance and virtual markings, improves the accuracy of operations, reduces errors and improves work efficiency; combined with AR technology, real-time operation guidance and learning prompts can be obtained through the AR interface without complicated training courses; virtual demonstrations can be used to quickly master the key points of operation, thereby reducing training costs and shortening adaptation time. AR combined with exoskeletons can realize health monitoring and motion assessment functions, and can display the body status (such as muscle fatigue, joint angle, etc.) in real time, and prompt in time to adjust posture or movement, thereby effectively avoiding problems such as excessive fatigue and sports injuries.

[0092] Embodiment 2:

[0093] This embodiment provides an AR-based wearable upper limb assist exoskeleton system. Figure 2 As shown, it includes an upper limb power-assisted exoskeleton, a LiDAR depth sensor, and an HMD head-mounted display device, wherein the upper limb power-assisted exoskeleton is equipped with an electromyographic sensor, a rotary encoder, and an MCU microcontroller.

[0094] The electromyographic sensor is a sensor used to measure and record muscle electrical activity. It can detect the electrical signal generated when the muscle contracts; it can reflect the working state, load and fatigue of the muscle. The rotary encoder is used to directly measure the change in the angle θ of the shoulder joint. The MCU microcontroller is used to calculate the degree of muscle fatigue based on the EMG signal obtained by the electromyographic sensor and the shoulder joint angle obtained by the rotary encoder, and transmit the degree of muscle fatigue to the HMD head-mounted display device.

[0095] LiDAR depth sensors are used to scan the working environment and collect three-dimensional spatial data of various components of the automobile chassis in real time. HMD head-mounted display devices are displays worn on the head. Based on AR technology, images of the virtual world can be directly seen while maintaining interaction with the real world. AR can be superimposed on the exoskeleton through graphics to guide operators to accurately find the parts that need to be repaired or installed (target parts), and display the best operation path in real time.

[0096] Graphics overlay refers to a technology that overlays virtual graphics information onto the real environment in real time, allowing operators to see important guidance information through the augmented reality interface in their field of view, such as the location, status, and operation path of the parts that need to be repaired or installed. This technology is not just a simple presentation of information on the display screen. Through the combination of head-mounted display devices (HMD) and exoskeletons, it can accurately position virtual elements in the three-dimensional space of the real world, thereby providing intuitive guidance for operators; in this way, operators can obtain auxiliary information at the virtual level without leaving the actual working environment.

[0097] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A wearable upper limb power-assisted exoskeleton control method based on AR, characterized in that: include: Acquire the three-dimensional spatial data and image data of the working area of ​​the automobile chassis; The depth image and image data converted from the three-dimensional spatial data are input into the neural network CNN model to extract feature information; Compare the target component image with the reference image in normal state to determine the working state of the target component; when the component is in abnormal working state, obtain the precise position of the target component in the image based on the neural network CNN model; The three-dimensional spatial data of the target component is converted into the device display coordinate system, and the preset standard virtual information library is superimposed on the target component based on the visual SLAM technology; Obtain EMG signals and shoulder joint angles to calculate muscle fatigue levels; Combined with the upper limb assisted exoskeleton, the operation steps are executed according to the standard virtual information library, and the operation sequence and action posture are adjusted in real time according to the virtual information and muscle fatigue level.

2. The AR-based wearable upper limb power-assisted exoskeleton control method according to claim 1, characterized in that: The three-dimensional spatial data is obtained in the following manner: The LiDAR depth sensor is used to scan the working area of ​​the automobile chassis, obtain the three-dimensional spatial data of the target components in real time, and generate three-dimensional point cloud data.

3. The AR-based wearable upper limb power-assisted exoskeleton control method according to claim 1, characterized in that: The image data is obtained in the following manner: Obtained based on HMD head mounted display device.

4. The AR-based wearable upper limb assist exoskeleton control method according to claim 1, characterized in that: The three-dimensional space data of the target component is converted into the device display coordinate system based on the coordinate conversion algorithm.

5. The AR-based wearable upper limb power-assisted exoskeleton control method according to claim 1, characterized in that: The EMG signal is obtained by an electromyographic sensor, and the shoulder joint angle is obtained by a rotary encoder; The EMG signal and shoulder joint angle are transmitted to the MCU microcontroller after signal processing. The MCU microcontroller calculates the degree of muscle fatigue and transmits the degree of muscle fatigue to the AR system interface via wireless transmission.

6. The AR-based wearable upper limb power-assisted exoskeleton control method according to claim 5, characterized in that: The degree of muscle fatigue is estimated by calculating the frequency spectrum of the EMG signal, where the average frequency is used as the fatigue indicator.

7. The AR-based wearable upper limb power-assisted exoskeleton control method according to claim 5, characterized in that: The muscle activation intensity is represented by calculating the absolute value of the electromyographic signal and averaging it over time.

8. The AR-based wearable upper limb power-assisted exoskeleton control method according to claim 1 or 5, characterized in that: The supporting force of the upper limb assisted exoskeleton is a dynamic function based on muscle load and working posture; the supporting force of the upper limb assisted exoskeleton is adjusted according to the degree of muscle fatigue.

9. The AR-based wearable upper limb power-assisted exoskeleton control method according to claim 1 or 5, characterized in that: Provide real-time feedback to the virtual environment in the HMD head mounted display device according to the muscle state, wherein the feedback includes posture correction and feedback, and real-time action guidance.

10. A wearable upper limb assist exoskeleton system based on AR, characterized in that: include: An upper limb assist exoskeleton, which is equipped with an electromyographic sensor, a rotary encoder and an MCU microcontroller. The MCU microcontroller is used to calculate the degree of muscle fatigue based on the EMG signal obtained by the electromyographic sensor and the shoulder joint angle obtained by the rotary encoder, and transmit the degree of muscle fatigue to the HMD head-mounted display device; LiDAR depth sensor, used to scan the working area of ​​the vehicle chassis to obtain three-dimensional spatial data; The HMD head-mounted display device is used to acquire image data, combine the image data with three-dimensional space data and input them into the neural network model, and generate operation instructions after processing.

Images