An assisted system adapted for teleoperation

Abstract

The application discloses an auxiliary system suitable for remote operation, comprising a display end and an operation end, and the auxiliary system is arranged on the display end or the operation end; the auxiliary system comprises a motion input unit, a general instruction unit, a self-learning instruction unit and a processor; the motion input unit comprises an instruction input module, a storage module and an instruction calling module; the general instruction unit comprises a real-time target positioning module, a trajectory planning module, a pose estimation module, an adjustment module, a command execution module, a data generation module and a model training data generation module; the self-learning instruction unit comprises a data acquisition module, a data processing module, a body intelligence model training module, a control strategy generation module, an operation execution module, a result feedback and analysis module. Compared with a traditional remote operation system without the auxiliary system, the system not only greatly improves operation efficiency and makes overall operation more refined, but also effectively reduces uncertainty caused by human operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of teleoperation assistance systems, and in particular to an assistance system adapted for teleoperation. BACKGROUND

[0002] With the continuous development of industrial automation and intelligence, teleoperation systems are becoming increasingly widely used in complex environments. These systems perform tasks by remotely controlling robotic arms or other actuators and have been widely used in manufacturing, medicine, rescue and other fields.

[0003] Currently, many innovations in assistance systems focus on achieving more precise control by adding various sensors. However, the core of the operation still relies on human operation. This means that the operator must carefully monitor the position of the grasp and drop points during each task execution, and once these points change, repositioning is required. This not only increases the complexity of the operation, but also reduces work efficiency.

[0004] To solve the above problems, we have developed an assistance system adapted for teleoperation. SUMMARY

[0005] The present application discloses an assistance system adapted for teleoperation, which aims to solve the technical problems in the background art.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] An assistance system adapted for teleoperation, comprising a display end and an operation end, the display end or operation end is provided with an assistance system;

[0008] The assistance system comprises an action input unit, a regular instruction unit, a self-learning instruction unit and a processor;

[0009] The action input unit comprises an instruction input module, a storage module and an instruction retrieval module;

[0010] The regular instruction unit comprises a real-time target positioning module, a trajectory planning module, a pose estimation module, an adjustment module, a command execution module, a data generation module and a model training data generation module;

[0011] The self-learning instruction unit comprises a data acquisition module, a data processing module, a body intelligence model training module, a control strategy generation module, an operation execution module, a result feedback and analysis module;

[0012] The processor is in bidirectional electrical connection with the regular instruction unit and the self-learning instruction unit, respectively;

[0013] The display end comprises an input camera A and a display module.

[0014] The operation end comprises a post-entry camera B, a sensor group and a driving assembly.

[0015] In a preferred scheme, the instruction entry module acquires the user's action information through the entry camera A, performs action recognition and processing relying on the visual nervous system, and converts visual information into human joint pose;

[0016] The storage module stores the information code generated by the instruction entry module, and classifies and labels the same;

[0017] The instruction calling module retrieves and calls the action information code stored in the storage module according to the calling command input by the display end.

[0018] In a preferred scheme, the real-time target positioning module relies on the visual nervous network, captures the position information of the target in real time through the entry camera B, analyzes the same through the visual nervous network, acquires the current position and state of the target object, and plans a suitable grasping position according to the corresponding instruction;

[0019] The trajectory planning module continuously calculates and generates the optimal operation trajectory by real-time positioning the position information of the target through the visual nervous network and combining the task requirements;

[0020] The pose estimation module optimizes the accuracy of identifying and positioning the target by analyzing the image data acquired by the entry camera B in cooperation with the trajectory planning module;

[0021] The adjustment module dynamically adjusts the operation path, force, speed and other parameters based on the data sensed by the sensor group and the real-time detection of the visual nervous of the trajectory planning module and the pose estimation module, so as to ensure that the operation strategy is adjusted in time according to the visual feedback during the task execution process;

[0022] The command execution module controls the driving assembly in the operation end to complete the operation task according to the action information code provided by the instruction calling module and the data obtained by the real-time target positioning module, the trajectory planning module, the pose estimation module and the adjustment module;

[0023] The data generation module is used to generate and record relevant operation data after and during the task execution, including the target positioning information provided by the visual nervous network and the execution feedback data;

[0024] The model training data generation module generates a data set for training the self-learning instruction unit by analyzing various real-time data generated by the operation end during the operation process.

[0025] In a preferred scheme, the data acquisition module enters real-time image, video and other data through the entry camera B, uses a visual neural network for preliminary processing, extracts key features including object position, shape, color and other information, and provides basic data for subsequent decision-making and operation;

[0026] The data processing module is used for further processing and analysis of the visual data acquired by the data acquisition module, and converts it into useful information for decision-making and operation. The data processing module can identify suitable operation positions and paths to provide input for subsequent control strategy generation;

[0027] The embodied intelligent model training module is used for continuous training and optimization of the large model. The module uses visual data generated by the data acquisition and data processing modules, and feedback data after execution of the control strategy generation module to continuously adjust and optimize the parameters of the large model. The latest trained model is transmitted to the control strategy generation module to ensure that the operation strategy is always based on the latest model inference result;

[0028] The control strategy generation module determines the best operation path and method based on the information provided by the data acquisition module and the data processing module, judges the state of the environment and the target object through the visual neural network, generates specific operation strategies automatically in combination with the embodied intelligent model training module, and transmits the operation strategies to the operation execution module;

[0029] The operation execution module is used for executing specific operation tasks according to the operation instructions provided by the control strategy generation module. During the execution process, the entry camera B will monitor the operation effect in real time, and generate new data through the data acquisition module;

[0030] The result feedback module evaluates the success rate, accuracy and other information of the operation through feedback information, and identifies the shortcomings in the operation;

[0031] The analysis module is used in cooperation with the result feedback module. Through analysis of the feedback data, optimization suggestions are proposed, which are transmitted to the embodied intelligent model training module together with the result feedback, so as to further train and optimize the model.

[0032] In a preferred scheme, the model training data generation module also synchronously transmits real-time data to the embodied intelligent model training module.

[0033] In a preferred scheme, the visual neural network uses a multi-layer convolutional neural network (CNN) to extract multi-scale features from the images acquired by the entry camera B, and accurately identifies and locates the position information of the target object.

[0034] In a preferred scheme, the multi-layer convolutional neural network is used for multi-layer convolution operation on image data obtained by the entry camera B, generates a feature map through feature extraction, pooling and activation operations, can process the displacement, scaling and rotation invariance of the image, and improves the accuracy of target recognition and positioning.

[0035] The auxiliary system for adapting remote operation provided by the application has the following advantages:

[0036] The application can automatically identify the current scene and generate a corresponding data chain by presetting common actions, digitizing these action data, and using a visual neural network to detect the operation in real time, thereby replacing the preset actions to perform operations. The system can quickly and efficiently complete the operation on the object by calling the corresponding instructions. In addition, the data collection of the preset actions provides continuous data support for the self-learning instruction unit, enabling the large model to grow rapidly. This design does not require frequent manual intervention when processing high-repetition and high-precision operation tasks. Compared with the traditional remote operation system without an auxiliary system, the system not only greatly improves the operation efficiency and makes the overall operation more refined, but also effectively reduces the uncertainty caused by human operation. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The structure of the auxiliary system for adapting remote operation provided by the application is shown in the schematic diagram. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the application will be described below in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments. The components of the embodiments of the application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the application. Based on the embodiments of the application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the application.

[0039] The auxiliary system for adapting remote operation disclosed by the application. EMBODIMENT

[0040] REFERENCE Figure 1As shown, an auxiliary system adapted for teleoperation includes a display end and an operation end, and the auxiliary system is arranged on the display end or the operation end; the auxiliary system includes an action input unit, a regular instruction unit, a self-learning instruction unit and a processor; the action input unit includes an instruction input module, a storage module and an instruction calling module; the regular instruction unit includes a real-time target positioning module, a trajectory planning module, a pose estimation module, an adjustment module, a command execution module, a data generation module and a model training data generation module; the display end includes an input camera A and a display module; the operation end includes an output camera B, a sensor group and a driving assembly. The instruction input module obtains the action information of the user through the input camera A, performs action recognition and processing relying on the visual nervous system, and converts visual information into action information code; the storage module stores the information code generated by the instruction input module, and classifies and names it; the instruction calling module retrieves and calls the action information code stored in the storage module according to the calling command input by the display end. The real-time target positioning module relies on the visual neural network, captures the position information of the target in real time through the input camera B, analyzes through the visual neural network, obtains the current position and state of the target object, and plans the appropriate grabbing position according to the corresponding instruction; the trajectory planning module locates the position information of the target in real time through the visual neural network, combines with the task requirements, continuously calculates and generates the best operation trajectory; the pose estimation module analyzes the image data obtained by the input camera B, cooperates with the trajectory planning module, calibrates the perception accuracy of the visual neural network, and optimizes the accuracy of identifying and positioning the target; the adjustment module dynamically adjusts the operation path, force, speed and other parameters based on the data perceived by the sensor group, and the real-time detection of the visual neural network of the trajectory planning module and the pose estimation module, ensures that the operation strategy is adjusted in time according to the visual feedback during the task execution process; the command execution module controls the driving assembly in the operation end to complete the operation task according to the action information code provided by the instruction calling module, and the data obtained by the real-time target positioning module, the trajectory planning module, the pose estimation module and the adjustment module; the data generation module is used to generate and record related operation data after the task is completed, including target positioning information provided by the visual neural network, execution feedback data, etc.; the model training data generation module generates a data set for training the self-learning instruction unit by analyzing various real-time data generated by the operation end during the operation process. The visual neural network extracts multi-scale features from the image obtained by the input camera B through a multi-layer convolutional neural network (CNN), and combines a target detection algorithm (such as Faster R-CNN or YOLO) to accurately identify and locate the position information of the target object. The multi-layer convolutional neural network is used for multi-layer convolution operation on the image data obtained by the input camera B, generates feature mapping through feature extraction, pooling and activation operations, can process the displacement, scaling and rotation invariance of the image, and improves the accuracy of target identification and positioning;

[0041] In this embodiment, the conventional instruction unit is responsible for converting user instructions into specific actions. First, the corresponding action is recorded through the recording camera 4, and the action data is stored in the storage module. When applicable, the corresponding instruction is retrieved through the instruction retrieval module.

[0042] The conventional instruction unit can set actions such as pressing, lifting, grabbing, twisting, and turning, etc. Specifically:

[0043] Press:

[0044] First, record the user's "press" action and generate corresponding data chain. In the subsequent use process, when the user selects to perform the "press" action, the system retrieves the stored "press" action code from the storage module. Then, the system obtains real-time image information of the operating end through the recording camera B, and analyzes the image through the real-time target positioning module to obtain the current position and state of the target object. Based on the real-time position information and state of the target object, the system calculates the best operation trajectory suitable for the current task requirements through the trajectory planning module. At this time, the pose estimation module analyzes the image data obtained by the recording camera B, and cooperates with the trajectory planning module to calibrate the perception accuracy of the visual neural network to ensure accurate identification and positioning in different coordinate systems. Then, the adjustment module adjusts the initial position of the operating end based on the environmental data perceived by the sensor group and the real-time detection results of the visual neural network. This module dynamically optimizes operation path, pressing force, speed, etc. to ensure that the "press" action can be accurately executed in the current scene. Finally, the command execution module controls the driving components in the operating end to execute the "press" action based on the "press" action code provided by the instruction retrieval module and the data support provided by the real-time target positioning module, trajectory planning module, pose estimation module, and adjustment module to ensure the smooth completion of the operation task.

[0045] Lift:

[0046] Firstly, the user's "lift" action is entered and the corresponding data chain is generated. In the subsequent use process, when the user selects to perform the "lift" action, the system retrieves the stored "lift" action code from the storage module. Subsequently, the system obtains real-time image information of the operating end through the entry camera B, and analyzes the image through the real-time target positioning module to obtain the current position, shape and weight of the target object, etc. According to the real-time state and position of the target object, the system calculates the best lifting path suitable for the current task requirement through the trajectory planning module. At this time, the pose estimation module will analyze the image data obtained by the entry camera B, and cooperate with the trajectory planning module to calibrate the perception accuracy of the visual neural network, so as to ensure accurate identification and positioning of the target object under different coordinate systems. Then, the adjustment module will adjust the initial position of the operating end based on the environmental data perceived by the sensor group and the real-time detection results of the visual neural network. This module will dynamically optimize the lifting path, lifting force, lifting speed and other parameters to ensure that the "lift" action can be safely and effectively performed in the current scene. Finally, the command execution module controls the driving components in the operating end to execute the "lift" action according to the "lift" action code provided by the instruction retrieval module, and combines the data support provided by the real-time target positioning module, the trajectory planning module, the pose estimation module and the adjustment module to ensure the accurate completion of the operation task.

[0047] Grab:

[0048] Firstly, the user's "grab" action is recorded and the corresponding data chain is generated. In the subsequent use process, when the user selects to perform the "grab" action, the system retrieves the stored "grab" action code from the storage module. The system obtains real-time image information of the operating end through the recording camera B, and analyzes the image through the real-time target positioning module to obtain the real-time position, shape and material of the target object, etc. According to the real-time state and position of the target object, the system calculates the best grabbing path suitable for the current task requirement through the trajectory planning module. At this time, the pose estimation module analyzes the image data obtained by the recording camera B, and cooperates with the trajectory planning module to calibrate the perception accuracy of the visual neural network, so as to ensure accurate identification, positioning and grabbing of the target object in different coordinate systems. Then, the adjustment module adjusts the initial position and grabbing parameters of the operating end based on the environmental data perceived by the sensor group and the real-time detection results of the visual neural network. This module dynamically optimizes the grabbing path, grabbing force and grabbing speed, etc. to ensure that the "grab" action can be accurately and safely executed in the current scene, avoiding damage to the target object or failure to grab. Finally, the command execution module controls the driving components in the operating end to execute the "grab" action according to the "grab" action code provided by the instruction retrieval module, and combines the data support provided by the real-time target positioning module, the trajectory planning module, the pose estimation module and the adjustment module to ensure the accurate completion of the operation task.

[0049] Turn:

[0050] Firstly, the "turn" action of the user is recorded, and the corresponding data chain is generated. In the subsequent use process, when the user selects to perform the "turn" action, the system retrieves the stored "turn" action code from the storage module. The system obtains real-time image information of the operating end through the recording camera B, and analyzes the image through the real-time target positioning module to obtain the real-time position, direction and current state of the target object, etc. According to the real-time state and position of the target object, the system calculates the best turning path and angle suitable for the current task requirement through the trajectory planning module. At this time, the pose estimation module analyzes the image data obtained by the recording camera B, and cooperates with the trajectory planning module to calibrate the perception accuracy of the visual neural network, so as to accurately control the turning direction and angle of the target object in different coordinate systems. Then, the adjustment module adjusts the initial position and turning parameters of the operating end based on the environmental data perceived by the sensor group and the real-time detection results of the visual neural network. This module dynamically optimizes the turning path, turning force, speed and other parameters to ensure that the "turn" action can be smoothly and accurately executed in the current scene, avoiding interference or damage to the target object during the turning process. Finally, the command execution module controls the driving components in the operating end to execute the "turn" action according to the "turn" action code provided by the instruction retrieval module, and combines the data support provided by the real-time target positioning module, the trajectory planning module, the pose estimation module and the adjustment module, to ensure the accurate completion of the operation task.

[0051] And when carrying some articles that may have a risk of falling, specifically:

[0052] Carrying multiple stacked boxes:

[0053] First, the real-time target positioning module captures the position information and relative pose of the stacked boxes through the visual neural network and real-time image capture of the input camera B. This module can analyze the precise position and pose of each box in the stack and determine the optimal grabbing point, thereby laying the foundation for subsequent stable handling. Next, the trajectory planning module calculates and generates the optimal operation trajectory based on the position information provided by the real-time target positioning module, continuously tracks the dynamic position of each box through the visual neural network, and combines the specific requirements of the operation task. This trajectory not only includes the handling path, but also covers the precise motion control of grabbing and releasing, ensuring the stability of the entire handling process. To further improve the accuracy of the operation, the pose estimation module analyzes the image data obtained from camera B and works in conjunction with the trajectory planning module to dynamically calibrate the perception accuracy of the visual neural network. The role of this module is to eliminate errors caused by external factors such as light and angle changes through multi-layer convolution processing of image data, ensuring high accuracy in identifying and positioning boxes. During the handling process, the adjustment module obtains the state data of the operation end through the sensor group in real time, and combines the visual feedback information provided by the trajectory planning module and the pose estimation module to dynamically adjust the operation path, the force applied, and the moving speed. This module ensures that the operation parameters are fine-tuned according to real-time feedback during the entire handling process, preventing the scattering of boxes due to sudden external disturbances or internal errors. Finally, the command execution module is responsible for integrating the data and instructions provided by the above modules to precisely control the drive components of the operation end to perform the handling task. This module will command the operation end to perform stable grabbing, handling, and releasing actions based on the calculated optimal trajectory and adjusted operation parameters, ensuring that the stacked boxes remain balanced throughout the operation process and do not scatter. Embodiment

[0054] Figure 1As shown, an auxiliary system adapted for teleoperation, further comprising a display end and an operation end, the auxiliary system is arranged on the display end or the operation end; the auxiliary system comprises an action input unit, a regular instruction unit, a self-learning instruction unit and a processor; the action input unit comprises an instruction input module, a storage module and an instruction calling module; the regular instruction unit comprises a real-time target positioning module, a trajectory planning module, a pose estimation module, an adjustment module, a command execution module, a data generation module and a model generation module; the self-learning instruction unit comprises a data acquisition module, a data processing module, a body intelligence model training module, a control strategy generation module, an operation execution module, a result feedback and analysis module; the processor is in bidirectional electrical connection with the regular instruction unit and the self-learning instruction unit; the display end comprises an input camera A and a display module; the operation end comprises an output camera B, a sensor group and a driving assembly. The instruction input module obtains the action information of the user through the input camera A, performs action recognition and processing relying on the visual nervous system, and converts visual information into action information code; the storage module stores the information code generated by the instruction input module, and classifies and names it; the instruction calling module retrieves and calls the action information code stored in the storage module according to the calling command input by the display end. The real-time target positioning module relies on the visual neural network, captures the position information of the target in real time through the input camera B, analyzes through the visual neural network, obtains the current position and state of the target object, and plans the appropriate grabbing position according to the corresponding instruction; the trajectory planning module locates the position information of the target in real time through the visual neural network, combines with the task requirements, continuously calculates and generates the best operation trajectory; the pose estimation module analyzes the image data obtained by the input camera B, cooperates with the trajectory planning module, achieves the perception accuracy of the visual neural network, and optimizes the accuracy of identifying and positioning the target; the adjustment module dynamically adjusts the operation path, force, speed and other parameters based on the data perceived by the sensor group, and the real-time detection of the visual neural network of the trajectory planning module and the pose estimation module, ensures that the operation strategy is adjusted in time according to the visual feedback in the task execution process; the command execution module controls the driving assembly in the operation end to complete the operation task according to the action information code provided by the instruction calling module, and the data obtained by the real-time target positioning module, the trajectory planning module, the pose estimation module and the adjustment module; the data generation module is used to generate and record related operation data after the task is completed, including target positioning information provided by the visual neural network, execution feedback data, etc.; the model training data generation module generates a data set for training the self-learning instruction unit by analyzing various real-time data generated by the operation end in the operation process.The data acquisition module enters real-time image, video and other data through the entry camera B, performs preliminary processing using the visual neural network, extracts key features including object position, shape, color and other information, and provides basic data for subsequent decision-making and operation; the data processing module is used for further processing and analysis of the visual data obtained by the data acquisition module, and converts it into useful information that can be used for decision-making and operation. The data processing module can identify suitable operation positions and paths to provide input for subsequent control strategy generation; the embodied intelligent model training module is used for continuous training and optimization of the large model. This module uses visual data generated by the data acquisition and data processing modules, as well as feedback data after execution of the control strategy generation module, to continuously adjust and optimize the parameters of the large model. The latest trained model will be passed to the control strategy generation module to ensure that the operation strategy is always based on the latest model inference result; the control strategy generation module determines the best operation path and method based on the information provided by the data acquisition module and the data processing module, judges the state of the environment and the target object through the visual neural network, and automatically generates specific operation strategies in combination with the embodied intelligent model training module, and passes the operation strategies to the operation execution module; the operation execution module is used to execute specific operation tasks according to the operation instructions provided by the control strategy generation module. During the execution process, the entry camera B will monitor the operation effect in real time, and generate new data through the data acquisition module; the result feedback module evaluates the success rate, accuracy and other information of the operation through the feedback information, and identifies the shortcomings in the operation; the analysis module is used in cooperation with the result feedback module. Through the analysis of the feedback data, optimization suggestions are put forward, and the optimization suggestions and result feedback are passed to the embodied intelligent model training module for further training and optimization. The real-time data generated by the model training data generation module is also synchronously transmitted to the embodied intelligent model training module. The visual neural network extracts multi-scale features from the image obtained by the entry camera B through a multi-layer convolutional neural network (CNN), and combines a target detection algorithm (such as Faster R-CNN or YOLO) to accurately identify and locate the position information of the target object. The multi-layer convolutional neural network is used for multi-layer convolution operation on the image data obtained by the entry camera B, generates feature maps through feature extraction, pooling and activation operations, can handle image displacement, scaling and rotation invariance, and improves the accuracy of target recognition and positioning.

[0055] In this embodiment, the conventional instruction unit generates information related to the operation each time it performs a task through the model training data generation module. This information is sent to the embodied intelligent model training module for continuous optimization and learning improvement of the system. At the same time, the self-learning instruction unit also generates operation data during its execution and sends it to the embodied intelligent model training module to improve the maturity speed of the large model. When the system faces tasks with high repetition and extremely high accuracy requirements, the staff can choose to activate the self-learning instruction unit. At this time, the staff only needs to retrieve the corresponding action command, and the self-learning instruction unit will automatically complete the corresponding operation according to the existing training data and the current task requirements. First, the staff selects and retrieves the specific action command of the self-learning instruction unit through the system interface or console according to the task requirements. Before the action is executed, the system will capture the image and video data of the current environment and target object in real time through the input camera B of the operation end. The visual neural network processes these data preliminarily and extracts key features such as the position, shape, and color of the object. These features provide basic data for subsequent operations. After data collection is completed, the data processing module analyzes the collected data in depth and converts these data into useful information that the system can understand and use. This process includes identifying suitable positions and paths for operation to provide necessary inputs for subsequent action execution. Based on the visual information provided by the data processing module, the control strategy generation module uses the training data and algorithms in the large model to automatically generate specific operation strategies, including judging the environmental state, evaluating the characteristics of the target object, and determining the best operation path and method. During the execution process, the system will monitor the operation effect in real time. If the visual nervous system detects changes in the position, state, or environmental conditions of the target object, the control strategy generation module will dynamically adjust the operation strategy according to the latest data to ensure the accuracy and efficiency of the action. When the control strategy generation is complete, the operation execution module transmits the specific operation command to the driving component of the operation end. At this time, the operation end completes the retrieved action command such as grabbing, moving, or other operations through the automated system. During the entire operation process, the input camera B continuously monitors the target object and the operating environment. If any changes or abnormalities occur, the control strategy generation module will immediately specify a new motion plan and execute it through the operation execution module until the target task is completed. After the operation is completed, the result feedback module and analysis module analyze the success rate and accuracy of the operation and generate data, which is transmitted to the embodied intelligent model training module as the basis for further training and optimization of the large model. Through continuous feedback and training, the parameters of the large model are continuously optimized to achieve better performance.

[0056] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto. The alternatives can be partial structure, device, method step alternatives, or complete technical solutions. According to the technical solutions of the present application and the inventive concept, equivalent replacement or change should be covered within the protection scope of the present application.

Claims

1. An assist system adapted for teleoperation, comprising a display side and an operation side, characterized in that, The display end or operation end is provided with an auxiliary system; The auxiliary system includes a motion input unit, a regular instruction unit, a self-learning instruction unit and a processor; The motion input unit includes an instruction input module, a storage module and an instruction calling module; The regular instruction unit includes a real-time target positioning module, a trajectory planning module, a pose estimation module, an adjustment module, a command execution module, a data generation module and a model training data generation module; The self-learning instruction unit includes a data acquisition module, a data processing module, a body intelligence model training module, a control strategy generation module, an operation execution module, a result feedback and analysis module; The processor is connected with the regular instruction unit and the self-learning instruction unit respectively; The display end includes an input camera A and a display module; The operation end includes an input camera B, a sensor group and a driving assembly; The data acquisition module inputs image and video data in real time through the input camera B, performs preliminary processing by using a visual neural network, extracts key features including object position, shape and color information, and provides basic data for subsequent decision and operation; The data processing module is used for further processing and analyzing the visual data acquired by the data acquisition module, and converting the visual data into useful information for decision and operation, and the data processing module can identify a suitable operation position and path to provide input for subsequent control strategy generation; The body intelligence model training module is used for continuously training and optimizing the model, and the module uses the visual data generated by the data acquisition and data processing modules and the feedback data after the execution of the control strategy generation module to continuously adjust and optimize the parameters of the large model, and the latest trained model is transmitted to the control strategy generation module to ensure that the operation strategy is always based on the latest body model inference result; The control strategy generation module determines the best operation path and method based on the information provided by the data acquisition module and the data processing module, judges the state of the environment and the target object through the visual neural network, automatically generates specific operation strategies in combination with the body intelligence model training module, and transmits the operation strategies to the operation execution module; The operation execution module is used for executing specific operation tasks according to the operation instructions provided by the control strategy generation module, and the input camera B monitors the operation effect in real time during the execution process and generates new data through the data acquisition module; The result feedback module evaluates the success rate and accuracy of the operation through feedback information, and identifies the shortcomings in the operation; The analysis module is used in cooperation with the result feedback module, analyzes the feedback data, proposes optimization suggestions, transmits the optimization suggestions and the result feedback to the body intelligence model training module, and makes the body intelligence model training module perform further training and optimization.

2. The teleoperation-adapted assistance system according to claim 1, characterized in that The instruction input module acquires motion information of a user through the input camera A, performs motion recognition and processing by relying on a visual neural network system, and converts visual information into human joint pose; The storage module stores the information generated by the instruction input module, and classifies and labels the information. The instruction calling module retrieves and calls the stored action information in the storage module according to the calling command and action input by the display end.

3. The teleoperation-adapted assistance system according to claim 1, characterized in that, The real-time target positioning module relies on the visual neural network to capture the target in real time through the input camera B, and to identify the target through the visual neural network, estimate the current position and state of the target object, and plan the position of the appropriate operation according to the corresponding instructions. The trajectory planning module calculates and generates the optimal operation trajectory by real-time positioning of the target position information through the visual neural network and combining the task requirements. The pose estimation module optimizes the accuracy of target identification and positioning and the accuracy of the end effector during operation by analyzing the image data obtained by the input camera B and cooperating with the trajectory planning module. The adjustment module dynamically adjusts the operation path, force and speed parameters based on the data perceived by the sensor group and the real-time detection of the visual neural network of the trajectory planning module and the pose estimation module, to ensure that the operation strategy is adjusted in a timely manner according to the visual feedback during the task execution process. The command execution module controls the driving components in the operation end to complete the operation task according to the action information provided by the instruction calling module and the data obtained by the real-time target positioning module, the trajectory planning module, the pose estimation module and the adjustment module. The data generation module is used to generate and record relevant operation data, including target positioning information and execution feedback data provided by the visual neural network, after the completion of the task execution and during the process. The model training data generation module generates a data set for training the self-learning instruction unit by analyzing various real-time data generated by the operation end during the operation process.

4. The teleoperation-adapted assist system according to claim 1, characterized by, The real-time data produced during the model runtime is synchronously transmitted to the embodied intelligent model for real-time decision-making, so that the model has anti-disturbance ability.

Images