Auxiliary system adaptive to teleoperation

Abstract

The invention discloses an auxiliary system adaptive to teleoperation, which comprises a display end and an operation end, and is characterized in that the display end or the operation end is provided with an auxiliary system; the auxiliary system comprises an action input unit, a conventional instruction unit, a self-learning instruction unit and a processor. The action input unit comprises an instruction input module, a storage module and an instruction calling module; the conventional instruction unit comprises a real-time target positioning module, a trajectory planning module, a pose estimation module, an adjustment module, a command execution module, a data generation module and a model training data generation module; the self-learning instruction unit comprises a data acquisition module, a data processing module, an intelligent model training module, a control strategy generation module, an operation execution module and a result feedback and analysis module. Compared with a traditional teleoperation system without an auxiliary system, the system has the advantages that the operation efficiency is greatly improved, the whole operation is more refined, and the uncertainty caused by manual operation can be effectively reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of remote operation auxiliary systems, and in particular to an auxiliary system adapted for remote operation. Background Art

[0002] With the continuous development of industrial automation and intelligence, the application of teleoperation systems in complex environments has become more and more extensive. These systems perform tasks by remotely controlling robotic arms or other actuators, and have been widely used in many fields such as manufacturing, medical treatment, and rescue.

[0003] At present, many innovations in auxiliary systems are mainly focused on achieving more refined control by adding various sensors. However, the core of the operation still relies on human operation. This means that the operator must pay close attention to the location of the grasping point and the placement point each time a task is performed, and once these points change, they need to be repositioned. This not only increases the complexity of the operation, but also reduces work efficiency.

[0004] To address the above issues, we have introduced an auxiliary system that is suitable for remote operation. Summary of the invention

[0005] The invention discloses an auxiliary system adapted for remote operation, aiming to solve the technical problems in the background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: An auxiliary system adapted for remote operation, comprising a display terminal and an operation terminal, wherein the display terminal or the operation terminal is provided with the auxiliary system; The auxiliary system includes an action entry unit, a conventional instruction unit, a self-learning instruction unit and a processor; The action entry unit includes an instruction entry module, a storage module and an instruction retrieval module; The conventional instruction unit includes a real-time target positioning module, a trajectory planning module, a posture estimation module, an adjustment module, a command execution module, a data generation module and a model training data generation module; The self-learning instruction unit includes a data acquisition module, a data processing module, an embodied intelligent model training module, a control strategy generation module, an operation execution module, and a result feedback and analysis module; The processor is electrically connected to the conventional instruction unit and the self-learning instruction unit in a bidirectional manner; The display end includes a recording camera A and a display module; The operating end includes a recording camera B, a sensor group and a driving component.

[0007] In a preferred solution, the instruction input module obtains the user's action information through the input camera A, relies on the visual nervous system to perform action recognition and processing, and converts the visual information into human joint posture; The storage module stores the information codes generated by the instruction input module, and classifies and labels them; The instruction calling module retrieves and calls the action information code stored in the storage module according to the calling command input by the display terminal.

[0008] In a preferred solution, the real-time target positioning module relies on a visual neural network to capture the position information of the target in real time by recording the camera B, and analyzes it through the visual neural network to obtain the current position and state of the target object, and plans a suitable grasping position according to the corresponding instructions; The trajectory planning module locates the target's position information in real time through a visual neural network, and continuously calculates and generates the best operation trajectory in combination with task requirements; The pose estimation module analyzes the image data obtained by the input camera B and cooperates with the trajectory planning module to optimize the accuracy of identifying and locating the target; The adjustment module dynamically adjusts the operation path, force, speed and other parameters based on the data sensed by the sensor group and the real-time detection of the visual nerves of the trajectory planning module and the posture estimation module, so as to ensure that the operation strategy is adjusted in time according to the visual feedback during the task execution; The command execution module controls the driving component in the operation terminal to complete the operation task according to the action information code provided by the instruction calling module and the data support obtained by the real-time target positioning module, the trajectory planning module, the posture estimation module and the adjustment module; The data generation module is used to generate and record relevant operation data after and during the task execution, including target positioning information provided by the visual neural network, execution feedback data, etc.; The model training data generation module generates a data set for training the self-learning instruction unit by analyzing various real-time data generated by the operating terminal during the operation process.

[0009] In a preferred solution, the data acquisition module records images, videos and other data in real time through the recording camera B, performs preliminary processing using a visual neural network, extracts key features, including the location, shape, color and other information of the object, and provides basic data for subsequent decision-making and operations; The data processing module is used to further process and analyze the visual data acquired by the data acquisition module, and convert it into useful information that can be used for decision-making and operation. The data processing module can identify appropriate operation positions and paths, and provide input for subsequent control strategy generation; The embodied intelligent model training module is used to continuously train and optimize the large model. The module uses the visual data generated by the data acquisition and data processing modules, as well as the feedback data after the control strategy generation module is executed, to continuously adjust and optimize the parameters of the large model. The latest trained model will be passed to the control strategy generation module to ensure that the operation strategy is always based on the latest model reasoning results; The control strategy generation module determines the state of the environment and the target object through the visual neural network based on the information provided by the data acquisition module and the data processing module, determines the best operation path and method, and automatically generates a specific operation strategy in combination with the embodied intelligent model training module, and passes the operation strategy to the operation execution module; The operation execution module is used to perform specific operation tasks according to the operation instructions provided by the control strategy generation module. During the execution process, the input camera B will monitor the operation effect in real time and generate new data through the data acquisition module; The result feedback module evaluates the success rate, accuracy, etc. of the operation through feedback information and identifies deficiencies in the operation; The analysis module is used in conjunction with the result feedback module to make optimization suggestions by analyzing the feedback data, and pass the optimization suggestions and result feedback to the embodied intelligent model training module for further training optimization.

[0010] In a preferred solution, the real-time data generated by the model training data generation module is also synchronously transmitted to the embodied intelligent model training module.

[0011] In a preferred solution, the visual neural network extracts multi-scale features from the image acquired by the recording camera B through a multi-layer convolutional neural network (CNN), and accurately identifies and locates the position information of the target object.

[0012] In a preferred embodiment, the multi-layer convolutional neural network is used to perform multi-layer convolution operations on the image data acquired by the input camera B, and generate feature maps through feature extraction, pooling and activation operations. It can handle the displacement, scaling and rotation invariance of the image and improve the accuracy of target recognition and positioning.

[0013] The auxiliary system for adaptive remote operation provided by the present invention has the following advantages: The present invention presets common actions and digitizes these actions, and then uses a visual neural network to detect the operation status in real time. It can automatically identify the current scene and generate a corresponding data chain to replace the preset actions for operation. The system only needs to call the corresponding instructions to quickly and efficiently complete the operation of the object. In addition, the data collection of preset actions provides continuous data support for the self-learning instruction unit, allowing large models to grow rapidly. This design does not require frequent manual intervention by staff when dealing with highly repeatable and high-precision operation tasks. Compared with traditional remote operation systems that are not equipped with auxiliary systems, this system not only greatly improves the operating efficiency and makes the overall operation more refined, but also effectively reduces the uncertainty caused by human operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 The present invention is a schematic diagram of the structural composition of an auxiliary system adapted for remote operation. DETAILED DESCRIPTION

[0015] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application usually described and marked in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0016] The invention discloses an auxiliary system adapted for remote operation. Example

[0017] Reference Figure 1As shown, an auxiliary system adapted for remote operation includes a display terminal and an operation terminal, and the display terminal or the operation terminal is provided with an auxiliary system; the auxiliary system includes an action entry unit, a conventional instruction unit, a self-learning instruction unit and a processor; the action entry unit includes an instruction entry module, a storage module and an instruction retrieval module; the conventional instruction unit includes a real-time target positioning module, a trajectory planning module, a posture estimation module, an adjustment module, a command execution module, a data generation module and a model training data generation module; the display terminal includes an entry camera A and a display module; the operation terminal includes a later entry camera B, a sensor group and a driving component. The instruction entry module obtains the user's action information through the entry camera A, relies on the visual nervous system to perform action recognition and processing, and converts the visual information into an action information code; the storage module stores the information code generated by the instruction entry module, and classifies and names it; the instruction retrieval module retrieves and retrieves the action information code stored in the storage module according to the retrieval command input by the display terminal. The real-time target positioning module relies on the visual neural network to capture the target's position information in real time by recording the camera B, and analyzes it through the visual neural network to obtain the current position and state of the target object, and plans a suitable grasping position according to the corresponding instructions; the trajectory planning module locates the target's position information in real time through the visual neural network, and continuously calculates and generates the best operation trajectory in combination with the task requirements; the posture estimation module analyzes the image data obtained by the recording camera B and cooperates with the trajectory planning module to calibrate the perception accuracy of the visual neural network and optimize the accuracy of identifying and locating the target; the adjustment module is based on the data perceived by the sensor group and the visual neural network of the trajectory planning module and the posture estimation module. Real-time detection, dynamically adjust the operation path, force, speed and other parameters to ensure that the operation strategy is adjusted in time according to the visual feedback during the task execution; the command execution module controls the driving components in the operation end to complete the operation task according to the action information code provided by the instruction retrieval module, and the data support obtained by the real-time target positioning module, trajectory planning module, posture estimation module and adjustment module; the data generation module is used to generate and record relevant operation data after and during the task execution, including the target positioning information and execution feedback data provided by the visual neural network; the model training data generation module generates a data set for training the self-learning instruction unit by analyzing the real-time data generated by the operation end during the operation. The visual neural network extracts multi-scale features from the image obtained by the input camera B through a multi-layer convolutional neural network (CNN), and combines the target detection algorithm (such as Faster R-CNN or YOLO) to accurately identify and locate the position information of the target object. The multi-layer convolutional neural network is used to perform multi-layer convolution operations on the image data obtained by the input camera B, and generates feature maps through feature extraction, pooling and activation operations. It can handle the displacement, scaling and rotation invariance of the image and improve the accuracy of target recognition and positioning; In this embodiment: the conventional instruction unit is responsible for converting the user instruction into a specific action, firstly recording the corresponding action through the recording camera 4, and digitizing the action, storing it in the storage module, and when applicable, calling out the corresponding instruction through the instruction calling module; Conventional instruction units can be set to press, lift, grab, twist, turn and other actions, specifically: according to: First, the user's "press" action is recorded and the corresponding data chain is generated. In the subsequent use process, when the user chooses to perform the "press" action, the system retrieves the stored "press" action code from the storage module. Subsequently, the system obtains the real-time image information of the operating end by recording camera B, and performs visual neural network analysis on the image through the real-time target positioning module to obtain the current position and state of the target object. Based on the real-time position information and state of the target object, the system calculates through the trajectory planning module to generate the best operation trajectory suitable for the current task requirements. At this time, the posture estimation module will analyze the image data obtained by the recording camera B, cooperate with the trajectory planning module, and calibrate the perception accuracy of the visual neural network to ensure accurate recognition and positioning in different coordinate systems. Then, the adjustment module will adjust the initial position of the operating end based on the environmental data perceived by the sensor group and the real-time detection results of the visual neural network. This module will dynamically optimize the operation path, pressure, speed and other parameters to ensure that the "press" action is accurately executed in the current scene. Finally, the command execution module controls the drive component in the operation end to execute the "press" action according to the "press" action code provided by the instruction call module, as well as the data support provided by the real-time target positioning module, trajectory planning module, posture estimation module and adjustment module, to ensure the smooth completion of the operation task. carry: First, the user's "lifting" action is recorded and the corresponding data chain is generated. In the subsequent use process, when the user chooses to perform the "lifting" action, the system retrieves the stored "lifting" action code from the storage module. Subsequently, the system obtains the real-time image information of the operating end by recording camera B, and performs visual neural network analysis on the image through the real-time target positioning module to obtain the current position, shape, weight and other state information of the target object. According to the real-time state and position of the target object, the system calculates the best extraction path suitable for the current task requirements through the trajectory planning module. At this time, the posture estimation module will analyze the image data obtained by the recording camera B, and cooperate with the trajectory planning module to calibrate the perception accuracy of the visual neural network to ensure accurate identification and positioning of the target object in different coordinate systems. Then, the adjustment module will adjust the initial position of the operating end based on the environmental data perceived by the sensor group and the real-time detection results of the visual neural network. This module will dynamically optimize the extraction path, extraction force, extraction speed and other parameters to ensure that the "lifting" action can be performed safely and effectively in the current scene. Finally, the command execution module calls the "lifting" action code provided by the instruction module, and combines the data support provided by the real-time target positioning module, trajectory planning module, posture estimation module and adjustment module to control the driving component in the operation end to perform the "lifting" action, ensuring the accurate completion of the operation task.

[0018] Grasp: First, the user's "grab" action is recorded and the corresponding data chain is generated. In the subsequent use process, when the user chooses to perform the "grab" action, the system retrieves the stored "grab" action code from the storage module. The system obtains the real-time image information of the operation end by recording camera B, and performs visual neural network analysis on the image through the real-time target positioning module to obtain the real-time position, shape, material and other state information of the target object. According to the real-time state and position of the target object, the system calculates the best grasping path suitable for the current task requirements through the trajectory planning module. At this time, the posture estimation module will analyze the image data obtained by the input camera B, and cooperate with the trajectory planning module to calibrate the perception accuracy of the visual neural network to ensure accurate identification, positioning and grasping of the target object in different coordinate systems. Then, the adjustment module will adjust the initial position and grasping parameters of the operation end based on the environmental data perceived by the sensor group and the real-time detection results of the visual neural network. This module will dynamically optimize parameters such as the grasping path, grasping force and grasping speed to ensure that the "grasping" action can be performed accurately and safely in the current scene, avoiding damage to the target object or grasping failure. Finally, the command execution module calls the "grasping" action code provided by the instruction module, and combines the data support provided by the real-time target positioning module, trajectory planning module, posture estimation module and adjustment module to control the driving component in the operation end to perform the "grasping" action, ensuring the accurate completion of the operation task.

[0019] change: First, the user's "turn" action is recorded and the corresponding data chain is generated. In the subsequent use process, when the user chooses to execute the "turn" action, the system retrieves the stored "turn" action code from the storage module. The system obtains the real-time image information of the operation end by recording camera B, and performs visual neural network analysis on the image through the real-time target positioning module to obtain the real-time position, direction and current state of the target object. According to the real-time state and position of the target object, the system calculates the optimal rotation path and angle suitable for the current task requirements through the trajectory planning module. At this time, the posture estimation module will analyze the image data obtained by the recording camera B, and work with the trajectory planning module to calibrate the perception accuracy of the visual neural network to ensure accurate control of the rotation direction and angle of the target object in different coordinate systems. Then, the adjustment module will adjust the initial position and rotation parameters of the operation end based on the environmental data perceived by the sensor group and the real-time detection results of the visual neural network. This module will dynamically optimize the rotation path, rotation force, speed and other parameters to ensure that the "rotation" action can be performed smoothly and accurately in the current scene, avoiding interference or damage to the target object during the rotation process. Finally, the command execution module calls the "rotation" action code provided by the module according to the instruction, and combines the data support provided by the real-time target positioning module, trajectory planning module, posture estimation module and adjustment module to control the drive component in the operation end to perform the "rotation" action, ensuring the accurate completion of the operation task.

[0020] Also, when moving items that may be at risk of tipping over, specifically: To move multiple stacked boxes: First, the real-time target positioning module obtains the position information and relative posture of the stacked boxes through the visual neural network and the real-time image capture of the input camera B. This module can analyze the precise position and posture of each box in the stack and determine the best grasping point, thus laying the foundation for the subsequent stable handling. Next, the trajectory planning module continuously tracks the dynamic position of each box through the visual neural network according to the position information provided by the real-time target positioning module, and calculates and generates the best operation trajectory in combination with the specific requirements of the operation task. This trajectory not only includes the handling path, but also covers the precise motion control of grasping and releasing to ensure the stability of the entire handling process. In order to further improve the accuracy of the operation, the posture estimation module will analyze the image data obtained from camera B, work with the trajectory planning module, and dynamically calibrate the perception accuracy of the visual neural network. The role of this module is to eliminate the errors caused by external factors such as light and perspective changes through multi-layer convolution processing of image data, so as to ensure the high accuracy of the system in identifying and locating boxes. During the handling process, the adjustment module obtains the status data of the operation end in real time through the sensor group, and dynamically adjusts the operation path, applied force and moving speed in combination with the visual feedback information provided by the trajectory planning module and the posture estimation module. This module ensures that during the entire handling process, the operating parameters are fine-tuned according to real-time feedback to prevent the boxes from scattering due to sudden external interference or internal errors. Finally, the command execution module is responsible for integrating the data and instructions provided by the above modules to accurately control the drive components of the operating end to perform the handling task. This module will command the operating end to perform stable grasping, handling and releasing actions based on the calculated optimal trajectory and adjusted operating parameters to ensure that the stacked boxes remain balanced throughout the operation and will not scatter. Example

[0021] Figure 1As shown, an auxiliary system adapted for remote operation also includes a display end and an operation end, and the display end or the operation end is provided with an auxiliary system; the auxiliary system includes an action entry unit, a conventional instruction unit, a self-learning instruction unit and a processor; the action entry unit includes an instruction entry module, a storage module and an instruction retrieval module; the conventional instruction unit includes a real-time target positioning module, a trajectory planning module, a posture estimation module, an adjustment module, a command execution module, a data generation module and a model generation module; the self-learning instruction unit includes a data acquisition module, a data processing module, an embodied intelligent model training module, a control strategy generation module, an operation execution module, a result feedback and analysis module; the processor is respectively electrically connected to the conventional instruction unit and the self-learning instruction unit in a bidirectional manner; the display end includes an input camera A and a display module; the operation end includes a later input camera B, a sensor group and a drive component. The instruction input module obtains the user's action information through the input camera A, relies on the visual nervous system to perform action recognition and processing, and converts the visual information into action information codes; the storage module stores the information codes generated by the instruction input module, and classifies and names them; the instruction retrieval module retrieves and retrieves the action information codes stored in the storage module according to the retrieval command input by the display terminal. The real-time target positioning module relies on the visual neural network to capture the target's position information in real time through the input camera B, and analyzes it through the visual neural network to obtain the current position and state of the target object, and plans a suitable grasping position according to the corresponding instructions; the trajectory planning module locates the target's position information in real time through the visual neural network, and combines the task requirements to continuously calculate and generate the best operation trajectory; the posture estimation module analyzes the image data obtained by the input camera B and cooperates with the trajectory planning module to calibrate the perception accuracy of the visual neural network and optimize the accuracy of identifying and locating the target; the adjustment module is based on the data perceived by the sensor group, as well as the visual neural real-time of the trajectory planning module and the posture estimation module. Detection, dynamically adjust the operation path, force, speed and other parameters to ensure that the operation strategy is adjusted in time according to the visual feedback during the task execution; the command execution module controls the driving components in the operation end to complete the operation task according to the action information code provided by the instruction retrieval module, and the data support obtained by the real-time target positioning module, trajectory planning module, posture estimation module and adjustment module; the data generation module is used to generate and record relevant operation data after and during the task execution, including the target positioning information and execution feedback data provided by the visual neural network; the model training data generation module generates a data set for training the self-learning instruction unit by analyzing the real-time data generated by the operation end during the operation.The data acquisition module records images, videos and other data in real time through the recording camera B, uses the visual neural network for preliminary processing, and extracts key features, including the location, shape, color and other information of the object, to provide basic data for subsequent decision-making and operations; the data processing module is used to further process and analyze the visual data obtained by the data acquisition module, and convert it into useful information that can be used for decision-making and operation. The data processing module can identify the appropriate operation position and path, and provide input for the subsequent control strategy generation; the embodied intelligent model training module is used to continuously train and optimize the large model. This module uses the visual data generated by the data acquisition and data processing modules, as well as the feedback data after the execution of the control strategy generation module, to continuously adjust and optimize the parameters of the large model. The latest trained model will be passed to the control strategy generation module to ensure that the operation strategy is always based on the latest model reasoning result. The control strategy generation module uses the visual neural network to judge the state of the environment and the target object based on the information provided by the data acquisition module and the data processing module, determines the best operation path and method, and automatically generates a specific operation strategy in combination with the embodied intelligent model training module, and passes the operation strategy to the operation execution module; the operation execution module is used to perform specific operation tasks according to the operation instructions provided by the control strategy generation module. During the execution process, the input camera B will monitor the operation effect in real time and generate new data through the data acquisition module; the result feedback module evaluates the success rate and accuracy of the operation through feedback information and identifies the shortcomings in the operation; the analysis module is used in conjunction with the result feedback module, and proposes optimization suggestions through the analysis of the feedback data, and passes the optimization suggestions and result feedback to the embodied intelligent model training module for further training and optimization. The real-time data generated by the model training data generation module will also be synchronously transmitted to the embodied intelligent model training module. The visual neural network extracts multi-scale features from the images obtained by the input camera B through a multi-layer convolutional neural network (CNN), and accurately identifies and locates the location information of the target object in combination with the target detection algorithm (such as Faster R-CNN or YOLO). The multi-layer convolutional neural network is used to perform multi-layer convolution operations on the image data obtained by the input camera B. It generates feature maps through feature extraction, pooling and activation operations. It can handle the displacement, scaling and rotation invariance of the image and improve the accuracy of target recognition and positioning.

[0022] This embodiment: Every time the conventional instruction unit performs a task, it will collect and generate information related to the operation through the model training data generation module. This information is sent to the embodied intelligent model training module for continuous optimization and learning improvement of the system. At the same time, the self-learning instruction unit will also generate operation data during its own execution and send it to the embodied intelligent model training module to improve the maturity speed of the large model. When the system faces a task with high repeatability and extremely high precision requirements, the staff can choose to activate the self-learning instruction unit. At this time, the staff only needs to call the corresponding action command, and the self-learning instruction unit will automatically complete the corresponding operation according to the existing training data and the current task requirements. First, the staff selects and calls the specific action command of the self-learning instruction unit through the system interface or console according to the task requirements. Before the action is executed, the system will capture the image and video data of the current environment and the target object in real time through the input camera B of the operation end. The visual neural network performs preliminary processing on these data and extracts key features, such as the position, shape, color, etc. of the object. These features provide basic data for subsequent operations. After the data collection is completed, the data processing module conducts in-depth analysis of the collected data and converts these data into useful information that the system can understand and use. This process includes identifying the position and path suitable for operation, providing the necessary input for the subsequent action execution. Based on the visual information provided by the data processing module, the control strategy generation module uses the training data and algorithms in the large model to automatically generate specific operation strategies, including judging the environmental state, evaluating the characteristics of the target object, and determining the best operation path and method. During the execution process, the system will monitor the operation effect in real time. If the visual nervous system detects that the position, state or environmental conditions of the target object have changed, the control strategy generation module will dynamically adjust the operation strategy according to the latest data to ensure the accuracy and efficiency of the action. When the control strategy is generated, the operation execution module will pass the specific operation command to the drive component of the operation end. At this time, the operating end completes the retrieved action commands through the automation system, such as grabbing, moving or other operations. During the entire operation process, the input camera B will continue to monitor the target object and the operating environment. If any changes or abnormalities occur, the control strategy generation module will immediately specify a new motion plan and execute it through the operation execution module until the target task is completed. After the operation is completed, the result feedback module and the analysis module will analyze the success rate and accuracy of the operation and generate data. The data will be passed to the embodied intelligent model training module as the basis for further training and optimization of the large model. Through continuous feedback and training, the parameters of the large model are continuously optimized to achieve better performance.

[0023] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. The replacement may be a replacement of a part of the structure, device, method step, or a complete technical solution. Any equivalent replacement or change according to the technical solution and the inventive concept of the present invention shall be covered within the protection scope of the present invention.

Claims

1. An auxiliary system adapted for remote operation, comprising a display terminal and an operation terminal, characterized in that: The display terminal or the operating terminal is provided with an auxiliary system; The auxiliary system includes an action entry unit, a conventional instruction unit, a self-learning instruction unit and a processor; The action entry unit includes an instruction entry module, a storage module and an instruction retrieval module; The conventional instruction unit includes a real-time target positioning module, a trajectory planning module, a posture estimation module, an adjustment module, a command execution module, a data generation module and a model training data generation module; The self-learning instruction unit includes a data acquisition module, a data processing module, an embodied intelligent model training module, a control strategy generation module, an operation execution module, and a result feedback and analysis module; The processor is respectively connected to the conventional instruction unit and the self-learning instruction unit; The display end includes a recording camera A and a display module; The operating end includes a recording camera B, a sensor group and a driving component.

2. The auxiliary system for adaptation remote operation according to claim 1, characterized in that: The command input module obtains the user's action information through the input camera A, performs action recognition and processing based on the visual neural network system, and converts the visual information into human joint posture; The storage module stores the information generated by the instruction input module and classifies and marks it; The instruction calling module retrieves and calls the action information stored in the storage module according to the calling command and action input by the display terminal.

3. The auxiliary system for adapting remote operation according to claim 1, characterized in that: The real-time target positioning module relies on a visual neural network to capture the target in real time by recording camera B, and recognizes the target through a visual neural network, estimates the current position and state of the target object, and plans a suitable operation position according to the corresponding instructions; The trajectory planning module locates the position information of the target in real time through the visual neural network, and calculates and generates the optimal operation trajectory in combination with the task requirements; The pose estimation module analyzes the image data obtained by the input camera B and cooperates with the trajectory planning module to optimize the accuracy of identifying and locating the target and the accuracy of the end effector during operation; The adjustment module dynamically adjusts the operation path, force, speed and other parameters based on the data sensed by the sensor group and the real-time detection of the visual nerves of the trajectory planning module and the posture estimation module, so as to ensure that the operation strategy is adjusted in time according to the visual feedback during the task execution; The command execution module controls the driving component in the operation terminal to complete the operation task according to the action information provided by the instruction calling module and the data support obtained by the real-time target positioning module, the trajectory planning module, the posture estimation module and the adjustment module; The data generation module is used to generate and record relevant operation data after and during the task execution, including target positioning information provided by the visual neural network, execution feedback data, etc.; The model training data generation module generates a data set for training the self-learning instruction unit by analyzing various real-time data generated by the operating terminal during the operation process.

4. The auxiliary system for adapting remote operation according to claim 1, characterized in that: The data acquisition module records images, videos and other data in real time through the recording camera B, performs preliminary processing using the visual neural network, extracts key features, including the location, shape, color and other information of the object, and provides basic data for subsequent decision-making and operations; The data processing module is used to further process and analyze the visual data acquired by the data acquisition module, and convert it into useful information that can be used for decision-making and operation. The data processing module can identify appropriate operation positions and paths, and provide input for subsequent control strategy generation; The embodied intelligent model training module is used to continuously train and optimize the model. The module uses the visual data generated by the data acquisition and data processing modules, as well as the feedback data after the control strategy generation module is executed, to continuously adjust and optimize the parameters of the large model. The latest trained model will be passed to the control strategy generation module to ensure that the operation strategy is always based on the latest embodied model reasoning results; The control strategy generation module determines the state of the environment and the target object through the visual neural network based on the information provided by the data acquisition module and the data processing module, determines the best operation path and method, and automatically generates a specific operation strategy in combination with the embodied intelligent model training module, and passes the operation strategy to the operation execution module; The operation execution module is used to perform specific operation tasks according to the operation instructions provided by the control strategy generation module. During the execution process, the input camera B will monitor the operation effect in real time and generate new data through the data acquisition module; The result feedback module evaluates the success rate, accuracy, etc. of the operation through feedback information and identifies deficiencies in the operation; The analysis module is used in conjunction with the result feedback module to make optimization suggestions by analyzing the feedback data, and pass the optimization suggestions and result feedback to the embodied intelligent model training module for further training optimization.

5. The auxiliary system for adapting remote operation according to claim 1, characterized in that: The real-time data generated when the model is running will be synchronously transmitted to the embodied intelligent model for real-time decision-making, making the model resistant to disturbances.

Images