Remote assistance system

Abstract

The invention provides a remote assistance system which comprises a cooperation device and a main control device, and the cooperation device is in communication connection with the main control device. The cooperation equipment is provided with a multi-degree-of-freedom camera, a multi-degree-of-freedom mechanical arm and a display The cooperative equipment receives a control signal of the main control equipment, controls the multi-degree-of-freedom camera to obtain workpiece images under different visual angles according to the control signal and controls the multi-degree-of-freedom mechanical arm to operate a workpiece according to the control signal, and a display screen of the cooperative equipment and a display screen of the main control equipment share a workpiece drawing based on the workpiece images; by means of the multi-degree-of-freedom camera and the multi-degree-of-freedom mechanical arm, remote personnel can conveniently operate the cooperation device through the main control device to conduct flexible operation, bidirectional communication is achieved through the shared screen, and remote cooperation convenience is improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of remote control, and in particular, to a remote assistance system. Background Art

[0002] Remote collaboration refers to remote collaboration through real-time annotation such as voice and video communication, AR technology, etc. At present, remote collaboration technology mainly uses a camera on the wearable device of on-site personnel to send the acquired images to remote technicians, and has a certain two-way interaction ability.

[0003] Existing camera gimbals usually have only 2 or 3 degrees of freedom, thus affecting the operator's ability to obtain a larger and better field of view. Although the device combining a robotic arm and a camera can obtain better field of view conditions, the robotic arm is generally controlled by buttons, joysticks, etc., and its control flexibility and intelligence are not high. The combination of a multi-rotor unmanned aerial vehicle and a gimbal camera has the ability of target tracking and flexible and smooth field of view conversion ability, but due to the influence of its own structure, the obtained field of view is also limited, and generally cannot work continuously for a long time. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail in this article. This overview is not intended to limit the scope of protection of the claims.

[0005] The purpose of the present application is to solve at least to some extent one of the technical problems existing in the related art. Embodiments of the present application provide a remote assistance system.

[0006] In an embodiment of the first aspect of the present application, a remote assistance system includes a collaboration device and a main control device, and the collaboration device is communicatively connected to the main control device; the collaboration device is provided with a multi-degree-of-freedom camera, a multi-degree-of-freedom robotic arm, and a display screen; the collaboration device receives a control signal from the main control device, controls the multi-degree-of-freedom camera to acquire workpiece images from different perspectives according to the control signal, controls the multi-degree-of-freedom robotic arm to operate the workpiece according to the control signal, and the display screen of the collaboration device shares a workpiece drawing based on the workpiece image with the display screen of the main control device.

[0007] According to certain embodiments of the first aspect of the present application, the collaboration device includes a first collaboration device, and the first collaboration device is provided with a first desktop and a bracket, the bracket supports the first desktop, and the multi-degree-of-freedom camera, the multi-degree-of-freedom robotic arm, and the display screen are arranged on the first desktop.

[0008] According to some embodiments of the first aspect of the present application, the collaborative device includes a second collaborative device, the second collaborative device is provided with a second desktop and a mobile module, the second desktop is arranged on the mobile module, the mobile module is provided with wheels, and the second desktop is provided with the multi-degree-of-freedom camera, the multi-degree-of-freedom robotic arm and the display screen.

[0009] According to some embodiments of the first aspect of the present application, the collaborative device includes a third collaborative device, the third collaborative device is provided with a backpack support structure, the backpack support structure is provided with a first strap, the multi-degree-of-freedom camera and the multi-degree-of-freedom robotic arm are arranged on the backpack support structure, and the display screen is provided with a second strap.

[0010] According to some embodiments of the first aspect of the present application, the multi-degree-of-freedom robotic arm is provided with a laser pointer and a robotic claw.

[0011] According to some embodiments of the first aspect of the present application, magnetic suction seats are arranged at the bottoms of the multi-degree-of-freedom camera and the multi-degree-of-freedom robotic arm.

[0012] According to some embodiments of the first aspect of the present application, the multi-degree-of-freedom camera includes a multi-degree-of-freedom arm rod and a camera, and the multi-degree-of-freedom arm rod drives the camera to perform the following motion modes: moving the camera closer to or farther away from the object to be photographed, rotating the camera around the center of the camera, orbiting the camera around the object to be photographed, and translating the camera.

[0013] According to some embodiments of the first aspect of the present application, audio information and video information are transmitted between the collaborative device and the main control device through an audio and video cloud server.

[0014] According to some embodiments of the first aspect of the present application, control signals are transmitted between the collaborative device and the main control device through the socket protocol and the TCP protocol.

[0015] According to some embodiments of the first aspect of the present application, shared drawings are transmitted between the collaborative device and the main control device through a front-end server, an Https server and a back-end server.

[0016] The above solution has at least the following beneficial effects: The remote assistance system includes a collaborative device and a master control device, and the collaborative device is communicatively connected to the master control device; the collaborative device is provided with a multi-degree-of-freedom camera, a multi-degree-of-freedom robotic arm, and a display screen; the collaborative device receives a control signal from the master control device, controls the multi-degree-of-freedom camera to acquire workpiece images from different perspectives according to the control signal, controls the multi-degree-of-freedom robotic arm to operate the workpiece according to the control signal, and the display screen of the collaborative device shares the workpiece drawing based on the workpiece image with the display screen of the master control device; by using the multi-degree-of-freedom camera and the multi-degree-of-freedom robotic arm, remote personnel can conveniently operate the collaborative device through the master control device for flexible operations, and realize two-way communication through screen sharing, improving the convenience of remote collaboration. Description of the Drawings

[0017] The drawings are used to provide a further understanding of the technical solutions of the present application, and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solutions of the present application, and do not constitute a limitation to the technical solutions of the present application.

[0018] Figure 1 is the architecture diagram of the remote assistance system;

[0019] Figure 2 is the structural diagram of the first collaborative device;

[0020] Figure 3 is the structural diagram of the second collaborative device;

[0021] Figure 4 is the structural diagram of the third collaborative device, and the display screen device of the third collaborative device has a second strap;

[0022] Figure 5 is the structural diagram of the third collaborative device, and the display screen of the third collaborative device is connected to the backpack support structure through a connecting arm rod;

[0023] Figure 6 is the schematic diagram of the audio and video call;

[0024] Figure 7 is the TCP protocol flow chart. Detailed Embodiments

[0025] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0026] It should be noted that although the functional modules are divided in the device schematic diagram and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be executed in a different module division in the device or a different order in the flowchart. Terms such as "first" and "second" in the specification, claims, or the above-mentioned drawings are used to distinguish similar objects and do not necessarily have to be used to describe a specific order or sequence.

[0027] The embodiments of the present application will be further described below with reference to the accompanying drawings.

[0028] An embodiment of the present application provides a remote assistance system.

[0029] The remote assistance system includes a collaborative device and a master device. The collaborative device is communicatively connected to the master device; the collaborative device is provided with a multi-degree-of-freedom camera 10, a multi-degree-of-freedom robotic arm 20, and a display screen 30; the collaborative device receives a control signal from the master device, controls the multi-degree-of-freedom camera 10 to acquire workpiece images from different perspectives according to the control signal, controls the multi-degree-of-freedom robotic arm 20 to operate the workpiece according to the control signal, and the display screen 30 of the collaborative device shares the workpiece drawing based on the workpiece image with the display screen 30 of the master device.

[0030] Refer to Figure 1 , the collaborative device and the master device communicate through a cloud server.

[0031] In this embodiment, the remote personnel operate the master device, and the on-site personnel at the workpiece site operate the collaborative device. The multi-degree-of-freedom camera 10 acquires the workpiece image and displays the workpiece image on the display screen 30 of the collaborative device and the display screen 30 of the master device for sharing. The remote personnel operate the master device to input a control signal. The collaborative device receives the control signal from the master device, controls the multi-degree-of-freedom camera 10 to acquire workpiece images from different perspectives according to the control signal, and controls the multi-degree-of-freedom robotic arm 20 to operate the workpiece according to the control signal; the remote personnel and the on-site personnel can draw a workpiece drawing based on the workpiece image through an input device and share the workpiece drawing on the display screen 30 of the collaborative device and the display screen 30 of the master device, which is convenient for communication.

[0032] By using the multi-degree-of-freedom camera 10 and the multi-degree-of-freedom robotic arm 20, the remote personnel can conveniently operate the collaborative device through the master device for flexible operation, and realize two-way communication through screen sharing, improving the convenience of remote collaboration.

[0033] Refer to Figure 2 , the collaborative device includes a first collaborative device 110. The first collaborative device 110 is provided with a first desktop 111 and a bracket 112. The bracket 112 supports the first desktop 111, and the multi-degree-of-freedom camera 10, the multi-degree-of-freedom robotic arm 20, and the display screen 30 are arranged on the first desktop 111.

[0034] The first collaborative device 110 is desktop-style, featuring light weight and compactness. The first collaborative device 110 is a foldable device, and it is convenient to be carried to the production site of workpieces after folding.

[0035] The first collaborative device 110 has a main control computer, and the main control computer communicates with the main control device through a cloud server. The first collaborative device 110 has one or two display screens 30, and one of the display screens 30 is touch-screen type. On-site personnel can conduct technical exchanges on workpiece drawing with technicians of the main control device through the first collaborative device 110. The multi-degree-of-freedom camera 10 is a six-degree-of-freedom camera, and the main control device can control the multi-degree-of-freedom camera 10 to obtain a better observation perspective. The multi-degree-of-freedom robotic arm 20 is equipped with a laser pointer or a roller, which can be used to indicate workpieces on-site. The bracket 112 is a telescopic bracket 112 that supports the first collaborative device 110. The multi-degree-of-freedom camera 10 and the multi-degree-of-freedom robotic arm 20 can be detached from the first collaborative device 110 and connected to the main control computer in a wired or wireless manner. The multi-degree-of-freedom camera 10 and the multi-degree-of-freedom robotic arm 20 are fixed on the magnetic surface of the first desktop 111 or on small-sized flat plates and tubes through magnetic suction seats.

[0036] Refer to Figure 3 , the collaborative device includes a second collaborative device 120. The second collaborative device 120 is provided with a second desktop 121 and a moving module 122. The second desktop 121 is arranged on the moving module 122. The moving module 122 is provided with wheels. The second desktop 121 is provided with a multi-degree-of-freedom camera 10, a multi-degree-of-freedom robotic arm 20, and a display screen 30.

[0037] The second collaborative device 120 is mobile, moving by crawlers or wheels, which expands the working range. The second collaborative device 120 has the ability of autonomous movement. The chassis can adopt a two-wheel differential drive mode, or methods such as Mecanum wheels, steering wheels, and crawlers. The second collaborative device 120 adopts a foldable design, and the multi-degree-of-freedom robotic arm 20 can be retracted. The moving mode of the second collaborative device 120 can adopt autonomous navigation technology or a mode controlled by the main control device. The second desktop 121 adopts a lifting design, enabling the multi-degree-of-freedom camera 10, the multi-degree-of-freedom robotic arm 20, and the display screen 30 on the second desktop 121 to move up and down. The multi-degree-of-freedom camera 10 and the multi-degree-of-freedom robotic arm 20 can be detached from the second collaborative device 120 and connected to the main control computer in a wired or wireless manner. The multi-degree-of-freedom camera 10 and the multi-degree-of-freedom robotic arm 20 are fixed on the magnetic surface of the second desktop 121 or on small-sized flat plates and tubes through magnetic suction seats.

[0038] Refer toFigure 4 , the third collaborative device 130 is provided with a backpack support structure 131. The backpack support structure 131 is provided with a first strap 132. The multi-degree-of-freedom camera 10 and the multi-degree-of-freedom robotic arm 20 are arranged on the backpack support structure 131, and the display screen 30 is provided with a second strap 133. The backpack support structure 131 is a box body, and electronic components such as a main control computer, a battery, and a communication module are arranged inside the box body. The backpack support structure 131 is separated from the display screen 30. The backpack support structure 131 has two first straps 132, and it can be carried on the shoulder through the two first straps 132. The display screen 30 is designed to be miniaturized, and the display screen 30 is tied to the arm through the second strap 133, which is convenient for carrying and viewing. The multi-degree-of-freedom camera 10 and the multi-degree-of-freedom robotic arm 20 can be detached from the backpack support structure 131 and connected to the main control computer in a wired or wireless manner. The multi-degree-of-freedom camera 10 and the multi-degree-of-freedom robotic arm 20 are fixed on the magnetic surface of the backpack support structure 131 or on a small-sized flat plate and tube through a magnetic suction seat.

[0039] Refer to Figure 5 , in addition, the display screen 30 of the third collaborative device 130 can also be connected to the backpack support structure 131 through a connecting arm rod, and the connecting arm rod extends the display screen 30 to a position in front of the user.

[0040] The remote assistance system can be used in cooperation with AR glasses and the like.

[0041] The multi-degree-of-freedom camera 10 includes a multi-degree-of-freedom arm rod and a camera. The multi-degree-of-freedom arm rod can adopt a structure combining a three-coordinate robot and three rotating axes, or a robotic arm with six rotating axis points for movement.

[0042] The multi-degree-of-freedom camera 10 is equipped with an IMU attitude sensor, a binocular camera, or a multi-line lidar, so that the system has the capabilities of autonomously keeping the camera level, anti-shake, obstacle avoidance during movement, target tracking, and path planning. In order to improve the flexibility of the robotic arm movement control, multiple methods such as a mouse, a touch-screen computer, AR glasses, and an eye tracker can be used for manipulation.

[0043] In addition, the multi-degree-of-freedom arm rod can also be replaced by devices such as a multi-degree-of-freedom serial robotic arm, a three-axis rectangular coordinate robot, and a three-axis pan-tilt head. The multi-degree-of-freedom camera 10 can also be composed of various unmanned aerial vehicles and the pan-tilt cameras carried by them, which is a shooting system with multiple degrees of freedom; it can also adopt a combination of a mobile platform and a pan-tilt camera, such as a combination of a wheeled mobile chassis and a pan-tilt camera.

[0044] The multi-degree-of-freedom arm rod drives the camera to perform the following movement modes: moving the camera closer to or farther away from the shooting object, rotating the camera around the center of the camera, orbiting the camera around the shooting object, and translating the camera.

[0045] For the zooming of the image, the position of the camera remains unchanged, so the movement of the arm is not involved. The image is enlarged and reduced simply by zooming the graph. In certain cases, the center point position of the zoomed image also needs to be selected, and then the image is zoomed with this point as the center.

[0046] For the camera moving closer to or farther away from the object being photographed, when the camera moves forward closer to the workpiece, the image becomes clearer and sharper. Especially when the camera is relatively close to the object being photographed, the effect of the image becoming larger is better. Therefore, moving the camera closer to the object being photographed can achieve the effect of enlarging the image. In fact, simply enlarging the image will reduce the clarity of the image. Additionally, when the workpiece is relatively far from the camera and the distance by which the camera moves closer to the workpiece is relatively small, the effect of the approaching action on making the image larger is limited, and mainly relies on the image magnification function to make the details of the workpiece clearer.

[0047] For the camera rotating around its own center (head movement mode), there are three rotational movements: pitch, yaw, and roll. The head movement is only related to the rotation of the rotating shaft. Additionally, when looking downwards, the camera will be blocked, so it is necessary to offset the center of the base rotating shaft from the camera by a certain distance.

[0048] For the camera moving in a circular motion around the object being photographed, it moves in a circular motion around the object being photographed with the object being photographed as the center of the sphere and a fixed radius length.

[0049] For the camera translation, a spatial coordinate axis is constructed, and the entire camera is translated along the spatial coordinate axis.

[0050] For the zooming process of special objects being photographed, for small objects, ordinary cameras cannot autofocus, and if the camera focuses on the background, the image quality of the small object will be blurred. Therefore, it is necessary for the camera to have a certain ability to autofocus according to the target object to eliminate the influence of the object size.

[0051] For the translational movement of the camera in a direction outside the field of view, the translation direction can be a straight line, a polyline within a plane, or a spatial line segment or curve.

[0052] For the edge distance shooting of special objects being photographed, the camera lens always points to the target object, but the distance between the camera and the target object can vary as needed.

[0053] For the circular motion based on special objects being photographed, when moving along the edge of the object being photographed and the size of the object being photographed is relatively large, the camera moves according to the surface shape of the object being photographed.

[0054] For the circular motion along the surface of the object being photographed, in order to carefully observe the surface condition of the object being photographed, the camera makes a circular motion according to the surface shape of the object being photographed.

[0055] For the motion mode of a multi-degree-of-freedom arm, the multi-degree-of-freedom arm has a total of 6 degrees of freedom, namely translational degrees of freedom X, Y, and Z, and rotational motions about each axis and

[0056] The correspondence between the degrees of freedom of the arm corresponding to the camera motion is shown in Table 1

[0057] Table 1 Correspondence Table of Degrees of Freedom of the Arm Corresponding to Camera Motion

[0058]

[0059] In addition, an attitude sensor IMU is added at the position of the camera for autonomous control of the camera's attitude. For example, the camera is kept horizontal, and the attitude of the robot is adjusted autonomously to play a role in anti-shake

[0060] A binocular camera and a multi-line lidar are added to the camera, so that it can recognize the three-dimensional shape of the target object and the spatial position relationship relative to the camera, thereby enabling the system to have the functions of autonomous obstacle avoidance and path planning

[0061] The camera has an automatic tracking function and can perform translational tracking

[0062] The camera can be a combination of multiple cameras, such as adding an infrared camera. Or a combination of multiple visible light cameras. If it is a combination of multiple visible light cameras, the angles can be outward-expanded, a spherical camera group, thus becoming a panoramic camera. If multiple cameras are inclined inward, a three-dimensional image can be obtained

[0063] The multi-degree-of-freedom camera 10 can be controlled by using the following devices as the input devices of the multi-degree-of-freedom camera 10. For example, a mouse, a touch-screen computer, a VR glasses, a wearable hand gesture sensor, a non-contact gesture sensor, a microphone, a binocular camera, an eye tracker, a tongue controller, a leg controller, etc

[0064] Exemplarily, the motion control mode of the multi-degree-of-freedom camera 10 based on a mouse is as follows

[0065] Zoom control mode: Place the mouse icon at a specific position on the screen, and this position is the center coordinate position of the image to be zoomed. Then, the zoom of the image can be controlled by using the scroll wheel. In addition, this control mode does not involve the actions of the six-axis robotic arm. Of course, a specific position can also be set as the center for magnification or reduction

[0066] Far - Near Mode: The manipulator is controlled by the mouse to move closer to or farther away from the observed object. The far - near mode is similar to the "zoom in - zoom out" control mode in terms of the intuitive feeling. Both are operations to magnify or reduce the observed object. However, the difference between them is that the "far - near" mode is achieved by the end of the manipulator equipped with a camera moving forward closer to or backward away from the object. When the camera moves forward closer to the object, the image becomes clearer. Especially when the distance between the camera and the object is relatively close, the magnification effect of the image is more obvious, thus achieving the effect of image magnification. In fact, although the simple "zoom in - zoom out" mode can also magnify the image, it may reduce the clarity of the image. In addition, when the object is far from the camera, even if the camera moves a short distance closer to the object, the image magnification effect brought by this approaching action is limited. In this case, mainly rely on the image scaling function in the "zoom in - zoom out" mode to enhance the clarity of the object details. In short, the approaching and retreating actions of the camera do not involve direct image scaling, but are related to the movement of the six - axis manipulator.

[0067] Translation Mode: Click the right - mouse button and drag the mouse to move, which controls the manipulator to perform translational motion along the plane of the camera lens. The "translation motion" mode means that the camera carried by the end of the manipulator moves translationally along a certain direction in the space plane, involving the displacement of the entire end - effector of the manipulator in the coordinate system. The "translation motion" mode can be divided into three forms: vertical translational motion, horizontal translational motion, and oblique translational motion (that is, the end of the manipulator moves simultaneously in the vertical and parallel directions in the plane) relative to the coordinate system of the camera at the end of the manipulator. These motions can be controlled by the mouse. By calculating the change amount of the coordinate system of the end of the manipulator relative to the world coordinate system (i.e., the position of the end of the six - axis manipulator), the numerical value of the translational motion of the manipulator can be accurately obtained.

[0068] Head Motion Mode: Drag the right - mouse button to control the manipulator to perform head - like motions. When the mouse moves left and right, the head of the manipulator turns left and right; when the mouse moves up and down, the head of the manipulator tilts up and down. The head motion of the manipulator simulates the head - observing action. According to the motion of the human head, three main motion modes of the manipulator head can be analogized: pitch motion (similar to the actions of raising and lowering the head), horizontal rotation motion (similar to the action of turning the head left and right), and roll motion (similar to the action of tilting the head, that is, the end - effector of the manipulator carrying the camera rotates around its Z - axis when facing the object). For a six - axis manipulator, the head action only controls the three rotational axes of the end of the manipulator. In addition, during the motion process, the interference problems that may occur to the manipulator need to be considered, such as the camera may be blocked when tilting down, or collisions may occur between parts of the manipulator.

[0069] Surrounding mode: Using a binocular camera or a multi-line lidar, determine the three-dimensional shape and spatial position of the target to be photographed. Hold down the left mouse button and drag the mouse to achieve the above three motions. At the start of the motion, the camera first scans the entire object and performs mathematical modeling based on the scan results. Subsequently, the robotic arm moves along the contour of the object surface according to this model.

[0070] Zoom mode: Control the focal length change of the camera through a pulley. In the control mode of "specific object zoom processing", this mode does not directly involve the motion control of the robotic arm. For small objects that require careful observation, ordinary imaging devices may not be able to achieve autofocus, resulting in a blurred image of the small object being photographed. This is because the imaging device fails to accurately identify the target object. Therefore, in order to eliminate the influence of object size on the imaging quality, the imaging device needs to have an autofocus function for the target object. This requires optimizing the vision algorithm for detecting tiny objects. When observing an object with a small size, the user can select the specific object and use the optimized vision algorithm to locate the object in the image captured by the imaging device. After the positioning is completed, the system will automatically focus on the selected object. When observing a position near the pen tip, but the size of the pen tip position is too small, the camera automatically focuses on the floor. Therefore, in specific situations, the focal length of the camera needs to be controlled instead of autofocus.

[0071] Translation mode in non-field-of-view direction: Achieved through three-dimensional reconstruction and then drawing a three-dimensional path in space. There is no direct correlation between the lens direction and the camera translation direction. The translation direction can be a straight line, a polyline in a plane, or a spatial line segment or curve.

[0072] Variable-distance shooting mode: Using a binocular camera or a multi-line lidar, determine the three-dimensional shape and spatial position of the target to be photographed. Click the left mouse button and the pulley to simultaneously control the direction change and distance control. During the process of the camera at the end of the robotic arm surrounding the object for shooting, if it is necessary to operate the observed object closer or farther away, the previous step is to first enable the motion mode and then switch to the "close / far" mode to carefully observe the area of the object. In this mode, during the motion of the robotic arm, the distance from the object can be adjusted in real time by rolling the mouse wheel (or other control means), so that while the lens continuously moves towards the observed object, it is also possible to approach or move away from the object in real time according to needs.

[0073] The motion control mode of the multi-degree-of-freedom camera 10 based on the touch-screen computer is as follows.

[0074] Zoom control mode: Touch the screen with two fingers at the same time. The area between the two fingers is the area to be zoomed. When the distance between the two fingers increases, the image is enlarged; when the distance decreases, the image is reduced.

[0075] Far - Near Mode: It can be selected through a button to enter this mode. After entering this mode, directly control the camera to move away from or towards the target object with two fingers. At this time, the camera moves forward or backward along its front - facing direction. When the distance between the two fingers increases, the camera approaches the object; conversely, it moves away from the target object.

[0076] Translation Mode: The camera moves in a plane perpendicular to the camera's line of sight. By swiping with one finger or two fingers, the camera can be controlled to translate in the corresponding direction in the plane.

[0077] It should be noted that the zoom control mode and the translation mode can be combined. When changing the distance between the two fingers and translating simultaneously, zooming and translation can be achieved; the far - near mode and the translation mode can be combined to achieve far - near movement and translation simultaneously.

[0078] Head Movement Mode: Control the head movement through the touch screen; the display can also show arrows in four directions, and the head movement can be achieved by clicking on the arrows.

[0079] Surrounding Mode: Use a binocular camera or a multi - line lidar to determine the three - dimensional shape and spatial position of the target to be photographed, and achieve movement by dragging the touch screen.

[0080] Zoom Mode: Control the focal length change of the camera by clicking a button.

[0081] Non - Field - of - View Direction Translation Mode: It can be achieved through three - dimensional reconstruction and then drawing a three - dimensional path in space.

[0082] Variable - Distance Shooting Mode: Use a binocular camera or a multi - line lidar to determine the three - dimensional shape and spatial position of the target to be photographed, click on the touch screen to simultaneously control the direction change and distance control.

[0083] Refer to Figure 6, this system uses cloud technology to achieve the data transmission of remote audio and video, signals, control instructions, and shared sketches. By invoking the cloud audio and video server (TRTC) and Websocket technology, an efficient and secure two-way data transmission system is built to ensure the stable transmission of data between the client and the server. Based on cloud technology, the two-way secure transmission of audio and video, control signals, acquisition information, and shared drawing board information is carried out. The cloud audio and video server is invoked for deployment on the client side, and the TRTC API library is called to develop the audio and video transmission function. A cloud server is set up as a control signal transfer station to receive control information from the client and forward it. The client remote end encapsulates the control signal and sends it to the cloud server, and the cloud server forwards it to the client on-site. The on-site end parses the encapsulated frame to complete the control. Three servers are deployed in the cloud server: the front-end server, the Https server, and the back-end server, which are used to display the content of the drawing board, establish a secure connection for secure transmission, and process and forward the drawing content of the drawing board respectively. Then the website content is published on the Internet, and a DNS domain name request is applied for so that both the PC side and the mobile side can access it.

[0084] The collaboration device creates an audio and video stream, obtains the media hardware and collects the audio and video stream data, renders the local video stream data, and sends the audio and video stream data to the TRTC cloud server. The TRTC cloud server pushes the stream data to the remote end of the master device; when the master device subscribes successfully, it forwards the audio and video stream data of the collaboration device to the master device. The master device listens to the remotely pushed stream data and renders the remote video stream data.

[0085] The audio and video information is transmitted between the collaboration device and the master device through the audio and video cloud server.

[0086] Exemplarily, the cloud audio and video server is selected as the core service for audio and video transmission. TRTC has powerful audio and video processing capabilities and stable network transmission performance, which can meet the requirements of the system for audio and video quality. The TRTC is deployed on the client side, and the TRTC API library is called to develop the audio and video transmission function. Through this API library, the client can easily implement operations such as audio and video acquisition, encoding, transmission, and decoding, providing users with a smooth audio and video interaction experience.

[0087] Refer to Figure 7 , the control signals are transmitted between the collaboration device and the master device through the socket protocol and the TCP protocol.

[0088] The collaborative device performs steps such as creating a socket, requesting a connection, receiving data from the server side, controlling the robotic arm, and closing the connection; the server side performs steps such as creating a socket, binding an IP and a communication port, listening for connection requests, accepting connection requests, establishing a connection, receiving data from the master device, forwarding data to the collaborative device, and closing the connection; the master device performs steps such as creating a socket, requesting a connection, sending control data, and closing the connection.

[0089] Exemplarily, the Tencent Cloud server is built using the Socket technology in combination with the TCP protocol as a transfer station for control signals. The reliability and stability of the TCP protocol can ensure the accurate transmission of control signals and ensure the normal operation of the remote control function of the system.

[0090] The master device encapsulates the control signal and then sends it to the cloud server. After receiving the control signal, the cloud server forwards it to the collaborative device. After receiving the encapsulated frame, the collaborative device parses it and then completes the corresponding control operations.

[0091] The shared drawing is transmitted between the collaborative device and the master device through the front-end server, the Https server, and the back-end server.

[0092] Exemplarily, three servers are deployed in the cloud server, namely the front-end server, the Https server, and the back-end processing server. The front-end server is used to display the content of the drawing board and provide an intuitive drawing board operation interface for users. The Https server is used to establish a secure connection to ensure the security of the drawing board data during transmission and prevent the data from being stolen or tampered with. The back-end server is used to process and forward the drawing content of the drawing board and effectively manage and distribute the drawing board data.

[0093] The website content is published to the Internet and a DNS domain name request is applied for. Through domain name resolution, convenient access between the PC side and the mobile side is achieved, and users can view and operate the shared sketch in real time on different devices.

[0094] The Canvas technology is used to develop various functions of the drawing board, such as drawing lines, graphics, text, etc. The Canvas technology can provide rich drawing interfaces to meet the diverse needs of users for the functions of the drawing board. The NodeJS combined with the Express technology is adopted. NodeJS has efficient asynchronous processing capabilities and good network performance and can quickly respond to front-end requests; the Express framework provides simple routing management and middleware support, facilitating the development and maintenance of back-end logic.

[0095] After the front-end server and the back-end server establish a secure connection using HTTPS technology, a long-term connection is established through the Websocket protocol. The Websocket protocol enables real-time two-way communication between the server and the client, ensuring real-time synchronization and interaction of the drawing board data, and providing a smooth collaboration experience for users.

[0096] During the data transmission process, an Https server is used to establish a secure connection and encrypt the transmitted data. Even if the encrypted data is intercepted during transmission, it cannot be easily interpreted, effectively protecting the privacy and security of user data.

[0097] The overall architecture of the remote assistance system is as follows.

[0098] Access layer: Mobile devices, PC devices, etc., which are terminal devices connected to the remote assistance system.

[0099] Monetization layer: The web side, which has functions such as a drawing canvas function, an audio and video call function, a file upload function, listening to and processing events sent by the back-end server, etc.; among them, workpiece images and workpiece drawings are displayed and drawn on the canvas.

[0100] Communication layer: It has the function of connecting the front-end and back-end servers and realizes information interaction between the front-end and back-end servers.

[0101] Service layer: Build the operating environments of the Https server and the back-end server, and has functions such as domain name management, security encryption, processing https requests and establishing https connections, cross-domain configuration, listening to and processing front-end server events, broadcasting events to the front-end server, and log detection.

[0102] Data layer: It has the function of saving the canvas to the client local.

[0103] The above is a specific description of the preferred embodiment of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of this application, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A remote assistance system, characterized in that: The invention comprises a collaborative device and a main control device, wherein the collaborative device is communicatively connected with the main control device; the collaborative device is provided with a multi-degree-of-freedom camera, a multi-degree-of-freedom robotic arm and a display screen; the collaborative device receives a control signal from the main control device, controls the multi-degree-of-freedom camera according to the control signal to obtain images of the workpiece under different viewing angles, controls the multi-degree-of-freedom robotic arm to operate the workpiece according to the control signal, and the display screen of the collaborative device shares a workpiece drawing based on the workpiece image with the display screen of the main control device.

2. The remote assistance system according to claim 1, characterized in that: The collaboration device includes a first collaboration device, which is provided with a first desktop and a bracket, the bracket supports the first desktop, and the first desktop is provided with the multi-degree-of-freedom camera, the multi-degree-of-freedom robotic arm and the display screen.

3. The remote assistance system according to claim 1, characterized in that: The collaboration device includes a second collaboration device, the second collaboration device is provided with a second desktop and a mobile module, the second desktop is arranged on the mobile module, the mobile module is provided with wheels, and the second desktop is provided with the multi-degree-of-freedom camera, the multi-degree-of-freedom robotic arm and the display screen.

4. The remote assistance system according to claim 1, characterized in that: The collaborative device includes a third collaborative device, the third collaborative device is provided with a backpack support structure, the backpack support structure is provided with a first strap, the multi-degree-of-freedom camera and the multi-degree-of-freedom robotic arm are provided on the backpack support structure, and the display screen is provided with a second strap.

5. The remote assistance system according to claim 1, characterized in that: The multi-degree-of-freedom mechanical arm is provided with a laser pointer and a mechanical claw.

6. The remote assistance system according to claim 1, characterized in that: The bottoms of the multi-degree-of-freedom camera and the multi-degree-of-freedom mechanical arm are both provided with magnetic seats.

7. The remote assistance system according to claim 1, characterized in that: The multi-degree-of-freedom camera includes a multi-degree-of-freedom arm and a camera, and the multi-degree-of-freedom arm drives the camera to perform the following movement modes: making the camera approach or move away from the object, rotating the camera around the center of the camera, moving the camera around the object, and moving the camera horizontally.

8. The remote assistance system according to claim 1, characterized in that: The audio information and video information are transmitted between the collaboration device and the master control device via the audio and video cloud service end.

9. The remote assistance system according to claim 1, characterized in that: The control signal is transmitted between the cooperation device and the main control device through the socket protocol and the TCP protocol.

10. The remote assistance system according to claim 1, characterized in that: The collaborative device and the main control device transmit the shared drawing via the front-end server, the Https server and the back-end server.

Images