Robot motion anti-dizziness and panorama trigger linkage control method

Abstract

The present disclosure provides a robot motion anti-dazzle and panorama trigger linkage control method, and relates to the technical field of artificial intelligence. In some embodiments of the present disclosure, joint posture data of a robot is acquired; according to the joint posture data, the motion state of the robot is determined; according to the motion state, the camera working mode is determined, the smoothness of mode switching is ensured, and the mode switching efficiency is improved; the camera working mode is sent to a panoramic camera, so that the panoramic camera collects monitoring video data according to the camera working mode; according to the received monitoring video data uploaded by the panoramic camera, anti-dazzle processing is performed on the monitoring video data to obtain target monitoring video data, and after the anti-dazzle processing, the anti-dazzle effect is improved and the degree of automation is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method for robot motion anti-dizziness and panoramic trigger linkage control. Background Technology

[0002] With the increasing demand for remote control and intelligent monitoring of industrial robots, the linkage control between vision systems and robots has become a core technological bottleneck in the industry. Current technologies for robot vision control mostly employ fixed-viewpoint output or simple follow-up modes, lacking the ability to dynamically adapt to the robot's motion state, resulting in several prominent pain points: when the robot moves at high speed, the panoramic view is prone to severe distortion, causing strong visual dizziness for the operator and seriously affecting the accuracy of remote control and the user experience; when the robot returns to its origin after completing its task, it needs to be manually switched to panoramic mode, failing to achieve automated full-area monitoring, resulting in low monitoring efficiency and difficulty in adapting to unattended production needs.

[0003] Currently, although some technologies attempt to optimize visual control logic, significant shortcomings remain: First, merely alleviating dizziness through image stabilization does not fundamentally combine the robot's motion state with the adjustment of the visual strategy, resulting in poor anti-dizziness effects and inability to adapt to high-speed motion scenarios; Second, the panoramic trigger condition is singular, relying solely on time threshold triggering, which cannot accurately synchronize with the robot's process state, thus lacking practicality; Third, the lack of a smooth switching mechanism leads to noticeable screen jumps during mode switching, further exacerbating visual discomfort; Fourth, the failure to achieve deep integration of motion parameters and visual processing means that the viewing angle cannot be dynamically adjusted based on real-time data such as joint speed and angle, making it difficult to meet the diverse needs of industrial production in terms of adaptability and intelligence.

[0004] In summary, robot monitoring methods suffer from technical problems such as poor anti-dizziness effects, low mode switching efficiency, and low degree of automation. Summary of the Invention

[0005] This disclosure provides a robot motion anti-dizziness and panoramic trigger linkage control method to at least solve the problems of poor anti-dizziness effect, low mode switching efficiency and low degree of automation in robot monitoring methods.

[0006] The technical solution disclosed herein is as follows: This disclosure provides a robot motion anti-dizziness and panoramic trigger linkage control method, including: Acquire the robot's joint posture data; the joint posture data includes: angle, angular velocity, and direction of motion; The motion state of the robot is determined based on the joint posture data; The camera operating mode is determined based on the described motion state; The camera's operating mode is sent to the panoramic camera so that the panoramic camera can collect monitoring video data according to the camera's operating mode; Based on the received monitoring video data uploaded by the panoramic camera, anti-dizziness processing is performed on the monitoring video data to obtain target monitoring video data.

[0007] Optionally, the joint posture data includes: joint angular velocity, and the motion state includes: high-speed motion, low-speed motion, and stationary state; determining the motion state of the robot based on the joint posture data includes: If the joint angular velocity is greater than the angular velocity threshold, then the motion state of the robot is determined to be the high-speed motion; If the joint angular velocity is less than or equal to the angular velocity threshold and greater than zero, then the motion state of the robot is determined to be the low-speed motion. If the joint angular velocity is zero and the duration of the zero joint angular velocity is greater than or equal to a set duration, then the motion state of the robot is determined to be the stationary state.

[0008] Optionally, the step of performing anti-dizziness processing on the surveillance video data to obtain the target surveillance video data includes: When the robot is moving at high speed, the joint posture data and kinematic model parameters of the robot are acquired, and the spatial position coordinates of the robot's end effector in the base coordinate system are calculated based on the kinematic model parameters. Differentiation and filtering are performed on the spatial position coordinates of multiple consecutive frames to obtain the spatial motion velocity vector of the end effector. Based on the velocity weights of each axis in the spatial motion velocity vector, the main motion plane is determined according to a preset main motion plane determination strategy; wherein, the main motion plane determination strategy includes: selecting two axes whose sum of velocity weights satisfies a preset threshold condition to form the main motion plane, or defaulting to designating a plane as the main motion plane when the preset threshold condition is not met; Based on the joint calibration relationship between the panoramic camera and the robot, the spatial range of the main motion plane in the base coordinate system is mapped to the cropping reference area in the pixel coordinate system of the panoramic camera, and the image corresponding to the cropping reference area is cropped from the monitoring video data to obtain the cropped image. The cropped image is subjected to distortion correction, projection mapping, and image stabilization to obtain the target surveillance video data.

[0009] Optionally, the step of performing anti-dizziness processing on the surveillance video data to obtain the target surveillance video data includes: When the robot is moving at low speed, the monitoring video data is used to generate a local wide-angle view. The local wide-angle view retains the scene within a preset safe distance around the operation, and the distortion rate of the target monitoring video data is less than a preset distortion threshold.

[0010] Optionally, the method further includes: Obtain the status signal of the robot; When the status signal indicates that the return to origin is complete or the operation is idle, the camera's working mode is determined to be panoramic mode; The surveillance video data is stitched together to form a panoramic view.

[0011] Optionally, the method further includes: Detect the joint switching angle of the robot; When the joint switching angle of the robot is greater than or equal to the set angle, initialize the current view parameters, target view parameters, and total number of transition steps; Based on the difference between the target viewpoint parameter and the current viewpoint parameter and the total number of transition steps, calculate the equal step adjustment amount of the viewpoint adjustment; The current viewpoint parameters are interpolated and updated frame by frame according to the video frame rate to generate a gradually changing viewpoint image. Determine the deviation between the updated viewpoint parameters and the target viewpoint parameters; The transition process is terminated when the deviation is less than or equal to the preset pixel threshold. If the angle change is detected again during the transition process and reaches the preset trigger threshold, the viewpoint smooth transition process is re-triggered and executed.

[0012] Optionally, the method further includes: Receive the emergency stop alarm signal uploaded by the robot; According to the emergency stop alarm signal, an abnormal scene lock command is sent to the panoramic camera; so that the panoramic camera locks the current viewpoint according to the abnormal scene lock command, captures the scene of the fault, and records the fault timestamp corresponding to the scene of the fault. Receive the fault moment image and fault timestamp returned by the panoramic camera.

[0013] This disclosure also provides a robot monitoring and control device, including: The acquisition module is used to acquire the robot's joint posture data; the joint posture data includes: angle, angular velocity, and direction of motion; The first determining module is used to determine the motion state of the robot based on the joint posture data; The second determining module is used to determine the camera working mode based on the motion state; The distribution module is used to distribute the camera working mode to the panoramic camera so that the panoramic camera can collect monitoring video data according to the camera working mode. The processing module is used to perform anti-dizziness processing on the monitoring video data uploaded by the received panoramic camera to obtain target monitoring video data.

[0014] This disclosure also provides an electronic device, including: processor; Memory used to store processor-executable instructions; The processor is configured to execute instructions to implement the steps in the above method.

[0015] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0016] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects: In some embodiments of this disclosure, the robot's joint posture data is acquired; the robot's motion state is determined based on the joint posture data; the camera's working mode is determined based on the motion state to ensure smooth mode switching and improve mode switching efficiency; the camera's working mode is sent to the panoramic camera so that the panoramic camera can collect monitoring video data according to the camera's working mode; based on the received monitoring video data uploaded by the panoramic camera, anti-dizziness processing is performed on the monitoring video data to obtain the target monitoring video data. After anti-dizziness processing, the anti-dizziness effect is improved and the degree of automation is increased.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0019] Figure 1 A system architecture diagram of a six-axis robot panoramic vision linkage monitoring system provided as an exemplary embodiment of this disclosure; Figure 2 A schematic diagram of the main interface of an interactive terminal provided for an exemplary embodiment of this disclosure; Figure 3 A schematic diagram of camera working mode switching provided for an exemplary embodiment of this disclosure; Figure 4A flowchart illustrating a robot motion anti-dizziness and panoramic trigger linkage control method provided for an exemplary embodiment of this disclosure; Figure 5 An overall flowchart of a robot motion anti-dizziness and panoramic trigger linkage control method provided for an exemplary embodiment of this disclosure; Figure 6 A flowchart for an anti-vertigo control system provided as an exemplary embodiment of this disclosure; Figure 7 This disclosure provides a schematic diagram of the structure of a robot monitoring and control device according to an exemplary embodiment. Figure 8 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0021] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0022] It should be noted that the user information involved in this disclosure includes, but is not limited to, user device information and user personal information; the collection, storage, use, processing, transmission, provision and disclosure of user information in this disclosure all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0023] To address the aforementioned technical issues, in some embodiments of this disclosure, the robot's joint posture data is acquired; the robot's motion state is determined based on the joint posture data; the camera's operating mode is determined based on the motion state, ensuring smooth mode switching and improving mode switching efficiency; the camera's operating mode is sent to the panoramic camera, allowing the panoramic camera to collect monitoring video data according to the camera's operating mode; based on the received monitoring video data uploaded by the panoramic camera, anti-dizziness processing is performed on the monitoring video data to obtain the target monitoring video data. The anti-dizziness processing improves the anti-dizziness effect and enhances the degree of automation.

[0024] The technical solutions provided by the embodiments of this disclosure are described in detail below with reference to the accompanying drawings.

[0025] Figure 1 This is a system architecture diagram of a six-axis robot panoramic vision linkage monitoring system 10 provided for an exemplary embodiment of this disclosure. Figure 1 As shown, the six-axis robot panoramic vision linkage monitoring system 10 includes: a robot 10a, a panoramic camera 10b, a video processing device 10c, and an interactive terminal 10d. The panoramic camera, mounted on the robot, is used to collect monitoring video data according to the camera working mode issued by the video processing device, and upload the monitoring video data to the video processing device. The robot is used to collect joint posture data and status signals of the robot's third and fourth axes, and upload the joint posture data and status signals to the video processing device. The joint posture data includes: angle, angular velocity, and direction of motion. The video processing device is used to receive the joint posture data and status signals uploaded by the robot, determine the robot's motion state based on the joint posture data, determine the camera working mode based on the motion state, send the camera working mode to the panoramic camera, receive the monitoring video data uploaded by the panoramic camera, process the monitoring video data to obtain processed monitoring video data, and send the processed monitoring video data to the interactive terminal. The interactive terminal is used to receive the processed monitoring video data sent by the video processing device and display the processed monitoring video data.

[0026] The robot described herein is a six-axis robot. This disclosure does not specify the exact type of robot and adjustments can be made based on actual circumstances.

[0027] The panoramic camera is an industrial-grade integrated panoramic camera with IP65 / 67 protection rating, adaptable to a wide temperature range of -10~60℃ in industrial environments, a frame rate of ≥30fps, and a resolution of 1080P / 4K. It supports global shutter and PoE power supply mode, and has a built-in attitude sensor and image stabilization module to effectively resist vibrations and shocks generated during robot movement, ensuring image stability. It can flexibly switch between 360-degree panoramic mode and local anti-dizziness mode, and can acquire high-definition video data of the entire field or specific angles in real time, perfectly adapting to different application scenarios such as robot motion monitoring and static state monitoring.

[0028] In some embodiments of this disclosure, the robot is equipped with a mounting adapter structure, through which a panoramic camera is mounted. The mounting adapter structure includes a lightweight shock-absorbing bracket and an anti-interference connector; wherein, the bracket is made of high-strength aluminum alloy, and its weight is strictly controlled to ≤300g. It is rigidly connected to the third or fourth axis joint shell of the six-axis robot through shock-absorbing pads. The height and angle of the bracket can be flexibly adjusted to ensure that the camera does not exceed the robot's motion envelope after installation, without any risk of motion interference; the connector adopts an anti-loosening design and has the characteristics of quick disassembly and replacement, which facilitates later maintenance and mass deployment.

[0029] The video processing equipment can be an industrial-grade edge computing gateway with a built-in dedicated motion vision linkage algorithm and image processing module. Its core functions include: (1) receiving robot joint posture data in real time and dynamically judging the robot's motion state (high speed, low speed, stationary); (2) automatically switching the camera working mode according to the different motion states of the robot, completing anti-dizziness perspective cropping and distortion correction, and ensuring clear images without dizziness; (3) receiving robot status signals in real time and accurately triggering the panoramic mode start-up or abnormal image lock function; among which, the abnormal image lock is only triggered by the emergency stop alarm signal, and there are no other triggering situations. After triggering, the current perspective is immediately locked, the fault moment image is frozen and marked with a precise timestamp, and synchronously transmitted to the interactive terminal for easy fault tracing; (4) compressing and optimizing the collected video data and storing it securely, supporting the association and binding of video data and process parameters for easy production tracing.

[0030] It should be noted that the camera's operating modes include, but are not limited to, the following: cockpit view mode, wide-angle working view mode, and panoramic mode.

[0031] In some embodiments of this disclosure, the robot monitors emergency stop alarm events; upon detecting an emergency stop alarm event, it generates an emergency stop alarm signal. It should be noted that emergency stop alarm events include any one of the following: robot malfunction, manual emergency stop, or emergency stop triggered by external equipment.

[0032] In some embodiments of this disclosure, the video processing device receives an emergency stop alarm signal uploaded by the robot; based on the emergency stop alarm signal, it performs an abnormal scene locking operation. One possible implementation is that the video processing device sends an abnormal scene locking command to the panoramic camera; the panoramic camera locks its current viewpoint according to the abnormal scene locking command, captures the moment of failure, records the corresponding fault timestamp, and returns the fault timestamp and fault images to the video processing device; the video processing device receives the fault timestamp and fault images uploaded by the panoramic camera; and sends the fault timestamp and fault images to the interactive terminal. The emergency stop alarm signal is the sole trigger condition for the video processing device to activate the abnormal scene locking function; there are no other trigger conditions. Specifically, it covers all scenarios that cause the robot to stop suddenly, such as robot malfunctions (stuck, collision), manual emergency stops, and emergency stops linked to external devices. All of these scenarios will output an "emergency stop alarm" signal to trigger the abnormal scene locking function. This function is not triggered in non-emergency stop scenarios.

[0033] In one embodiment, the robot, video processing device, and interactive terminal integrate a communication transmission module, adopting a gigabit Ethernet architecture and supporting mainstream industrial bus protocols such as Profinet and EtherNet / IP, as well as RTSP and ONVIF video transmission protocols. The robot controller accurately transmits the robot's real-time joint posture data (including angle, angular velocity, and direction of movement) and status signals (including in motion, return to origin completed, emergency stop alarm, and idle operation) to the vision processing device via the industrial bus. The vision processing device synchronously transmits the processed high-definition video signal to the interactive terminal. The core communication latency is ≤200ms, ensuring the real-time linkage between motion and vision.

[0034] The interactive terminal, including a local industrial control computer and a remote web terminal, supports multi-mode display and diverse operation control. It can realize core functions such as single-view switching, panoramic drag-and-drop zoom, picture-in-picture synchronous display, video playback, and parameter setting, perfectly adapting to various scenarios such as workshop on-site operation, remote operation and maintenance management, and visitor display.

[0035] Figure 2 This is a schematic diagram of the main interface of an interactive terminal provided for an exemplary embodiment of this disclosure. For example... Figure 2 As shown, the main interface of the interactive terminal includes: a status display area, a screen display area, and an operation control area. It should be noted that this disclosure does not limit the specific implementation of the main interface of the interactive terminal, and adjustments can be made according to the actual situation.

[0036] Figure 3 This is a schematic diagram illustrating a camera operating mode switching method, provided as an exemplary embodiment of this disclosure. Figure 3 As shown, when the system starts, the robot controller collects joint posture data and status signals to determine the robot's motion state. When the robot has completed its return to the origin or is idle for ≥5 seconds, the panoramic mode start signal is triggered, and the vision processing unit (i.e., the vision processing device) starts panoramic stitching and switches to 360-degree panoramic mode. When the robot is in motion, the vision processing unit determines the joint angular velocity. If the joint angular velocity is greater than the angular velocity threshold of 5° / s, it is considered high-speed motion, and the camera working mode is switched to cockpit view mode. If the joint angular velocity is less than or equal to the angular velocity threshold of 5° / s, it is considered low-speed motion, and the camera working mode is switched to wide-angle working view mode.

[0037] like Figure 3 As shown, the system receives an emergency stop alarm signal uploaded by the robot; based on the emergency stop alarm signal, it performs an abnormal scene lock operation. The video processing equipment returns the instantaneous image of the fault, the fault timestamp, and the processed monitoring video data to the interactive terminal for display.

[0038] The following provides specific embodiments of the robot monitoring system to illustrate this disclosure.

[0039] Example 1: Application of robotic sample preparation system in the coal industry.

[0040] (1) Component parameters: A 4K panoramic camera with IP67 protection level is selected, with a frame rate set to 30fps, which is suitable for the harsh environment of dust and vibration in the coal sample preparation workshop; the mounting bracket is made of 6061 aluminum alloy, weighs 280g, and the shock-absorbing pad is 5mm thick to enhance the shock absorption effect; the vision processing unit is an industrial-grade edge gateway that supports the Profinet protocol and is compatible with KUKA robot controllers; the interactive terminal includes a local industrial control computer in the workshop and a remote Web terminal to achieve dual control, and adds an emergency stop alarm pop-up and abnormal image traceability function.

[0041] (2) Assembly method: The panoramic camera is firmly installed on the shell of the third axis joint of the six-axis robot through anti-interference connectors. The bracket angle is adjusted to the horizontal orientation of the camera lens to ensure that there is no interference from any parts during the robot's movement. It is suitable for the entire process of coal sample preparation, such as transfer, reduction, and retention. The camera is powered by the robot body with 24V PoE. The wiring is arranged in accordance with the robot's cable chain to avoid wiring tangling and pulling, and to resist the corrosion of workshop dust. The vision processing unit is stably connected to the robot controller, camera and interactive terminal through Gigabit Ethernet. The core communication delay is 180ms (which meets the requirement of core communication delay ≤200ms) to ensure the real-time transmission of emergency stop alarm signals.

[0042] (3) Operational effect: When the robot moves at high speed (three-axis / four-axis speed > 5° / s) to transfer coal materials and switch between sample preparation stations (crushing → reduction), the vision processing unit receives joint posture data in real time and automatically crops the XY plane orthographic projection image. The operator does not experience any dizziness and can accurately monitor material transfer to avoid spillage. When the robot moves at low speed (three-axis / four-axis speed ≤ 5° / s) to perform precise sample preparation operations such as coal reduction, sample retention, and sample disposal, the system automatically outputs a local wide-angle view, focusing on core stations such as the reduction port, sample retention box, and crushing equipment feed port to ensure the accuracy monitoring requirements of sample preparation. When the robot completes batch sample preparation and returns to its origin (with the three or four axes raised to their highest positions), it receives a "return to origin complete" signal and quickly switches to 360° panoramic mode within one second, fully covering the entire sample preparation station. It supports AI-powered inspection of remaining material and equipment operating status. During sample preparation, only the "emergency stop alarm" signal (the sole trigger condition) can lock the abnormal view. If the robot triggers an emergency stop due to equipment jamming or material spillage, the system immediately locks the current viewpoint, freezes the moment of the fault, marks a precise timestamp, and simultaneously displays pop-up alarms on both the local and remote web interfaces. Staff can quickly locate the fault point through the frozen image. The system ran continuously for 72 hours without any malfunctions, perfectly meeting the unattended operation requirements of coal sample preparation workshops.

[0043] Example 2: Application of vehicle-mounted mobile intelligent robot sampling system in coal sampling scenarios.

[0044] (1) Component parameters: A 4K panoramic camera with IP67 protection rating is selected, with a frame rate set to 30fps, which is suitable for the dust, vibration, and harsh outdoor wide temperature environment of vehicle-mounted mobile operations such as coal collection and transportation stations and mining areas; the mounting bracket is made of 6061 aluminum alloy, weighs 280g, and has a shock-absorbing pad thickness of 5mm to enhance the shock absorption effect of dual vibration of vehicle-mounted mobile and robot operations; the vision processing unit is selected as an industrial-grade edge gateway, which supports the Profinet protocol and is compatible with the six-axis collaborative robotic arm controller of the vehicle-mounted sampling and preparation system, while also being compatible with the system's 4G / 5G wireless transmission module; the interactive terminal includes a vehicle-mounted local industrial control computer and a remote Web terminal, realizing dual control of on-site operation at the vehicle end and remote cloud control, and adding emergency stop alarm pop-up and abnormal image traceability functions, which can be linked to the coal quality analysis data and video images of the sampling and preparation system for binding and storage.

[0045] (2) Assembly method: The panoramic camera is firmly installed on the shell of the third axis joint of the six-axis robot of the vehicle-mounted sampling system through anti-interference connectors. The bracket angle is adjusted to the horizontal orientation of the camera lens to ensure that there is no interference from any parts during the operation of the robot driving the sampler and the movement of the vehicle platform. It is compatible with the entire process of drilling, sampling, material transfer, crushing and reduction, sealing and analysis during coal sampling. The camera is powered by the 24V PoE power supply of the vehicle-mounted sampling system. The wiring is arranged in a standard manner along the robot drag chain and the pipeline trough of the vehicle platform to avoid wiring entanglement and pulling caused by vehicle movement and robot movement, and to resist dust erosion in the mining area and collection station. The vision processing unit is stably connected to the robot controller, camera, interactive terminal and coal storage device and air compressor of the vehicle-mounted sampling system through gigabit Ethernet. The core communication delay is 180ms (which meets the requirement of core communication delay ≤200ms) to ensure the real-time transmission of emergency stop alarm signals and sampling equipment status signals.

[0046] (3) Operational effect: When the robot drives the sampler to move at high speed (three-axis / four-axis speed > 5° / s) to drill and sample coal cars and coal piles, and transfer materials across workstations, the vision processing unit receives joint posture data in real time and automatically crops the XY plane orthographic projection image with a distortion rate of <1%. The operator does not experience any dizziness. The system accurately monitors the working position of the sampling drill bit, the coal sample lifting and transfer process, and avoids coal sample spillage and drill bit collision with the car / coal pile. When the robot moves at low speed (three-axis / four-axis speed ≤ 5° / s) to unload coal samples to the crusher, perform precision operation of the divider, and grab sample buckets for sealing, the system automatically outputs a local wide-angle view, focusing on the core workstations such as the unloading port, crusher feed port, divider and sealing device, to ensure the accuracy monitoring requirements of sampling and preparation. When the robot returns to the origin (three-axis and four-axis rise to the highest position) after completing a single batch of sampling and preparation operations, the system receives the "return to origin completed" signal and quickly switches to 360° within 1 second. The panoramic mode fully covers the vehicle-mounted platform and surrounding sampling and preparation areas, supporting AI-powered inspection of the operating status of equipment such as samplers, crushers, and dividers, as well as coal sample storage. During sampling and preparation, only the "emergency stop alarm" signal (the sole trigger condition) can activate the abnormal screen lock. If the robot triggers an emergency stop due to a stuck sampling drill bit, coal sample blockage, air compressor failure in the vehicle-mounted sampling and preparation system, or abnormal coal storage device, the system immediately locks the current viewpoint, freezes the moment of the fault, marks a precise timestamp, and simultaneously displays an alarm pop-up on the vehicle's local industrial control computer and the remote web terminal. Staff can quickly locate the fault point through the frozen screen. The system has been running continuously for 72 hours without any faults while moving with the vehicle-mounted platform, perfectly adapting to the mobile unattended sampling and preparation needs of coal collection and transportation stations and mining areas.

[0047] Example 3: Application in automotive parts assembly workshops.

[0048] (1) Component parameters: A 4K panoramic camera with IP67 protection level is selected, and the frame rate is set to 30fps; the mounting bracket is made of 6061 aluminum alloy, weighs 280g, and the shock-absorbing pad is 5mm thick; the vision processing unit is an industrial-grade edge gateway that supports the Profinet protocol; the interactive terminal includes a local industrial control computer in the workshop and a remote Web terminal to realize dual control, and adds emergency stop alarm pop-up prompts and abnormal image retrieval functions.

[0049] (2) Assembly method: The panoramic camera is firmly installed on the shell of the third axis joint of the six-axis robot through the anti-interference connector. The bracket angle is adjusted to the horizontal orientation of the camera lens to ensure that there is no interference from any parts during the robot's movement. The camera is powered by the robot body with 24V PoE. The wiring is arranged in a standard manner along the robot's cable chain to avoid tangling and pulling of the wiring. The vision processing unit is stably connected to the robot controller, camera and interactive terminal through Gigabit Ethernet to ensure that the emergency stop alarm signal is transmitted without delay or packet loss.

[0050] (3) Operation effect: When the robot moves at high speed (three-axis / four-axis speed > 5° / s), the vision processing unit receives joint posture data in real time and automatically crops the XY plane orthographic projection image, so the operator does not feel dizzy. When the assembly process is completed and the robot returns to the origin (three-axis and four-axis rise to the highest position), the system receives the "return to origin completed" signal and quickly switches to 360° panoramic mode within 1 second, fully covering the entire assembly station and supporting AI inspection of the remaining material quantity and equipment operating status. Remote engineers can view the screen in real time through the Web terminal and manually switch the view. The equipment runs continuously for 72 hours without any faults. During the assembly process, if the robot collision, component jamming or other abnormalities trigger an emergency stop, the system immediately triggers the abnormal screen lock (only the emergency stop can trigger this), freezes the fault screen and marks the timestamp, which helps to quickly troubleshoot the fault.

[0051] Example 4: Application of precision machining of electronic components.

[0052] (1) Component parameters: A 1080P panoramic camera with IP65 protection level is selected, and the frame rate is set to 25fps; the mounting bracket weighs 250g; the vision processing unit integrates AI recognition algorithm, which can realize automatic recognition of equipment status and adds emergency stop alarm linkage processing function.

[0053] (2) Assembly method: The camera is installed 15cm away from the joint shell. The shock-absorbing bracket and the joint contact surface are designed with anti-slip texture to ensure that the installation is firm and without loosening. The communication adopts EtherNet / IP protocol to realize the real-time and packet-free transmission of robot joint data and visual signals, and to ensure the real-time linkage of emergency stop alarm signals.

[0054] (3) Operation effect: When the robot performs low-speed precision machining (three-axis / four-axis speed ≤ 5° / s), the system automatically outputs a local wide-angle view, accurately focuses on the machining area, and the image clarity is ≥ 1080P, ensuring the machining accuracy monitoring requirements; after returning to the origin, the panoramic mode can support leaders to visit and demonstrate, and the details of specific machining equipment can be magnified through the interactive terminal to intuitively display the production process; during the production process, video data is bound and stored with process parameters; during the machining process, when only tool breakage, workpiece displacement, etc. trigger an emergency stop, the abnormal image lock is activated (no other triggering situation), the module terminates the vision algorithm, locks the machining view, freezes the fault image and marks the timestamp, and alarms are simultaneously triggered on multiple terminals to quickly locate the fault point.

[0055] The embodiments disclosed herein have the following technical effects: Dual protection against dizziness and full-area monitoring: By linking the robot's motion state with the camera's perspective mode, it outputs a distortion-free local perspective when moving at high speeds and automatically triggers a 360° panoramic mode when returning to the origin. This not only completely solves the problem of motion-induced visual dizziness but also effectively eliminates blind spots in monitoring and improves product adaptability.

[0056] Highly adaptable to various applications: The lightweight shock-absorbing bracket and anti-interference design will not affect the robot's load and motion accuracy. It can be directly adapted to various mainstream six-axis robots without the need to modify the robot's structure, which greatly reduces the threshold for industrial application.

[0057] High level of intelligence: It can achieve fully automatic linkage of motion, perspective and status without human intervention, and supports value-added functions such as production traceability and automatic inspection. It effectively improves the continuous operation efficiency of equipment, and clearly defines the emergency stop alarm as the only trigger condition for locking abnormal screen. The linkage is precise and efficient, reducing production downtime losses.

[0058] Significant cost advantages: A single panoramic camera can replace 3-5 fixed cameras, greatly simplifying wiring and installation costs. The vision processing unit integrates multiple algorithm functions, eliminating the need for additional computing power equipment.

[0059] Multi-scenario adaptability: It can meet the diverse needs of industrial production such as safety monitoring, remote operation and maintenance, visit and display, and industrial training, effectively expanding the boundaries of the vision application of industrial robots and helping enterprises improve their digital image.

[0060] It should be noted that this disclosure also provides a robot monitoring and control method. The robot monitoring and control method provided by this disclosure can be applied to the robot monitoring system described above, and can also be applied to other types of robot monitoring systems. This disclosure does not limit it.

[0061] Figure 4 This is a flowchart illustrating a robot motion anti-dizziness and panoramic trigger linkage control method provided as an exemplary embodiment of this disclosure. Figure 4As shown, the robot's motion anti-vertigo and panoramic trigger linkage control method includes: S401: Acquire the robot's joint posture data; joint posture data includes: angle, angular velocity, and direction of motion; S402: Determine the robot's motion state based on joint posture data; S403: Determines the camera's operating mode based on the motion state; S404: Sends the camera's operating mode to the panoramic camera so that the panoramic camera can collect monitoring video data according to the camera's operating mode; S405: Based on the received panoramic camera's uploaded surveillance video data, perform anti-glare processing on the surveillance video data to obtain the target surveillance video data.

[0062] In this embodiment, the execution entity of the above method is a video processing device. The video processing device can be a server, and this disclosure does not limit the implementation form of the server. For example, the server can be a conventional server, a cloud server, a cloud host, a virtual center, or other server equipment. The server mainly consists of a processor, hard disk, memory, system bus, and other common computer architecture types.

[0063] In this embodiment, the robot's joint posture data is acquired; the robot's motion state is determined based on the joint posture data; the camera's working mode is determined based on the motion state; the camera's working mode is sent to the panoramic camera so that the panoramic camera can collect monitoring video data according to the camera's working mode; based on the received monitoring video data uploaded by the panoramic camera, anti-dizziness processing is performed on the monitoring video data to obtain the target monitoring video data.

[0064] In some embodiments of this disclosure, joint posture data and status signals of the robot are acquired. Specifically, joint posture data and status signals of the six-axis robot are acquired in real time via an industrial bus. The joint posture data focuses on the real-time angles, angular velocities, and directions of motion of the third and fourth axes, which are the core axes relating to the robot's motion posture and accurately reflect its motion state. The status signals cover four core states: "in motion," "return to origin completed," "emergency stop alarm," and "idle operation," comprehensively covering the entire robot's workflow. The acquired joint motion data and status signals undergo filtering preprocessing to effectively remove outliers and fluctuations caused by electromagnetic interference in the industrial environment, ensuring the accuracy and stability of the data acquisition. The preprocessing delay is strictly controlled to ≤50ms to ensure the real-time performance of the linkage control and avoid problems such as untimely viewpoint switching and poor dizziness relief due to data delays.

[0065] In some embodiments of this disclosure, the robot's motion state is determined based on joint posture data. One possible approach is to determine the robot's motion state as high-speed motion when the joint angular velocity is greater than a threshold value; to determine the robot's motion state as low-speed motion when the joint angular velocity is less than or equal to the threshold value but greater than zero; and to determine the robot's motion state as stationary when the joint angular velocity is zero and the duration of zero angular velocity is greater than or equal to a set duration. It should be noted that this disclosure does not limit the angular velocity threshold value, and the threshold value can be adjusted according to actual conditions. For example, the angular velocity threshold value can be 5° / s.

[0066] For example, the robot's motion state can be accurately determined based on joint angular velocities, where the joint angle can be the angular velocity of the third or fourth axis. When the joint angular velocity is greater than 5° / s, the robot's motion state is determined to be high-speed motion; when the joint angular velocity is less than or equal to 5° / s, the robot's motion state is determined to be low-speed motion; when the joint angular velocity is 0° / s and the duration is greater than or equal to 3s, the robot's motion state is determined to be stationary. This determination criterion can be flexibly adjusted according to the actual application scenario to adapt to different robot motion characteristics.

[0067] In some embodiments of this disclosure, when the robot is in high-speed motion, anti-dizziness processing is performed on the monitoring video data to obtain target monitoring video data. One possible approach is to acquire the robot's joint posture data and kinematic model parameters when the robot is in high-speed motion, and calculate the spatial position coordinates of the robot's end effector in the base coordinate system based on the kinematic model parameters; perform differentiation and filtering on the spatial position coordinates of multiple consecutive frames to obtain the spatial motion velocity vector of the end effector; determine the main motion plane based on the velocity weights of each axis in the spatial motion velocity vector according to a preset main motion plane determination strategy; wherein, the main motion plane determination strategy includes: selecting two axes whose sum of velocity weights meets a preset threshold condition to form the main motion plane, or defaulting to designating a plane as the main motion plane when the preset threshold condition is not met; based on the joint calibration relationship between the panoramic camera and the robot, mapping the spatial range of the main motion plane in the base coordinate system to a cropping reference area in the panoramic camera pixel coordinate system, and cropping the image corresponding to the cropping reference area from the monitoring video data to obtain the cropped image; performing distortion correction, projection mapping, and anti-shake processing on the cropped image to obtain the target monitoring video data.

[0068] For example, the vision processing device first receives pre-processed real-time joint posture data from the robot controller via an industrial bus, pre-loads the DH parameter table of the target six-axis robot, and calculates the real-time spatial position coordinates of the robot's end effector in the base coordinate system based on the DH homogeneous transformation matrix. The real-time spatial velocity vector of the end effector is obtained by taking the first-order difference derivative of the position coordinates of three consecutive frames and then filtering it through a sliding window. This process accurately determines the robot's real-time movement direction. Then, by calculating the weight of the absolute value of the velocity along each axis, the two axes with a sum of weights ≥ 80% are determined to constitute the main motion plane (XY / YZ / XZ plane). If the sum of weights for all two axes is < 80%, the XY plane is defaulted as the main motion plane. Subsequently, based on the joint calibration results of the panoramic camera and the robot, the spatial range of the main motion plane in the robot's base coordinate system is mapped to the cropping reference area in the panoramic camera's pixel coordinate system. This area is precisely cropped from the panoramic video stream. The cropped image undergoes inverse fisheye distortion correction and orthographic mapping, simultaneously completing electronic image stabilization to output a dizzying cockpit view that focuses on the core area of ​​the robot's movement. The frame rate is synchronized with the robot's movement speed (25-30fps), and the determination of the main motion plane and the cropping area are dynamically updated with the robot's movement direction, with an adjustment delay ≤ 10ms, ensuring smooth, stutter-free, and ghosting-free visuals, completely alleviating the operator's visual dizziness.

[0069] In some embodiments of this disclosure, when the robot is moving at low speed, anti-dizziness processing is applied to the monitoring video data to obtain target monitoring video data. One possible approach is to generate a local wide-angle view from the monitoring video data when the robot is moving at low speed. The local wide-angle view retains the scene within a preset safe distance around the work area, and the distortion rate of the target monitoring video data is less than a preset distortion threshold. For example, the local wide-angle view is automatically output, accurately focusing on the robot's work area while retaining the scene within a 1.5m radius around it, balancing operational accuracy with the safety monitoring requirements of the work area; the image distortion rate is strictly controlled to <1%, ensuring that work details are clearly visible and meeting the monitoring needs of scenarios such as precision assembly and precision machining.

[0070] In some embodiments of this disclosure, the robot's joint switching angle is detected; when the robot's joint switching angle is greater than or equal to a set angle, the current viewpoint parameters, target viewpoint parameters, and total transition steps are initialized; based on the difference between the target viewpoint parameters and the current viewpoint parameters and the total transition steps, the equal-step adjustment amount of the viewpoint adjustment is calculated; the current viewpoint parameters are interpolated and updated frame by frame according to the video frame rate to generate a gradual viewpoint image; the deviation between the updated viewpoint parameters and the target viewpoint parameters is determined; when the deviation is less than or equal to a preset pixel threshold, the transition process is terminated; if the angle change is detected again to reach a preset trigger threshold during the transition process, the viewpoint smooth transition processing step is re-triggered and executed. It should be noted that this disclosure does not limit the set angle, and the set angle can be adjusted according to the actual situation, such as a set angle of 30°. For example, when the angle of the third / fourth axis changes by ≥30°, the view transition algorithm is automatically activated. This algorithm, based on the real-time rate of robot joint angle switching, performs equal-step interpolation to gradually change the view of the image output by the vision processing unit. The transition time is controlled between 0.5-1s (adaptively adjusted according to the joint angle switching rate; the faster the angle switching, the closer the transition time is to 1s, and vice versa). The specific execution steps are as follows: trigger judgment and parameter initialization: when an angle switch of ≥30° is detected, the algorithm is triggered, and the current / target view parameters and the total number of transition steps are initialized; view transition step size calculation: the equal-step adjustment amount is calculated based on the difference between the target and current parameters and the total number of transition steps; frame-by-frame interpolation and gradual output: the view parameters are adjusted frame by frame according to the video frame rate, while simultaneously preserving distortion correction and anti-shake effects; transition termination judgment: the algorithm terminates when the view parameter deviation is ≤1 pixel. If the joint angle changes again, the gradual transition is re-triggered. The above algorithm effectively avoids image jumps caused by rapid joint rotation, further improving visual comfort and reducing operator visual fatigue.

[0071] In some embodiments of this disclosure, the robot's status signals are acquired; when the status signal indicates completion of homing or idle operation, the camera's working mode is determined to be panoramic mode; and the monitoring video data is stitched together to form a panoramic image. For example, the robot's status signals are monitored in real time. When a homing completion signal is received (three or four axes rise to the preset highest position, ensuring an unobstructed panoramic view) or an "idle operation" signal (duration ≥ 5s), the panoramic mode is automatically triggered without manual intervention, achieving automated full-area monitoring switching. The vision processing unit immediately starts the 360° panoramic stitching algorithm, quickly stitching the full-area video data collected by the camera into a complete panoramic image. The stitching delay is ≤ 1s, and the image resolution remains at 1080P / 4K level, ensuring a clear panoramic image without stitching gaps, fully covering the work area and surrounding environment, and eliminating monitoring blind spots.

[0072] In the above embodiments, after the panoramic mode is activated, if no alarm signal for movement or emergency stop is received, the panoramic image will continue to be output to adapt to scenarios such as unattended inspection and exhibition; if any of the above signals are received, the mode will immediately switch to the corresponding anti-dizziness mode to ensure the continuity of monitoring and prevent problems such as image interruption or monitoring gaps.

[0073] In some embodiments of this disclosure, the system receives an emergency stop alarm signal uploaded by the robot; based on the emergency stop alarm signal, it sends an abnormal scene lock command to the panoramic camera; the panoramic camera locks the current viewpoint according to the abnormal scene lock command, captures the moment of the fault, and records the fault timestamp corresponding to the moment of the fault; and receives the fault moment image and fault timestamp returned by the panoramic camera. For example, when an emergency stop alarm signal is received, the vision processing device immediately locks the current viewpoint (high speed / low speed / panoramic), freezes the image at the moment of the alarm and marks the precise timestamp, and simultaneously outputs an alarm prompt in a pop-up window, synchronizing it to the local industrial control computer and the remote web-based interactive terminal, facilitating staff to quickly trace the cause of the fault, locate the fault location, reduce downtime losses, and improve operation and maintenance efficiency.

[0074] In some embodiments of this disclosure, when switching between anti-dizziness mode and panoramic mode, a fade-in / fade-out transition algorithm commonly used in the field of industrial vision is adopted. This algorithm achieves mode switching by gradually changing the transparency of the anti-dizziness image and the panoramic image through step interpolation. The transition time is controlled within 0.3-0.5s (which can be customized according to the actual application scenario). The single-frame transparency adjustment step is evenly distributed according to the video output frame rate (25-30fps) to ensure that there are no image gaps or sudden jumps during the switching process, further improving visual comfort and adapting to the needs of long-term monitoring operations.

[0075] The embodiments disclosed herein have the following beneficial effects: Significant anti-dizziness effect: Through dynamic adaptation of joint motion data and visual processing strategies, the main motion plane is accurately determined before targeted cropping and orthographic projection conversion during high-speed motion, improving the distortion correction rate. Local viewpoints are accurately focused during low-speed motion. When switching postures, a customized viewpoint gradient transition algorithm is used to achieve smooth screen switching without jumps or stutters, reducing operator dizziness and greatly improving the remote control experience and operation accuracy.

[0076] Intelligent panoramic triggering: The panoramic mode is automatically triggered based on the robot's process status signal, without the need for manual intervention. The response time for panoramic start-up at the origin is ≤1 second, which perfectly adapts to the needs of unattended monitoring, automatic inspection, and visitor display, and greatly improves monitoring efficiency.

[0077] Smooth mode switching: The innovative combination of a self-developed perspective gradient transition algorithm and a general fade-in / fade-out transition algorithm solves the problems of local perspective jumps during joint rotation and sudden screen changes during global mode switching. The two types of algorithms perform their respective functions and work together to improve visual comfort, significantly reduce operator visual fatigue, and adapt to long-term industrial monitoring scenarios.

[0078] High adaptability: It supports custom settings of core parameters such as motion speed threshold, panoramic triggering conditions, transition time, and viewpoint gradient step size according to actual application scenarios, with an adaptability of over 90% and a low threshold for industrialization and promotion.

[0079] It has good functional scalability: it can be linked with AI recognition algorithms to realize value-added functions such as panoramic equipment inspection, personnel intrusion warning, and automatic fault identification, further improving the level of intelligence in industrial production and expanding the boundaries of application scenarios.

[0080] Based on the descriptions of the above embodiments, Figure 5 This is an overall flowchart of a robot motion anti-dizziness and panoramic trigger linkage control method provided as an exemplary embodiment of this disclosure. Figure 5 As shown in the diagram, the complete logical chain of data acquisition, status judgment, mode control, and anomaly handling is clearly marked, demonstrating the connection between each step and the data flow.

[0081] Figure 6 A flowchart illustrating an anti-vertigo control method is provided for an exemplary embodiment of this disclosure. (See attached flowchart.) Figure 6 As shown in the figure, the criteria for judging three motion states—high speed, low speed, and stationary—corresponding anti-vertigo processing strategies and posture compensation logic are clearly displayed, and the execution order of each sub-step is clearly defined.

[0082] The following are specific embodiments of the robot monitoring and control method to illustrate this disclosure.

[0083] Example 1: Application of robotic sample preparation system (coal industry).

[0084] Control parameter settings: Based on the motion characteristics of the coal industry robot sampling system (high load, fixed material transfer trajectory, large joint rotation range, suitable for all processes of crushing, reducing, retaining, and discarding samples), the high-speed motion threshold is set to 5° / s, the low-speed motion threshold is set to ≤5° / s, the attitude switching transition time is 0.8s, the panoramic trigger condition is set to the "return to origin completed" signal and the coal sampling-specific "sample preparation completed" signal, the fade-in / fade-out transition algorithm transition time is 0.4s, the transparency interpolation step size is automatically allocated according to the 30fps frame rate, the main motion plane judgment weight threshold is set to 80%, the main motion plane adjustment delay is ≤10ms, and the single-step deviation threshold of the viewpoint gradient transition algorithm is ≤1 pixel, adapting to the requirements of remote monitoring and unattended operation of coal sampling.

[0085] Operation process: Data Acquisition: The Profinet industrial bus is used to collect angle and angular velocity data of the third and fourth axes, as well as robot status signals (including dedicated signals for "sample preparation completed" and "material transfer in progress"). The preprocessing delay is controlled within 30ms to resist dust and vibration interference in the coal sample preparation workshop and ensure stable and accurate data transmission without packet loss or abnormalities.

[0086] High-speed motion control: When the robot performs high-speed coal material transfer and switches between sample preparation stations (crushing station → reduction station), the angular velocity of the third axis reaches 6° / s, which the system quickly determines as high-speed motion. The vision processing unit receives real-time joint motion data, preloads the robot's DH parameter table, calculates the end motion velocity vector, and determines the main motion plane as the XY plane. Then, the XY plane spatial range is mapped to the cropping reference area of ​​the panoramic camera pixel coordinate system (focusing on the material transfer path and the core area of ​​the sample preparation station). The corresponding area is cropped from the panoramic video stream and the orthographic projection conversion and electronic image stabilization are completed to output a dizzying cockpit view. The operator does not feel any dizziness and can accurately monitor the material transfer status to avoid material spillage.

[0087] Low-speed motion control: When the robot performs precision sample preparation operations such as coal reduction, retention, and disposal, the angular velocity of the fourth axis is 3° / s. The system automatically outputs a local wide-angle view, accurately focusing on core workstations such as the reduction port, the retention box, and the feed port of the crushing equipment. The picture is clear and distortion-free, and details such as material particle size and retention amount can be clearly observed, effectively avoiding sample preparation errors.

[0088] Posture switching compensation: During the sample preparation process, when the robot switches from the transfer station to the reduction station, the third axis angle switches by 35° in a single switch. The system immediately triggers the view transition algorithm, initializes the parameters of the current cutting area and the target cutting area, and calculates the total number of transition steps based on a transition time of 0.8s and a frame rate of 30fps. After calculating the offset of the single-step cutting area, the system outputs the transition frame by frame, with no jumps in the image throughout the process. This adapts to the frequent posture switching requirements of the sample preparation process and improves the operator's visual experience.

[0089] Panoramic Trigger and Mode Switching: After completing a single batch of coal sample preparation and returning to the origin, the system promptly receives two signals: "Return to origin completed" and "Sample preparation completed." Panoramic stitching is initiated within 0.8 seconds, and a fade-in / fade-out transition algorithm is triggered simultaneously. The transparency of the anti-dizziness image decreases linearly frame by frame at a frame rate of 30fps, while the transparency of the panoramic image increases linearly frame by frame. The smooth switching is completed in 0.4 seconds, quickly outputting a 360° panoramic image. This allows remote engineers to inspect the operating status of the sample preparation equipment, the remaining material quantity, and the cleanliness of the workstation without the need for on-site inspection, making it suitable for unattended coal sample preparation scenarios.

[0090] Operational performance: Mode and posture switching are smooth and seamless. The main motion plane is accurately determined under high-speed motion, and the orthographic projection image is output in real time. The anti-dizziness effect fully meets the standards. The panoramic trigger and mode switching response are timely and accurate, greatly improving the sample preparation accuracy and production efficiency, and perfectly adapting to the needs of the robot sample preparation system in the coal industry.

[0091] Example 2: Application of a six-axis collaborative robotic arm in a vehicle-mounted mobile intelligent robot sampling system (coal collection and transportation station / mining area scenario).

[0092] (1) Control parameter settings: Based on the motion characteristics of the six-axis collaborative robotic arm of the vehicle-mounted mobile intelligent robot sampling system (vehicle-mounted mobile operation, high load adaptability requirements, operation trajectory covering the entire process of coal pile / carriage drilling and sampling, material transfer, crushing and reduction, and sealing analysis, and joint rotation is affected by the vehicle-mounted bumpy environment), the high-speed motion threshold is set to 5° / s, the low-speed motion threshold is set to ≤5° / s, the attitude switching transition time is 0.9s (to adapt to the small and frequent joint rotation caused by vehicle-mounted bumps and improve the smoothness of the transition), and the panoramic trigger condition is set to the "return to origin completed" signal and the coal sampling exclusive signal. The "Sampling and sample preparation complete" and "Vehicle-mounted work position switching" signals have a fade-in / fade-out transition algorithm with a transition time of 0.5s. The transparency interpolation step size is automatically allocated according to a 30fps frame rate. The main motion plane judgment weight threshold is set to 80%, the main motion plane adjustment delay is ≤10ms, and the single-step deviation threshold of the viewpoint gradient transition algorithm is ≤1 pixel. At the same time, a new vehicle-mounted equipment linkage signal compatibility logic has been added, which can receive status signals from the coal storage device, air compressor, and coal quality analysis unit, adapting to the vehicle-mounted mobile unattended sampling and preparation needs of coal collection and transportation stations / mine areas.

[0093] Operation process: a) Data Acquisition: Real-time acquisition of angle and angular velocity data of the third and fourth axes, as well as robot status signals (including dedicated signals for "sampling completed", "material transfer in progress", and "vehicle work position switching") via Profinet industrial bus (adapted to the six-axis collaborative robotic arm controller protocol of the vehicle-mounted sampling and preparation system). Simultaneously, 3D modeling data of the coal storage device and air compressor operating status signals of the vehicle-mounted sampling and preparation system are acquired. The preprocessing delay is controlled within 25ms (optimized for electromagnetic interference and vibration data filtering in the vehicle environment). 25ms meets the requirement of preprocessing delay ≤50ms, resisting multiple interferences from dust, vibration, and vehicle bumps in coal collection and transportation stations / mine areas, ensuring stable and accurate data transmission without packet loss or anomalies, and synchronous linkage with data from various devices in the vehicle system.

[0094] b) High-speed motion control: When the six-axis collaborative robotic arm of the vehicle-mounted sampling system drives the sampler to drill and sample coal in coal cars / coal piles, and to transfer materials at high speed across vehicle workstations (sampling position → crushing position → shrinking position), the angular velocity of the third axis reaches 6.5° / s, which the system quickly determines as high-speed motion. The vision processing unit receives real-time joint motion data, preloads the DH parameter table of the six-axis collaborative robotic arm of the vehicle-mounted sampling system, calculates the real-time spatial motion velocity vector of the end sampler, and determines the main motion plane as the XY plane. Then, the XY plane spatial range is mapped to the cropping reference area of ​​the panoramic camera pixel coordinate system (focusing on the sampling drill bit operation position, coal sample lifting path, and core areas of each workstation of the vehicle-mounted sampling system). The corresponding area is cropped from the panoramic video stream and completed with orthographic projection conversion + electronic image stabilization dual processing to output a dizzying cockpit view. The operator does not experience any dizziness and can accurately monitor the drilling depth of the sampling drill bit, the coal sample lifting status, and the material transfer process, avoiding coal sample spillage and drill bit collision with the car / coal pile. Coal piles or vehicle-mounted equipment.

[0095] c) Low-speed motion control: When the six-axis collaborative robotic arm of the vehicle-mounted sampling system performs low-speed operations such as unloading coal samples to the crusher and reducing device, and grabbing sample buckets to the sealing device / coal quality analysis unit, the angular velocity of the fourth axis is 2.8° / s. The system automatically outputs a local wide-angle view to accurately focus on core work positions such as the unloading port, crusher feed port, reducing device, sealing device, and coal quality analysis unit detection port. It can clearly observe details such as material particle size, sample retention amount, and coal sample detection status, effectively avoiding sample preparation errors and detection deviations.

[0096] d) Attitude switching compensation: During the sample preparation process, when the six-axis collaborative robotic arm of the vehicle-mounted sampling and preparation system switches from the sampling position to the crushing position, or from the reduction position to the sealing position, the third axis angle switches by 40° at a time. At the same time, it is affected by the slight bumps of the vehicle platform and rotates slightly within 5°. The system immediately triggers the view transition algorithm, initializes the parameters of the current clipping area and the target clipping area, and calculates the total number of transition steps of 27 based on a transition time of 0.9s and a frame rate of 30fps. After calculating the offset of the clipping area in a single step, the system outputs the transition frame by frame. The entire process is smooth and lag-free, adapting to the dual requirements of frequent attitude switching and slight bumps in vehicle-mounted mobile operations, and greatly improving the operator's visual experience.

[0097] e) Panoramic Trigger and Mode Switching: After completing a single batch of coal sampling operations, the six-axis collaborative robotic arm of the vehicle-mounted sampling system returns to its origin (the third and fourth axes rise to the highest preset position on the vehicle platform to ensure that the panoramic view is unobstructed by vehicle-mounted equipment). The system promptly receives the dual signals "Return to Origin Completed" and "Sampling Completed," and initiates panoramic stitching within 0.7 seconds. Simultaneously, it triggers a fade-in / fade-out transition algorithm, with the anti-dizziness image transparency decreasing linearly frame by frame at 30fps, and the panoramic image transparency increasing linearly frame by frame. The smooth switch is completed in 0.5 seconds, quickly outputting a 360° panoramic view that covers the entire area of ​​the vehicle platform and the surrounding sampling environment. This allows remote engineers to inspect the operating status of equipment such as the vehicle-mounted sampler, crusher, and divider, as well as the coal sample storage status and the safety of the surrounding operating environment of the vehicle platform. No on-site inspection is required, making it suitable for unattended mobile sampling scenarios in coal collection and transportation stations / mine areas. If a "Vehicle Operation Position Switching" signal is received, the system also triggers panoramic mode, facilitating remote monitoring of the equipment and environmental status during the movement of the vehicle platform.

[0098] f) Onboard equipment abnormal linkage control: During the sampling process, if the air compressor of the onboard sampling system fails, the coal storage device malfunctions, or the sampler drill bit gets stuck / the coal sample gets blocked, causing the robotic arm to trigger an emergency stop alarm, the system will immediately lock the current view, accurately mark the timestamp and output an alarm prompt in a pop-up window. At the same time, it will simultaneously display the status data of the onboard faulty equipment and push it to the onboard local industrial control computer and the remote Web-based maintenance terminal. Staff can quickly locate the fault point and trace the cause of the fault through the freeze-frame image and equipment status data, which will greatly shorten the troubleshooting time.

[0099] Operational Results: Mode and attitude switching remained smooth and seamless even under bumpy vehicle conditions. The main motion plane was accurately determined at high speeds, and the orthographic projection output was real-time. The anti-dizziness effect fully met the standards. Panoramic triggering and mode switching responses were timely and accurate, and data linkage was achieved with various devices in the vehicle-mounted sampling system. The system operated continuously for 72 hours without any malfunctions while moving with the vehicle platform, perfectly adapting to the six-axis collaborative robotic arm monitoring requirements of the vehicle-mounted mobile intelligent robot sampling system in coal collection and transportation stations / mine areas.

[0100] Example 3: Application of FANUC 0i-F robot in machining monitoring (precision machining of electronic components).

[0101] (1) Control parameter settings: Based on the characteristics of robot processing, the high-speed motion threshold is set to 6° / s, the posture switching transition time is 0.5s, the panoramic trigger condition is set to the "idle" signal (lasting 5s), the fade-in and fade-out transition algorithm transition time is 0.3s, the transparency interpolation step size is automatically allocated according to the 30fps frame rate, the main motion plane judgment weight threshold is set to 80%, the main motion plane adjustment delay is ≤10ms, and the single-step deviation threshold of the viewpoint gradient transition algorithm is ≤1 pixel, which is suitable for precision processing monitoring scenarios.

[0102] (2) Operation process: 1. Data Acquisition: Joint motion data and status signals are acquired in real time via EtherNet / IP industrial bus, with a preprocessing delay of 40ms to ensure stable data transmission without packet loss and accurately reflect the robot's processing status.

[0103] 2. Abnormal linkage: When an emergency stop alarm occurs during processing, the system immediately locks the current local view, accurately marks the timestamp, and outputs an alarm prompt in a pop-up window. It is synchronized to the local industrial control computer and the remote operation and maintenance terminal, so that the staff can quickly locate the fault location and trace the cause of the fault.

[0104] 3. High-speed motion control: When the robot switches between processing stations, the angular velocity of the fourth axis reaches 7° / s. The system judges this as high-speed motion. After the vision processing unit calculates the end motion velocity vector, it determines that the main motion plane is the XZ plane. It quickly completes the corresponding area clipping and orthographic projection conversion, outputting a stable cockpit view. The entire process is smooth and distortion-free, and the operator can clearly monitor the robot's motion trajectory.

[0105] 4. Posture switching compensation: When the fourth axis angle changes by 40° during workstation switching, the system triggers the viewpoint gradual transition algorithm, which completes 15 steps of frame-by-frame gradual change with a transition time of 0.5s and a frame rate of 30fps. The viewpoint switches smoothly with the joint rotation, without any sudden screen changes, ensuring the continuity of monitoring.

[0106] 5. Panoramic Trigger and Mode Switching: After a single processing step is completed, the robot enters an idle state. After 5 seconds, the system automatically triggers the panoramic mode, starts panoramic stitching, and triggers a fade-in / fade-out transition algorithm. At a frame rate of 30fps, the transparency interpolation gradually switches from anti-dizziness mode to panoramic mode within 0.3 seconds, quickly stitching together a complete panoramic image for equipment status inspection and safety checks of the processing area without the need for manual switching of the viewpoint.

[0107] (3) Operational effect: Abnormal screen locking is accurate and timely, with no screen loss; the anti-dizziness effect is significant under high-speed movement, the main motion plane is dynamically adjusted with the robot's movement direction, the screen is smooth and without jumps when switching postures, and there are no blind spots in monitoring; the whole-point inspection coverage has no blind spots, the mode switching is smooth and without sudden changes, effectively improving production continuity and processing safety, and adapting to the needs of precision processing monitoring scenarios for electronic components.

[0108] Figure 7 This disclosure provides a schematic diagram of the structure of a robot monitoring and control device 70, as shown in the exemplary embodiments. Figure 7 As shown, the robot monitoring and control device 70 includes: an acquisition module 71, a first determination module 72, a second determination module 73, a distribution module 74, and a processing module 75.

[0109] The acquisition module 71 is used to acquire the robot's joint posture data; the joint posture data includes: angle, angular velocity and direction of motion; The first determining module 72 is used to determine the motion state of the robot based on the joint posture data; The second determining module 73 is used to determine the camera working mode based on the motion state; The sending module 74 is used to send the camera's working mode to the panoramic camera so that the panoramic camera can collect monitoring video data according to the camera's working mode. The processing module 75 is used to perform anti-glare processing on the monitoring video data uploaded by the received panoramic camera to obtain the target monitoring video data.

[0110] Optionally, the joint posture data includes: joint angular velocity, and the motion state includes: high-speed motion, low-speed motion, and stationary state; when determining the robot's motion state based on the joint posture data, the first determining module 72 is used for: If the joint angular velocity is greater than the angular velocity threshold, the robot's motion state is determined to be high-speed motion; If the joint angular velocity is less than or equal to the angular velocity threshold and greater than zero, the robot's motion state is determined to be low-speed motion. If the joint angular velocity is zero and the duration of the zero joint angular velocity is greater than or equal to the set duration, then the robot's motion state is determined to be a stationary state.

[0111] Optionally, when processing the surveillance video data to obtain the target surveillance video data, the processing module 75 is used to: When the robot is moving at high speed, the robot's joint posture data and kinematic model parameters are acquired, and the spatial position coordinates of the robot's end effector in the base coordinate system are calculated based on the kinematic model parameters. Differentiation and filtering are performed on the spatial position coordinates of multiple consecutive frames to obtain the spatial motion velocity vector of the end effector. Based on the velocity weights of each axis in the spatial motion velocity vector, the main motion plane is determined according to a preset main motion plane determination strategy. The main motion plane determination strategy includes: selecting two axes whose sum of velocity weights meets a preset threshold condition to form the main motion plane, or defaulting to designating a plane as the main motion plane when the preset threshold condition is not met. Based on the joint calibration relationship between the panoramic camera and the robot, the spatial range of the main motion plane in the base coordinate system is mapped to the cropping reference area in the pixel coordinate system of the panoramic camera, and the image corresponding to the cropping reference area is cropped from the monitoring video data to obtain the cropped image. The cropped image is then subjected to distortion correction, projection mapping, and image stabilization to obtain the target surveillance video data.

[0112] Optionally, when processing the surveillance video data to obtain the target surveillance video data, the processing module 75 is used to: When the robot is moving at low speed, the monitoring video data is used to generate a local wide-angle view. The local wide-angle view retains the scene within a preset safe distance around the operation, and the distortion rate of the target monitoring video data is less than the preset distortion threshold.

[0113] Optionally, the processing module 75 can also be used for: Acquire the robot's status signals; When the status signal indicates that the return to origin is complete or the operation is idle, the camera working mode is determined to be panoramic mode; The surveillance video data is stitched together to create a panoramic view.

[0114] Optionally, the processing module 75 can also be used for: Detect the joint switching angles of the robot; If the robot's joint switching angle is greater than or equal to the set angle, initialize the current view parameters, target view parameters, and total number of transition steps; Based on the difference between the target view parameters and the current view parameters, and the total number of transition steps, calculate the equal step adjustment amount of the view adjustment; The current viewpoint parameters are interpolated and updated frame by frame according to the video frame rate to generate a gradually changing viewpoint image. Determine the deviation between the updated viewpoint parameters and the target viewpoint parameters; The transition process terminates when the deviation is less than or equal to the preset pixel threshold. If the angle change is detected again during the transition process and reaches the preset trigger threshold, the viewpoint smooth transition process is re-triggered and executed.

[0115] Optionally, the processing module 75 can also be used for: Receive emergency stop alarm signals uploaded by the robot; Based on the emergency stop alarm signal, an abnormal scene lock command is sent to the panoramic camera; so that the panoramic camera can lock the current viewpoint according to the abnormal scene lock command, capture the scene of the fault, and record the fault timestamp corresponding to the scene of the fault. Receive the instantaneous image of the failure and the failure timestamp returned by the panoramic camera.

[0116] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0117] Figure 8 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of the present disclosure. For example... Figure 8 As shown, the electronic device includes a memory 81 and a processor 82. Additionally, the electronic device also includes a power supply component 83 and a communication component 84.

[0118] Memory 81 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device.

[0119] The memory 81 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0120] Communication component 84 is used for data transmission with other devices.

[0121] The processor 82 can execute computer instructions stored in the memory 81 to: acquire joint posture data of the robot, including angle, angular velocity and direction of motion; determine the motion state of the robot based on the joint posture data; determine the camera working mode based on the motion state; send the camera working mode to the panoramic camera so that the panoramic camera can collect monitoring video data according to the camera working mode; and perform anti-dizziness processing on the monitoring video data uploaded by the panoramic camera to obtain the target monitoring video data.

[0122] Accordingly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program. When the computer-readable storage medium stores a computer program, and the computer program is executed by one or more processors, it causes one or more processors to perform... Figure 4 Each step in the method embodiment.

[0123] Accordingly, this disclosure also provides a computer program product, which includes a computer program / instructions that are executed by a processor. Figure 4 Each step in the method embodiment.

[0124] The above Figure 8 The communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0125] The above Figure 8 The power supply component provides power to various components within the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0126] The aforementioned electronic devices also include a display screen and audio components.

[0127] The display includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions, but also the duration and pressure associated with the touch or swipe operation.

[0128] An audio component may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals may be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0129] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0131] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0133] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0134] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0135] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0136] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0137] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for robot motion anti-vertigo and panoramic trigger linkage control, characterized in that, include: Acquire the robot's joint pose data; The joint posture data includes: angle, angular velocity, and direction of motion; The motion state of the robot is determined based on the joint posture data; The camera operating mode is determined based on the described motion state; The camera's operating mode is sent to the panoramic camera so that the panoramic camera can collect monitoring video data according to the camera's operating mode; Based on the received monitoring video data uploaded by the panoramic camera, anti-dizziness processing is performed on the monitoring video data to obtain target monitoring video data.

2. The method according to claim 1, characterized in that, The joint posture data includes: joint angular velocity; the motion state includes: high-speed motion, low-speed motion, and stationary state; determining the robot's motion state based on the joint posture data includes: If the joint angular velocity is greater than the angular velocity threshold, then the motion state of the robot is determined to be the high-speed motion; If the joint angular velocity is less than or equal to the angular velocity threshold and greater than zero, then the motion state of the robot is determined to be the low-speed motion. If the joint angular velocity is zero and the duration of the zero joint angular velocity is greater than or equal to a set duration, then the motion state of the robot is determined to be the stationary state.

3. The method according to claim 1, characterized in that, The process of performing anti-dizziness processing on the surveillance video data to obtain target surveillance video data includes: When the robot is moving at high speed, the joint posture data and kinematic model parameters of the robot are acquired, and the spatial position coordinates of the robot's end effector in the base coordinate system are calculated based on the kinematic model parameters. Differentiation and filtering are performed on the spatial position coordinates of multiple consecutive frames to obtain the spatial motion velocity vector of the end effector. Based on the velocity weights of each axis in the spatial motion velocity vector, the main motion plane is determined according to a preset main motion plane determination strategy; wherein, the main motion plane determination strategy includes: selecting two axes whose sum of velocity weights satisfies a preset threshold condition to form the main motion plane, or defaulting to designating a plane as the main motion plane when the preset threshold condition is not met; Based on the joint calibration relationship between the panoramic camera and the robot, the spatial range of the main motion plane in the base coordinate system is mapped to the cropping reference area in the pixel coordinate system of the panoramic camera, and the image corresponding to the cropping reference area is cropped from the monitoring video data to obtain the cropped image. The cropped image is subjected to distortion correction, projection mapping, and image stabilization to obtain the target surveillance video data.

4. The method according to claim 1, characterized in that, The process of performing anti-dizziness processing on the surveillance video data to obtain target surveillance video data includes: When the robot is moving at low speed, the monitoring video data is used to generate a local wide-angle view. The local wide-angle view retains the scene within a preset safe distance around the operation, and the distortion rate of the target monitoring video data is less than a preset distortion threshold.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the status signal of the robot; When the status signal indicates that the return to origin is complete or the operation is idle, the camera's working mode is determined to be panoramic mode; The surveillance video data is stitched together to form a panoramic view.

6. The method according to claim 1, characterized in that, The method further includes: Detect the joint switching angle of the robot; When the joint switching angle of the robot is greater than or equal to the set angle, initialize the current view parameters, target view parameters, and total number of transition steps; Based on the difference between the target viewpoint parameter and the current viewpoint parameter and the total number of transition steps, calculate the equal step adjustment amount of the viewpoint adjustment; The current viewpoint parameters are interpolated and updated frame by frame according to the video frame rate to generate a gradually changing viewpoint image. Determine the deviation between the updated viewpoint parameters and the target viewpoint parameters; The transition process is terminated when the deviation is less than or equal to the preset pixel threshold. If the angle change is detected again during the transition process and reaches the preset trigger threshold, the viewpoint smooth transition process is re-triggered and executed.

7. The method according to claim 1, characterized in that, The method further includes: Receive the emergency stop alarm signal uploaded by the robot; According to the emergency stop alarm signal, an abnormal scene lock command is sent to the panoramic camera; so that the panoramic camera locks the current viewpoint according to the abnormal scene lock command, captures the scene of the fault, and records the fault timestamp corresponding to the scene of the fault. Receive the fault moment image and fault timestamp returned by the panoramic camera.

Images