Real-time processing system based on virtual reality

Abstract

The invention discloses a real-time processing system based on virtual reality, and the operation method of the system comprises the steps: 1, building a teaching model according to the actual work operation of a real industrial robot; 2, an operator completes a teaching process and generates a teaching path track; 3, monitoring and recording muscle surface electromyographic signal data of the operator in real time by using an electromyographic sensor worn on the operation muscle, and extracting parameters related to muscle fatigue from collected muscle surface electromyographic signals; 4, whether track deviation exists in the teaching process or not is judged based on the path track data of the industrial robot in the teaching process, and deviation adjustment is conducted based on the judgment result; the fatigue factor is introduced to perform real-time judgment and adjustment on the basis of the teaching deviation, the problem that an operator performs operation again under the condition that the teaching path track deviates due to the fact that the operator of an existing system has an operation fatigue influence factor is solved, and the method has the advantages of improving teaching accuracy and improving overall applicability.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial virtual reality, and in particular to a real-time processing system based on virtual reality. Background Art

[0002] In the context of industrial automation, most jobs can only or try to use machines and robots to enter the actual working environment to replace humans to complete tasks. Industrial robot teaching refers to the operator guiding the robot end effector or a mechanical simulation device by hand in the actual working environment, or using a teaching box to operate the robot to complete the required posture of the job, and record the posture data of each teaching point, and use the robot language for online programming. When the program is played back, the robot executes the trajectory movement required by the program. In the virtual environment, the virtual robot is used for repeated demonstration, modification and adjustment to achieve real operation drills and control simulation. In the existing virtual reality-based teaching system, the guidance of industrial robot movement is through teaching reproduction or offline programming. The trajectory is described based on the feature points in the trajectory. However, during the operator's teaching operation, it needs to perform repeated operations many times, which will generate operation fatigue based on the operator himself, and operation fatigue will form a necessary influencing factor that causes the deviation of the teaching path trajectory, resulting in low overall teaching accuracy and poor effect. The existing system only allows the operator to perform the operation again to ensure accuracy, but it not only wastes time but also increases the operating cost of the entire system. Therefore, it is necessary to design a real-time processing system based on virtual reality to improve teaching accuracy and enhance overall applicability. Summary of the invention

[0003] The purpose of the present invention is to provide a real-time processing system based on virtual reality to solve the problems raised in the above background technology.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions: a real-time processing system based on virtual reality, comprising a teaching model building module, a teaching path trajectory analysis module and a trajectory deviation adjustment processing module, characterized in that: the teaching model building module is used to build a teaching model according to the actual working operation of a real industrial robot; the teaching path trajectory analysis module is used to perform deviation analysis and calculation on the teaching path trajectory generated by the operator; the trajectory deviation adjustment processing module is used to perform real-time adjustment processing according to the judgment result of whether there is a trajectory deviation.

[0005] According to the above technical solution, the teaching model building module includes: a three-dimensional construction module, an operation model building module, and a dynamic scene understanding module. The three-dimensional construction module is used to perform three-dimensional construction of the working operation of a real industrial robot based on the teaching system; the operation model building module is used to establish an operation model corresponding to the operation performed by the industrial robot in a virtual reality interface; the dynamic scene understanding module is used to use visual understanding to enable the built industrial robot to understand the dynamic scene.

[0006] According to the above technical solution, the teaching path trajectory analysis module includes: an operation trigger mechanism module, a teaching information data conversion module, a teaching decomposition module, and a point-by-point motion module. The operation trigger mechanism module is used for the industrial robot to move according to the operator's intention and define the operation trigger mechanism in the teaching system; the teaching information data conversion module is used to convert the collected action state into motion information data of the teaching part through a visual image processing algorithm; the teaching decomposition module is used to decompose the teaching of a single trajectory into two or more teachings for the industrial robot; the point-by-point motion module is used to operate the industrial robot control execution strategy to perform point-by-point motion according to the selected trajectory.

[0007] According to the above technical scheme, the trajectory deviation adjustment processing module includes: a muscle fatigue data monitoring module, a signal characteristic calculation module, a fatigue factor introduction module, a trajectory difference calculation module, a teaching deviation judgment module, and a height and deflection real-time adjustment module. The muscle fatigue data monitoring module is used to obtain the operator's muscle surface electromyography data through the electromyography sensor worn on the operating muscle; the signal characteristic calculation module is used to describe the signal characteristics of muscle fatigue using the relevant characteristics of the muscle surface electromyography signal; the fatigue factor introduction module is used to quantify the fatigue index according to the power spectrum density and introduce the fatigue factor of the operating muscle; the trajectory difference calculation module is used to calculate the difference between the ideal posture output by the operating end and the actual posture of the execution end in the teaching trajectory along the target trajectory; the teaching deviation judgment module is used to judge whether there is a teaching deviation between the teaching path and the actual path of the robot teaching path based on the deviation threshold; the height and deflection real-time adjustment module is used to make real-time adjustments to the height and deflection of the teaching posture within the range based on the specific value of the increment.

[0008] According to the above technical solution, a real-time processing method based on virtual reality is performed, and the method comprises the following steps:

[0009] Step 1: Build a teaching model based on the actual working operation of a real industrial robot;

[0010] Step 2: The operator completes the teaching process and generates a teaching path trajectory;

[0011] Step 3: Use the electromyographic sensor worn on the operating muscle to monitor and record the operator's muscle surface electromyographic signal data in real time, and extract parameters related to muscle fatigue from the collected muscle surface electromyographic signal;

[0012] Step 4: Based on the path trajectory data of the industrial robot during teaching, determine whether there is a trajectory deviation in the teaching process, and adjust the deviation based on the judgment result.

[0013] According to the above technical solution, the steps of building the teaching model include:

[0014] The teaching system is divided into two parts according to its functional composition: the operation end and the execution end. The operation end is the virtual reality operation device, including the virtual reality display terminal, the handle control terminal and the gesture recognition terminal. The execution end is the teaching robot terminal. Based on the teaching system, the working operation of the real industrial robot is constructed in three dimensions, and the operation model corresponding to the industrial robot execution operation is established in the virtual reality interface.

[0015] By importing the industrial robot model and model simulation motion, the constructed industrial robot model data file is imported into the simulation terminal, and the machine joint axis posture is initialized according to the industrial robot parameters. The model simulation motion is that when the industrial robot angle data in the virtual controller changes, the joint angle of the robot model also changes to form a continuous motion. When the industrial robot operates in the tool coordinate system mode, the robot is translated or rotated in the x, y, and z directions of the tool coordinate system through translation transformation along the axis direction or rotation transformation around the axis direction. When the robot translates along the x direction in the tool coordinate system, the execution end translates Δx along the x direction in the tool hand coordinate system, and then the translation transformation is calculated to obtain the translation vector in the base coordinate system, and the translation transformation Δy and Δz in the y and z directions in the tool coordinate system are obtained synchronously based on this.

[0016] According to the above technical solution, the step of the operator completing the teaching process and generating the teaching path trajectory includes:

[0017] After the instructor performs the teaching behavior, the visual sensor records the instructor's current behavior and action. Through the visual image processing algorithm, the collected action state is converted into motion information data of the teaching part. Through the industrial robot motion control algorithm, the recorded motion information data is converted into action information understood by the industrial robot. The trajectory data of the industrial robot generated by the virtual environment teaching system includes trajectory data in the form of tool coordinate point posture values ​​to control the movement of the industrial robot, that is, the teaching of a single trajectory is decomposed into two or multiple teachings of the industrial robot, wherein the operation process of the two teachings is: in the first teaching, the position trajectory in the action is collected, and in the second teaching process, the posture teaching is performed on the trajectory generated by the first teaching; after the position trajectory in the execution operation action is collected in the first teaching, the joints corresponding to each posture point on the trajectory are provided to the industrial robot in the virtual environment, the industrial robot selects the posture point on the operation trajectory according to the set density, decomposes the joint angle value of the selected posture point, and then enters the second teaching, and the posture teaching is performed on the trajectory generated by the first teaching, and the industrial robot control execution strategy is operated to move point by point according to the selected trajectory.

[0018] According to the above technical solution, the step of extracting parameters related to muscle fatigue from the collected muscle surface electromyography includes:

[0019] During the operator teaching process, the fatigue data of the operator's operating muscles is monitored, that is, the operator's muscle surface electromyography data is obtained through the electromyography sensor worn on the operating muscles. During the teaching process, the fatigue of the operating muscles is used to define the operator's muscle fatigue, and the data changes of the muscle surface electromyography signal are used to describe the potential muscle activity;

[0020] The extraction and analysis process of parameters related to muscle fatigue is as follows:

[0021] Extract the instantaneous amplitude and frequency variables of the time-varying sequence of the muscle surface electromyography signal from the collected muscle surface electromyography signal data records, and define the two variables as indicators for judging the fatigue state of the teaching operator;

[0022] Muscle surface electromyography (EMG) signals are used to measure muscle fatigue. Changes in muscle length cause changes in the amplitude and frequency components of muscle surface electromyography signals during the entire process of fatigue exercise. Therefore, the recorded raw muscle surface electromyography signals are bandpass filtered to remove noise and retain useful information.

[0023] The signal characteristics of muscle fatigue are described by using the relevant characteristics of muscle surface electromyographic signals, by calculating the values ​​of signal amplitude and frequency within a limited time period of the signal;

[0024] The average frequency and median frequency of the muscle surface electromyographic signal are further estimated, and the Fourier transform related to the muscle surface electromyographic signal is obtained by the periodogram method. The median frequency is obtained by dividing the sum of the muscle surface electromyographic power spectrum by the frequencies of two segments with equal amplitudes, that is, the power spectrum or power spectrum density of the muscle surface electromyographic signal is obtained;

[0025] The fatigue index is quantified according to the power spectral density, the fatigue factor of the operating muscle is introduced, and the weight of the signal characteristics describing muscle fatigue is used as input data to obtain the impact parameters caused by muscle fatigue.

[0026] According to the above technical solution, the step of judging whether there is a trajectory deviation in the teaching process based on the path trajectory data of the industrial robot during teaching includes:

[0027] The acquired path information is sent to the industrial robot drive controller through communication, and then the drive controller controls the servo motor to perform corresponding actions to complete the teaching. After the teaching action is completed, the tracking path is obtained by frame processing using the trajectory tracking algorithm according to the teaching target. The path information is the teaching path. The path information in the teaching path is extracted and compared with the actual planned path. That is, the translation and rotation experienced from the actual posture to the ideal posture along the target trajectory at the specified time are calculated, and the difference between the ideal posture output by the operating end in the teaching trajectory and the actual posture of the execution end along the target trajectory is calculated, that is, the distance between the two positions or the angle d between the two postures.

[0028]

[0029] Among them, x0 and x1 represent the x-coordinate values ​​of the operating end and the executing end, y0 and y1 represent the y-coordinate values ​​of the operating end and the executing end, z0 and z1 represent the z-coordinate values ​​of the operating end and the executing end, and set the deviation threshold between the actual planned path and the taught path. Based on the deviation threshold, it is judged whether there is a teaching deviation between the taught path and the actual path of the robot's teaching path.

[0030] According to the above technical solution, the step of adjusting the deviation based on the judgment result includes: when there is a teaching deviation, using the calculated change of the influencing parameter over time to judge the muscle fatigue state of the operator at this time, and calculating the following error of the robot following the operator at the initial moment. The calculation process includes: based on the operation time period t0-t1, recording the path trajectory data increment Δk of the industrial robot at the initial time period t0, And the joint angle data x of the industrial robot at the end time t1 within the time period, that is, the difference between the actual path trajectory s0 and the taught path trajectory s1 is represented by S, and the increment Δk1 of the joint angle of the industrial robot at the end time t1 is calculated according to the incremental data and the joint angle data of the industrial robot. At the same time, the fatigue factor θ of the operating muscle is introduced, that is, Δk1=(Δk+x)*θ, and the homogeneous transformation matrix of the joint angle motion posture of the industrial robot is performed. The increment at the end time of the industrial robot and the size of the displacement increment in the x-axis direction, y-axis direction, and z-axis direction are calculated, and the height and deflection angle of the taught posture within the range are adjusted in real time based on the specific value of the increment.

[0031] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the present invention builds a teaching model according to the actual working operation of a real industrial robot, the operator completes the teaching process and generates a teaching path trajectory, and then uses the electromyography sensor worn on the operating muscle to monitor and record the operator's muscle surface electromyography signal data in real time, and extracts parameters related to muscle fatigue from the collected muscle surface electromyography; finally, based on the path trajectory data of the industrial robot during teaching, it is judged whether there is a trajectory deviation in the teaching process, and the deviation is adjusted based on the judgment result; that is, by introducing the fatigue factor to make real-time judgment and adjustment based on the teaching deviation, the problem of requiring the operator to operate again to improve the accuracy when the teaching path trajectory is offset due to the influence of operation fatigue in the existing system is solved, thereby improving the overall teaching accuracy and overall applicability while saving time and operating costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0033] Figure 1 A flowchart of a real-time processing method based on virtual reality provided in Embodiment 1 of the present invention;

[0034] Figure 2 A schematic diagram of the module composition of a real-time processing system based on virtual reality provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Embodiment 1: Figure 1 Flow chart of a real-time processing method based on virtual reality provided in Embodiment 1 of the present invention. This embodiment can be applied to the scenario of industrial robot teaching under virtual reality. This method can be executed by a real-time processing system based on virtual reality provided in this embodiment. Figure 1 As shown, the method specifically comprises the following steps:

[0037] Step 1: Build a teaching model based on the actual working operation of a real industrial robot;

[0038] In the embodiment of the present invention, the system collects data information of the actual working operation of the actual industrial robot, and the teaching system divides it into two parts according to the functional composition: the operating end and the execution end. The operating end is the virtual reality operating device, including the virtual reality display terminal, the handle control terminal and the gesture recognition terminal. The execution end is the teaching robot terminal. The working operation of the real industrial robot is constructed in three dimensions based on the teaching system, and the operation model corresponding to the industrial robot execution operation is established in the virtual reality interface.

[0039] Exemplarily, by importing the industrial robot model and model simulation motion, the constructed industrial robot model data file is imported into the simulation terminal, and the machine joint axis posture is initialized according to the industrial robot parameters. The model simulation motion is that when the industrial robot angle data in the virtual controller changes, the joint angle of the robot model also changes to form a continuous motion. When the industrial robot operates in the tool coordinate system mode, the robot is translated or rotated in the x, y, and z directions of the tool coordinate system through a translation transformation along the axis direction or a rotation transformation around the axis direction. When the robot is translated in the x direction of the tool coordinate system, the execution end is translated Δx in the x direction in the tool hand coordinate system, and then the translation transformation is calculated to obtain the translation vector in the base coordinate system, and the translation transformation Δy and Δz in the y and z directions in the tool coordinate system are obtained synchronously based on this.

[0040] Exemplarily, the teaching system uses visual understanding to enable the built industrial robot to understand dynamic scenes, performs semantic segmentation and scene space information extraction based on RGB cameras, extracts the operator's teaching trajectory samples collected by the camera, uses the RGB camera to transmit teaching information to the industrial robot, and through modeling, enables the industrial robot to learn the set control strategy and form a teaching operation process for performing work tasks.

[0041] Step 2: The operator completes the teaching process and generates a teaching path trajectory;

[0042] In an embodiment of the present invention, the industrial robot moves according to the operator's intention, defines an operation trigger mechanism in the teaching system, combines the operator's left hand gesture information with the control terminal on the right hand control handle, defines the opening and closing of the user's left palm as a trigger signal, and when the operator's left palm is detected to be closed, tracks the position of the left palm, and uses a conversion algorithm to convert the position information of the operator's left palm into coordinates in the robot coordinate system in the teaching system. A control command is generated according to the result obtained by the conversion algorithm, and is transmitted to the robot terminal in the teaching system in real time through UDP communication. The operator uses the right hand to control the robot to perform the operation using the handle;

[0043] Exemplarily, after the instructor performs the teaching behavior, the visual sensor records the instructor's current behavior and action, and through the visual image processing algorithm, the collected action state is converted into motion information data of the teaching part, and the recorded motion information data is converted into action information understood by the industrial robot through the industrial robot motion control algorithm. The trajectory data of the industrial robot generated by the virtual environment teaching system includes trajectory data in the form of tool coordinate point posture values ​​to control the movement of the industrial robot, that is, the teaching of a single trajectory is decomposed into two or more teachings of the industrial robot, and the operation process of the two teachings is: in the first teaching, the position trajectory in the action is collected, and in the second teaching process, the posture teaching is performed on the trajectory generated by the first teaching; after the position trajectory in the execution operation action is collected in the first teaching, the joints corresponding to each posture point on the trajectory are provided to the industrial robot in the virtual environment, and the industrial robot selects posture points on the operation trajectory according to the set density, decomposes the joint angle values ​​of the selected posture points, and then enters the second teaching, and through the posture teaching of the trajectory generated by the first teaching, the industrial robot control execution strategy is operated to move point by point according to the selected trajectory.

[0044] Step 3: Use the electromyographic sensor worn on the operating muscle to monitor and record the operator's muscle surface electromyographic signal data in real time, and extract parameters related to muscle fatigue from the collected muscle surface electromyographic signal;

[0045] In the embodiment of the present invention, the fatigue data of the operator's operating muscles are monitored during the operator's teaching process, that is, the operator's muscle surface electromyography data is obtained by an electromyography sensor worn on the operating muscle, and the fatigue of the operator's muscles is defined during the teaching operation using the fatigue of the operating muscles, and the potential muscle activity is described using the data changes of the muscle surface electromyography signal;

[0046] The extraction and analysis process of parameters related to muscle fatigue is as follows:

[0047] Extract the instantaneous amplitude and frequency variables of the time-varying sequence of the muscle surface electromyography signal from the collected muscle surface electromyography signal data records, and define the two variables as indicators for judging the fatigue state of the teaching operator;

[0048] Muscle surface electromyography (EMG) signals are used to measure muscle fatigue. Changes in muscle length cause changes in the amplitude and frequency components of muscle surface electromyography signals during the entire process of fatigue exercise. Therefore, the recorded raw muscle surface electromyography signals are bandpass filtered to remove noise and retain useful information.

[0049] The signal characteristics of muscle fatigue are described by using the relevant characteristics of muscle surface electromyographic signals, by calculating the values ​​of signal amplitude and frequency within a limited time period of the signal;

[0050] The average frequency and median frequency of the muscle surface electromyographic signal are further estimated, and the Fourier transform related to the muscle surface electromyographic signal is obtained by the periodogram method. The median frequency is obtained by dividing the sum of the muscle surface electromyographic power spectrum by the frequencies of two segments with equal amplitudes, that is, the power spectrum or power spectrum density of the muscle surface electromyographic signal is obtained;

[0051] The fatigue index is quantified according to the power spectral density, the fatigue factor of the operating muscle is introduced, and the weight of the signal characteristics describing muscle fatigue is used as input data to obtain the impact parameters caused by muscle fatigue.

[0052] Step 4: Based on the path trajectory data of the industrial robot during teaching, determine whether there is a trajectory deviation in the teaching process, and adjust the deviation based on the judgment result.

[0053] In an embodiment of the present invention, the acquired path information is sent to the industrial robot drive controller through communication, and then the drive controller controls the servo motor to perform corresponding actions to complete the teaching. After the teaching action is completed, the tracking path is obtained by frame processing using a trajectory tracking algorithm according to the teaching target. The path information is the teaching path. The path information in the teaching path is extracted and compared with the actual planned path. That is, the translation and rotation experienced from the actual posture to the ideal posture along the target running trajectory at a specified time are calculated, and the difference between the ideal posture output by the operating end in the teaching trajectory and the actual posture of the execution end along the target trajectory is calculated, that is, the distance between the two positions or the angle d between the two postures.

[0054]

[0055] Wherein, x0 and x1 represent the x-coordinate values ​​of the operation end and the execution end, y0 and y1 represent the y-coordinate values ​​of the operation end and the execution end, and z0 and z1 represent the z-coordinate values ​​of the operation end and the execution end. The deviation threshold between the actual planned path and the teaching path is set, and based on the deviation threshold, it is judged whether there is a teaching deviation between the teaching path and the actual path of the robot teaching path;

[0056] Exemplarily, when there is a teaching deviation, the calculated influence parameter changes over time to judge the muscle fatigue state of the operator at this time, and the following error of the robot following the operator at the initial moment is calculated. The calculation process includes: based on the operation time period t0-t1, the path trajectory data increment Δk of the industrial robot at the initial time period t0 is recorded, And the joint angle data x of the industrial robot at the end time t1 within the time period, that is, the difference between the actual path trajectory s0 and the taught path trajectory s1 is represented by S, and the increment Δk1 of the joint angle of the industrial robot at the end time t1 is calculated according to the incremental data and the joint angle data of the industrial robot. At the same time, the fatigue factor θ of the operating muscle is introduced, that is, Δk1=(Δk+x)*θ, and the homogeneous transformation matrix of the joint angle motion posture of the industrial robot is performed. The increment at the end time of the industrial robot and the size of the displacement increment in the x-axis direction, y-axis direction, and z-axis direction are calculated, and the height and deflection angle of the taught posture within the range are adjusted in real time based on the specific value of the increment.

[0057] Embodiment 2: Embodiment 2 of the present invention provides a real-time processing system based on virtual reality. Figure 2 A schematic diagram of the module composition of a real-time processing system based on virtual reality provided in the second embodiment of the present invention, such as Figure 2 As shown, the system includes:

[0058] The teaching model building module is used to build the teaching model according to the actual working operation of the real industrial robot;

[0059] The teaching path trajectory analysis module is used to perform deviation analysis and calculation on the teaching path trajectory generated by the operator;

[0060] The trajectory deviation adjustment processing module is used to perform real-time adjustment processing according to the judgment result of whether there is a trajectory deviation.

[0061] In some embodiments of the present invention, the teaching model building module includes:

[0062] The 3D construction module is used to construct the working operation of the real industrial robot in 3D based on the teaching system;

[0063] An operation model building module is used to build an operation model corresponding to the operation performed by the industrial robot in the virtual reality interface;

[0064] The dynamic scene understanding module is used to use visual understanding to enable the built industrial robot to understand dynamic scenes.

[0065] In some embodiments of the present invention, the teaching path trajectory analysis module includes:

[0066] The operation trigger mechanism module is used to make the industrial robot move according to the operator's intention and define the operation trigger mechanism in the teaching system;

[0067] The teaching information data conversion module is used to convert the collected motion state into motion information data of the teaching part through a visual image processing algorithm;

[0068] A teaching decomposition module is used to decompose the teaching of a single trajectory into two or more teachings for the industrial robot;

[0069] The point-by-point motion module is used to operate the industrial robot control execution strategy to perform point-by-point motion according to the selected trajectory.

[0070] In some embodiments of the present invention, the trajectory deviation adjustment processing module includes:

[0071] A muscle fatigue data monitoring module is used to obtain the operator's muscle surface electromyography data through an electromyography sensor worn on the operating muscle;

[0072] A signal characteristic calculation module, used to describe the signal characteristics of muscle fatigue by using the relevant characteristics of muscle surface electromyography signals;

[0073] The fatigue factor introduction module is used to quantify the fatigue index according to the power spectrum density and introduce the fatigue factor of the operating muscle;

[0074] A trajectory difference calculation module is used to calculate the difference between the ideal posture output by the operating end and the actual posture of the execution end along the target trajectory in the teaching trajectory;

[0075] A teaching deviation judgment module is used to judge whether there is a teaching deviation between the teaching path and the actual path of the robot based on a deviation threshold;

[0076] The height and deflection real-time adjustment module is used to adjust the height and deflection of the teaching posture in real time based on the specific value of the increment.

[0077] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0078] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A real-time processing system based on virtual reality, comprising a teaching model building module, a teaching path trajectory analysis module and a trajectory deviation adjustment processing module, characterized in that: The teaching model building module is used to build a teaching model according to the actual working operation of a real industrial robot; the teaching path trajectory analysis module is used to perform deviation analysis and calculation on the teaching path trajectory generated by the operator; the trajectory deviation adjustment processing module is used to perform real-time adjustment processing based on the judgment result of whether there is a trajectory deviation.

2. A real-time processing system based on virtual reality according to claim 1, characterized in that: The teaching model building module includes: a three-dimensional construction module, an operation model building module, and a dynamic scene understanding module. The three-dimensional construction module is used to perform three-dimensional construction of the working operation of a real industrial robot based on the teaching system; the operation model building module is used to establish an operation model corresponding to the operation performed by the industrial robot in a virtual reality interface; and the dynamic scene understanding module is used to use visual understanding to enable the built industrial robot to understand the dynamic scene.

3. A real-time processing system based on virtual reality according to claim 2, characterized in that: The teaching path trajectory analysis module includes: an operation trigger mechanism module, a teaching information data conversion module, a teaching decomposition module, and a point-by-point motion module. The operation trigger mechanism module is used for the industrial robot to move according to the operator's intention and define the operation trigger mechanism in the teaching system; the teaching information data conversion module is used to convert the collected action state into motion information data of the teaching part through a visual image processing algorithm; the teaching decomposition module is used to decompose the teaching of a single trajectory into two or more teachings for the industrial robot; the point-by-point motion module is used to operate the industrial robot control execution strategy to perform point-by-point motion according to the selected trajectory.

4. A real-time processing system based on virtual reality according to claim 3, characterized in that: The trajectory deviation adjustment processing module includes: a muscle fatigue data monitoring module, a signal characteristic calculation module, a fatigue factor introduction module, a trajectory difference calculation module, a teaching deviation judgment module, and a height and deflection real-time adjustment module. The muscle fatigue data monitoring module is used to obtain the operator's muscle surface electromyography data through the electromyography sensor worn on the operating muscle; the signal characteristic calculation module is used to describe the signal characteristics of muscle fatigue using the relevant characteristics of the muscle surface electromyography signal; the fatigue factor introduction module is used to quantify the fatigue index according to the power spectrum density and introduce the fatigue factor of the operating muscle; the trajectory difference calculation module is used to calculate the difference between the ideal posture output by the operating end and the actual posture of the execution end along the target trajectory in the teaching trajectory; the teaching deviation judgment module is used to judge whether there is a teaching deviation between the teaching path and the actual path of the robot teaching path based on the deviation threshold; the height and deflection real-time adjustment module is used to make real-time adjustments to the height and deflection of the teaching posture within the range based on the specific value of the increment.

5. The real-time processing system based on virtual reality according to claim 4, performing a real-time processing method based on virtual reality, characterized in that: The method comprises the following steps: Step 1: Build a teaching model based on the actual working operation of a real industrial robot; Step 2: The operator completes the teaching process and generates a teaching path trajectory; Step 3: Use the electromyographic sensor worn on the operating muscle to monitor and record the operator's muscle surface electromyographic signal data in real time, and extract parameters related to muscle fatigue from the collected muscle surface electromyographic signal; Step 4: Based on the path trajectory data of the industrial robot during teaching, determine whether there is a trajectory deviation in the teaching process, and adjust the deviation based on the judgment result.

6. A real-time processing system based on virtual reality according to claim 5, characterized in that: The steps of building the teaching model include: The teaching system is divided into two parts according to its functional composition: the operation end and the execution end. The operation end is the virtual reality operation device, including the virtual reality display terminal, the handle control terminal and the gesture recognition terminal. The execution end is the teaching robot terminal. Based on the teaching system, the working operation of the real industrial robot is constructed in three dimensions, and the operation model corresponding to the industrial robot execution operation is established in the virtual reality interface. By importing the industrial robot model and model simulation motion, the constructed industrial robot model data file is imported into the simulation terminal, and the machine joint axis posture is initialized according to the industrial robot parameters. The model simulation motion is that when the industrial robot angle data in the virtual controller changes, the joint angle of the robot model also changes to form a continuous motion. When the industrial robot operates in the tool coordinate system mode, the robot is translated or rotated in the x, y, and z directions of the tool coordinate system through translation transformation along the axis direction or rotation transformation around the axis direction. When the robot translates along the x direction in the tool coordinate system, the execution end translates Δx along the x direction in the tool hand coordinate system, and then the translation transformation is calculated to obtain the translation vector in the base coordinate system, and the translation transformation Δy and Δz in the y and z directions in the tool coordinate system are obtained synchronously based on this.

7. The real-time processing method based on virtual reality according to claim 6, characterized in that: The step of the operator completing the teaching process and generating a teaching path trajectory includes: After the instructor performs the teaching behavior, the visual sensor records the instructor's current behavior and action. Through the visual image processing algorithm, the collected action state is converted into motion information data of the teaching part. Through the industrial robot motion control algorithm, the recorded motion information data is converted into action information understood by the industrial robot. The trajectory data of the industrial robot generated by the virtual environment teaching system includes trajectory data in the form of tool coordinate point posture values ​​to control the movement of the industrial robot, that is, the teaching of a single trajectory is decomposed into two or multiple teachings of the industrial robot, wherein the operation process of the two teachings is: in the first teaching, the position trajectory in the action is collected, and in the second teaching process, the posture teaching is performed on the trajectory generated by the first teaching; after the position trajectory in the execution operation action is collected in the first teaching, the joints corresponding to each posture point on the trajectory are provided to the industrial robot in the virtual environment, the industrial robot selects the posture point on the operation trajectory according to the set density, decomposes the joint angle value of the selected posture point, and then enters the second teaching, and the posture teaching is performed on the trajectory generated by the first teaching, and the industrial robot control execution strategy is operated to move point by point according to the selected trajectory.

8. The real-time processing method based on virtual reality according to claim 7, characterized in that: The step of extracting parameters related to muscle fatigue from the collected muscle surface electromyography comprises: During the operator teaching process, the fatigue data of the operator's operating muscles is monitored, that is, the operator's muscle surface electromyography data is obtained through the electromyography sensor worn on the operating muscles. During the teaching process, the fatigue of the operating muscles is used to define the operator's muscle fatigue, and the data changes of the muscle surface electromyography signal are used to describe the potential muscle activity; The extraction and analysis process of parameters related to muscle fatigue is as follows: Extract the instantaneous amplitude and frequency variables of the time-varying sequence of the muscle surface electromyography signal from the collected muscle surface electromyography signal data records, and define the two variables as indicators for judging the fatigue state of the teaching operator; Muscle surface electromyography (EMG) signals are used to measure muscle fatigue. Changes in muscle length cause changes in the amplitude and frequency components of muscle surface electromyography signals during the entire process of fatigue exercise. Therefore, the recorded raw muscle surface electromyography signals are bandpass filtered to remove noise and retain useful information. The signal characteristics of muscle fatigue are described by using the relevant characteristics of muscle surface electromyographic signals, by calculating the values ​​of signal amplitude and frequency within a limited time period of the signal; The average frequency and median frequency of the muscle surface electromyographic signal are further estimated, and the Fourier transform related to the muscle surface electromyographic signal is obtained by the periodogram method. The median frequency is obtained by dividing the sum of the muscle surface electromyographic power spectrum by the frequencies of two segments with equal amplitudes, that is, the power spectrum or power spectrum density of the muscle surface electromyographic signal is obtained; The fatigue index is quantified according to the power spectral density, the fatigue factor of the operating muscle is introduced, and the weight of the signal characteristics describing muscle fatigue is used as input data to obtain the impact parameters caused by muscle fatigue.

9. A real-time processing system based on virtual reality according to claim 8, characterized in that: The step of judging whether there is a trajectory deviation in the teaching process based on the path trajectory data of the industrial robot during teaching comprises: The acquired path information is sent to the industrial robot drive controller through communication, and then the drive controller controls the servo motor to perform corresponding actions to complete the teaching. After the teaching action is completed, the tracking path is obtained by frame processing using the trajectory tracking algorithm according to the teaching target. The path information is the teaching path. The path information in the teaching path is extracted and compared with the actual planned path. That is, the translation and rotation experienced from the actual posture to the ideal posture along the target trajectory at the specified time are calculated, and the difference between the ideal posture output by the operating end in the teaching trajectory and the actual posture of the execution end along the target trajectory is calculated, that is, the distance between the two positions or the angle d between the two postures. Among them, x0 and x1 represent the x-coordinate values ​​of the operating end and the executing end, y0 and y1 represent the y-coordinate values ​​of the operating end and the executing end, z0 and z1 represent the z-coordinate values ​​of the operating end and the executing end, and set the deviation threshold between the actual planned path and the taught path. Based on the deviation threshold, it is judged whether there is a teaching deviation between the taught path and the actual path of the robot's teaching path.

10. A real-time processing system based on virtual reality according to claim 9, characterized in that: The step of adjusting the deviation based on the judgment result includes: when there is a teaching deviation, using the calculated change of the influencing parameter over time to judge the muscle fatigue state of the operator at this time, and calculating the following error of the robot following the operator at the initial moment. The calculation process includes: based on the operation time period t0-t1, recording the path trajectory data increment Δk of the industrial robot at the initial time period t0, And the joint angle data x of the industrial robot at the end time t1 within the time period, that is, the difference between the actual path trajectory s0 and the taught path trajectory s1 is represented by S, and the increment Δk1 of the joint angle of the industrial robot at the end time t1 is calculated according to the incremental data and the joint angle data of the industrial robot. At the same time, the fatigue factor θ of the operating muscle is introduced, that is, Δk1=(Δk+x)*θ, and the homogeneous transformation matrix of the joint angle motion posture of the industrial robot is performed. The increment at the end time of the industrial robot and the size of the displacement increment in the x-axis direction, y-axis direction, and z-axis direction are calculated, and the height and deflection angle of the taught posture within the range are adjusted in real time based on the specific value of the increment.

Images