Live working robot remote control method and system based on virtual reality

Abstract

The present application relates to the technical field of mechanical arm control, and particularly relates to a live working mechanical arm remote control method and system based on virtual reality. The present application judges the space occupation result of the target object grabbed on the workable area in real time, and obtains the form complexity parameter of the target object; then analyzes the state transition of the operation instruction information of the live working mechanical arm and the response motion information of the live working mechanical arm, obtains the response effective coefficient of the live working mechanical arm, and then controls the live working mechanical arm in combination with the form complexity parameter and the space occupation result. The present application analyzes the space occupation of the target object grabbed in real time, so as to trigger the protection mechanism, and at the same time, evaluates the response deviation in combination with the state transition of the instruction and the motion of the mechanical arm, and then regulates and controls the motion of the mechanical arm in real time, reduces the collision risk and improves the remote control effect of the live working mechanical arm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arm control, in particular to a live working mechanical arm remote control method and system based on virtual reality. BACKGROUND

[0002] The live working mechanical arm is a robot device specially used for live working on high-voltage electrical equipment, which uses force feedback and master-slave control technology to replace manual work for dangerous or high-precision work tasks; however, since the operator may have a perception error for the actual working environment of the mechanical arm, the current virtual reality (VR) technology is usually used to provide the operator with a three-dimensional virtual environment highly similar to the live working scene where the mechanical arm is located, thereby enhancing the environmental perception, and distance collision monitoring is performed based on the sensor to assist risk warning, so as to remotely control the mechanical arm.

[0003] However, when the mechanical arm moves to its physical limit position or cannot continue to move due to environmental restrictions, the mechanical arm cannot respond to the operation instruction, and when the mechanical arm stops due to obstacles or restrictions, if the operator does not timely adjust the operation instruction and inputs a continue-to-move instruction, not only the operation experience will be affected, but also the mechanical arm may have unpredictable actions after resuming movement due to the accumulation of operation instructions, increasing the safety risks such as collision, thereby resulting in poor remote control effect of the live working mechanical arm. SUMMARY

[0004] In order to solve the technical problem of poor remote control effect of the live working mechanical arm, the purpose of the present application is to provide a live working mechanical arm remote control method and system based on virtual reality, and the technical solution adopted is as follows:

[0005] The live working mechanical arm remote control method based on virtual reality, the method comprises:

[0006] real-time acquisition of a grabbing image and a workable area of the live working mechanical arm when grabbing a target object, and real-time acquisition of operation instruction information of the live working mechanical arm and response motion information of the live working mechanical arm;

[0007] judging a space occupation result of the grabbed target object on the workable area in the grabbing image; acquiring a shape complexity parameter of the target object according to a shape feature of a corresponding area of the target object in the grabbing image;

[0008] According to the state transition of the operation instruction information and the response motion information, a command significant transition vector of the operation instruction information and a motion significant transition vector of the response motion information are constructed respectively; according to the similarity between the command significant transition vector and the motion significant transition vector, a response effective coefficient of the live-line work mechanical arm is obtained; according to the response effective coefficient and the shape complexity parameter, the live-line work mechanical arm is remotely controlled in combination with the space occupation result.

[0009] Further, the judgment method of the space occupation result comprises:

[0010] All segmentation regions in the captured image are obtained based on a threshold segmentation algorithm, and the segmentation region containing the image center pixel point of the captured image is taken as a target object region; when the area of the target object region is greater than the area of the workable region, it is determined that the captured target object exists space occupation, otherwise it is determined that it does not exist.

[0011] Further, the method for obtaining the shape complexity parameter comprises:

[0012] In the captured image, any boundary pixel point on the region boundary of the target object region is taken as a target pixel point, and the target pixel point is taken as a starting point to start traversing along the region boundary from any direction, the coordinate distance between the position coordinates corresponding to all adjacent boundary pixel points is obtained, and a coordinate distance sequence is constructed by sorting in the traversal order; the shape complexity parameter of the target object is obtained according to the coordinate distance sequence and a first-order difference sequence thereof.

[0013] Further, the method for obtaining the shape complexity parameter of the target object according to the coordinate distance sequence and the first-order difference sequence thereof comprises:

[0014] The variance of all sequence elements in the coordinate distance sequence is taken as a first shape complexity parameter, and the total number of non-zero elements in the first-order difference sequence is taken as a second shape complexity parameter; the shape complexity parameter of the target object is obtained by fusing the first shape complexity parameter and the second shape complexity parameter.

[0015] Further, the operation instruction information is an operation instruction vector whose vector elements are the displacement, linear acceleration and angular velocity set for each axis of the live-line work mechanical arm, and the response motion information is a work motion vector whose vector elements are the displacement, linear acceleration and angular velocity when each axis of the live-line work mechanical arm moves.

[0016] Further, the method for obtaining the command significant transition vector and the motion significant transition vector comprises:

[0017] Based on the extended Kalman filtering algorithm, the instruction state transition matrix of the operation instruction vector and the motion state transition matrix of the operation motion vector are obtained respectively; the maximum matrix elements in each row of the instruction state transition matrix are screened and a row vector is constructed to obtain an instruction significant transition vector; the maximum matrix elements in each row of the motion state transition matrix are screened and a row vector is constructed to obtain a motion significant transition vector.

[0018] Further, the method for obtaining the response effective coefficient comprises:

[0019] The negative correlation normalization result of the module of the difference vector between the instruction significant transition vector and the motion significant transition vector is taken as the response effective coefficient of the live working mechanical arm.

[0020] Further, the method for controlling the live working mechanical arm comprises:

[0021] The normalization result of the shape complexity parameter of the target object is taken as a work complexity weight; the response effective coefficient is weighted by using the work complexity weight, and the weighted result is taken as a response control coefficient of the live working mechanical arm.

[0022] According to the space occupation result, the control level symbol of the live working mechanical arm is determined; the negative correlation normalization result of the response control coefficient is weighted by using a preset gain coefficient, the weighted result is amplified after a preset basic control level is added, the control level is determined in combination with the control level symbol to control the live working mechanical arm.

[0023] Further, the method for determining the control level symbol comprises:

[0024] When the target object has space occupation, the control level symbol is set as a negative sign; when the target object has no space occupation, the control level symbol is set as a positive sign.

[0025] The live working mechanical arm remote control system based on virtual reality comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the live working mechanical arm remote control method based on virtual reality when executing the computer program.

[0026] The present application has the following advantages:

[0027] The application provides data preparation for subsequent analysis by acquiring a grabbing image and a workable area of the live-line work mechanical arm when grabbing a target object in real time, and acquiring operation instruction information of the live-line work mechanical arm and response motion information of the live-line work mechanical arm; the spatial invasion result of the grabbed target object in the grabbing image to the workable area is judged, so as to determine whether the protection mechanism needs to be triggered to adjust the control level; then the morphological complexity parameter of the target object is acquired according to the morphological features of the corresponding area of the target object in the grabbing image, to prepare for subsequent regulation and control; the instruction significant transition vector of the operation instruction information and the motion significant transition vector of the response motion information are respectively constructed according to the state transition of the operation instruction information and the response motion information, to prepare for measuring the response deviation according to the similarity of the instruction significant transition vector and the motion significant transition vector, and acquiring the response effective coefficient of the live-line work mechanical arm, and finally the live-line work mechanical arm is controlled in combination with the morphological complexity parameter and the spatial invasion result. The application analyzes the spatial invasion of the grabbed target object in real time, so as to trigger the protection mechanism, and the response deviation is evaluated according to the state transition of the instruction and the motion of the mechanical arm, and then the motion of the mechanical arm is regulated and controlled in real time, so as to reduce the collision risk and improve the remote control effect of the live-line work mechanical arm. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, and the advantages thereof, below briefly introduces the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on these drawings.

[0029] Figure 1 A flow chart of a live-line work mechanical arm remote control method based on virtual reality provided by an embodiment of the present application;

[0030] Figure 2 A method flow chart for controlling a live-line work mechanical arm provided by an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to further illustrate the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following describes a live-line work mechanical arm remote control method and system based on virtual reality according to the present application, its specific implementation, structure, features and effects in detail, as shown in the drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0033] The application provides a live-line work mechanical arm remote control method and system based on virtual reality.

[0034] Please refer to Figure 1 which shows a flowchart of a live-line work mechanical arm remote control method based on virtual reality provided by an embodiment of the application, and specifically includes the following steps.

[0035] In step S1, the live-line work mechanical arm is powered on, and a head-mounted VR device is configured for the operator of the live-line work mechanical arm.

[0036] In an embodiment of the application, the live-line work mechanical arm is powered on first, and a head-mounted VR device is configured for the operator of the live-line work mechanical arm.

[0037] The live-line work mechanical arm mainly includes a base, a large arm, a small arm and a gripper, the small arm is integrated with a camera, and a weight sensor is installed at the position of the gripper and the small arm.

[0038] At the same time, the panoramic image in the VR device is used to obtain the workable area of the live-line work mechanical arm in real time when the target object is grabbed.

[0039] Then, in a preferred embodiment of the application, the VR device is used to obtain the operator's operation instruction information, which includes the displacement, linear acceleration and angular velocity set for each axis of the live-line work robot arm, and the information is used as vector elements to construct an operation instruction vector; the response motion information of the live-line work robot arm to the operation instruction is obtained based on the attitude sensor on the live-line work robot arm, i.e., the displacement, linear acceleration and angular velocity of each axis are collected as vector elements to construct a work motion vector.

[0040] It should be noted that the acquisition of operation instruction information and response motion information is a prior art and will not be described again; the arrangement order of the vector elements in the operation instruction vector and the work motion vector is consistent to ensure the effectiveness of subsequent comparative analysis.

[0041] In step S2, the spatial invasion result of the target object grabbed in the grabbing image to the workable area is determined; and the shape complexity parameter of the target object is obtained according to the shape feature of the target object corresponding area in the grabbing image.

[0042] Due to the scene conditions such as small work space, the scene facilities will interfere with the robot arm, thereby causing collision risk or affecting the stability of operation, and the invasion of the size of the target object grabbed by the robot arm to the workable area can help to determine the activity space redundancy of the robot arm to adjust the operation instruction to avoid obstacles.

[0043] Preferably, in an embodiment of the application, considering that the target object corresponding area in the grabbing image preliminarily reflects the size of the target object, the area area difference between the target object corresponding area and the workable area preliminarily reflects the spatial redundancy of the target object in the workable area, and then the spatial invasion result of the target object is determined to evaluate whether there is a collision risk, so as to prepare for the subsequent triggering of the protection mechanism and the timely obstacle avoidance and stopping of the robot arm; and considering that the center of the grabbing image coincides with the center of the gripper, and the target object in the grabbing image appears as a connected domain, the target object in the image can be determined through the image center, and further when the area area of the target object corresponding area is greater than the area area of the workable area, it can be evaluated that there is no redundant space; therefore, the determination method of the spatial invasion result includes:

[0044] All the segmentation areas in the grabbing image are obtained based on the threshold segmentation algorithm, and the segmentation area containing the image center pixel point of the grabbing image is taken as the target object area; when the area area of the target object area is greater than the area area of the workable area, it is determined that the target object exists spatial invasion, otherwise it is determined that it does not exist.

[0045] As an example, first based on the Otsu threshold segmentation algorithm, a plurality of segmentation regions in the captured image are obtained, and then the target object region can be obtained, and the spatial invasion result is determined; when it is determined that there is invasion, subsequent obstacle avoidance stop control needs to be performed, and when there is no invasion, the operation instruction can be continued to be responded.

[0046] In other examples, the implementer can also use other threshold segmentation methods or other pre-trained labeling models to determine the target object region, which is also a prior art known to those skilled in the art as Otsu threshold segmentation, and will not be repeated here.

[0047] In addition, considering that the irregular structure of the target object may have greater spatial interference possibility when it is moved by the live working robot arm in each direction, and is more unfavorable for the response movement of the robot arm, the embodiment of the present application analyzes the size uniformity of the target object structure according to the boundary shape features of the corresponding region of the target object in the captured image, obtains the shape complexity parameter of the target object, and then facilitates the subsequent movement interference of the robot arm to control the robot arm.

[0048] Preferably, in an embodiment of the present application, it is considered that the boundary profile change of the target object region can reflect its shape complexity, and the change of the coordinate distance between adjacent boundary pixel points on the boundary profile basically reflects the complex situation of the boundary profile change. When the coordinate distances between all adjacent boundary pixel points are more consistent, it means that the boundary is more regular or smooth, otherwise it is more rugged and complex. Based on this, first, the change sequence of the coordinate distances between all adjacent boundary pixel points can be obtained by traversing along the boundary, and then the change of the sequence is analyzed to evaluate the shape complexity parameter of the target object. Therefore, the method for obtaining the shape complexity parameter comprises:

[0049] In the captured image, any boundary pixel point on the region boundary of the target object region is taken as a target pixel point, the target pixel point is taken as a starting point, and the coordinate distance between the position coordinates of all adjacent boundary pixel points is obtained by traversing along the region boundary from any direction, and the coordinate distance sequence is constructed by sorting according to the traversal order; and the shape complexity parameter of the target object is obtained according to the coordinate distance sequence and the first-order difference sequence thereof.

[0050] In addition, it is considered that the fluctuation change of the coordinate distance in the coordinate distance sequence can reflect the change or ruggedness of the boundary profile. When the fluctuation change of the sequence elements is greater, it means that the boundary profile is more rugged. And the variance can help to evaluate the fluctuation degree of the sequence elements in the sequence. It is also considered that if the adjacent sequence elements in the coordinate distance sequence are consistent, it means that the corresponding boundary pixel points are more likely to be smooth boundaries, and on the contrary, if there is a difference between the adjacent sequence elements, it means that the boundary is rugged, there may be recesses or protrusions, and the boundary shape is more complex.

[0051] Therefore, in a preferred embodiment of the present application, the variance of all sequence elements in the coordinate distance sequence is taken as the first morphological complexity parameter, and the total number of non-zero elements in the first-order difference sequence is taken as the second morphological complexity parameter; the morphological complexity parameter of the target object is obtained by fusing the first morphological complexity parameter and the second morphological complexity parameter.

[0052] As an example, first, a coordinate system is constructed with the center of the captured image as the coordinate origin to determine the position coordinates of each pixel point; then a target pixel point is determined on the region boundary, and the traversal is started in the counterclockwise direction along the region boundary, the Euclidean distance, i.e., the coordinate distance, between adjacent boundary pixel points is evaluated based on the position coordinates, and then the coordinate distances are sorted according to the traversal acquisition order to obtain a coordinate distance sequence, and further obtain a first-order difference sequence of the coordinate distance sequence; then the first morphological complexity parameter and the second morphological complexity parameter are obtained, and the first morphological complexity parameter and the second morphological complexity parameter are multiplied and fused to obtain the morphological complexity parameter.

[0053] In another embodiment of the present application, the implementer can also evaluate the ruggedness of the boundary through the boundary curvature variation; specifically, the boundary curve is fitted based on the curve fitting algorithm, and the local curvature at each boundary pixel point is calculated, wherein the boundary pixel point with a curvature greater than 0 corresponds to a protruding part, i.e., a convex point, and the boundary pixel point with a curvature less than 0 corresponds to a recessed part, i.e., a concave point; considering that the more the number of convex points and concave points and the closer they are, the greater the possibility of boundary ruggedness, the absolute value of the difference between the number of convex points and the number of concave points is negatively related, such as adding a minimum non-zero positive parameter 0.1 and then taking the reciprocal to obtain the first morphological complexity parameter; and considering that the farther the distance of the convex point from the center of the captured image and the closer the distance of the concave point from the center of the captured image, the greater the degree of boundary ruggedness, the difference between the average distance of all convex points from the center of the captured image and the average distance of all concave points from the center of the captured image is taken as the second morphological complexity parameter; then the first morphological complexity parameter and the second morphological complexity parameter are multiplied and fused to obtain the morphological complexity parameter.

[0054] In another embodiment of the present application, the target object region can also be subjected to morphological operations to further determine the number of eroded regions and dilated regions, and then the eroded regions are taken as the protruding regions and the dilated regions are taken as the recessed regions, and the greater the total number of the recessed regions and the dilated regions, the greater the morphological complexity parameter.

[0055] It should be noted that in other embodiments, the implementer can also use other fusion means such as addition or weighted summation to fuse the first morphological complexity parameter and the second morphological complexity parameter, and can also use the boundary line segment angle method to evaluate the boundary concave-convex situation of the target object region, which is well known to those skilled in the art as prior art and will not be described in detail.

[0056] Step S3, according to the state transition of the operation instruction information and the response motion information, respectively constructing the instruction significant transition vector of the operation instruction information and the motion significant transition vector of the response motion information; according to the similarity of the instruction significant transition vector and the motion significant transition vector, obtaining the response effective coefficient of the live working mechanical arm; according to the response effective coefficient and the morphological complexity parameter, combining the space occupation result to control the live working mechanical arm.

[0057] Considering the operation instruction of the operator and the response motion of the live working mechanical arm, certain state transition will occur in the remote control operation process. By analyzing the state transition information, the change difference between the instruction and the response can be understood, so as to accurately locate the response delay and deviation, which will help to optimize the performance of the mechanical arm control system in the subsequent, and reduce the delay and deviation. Therefore, in the embodiment of the present application, the instruction significant transition vector of the operation instruction information and the motion significant transition vector of the response motion information are first constructed. The two significant transition vectors reflect the main transition of the state respectively.

[0058] Preferably, in an embodiment of the present application, considering that the Extended Kalman Filter (EKF) is a recursive algorithm for state estimation of nonlinear systems, it can help to estimate the state transition. Therefore, the instruction state transition matrix and the motion state transition matrix can be obtained based on the Extended Kalman Filter algorithm. Considering that each matrix element in each row of the state transition matrix represents the probability distribution of the transition from the initial state to different states, and the maximum state transition probability in each row can reflect the significant transition intention or maximum transition information of the state, the significant transition vector can be constructed to pay more attention to the variables that have greater influence on the system state change, and then the control strategy can be better designed. Therefore, the method for obtaining the instruction significant transition vector and the motion significant transition vector comprises:

[0059] Based on the Extended Kalman Filter algorithm, the instruction state transition matrix of the operation instruction vector and the motion state transition matrix of the operation motion vector are obtained. The maximum matrix elements in each row of the instruction state transition matrix are screened and a row vector is constructed to obtain the instruction significant transition vector. The maximum matrix elements in each row of the motion state transition matrix are screened and a row vector is constructed to obtain the motion significant transition vector.

[0060] It should be noted that the acquisition of the instruction state transition matrix and the motion state transition matrix based on the Extended Kalman Filter algorithm, as well as the construction of the new row vector are all existing technologies familiar to those skilled in the art, and will not be repeated here.

[0061] If the main state transition of the operation instruction is similar to the main state transition of the mechanical arm response motion, it indicates that the current mechanical arm execution action can accurately respond to the operator's operation instruction, and the response effect is better. Therefore, the embodiment of the present application obtains the response effective coefficient of the live-line work mechanical arm according to the similarity of the instruction significant transition vector and the motion significant transition vector. The response effective coefficient reflects the response deviation degree of the mechanical arm to the control instruction, and provides a certain reference value for subsequent regulation and control of the mechanical arm motion, so as to improve the control effect.

[0062] Preferably, in an embodiment of the present application, it is considered that the smaller the difference vector length between the two vectors, the more similar the vectors, and further, the better the response effect of the motion to the instruction. The logical relationship can be adjusted through negative correlation mapping. Therefore, the method for obtaining the response effective coefficient comprises:

[0063] The negative correlation normalization result of the length of the difference vector between the instruction significant transition vector and the motion significant transition vector is taken as the response effective coefficient of the live-line work mechanical arm.

[0064] As an example, the reciprocal is taken for negative correlation normalization. In order to avoid the case that the length of the difference vector is 0, which leads to meaningless denominator, a very small positive parameter such as 0.0001 is added to the length before taking the reciprocal, and the response effective coefficient is obtained. Implementers can also use other negative correlation normalization means, which are prior art and will not be described in detail.

[0065] In another embodiment of the present application, the implementer can also measure the Pearson correlation coefficient between the instruction significant transition vector and the motion significant transition vector, and then normalize the Pearson correlation coefficient to obtain the response effective coefficient. The Pearson correlation coefficient is also prior art and will not be described in detail.

[0066] It is considered that when the target object has space occupation, that is, there is a risk of collision and interference with other objects in the work scene, the object needs to be stopped. It is also considered that the response of the morphological complexity parameter of the target object to the operation instruction has certain interference influence, that is, the more complex the morphological structure of the target object and the greater the possibility of space interference, the slower the response speed of the mechanical arm in the work motion process should be, so as to reduce the possibility of interference and collision accidents. Therefore, after the embodiment of the present application obtains the response effective coefficient, the live-line work mechanical arm can be further controlled according to the response effective coefficient and the morphological complexity parameter in combination with the space occupation result.

[0067] Preferably, in an embodiment of the present application, the method for controlling the live-line work mechanical arm comprises:

[0068] Please refer to Figure 2 which shows a method flowchart for controlling a live-line work mechanical arm according to an embodiment of the present application, and specifically comprises:

[0069] Step S301, the normalized result of the form complexity parameter of the target object is taken as a work complexity weight; the work complexity weight is used to weight the response effective coefficient, and the weighted result is taken as a response regulation coefficient of the live-line work mechanical arm.

[0070] Considering that the gripper may still rotate after grabbing the target object, the target object region in the grabbing image collected at different times may show the state of the target object under different viewing angles, and the relative size of the form complexity parameter of the target object region in the current real-time acquired grabbing image reflects the relative work complexity at the real-time moment; the response regulation coefficient of the live-line work mechanical arm is further determined in combination with the response effective coefficient of the mechanical arm.

[0071] As an example, the form complexity parameter of the target object currently acquired in real time is taken as a numerator, and the maximum value of the form complexity parameter of the target object acquired at all historical moments up to the current moment is taken as a denominator; since the denominator is always greater than or equal to the numerator, normalization can be performed, and the fractional ratio is taken as a work complexity weight; then the work complexity weight is multiplied by the response effective coefficient to obtain the response regulation coefficient.

[0072] Step S302, the control level symbol of the live-line work mechanical arm is determined according to the space occupation result; the negative correlation normalized result of the response regulation coefficient is weighted by using a preset gain coefficient, the weighted result is added to a preset basic control level and then is limited in amplitude, and the control level is determined in combination with the control level symbol to control the live-line work mechanical arm.

[0073] Considering that when there is space occupation, the mechanical arm needs to be controlled to stop avoiding obstacles in time, and a negative control level can be output to trigger a protection mechanism to make the mechanical arm stop avoiding obstacles; when there is no space occupation, a positive control level can be output to make the mechanical arm continue to execute the operation instruction; therefore, in a preferred embodiment of the present application, the method for determining the control level symbol comprises: when the target object has space occupation, the control level symbol is set to a negative sign; and when the target object has no space occupation, the control level symbol is set to a positive sign.

[0074] Further considering that the larger the response regulation coefficient is, the greater the space interference influence on the current target object grabbing is, and the smaller the response deviation of the mechanical arm is; therefore, on the premise of ensuring the control level required for normal grabbing movement, the control level needs to be timely regulated to reduce the collision risk; at the same time, it is also necessary to ensure that the regulated control level is within the normal working level range of the mechanical arm, and further needs to be limited in amplitude.

[0075] As an example, since the value range of the response regulation coefficient is 0-1, the negative correlation normalization can be directly performed by subtracting the regulation effective coefficient from 1, then multiplying the preset gain coefficient with the negative correlation normalization result, adding the product to the preset basic control level to obtain the control level, and then limiting the control level in the normal working level range by using the clamp function, so as to assign the control level symbol to the limited control level to control the robot arm.

[0076] It should be noted that the preset gain coefficient is set to 0.7, the preset basic control level is set to 7.4V, and the normal working level of the robot arm needs to be determined according to the design parameters of the robot arm. The implementer can also determine the preset gain coefficient and the preset basic control level according to the actual application, which will not be described here.

[0077] The application also provides a live working robot arm remote control system based on virtual reality, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the live working robot arm remote control method based on virtual reality described in steps S1-S3 when executing the computer program.

[0078] To sum up, the application judges the space occupation result of the grasped target object to the workable area in real time, and obtains the shape complexity parameter of the target object. Then, according to the state transition of the operation instruction information of the live working robot arm and the response motion information of the live working robot arm, the instruction significant transition vector of the operation instruction information and the motion significant transition vector of the response motion information are constructed respectively, and then the response effective coefficient of the live working robot arm is obtained according to the similarity of the instruction significant transition vector and the motion significant transition vector. The live working robot arm is controlled in combination with the shape complexity parameter and the space occupation result. The application analyzes the space occupation of the grasped target object in real time, so as to trigger the protection mechanism, and at the same time, the response deviation is evaluated in combination with the state transition of the instruction and the motion of the robot arm, and then the motion of the robot arm is regulated in real time, so as to reduce the collision risk and improve the remote control effect of the live working robot arm.

[0079] It should be noted that the above-mentioned embodiment sequence of the application is only for description, and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0080] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment mainly describes the differences from other embodiments.

Claims

1. A virtual reality-based remote control method for a live working robot arm, characterized in that, The method comprises: Real-time acquisition of a grabbing image and a workable area of the live-line work robot arm when grabbing a target object, and real-time acquisition of operation instruction information of the live-line work robot arm and response motion information of the live-line work robot arm; Judgment of a space occupation result of the grabbed target object to the workable area in the grabbing image; acquisition of a shape complexity parameter of the target object according to a shape feature of a corresponding area of the target object in the grabbing image; According to the state transition of the operation instruction information and the response motion information, a command significant transition vector of the operation instruction information and a motion significant transition vector of the response motion information are respectively constructed; according to the similarity between the command significant transition vector and the motion significant transition vector, a response effective coefficient of the live-line work robot arm is acquired; and according to the response effective coefficient and the shape complexity parameter, the live-line work robot arm is remotely controlled in combination with the space occupation result. The judgment method of the space occupation result comprises: All segmentation areas in the grabbing image are acquired based on a threshold segmentation algorithm, and the segmentation area containing the image center pixel point of the grabbing image is taken as a target object area; when the area of the target object area is greater than the area of the workable area, it is determined that the grabbed target object has space occupation, otherwise it is determined that there is no space occupation; The operation instruction information is an operation instruction vector of the displacement, linear acceleration and angular velocity of each axis of the live-line work robot arm as vector elements, and the response motion information is a work motion vector of the displacement, linear acceleration and angular velocity of each axis of the live-line work robot arm as vector elements; The acquisition method of the command significant transition vector and the motion significant transition vector comprises: Based on an extended Kalman filtering algorithm, a command state transition matrix of the operation instruction vector and a motion state transition matrix of the work motion vector are respectively acquired; the maximum matrix elements in each row of the command state transition matrix are screened and a row vector is constructed to obtain the command significant transition vector; the maximum matrix elements in each row of the motion state transition matrix are screened and a row vector is constructed to obtain the motion significant transition vector; The acquisition method of the response effective coefficient comprises: The negative correlation normalization result of the module of the difference vector between the command significant transition vector and the motion significant transition vector is taken as the response effective coefficient of the live-line work robot arm.

2. The virtual reality-based remote control method of a live working robot arm according to claim 1, characterized in that, The acquisition method of the shape complexity parameter comprises: In the grabbing image, any boundary pixel point on the area boundary of the target object area is taken as a target pixel point, the target pixel point is taken as a starting point, and the coordinate distance between the position coordinates of all adjacent boundary pixel points is acquired by traversing from any direction along the area boundary, and a coordinate distance sequence is constructed by sorting in the traversal order; the shape complexity parameter of the target object is acquired according to the coordinate distance sequence and a first-order difference sequence thereof.

3. The virtual reality-based remote control method of a live working robot arm according to claim 2, characterized in that, The method for acquiring the shape complexity parameter of the target object according to the coordinate distance sequence and the first-order difference sequence thereof comprises: A variance of all sequence elements in the coordinate distance sequence is taken as a first morphological complexity parameter, and a total number of non-zero elements in the first-order difference sequence is taken as a second morphological complexity parameter; a morphological complexity parameter of the target object is obtained by fusing the first morphological complexity parameter and the second morphological complexity parameter.

4. The virtual reality-based remote control method of a hot-line work robot arm according to claim 1, characterized by, The method for controlling the live-line work mechanical arm comprises: A normalized result of the morphological complexity parameter of the target object is taken as a work complexity weight; the response effective coefficient is weighted by using the work complexity weight, and a weighted result is taken as a response regulation coefficient of the live-line work mechanical arm; A control level symbol of the live-line work mechanical arm is determined according to the space occupation result; a negative correlation normalized result of the response regulation coefficient is weighted by using a preset gain coefficient, an amplitude of a result after the preset basic control level is added to the weighted result is limited, the control level is determined by combining the control level symbol, and the live-line work mechanical arm is controlled.

5. The virtual reality-based remote control method of a hot-line work robot arm according to claim 4, characterized by, The method for determining the control level symbol comprises: When the target object has the space occupation, the control level symbol is set as a negative sign; and when the target object does not have the space occupation, the control level symbol is set as a positive sign.

6. A virtual reality based remote control system for a live working robot, the system comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the steps of the live-line work mechanical arm remote control method based on virtual reality according to any one of claims 1-5 when executing the computer program.

Images