Visual servo control method, device and equipment for tool changing of cutterhead of tunnel boring machine and nonvolatile storage medium

Abstract

The invention relates to a visual servo control method, device and equipment for tool changing of a cutterhead of a tunnel boring machine and a nonvolatile storage medium. The visual servo control method comprises the steps that image data are collected, and a to-be-processed image is obtained; processing and compressing the to-be-processed image, and outputting a feature vector and a compressed image; processing the feature vector and the compressed image and outputting a motion control instruction; performing feedback check on the execution action of the motion control instruction, acquiring a first feedback error, and judging the first feedback error; and acquiring a current execution action, acquiring a second feedback error, and judging the second feedback error. Under severe conditions of illumination variation, pollution, partial shielding and the like, the servo control system can realize high-precision and high-robustness visual servo control, and the working capability of the tunnel boring machine cutterhead tool changing robot is effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of servo motion control and is mainly used for visual servo control of a cutter head tool changing robot for a tunnel boring machine, and in particular to a visual servo control method, device, equipment and non-volatile storage medium for a cutter head tool changing robot for a tunnel boring machine. Background Art

[0002] Servo motion control is a motion control technology, which is a closed-loop motion control technology in short. It has the characteristics of closed-loop control that combines the output of motion control instructions with the feedback of motion control execution effects.

[0003] Currently, the execution effect feedback of the servo motion control system is usually obtained by position feedback through encoders, or by capturing images or videos through visual systems as feedback effects.

[0004] Currently, visual servo control systems rely on the extraction of geometric features for motion control. However, due to the complex working conditions of the visual system, such as high pollution, occlusion, and complex lighting conditions, the above geometric features may not be accurately extracted, resulting in reduced accuracy and robustness of visual servoing. Common visual servo control methods: Position-based visual servoing and image-based visual servoing both have the problem of being highly dependent on environmental conditions.

[0005] In order to solve these problems, mainly to solve the visual servo control problems under high pollution, occlusion and complex lighting conditions, data-driven methods are introduced. However, most of the existing methods rely on large amounts of data collection and lack real-time and flexibility optimization. Therefore, it is urgent to develop a visual servo method that can cope with complex environments and has high real-time and robustness. Summary of the invention

[0006] In view of this, the present application proposes a visual servo control method, device, equipment and non-volatile storage medium for cutting tool changing of a tunnel boring machine cutter head.

[0007] According to one aspect of the present application, a visual servo control method for changing cutter heads of tunnel boring machines is provided, comprising:

[0008] Step S1: collecting image data and obtaining an image to be processed;

[0009] Step S2: Processing and compressing the image to be processed and outputting a feature vector and a compressed image;

[0010] Step S3: Processing the feature vector and the compressed image and outputting a motion control instruction;

[0011] Step S4: performing feedback check on the execution action of the motion control instruction, and obtaining a first feedback error, and making a judgment on the first feedback error. If the first feedback error result meets the first standard, executing step S41; if it does not meet the first standard, executing step S42;

[0012] Step S41: the first feedback error meets the first standard, and the whole process ends;

[0013] Step S42: if the first feedback error does not meet the first standard, execute step S5;

[0014] Step S5: obtaining the execution action in the current step S4, and obtaining the second feedback error, determining the second feedback error, and if the second feedback error result meets the second standard, executing step S51, and if it does not meet the second standard, executing step S52;

[0015] Step S51: the second feedback error meets the second standard, and the whole process ends;

[0016] Step S52: If the second feedback error does not meet the second standard, execute step S1.

[0017] In a possible implementation, step S2 further includes:

[0018] Step S21: gridding the unprocessed image and outputting a first image to be processed;

[0019] Step S22: acquiring a first moving direction of the first image to be processed;

[0020] Step S23: Encode the first image to be processed and combine it with the first motion direction to form the data to be processed.

[0021] In a possible implementation, step S2 further includes:

[0022] Step S24: recording each grid area in the image to be processed, and obtaining first motion feature vector data of at least two motion dimensions;

[0023] Step S25: performing self-encoding on the image data to be processed, and obtaining the feature vector and the compressed image.

[0024] In a possible implementation, step S3 further includes:

[0025] Step S31: Acquire a first adjacent feature label by combining the vector data and the compressed image with known training data;

[0026] Step S32: determining the movement direction of the first adjacent tag and obtaining the movement direction of the first grid;

[0027] Step S33: outputting the first grid movement direction as the movement control instruction.

[0028] In a possible implementation, step S4 further includes:

[0029] Step S401: establishing an oscillation counter for obtaining the change of motion speed during the motion process and outputting the first feedback error;

[0030] Step S402: establishing a first error determiner for monitoring whether the first feedback error meets a first standard.

[0031] In a possible implementation, step S5 further includes:

[0032] Step S501: establishing an operator for decomposing the motion control instruction into a first motion speed and a first motion direction;

[0033] Step S502: establishing a second error operator to output the first movement speed and the first movement direction as the second feedback error;

[0034] Step S503: establishing a second error determiner for monitoring whether the second feedback error meets a second standard.

[0035] In a possible implementation, step S4 further includes:

[0036] The operation process includes:

[0037] Step S4011: setting a counting variable and a maximum motion value;

[0038] Step S4012: if the first motion feature vector and / or the first adjacent label and / or the image to be processed and / or the motion control instruction are inverted and / or have opposite signs, counting the counting variable;

[0039] Step S4013: specifying a counting time for counting the counting variable, and converting the counting variable and the counting time into a first oscillation frequency;

[0040] Step S4014: Calculate the first oscillation frequency and the maximum motion value, and output the result as the first feedback error.

[0041] According to another aspect of the present application, a visual servo control device for changing cutters of a tunnel boring machine cutter head is provided, characterized in that it comprises: an image acquisition module, an image processing module, a servo control module and an oscillation detection module;

[0042] The image acquisition module is used to acquire image data;

[0043] The image processing module is used to perform network division, encoding, vector calculation and compression on the image and output motion control instructions;

[0044] The servo control module is used to execute the motion control instruction and determine whether the motion control instruction is executed to the set parameters;

[0045] The oscillation detection module is used to detect the oscillation frequency and make judgments.

[0046] According to another aspect of the present application, a visual servo control device for tool change of a tunnel boring machine cutter head is provided, characterized in that it comprises: a processor and a memory for storing processor executable instructions, wherein the processor is configured to implement any one of the methods described above when executing the executable instructions.

[0047] According to another aspect of the present application, a non-volatile storage medium for visual servo control of cutter change of a tunnel boring machine cutter head is provided, characterized in that the non-volatile storage medium includes stored computer program instructions, and when the computer stored instructions are executed by a processor, any one of the methods described above is implemented.

[0048] Other features and other aspects of the present application will become more apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

[0049] The present application is suitable for improving the accuracy and robustness of the visual servo system during the cutter changing process of a tunnel boring machine in complex scenarios. Under harsh conditions such as lighting changes, pollution, and partial occlusion, the servo control system of the present application can achieve high-precision and high-robustness visual servo control, effectively improving the operating capability of the cutter changing robot of the tunnel boring machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present application and, together with the description, serve to explain the principles of the present application.

[0051] Figure 1 A flow chart showing a visual servo control method for changing cutters of a tunnel boring machine cutter head according to an embodiment of the present application;

[0052] Figure 2A main structural diagram showing a visual servo control device for changing cutter heads of a tunnel boring machine according to an embodiment of the present application;

[0053] Figure 3 A main structural diagram of a visual servo control device for changing cutter heads of a tunnel boring machine according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0054] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0055] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0056] In addition, in order to better illustrate the present disclosure, numerous specific details are provided in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0057] [Method Example]

[0058] Figure 1 The flowchart of the visual servo control method for changing the cutter head of a tunnel boring machine according to an embodiment of the present application is shown. Figure 1 As shown, the method includes: step S1: collecting image data and obtaining an image to be processed; step S2: processing the image to be processed, compressing it and outputting a feature vector and a compressed image; step S3: processing the feature vector and the compressed image and outputting a motion control instruction; step S4: performing feedback check on the execution action of the motion control instruction, and obtaining a first feedback error, judging the first feedback error, if the first feedback error result meets the first standard, executing step S41, if it does not meet the first standard, executing step S42; step S41: the first feedback error meets the first standard, and the whole process ends; step S42: the first feedback error does not meet the first standard, and step S5 is executed; step S5: obtaining the execution action in the current step S4, and obtaining the second feedback error, judging the second feedback error, if the second feedback error result meets the second standard, executing step S51, if it does not meet the second standard, executing step S52; step S51: the second feedback error meets the second standard, and the whole process ends; step S52: the second feedback error does not meet the second standard, and step S1 is executed.

[0059] In another preferred embodiment, the image data is collected by sampling using a monocular vision device.

[0060] In another preferred embodiment, the image data is collected by sampling using a binocular vision device.

[0061] In another preferred embodiment, the image data is collected by sampling using a charge coupled device (CCD) image acquisition device.

[0062] In another preferred embodiment, the image data is collected by sampling using a solid-state imaging device (CMOS) image acquisition device.

[0063] In another preferred embodiment, the acquired images are sampled by a position-based visual servo system.

[0064] In another preferred embodiment, the acquired images are sampled by an image-based visual servoing.

[0065] In another preferred embodiment, the image to be processed is further processed by edge finding and / or halftone binarization and / or error diffusion binarization and / or edge extraction and / or grayscale correction and / or feature finding and / or brightness gain and / or enhanced denoising and / or eliminated denoising.

[0066] In another preferred embodiment, the first feedback error is: ±0.05% to 30% of the difference between the oscillation frequency and the maximum value of the movement.

[0067] In another preferred embodiment, the second feedback error is: ±0.05% to 30% of the difference between the current speed and the set speed change and ±0.05% to 30% of the difference between the current movement direction and the actual direction change.

[0068] Furthermore, in another preferred embodiment, step S2 also includes: step S21: gridding the unprocessed image and outputting a first image to be processed; step S22: obtaining a first motion direction of the first image to be processed; step S23: encoding the first image to be processed and combining it with the first motion direction to form data to be processed.

[0069] In another preferred embodiment, the image to be processed is divided into grid areas. Further, the image to be processed is divided into 4 grid areas; further, the image to be processed is divided into 8 grid areas; further, the image to be processed is divided into 9 grid areas; further, the image to be processed is divided into 16 grid areas; further, the image to be processed is divided into 25 grid areas; further, the image to be processed is divided into 36 grid areas; further, the image to be processed is divided into 49 grid areas; further, the image to be processed is divided into 64 grid areas; further, the image to be processed is divided into 81 grid areas; further, the image to be processed is divided into at most 4096 grid areas; after the image to be processed is divided into grid areas, the first image to be processed is output.

[0070] In another preferred embodiment, the images to be processed acquired at different positions are obtained and divided into grid areas.

[0071] In another preferred embodiment, the first movement direction of each grid is obtained by comparing the images to be processed acquired at different positions and the images of different grid areas.

[0072] In another preferred embodiment, each grid area has four different position motion feature data directions, and further, the first motion direction includes: each motion direction of the upward direction and / or the downward direction and / or the left direction and / or the right direction and / or the front direction and / or the backward direction.

[0073] In another preferred embodiment, each grid area outputs a first movement direction through an expert system; further, the first movement direction includes a vector combination of different directions of two movement axes that can be realized in any adjacent three-dimensional space.

[0074] Furthermore, in another preferred embodiment, step S2 also includes: step S24: recording each grid area in the image to be processed, and obtaining first motion feature vector data of at least two motion dimensions; step S25: self-encoding the image data to be processed, and obtaining feature vectors and compressed images.

[0075] In another preferred embodiment, the first motion vector data includes the speed of motion and the direction of motion.

[0076] In another preferred embodiment, the direction of movement includes: each direction of movement of an upward direction and / or a downward direction and / or a left direction and / or a right direction and / or a front direction and / or a rear direction.

[0077] In another preferred embodiment, the grid area image and the first motion vector are encoded by an autoencoder to generate data to be processed.

[0078] In another preferred embodiment, the autoencoder is an autoencoder based on machine learning, and further, the autoencoding is performed through a back propagation algorithm and an optimization algorithm.

[0079] In another preferred embodiment, the optimization algorithm adopts a gradient descent algorithm.

[0080] In another preferred embodiment, the autoencoder is an encoding algorithm performed via a feedforward neural network.

[0081] In another preferred embodiment, the autoencoder adopts a supervised learning computer training algorithm.

[0082] In another preferred embodiment, the autoencoder adopts an unsupervised learning computer training algorithm.

[0083] In another preferred embodiment, the autoencoder is a stacked autoencoder.

[0084] In another preferred embodiment, the autoencoder is a variational autoencoder.

[0085] In another preferred embodiment, the first motion direction is generated by using clustering direction comparison based on the feature vector after the autoencoder compression operation.

[0086] Furthermore, in another preferred embodiment, step S3 also includes: step S31: obtaining a first adjacent feature label by combining the vector data and the compressed image with known training data; step S32: determining the movement direction of the first adjacent label and obtaining the first grid movement direction; step S33: outputting the first grid movement direction as a motion control instruction.

[0087] In another preferred embodiment, the feature vector and the compressed image are input into a K nearest neighbor classifier (hereinafter referred to as a KNN classifier). The KNN classifier compares the feature vector of the current grid with the nearest neighbor data in the big data and / or the trained data, and obtains the first adjacent feature label by judging the label in the neighbor data.

[0088] In another preferred embodiment, the KNN classifier obtains the first movement direction through the direction of the first neighboring tag.

[0089] In another preferred embodiment, the KNN classifier outputs a motion control instruction via a first motion direction, and further, the motion control instruction includes: a motion speed and a motion direction.

[0090] In another preferred embodiment, the first motion direction outputs the first grid motion direction as a motion control instruction through a Jacobian matrix.

[0091] In another preferred embodiment, the Jacobian matrix is ​​calculated in real time in each cycle.

[0092] In another preferred embodiment, the Jacobian matrix is ​​obtained through offline sampling and training.

[0093] Furthermore, in another preferred embodiment, step S4 also includes: step S401: establishing an oscillation counter for obtaining the change in movement speed during the movement process and outputting a first feedback error; step S402: establishing a first error determiner for monitoring whether the first feedback error meets the first standard.

[0094] Furthermore, in another preferred embodiment, the calculation process in step S4 also includes: step S4011: setting a counting variable and a maximum motion value; step S4012: if the first motion feature vector and / or the first adjacent label and / or the image to be processed and / or the motion control instruction are inverted and / or have opposite signs, counting the counting variable; step S4013: specifying a counting time for counting the counting variable, and combining the counting variable with the counting time to convert it into a first oscillation frequency; step S4014: calculating the first oscillation frequency and the maximum motion value, and outputting them as a first feedback error.

[0095] In another preferred embodiment, a counting variable and a threshold are set to record the number of oscillations of the motion control instruction.

[0096] In another preferred embodiment, when the count variable reaches the number of oscillations, it is determined that the first feedback error meets the first standard.

[0097] In another preferred embodiment, each time it is detected that the first motion feature vector and / or the first adjacent label and / or the image to be processed and / or the motion control instruction have opposite signs to the previous first motion feature vector and / or the first adjacent label and / or the image to be processed and / or the motion control instruction components, the counting variable is counted.

[0098] Furthermore, in another preferred embodiment, step S5 also includes: step S501: establishing an operator for decomposing the motion control instruction into a first motion speed and a first motion direction; step S502: establishing a second error operator for outputting the first motion speed and the first motion direction as a second feedback error; step S503: establishing a second error determiner for monitoring whether the second feedback error meets the second standard.

[0099] In another preferred embodiment, the first motion speed and the first motion direction are sent to the motion actuator via an inverse kinematics algorithm.

[0100] In another preferred embodiment, the second criterion includes: an error in motion speed and an error in motion direction.

[0101] In another preferred embodiment, the error of the movement speed is: ±0.00% to 50.00%.

[0102] In another preferred embodiment, the error of the moving direction is: ±0.00% to 50.00%.

[0103] [Device embodiment]

[0104] Figure 2 A device according to an embodiment of the present application is shown, including: an image acquisition module 100, an image processing module 200, a servo control module 300 and an oscillation detection module 400; the image acquisition module 100 is used to acquire image data; the image processing module 200 is used to perform network division, encoding, vector calculation and compression on the image and output motion control instructions; the servo control module 300 is used to execute the motion control instructions and determine whether the motion control instructions are executed to the set parameters; the oscillation detection module 400 is used to detect the oscillation frequency and make judgments.

[0105] [Equipment Example]

[0106] Figure 3 FIG. 1 shows a device according to an embodiment of the present application. Figure 3 As shown, the device 800 includes: a processor 810 and a memory 820 for storing executable instructions of the processor 810. The processor 810 is configured to implement any of the above-mentioned visual servo control methods for changing cutters of a tunnel boring machine cutter head when executing the executable instructions.

[0107] Here, it should be noted that the number of processors 810 may be one or more. At the same time, the device 800 in the embodiment of the present application may also include an input device 830 and an output device 840. The processor 810, the memory 820, the input device 830, and the output device 840 may be connected via a bus or in other ways, which are not specifically limited here.

[0108] The memory 820, as a computer-readable storage medium, can be used to store software programs, computer executable programs and various modules, such as: the program or module corresponding to the visual servo control method of the cutter head of a tunnel boring machine in a complex scene in the embodiment of the present application. The processor 810 runs the software program or module stored in the memory 820, thereby implementing various functional applications and data processing of the device 800. The input device 830 can be used to receive input numbers or signals. Among them, the signal can be a key signal generated related to the user settings and function control of the device / terminal / server. The output device 840 can include various types of execution devices such as display screens and motion actuators.

[0109] [Storage Medium Embodiment]

[0110] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is also provided, on which computer program instructions are stored. When the computer program instructions are executed by the processor 810, any of the above-mentioned visual servo control methods for changing the cutter head of a tunnel boring machine is implemented.

[0111] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A visual servo control method for cutting tool change of a tunnel boring machine cutter head, characterized in that: Step S1: collecting image data and obtaining an image to be processed; Step S2: Processing and compressing the image to be processed and outputting a feature vector and a compressed image; Step S3: Processing the feature vector and the compressed image and outputting a motion control instruction; Step S4: performing feedback check on the execution action of the motion control instruction, and obtaining a first feedback error, and making a judgment on the first feedback error. If the first feedback error result meets the first standard, executing step S41; if it does not meet the first standard, executing step S42; Step S41: the first feedback error meets the first standard, and the whole process ends; Step S42: if the first feedback error does not meet the first standard, execute step S5; Step S5: obtaining the execution action in the current step S4, and obtaining the second feedback error, determining the second feedback error, and if the second feedback error result meets the second standard, executing step S51, and if it does not meet the second standard, executing step S52; Step S51: the second feedback error meets the second standard, and the whole process ends; Step S52: If the second feedback error does not meet the second standard, execute step S1.

2. The method according to claim 1, characterized in that: The step S2 further comprises: Step S21: gridding the unprocessed image and outputting a first image to be processed; Step S22: acquiring a first moving direction of the first image to be processed; Step S23: Encode the first image to be processed and combine it with the first motion direction to form the data to be processed.

3. The method according to claims 1 to 2, characterized in that: The step S2 further comprises: Step S24: recording each grid area in the image to be processed, and obtaining first motion feature vector data of at least two motion dimensions; Step S25: performing self-encoding on the image data to be processed, and obtaining the feature vector and the compressed image.

4. The method according to claims 1 to 3, characterized in that: The step S3 further comprises: Step S31: Acquire a first adjacent feature label by combining the vector data and the compressed image with known training data; Step S32: determining the movement direction of the first adjacent tag and obtaining the movement direction of the first grid; Step S33: outputting the first grid movement direction as the movement control instruction.

5. The method according to claims 1 to 4, characterized in that: The step S4 also includes: Step S401: establishing an oscillation counter for obtaining the change of motion speed during the motion process and outputting the first feedback error; Step S402: establishing a first error determiner for monitoring whether the first feedback error meets a first standard.

6. The method according to claims 1 to 5, characterized in that: The step S5 further comprises: Step S501: establishing an operator for decomposing the motion control instruction into a first motion speed and a first motion direction; Step S502: establishing a second error operator to output the first movement speed and the first movement direction as the second feedback error; Step S503: establishing a second error determiner for monitoring whether the second feedback error meets a second standard.

7. The method according to claims 1 to 6, characterized in that: The step S4 further comprises: The operation process includes: Step S4011: setting a counting variable and a maximum motion value; Step S4012: if the first motion feature vector and / or the first adjacent label and / or the image to be processed and / or the motion control instruction are inverted and / or have opposite signs, counting the counting variable; Step S4013: specifying a counting time for counting the counting variable, and converting the counting variable and the counting time into a first oscillation frequency; Step S4014: Calculate the first oscillation frequency and the maximum motion value, and output the result as the first feedback error.

8. A visual servo control device for changing cutter heads of tunnel boring machines, characterized in that: include: Image acquisition module, image processing module, servo control module and vibration detection module; The image acquisition module is used to acquire image data; The image processing module is used to perform network division, encoding, vector calculation and compression on the image and output motion control instructions; The servo control module is used to execute the motion control instruction and determine whether the motion control instruction is executed to the set parameters; The oscillation detection module is used to detect the oscillation frequency and make judgments.

9. A visual servo control device for changing cutter heads of tunnel boring machines, characterized in that: include: A processor and a memory for storing processor-executable instructions, wherein the processor is configured to implement the method of any one of claims 1 to 7 when executing the executable instructions.

10. A non-volatile storage medium for visual servo control of cutterhead tool change of a tunnel boring machine, characterized in that: The non-volatile storage medium includes stored computer program instructions, and when the computer program instructions are executed by a processor, the method of any one of claims 1 to 7 is implemented.

Images