Method and system for rapidly selecting and tracking dynamic visual servo target under complex background

Abstract

The invention relates to the technical field of machine vision, and provides a method and a system for quickly selecting and tracking a dynamic visual servo target under a complex background, which ensure that the target is always in a picture by continuously capturing a screen and allowing an operator to control a robot to move in a target selection period, thereby effectively reducing the risk of target loss. Meanwhile, the provided step-by-step outward expansion automatic selection mode simplifies the selection operation process, and human error operation is avoided to a certain extent. The target selection error-tolerant rate is automatically adjusted according to the moving speed of the interaction tool so as to adapt to the dynamic change of the target, the burden of an operator is further relieved, and therefore more accurate and reliable target selection is achieved. The method achieves the efficient selection and real-time tracking of a target, and provides an innovative solution for the application of a dynamic visual servo system.

Description

Technical Field

[0001] The present disclosure relates to the technical field related to machine vision, and in particular to a method and system for quickly selecting a dynamic visual servoing target under a complex background. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] In the field of robotics and automation control, visual servo technology is one of the important means to achieve high-precision target positioning and motion control. Visual servo systems guide robots to perform precise operation tasks by acquiring visual images of target objects. They are widely used in industrial manufacturing, autonomous driving, medical surgery and other fields. However, traditional visual servo systems face some technical difficulties and limitations in practical applications.

[0004] First, due to the complexity of the background in the environment and the dynamic movement of the target object, the visual servo system faces severe challenges in rapid target recognition and high-precision positioning. Traditional target selection methods mainly rely on static or semi-dynamic scenes, which are suitable for application scenarios with a relatively simple background. When the background interference is large or the target object moves quickly, this method is often difficult to adapt, which can easily lead to target recognition failure or increased positioning error. In the target selection stage, it is common for the target to be out of the picture, that is, the target is lost, which seriously affects the robustness and practicality of the system.

[0005] Secondly, as the target object moves non-uniformly in space, the visual servo system needs to have dynamic following and adaptive processing capabilities. However, traditional methods usually lack effective modeling of the dynamic motion characteristics of the target, resulting in insufficient system response speed and anti-interference ability. This not only limits the application of visual servo technology in complex dynamic scenes, but also puts higher requirements on improving the real-time performance of the system. Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a method and system for quickly selecting and tracking dynamic visual servo targets in complex backgrounds. By combining image texture analysis and deep learning technology, efficient selection and real-time tracking of targets can be achieved under limited time and computing resources, providing an innovative solution for the application of dynamic visual servo systems.

[0007] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0008] One or more embodiments provide a method for quickly selecting and tracking a dynamic visual servoing target under a complex background, which is applied to a dynamic visual servoing system, including a master end and a slave end, wherein the master end controls the slave end to track the target, including the following steps:

[0009] Acquire the master's position and posture in real time, and send follow-up instructions to the slave, so that the slave follows the master's movements and maintains the same position and posture;

[0010] Detect and map the interactive tools on the master end to the coordinate system of the screen collected by the slave end robot;

[0011] The sampling time interval is dynamically adjusted according to the moving speed of the interactive tool to sample the images collected by the slave robot;

[0012] For the sampled image, continuous screenshots are taken from small to large, with the interactive tool coordinates as the center, to obtain multi-scale regional images, perform layer texture analysis and similarity matching, and obtain the final set of matching results for the selected target;

[0013] Target tracking is performed on the matching result set of the selected target.

[0014] One or more embodiments provide a fast selection and tracking system for a dynamic visual servoing target under a complex background, including a master and slave robots in communication connection;

[0015] The master end includes a master end microprocessor, and the master end microprocessor is configured to execute the steps of the above-mentioned method for fast selection and tracking of a dynamic visual servoing target under a complex background.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] In the present disclosure, by continuously taking screenshots and allowing the operator to control the movement of the robot (or camera) during target selection, the target is always kept in the picture, effectively reducing the risk of target loss. At the same time, the proposed automatic selection method that expands outward step by step simplifies the selection operation process and avoids human error operations to a certain extent. In addition, the target selection tolerance is automatically adjusted according to the movement speed of the interactive tool (such as a finger) to adapt to the dynamic changes of the target, further reducing the burden on the operator, thereby achieving more accurate and reliable target selection, which is particularly suitable for the recognition and tracking of fast-moving objects.

[0018] The advantages of the present disclosure and the advantages of additional aspects will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and the description thereof are used to explain the present disclosure but do not constitute a limitation of the present disclosure.

[0020] Figure 1 The main end selection control interaction system structure block diagram of the embodiment 1 of the present disclosure;

[0021] Figure 2 This is a structural block diagram of a slave robot system according to Embodiment 1 of the present disclosure;

[0022] Figure 3 A schematic diagram of quickly selecting a target in a method for quickly selecting and tracking a dynamic visual servoing target under a complex background according to Embodiment 2 of the present disclosure;

[0023] Figure 4 This is an overall flow chart of a method for quickly selecting and tracking a dynamic visual servoing target under a complex background according to Embodiment 2 of the present disclosure;

[0024] Figure 5 Obtaining a region division map for the layer of Embodiment 2 of the present disclosure;

[0025] Figure 6 This is a flow chart of layer texture analysis of Embodiment 2 of the present disclosure;

[0026] Figure 7 This is a flowchart of image feature similarity matching in Example 2 of the present disclosure. DETAILED DESCRIPTION

[0027] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0028] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0029] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments in the present disclosure and the features in the embodiments can be combined with each other. The embodiments will be described in detail below in conjunction with the accompanying drawings.

[0030] Example 1

[0031] In the technical solutions disclosed in one or more embodiments, Figure 1 to Figure 2 As shown, a fast selection and tracking system for dynamic visual servoing targets under complex backgrounds includes: a master robot and a slave robot;

[0032] The master end includes a master end microprocessor, a wireless communication unit, an image display unit, an image processing unit, a front camera unit and an inertial measurement unit; the master end microprocessor is connected to the image processing unit, the image display unit, the wireless communication unit, the front camera unit and the inertial measurement unit respectively;

[0033] A feasible technical solution is that the master end can be set on a wearable device, which can be conveniently worn on the operator's body, such as on the head, to send control instructions and receive slave robot system information.

[0034] A master-side wireless communication unit, used to realize data and image transmission between the master and slave ends;

[0035] An image display unit, used to display the real-time images collected from the end robot system;

[0036] Image processing unit, used for texture and similarity analysis of the images collected by the slave robot;

[0037] The front camera unit is used to capture images and recognize and detect human hands, and determine the relative positions of fingers;

[0038] Inertial measurement unit, used to analyze the operator's motion state and calculate the operator's posture;

[0039] The master-side microprocessor unit is used to generate instructions and information for controlling each unit, and send the processed results from the master to the slave-side robot system via wireless communication;

[0040] The slave robot comprises an image acquisition unit, a robot motion control unit and a slave wireless communication unit, wherein the robot motion control unit is connected to the image acquisition unit and the slave wireless communication unit respectively;

[0041] The image acquisition unit is used to collect information about the robot's surrounding environment, and the robot motion control unit is used to control the robot's motion.

[0042] Example 2

[0043] Based on Example 1, this embodiment provides a method for quickly selecting and tracking a dynamic visual servoing target under a complex background, which can be configured to be implemented in a host microprocessor, such as Figures 3 to 7 As shown, the following steps are included:

[0044] Step 1: Obtain the position and posture of the master end in real time, and send a follow-up instruction to the slave end so that the slave end follows the action of the master end and maintains the same position and posture;

[0045] Step 2: Detect and map the interactive tool on the master end to the coordinate system of the screen collected by the slave end robot;

[0046] Step 3: dynamically adjust the sampling time interval according to the moving speed of the interactive tool to sample the images collected by the slave robot;

[0047] For the sampled image, we continuously take screenshots from small to large with the interactive tool coordinates as the center, obtain multi-scale regional images, perform layer texture analysis and similarity matching, and obtain the final matching result set of the selected target. f2 ;

[0048] Step 4: Matching result set O for the selected target f2 Conduct target tracking;

[0049] In this embodiment, by continuously taking screenshots and allowing the operator to control the movement of the robot (or camera) during target selection, the target is always kept in the picture, effectively reducing the risk of target loss. At the same time, the proposed step-by-step outward expansion of the automatic selection method simplifies the selection operation process and avoids human error operations to a certain extent. In addition, the target selection tolerance is automatically adjusted according to the movement speed of the interactive tool (such as a finger) to adapt to the dynamic changes of the target, further reducing the burden on the operator, thereby achieving more accurate and reliable target selection, which is particularly suitable for the recognition and tracking of fast-moving objects.

[0050] Step 1: Get the position and posture of the master end in real time, and send a follow-up instruction to the slave end so that the slave end follows the action of the master end and keeps the position and posture consistent; specifically:

[0051] Step 11: Detect the operator's head posture T through the inertial measurement unit on the wearable device of the master end;

[0052] The inertial measurement unit on the wearable device detects the operator's head posture and sends the head posture T as the target posture to the slave robot system;

[0053] Step 12: Control the end posture of the slave robot to be consistent with the head posture of the operator, so as to achieve synchronization between the slave robot and the operator;

[0054] The master sends the target posture T to the slave robot system and controls it to follow the movement of the human head so that the robot's terminal posture is consistent with the operator's head posture T;

[0055] In this step, by following the master end robot through the slave end robot, the operator can control the movement of the robot (camera) during target selection, thereby ensuring that the target is always in the picture, effectively reducing the risk of target loss.

[0056] Step 2: Detect and map the interactive tool on the master end to the image captured by the slave end robot to obtain the coordinates of the interactive tool;

[0057] The interactive tool is used to point to the target object, which can be a human finger or a handheld object. The master end is generally a device worn by the operator, and the slave end is an intelligent body such as a robot that performs certain operations based on the control of the master end.

[0058] Specifically, using the index finger of the hand as an interaction tool, finger coordinate detection and mapping include the following steps:

[0059] Step 21: Use the front camera unit of the wearable device at the master end to identify the index finger of the hand and determine its coordinate p1 (x1, y1) in the first picture of the front camera unit;

[0060] Step 22, slave end image acquisition and display: obtain image 2 captured by the image acquisition unit installed at the end of the slave end robot;

[0061] Specifically, an image acquisition unit installed at the end of the slave robot is used to capture a frame of real-time image, and the image is transmitted back to an image display unit worn by the operator through a wireless communication unit;

[0062] Step 23, coordinate mapping: map the coordinate p1 (x1, y1) in the first image to the second image in proportion to obtain the coordinate p2 (x2, y2) of the index finger of the hand;

[0063] Specifically, the coordinate position p1 (x1, y1) is mapped in a proportional manner to the picture displayed in real time by the image display unit of the master end, and its coordinate p2 (x2, y2) in the image display system is obtained. The mapping calculation formula is as follows:

[0064]

[0065] in, and Indicates the range of finger movement in the front camera unit screen; and Indicates the image display unit screen display range; defines that the screen origin of the image display unit of the master end is located at the upper left corner;

[0066] In this embodiment, fingers are used as auxiliary tools. The role of fingers in the method is to serve as an interactive tool between the operator and the robot system. By identifying the coordinates of the fingers, the system on the main end can convert the operator's intention (pointing to the target area) into coordinates that the system can process. The coordinates of the fingers are used to guide the visual servo system to lock the target object.

[0067] Step 3: Dynamically adjust the sampling time interval to sample the images collected by the slave robot; for the sampled images, continuously take screenshots from small to large, with the interactive tool coordinates as the center, to obtain multi-scale regional images, and identify and obtain the initially selected targets;

[0068] Step 31, the master end dynamically adjusts the sampling time interval to sample the images collected by the slave end robot;

[0069] Step 32: Continuously take screenshots of the acquired sampled image, centering on the coordinates of the interactive tool, and gradually expanding outward from small to large, and select regions of multiple scales to form a layer set V of regions to be selected;

[0070] Step 33, select the target based on texture analysis: perform layer texture analysis on each layer in the set and calculate the wrapping rate; if the wrapping rate requirement is met, determine the minimum layer of the target object as the selected target; if the requirement is not met, return to step 1 and reselect.

[0071] Specifically, a screenshot of the real-time image displayed by the image display unit is taken at a set time interval ΔT seconds, and the currently acquired screenshot is defined as img, wherein ΔT is the sampling time interval; multiple scales of regions are selected with the coordinate p2 (x2, y2) as the center in the screenshot img, which together constitute a layer set V of the region to be selected; the system performs texture analysis on the layers of different sizes in the set V, and calculates the wrapping rate. If the analysis result meets the requirements, and the minimum layer w that wraps the target object is determined, the next step (step 34) is continued; if the analysis result does not meet the requirements, the system returns to step 1 and reselects the target;

[0072] In the above steps, the selection process is first simplified through the automatic selection method that expands outward step by step, which avoids human error to a certain extent. In addition, the target selection tolerance is automatically adjusted according to the movement speed of the finger to achieve dynamic adjustment of the sampling interval to adapt to the dynamic changes of the target, further reducing the burden on the operator, thereby achieving more accurate and reliable target selection. The operator only needs to keep moving the finger to the appropriate position in the picture, and the system will start to expand outward from the fingertip point step by step from small to large to perform texture analysis and wrapping rate calculation, and automatically select the area containing the target object;

[0073] As a further technical solution, in step 31, the sampling time interval is dynamically adjusted according to the moving speed of the interactive tool. The sampling time interval ΔT can be dynamically determined based on the frame reduction parameter d at the previous moment. The specific process is as follows:

[0074] Step 311, calculating the position change Δp of the interactive tool (finger tip) in the display image uploaded by the slave robot at the previous and next moments;

[0075] The calculation formula of position change Δp is:

[0076] Δp=||p1-p′1||;

[0077] Among them, p′1 represents the finger coordinate position at the previous moment, ||·|| represents the Euclidean norm calculation symbol;

[0078] Step 312: According to the position change Δp and the previous sampling time interval ΔT last , calculate the average speed v of the interactive tool (finger tip) moving in the screen, the calculation formula is:

[0079]

[0080] Step 313: Calculate the frame reduction parameter d according to the moving speed v of the interactive tool (finger tip). The specific calculation formula is as follows:

[0081]

[0082] in, Indicates rounding down to ensure that d is an integer; appropriately select parameters a and b to control the growth rate of d; the frame reduction parameter f is a non-negative integer and ranges from 0 to f-1; f represents the set system standard frame number;

[0083] In this embodiment, the frame reduction parameter f is calculated using an exponential growth relationship, so that the growth rate of d increases as the average speed v of the fingertip moving in the picture increases;

[0084] Step 314: Calculate the current sampling time interval ΔT according to the frame reduction parameter d and the set system standard frame number. The specific calculation formula is:

[0085]

[0086] In step 32, the method for constructing the layer set V of the area to be selected is as follows:

[0087] Step 321, setting the shape of the selected area to a rectangle, and determining the aspect ratio div of the rectangle;

[0088] The image resolution of the main end image display unit is m×n, and the method for determining the aspect ratio of the rectangle div is as follows:

[0089]

[0090] Among them, max(·) and min(·) represent the maximum value and minimum value operations respectively; Figure 5 As shown, D1, D2, D3 and D4 respectively represent quadrant areas divided in the image display unit screen; p2∈Dn Indicates that coordinate p2 is located in region D n Inside;

[0091] The D1 region can be represented by the following constraints:

[0092]

[0093] The D2 region can be represented by the following constraints:

[0094]

[0095] The D3 region can be represented by the following constraints:

[0096]

[0097] The D4 region can be represented by the following constraints:

[0098]

[0099] Step 322, set the number q of rectangular areas of the screenshot to be expanded outward, and calculate the side length increment Δl(Δx, Δy) of the rectangular area, and the calculation formula is as follows:

[0100]

[0101] Step 323, taking the interactive tool (operator's finger position) coordinate p2 (x2, y2) as the center, determine a rectangle that gradually expands outward according to the aspect ratio div, wherein the size of each rectangular area differs by Δl until it reaches the screen boundary;

[0102] Step 324, taking screenshots of the rectangles from small to large, forming q layers in total, and the q layers together constitute a layer set V of the area to be selected;

[0103] Further, in step 33, a target is selected based on layer texture analysis, and layer texture analysis is performed, including the following steps:

[0104] (1) Traverse the set of layers V of the area to be selected in order from small to large image areas, and select the current target analysis layer;

[0105] (2) Use the gray level co-occurrence matrix (GLCM) algorithm to analyze the texture characteristics (such as contrast, homogeneity, energy, etc.) of the current target analysis layer to separate the target object from the background;

[0106] (3) Calculate the target envelopment rate cov by using the area ratio of the target and the background in the current target analysis layer. The calculation formula is as follows:

[0107]

[0108] Among them, S t Represents the area of ​​the image that is similar to the target texture feature, S b Represents the area occupied by the background;

[0109] Determine whether the target wrapping rate is less than the set threshold. If the condition is met, the target separation is considered successful, the texture analysis is ended, and the selected layer w obtained by the texture analysis is added to the target preliminary selection set O. f ; Otherwise, continue with the texture analysis of the next layer; If the target separation fails to be completed for multiple consecutive times, and the number exceeds the texture analysis frame skipping parameter, the texture analysis process ends; The specific implementation process is as follows:

[0110] (4) If the wrapping rate is greater than the threshold ε1, execute step (5); if the wrapping rate is less than the threshold ε1, it is determined that the texture analysis result based on the current set V can meet the requirements, the target separation is completed, and the layer texture analysis process ends;

[0111] (5) Determine whether the traversal in step (1) is completed. If the traversal has not yet been completed, execute step (6); if all layers have been traversed, determine that the target separation cannot be completed based on the current set V, and execute step (7);

[0112] (6) Execute the traversal process of step (1), update the analysis target layer, and then execute step (2);

[0113] (7) Determine whether the number of consecutive occurrences of this situation is greater than the texture analysis frame skipping parameter n1. If it is less than n1, execute step (8); if it is greater than n1, execute step (9);

[0114] The texture analysis frame skipping parameter n1 is related to the average moving speed v of the interactive tool (finger) in the screen, and a logarithmic function normalization method is used. The specific calculation formula is as follows:

[0115]

[0116] in, Indicates the maximum setting value of n1. The range of texture analysis frame skip parameter n1 is v max Indicates the maximum value of the average moving speed v of the finger in the screen, and the range of v is [0,v max ]; c1 is a positive constant, which is designed to prevent the logarithmic function from being undefined when v = 0;

[0117] (8) It is determined that the texture analysis result based on the current set V fails to meet the requirements and fails to achieve the separation of the target object and the background, and the layer texture analysis process is terminated; and the target preliminary selection set O is cleared. f; Target initial selection set O f Used to store the selected target image;

[0118] (9) Based on step (8), it is also necessary to clear the set O f ;

[0119] Furthermore, in step 3, matching is performed on the selected target to obtain the final selected target, using a similarity matching method, including the following steps:

[0120] Step 34: Add the selected layer w obtained based on texture analysis to the target preliminary selection set O f ;

[0121] Step 35: Preliminary selection of target set O f The layers in the image feature similarity matching are used to extract images whose similarity is greater than the similarity matching threshold ε2 as the final target image matching result set O f2 ;

[0122] Initially select the target set O f The layers in the image feature similarity matching are performed, if the similarity meets the requirements, and the final matching result set O is obtained f2 , then proceed to step 4; if the matching is not completed, return to step 1; if the analysis result does not meet the requirements, end the matching process;

[0123] In step 35, a set O is initially selected for the target f In the layer, when the target is initially selected set O f The number of layers meets the requirements, and the steps for image feature similarity matching are as follows:

[0124] Step 351: Preliminary selection of target set O f The graphs in the figure are matched with each other and the similarity is calculated respectively;

[0125] Step 352: If the number of similarities less than the similarity matching threshold ε2 is less than the similarity analysis frame skipping parameter n2, extract the target preliminary selection set O f The layers whose similarity is greater than the threshold ε2 form a set O f2 , as the final matching result set O of the selected target f2 ;

[0126] Specifically, the example steps for implementing step 35 are as follows:

[0127] (35.1) Set the number of layers n3 required for similarity analysis and matching, and judge the set O fWhether the number of existing layers in is greater than n3, if it is greater than n3, then continue to step (352); otherwise, it is determined that the number of collected layers has not yet met the requirement, that is, the similarity matching process has not been completed, and the current similarity analysis process ends;

[0128] (35.2) Using the structural similarity (SSIM) algorithm, we can f The images in the figure are matched with each other for similarity, and the similarity is calculated respectively. The specific calculation formula is as follows:

[0129]

[0130] Among them, s ij Represents the set O f The similarity between the i-th layer and the j-th layer in , and similarly, Indicates the similarity between the n3-1th layer and the n3th layer; s ij The range of s is [0,1], ij The larger the value, the higher the similarity between the i-th layer and the j-th layer; i With μ j They represent the average brightness of the i-th layer and the j-th layer, respectively, reflecting their brightness levels; and They represent the brightness variance of the i-th layer and the j-th layer, respectively, reflecting their contrast levels; σ ij It represents the covariance between the i-th layer and the j-th layer, reflecting its structural information; c3 and c4 are positive constants, and their design purpose is to avoid the situation where the denominator is zero during the calculation process;

[0131] Where i and j are both integers and satisfy the following relationship:

[0132]

[0133] (35.3) ensure that the set O f The continuity and stability of the similarity of the layers in the Is the number of frames less than the similarity matching threshold ε2 less than the similarity analysis frame skipping parameter n2? If so, proceed to step (35.4); otherwise, proceed to step (35.6);

[0134] The calculation method of the similarity analysis frame skipping parameter n2 is consistent with the calculation method of the texture analysis frame skipping parameter n1. The specific calculation formula is as follows:

[0135]

[0136] in, Indicates the maximum setting value of n2. The range of texture analysis frame skip parameter n2 is v max Indicates the maximum value of the average moving speed v of the finger in the screen, and the range of v is [0,v max ]; c2 is a positive constant, which is designed to prevent the logarithmic function from being undefined when v = 0;

[0137] (35.4) Extract set O f The layers whose similarity is greater than the threshold ε2 form a set O y2 , the final matching result set O of the selected target f2 The number of layers in is n4;

[0138] (35.5) Ensure that the set O f2 The overall similarity of the middle layer is calculated The average value E of the similarity is determined to be greater than the average value threshold ε3. If it is greater than ε3, the set O is determined. f The image feature similarity matching process of the layer is completed, that is, the layer similarity analysis meets the requirements, and the current similarity analysis process ends; otherwise, continue to execute step (35.6);

[0139] in, Represents all the different combinations of selecting 2 layers from n4 layers;

[0140] (35.6) Determine based on the current set O f The layer similarity analysis and matching cannot be completed, that is, the analysis result does not meet the requirements, the current similarity analysis process ends, and step 1 is executed to re-collect the image;

[0141] Step 4: Matching result set O for the selected target f2 Target tracking includes the following steps:

[0142] Step 41: Matching result set O for selected target f2 Extract features from the layers in the image and obtain deep features as the target feature library;

[0143] Specifically, the final matching result set O f2 The layers in are input into the lightweight feature extraction network, and the deep features of each layer are extracted and stored to form a target feature library;

[0144] Step 42: Obtain each frame of real-time image uploaded by the slave robot, perform feature extraction, calculate the similarity between the extracted features and the features in the target feature library, and locate and track the targets whose similarity is higher than a set threshold;

[0145] In this step, the target detection and tracking phase is entered. For each frame of real-time image A displayed by the image display unit, 11 , cosine similarity can be used to calculate the real-time image A 11 The included angle between the feature vector and the feature vector in the target feature library is used to determine their similarity and thereby realize target detection, that is, to detect the position of the target in the picture in real time, thus completing preparations for subsequent visual servo control;

[0146] The present embodiment adopts a real-time fast dynamic target selection method, which can reduce the risk of target loss after target selection. Some traditional target selection methods require taking a screenshot of the display screen, and then performing a relatively slow target selection process on the static screenshot. When the selection is completed and enters the detection stage, it is very likely that the target has left the camera screen, that is, the target is lost. Especially when facing fast-moving targets, the probability of the above situation is particularly prominent. The method of this embodiment continuously takes screenshots of the real-time camera screen in step 3 during the entire target selection process, and during this period the operator can control the movement of the robot (camera) to ensure that the target always exists in the camera screen. This can effectively solve the above-mentioned target loss problem;

[0147] The automatic selection method of this embodiment that expands outward step by step can reduce the difficulty of operations during the selection process. In some traditional target selection processes, a large number of tedious manual operations are required, such as gesture frame selection and other processes. During this period, human subjective factors are large, and operational errors are prone to occur. Even if the system continues to take screenshots, when facing a fast-moving target, due to the rapidity and uncertainty of the target's movement, it is easy for the previous selection operation to fail, such as the target leaving the selection area that is about to be closed by the gesture frame. In the method proposed in this embodiment, the operator only needs to keep moving the finger to the appropriate position in the picture, and the system will begin to perform texture analysis and wrapping rate calculation from the fingertip point position, expanding outward step by step from small to large, and automatically select the area containing the target object;

[0148] The method proposed in this embodiment can change the system's dynamic selection error tolerance rate according to the speed of the finger movement, thereby reducing the operational burden of the selection. Because the position of the finger in the picture needs to change with the movement of the target object, the finger movement speed can indirectly reflect the movement speed of the target object; for example, when the finger moves quickly in the picture, it means that the target object is likely to be moving relatively quickly. At this time, the probability of target selection errors increases, and it is necessary to dynamically adjust the relevant parameters to affect the system's error tolerance rate during the dynamic selection process, thereby reducing the finger's movement burden and avoiding or reducing the adverse effects of operational errors on target selection.

[0149] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

[0150] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A fast selection and tracking method for dynamic visual servo targets under complex backgrounds, applied to a dynamic visual servo system, including a master end and a slave end, wherein the master end controls the slave end to track the target, and is characterized in that: The steps include: Acquire the master's position and posture in real time, and send follow-up instructions to the slave, so that the slave follows the master's movements and maintains the same position and posture; Detect and map the interactive tools on the master end to the coordinate system of the screen collected by the slave end robot; The sampling time interval is dynamically adjusted according to the moving speed of the interactive tool to sample the images collected by the slave robot; For the sampled image, continuous screenshots are taken from small to large, with the interactive tool coordinates as the center, to obtain multi-scale regional images, perform layer texture analysis and similarity matching, and obtain the final set of matching results for the selected target; Target tracking is performed on the matching result set of the selected target.

2. The method for fast selection and tracking of dynamic visual servo targets under complex backgrounds as claimed in claim 1, characterized in that: The interactive tool is a finger. Detecting and mapping the interactive tool of the master end includes the following steps: Using the front camera unit of the wearable device at the master end to identify the finger, determine the coordinate p1 of the finger in the picture 1 of the front camera unit; Acquire an image 2 captured by an image acquisition unit installed at the end of the slave robot; The coordinate p1 in the first image is mapped to the second image in proportion to obtain the coordinate p2 of the index finger of the hand.

3. The method for fast selection and tracking of dynamic visual servoing targets under complex backgrounds as claimed in claim 1, characterized in that: Dynamically adjust the sampling time interval based on the movement speed of the interactive tool, including the following steps: Calculate the position change Δp of the interactive tool in the display screen uploaded by the slave robot before and after; According to the position change Δp and the previous sampling time interval ΔT last , calculate the average speed v of the interactive tool moving in the picture; Calculate the frame reduction parameter d according to the average speed of the interactive tool; The current sampling time interval ΔT is calculated according to the frame reduction parameter d and the set system standard frame number.

4. The method for fast selection and tracking of dynamic visual servoing targets under complex backgrounds as claimed in claim 1, characterized in that: For the acquired sampled image, with the interactive tool coordinates as the center, continuous screenshots are taken from small to large, and regions of multiple scales are selected to form a layer set V of the region to be selected. The construction method of the layer set V of the region to be selected is as follows: Set the shape of the selection area to a rectangle and determine the aspect ratio of the rectangle div; Set the number of rectangular areas q that expand outwards to capture the screenshot, and calculate the increment of the side length of the rectangular area; With the interactive tool coordinates as the center, determine a rectangle that expands outward step by step according to the aspect ratio div, where the size of each rectangular area differs by Δl until it reaches the screen boundary; Screenshots are taken for the rectangles from small to large, forming q layers in total, and the q layers together constitute a layer set V of the area to be selected.

5. The method for fast selection and tracking of dynamic visual servoing targets under complex backgrounds as claimed in claim 4, characterized in that: Selecting a target based on layer texture analysis includes the following steps: Traverse the set of layers V of the area to be selected in order from small to large image areas, and select the current target analysis layer; Use the gray-level co-occurrence matrix algorithm to analyze the texture features of the current target analysis layer and separate the target object from the background of the picture; Calculate the target envelopment rate by using the area ratio of the target and the background in the current target analysis layer screen; Determine whether the target wrapping rate is greater than the set threshold. If the condition is met, the target separation is considered successful, the texture analysis is ended, and the selected layer w obtained by the texture analysis is added to the target preliminary selection set O. f ; Otherwise, continue with the texture analysis of the next layer; if the target separation cannot be completed for multiple consecutive times and the number exceeds the texture analysis frame skipping parameter, the texture analysis process is terminated.

6. The method for fast selection and tracking of dynamic visual servoing targets under complex backgrounds as claimed in claim 1, characterized in that: The similarity matching method is used to match the final selected target, including the following steps: Add the selected layer w obtained based on texture analysis to the target preliminary selection set O f ; Initially select the target set O f The graphs in the figure are matched with each other and the similarity is calculated respectively; If the number of similarities less than the similarity matching threshold ε2 is less than the similarity analysis frame skipping parameter n2, the target preliminary selection set O is extracted. f The layers whose similarity is greater than the threshold ε2 form a set O f2 , as the final matching result set O of the selected target f2 .

7. The method for fast selection and tracking of dynamic visual servoing targets under complex backgrounds as claimed in claim 1, characterized in that: Similarity analysis frame skipping parameter n2, the calculation formula is as follows: in, Indicates the maximum setting value of n2. The range of texture analysis frame skip parameter n2 is v max Indicates the maximum value of the average moving speed v of the finger in the screen, and the range of v is [0,v max ]; c2 is a positive constant.

8. The method for fast selection and tracking of dynamic visual servoing targets under complex backgrounds as claimed in claim 1, characterized in that: Target tracking is performed for the matching result set of the selected target, including the following steps: Perform feature extraction on the layers in the matching result set of the selected target to obtain deep features as the target feature library; Obtain each frame of real-time image uploaded by the end robot, perform feature extraction, calculate the similarity between the extracted features and the features in the target feature library, and locate and track targets whose similarity is higher than the set threshold.

9. A fast selection and tracking system for dynamic visual servoing targets under complex backgrounds, characterized by: including a master and slave robot connected in communication; The master end includes a master end microprocessor, and the master end microprocessor is configured to execute the steps of the method for fast selection and tracking of a dynamic visual servoing target under a complex background as described in any one of claims 1-8.

10. The fast selection and tracking system for dynamic visual servoing targets under complex backgrounds as claimed in claim 9, characterized in that: The master end further includes a wireless communication unit, an image display unit, an image processing unit, a front camera unit and an inertial measurement unit; the master end microprocessor is respectively connected to the image processing unit, the image display unit, the wireless communication unit, the front camera unit and the inertial measurement unit; The slave robot comprises an image acquisition unit, a robot motion control unit and a slave wireless communication unit, wherein the robot motion control unit is connected to the image acquisition unit and the slave wireless communication unit respectively; The image acquisition unit is used to collect information about the robot's surrounding environment, and the robot motion control unit is used to control the robot's motion.

Images