Collaborative robot target tracking method, device and computer readable storage medium

Abstract

The embodiment of the present application provides a kind of collaborative robot target tracking method, device and computer readable storage medium, the method comprises the following steps: from the real-time image frame of the camera device of collaborative robot transmission extraction predetermined starting image frame as current image frame;Actual hue and actual saturation of image in target tracking frame are established two-dimensional joint histogram according to;Two-dimensional joint probability map is obtained by the inverse projection of two-dimensional joint histogram;Optimized probability map is obtained by using adaptive filter to two-dimensional joint probability map spatial filtering;With the next image frame of current image frame update current image frame;Based on the mean shift algorithm model, the target tracking frame is re-determined in the updated current image frame according to optimized probability map and jumps back to the previous step.This embodiment can effectively provide the accuracy of target tracking.

Description

Technical Field

[0001] This invention relates to the field of intelligent control technology for collaborative robots, and in particular to a collaborative robot target tracking method, apparatus, and computationally readable storage medium. Background Technology

[0002] Currently, in order to grasp dynamic targets, collaborative robots typically use target dynamic recognition and tracking algorithms to perform real-time dynamic tracking of dynamic targets.

[0003] Currently, the industry typically uses the Camshift algorithm to identify and track moving targets. It is a continuous MeanShift algorithm. Its main working principle is to select a predetermined moving target within the current image frame provided by the camera device of the collaborative robot (usually by manual selection or automatic recognition by the algorithm) to obtain the target tracking box corresponding to the current image frame. Based on a color feature of the image within the target tracking box, a histogram model of the predetermined moving target is established. The histogram model is back-projected to obtain a probability map. Then, based on the mean shift algorithm model and the probability map, the predetermined moving target is selected in subsequent image frames to obtain the target tracking box, thereby realizing real-time tracking of the predetermined moving target.

[0004] However, the inventors discovered during implementation that the traditional Camshift algorithm, which constructs a histogram model based on a single color feature, cannot effectively distinguish between the intended moving target and the background when subjected to complex background interference, easily leading to tracking failure. At the same time, during the tracking of the intended moving target, the target is easily affected by factors such as illumination, scale, and shape changes, resulting in poor tracking performance. In addition, when the intended moving target is occluded, the algorithm will lose the target, and even if the intended moving target reappears, the algorithm cannot relocate and continue tracking. Summary of the Invention

[0005] The technical problem to be solved by the embodiments of the present invention is to provide a collaborative robot target tracking method that can effectively improve the accuracy of target tracking.

[0006] A further technical problem to be solved by the embodiments of the present invention is to provide a collaborative robot target tracking device that can effectively improve the accuracy of target tracking.

[0007] A further technical problem to be solved by the embodiments of the present invention is to provide a computationally readable storage medium for storing a computer program that can effectively improve the accuracy of target tracking.

[0008] To address the aforementioned technical problems, embodiments of the present invention provide the following technical solution: a collaborative robot target tracking method, comprising the following steps executed sequentially:

[0009] Step S1: Extract a predetermined starting image frame from the real-time image frames acquired and transmitted by the camera device of the collaborative robot as the current image frame, and determine a target tracking box containing the predetermined moving target from the current image frame.

[0010] Step S2: Establish a two-dimensional joint histogram based on the actual hue and actual saturation of the image within the target tracking box;

[0011] Step S3: Back-project the two-dimensional joint histogram to obtain a two-dimensional joint probability map;

[0012] Step S4: Use an adaptive filter to perform spatial filtering on the two-dimensional joint probability map to obtain an optimized probability map;

[0013] Step S5, update the current image frame with the next image frame; and

[0014] Step S6: Based on the mean shift algorithm model, determine the target tracking box containing the predetermined moving target in the updated current image frame according to the optimized probability map, and jump back to step S5.

[0015] Furthermore, the adaptive filter is a spatial filter of size m×n, where m = 2a + 1, n = 2b + 1, a and b are both positive integers, and m and n are respectively smaller than the dimensions of the target tracking box in the two image coordinate directions. Then, the two-dimensional joint probability map after spatial filtering is represented as:

[0016]

[0017] Where, p new (x, y) represents the spatially filtered two-dimensional joint probability map, p f (x, y) represents the two-dimensional joint probability graph.

[0018] Furthermore, in step S6, after determining the target tracking box containing the predetermined moving target in the updated current image frame based on the mean shift algorithm model and the optimized probability map, the target tracking box is shrunk by a predetermined ratio while keeping the center coordinates of the target tracking box unchanged, and then the shrunk target tracking box is used to jump back to step S2.

[0019] Furthermore, in step S6, after redetermining the target tracking box containing the predetermined moving target, the actual similarity between the images within the target tracking boxes of the current image frame and the previous image frame is analyzed. When the actual similarity is less than a first predetermined similarity threshold, the two-dimensional joint histogram is corrected according to the following formula, and then the process jumps back to step S3:

[0020] qnext =(1-α)q pre +αq curr , where q curr q represents the actual two-dimensional joint histogram of the image within the target tracking box corresponding to the current image frame. pre q represents the actual two-dimensional joint histogram of the image within the target tracking bounding box corresponding to the previous image frame of the current image frame. next This represents the modified two-dimensional joint histogram, where α is a predetermined constant.

[0021] Furthermore, when performing step S5, after updating the current image frame with the next image frame of the current image frame, it is also analyzed and determined whether the predetermined moving target is missing in the current image frame. If so, the subsequent image frames of the current image frame are extracted frame by frame in sequence, and the predetermined moving target is searched in each subsequent image frame based on a predetermined feature matching algorithm model. The subsequent image frame in which the predetermined moving target is found is determined as the current image frame.

[0022] Furthermore, the analysis and determination of whether the predetermined moving target is missing in the current image frame specifically includes: determining the actual similarity between the target tracking box images corresponding to the current image frame and the previous image frame;

[0023] Calculate the ratio of the actual image area of ​​the target tracking box in the current image frame to that in the previous image frame; and determine that the predetermined moving target is missing in the current image frame when the actual similarity and the actual image area ratio satisfy one of the following predetermined conditions: the actual similarity is greater than a second predetermined similarity threshold, and the actual image area ratio is greater than a predetermined image area ratio; or, the actual image area is less than the reciprocal of the predetermined image area ratio.

[0024] Furthermore, the predetermined feature matching algorithm model is the Brisk feature matching algorithm model.

[0025] Furthermore, the actual similarity is determined by calculating the Bach distance of the two-dimensional joint histogram of the target tracking box images corresponding to the current image frame and the previous image frame.

[0026] On the other hand, in order to solve the above-mentioned technical problems, the present invention provides the following technical solution: a collaborative robot target tracking device, connected to the camera device of the collaborative robot, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the collaborative robot target tracking method as described in any one of the above.

[0027] Furthermore, in order to solve the aforementioned technical problems, the present invention provides the following technical solution: a computer-readable storage medium, including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the collaborative robot target tracking method as described in any of the above.

[0028] After adopting the above technical solution, the embodiments of the present invention have at least the following beneficial effects: The embodiments of the present invention use real-time image frames acquired from the camera device of the collaborative robot as the current image frame and determine the target containing the predetermined moving target. Compared to the traditional method of constructing a histogram based solely on a single color feature of the image within the target tracking box, this invention uses the actual hue and saturation of the image within the target tracking box to establish a two-dimensional joint histogram, which provides richer color information and improves the accuracy of subsequent tracking. After obtaining a two-dimensional joint probability map by back-projecting the two-dimensional joint histogram, an adaptive filter is used to spatially filter the two-dimensional joint probability map to obtain an optimized probability map. The adaptive filter can significantly suppress the interference of complex backgrounds on target tracking while maintaining the target region probability as unchanged as possible, thus improving tracking accuracy. After updating the current image frame, based on the mean-shift algorithm model and the optimized probability map, the target tracking box containing the predetermined moving target is determined in the updated current image frame, thereby achieving continuous tracking of the predetermined moving target in each real-time image frame and effectively improving the accuracy of target tracking. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the steps of an optional embodiment of the collaborative robot target tracking method of the present invention.

[0030] Figure 2 The images provided are RGB images, HSV images, hue feature images, and saturation feature images, which are optional embodiments of the collaborative robot target tracking method of the present invention.

[0031] Figure 3 This is a two-dimensional joint histogram of an optional embodiment of the collaborative robot target tracking method of the present invention.

[0032] Figure 4 This is a schematic diagram illustrating the principle of shrinking the target tracking box in an optional embodiment of the collaborative robot target tracking method of the present invention.

[0033] Figure 5 This is a schematic diagram illustrating the principle of the neighborhood sampling mode of the Brisk algorithm, an optional embodiment of the collaborative robot target tracking method of the present invention.

[0034] Figure 6 This is a schematic diagram of an optional embodiment of the collaborative robot target tracking device of the present invention.

[0035] Figure 7 This is a functional block diagram of an optional embodiment of the collaborative robot target tracking device of the present invention. Detailed Implementation

[0036] The present application will now be described in further detail with reference to the accompanying drawings and specific embodiments. It should be understood that the following illustrative embodiments and descriptions are only used to explain the present invention and are not intended to limit the present invention. Moreover, the embodiments and features in the embodiments of the present application can be combined with each other unless otherwise specified.

[0037] like Figure 1 As shown, an optional embodiment of the present invention provides a collaborative robot target tracking method, comprising the following steps performed sequentially:

[0038] Step S1: Extract a predetermined starting image frame from the real-time image frames acquired and transmitted by the camera device 1 of the collaborative robot as the current image frame, and determine a target tracking box containing the predetermined moving target from the current image frame.

[0039] Step S2: Establish a two-dimensional joint histogram based on the actual hue and actual saturation of the image within the target tracking box;

[0040] Step S3: Back-project the two-dimensional joint histogram to obtain a two-dimensional joint probability map;

[0041] Step S4: Use an adaptive filter to perform spatial filtering on the two-dimensional joint probability map to obtain an optimized probability map;

[0042] Step S5, update the current image frame with the next image frame; and

[0043] Step S6: Based on the mean shift algorithm model, the target tracking box containing the predetermined moving target is re-determined in the updated current image frame according to the optimized probability map, and the process jumps back to step S5.

[0044] This invention, in its embodiments, uses real-time image frames acquired from the camera device 1 of the collaborative robot as the current image frame and determines those containing a predetermined moving target. Compared to the traditional method of constructing a histogram based solely on a single color feature of the image within the target tracking box, this invention establishes a two-dimensional joint histogram using the actual hue and saturation of the image within the target tracking box. This results in a two-dimensional joint histogram with richer color information, improving the accuracy of subsequent tracking. After back-projecting the two-dimensional joint histogram to obtain a two-dimensional joint probability map, an adaptive filter is used to spatially filter the two-dimensional joint probability map to obtain an optimized probability map. The adaptive filter significantly suppresses interference from complex backgrounds on target tracking while maintaining the target region probability as unchanged as possible, improving tracking accuracy. After updating the current image frame, based on the mean-shift algorithm model and the optimized probability map, a target tracking box containing the predetermined moving target is determined in the updated current image frame. This enables continuous tracking of the predetermined moving target in each real-time image frame, effectively improving the accuracy of target tracking.

[0045] In a specific embodiment, the image within the target tracking box is first converted from the RGB color space to the HSV color space using the following formulas 1 and 2, thereby extracting two color feature images, the actual hue (H) and the actual saturation (S), from the image within the target tracking box. Figure 2 As shown:

[0046]

[0047]

[0048] Where r, g, and b represent the corresponding color channel images of the image within the target tracking box, and max and min represent the maximum and minimum values ​​of r, g, and b, respectively.

[0049] Then, the actual hue H is divided into m bins, and the actual saturation is divided into n bins. The actual hue H and actual saturation S are then merged to construct a two-dimensional joint histogram according to the following formulas 3 and 4, as follows: Figure 3 As shown:

[0050] q={q uv}, u = 1, 2, ..., m; v = 1, 2, ..., n (Formula 3)

[0051]

[0052] Among them, b h (x i y i ) represents the tone feature map (x i y i b) Tonal feature value at image coordinates;s (x i y i ) represents the saturation feature map (x) i y i The saturation feature value at the image coordinates;

[0053] The actual hue H ranges from [0, 180]. In this embodiment of the invention, the hue value is quantized into 60 bins (each bin is 3 in length), and the saturation S ranges from [0, 255] and is quantized into 30 bins (each bin is 2 / 17 in length). A two-dimensional joint histogram is established for the image within the initial tracking box or the target tracking box, with a total of 60 × 30 = 1800 bins.

[0054] Furthermore, by back-projecting the two-dimensional joint histogram of chroma-saturation features, a two-dimensional joint probability map of chroma-saturation of the search image is established. The hue feature value of the pixel at coordinates (x, y) in the two-dimensional joint probability map is set to u, and the saturation feature value to v. Then, the two-dimensional joint probability map can be represented as:

[0055]

[0056] In an optional embodiment of the present invention, the adaptive filter is a spatial filter of size m×n, where m = 2a + 1, n = 2b + 1, a and b are both positive integers, and m and n are respectively smaller than the dimensions of the image in the two image coordinate directions within the target tracking box. The two-dimensional joint probability map after spatial filtering is then represented as:

[0057]

[0058] Where, p new (x, y) represents the spatially filtered two-dimensional joint probability map, p f (x, y) represents the two-dimensional joint probability graph.

[0059] In this embodiment, since m = 2a + 1 and n = 2b + 1, that is, an odd-sized spatial filter is used to perform spatial filtering on the two-dimensional joint probability map (size M × N). The resulting optimized probability map can significantly suppress the probability values ​​of interfering pixels in the background region, thereby achieving the purpose of distinguishing between the target and the background region and improving the accuracy of target recognition and tracking.

[0060] Formula 6 can also be broken down into:

[0061]

[0062] in,

[0063] Mapping the probability values ​​obtained from Formula 6 to grayscale values ​​in the range [0, 255] yields the final optimized probability map p. unew (x, y) can be represented as:

[0064]

[0065] In an optional embodiment of the present invention, in step S6, after determining the target tracking box containing the predetermined moving target in the updated current image frame based on the mean drift algorithm model and the optimized probability map, the target tracking box is shrunk by a predetermined ratio while keeping the center coordinates of the target tracking box unchanged, and then the shrunk target tracking box is used to jump back to step S2.

[0066] Through specific experiments, it was found that since the target tracking bounding box in the initial image frame is usually selected manually or determined by a machine using a target recognition algorithm, the range for recognizing the predetermined moving target is usually large. This results in the target tracking bounding box often containing a small portion of background pixels at its edges, in addition to the predetermined moving target. To remove these background pixels and avoid interference with target tracking, this embodiment, after determining the target tracking bounding box, keeps the center coordinates of the target tracking bounding box unchanged while reducing the target tracking bounding box by a predetermined ratio, such as... Figure 4 As shown, the outer box represents the target tracking box before shrinking, and the inner box represents the target tracking box after shrinking. It is clear from the figure that the center coordinates of the target tracking box remain unchanged, while shrinking the target tracking box to a certain area reduces interference from background pixels. Finally, the process jumps back to step S2, sequentially calculating and updating the two-dimensional joint histogram, the two-dimensional joint probability map, and the optimized probability map, ensuring that the aforementioned effect is achieved when determining the predetermined moving target for each subsequent image frame. In specific implementation, experiments have shown that the predetermined ratio is typically selected from 0.6 to 0.8, preferably 0.73.

[0067] In practice, the target tracking box can be reduced according to the following formula:

[0068] S u =k·S t (Formula 10)

[0069] Among them, S t and S u These represent the image area dimensions of the target tracking bounding box before and after scaling.

[0070] In an optional embodiment of the present invention, in step S6, after redetermining the target tracking box containing the predetermined moving target, the actual similarity between the images within the target tracking boxes corresponding to the current image frame and the previous image frame is analyzed. When the actual similarity is less than a first predetermined similarity threshold, the two-dimensional joint histogram is corrected according to the following formula, and then the process jumps back to step S3:

[0071] q next =(1-α)q pre +q curr , where q curr q represents the actual two-dimensional joint histogram of the image within the target tracking box corresponding to the current image frame. pre q represents the actual two-dimensional joint histogram of the image within the target tracking bounding box corresponding to the previous image frame of the current image frame. next This represents the modified two-dimensional joint histogram, where α is a predetermined constant.

[0072] When the actual similarity between the target tracking bounding box images of the current image frame and the previous image frame is less than a first predetermined similarity threshold, it indicates that the target has changed (due to factors such as illumination, scale, and shape). If target tracking is still performed based on the optimized probability map calculated from the original two-dimensional joint histogram, it will obviously lead to a decrease in tracking accuracy. Therefore, in this case, this embodiment dynamically updates the original two-dimensional joint histogram by referring to the actual two-dimensional joint histogram of the target tracking bounding box images of the current image frame and the previous image frame, thereby improving the accuracy of target tracking in subsequent image frames.

[0073] In an optional embodiment of the present invention, when performing step S5, after updating the current image frame with the next image frame of the current image frame, it is further analyzed and determined whether the predetermined moving target is missing in the current image frame. If so, subsequent image frames of the current image frame are extracted frame by frame in sequence, and the predetermined moving target is searched in each subsequent image frame based on a predetermined feature matching algorithm model. The subsequent image frame in which the predetermined moving target is found is determined as the current image frame. In this embodiment, it is also determined whether the predetermined moving target is lost from the current image frame, and after the predetermined moving target is lost, the predetermined feature matching algorithm model is used to search for the predetermined moving target, so that when the predetermined moving target reappears, it can be tracked again, thereby improving the efficiency of target tracking.

[0074] In an optional embodiment of the present invention, the step of analyzing and determining whether the predetermined moving target is missing in the current image frame specifically includes:

[0075] Determine the actual similarity between the target tracking bounding boxes of the current image frame and the previous image frame;

[0076] Calculate the ratio of the actual image area of ​​the target tracking box in the current image frame to that in the previous image frame; and determine that the predetermined moving target is missing in the current image frame when the actual similarity and the actual image area ratio satisfy one of the following predetermined conditions: the actual similarity is greater than a second predetermined similarity threshold, and the actual image area ratio is greater than a predetermined image area ratio; or, the actual image area is less than the reciprocal of the predetermined image area ratio.

[0077] In this embodiment, the actual similarity and actual image area ratio of the target tracking box images corresponding to the current image frame and the previous image frame are compared with the corresponding second predetermined similarity threshold and predetermined image area ratio. Thus, when the actual similarity and actual image area ratio meet the predetermined conditions, it is determined that the predetermined moving target is lost from the current image frame. The judgment method is relatively simple and fast, and the judgment efficiency is high.

[0078] In an optional embodiment of the present invention, the predetermined feature matching algorithm model is the Brisk feature matching algorithm model. In this embodiment, the Brisk feature matching algorithm is a binary feature description algorithm, which has rotation invariance, scale invariance, and good robustness. For feature point matching and recognition of predetermined moving targets, it can accurately find predetermined moving targets in image frames and select target tracking boxes.

[0079] Specifically, the Brisk algorithm primarily utilizes FAST9-16 for feature point detection. To achieve scale invariance, the algorithm first constructs a scale space, which contains n octave layers (c i ) and n intra-octave layers (d i The scale relationship between the octave layer, the intra-octave layer, and the original image can be expressed as:

[0080] t(c i ) = 2 i ,t(d i ) = 2 i ×1.5 (Formula 11)

[0081] By performing corner detection on each layer, and then performing non-maximum suppression on the images of each layer after corner detection, coarser feature points are obtained. Finally, a two-dimensional quadratic function interpolation method is used to interpolate the layer where the feature points are located and the corresponding upper and lower layers to obtain accurate feature points.

[0082] Furthermore, a neighborhood sampling pattern is adopted, in which N concentric circles and 60 sampling points are arranged around the feature point at equal intervals, such as... Figure 5 As shown, the set of sampling points A can be represented as:

[0083] A={(p i p j )∈R 2 ×P 2 |i<N∧j<i∧i,j∈R} (Formula 12)

[0084] For any sampling point (p) i p j Let the subset of short-distance point pairs S and the subset of long-distance point pairs L be represented as follows:

[0085]

[0086]

[0087] Where, δ max =9.57t, δ min =13.67t, where t is the scale of the feature point;

[0088] Furthermore, calculate the principal direction of the feature points:

[0089]

[0090] α=arctan2(g x g y ) (Formula 16)

[0091] Before describing the feature points, the sampling template needs to be rotated clockwise by an angle α, and then sampling point pairs are selected from the set S. By comparing their strengths, a 512-bit binary feature descriptor is obtained. The judgment method is as follows:

[0092]

[0093] Finally, Hamming distance is used to achieve feature point matching, which is to achieve target matching.

[0094] In practical implementation, to verify the performance of the Brisk algorithm, this embodiment of the invention compares Fast+FREAK, ORB (Oriented Fast and Rotated Brief, accelerating feature point extraction and description), SURF (Speeded Up RobustFeatures, accelerating robust features), SIFT (Scale-invariant feature transform), and the Brisk algorithm used in this embodiment of the invention. The performance of these algorithms is evaluated from the number of detected feature points, algorithm speed, and matching accuracy. Since the Fast algorithm does not have feature descriptors, it is combined with the FREAK algorithm to describe and match feature points. Two images are compared: a target template image and a matching image containing the target. In the matching image, the target has undergone scale changes. Feature point detection and feature matching are performed on these two images using the various algorithms mentioned above. The performance comparison is shown in the table below. The table shows the number of feature points detected by each algorithm, the running time, and the matching accuracy. The experiment was conducted 50 times, and the average value was taken.

[0095]

[0096]

[0097] Based on the comparative experimental data above, it can be seen that when there is a significant scale difference between the target template and the target in the image to be matched, the Fast algorithm has low matching accuracy because it lacks scale invariance. In contrast, ORB, SURF, SIFT, and Brisk all have scale invariance and high matching accuracy. Considering the comprehensive comparative experimental results, it can be seen that the Brisk algorithm has a fast running speed and high matching accuracy, which meets the requirements for relocalization under occlusion interference. Therefore, this embodiment of the invention uses the Brisk matching algorithm for target relocalization.

[0098] In an optional embodiment of the present invention, the actual similarity is determined by calculating the Bach distance of the two-dimensional joint histogram of the target tracking box images corresponding to the current image frame and the previous image frame. In this embodiment, the Bach distance of the two-dimensional joint histogram of the target tracking box images corresponding to the current image frame and the previous image frame is used to determine the actual similarity. The Bach distance can be used to measure the separability of two discrete probability distributions. When calculating histogram similarity, the Bach distance yields the best results and has higher accuracy in calculating the similarity between the target tracking box images corresponding to the current image frame and the previous image frame. In specific implementations, it is understood that cosine similarity principle, hash algorithms, structural similarity measurement algorithms, etc., can also be used to calculate the similarity.

[0099] In practice, Bhattacharyya distance measures the similarity between two color histograms. A smaller Bhattacharyya value indicates higher similarity, and vice versa. A Bhattacharyya distance of zero indicates that the two histograms are completely identical. The formula for calculating Bhattacharyya distance is as follows:

[0100]

[0101] Wherein, H1(I) represents the actual two-dimensional joint histogram of the image within the target tracking box corresponding to the previous image frame, and H2(I) represents the actual two-dimensional joint histogram of the image within the target tracking box corresponding to the current image frame.

[0102] Furthermore, the ratio ΔS of the actual image area within the target tracking box corresponding to the current image frame and the previous image frame can be expressed as:

[0103] ΔS=S curr / S pre (Formula 19)

[0104] Among them, S curr S represents the image area within the target tracking bounding box corresponding to the current image frame. pre The image area of ​​the target tracking box in the previous image frame.

[0105] The present invention assumes that the motion state of the predetermined moving target will not change much in two adjacent frames, so the change in the area of ​​the tracking box in two adjacent frames will not be too large; if the size of the target tracking box corresponding to the current image frame changes abruptly and the change is too large compared to the previous image frame, it is determined that the tracking result is inaccurate and the target is lost.

[0106] Therefore, a second predetermined similarity threshold T is set. occ The ratio of the predetermined image area to T area When d(H1, H2) > T occ ΔS>T area ||ΔS<1 / T area When this occurs, it indicates that the intended moving target has been lost. In this case, the Brisk feature matching algorithm is introduced for model relocation. Experiments have verified that the second predetermined similarity threshold T... occ The value range is 0.4 to 0.5, preferably 0.45, and the predetermined image area ratio T area The value range is 3 to 5, preferably 4.

[0107] In specific implementation, for the embodiments of the present invention, a tracking and grasping system for collaborative robots can be used. The system can employ a FrankaEmika robot, a Kinect2, a Realsense D435i RGB-D camera (i.e., the camera device 1 of the collaborative robot), a Robotiq 2f-85 two-finger gripper, and a conveyor belt for moving objects. The tracking and grasping system can be developed based on ROS (Robot Operating System) under Ubuntu 16.04.

[0108] This invention demonstrates the tracking and grasping of an object moving at a constant speed on a conveyor belt through experiments. The conveyor belt speeds were set to 0.1 m / s, 0.15 m / s, 0.2 m / s, and 0.25 m / s, respectively. The tracking and grasping experiments were repeated 30 times at each speed, and the grasping success rate was calculated. The table below lists the grasping success rate and average tracking time of the robotic arm under different conditions:

[0109] Speed ​​(m / s) Number of fetches Successful capture count Success rate Average tracking time (s) 0.1 30 28 93.33％ 2.81 0.15 30 28 93.33％ 2.03 0.2 30 26 86.67％ 1.63 0.25 30 25 83.33％ 1.05

[0110] As shown in the table above, at motion speeds of 0.15 m / s and below, the tracking time is longer and the grasping success rate is higher. When the object's motion speed is greater than 0.2 m / s, the tracking time decreases due to limitations imposed by the conveyor belt and the robotic arm's working range, resulting in a slight decrease in the grasping success rate.

[0111] On the other hand, this embodiment of the invention provides a collaborative robot target tracking device 3, which is connected to the camera device 1 of the collaborative robot. It includes a processor 30, a memory 32, and a computer program stored in the memory 32 and configured to be executed by the processor 30. When the processor executes the computer program, it implements the collaborative robot target tracking method as described in the above embodiments.

[0112] For example, the computer program can be divided into one or more modules / units, which are stored in the memory 32 and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the collaborative robot target tracking device 3. For example, the computer program can be divided into... Figure 7 The functional modules in the collaborative robot target tracking device 3 include the target preliminary determination module 41, the histogram establishment module 42, the probability graph calculation module 43, the probability graph optimization module 44, the image frame update module 45, and the target tracking module 46, which respectively perform the above steps S1-S6.

[0113] The collaborative robot target tracking device 3 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. The collaborative robot target tracking device 3 may include, but is not limited to, a processor 30 and a memory 32. Those skilled in the art will understand that the schematic diagram is merely an example of the collaborative robot target tracking device 3 and does not constitute a limitation on the collaborative robot target tracking device 3. It may include more or fewer components than shown, or combine certain components, or use different components. For example, the collaborative robot target tracking device 3 may also include input / output devices, network access devices, buses, etc.

[0114] The processor 30 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 30 is the control center of the collaborative robot target tracking device 3, connecting all parts of the collaborative robot target tracking device 3 via various interfaces and lines.

[0115] The memory 32 can be used to store the computer program and / or modules. The processor 30 implements various functions of the collaborative robot target tracking device 3 by running or executing the computer program and / or modules stored in the memory 32 and calling the data stored in the memory 32. The memory 32 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as image recognition function, image overlay function, etc.), etc.; the data storage area may store data created based on the use of the collaborative robot target tracking device 3 (such as image data, etc.). In addition, the memory 32 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0116] If the functions described in the embodiments of the present invention are implemented in the form of software functional modules or units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the embodiments of the present invention can implement all or part of the processes in the methods described above, or they can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 30, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0117] In another aspect, embodiments of the present invention provide a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the collaborative robot target tracking method as described in the above embodiments.

[0118] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0119] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the scope of protection of the present invention.

Claims

1. A method of collaborative robot target tracking, the method comprising: The method comprises the following steps executed in sequence: Step S1, extracting a predetermined starting image frame as a current image frame from real-time image frames collected and transmitted by a camera of a collaborative robot, and determining a target tracking box containing a predetermined moving target from the current image frame; Step S2, establishing a two-dimensional joint histogram according to actual hue and actual saturation of images in the target tracking box; Step S3, performing inverse projection on the two-dimensional joint histogram to obtain a two-dimensional joint probability map; Step S4, performing spatial filtering on the two-dimensional joint probability map by using an adaptive filter to obtain an optimized probability map; Step S5, updating the current image frame with a next image frame of the current image frame; and Step S6, re-determining a target tracking box containing the predetermined moving target in the updated current image frame according to the optimized probability map based on a mean shift algorithm model, then analyzing actual similarity of images in the respective target tracking boxes corresponding to the current image frame and its previous image frame, jumping back to step S5 when the actual similarity is not less than a first predetermined similarity threshold, and jumping back to step S3 after correcting the two-dimensional joint histogram according to the following formula when the actual similarity is less than the first predetermined similarity threshold: ， wherein, represents the actual two-dimensional joint histogram of the image within the target tracking frame corresponding to the current image frame, represents the actual two-dimensional joint histogram of the image within the target tracking frame corresponding to the previous image frame of the current image frame, represents the corrected two-dimensional joint histogram, and a is a predetermined constant.

2. The collaborative robotic target tracking method of claim 1, wherein, The adaptive filter is a spatial filter with a size of mxn, wherein m=2a+1, n=2b+1, a and b are both positive integers, and m and n are respectively smaller than the size of the target tracking frame in two image coordinate directions, and the two-dimensional joint probability graph after spatial filtering is represented as: , wherein, denotes the two-dimensional joint probability map after spatial filtering, denotes the two-dimensional joint probability map.

3. The method of claim 1, wherein, In step S6, after determining the target tracking box containing the predetermined moving target in the updated current image frame according to the optimized probability map based on the mean shift algorithm model for the first time, the target tracking box is reduced by a predetermined proportion while keeping the center coordinates of the target tracking box unchanged, and then jumping back to step S2 with the reduced target tracking box.

4. The method of claim 1, wherein, When performing step S5, after updating the current image frame with a next image frame of the current image frame, it is further analyzed and judged whether the predetermined moving target is missing in the current image frame, and if so, subsequent image frames of the current image frame are extracted in sequence and the predetermined moving target is searched for in each subsequent image frame based on a predetermined feature matching algorithm model, and the subsequent image frame in which the predetermined moving target is re-searched for is determined as the current image frame.

5. The method of claim 4, wherein, The analysis and judgment of whether the predetermined moving target is missing in the current image frame specifically comprises: determining actual similarity of images in the respective target tracking boxes corresponding to the current image frame and its previous image frame; calculating an actual image area ratio of images in the respective target tracking boxes corresponding to the current image frame and its previous image frame; and determining that the predetermined moving target is missing in the current image frame when the actual similarity and the actual image area ratio satisfy one of the following predetermined conditions: the actual similarity is greater than a second predetermined similarity threshold, and at the same time, the actual image area ratio is greater than a predetermined image area ratio; or the actual image area is less than an inverse of the predetermined image area ratio.

6. The method of claim 4, wherein, The predetermined feature matching algorithm model is a Brisk feature matching algorithm model.

7. The method of claim 1 or 5, wherein, The actual similarity is determined by calculating Bhattacharyya distance of two-dimensional joint histograms of images in the respective target tracking boxes corresponding to the current image frame and its previous image frame.

8. A cooperative robot target tracking device connected to a camera device of a cooperative robot, characterized by, The collaborative robot target tracking apparatus comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the collaborative robot target tracking method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to execute the collaborative robot target tracking method according to any one of claims 1 to 7 when the computer program runs.

Images