Target object tracking method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a target object tracking method, a target object tracking device, electronic equipment and a readable storage medium, and belongs to the technical field of computers. The target object tracking method comprises the steps that at least two target objects are provided, each target object comprises a mark pattern and a reference pattern which are located in the same plane, the reference patterns are close to the peripheries of the target objects, and the reference patterns of the target objects are different; acquiring an RGB image of the target object through a camera; identifying a reference pattern of the target object based on the RGB image of the target object; and determining a target object corresponding to the identified reference pattern of the target object based on the identified reference pattern of the target object. According to the embodiment of the invention, a plurality of target objects can be determined in one image.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a target tracking method, a target tracking device, an electronic device and a readable storage medium. Background Art

[0002] With the development of computer image processing technology, calibration plates play an increasingly important role in applications such as machine vision, image measurement, photogrammetry, and 3D reconstruction. For example, they can correct lens distortion, determine the conversion relationship between physical size and pixels, determine the relationship between the 3D geometric position of a point on the surface of a spatial object and its corresponding point in the image, and establish a geometric model of camera imaging.

[0003] However, currently most applications based on calibration plates are for the recognition of calibration patterns, such as squares, circles, or specified graphics. It is not possible to meet the scenario of simultaneously recognizing multiple calibration plates in the same image. For example, when the camera needs to track multiple similar objects at the same time or distinguish different surfaces of the same object (such as the surfaces in the front, back, left, right, up, and down directions), if the same calibration plate is used, the tracked objects or different surfaces of the same object cannot be distinguished; if different types of calibration plates are used, the maximum number of objects that can be tracked is limited. Summary of the invention

[0004] In order to solve at least one aspect of the above-mentioned problems and defects existing in the prior art, an embodiment of the present invention provides a target tracking method, a target tracking device, an electronic device and a readable storage medium to solve the problem of how to determine or identify multiple calibration plates (for example, at least three) in one image.

[0005] An object of the present invention is to provide a target tracking method.

[0006] Another object of the present invention is to provide a target tracking device.

[0007] Another object of the present invention is to provide an electronic device.

[0008] Another object of the present invention is to provide a readable storage medium.

[0009] According to one aspect of the present invention, a target tracking method is provided, comprising:

[0010] Providing at least two targets, each target comprising a marking pattern and a reference pattern located in the same plane, the reference pattern being close to the periphery of the target, and the reference pattern of each target being different;

[0011] Collect the RGB image of the target object through the camera;

[0012] Recognize a reference pattern of the target object based on an RGB image of the target object;

[0013] An object corresponding to the reference pattern of the identified object is determined based on the reference pattern of the identified object.

[0014] In some embodiments, the marking pattern of the target object is a regular pattern, wherein the regular pattern includes at least one of a circle, an ellipse, a rectangle, a square, a polygon, a triangle and ChArUco arranged in a regular pattern; or

[0015] The marking pattern of the target object is an irregular pattern, and the irregular pattern includes logos arranged in a regular pattern.

[0016] The reference pattern of the target object includes at least one of a QR code, a triangle, a rhombus, a five-pointed star, a six-pointed star and a cartoon graphic.

[0017] In some embodiments, identifying a reference pattern of a target object based on an RGB image of the target object includes:

[0018] Determine a grayscale image of the target object based on the RGB image of the target object;

[0019] Determine a binary image of the target object based on the grayscale image of the target object;

[0020] A reference pattern of the target object is recognized based on the binary image of the target object.

[0021] In some embodiments, identifying a reference pattern of a target object based on a binary image of the target object includes:

[0022] When the reference pattern is a two-dimensional code, different two-dimensional codes are distinguished by decoding the two-dimensional codes in the binary image.

[0023] When the reference pattern is at least one of a triangle, a rhombus, a five-pointed star, a six-pointed star and a cartoon figure,

[0024] providing a binary image of a reference pattern as a recognition template;

[0025] Providing a moving template, and moving the moving template on the binary image of the target object to obtain overlapping sub-images;

[0026] The reference pattern of the target object is recognized based on the overlapping sub-images and the recognition template.

[0027] In some embodiments, identifying a reference pattern of a target object based on overlapping sub-images and a recognition template includes:

[0028] Determining the matching degree between the overlapping sub-images and the recognition template;

[0029] When the matching degree indicates that the two are matched, the image in the overlapping sub-image is identified as a reference pattern of the target object.

[0030] In some embodiments, identifying a reference pattern of the target object based on the RGB image of the target object includes identifying an image of the reference pattern,

[0031] Determining a target object corresponding to the reference pattern of the identified target object based on the reference pattern of the identified target object includes:

[0032] Determine position information of a pixel point of a first center point of an image of the identified reference pattern;

[0033] Determine whether there is a target object in the search area with R as the domain based on each pixel point of the first center point;

[0034] When it is determined that there is a target object in the search area, extract image information from the recognized image in a search area centered on the pixel point of the first center point and with R as the area, and determine the image information as a target object corresponding to the reference pattern of the recognized target object,

[0035] The initial value of R is determined based on the target object and the camera.

[0036] In some embodiments, the initial value of R is determined by the following steps:

[0037] Placing at least one target object at a preset distance from the camera, and capturing an RGB image of the at least one target object through the camera;

[0038] Determine the position information of the target object in the captured RGB image of the target object;

[0039] determining a length and a width of the at least one target object based on the position information of the at least one target object;

[0040] An initial value of R is determined based on the length and width of the at least one target object.

[0041] In some embodiments, determining an initial value of R based on the length and width of the at least one target object comprises:

[0042] Determine a first pixel value corresponding to the length and a second pixel value corresponding to the width based on the length and the width of the at least one target object;

[0043] Determine a maximum value among the first pixel value and the second pixel value of all the target objects in the at least one target object;

[0044] The maximum value is determined as the initial value of R.

[0045] In some embodiments, determining the target object corresponding to the reference pattern of the identified target object based on the reference pattern of the identified target object further includes:

[0046] When it is determined that there is no target object in the search area, R is increased and a new search area is formed with the increased R as the area, and whether there is a target object in the new search area is re-determined based on the pixel points of each of the first center points until a target object corresponding to the reference pattern of the identified target object is identified,

[0047] The enlarged area R includes a gradual increase in size of 1 pixel.

[0048] In some embodiments, determining the target object corresponding to the reference pattern of the identified target object based on the reference pattern of the identified target object further includes:

[0049] Determine the position information of the second center point of the marking pattern of the identified target object in the image coordinate system;

[0050] Determining whether the identified target object is repeated based on the position information of the second center point of the marking pattern of the identified target object in the image coordinate system;

[0051] When there are duplicates among the identified objects, all the duplicate identified objects are deleted.

[0052] In some embodiments, the target tracking method further includes:

[0053] determining a first number of objects identified in the initial image of the objects;

[0054] Determining a second number of the target object in a next frame image of the target object;

[0055] A relationship between the first number and the second number is determined, and the target object is tracked based on the relationship.

[0056] In some embodiments, tracking the target object according to the relationship includes:

[0057] When the second number is less than or equal to the first number,

[0058] Determine the position data of each target object in the initial image in the image coordinate system and the position data of each target object in the next frame image in the image coordinate system;

[0059] Determine the distance between the position data of the target object in the initial image in the image coordinate system and the position data of the target object in the next frame image in the image coordinate system, and use the position data with the closest distance as the pairing data;

[0060] The position data of the target object in the next frame image in the image coordinate system is used to update the position data of the target object in the initial image paired with the position data of the target object in the next frame image in the image coordinate system, and the position data of the unpaired target object is deleted.

[0061] In some embodiments, tracking the target object according to the relationship includes:

[0062] When the second number is greater than the first number,

[0063] Recognize a reference pattern of the target object based on the next frame RGB image of the target object;

[0064] Determining a target object corresponding to the reference pattern of the identified target object based on the reference pattern of the identified target object;

[0065] Determine the position data of all targets in the next frame of RGB image in the image coordinate system and the number of targets.

[0066] According to another aspect of the present invention, a target tracking device is provided, the target tracking device is suitable for tracking at least two targets, each target includes a marking pattern and a reference pattern located in the same plane, the reference pattern is close to the periphery of the target, and the reference pattern of each target is different, the target tracking device includes:

[0067] A camera configured to capture an RGB image of a target object;

[0068] a recognition module in communication with the camera, the recognition module being configured to recognize a reference pattern of the target object based on an RGB image of the target object from the camera;

[0069] A tracking module is communicatively connected to the camera and the recognition module, respectively, and the tracking module is configured to determine a target corresponding to the reference pattern of the recognized target based on the reference pattern of the target recognized by the recognition module.

[0070] In some embodiments, the tracking module is further configured to determine a first number of targets in an initial image of the target and a second number of targets in a next frame of image, determine a relationship between the first number and the second number, and track the target according to the relationship.

[0071] According to another aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a program, wherein the processor implements the target tracking method described in any of the aforementioned embodiments when executing the program on the memory.

[0072] According to another aspect of the present invention, a readable storage medium is provided, in which a computer-readable program or instruction is stored. When the computer-readable program or instruction is executed by a processor, the target object tracking method described in any of the aforementioned embodiments is implemented.

[0073] The target tracking method, target tracking device, electronic device and readable storage medium according to the present invention have at least one of the following advantages:

[0074] (1) The target tracking method, target tracking device, electronic device, and readable storage medium of the present invention can simultaneously determine or identify targets that need to be distinguished by determining or identifying targets (such as calibration plates or tracked objects with calibration plates) provided with different reference patterns, thereby meeting the scenario of simultaneously identifying multiple targets (such as calibration plates) in the same image;

[0075] (2) The target tracking method, target tracking device, electronic device and readable storage medium of the present invention can meet the requirement for identifying multiple calibration plates by setting the number of different reference patterns, eliminating the requirement for the maximum number of trackable objects;

[0076] (3) The target tracking method, target tracking device, electronic device and readable storage medium of the present invention can realize dynamic tracking of multiple targets;

[0077] (4) The target tracking method, target tracking device, electronic device and readable storage medium of the present invention are real-time and robust in the determination or identification process of multiple calibration plates. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] These and / or other aspects and advantages of the present invention will become apparent and readily understood from the following description of the preferred embodiments in conjunction with the accompanying drawings, in which:

[0079] Figure 1 A flow chart of a target object tracking method according to an embodiment of the present invention is shown;

[0080] Figure 2 A flow chart of a target tracking method according to another embodiment of the present invention is shown;

[0081] Figure 3-6 Shows the application Figure 1 The target object of the target object tracking method shown;

[0082] Figure 7 A flowchart of identifying a reference pattern of a target object based on an RGB image of the target object according to an embodiment of the present invention is shown;

[0083] Figure 8A schematic diagram of dynamic tracking according to an embodiment of the present invention is shown;

[0084] Fig. 9 A target tracking device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0085] The technical solution of the present invention is further specifically described below by examples and in conjunction with the accompanying drawings. In the specification, the same or similar reference numerals indicate the same or similar components. The following description of the embodiments of the present invention with reference to the accompanying drawings is intended to explain the overall inventive concept of the present invention and should not be construed as a limitation of the present invention.

[0086] In an embodiment of the present invention, a target object tracking method is provided, which can simultaneously identify or distinguish multiple targets to achieve tracking of the targets.

[0087] The target object in the embodiment of the present invention includes an object that is expected to be tracked, such as a calibration plate, an object provided with a calibration plate, or different surfaces of the same object provided with a calibration plate.

[0088] like Figure 1 and Figure 2 As shown, the target tracking method includes:

[0089] Step S1 provides at least two targets, each target includes a marking pattern and a reference pattern located in the same plane, the reference pattern is close to the periphery of the target, and the reference pattern of each target is different;

[0090] Step S2 collects the RGB image of the target object through the camera;

[0091] Step S3 identifies a reference pattern of the target object based on the RGB image of the target object;

[0092] Step S4 determines the target object corresponding to the reference pattern of the identified target object based on the reference pattern of the identified target object.

[0093] The embodiments of the present invention simultaneously determine or identify targets that need to be distinguished by determining or identifying targets (such as calibration plates or tracked objects with calibration plates) with different reference patterns, thereby meeting the scenario of simultaneously identifying multiple targets (such as calibration plates) in the same image. For example, when tracking multiple objects through calibration plates, or when different areas on an object need to be distinguished through calibration plates, these scenarios require simultaneous identification of multiple calibration plates.

[0094] The embodiment of the present invention meets the requirement for identifying multiple calibration plates by setting the number of different reference patterns, thereby eliminating the requirement for the maximum number of trackable objects.

[0095] The embodiment of the present invention has real-time and robustness for the determination or identification process of multiple calibration plates.

[0096] It should be noted here that Figure 2 The target object tracking method is shown using a calibration plate as an example. However, the embodiments of the present invention are not limited thereto, and the target object may also be other objects that need to be tracked.

[0097] The embodiment of the present invention needs to provide at least two targets so that different targets can be distinguished by setting different reference patterns. Of course, the embodiment of the present invention does not limit the specific number of targets, and those skilled in the art can set it as needed.

[0098] Specifically, each target (e.g. Figure 3 The reference numeral 1 in the figure includes marking patterns located in the same plane (e.g. Figure 3 11 in the figure) and the reference pattern (eg Figure 3 12 in the figure).

[0099] The marking pattern of the target object is a regular pattern or an irregular pattern.

[0100] The regular graphics include at least one of a circle, an ellipse, a rectangle, a square, a polygon, a triangle, and ChArUco arranged in a regular pattern, such as Figure 3-6 shown.

[0101] Figure 3 An embodiment of the present invention is a checkerboard pattern. Figure 3 The pattern of the embodiment has the characteristics of high precision in the post-processing process, but it is necessary to ensure that the camera obtains a complete pattern image. Moreover, as the distance from the camera increases, the stability during the processing will decrease.

[0102] Figure 4 and Figure 5 The embodiment of the present invention is a circular pattern, and the difference between the two is that the arrangement rules of the circles are different. Figure 3 Compared to the example using Figure 4 and Figure 5 The pattern in the embodiment can improve the stability during the processing. Figure 4 and Figure 5 When you are viewing a pattern in the image, you also need to ensure that the camera obtains a complete image of the pattern.

[0103] Figure 6 An embodiment of is ChArUco. Figure 3-5 Compared with the pattern in the embodiment, Figure 6 The pattern of the embodiment allows processing of a portion of the image based on the pattern. That is, Figure 6The embodiment does not require that the pattern image obtained by the camera must be complete.

[0104] The irregular graphics include asymmetrical polygons or asymmetrical curved edges arranged in a regular pattern. For example, the irregular graphics include logos arranged in a regular pattern. For patterns using irregular graphics, the freedom and recognition of product design can be increased.

[0105] The reference patterns of the target include five-pointed stars (such as Figure 3 As shown), triangle (as Figure 4 As shown), QR code (as Figure 5 As shown), diamond (as Figure 6 The reference pattern is located near the periphery of the target object. For example, the reference pattern is located at one of the four corners of the target object (such as Figure 3 , 4 and 6), or the reference pattern is located at the upper periphery, or the reference pattern is located at the lower periphery (as shown in Figure 5 The embodiment of the present invention does not limit the specific position of the reference pattern, and those skilled in the art can set it as needed.

[0106] The embodiment of the present invention needs to ensure that the reference pattern of each target object is different so that different targets can be distinguished at the same time, and the reference pattern cannot be repeated with the pattern existing in the environment to avoid the problem of recognition failure. However, the embodiment of the present invention does not require that the marking pattern of each target object is different. After determining or selecting the marking pattern and the reference pattern for each target object, the target object is designed, that is, the position of the reference pattern on the target object is determined, and the shape of the marking pattern of each target object and the shape of the reference pattern are recorded for subsequent identification. The embodiment of the present invention does not limit the specific shapes of the marking pattern and the reference pattern, and those skilled in the art can set them as needed.

[0107] In one example, after the target object is designed, the target object can be printed as a sticker and pasted on a flat surface of the object, or printed on a metal surface or a plastic surface by laser.

[0108] After the design of the target is completed, the image stream or video stream of the target is obtained. The designed target is placed in the space, and the RGB image of the target is collected by the camera. The image at this time can be an image obtained in real time, or it can be a pre-stored image. The image can be in picture format or video format. This collection process can be triggered manually or automatically.

[0109] After acquiring the RGB image of the target object, the reference pattern is recognized. Specifically, Figure 7 As shown, step S3 includes:

[0110] S31 determines a grayscale image of the target object based on the RGB image of the target object;

[0111] S32 determines a binary image of the target object based on the grayscale image of the target object;

[0112] S33 recognizes a reference pattern of the target object based on the binary image of the target object.

[0113] A grayscale image is an image with only one sampled color per pixel. Such images are usually displayed as grayscales from black to white. There are at least multiple levels of color depth between black and white in a grayscale image. Converting an RGB image to a grayscale image can reduce the data that needs to be processed. The averaging method, the maximum-minimum-average method, the weighted average method, etc. can be used to convert an RGB image to a grayscale image. The averaging method is to average the RGB values ​​of the three channels of the same pixel. The maximum-minimum-average method is to average the values ​​with the highest brightness and the lowest brightness in the RGB at the same pixel position. The weighted average method is to use the three weighting coefficients of 0.3, 0.59, and 0.11 to calculate the weighted sum of the RGB values ​​of the three channels of the same pixel. These three weighting coefficients are parameters adjusted according to the human brightness perception system. Of course, those skilled in the art can use other known methods to implement the conversion process.

[0114] After obtaining the grayscale image, it is necessary to perform a binarization process on the grayscale image. The binarization process can set the grayscale value of the pixel on the image to 0 or 255. In this way, the entire image presents a visual effect of only black and white, so as to identify the reference pattern. The embodiments of the present invention can use the average method, the bimodal method, the Otsu algorithm (OTSU) and the like to obtain a binary image.

[0115] The recognition process of the reference pattern varies depending on the graphics of the reference pattern.

[0116] In one example, when the reference pattern is a two-dimensional code, step S33 includes decoding the two-dimensional code in the binary image to distinguish different two-dimensional codes. The reference pattern of each target object is different, and the graphics of the reference pattern of each target object are known. When the reference pattern is a two-dimensional code, the information that the two-dimensional code can be identified is known. By decoding the two-dimensional code in the binary image, the information of the identified two-dimensional code can be determined, so that the two-dimensional code can be identified based on the decoded information. The embodiments of the present invention can use existing technology for decoding, which is not described in detail here.

[0117] In one example, when the reference pattern is at least one of a triangle, a rhombus, a five-pointed star, a six-pointed star, and a cartoon figure, a template matching algorithm may be used to identify the reference pattern. Specifically, step S33 includes:

[0118] S331 provides a binary image of a reference pattern as a recognition template;

[0119] S332 provides a moving template, and moves the moving template on the binary image of the target object to obtain overlapping sub-images;

[0120] S333 recognizes a reference pattern of the target object based on the overlapping sub-images and the recognition template.

[0121] In step S331, a binary image of the reference pattern is set based on the image of the reference pattern. For example, a five-pointed star is used as the reference pattern, the inside of the five-pointed star pattern is set to black, and the outside is set to white, and this image forms a recognition template.

[0122] In step S332, the moving template includes a rectangular frame. In one example, the rectangular frame is a rectangular frame representing pixel units, such as 3*3, 3*6, 3*9, 6*3, 9*3, etc. In one example, the rectangular frame is a rectangular frame obtained by scaling the rectangular frame representing pixel units, such as 6*6, 6*12, 6*18, 12*6, 18*6, etc.

[0123] The moving template is moved on the binary image obtained in step S32, and the image framed by the rectangular frame on the binary image constitutes an overlapping sub-image. In one example, the moving template can be moved from left to right and from top to bottom in pixels. Of course, those skilled in the art can set a specific moving method as needed.

[0124] Step S333 includes: determining the degree of matching between the overlapping sub-image and the recognition template; when the degree of matching indicates that the two are matched, identifying the image in the overlapping sub-image as the reference pattern of the target object. That is, the reference pattern for identifying the target object includes the image of the recognition reference pattern (overlapping sub-image).

[0125] The greater the degree of matching, the greater the possibility that the overlapping sub-image and the recognition template are the same. The method for determining the degree of matching can be to use the square difference for matching, and the result value of 0 indicates the highest degree of matching between the two, and the larger the value of the result, the lower the degree of matching between the two. The method for determining the degree of matching can also be to use the multiplication operation between the recognition template and the overlapping sub-image, and the larger the value of the result, the better the degree of matching, and the value of the result is 0, which indicates the lowest degree of matching. The method for determining the degree of matching can also be to match the relative value of the recognition template to its mean with the correlation value of the overlapping sub-image to its mean, and the result value of 1 indicates the highest degree of matching, the result value of -1 indicates the lowest degree of matching, and the result value of 0 indicates that there is no correlation (random sequence).

[0126] In one example, step S3 further includes:

[0127] S34 determines the position information of the pixel point of the first center point of the image of the reference pattern identified in step S33.

[0128] After the reference pattern is identified in the overlapping sub-image, the first center point of the image of the reference pattern can be determined; the pixel point of the first center point can be determined based on the first center point; the position coordinates in the image coordinate system can be determined based on the pixel point of the first center point. The position coordinates can be used for subsequent target object extraction.

[0129] After the reference pattern is identified, the target object corresponding to the reference pattern is extracted. Specifically, step S4 includes:

[0130] S41 determines whether there is a target object in the search area with R as the domain based on each pixel point of the first center point (determined by step S34); when it is determined that there is a target object in the search area, the image information in the identified image is extracted from the search area with the pixel point of the first center point as the center and R as the domain, and the image information is determined as a target object corresponding to the reference pattern of the identified target object.

[0131] The pixel point of the first center point can be used as the center of the search area to determine whether there is a target object in the image of the reference pattern framed by the search area. For example, when the target object is a calibration plate, a calibration plate recognition method can be used to check whether there is a pattern of the calibration plate in the selected area.

[0132] When it is determined that there is a target object in the search area, the image information in the identified image is extracted from the search area with the pixel point of the first center point as the center and R as the area, and the image information is determined as a target object corresponding to the reference pattern of the identified target object. After the target object is identified, the position information of the second center point of the marking pattern of the identified target object in the image coordinate system can be determined. For example, the position information includes the coordinates of the pixel point of the second center point in the image coordinate system.

[0133] When it is determined that there is no target in the search area, R is increased and a new search area is formed with the increased R as the area, and whether there is a target in the new search area is re-determined based on the pixel points of each of the first center points until a target corresponding to the reference pattern of the identified target is identified. In one example, increasing the area R includes gradually increasing by 1 pixel.

[0134] In one example, the initial value of R is set to 1 pixel, 2 pixels, or more pixels.

[0135] In one example, the initial value of R is determined based on the target object and the camera. Specifically, the initial value of R is determined by the following steps:

[0136] Placing at least one target object at a preset distance from the camera, and capturing an RGB image of the at least one target object through the camera;

[0137] Determine the position information of the target object in the captured RGB image of the target object;

[0138] determining a length and a width of the at least one target object based on the position information of the at least one target object;

[0139] An initial value of R is determined based on the length and width of the at least one target object.

[0140] The at least one target described here is at least one target of the at least two targets provided. In one example, when the marking pattern of each target in the at least two targets is different, the at least one target can be all the targets in the at least two targets, or some of the targets in the at least two targets. Of course, when the at least one target is all the targets, the determined initial value of R will be more accurate.

[0141] When some of the at least two targets have the same marking pattern, the initial value of R can be determined for the targets with different marking patterns. That is, when there are M targets and the M targets have N different marking patterns (M>N>0, M and N are both positive integers), the number of at least one target can be N, or any positive integer less than N.

[0142] The preset distance is the maximum distance between the camera and the target object in the application scenario, and the camera can still clearly capture the target object at the maximum distance. The embodiment of the present invention does not limit the specific value of the preset distance, and those skilled in the art can set it as needed.

[0143] According to the existing method, the position information of the target object can be identified in the RGB image of the captured target object. The position information of the target object includes the coordinates of the pixel points of the marking pattern of the target object in the image coordinate system.

[0144] The length and width of the target object can be determined based on the position information of the target object. For example, the length and width of the target object can be determined based on the coordinates of the pixel points of the marking pattern in the image coordinate system.

[0145] An initial value of R is determined based on the length and width of the at least one target object. This process includes:

[0146] Determine a first pixel value corresponding to the length and a second pixel value corresponding to the width based on the length and the width of the at least one target object;

[0147] Determine a maximum value among the first pixel value and the second pixel value of all the target objects in the at least one target object;

[0148] The maximum value is determined as the initial value of R.

[0149] The process of determining the length and width of the target object includes determining a first pixel value corresponding to the length and a second pixel value corresponding to the width. The first pixel value and the second pixel value of all the targets in at least one target object are compared respectively to determine the maximum value of the first pixel value and the second pixel value. The maximum value is determined as the initial value of R.

[0150] In some cases, in the recognized image, the reference pattern P of the first target is closer to the second target. During the recognition process, the second target may be recognized based on the reference pattern P, and the second target may also be recognized based on the reference pattern Q of the second target. In this case, inaccurate recognition may occur, which is not conducive to the subsequent tracking process. In view of this, the embodiment of the present invention can determine whether the recognition result is correct. Specifically, step S4 includes:

[0151] S42 determines whether the identified target objects are repeated based on the position of the second center point of the marking pattern of the identified target object in the image coordinate system; if there are repeated identified targets, delete all the repeated identified targets.

[0152] As mentioned above, after the target object is identified, the coordinates of the pixel point of the second center point of the target object's marking pattern in the image coordinate system can be recorded. It is determined whether the coordinates of the second center point of the marking pattern of the identified target object are the same. When there are at least two targets whose second center point coordinates of the marking pattern are the same, it is considered that the identified target objects are repeated, and all the repeated identified targets are deleted. It can be determined whether the coordinates of the second center point of the marking pattern of the target object are the same based on the existing method, and the embodiments of the present invention are not limited to this.

[0153] In one example, step S4 further includes:

[0154] In the result after processing in step S42, S43 takes the coordinates of the pixel point of the first center point recorded in step S34 as the center and R calculated in S41 as the area, extracts the image information at this time, and obtains the image information of the designed target object, and records the coordinate information of the pixel point of the second center point of the marking pattern of the identified target object in the image coordinate system for subsequent tracking algorithms.

[0155] In another example of the present invention, Figure 2 As shown, the target object tracking method also includes:

[0156] determining a first number of objects identified in the initial image of the objects;

[0157] Determining a second number of the target object in a next frame image of the target object;

[0158] A relationship between the first number and the second number is determined, and the target object is tracked based on the relationship.

[0159] In this way, the embodiment of the present invention realizes a dynamic tracking process by tracking the current target object according to the relationship between the first number and the second number.

[0160] After the target objects in the initial image are determined or recognized according to the aforementioned embodiment, the first number of the determined or recognized target objects may be known.

[0161] For the determination or recognition process of the target in the next frame of image, the target is not recognized based on the difference of the reference pattern (such as the solution of the aforementioned embodiment), but is recognized based on the marking pattern of the target. For example, feature points can be set on the target, and the target is recognized based on the spatial data of the feature points in the pattern coordinate system, the intrinsic parameter data of the camera, the world coordinate system and the camera coordinate system. Of course, the embodiments of the present disclosure do not limit the specific recognition process, and those skilled in the art can also use other disclosed technologies to identify the number of targets in the next frame of image.

[0162] The target object is tracked by comparing the relationship (eg, size relationship) between the first number and the second number.

[0163] Specifically, when the second number is less than or equal to the first number, the target object determined in the next frame image has been determined or recognized in the initial image, and includes the following steps:

[0164] Determine the position data of each target object in the initial image in the image coordinate system and the position data of each target object in the next frame image in the image coordinate system;

[0165] Determine the distance between the position data of the target object in the initial image in the image coordinate system and the position data of the target object in the next frame image in the image coordinate system, and use the position data with the closest distance as the pairing data;

[0166] The position data of the target object in the next frame image in the image coordinate system is used to update the position data of the target object in the initial image paired with the position data of the target object in the next frame image in the image coordinate system, and the position data of the unpaired target object is deleted.

[0167] The position data of the target object includes the coordinate data of the second center point of the marking pattern of the target object in the image coordinate system. After the target object is determined, the coordinate value of the area where the marking pattern is located in the image coordinate system can be determined, such as the O-XY rectangular coordinate system; the coordinate value of the second center point of the area where the marking pattern is located can be determined, x mid =(x max +x min ) / 2,y mid =(y max +y min ) / 2,x max is the maximum coordinate value of the region on the x-axis of the image coordinate system, x min is the minimum coordinate value of the region on the x-axis of the image coordinate system, and y max is the maximum coordinate value of the region on the y-axis of the image coordinate system, y min is the minimum coordinate value of the region on the y-axis of the image coordinate system; the coordinate value of the second center point of the region is determined as the position data of the target object in the image coordinate system. In this way, the position data of each target object in the initial image in the image coordinate system and the position data of each target object in the next frame of image in the image coordinate system can be determined.

[0168] After determining the position data of the target object in the initial image and the position data of the target object in the next frame of image, determine the distance (e.g., Euclidean distance) between the position data in the two images. The Euclidean distance between the center points of the target areas in the two images can be calculated respectively. The data of the position with the closest distance (i.e., the smallest Euclidean distance value) is determined as the paired data.

[0169] For the paired data, the position data of the target object in the next frame image in the pairing is used to update the position data of the target object in the initial image in the pairing, and the position data of the unpaired target object is deleted, thereby realizing dynamic tracking of the target object.

[0170] by Figure 8 For example, three targets A, B and C are identified in the initial image, and the first number of targets in the initial image is three; three targets A', B' and C' are identified in the next frame image, and the second number of targets in the next frame image is three, and the first number is equal to the second number; the Euclidean distance between the second center point of A and the second center point of each target (A', B' and C') in the next frame image is determined, which can be expressed as AA', AB' and AC' respectively; the Euclidean distance between the second center point of B and the second center point of each target in the next frame image is determined, which can be expressed as BA', BB' and BC' respectively; the Euclidean distance between the second center point of C and the second center point of each target in the next frame image is determined, which can be expressed as CA', CB' and CC' respectively. Figure 8 In the example, AA' is the shortest among AA', AB' and AC', so the target A and A' are paired data, and the position data of A is updated by using the position data of A; BB' is the shortest among BA', BB' and BC', so the target B and B' are paired data, and the position data of B is updated by using the position data of B; CC' is the shortest among CA', CB' and CC', so the target C and C' are paired data, and the position data of C is updated by using the position data of C', thereby completing the tracking of the target.

[0171] When the second number is greater than the first number, a target object that is not determined or identified in the initial image appears in the next frame image, so it is necessary to identify the target object based on the difference in the reference pattern according to the method of the above embodiment. Specifically, the following process is included:

[0172] Identify the reference pattern of the target object based on the next frame RGB image of the target object. This process is referred to in the above content and will not be repeated here;

[0173] Determine the target object corresponding to the reference pattern of the identified target object based on the reference pattern of the identified target object. This process is referred to the above content and will not be described in detail here;

[0174] Determine the position data of all targets in the next frame of RGB image in the image coordinate system and the number of targets.

[0175] After all the targets in the next frame image are identified based on the reference pattern, it is necessary to determine the position data of all the targets in the next frame image and the number of the targets for use in subsequent dynamic tracking.

[0176] In this way, the image updated later can use the above method to update the position data of the target object, thereby completing the dynamic tracking of the target object.

[0177] In an embodiment of the present invention, a target tracking device is provided, which is suitable for tracking at least two targets. Each target includes a marking pattern and a reference pattern located in the same plane, the reference pattern is close to the periphery of the target, and the reference pattern of each target is different. The target tracking device can execute the target tracking method described in any of the above embodiments.

[0178] like Fig. 9 As shown, the target tracking device 100 includes a camera 10 , a recognition module 20 and a tracking module 30 .

[0179] The camera 10 is a camera capable of acquiring RGB images of a target object, and may include a monocular camera, a binocular camera, and the like.

[0180] The recognition module 20 is in communication with the camera 10. The recognition module 20 is configured to recognize a reference pattern of the target object based on the RGB image of the target object from the camera 10. The specific process of recognizing the reference pattern can be referred to the above embodiment, and will not be described in detail here.

[0181] The tracking module 30 is respectively connected to the camera 10 and the recognition module 20. The tracking module 30 is configured to determine a target corresponding to the reference pattern of the recognized target based on the reference pattern of the target recognized by the recognition module 20. The specific process of determining the target can be referred to the above embodiment, and will not be repeated here.

[0182] In one example, the tracking module 30 is further configured to determine a first number of targets in an initial image of the target and a second number of targets in a next frame of image, determine a relationship between the first number and the second number, and track the target according to the relationship. The specific process of tracking the target can be referred to the above embodiment, and will not be repeated here.

[0183] In an embodiment of the present invention, a readable storage medium is provided, wherein the readable storage medium stores a program or an instruction, and when the program or the instruction is executed by a processor, the target object tracking method described in any of the above embodiments is implemented.

[0184] The "readable storage medium" of an embodiment of the present invention refers to any medium that participates in providing a program or instruction to a processor for execution. The medium can take a variety of forms, including but not limited to non-volatile media, volatile media and transmission media. Non-volatile media include, for example, optical disks or disks, such as storage devices. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wires and optical fibers, including wires containing buses. Transmission media can also take the form of sound waves or light waves, such as sound waves or light waves generated during radio frequency (RF) and infrared (IR) data communications. Common forms of readable storage media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punch cards, paper tapes, any other physical media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or boxes, carriers as described below, or any other media from which a computer can read.

[0185] In an embodiment of the present invention, a cloud server is also provided. The cloud server includes a memory and a processor. The processor may be a central processing unit (CPU). The memory stores programs or instructions, and when the programs or instructions are executed by the processor, the above-mentioned target tracking method is executed. In one example, the cloud server may be a virtual server formed by mapping a physical server through virtualization technology. Among them, there may be one or more physical servers. When there are multiple physical servers, the cloud server may be a virtual server formed by mapping a server cluster through virtualization technology. The virtual server may also be one or more. In one example, the cloud server can be provided to users through a cloud platform. In one example, the memory may be a readable storage medium.

[0186] In an embodiment of the present invention, an electronic device is also provided. The electronic device (not shown) includes a processor (not shown) and a memory (not shown). The memory stores a program, and when the program is executed by the processor, any of the target tracking methods in the above examples can be implemented.

[0187] In one example, the processor may be a microprocessor, such as a general-purpose processor such as a graphics processing unit (GPU), a central processing unit (CPU), a digital signal processor (DSP), etc. In one example, the processor may also be a microprocessor core implemented by a hardware circuit, such as a microprocessor core implemented in a hardware logic component by a reconfigurable logic, and the hardware logic component includes a field programmable gate array (FPGA), a complex programmable logic device (CPLD), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system on a chip (SOC), etc.

[0188] In one example, the processor may also be a virtual processor, which may be a virtual processor with the characteristics of an Intel x86 processor, or a virtual processor with the characteristics of a PowerPC processor. Preferably, the processor is a graphics processor. In one example, the processor may be a single-core processor or a multi-core processor.

[0189] In one example, the memory includes a volatile memory (i.e., a random access memory) and a non-volatile memory. The volatile memory includes a main memory, a cache, etc., and the non-volatile memory includes an auxiliary memory, etc. In one example, the memory can be set as a remote memory, and the remote memory can be connected to the processor via a network (wired network or wireless network). The network includes, but is not limited to, a wide area network, a local area network, a metropolitan area network, a personal area network, the Internet, a satellite communication network, and any combination thereof.

[0190] In one example, the processor executes a program based on the program obtained from the memory to create a corresponding task thread and execute the thread. In one example, the processor obtains a program from the external memory based on the read instruction in the memory to create a corresponding task thread and execute the thread. The above program is used to implement a control method for tracking a target object.

[0191] Although the subject matter described herein is provided in the general context of being executed in conjunction with the execution of an operating system and an application program on a computer system, it will be appreciated by those skilled in the art that it can also be implemented in conjunction with other types of program modules. Generally speaking, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. It will be appreciated by those skilled in the art that the method steps described in conjunction with any one of the examples herein can all be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether to perform in hardware or software mode depends mainly on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0192] When the method steps are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Therefore, the technical solution of the present invention, or the part that contributes to the original technology, or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each example of the present invention.

[0193] Although some embodiments of the present general inventive concept have been shown and described, it will be appreciated by those skilled in the art that changes may be made to these embodiments without departing from the principles and spirit of the present general inventive concept, the scope of which is defined by the claims and their equivalents.

Claims

1. A target tracking method, comprising: Providing at least two targets, each target comprising a marking pattern and a reference pattern located in the same plane, the reference pattern being close to the periphery of the target, and the reference pattern of each target being different; Collect the RGB image of the target object through the camera; Recognize a reference pattern of the target object based on an RGB image of the target object; An object corresponding to the reference pattern of the identified object is determined based on the reference pattern of the identified object.

2. The target tracking method according to claim 1, wherein: The marking pattern of the target object is a regular shape. The regular graphics include at least one of a circle, an ellipse, a rectangle, a square, a polygon, a triangle, and ChArUco arranged in a regular pattern; or The marking pattern of the target object is an irregular pattern, and the irregular pattern includes logos arranged in a regular pattern. The reference pattern of the target object includes at least one of a QR code, a triangle, a rhombus, a five-pointed star, a six-pointed star and a cartoon graphic.

3. The target tracking method according to claim 2, wherein: Reference patterns for identifying a target based on its RGB image include: Determine a grayscale image of the target object based on the RGB image of the target object; Determine a binary image of the target object based on the grayscale image of the target object; A reference pattern of the target object is recognized based on the binary image of the target object.

4. The target tracking method according to claim 3, wherein: Reference patterns for identifying a target based on its binary image include: When the reference pattern is a two-dimensional code, different two-dimensional codes are distinguished by decoding the two-dimensional codes in the binary image. When the reference pattern is at least one of a triangle, a rhombus, a five-pointed star, a six-pointed star and a cartoon figure, providing a binary image of a reference pattern as a recognition template; Providing a moving template, and moving the moving template on the binary image of the target object to obtain overlapping sub-images; The reference pattern of the target object is recognized based on the overlapping sub-images and the recognition template.

5. The target tracking method according to claim 4, wherein: Reference patterns for identifying targets based on overlapping sub-images and recognition templates include: Determining the matching degree between the overlapping sub-images and the recognition template; When the matching degree indicates that the two are matched, the image in the overlapping sub-image is identified as a reference pattern of the target object.

6. The target tracking method according to any one of claims 1 to 5, wherein: Recognizing a reference pattern of a target object based on an RGB image of the target object includes recognizing an image of the reference pattern, Determining a target object corresponding to the reference pattern of the identified target object based on the reference pattern of the identified target object includes: Determine position information of a pixel point of a first center point of an image of the identified reference pattern; Determine whether there is a target object in the search area with R as the domain based on each pixel point of the first center point; When it is determined that there is a target object in the search area, extract image information from the recognized image in a search area centered on the pixel point of the first center point and with R as the area, and determine the image information as a target object corresponding to the reference pattern of the recognized target object, The initial value of R is determined based on the target object and the camera.

7. The target tracking method according to claim 6, wherein: The initial value of R is determined by the following steps: Placing at least one target object at a preset distance from the camera, and capturing an RGB image of the at least one target object through the camera; Determine the position information of the target object in the captured RGB image of the target object; determining a length and a width of the at least one target object based on the position information of the at least one target object; An initial value of R is determined based on the length and width of the at least one target object.

8. The target tracking method according to claim 7, wherein: Determining an initial value of R based on the length and width of the at least one target object includes: Determine a first pixel value corresponding to the length and a second pixel value corresponding to the width based on the length and the width of the at least one target object; Determine a maximum value among the first pixel value and the second pixel value of all the target objects in the at least one target object; The maximum value is determined as the initial value of R.

9. The target tracking method according to claim 8, wherein: Determining the target object corresponding to the reference pattern of the identified target object based on the reference pattern of the identified target object further includes: When it is determined that there is no target object in the search area, R is increased and a new search area is formed with the increased R as the area, and whether there is a target object in the new search area is re-determined based on the pixel points of each of the first center points until a target object corresponding to the reference pattern of the identified target object is identified, The enlarged area R includes a gradual increase in size of 1 pixel.

10. The target tracking method according to claim 9, wherein: Determining the target object corresponding to the reference pattern of the identified target object based on the reference pattern of the identified target object further includes: Determine the position information of the second center point of the marking pattern of the identified target object in the image coordinate system; Determining whether the identified target object is repeated based on the position information of the second center point of the marking pattern of the identified target object in the image coordinate system; When there are duplicates among the identified objects, all the duplicate identified objects are deleted.

11. The target tracking method according to claim 10, wherein: The target object tracking method further includes: determining a first number of objects identified in the initial image of the objects; Determining a second number of the target object in a next frame image of the target object; A relationship between the first number and the second number is determined, and the target object is tracked based on the relationship.

12. The target tracking method according to claim 11, wherein: Tracking targets according to the relationship includes: When the second number is less than or equal to the first number, Determine the position data of each target object in the initial image in the image coordinate system and the position data of each target object in the next frame image in the image coordinate system; Determine the distance between the position data of the target object in the initial image in the image coordinate system and the position data of the target object in the next frame image in the image coordinate system, and use the position data with the closest distance as the pairing data; The position data of the target object in the next frame image in the image coordinate system is used to update the position data of the target object in the initial image matched with the position data of the target object in the next frame image in the image coordinate system, and the position data of the unmatched target object is deleted.

13. The target tracking method according to claim 12, wherein: Tracking targets according to the relationship includes: When the second number is greater than the first number, Recognize a reference pattern of the target object based on the next frame RGB image of the target object; Determining a target object corresponding to the reference pattern of the identified target object based on the reference pattern of the identified target object; Determine the position data of all targets in the next frame of RGB image in the image coordinate system and the number of targets.

14. A target tracking device, characterized in that: The target tracking device is suitable for tracking at least two targets, each target includes a marking pattern and a reference pattern located in the same plane, the reference pattern is close to the periphery of the target, and the reference pattern of each target is different, and the target tracking device includes: A camera configured to capture an RGB image of a target object; a recognition module in communication with the camera, the recognition module being configured to recognize a reference pattern of the target object based on an RGB image of the target object from the camera; A tracking module is communicatively connected to the camera and the recognition module, respectively, and the tracking module is configured to determine a target corresponding to the reference pattern of the recognized target based on the reference pattern of the target recognized by the recognition module.

15. The target tracking device according to claim 14, characterized in that: The tracking module is also configured to determine a first number of targets in an initial image of the target and a second number of targets in a next frame of image, determine a relationship between the first number and the second number, and track the target according to the relationship.

16. An electronic device, characterized in that: The electronic device comprises: A memory and a processor, wherein the memory stores a program, wherein the processor implements the target tracking method according to any one of claims 1-13 when executing the program on the memory.

17. A readable storage medium, characterized in that: The readable storage medium stores a computer-readable program or instruction, and when the computer-readable program or instruction is executed by a processor, the target tracking method according to any one of claims 1-13 is implemented.

Images