Target tracking control method and device based on image recognition and camera

Abstract

The invention provides a target tracking control method and device based on image recognition and a camera, and relates to the technical field of automatic control. The method comprises the steps of performing color recognition on a target image through an adaptive color recognition algorithm, recognizing a color region and calculating a color center coordinate; identifying the target image through a pre-trained YOLOv2 target detection model, and calculating object center coordinates according to the identified object area; and calculating a tracking center coordinate according to the color center coordinate and the object center coordinate, converting the tracking center coordinate into a motion control signal to drive the visual sensor to rotate, calculating an actual rotation angle of the visual sensor through a PID control algorithm, and correcting the rotated visual sensor according to a calculation result to realize target tracking. The central position of the target is found through color recognition and object recognition, and the visual sensor is driven to rotate to track the target, so that the target is located at the central position of the image in real time.

Description

Technical Field

[0001] The present invention mainly relates to the field of automation control technology, and specifically relates to a target tracking control method, device and camera based on image recognition. Background Art

[0002] When tracking a target, the main thing is to accurately find the characteristics of the target in the target scene, and then locate the position of the target based on its characteristics. Usually, precision sensor technology is used for target detection and tracking. During the movement, the tracking center and the target will be offset, and the error is large, which makes it very easy to lose the target during tracking. In other words, the accuracy and timeliness of this technology are very low, and the cost of precision sensors is high. Therefore, while the target tracking technology is developing rapidly, higher requirements are placed on its stability and accurate tracking performance in motion. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a target tracking control method, device and camera based on image recognition in view of the deficiencies in the prior art.

[0004] The technical solution of the present invention to solve the above technical problems is as follows:

[0005] A target tracking control method based on image recognition comprises the following steps:

[0006] Performing color recognition on the target image collected by the visual sensor through an adaptive color recognition algorithm to obtain a color area, and performing center calculation on the color area to obtain color center coordinates;

[0007] Performing object recognition on the target image through a pre-trained YOLOv2 target detection model to obtain an object region, performing center calculation on the object region to obtain the object center coordinates;

[0008] The color center coordinates and the object center coordinates are averaged to obtain the tracking center coordinates, which are converted into motion control signals. The visual sensor is driven to rotate according to the motion control signals, and the actual rotation angle of the visual sensor is read. The actual rotation angle is calculated through a PID control algorithm, and the rotation of the visual sensor is corrected according to the calculation result to achieve target tracking.

[0009] Furthermore, the color recognition of the target image collected by the visual sensor is performed by the adaptive color recognition algorithm to obtain the color area, specifically:

[0010] S1. Obtain LAB threshold;

[0011] S2, performing color recognition on the target image collected by the visual sensor according to the LAB threshold through the color recognition function, and determining whether a color area corresponding to the threshold is recognized, if so, obtaining the color area, if not, executing S3;

[0012] S3, adjusting the parameters of the LAB threshold through the set adaptive adjustment scheme to obtain an updated LAB threshold, and executing S4;

[0013] S4, performing color recognition on the target image according to the updated LAB threshold through a color recognition function, and determining whether a color region corresponding to the threshold is recognized, if so, obtaining the color region, if not, executing S3 until the color region is obtained.

[0014] Further, the adaptive adjustment scheme includes a brightness adjustment scheme, a first color adjustment scheme and a second color adjustment scheme;

[0015] The S3 includes:

[0016] S31, adjusting the second color parameter B of the LAB threshold by using the second color adjustment scheme, and when the color recognition function cannot recognize the color area corresponding to the threshold according to the adjusted LAB threshold, executing the first color adjustment scheme;

[0017] S32, adjusting the first color parameter A and the second color parameter B of the LAB threshold by using the first color adjustment scheme, and executing the brightness adjustment scheme when the color recognition function cannot recognize the color area corresponding to the threshold according to the adjusted LAB threshold;

[0018] S33: Adjust the brightness parameter L of the LAB threshold, the first color parameter A, and the second color parameter B by using the brightness adjustment scheme.

[0019] Further, in S31, adjusting the second color parameter B of the LAB threshold by using the second color adjustment scheme includes:

[0020] The brightness parameter L and the first color parameter A are fixed, and the maximum value and / or the minimum value of the second color parameter B are adjusted multiple times, and an updated LAB threshold is obtained according to the brightness parameter L, the first color parameter A and the corresponding adjusted second color parameter B;

[0021] In S32, adjusting the first color parameter A and the second color parameter B of the LAB threshold by using the first color adjustment scheme includes:

[0022] The brightness parameter L is fixed, and the maximum value and / or the minimum value of the first color parameter A are adjusted multiple times. When the maximum value and / or the minimum value of the first color parameter A are adjusted each time, the maximum value and / or the minimum value of the second color parameter B are adjusted multiple times, and an updated LAB threshold is obtained according to the brightness parameter L, the corresponding adjusted first color parameter A, and the corresponding adjusted second color parameter B;

[0023] The S33 comprises:

[0024] The maximum value and / or minimum value of the brightness parameter L is adjusted multiple times respectively. Each time the maximum value and / or minimum value of the brightness parameter L is adjusted once, the maximum value and / or minimum value of the first color parameter A is adjusted multiple times respectively. Each time the maximum value and / or minimum value of the first color parameter A is adjusted once, the maximum value and / or minimum value of the second color parameter B is adjusted multiple times respectively. The updated LAB threshold is obtained from the corresponding adjusted brightness parameter L, the corresponding adjusted first color parameter A and the corresponding adjusted second color parameter B.

[0025] Further, the adjusting the maximum value and / or the minimum value of the second color parameter B for multiple times respectively includes:

[0026] Fixing the maximum value of the second color parameter B and adjusting the minimum value of the second color parameter B include:

[0027] Determine whether the minimum value reaches a first preset minimum threshold value, if not, reduce the minimum value of the second color parameter B according to the set value, obtain an updated LAB threshold value by the maximum value of the brightness parameter L, the first color parameter A, the second color parameter B and the corresponding adjusted minimum value of the second color parameter B, and when the color recognition function cannot recognize the color area corresponding to the threshold value according to the updated LAB threshold value, repeat the operation of adjusting the minimum value of the second color parameter B;

[0028] If yes, then adjusting the maximum value of the second color parameter B includes:

[0029] Determine whether the maximum value reaches a first preset maximum threshold value. If not, increase the maximum value of the second color parameter B according to the set value, and perform an operation of adjusting the minimum value of the second color parameter B. The updated LAB threshold value is obtained by the brightness parameter L, the first color parameter A, and the corresponding maximum and minimum values ​​of the adjusted second color parameter B. When the color recognition function cannot identify the color area corresponding to the threshold value according to the updated LAB threshold value, repeat the operation of adjusting the maximum value of the second color parameter B.

[0030] Furthermore, the color region is a pixel matrix composed of a plurality of pixel points;

[0031] The center of the color area is calculated to obtain the color center coordinates, specifically:

[0032] The maximum and minimum values ​​of the horizontal coordinates of multiple pixel points in the pixel matrix are averaged to obtain the central horizontal coordinate, the maximum and minimum values ​​of the vertical coordinates of multiple pixel points in the pixel matrix are averaged to obtain the central vertical coordinate, and the color center coordinate is obtained from the central horizontal coordinate and the central vertical coordinate.

[0033] Furthermore, the converting the tracking center coordinates into a motion control signal and driving the visual sensor to rotate according to the motion control signal is specifically as follows:

[0034] Acquire the image center coordinates of the target image, calculate the difference between the image center coordinates and the tracking center coordinates, and obtain the coordinate difference;

[0035] The coordinate difference is converted into a pulse width modulation signal of a set frequency and a direction control signal, the rotation speed of the visual sensor is controlled according to the pulse width modulation signal, and the rotation angle of the visual sensor is controlled according to the direction control signal.

[0036] Another technical solution of the present invention to solve the above technical problems is as follows:

[0037] A target tracking control device based on image recognition, comprising:

[0038] A coordinate operator, connected to a visual sensor for outputting a target image, for performing color recognition on the target image acquired by the visual sensor through an adaptive color recognition algorithm to obtain a color area, performing center calculation on the color area to obtain color center coordinates, performing object recognition on the target image through a pre-trained YOLOv2 target detection model to obtain an object area, performing center calculation on the object area to obtain object center coordinates;

[0039] A tracking controller is connected after the coordinate operator, and is used to calculate the mean of the color center coordinates and the object center coordinates to obtain the tracking center coordinates, convert the tracking center coordinates into a motion control signal, drive the visual sensor to rotate according to the motion control signal, and read the actual rotation angle of the visual sensor, calculate the actual rotation angle through a PID control algorithm, and correct the rotation of the visual sensor after rotation according to the calculation result.

[0040] Another technical solution of the present invention to solve the above technical problems is as follows:

[0041] A camera comprises a visual sensor and a magnetic encoder respectively connected to a two-dimensional pan-tilt platform, and a target tracking control device for image recognition as described above, wherein the device is used to execute the target tracking control method for image recognition as described above.

[0042] Furthermore, a two-dimensional pan-tilt platform is constructed by a plurality of stepper motors, and the two-dimensional pan-tilt platform is driven to rotate the visual sensor according to the motion control signal, and the actual rotation angle of the two-dimensional pan-tilt platform is read by a magnetic encoder.

[0043] The beneficial effects of the present invention are as follows: by adaptively debugging the target image through an adaptive color recognition algorithm, the target color to be recognized can be continuously approached, and the coordinates of the target color features can be obtained. Then, the shape of the object in the target image is detected through a target detection model, and the target shape can be found in a large number of objects, and the coordinates of the target shape features can be obtained. The coordinate recognition accuracy of the target object is improved according to color recognition and object recognition, and the PID algorithm is used to perform closed-loop control on the stepper motor of the two-dimensional pan-tilt head, that is, after controlling the rotation of the visual sensor according to the target coordinates, the angle after rotation is adjusted, and the angle of the visual sensor is accurately controlled, which overcomes the offset between the tracking center and the target center caused by jitter or external factors during the movement, and achieves the purpose of tracking and identifying the target object by always being in the center position of the image collected by the visual sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flow chart of a target tracking control method based on image recognition provided by an embodiment of the present invention;

[0045] Figure 2 A structural block diagram of a target tracking control device based on image recognition provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the structure of camera target tracking provided by an embodiment of the present invention;

[0047] Figure 4 A structural block diagram of a camera target tracking control provided by an embodiment of the present invention;

[0048] Figure 5 A flowchart of a target tracking control process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0050] The motion target tracking technology has shown a wide range of application value in many fields, including: in terms of intelligent transportation, it can track and photograph vehicles and pedestrians that violate traffic regulations, automatically record violations and report them to the traffic safety system, saving a lot of manpower and material resources. In terms of intelligent driving, cars face complex traffic environments, changeable weather conditions, unknown driving behaviors, and sudden traffic conditions on the road; intelligent driving can identify abnormal behaviors of vehicles and pedestrians in the surrounding movement, so as to take timely evasive actions. In terms of security monitoring, public facilities, fire conditions, abnormal personnel and other situations can be intelligently monitored and tracked, and abnormalities can be reported in time to prevent them from happening. In terms of crowd control, the flow of people in crowded areas such as large shopping malls and tourist attractions during specific time periods is counted to facilitate relevant managers to control the flow of people and plan the site.

[0051] However, when identifying and tracking targets, precision sensor technology is usually used to improve the accuracy of target detection and tracking, which is costly but not accurate and timely. Therefore, in order to improve the stability of the motion state and the accuracy of tracking, it is necessary to solve the problem of the deviation between the tracking center and the target center caused by the motion process, which leads to increased errors.

[0052] like Figure 1 As shown, an embodiment of the present invention provides a target tracking control method based on image recognition, comprising the following steps:

[0053] Performing color recognition on the target image collected by the visual sensor through an adaptive color recognition algorithm to obtain a color area, and performing center calculation on the color area to obtain color center coordinates;

[0054] Performing object recognition on the target image through a pre-trained YOLOv2 target detection model to obtain an object region, performing center calculation on the object region to obtain the object center coordinates;

[0055] The color center coordinates and the object center coordinates are averaged to obtain the tracking center coordinates, which are converted into motion control signals. The visual sensor is driven to rotate according to the motion control signals, and the actual rotation angle of the visual sensor is read. The actual rotation angle is calculated through a PID control algorithm, and the rotation of the visual sensor is corrected according to the calculation result to achieve target tracking.

[0056] Specifically, the YOLOv2 target detection model is an upgraded version based on YOLO, integrating the characteristics of some other network structures. Its detection principle is to use a convolutional network as the backbone network, and the output layer adopts a specific three-dimensional structure. The trained model can identify and locate different categories of target objects from the input image. The pre-training process of the YOLOv2 target detection model includes: sample collection, sample upload, data annotation, online model training on the MaixHub platform, and model deployment. In order to improve the recognition accuracy of the model, a variety of image enhancement operations are selected during the training process, including random rotation, random mirroring, and random blurring. It can be seen from the training report output by the MaixHub platform that the prediction accuracy (val_acc, val_acc_pig, val_acc_dunk) of the YOLOv2 target detection model for the validation set is stable at around 0.9, that is, the prediction recognition accuracy is high. Among them, sample collection and data labeling can be selected according to needs. You can choose vehicles on the road in the traffic scene and label the vehicles, or you can choose pedestrians in the store in the shopping mall and label the people, that is, identify the target object according to the target labeling features.

[0057] It should be understood that the real-time and continuous image acquisition by the visual sensor, that is, acquisition at a frame rate, is 10-20 frames per second depending on the usage scenario, in order to achieve tracking of moving targets. When it is necessary to locate and track a person wearing red clothes, the scene is imaged at a frame rate, and the YOLOv2 target detection model is used to identify multiple human bodies in the image, and then the adaptive color recognition algorithm is integrated to identify the red area of ​​the image color to achieve target positioning, that is, the person wearing red clothes is accurately identified through red recognition and human body recognition.

[0058] In the embodiment of the present invention, the adaptive color recognition algorithm is used to adaptively debug the target image, so that the target color to be recognized can be continuously approached, and the coordinates of the target color feature can be obtained. Then, the shape of the object in the target image is detected by the target detection model, and the target shape can be found in a large number of objects, and the coordinates of the target shape feature can be obtained. After the visual sensor is controlled to rotate according to the target coordinates, the rotation angle of the visual sensor is adjusted by the PID control algorithm, thereby avoiding the rotation error caused by jitter or external factors when the visual sensor rotates for the first time, so that the target object is always in the center of the image collected by the visual sensor.

[0059] Preferably, the color recognition of the target image collected by the visual sensor is performed by an adaptive color recognition algorithm to obtain a color area, specifically:

[0060] S1. Obtain LAB threshold;

[0061] S2, performing color recognition on the target image collected by the visual sensor according to the LAB threshold through the color recognition function, and determining whether a color area corresponding to the threshold is recognized, if so, obtaining the color area, if not, executing S3;

[0062] S3, adjusting the parameters of the LAB threshold through the set adaptive adjustment scheme to obtain an updated LAB threshold, and executing S4;

[0063] S4, performing color recognition on the target image according to the updated LAB threshold through a color recognition function, and determining whether a color region corresponding to the threshold is recognized, if so, obtaining the color region, if not, executing S3 until the color region is obtained.

[0064] Specifically, an arbitrary red LAB threshold is sent to the adaptive color recognition algorithm via Bluetooth / wifi on the set mobile phone APP; the color recognition function is img.find_blobs. When searching for red blocks (i.e., color areas) within the LAB threshold in the target image, an approximate red LAB threshold is input as a parameter in the color recognition function, i.e., red_blobs = img.find_blobs (red_threshold, merge = True), where red_threshold is the LAB threshold parameter, and then the parameters of the LAB threshold are adjusted according to the set adaptive adjustment scheme, so that the color recognition function can recognize the desired color in the target image.

[0065] When multiple identical color regions are identified, a color threshold can be added to increase the recognition parameters of the color recognition function, so that the color target can be located by identifying multiple different target colors through the color recognition function. If a red region is identified for the first time, and a yellow region is identified again, the distance between the two color center coordinates is calculated. The two color regions corresponding to the smallest distance are the same object, which is set as the target, and the mean of the two color center coordinates is calculated to obtain the color center coordinates of the target.

[0066] It should be understood that general color recognition is achieved by searching for target color blocks in an image based on a calibrated color threshold. Color threshold debugging is an indispensable step, and understanding the characteristics of the LAB color space is the basis for color threshold debugging. The LAB color model is a device-independent color model based on physiological characteristics, consisting of three elements: brightness (L) and two color channels (A and B). Color recognition involves color threshold setting, which also depends on the LAB color model. In the LAB color space, the L component describes the brightness of the pixel, the larger the value, the brighter it is, and the value range is [0, 100], representing from pure black to pure white; the A component describes the range from green to red, and the value range is [-128, 127], where positive values ​​indicate a bias towards red, and negative values ​​indicate a bias towards green; the B component describes the range from blue to yellow, and the value range is [-128, 127], where positive values ​​indicate a bias towards yellow, and negative values ​​indicate a bias towards blue. According to the characteristics of LAB color space, the color threshold can be debugged through the color threshold editor to obtain the color threshold range in the format of [(Lmin, Lmax, Amin, Amax, Bmin, Bmax)]. For example, when the color threshold is [(3, 100, -41, -6, -13, 15)], only the green block is left in the debugging screen, and when the color threshold is [(3, 100, 10, 44, -7, 17)], only the red block is left in the debugging screen. After color threshold debugging, both red and green can be correctly identified.

[0067] In other words, you can also input the range of the three component values ​​of L, A and B of the LAB threshold in the color threshold editor, and the color recognition function will obtain the color threshold range of the target image, and filter out other colors in the target image according to the input threshold, and only keep the target color to be tracked, and the other areas are black. However, this recognition scheme of inputting a fixed LAB threshold is relatively rigid and can only recognize the set color. If you want to recognize other colors, you also need to find the thresholds of other colors in the color editor and re-modify the parameters of the LAB threshold. It is not possible to change the color to be recognized online.

[0068] In the embodiment of the present invention, the input color threshold is continuously adjusted through an adaptive color recognition algorithm, so that the color recognition function can recognize the target color area, and then obtain the coordinates of the target color, and perform tracking behavior according to the coordinates.

[0069] Preferably, the adaptive adjustment scheme includes a brightness adjustment scheme, a first color adjustment scheme and a second color adjustment scheme;

[0070] The S3 includes:

[0071] S31, adjusting the second color parameter B of the LAB threshold by using the second color adjustment scheme, and when the color recognition function cannot recognize the color area corresponding to the threshold according to the adjusted LAB threshold, executing the first color adjustment scheme;

[0072] S32, adjusting the first color parameter A and the second color parameter B of the LAB threshold by using the first color adjustment scheme, and executing the brightness adjustment scheme when the color recognition function cannot recognize the color area corresponding to the threshold according to the adjusted LAB threshold;

[0073] S33: Adjust the brightness parameter L of the LAB threshold, the first color parameter A, and the second color parameter B by using the brightness adjustment scheme.

[0074] In the embodiment of the present invention, the parameters of the LAB threshold are adjusted through multiple schemes, and the adjustment is performed layer by layer so that the color recognition function can accurately identify the target color area.

[0075] Preferably, in S31, adjusting the second color parameter B of the LAB threshold by using the second color adjustment scheme includes:

[0076] The brightness parameter L and the first color parameter A are fixed, and the maximum value and / or the minimum value of the second color parameter B are adjusted multiple times, and an updated LAB threshold is obtained according to the brightness parameter L, the first color parameter A and the corresponding adjusted second color parameter B;

[0077] In S32, adjusting the first color parameter A and the second color parameter B of the LAB threshold by using the first color adjustment scheme includes:

[0078] The brightness parameter L is fixed, and the maximum value and / or the minimum value of the first color parameter A are adjusted multiple times. When the maximum value and / or the minimum value of the first color parameter A are adjusted each time, the maximum value and / or the minimum value of the second color parameter B are adjusted multiple times, and an updated LAB threshold is obtained according to the brightness parameter L, the corresponding adjusted first color parameter A, and the corresponding adjusted second color parameter B;

[0079] The S33 comprises:

[0080] The maximum value and / or minimum value of the brightness parameter L is adjusted multiple times respectively. Each time the maximum value and / or minimum value of the brightness parameter L is adjusted once, the maximum value and / or minimum value of the first color parameter A is adjusted multiple times respectively. Each time the maximum value and / or minimum value of the first color parameter A is adjusted once, the maximum value and / or minimum value of the second color parameter B is adjusted multiple times respectively. The updated LAB threshold is obtained from the corresponding adjusted brightness parameter L, the corresponding adjusted first color parameter A and the corresponding adjusted second color parameter B.

[0081] Preferably, the adjusting the maximum value and / or the minimum value of the second color parameter B for multiple times comprises:

[0082] Fixing the maximum value (or the minimum value) of the second color parameter B and adjusting the minimum value of the second color parameter B include:

[0083] Determine whether the minimum value reaches a first preset minimum threshold value, if not, reduce the minimum value of the second color parameter B according to the set value, obtain an updated LAB threshold value by the maximum value of the brightness parameter L, the first color parameter A, the second color parameter B and the corresponding adjusted minimum value of the second color parameter B, and when the color recognition function cannot recognize the color area corresponding to the threshold value according to the updated LAB threshold value, repeat the operation of adjusting the minimum value of the second color parameter B;

[0084] If yes, then adjusting the maximum value of the second color parameter B includes:

[0085] Determine whether the maximum value reaches a first preset maximum threshold value. If not, increase the maximum value of the second color parameter B according to the set value, and perform an operation of adjusting the minimum value of the second color parameter B. The updated LAB threshold value is obtained by the brightness parameter L, the first color parameter A, and the corresponding maximum and minimum values ​​of the adjusted second color parameter B. When the color recognition function cannot identify the color area corresponding to the threshold value according to the updated LAB threshold value, repeat the operation of adjusting the maximum value of the second color parameter B.

[0086] It should be understood that the steps of adjusting the maximum value and / or minimum value of the first color parameter A multiple times and the steps of adjusting the maximum value and / or minimum value of the brightness parameter L multiple times in the above steps are the same as the steps of adjusting the maximum value and / or minimum value of the second color parameter B multiple times.

[0087] Specifically, the format of the color threshold range of the LAB threshold is [(Lmin, Lmax, Amin, Amax, Bmin, Bmax)]. The value of Bmin or Bmax is adjusted once through the second color adjustment scheme to obtain an updated LAB threshold, and the color recognition function is used to perform color recognition on the template image according to the updated LAB threshold. If the color area is not recognized, Bmin or Bmax is adjusted again, and this is repeated until Bmin or Bmax reaches the threshold. In addition, when adjusting the value of Bmin or Bmax once, one of Bmin or Bmax is first fixed. If Bmax is fixed, Bmin is first reduced by 1 each time it is adjusted until it is reduced to the preset minimum threshold. When the color area cannot be recognized after adjusting Bmin, Bmax is added by 1 and then fixed, and the original Bmin is reduced by 1 each time it is adjusted.

[0088] That is, when the input random red threshold is [(3, 100, 10, 44, -7, 17)], the random red threshold is adjusted by the brightness adjustment scheme, the first color adjustment scheme, and the second color adjustment scheme.

[0089] The second color adjustment scheme includes: first fixing the brightness parameter L and the first color parameter A, changing the second color parameter B, the range of the second color parameter B of the red color system is approximately -70 to +70, taking the given parameter B as the starting point, changing a value of parameter B each time, adding 1 to the large value Bmax, or subtracting 1 from the small value Bmin, where the range of the large value change is 1 to 70, and the range of the small value change is 1 to -70.

[0090] The first color adjustment scheme includes: first fixing the brightness parameter L, changing the first color parameter A and the second color parameter B, the range of the first color parameter A of the red color system is approximately 0 to +127, and the range of the second color parameter B is approximately -70 to +70, taking the given parameter A as the starting point, each time a value of parameter A is changed, 1 is added to the large value Amax, or 1 is subtracted from the small value Amin, wherein the range of the large value change is 1 to 127, and the range of the small value change is 1 to 0; and each time the value of the A parameter changes, the B parameter must be traversed once from -70 to +70 (i.e., the second color adjustment scheme is executed once).

[0091] The brightness adjustment scheme includes: changing the brightness parameter L, the first color parameter A, and the second color parameter B. The range of the brightness parameter L for the red color system is approximately 25 to 75, the range of the first color parameter A is approximately 0 to +127, and the range of the second color parameter B is approximately -70 to +70. Starting from the given L parameter, each time a value of the parameter L is changed, add 1 to the maximum value Amax or subtract 1 from the minimum value Amin, where the range of the change in the maximum value is from 1 to 25, and the range of the change in the minimum value is from 26 to 100; moreover, every time the L parameter is changed by one value, the A parameter needs to traverse once within the range of 0 to +127 (i.e., execute the first color adjustment scheme once); every time the A parameter is changed by one value, the B parameter needs to traverse once within the range of -70 to +70 (i.e., execute the second color adjustment scheme once).

[0092] In an embodiment of the present invention, when the color recognition function still cannot recognize the color area corresponding to the threshold value according to the adjusted LAB threshold value, the multiple parameters of the LAB threshold value are adjusted one by one for multiple times, so that the target color required can be recognized in the image by dynamically adjusting the LAB threshold value.

[0093] Preferably, the color area is a pixel matrix composed of multiple pixel points;

[0094] The central calculation of the color area to obtain the color center coordinates is specifically as follows:

[0095] Perform a mean value calculation on the maximum value and the minimum value of the abscissas of the multiple pixel points in the pixel matrix to obtain the central abscissa, perform a mean value calculation on the maximum value and the minimum value of the ordinates of the multiple pixel points in the pixel matrix to obtain the central ordinate, and obtain the color center coordinates from the central abscissa and the central ordinate.

[0096] Specifically, the pixels in the color area of the image can be regarded as a matrix with x rows and y columns. The coordinates of the pixel point are (n, m), and 0 < n < x, 0 < m < y, indicating that this pixel point is a point in the nth row and the mth column. If a red area is recognized, then this red area is a coordinate set of several (n, m). In this set, the abscissa n has a maximum value and a minimum value, and the ordinate m has a maximum value and a minimum value. Respectively sum the maximum value and the minimum value of the abscissa n and the ordinate m and then divide by 2 to obtain the center point coordinates of the red area (i.e., the color center coordinates).

[0097] It should be understood that the steps of performing central calculation on the object area to obtain the object center coordinates are the same as the steps of obtaining the color center coordinates.

[0098] In an embodiment of the present invention, the corresponding center point coordinates are obtained according to the multiple pixel values of the recognized feature target, so as to achieve precise positioning of the target.

[0099] Preferably, before the step of converting the tracking center coordinates into a motion control signal, the method further comprises:

[0100] The tracking center coordinates are packaged according to the set data frame header and data frame footer to obtain a data frame, and the data frame is sent through the set serial port, and it is verified whether the data frame is complete.

[0101] Specifically, the tracking center coordinates are packaged by formulating specific data frame headers and frame tails, and a pre-configured serial port is used for communication. The packaged target coordinate information (i.e., the tracking center coordinates) is sent to the tracking controller integrated in the single-chip microcomputer main control. The single-chip microcomputer main control determines whether the received data frame header and frame tail are correct data frames, that is, when 55 represents the frame header and AA represents the frame tail, when the AA byte is received, it indicates that the transmission of this frame of data has ended, thereby ensuring the correctness of the target coordinate information.

[0102] In the embodiment of the present invention, the transmitted coordinate information is verified when it is received to avoid incomplete data transmission and erroneous tracking.

[0103] Preferably, the converting the tracking center coordinates into a motion control signal, and driving the visual sensor to rotate according to the motion control signal, is specifically:

[0104] Acquire the image center coordinates of the target image, calculate the difference between the image center coordinates and the tracking center coordinates, and obtain the coordinate difference;

[0105] The coordinate difference is converted into a pulse width modulation signal of a set frequency and a direction control signal, the rotation speed of the visual sensor is controlled according to the pulse width modulation signal, and the rotation angle of the visual sensor is controlled according to the direction control signal.

[0106] Specifically, when the target image is collected, the image center coordinates are calculated to obtain the initial center coordinates of the image positioning by the visual sensor, and the difference between the image center coordinates and the tracking center coordinates is calculated to obtain the coordinate difference. According to the coordinate difference, a pulse width modulation signal and a direction signal are generated. The frequency of the PWM wave determines the speed of the motor, and the state of the direction signal is set according to the direction of the coordinate difference. For example, if the difference of the X coordinate is positive, the direction signal may need to be set to the right. That is, if the object is in the lower left corner of the image, a signal is generated to make the visual sensor turn left and turn downward, so that the object in the image captured by the visual sensor is in the center of the image; conversely, if the object is in the upper right corner of the image, a signal is generated to make the visual sensor turn right and turn upward, so that the object in the image captured by the visual sensor is in the center of the image; by setting the direction signal of rotation, the left or right rotation, up or down rotation is controlled, and the speed of rotation is controlled by setting the frequency of the pulse width modulation signal.

[0107] In the embodiment of the present invention, the coordinate difference is converted into a motion control signal to drive the visual sensor to rotate, so that the target object in the image captured by the visual sensor is always located at the center of the image, thereby achieving the purpose of tracking.

[0108] Preferably, the actual rotation angle is calculated by a PID control algorithm, and the rotation of the visual sensor after rotation is corrected according to the calculation result, specifically:

[0109] The difference between the rotation angle and the actual rotation angle is calculated by a PID control algorithm to obtain an angle difference, and the rotation of the rotated visual sensor is corrected according to the angle difference.

[0110] In the embodiment of the present invention, a PID control algorithm is used to perform closed-loop control on the visual sensor, thereby improving the control accuracy of the rotation angle of the visual sensor and overcoming the problem of offset between the tracking center and the target center caused during the movement.

[0111] like Figure 2 As shown, an embodiment of the present invention provides a target tracking control device based on image recognition, comprising:

[0112] A coordinate operator, connected to a visual sensor for outputting a target image, for performing color recognition on the target image acquired by the visual sensor through an adaptive color recognition algorithm to obtain a color area, performing center calculation on the color area to obtain color center coordinates, performing object recognition on the target image through a pre-trained YOLOv2 target detection model to obtain an object area, performing center calculation on the object area to obtain object center coordinates;

[0113] A tracking controller is connected after the coordinate operator, and is used to calculate the mean of the color center coordinates and the object center coordinates to obtain the tracking center coordinates, convert the tracking center coordinates into a motion control signal, drive the visual sensor to rotate according to the motion control signal, and read the actual rotation angle of the visual sensor, calculate the actual rotation angle through a PID control algorithm, and correct the rotation of the visual sensor after rotation according to the calculation result.

[0114] It should be understood that the visual sensor can be installed on a driving platform (such as a two-dimensional pan-tilt platform), and the two-dimensional pan-tilt platform can be driven to rotate according to a motion control signal, thereby driving the visual sensor to rotate, which is described in detail in the following embodiments.

[0115] A camera provided by an embodiment of the present invention includes a visual sensor and a magnetic encoder respectively connected to a two-dimensional pan-tilt head, and a target tracking control device for image recognition as described above, wherein the device is used to execute the target tracking control method for image recognition as described above.

[0116] Preferably, a two-dimensional pan-tilt platform is constructed by multiple stepper motors, and the stepper motors of the two-dimensional pan-tilt platform are driven to rotate according to the motion control signal to drive the visual sensor to rotate, and the actual rotation angle of the two-dimensional pan-tilt platform is read by a magnetic encoder.

[0117] It should be understood that the absolute position encoder (i.e., magnetic encoder) records the position of the encoder in an absolute coordinate system, and then detects parameters such as rotation position, speed and direction, and reads the current rotation angle of the stepper motor (i.e., the actual rotation angle of the two-dimensional gimbal) by outputting an electrical pulse signal.

[0118] An experimental process of tracking a target with a camera includes:

[0119] like Figure 3 and Figure 4 As shown in the figure, STM32F103C8T6 is used as the control core (i.e., the main microcontroller), and the processing process of the tracking controller for calculating the tracking center coordinates, converting the motion control signal, and driving the visual sensor to rotate is integrated into the main microcontroller (i.e., the processing process of the tracking controller is integrated into the main microcontroller), V831 visual sensor (i.e., visual module), the processing process of color recognition, object recognition, color center coordinates, and object center coordinates calculation is integrated into the visual module (i.e., the processing process of the coordinate operator is integrated into the visual module), MT6701 is a magnetic encoder, and two 42 stepper motors are used as two-dimensional cloud The X-axis and Y-axis actuators of the stage, the stepper motor drive system drives the corresponding stepper motor movement according to the motion control signal; the visual sensor is installed on the two-dimensional pan-tilt, the green laser pen is fixed on one end of the visual sensor, and the angle of the laser is adjusted so that the laser spot is the center of the image captured by the visual sensor; it is powered by a lithium battery, and the buck module is used to step down the voltage to 5V to power other modules; a key circuit or a set mobile phone APP is used for human-computer interaction to realize function selection and generate control instructions; finally, the single-chip main control drives the display module to display time-consuming information, and drives the prompt module to sound prompts for the tracking effect to build a tracking model. Another two-dimensional pan-tilt controls the rotation of the red laser pen (you can also hold a red laser pen in your hand to rotate), and the red spot on the screen will move according to the set recognition feature route to simulate the instructions for the target to be tracked.

[0120] Among them, the mobile phone APP has the same function as the key circuit. The mobile phone APP can be understood as a remote control. The key circuit is set on the microcontroller circuit board. It sends instructions to the microcontroller main control, and the instructions are pre-defined. For example, the key circuit can be defined as the first button is a reset instruction, the second button is a tracking instruction, and the third button is a display page turning instruction. The mobile phone APP can be defined as sending 00 is a reset instruction, sending 01 is a tracking instruction, and sending 02 is a display page turning instruction.

[0121] like Figure 5 As shown in the figure, the initialization phase is first entered, such as the initialization of the microcontroller main control peripherals, the initialization of the V831 visual sensor, the initialization of each module, the reset of the two-dimensional pan-tilt, and the setting of various parameters. Then, the main loop is entered, and visual recognition is continuously performed through the visual module, and the microcontroller main control receives instructions and refreshes the screen. The instructions are input by the key circuit or the mobile phone APP. Pressing different keys or buttons will generate different instructions, such as reset instructions, tracking instructions, and display page turning instructions. When the tracking instruction is received, the tracking operation is executed, and the two-dimensional pan-tilt is driven to rotate according to the received tracking center coordinates to track the moving target; when the reset instruction is received, the reset operation is executed to drive the two-dimensional pan-tilt to reset to the set origin. When the page turning instruction is received, the display page turning operation is executed, and various information on the display screen is viewed through the display module, and the tracking effect is sounded through the buzzer of the prompt module.

[0122] Among them, an arbitrary red color LAB threshold is sent to the MCU main control via Bluetooth / wifi on the set mobile phone APP, and the MCU main control then sends it to the adaptive color recognition algorithm of the visual module through the serial port; the visual module recognizes the image in real time and continuously sends the center coordinates to the MCU main control. When the MCU main control does not receive the tracking command, it does not perform tracking; that is, it only accepts the coordinate information and does not take action. When the tracking command is received, it tracks according to the coordinate information. The visual module calculates the recognition time, displays it on the display screen of the visual module, and sends it to the MCU main control at the same time, and also displays it on the display screen of the display module of the MCU main control.

[0123] The tracking process is as follows: the V831 visual sensor performs color recognition on the collected target image based on the adaptive color recognition algorithm, searches for the target color block (i.e., color area), identifies the color center coordinates according to the color area, and sends the target coordinate information (i.e., color center coordinates) to the MCU main control; and uses the trained YOLOv2 target detection model to perform object recognition on the collected target image, obtain the object area, identify the object center coordinates according to the object area, and send the target coordinate information (i.e., object center coordinates) to the MCU main control; communicates with the MCU main control using the serial port, and packages the target coordinate information by formulating specific data frame headers and frame tails. The MCU main control determines whether it is a correct data frame based on the received data frame header and frame tail to ensure the correctness of the target coordinate information. According to the target coordinate information, the single-chip microcomputer generates a pulse width modulation signal of a specific frequency and a direction control signal, and inputs the signal to the motor drive module to drive the stepper motor on the two-dimensional pan-tilt platform to rotate. At the same time, the magnetic encoder is used to read the current rotation angle information of the stepper motor to realize the rotation position feedback of the stepper motor. The PID algorithm is used to control the movement of the two-dimensional pan-tilt platform so that the green light spot tracks the red light spot and makes the green light spot overlap with the red light spot to verify the tracking of the target.

[0124] The reset process is as follows: the MCU main control reads the rotation position information (i.e. the actual rotation angle) of the stepper motor from the MT6701 magnetic encoder, and controls the two-dimensional pan-tilt head to reset to the set origin through the PID algorithm.

[0125] This solution can be applied in home monitoring and identification scenarios, and can locate and track objects in the home, such as for child care.

[0126] The above-mentioned target tracking control device and camera based on image recognition can refer to the implementation content and beneficial effects of the target tracking control method based on image recognition specifically described above, which will not be repeated here.

[0127] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0128] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A target tracking control method based on image recognition, characterized in that: The steps include: Performing color recognition on the target image collected by the visual sensor through an adaptive color recognition algorithm to obtain a color area, and performing center calculation on the color area to obtain color center coordinates; Performing object recognition on the target image through a pre-trained YOLOv2 target detection model to obtain an object region, performing center calculation on the object region to obtain the object center coordinates; The color center coordinates and the object center coordinates are averaged to obtain the tracking center coordinates, which are converted into motion control signals. The visual sensor is driven to rotate according to the motion control signals, and the actual rotation angle of the visual sensor is read. The actual rotation angle is calculated through a PID control algorithm, and the rotation of the visual sensor is corrected according to the calculation result to achieve target tracking.

2. The target tracking control method based on image recognition according to claim 1, characterized in that: The color recognition is performed on the target image collected by the visual sensor by using the adaptive color recognition algorithm to obtain the color area, which is specifically: S1. Obtain LAB threshold; S2, performing color recognition on the target image collected by the visual sensor according to the LAB threshold through the color recognition function, and determining whether a color area corresponding to the threshold is recognized, if so, obtaining the color area, if not, executing S3; S3, adjusting the parameters of the LAB threshold through the set adaptive adjustment scheme to obtain an updated LAB threshold, and executing S4; S4, performing color recognition on the target image according to the updated LAB threshold through a color recognition function, and determining whether a color region corresponding to the threshold is recognized, if so, obtaining the color region, if not, executing S3 until the color region is obtained.

3. The target tracking control method based on image recognition according to claim 2, characterized in that: The adaptive adjustment scheme includes a brightness adjustment scheme, a first color adjustment scheme, and a second color adjustment scheme; The S3 includes: S31, adjusting the second color parameter B of the LAB threshold by using the second color adjustment scheme, and when the color recognition function cannot recognize the color area corresponding to the threshold according to the adjusted LAB threshold, executing the first color adjustment scheme; S32, adjusting the first color parameter A and the second color parameter B of the LAB threshold by using the first color adjustment scheme, and executing the brightness adjustment scheme when the color recognition function cannot recognize the color area corresponding to the threshold according to the adjusted LAB threshold; S33: Adjust the brightness parameter L of the LAB threshold, the first color parameter A, and the second color parameter B by using the brightness adjustment scheme.

4. The target tracking control method based on image recognition according to claim 3 is characterized in that: In S31, adjusting the second color parameter B of the LAB threshold by using the second color adjustment scheme includes: The brightness parameter L and the first color parameter A are fixed, and the maximum value and / or the minimum value of the second color parameter B are adjusted multiple times, and an updated LAB threshold is obtained according to the brightness parameter L, the first color parameter A and the corresponding adjusted second color parameter B; In S32, adjusting the first color parameter A and the second color parameter B of the LAB threshold by using the first color adjustment scheme includes: The brightness parameter L is fixed, and the maximum value and / or the minimum value of the first color parameter A are adjusted multiple times. When the maximum value and / or the minimum value of the first color parameter A are adjusted each time, the maximum value and / or the minimum value of the second color parameter B are adjusted multiple times, and an updated LAB threshold is obtained according to the brightness parameter L, the corresponding adjusted first color parameter A, and the corresponding adjusted second color parameter B; The S33 comprises: The maximum value and / or minimum value of the brightness parameter L is adjusted multiple times respectively. Each time the maximum value and / or minimum value of the brightness parameter L is adjusted once, the maximum value and / or minimum value of the first color parameter A is adjusted multiple times respectively. Each time the maximum value and / or minimum value of the first color parameter A is adjusted once, the maximum value and / or minimum value of the second color parameter B is adjusted multiple times respectively. The updated LAB threshold is obtained from the corresponding adjusted brightness parameter L, the corresponding adjusted first color parameter A and the corresponding adjusted second color parameter B.

5. The target tracking control method based on image recognition according to claim 4, characterized in that: The step of adjusting the maximum value and / or the minimum value of the second color parameter B for multiple times comprises: Fixing the maximum value of the second color parameter B and adjusting the minimum value of the second color parameter B include: Determine whether the minimum value reaches a first preset minimum threshold value, if not, reduce the minimum value of the second color parameter B according to the set value, obtain an updated LAB threshold value by the maximum value of the brightness parameter L, the first color parameter A, the second color parameter B and the corresponding adjusted minimum value of the second color parameter B, and when the color recognition function cannot recognize the color area corresponding to the threshold value according to the updated LAB threshold value, repeat the operation of adjusting the minimum value of the second color parameter B; If yes, then adjusting the maximum value of the second color parameter B includes: Determine whether the maximum value reaches a first preset maximum threshold value. If not, increase the maximum value of the second color parameter B according to the set value, and perform an operation of adjusting the minimum value of the second color parameter B. The updated LAB threshold value is obtained by the brightness parameter L, the first color parameter A, and the corresponding maximum and minimum values ​​of the adjusted second color parameter B. When the color recognition function cannot identify the color area corresponding to the threshold value according to the updated LAB threshold value, repeat the operation of adjusting the maximum value of the second color parameter B.

6. The target tracking control method based on image recognition according to claim 1, characterized in that: The color area is a pixel matrix composed of a plurality of pixel points; The center of the color area is calculated to obtain the color center coordinates, specifically: The maximum and minimum values ​​of the horizontal coordinates of multiple pixel points in the pixel matrix are averaged to obtain the central horizontal coordinate, the maximum and minimum values ​​of the vertical coordinates of multiple pixel points in the pixel matrix are averaged to obtain the central vertical coordinate, and the color center coordinate is obtained from the central horizontal coordinate and the central vertical coordinate.

7. The target tracking control method based on image recognition according to claim 1, characterized in that: The step of converting the tracking center coordinates into a motion control signal and driving the visual sensor to rotate according to the motion control signal is specifically as follows: Acquire the image center coordinates of the target image, calculate the difference between the image center coordinates and the tracking center coordinates, and obtain the coordinate difference; The coordinate difference is converted into a pulse width modulation signal of a set frequency and a direction control signal, the rotation speed of the visual sensor is controlled according to the pulse width modulation signal, and the rotation angle of the visual sensor is controlled according to the direction control signal.

8. An image recognition target tracking control device, characterized in that: include: A coordinate operator, connected to a visual sensor for outputting a target image, for performing color recognition on the target image acquired by the visual sensor through an adaptive color recognition algorithm to obtain a color area, performing center calculation on the color area to obtain color center coordinates, performing object recognition on the target image through a pre-trained YOLOv2 target detection model to obtain an object area, performing center calculation on the object area to obtain object center coordinates; A tracking controller is connected after the coordinate operator, and is used to calculate the mean of the color center coordinates and the object center coordinates to obtain the tracking center coordinates, convert the tracking center coordinates into a motion control signal, drive the visual sensor to rotate according to the motion control signal, and read the actual rotation angle of the visual sensor, calculate the actual rotation angle through a PID control algorithm, and correct the rotation of the visual sensor after rotation according to the calculation result.

9. A camera, characterized in that: It includes a visual sensor and a magnetic encoder respectively connected to the two-dimensional pan-tilt head, and an image recognition target tracking control device as described in claim 8, wherein the device is used to execute the image recognition target tracking control method as described in any one of claims 1-7.

10. The camera according to claim 9, characterized in that: A two-dimensional pan-tilt platform is constructed by multiple stepper motors. The two-dimensional pan-tilt platform is driven to rotate the visual sensor according to the motion control signal, and the actual rotation angle of the two-dimensional pan-tilt platform is read by a magnetic encoder.

Images