A target following method and system

Abstract

The application provides a target following method and system, the method comprises: generating at least one detection frame, identifying the target object framed in the current frame of the acquired video and determining the category thereof; generating at least one prediction frame, predicting the position and moving speed of the target object in the next frame of the video; based on the position and moving speed of the target object, comparing the prediction frame with the detection frame of the next frame of the video, verifying the correctness of the category classification of the target object; judging the similarity degree between the classified target object and the pre-set object, and determining the target object to be followed as the pre-set object. By using the technical scheme of the application, the functions of following and obstacle avoidance can be realized while using hardware assembly with lower cost; the accuracy of following and identifying the target object can be improved, and the performance of following and identifying the target object is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual following, and particularly relates to a target following method and system. BACKGROUND

[0002] Mobile robots have been widely applied. At present, there are mobile robots in the forms of wheeled / track type, leg-foot type and humanoid type, which can be applied to industrial manufacturing, home cleaning and other scenes. In various application scenes, the improvement of the identification, control and obstacle avoidance functions of the mobile robots can improve the performance of the mobile robots in intelligently following a specific target. The identification module can make the mobile robot follow a specific target and serve the same object. The control module can make the mobile robot smoothly follow the specific target. When the specific target moves faster, the control module controls the mobile robot to increase the speed. When the specific target moves slower, the control module controls the mobile robot to reduce the speed or pause. The obstacle avoidance module can make the mobile robot freely travel in a complex and changeable scene and avoid crowds or various obstacles in the scene.

[0003] One of the application scenes with high requirements for the performance of the mobile robots in intelligently following a specific target is in an airport or a supermarket, when a customer needs to put luggage or articles into a cart for carrying. The cart plus the articles are relatively heavy and inconvenient to operate. If the cart can automatically follow the user and keep a certain distance, the user does not need to manually push the cart, so that the passengers in the airport and train station and the customers in the supermarket will be more relaxed and convenient. This function will have more universal significance under the trend of an aging society.

[0004] The existing intelligent following scheme of mobile robots is realized by using transmitter and receiver, ultrasonic sensor and camera, etc. For the way of guiding the intelligent following of the mobile robot to the user by the relative position of the transmitter and the receiver, the user holds the receiver or wears a specially designed wearable receiving device to communicate with the transmitter installed on the mobile robot to achieve the following effect. The accuracy of this following method is high, and the position information of the user relative to the mobile robot can be accurately known, but it lacks unique identification and is difficult to avoid obstacles. When multiple users are close to the corresponding mobile robots, the mobile robots cannot effectively determine which user to follow due to the lack of individual identification. Since only the transmitter and the receiver are used, other obstacles besides the user cannot be judged, and other devices need to be installed to achieve the obstacle avoidance function. For the way of using ultrasonic sensors, multiple ultrasonic sensors are loaded on the mobile robot to detect the position of the user and control the distance between the mobile robot and the user. This method cannot distinguish different users and is usually better applied in open warehouses, but has poor effect in application scenarios such as supermarkets with many users and complex terrain. For the way of using the camera, the following can be performed according to color block recognition. This method has low following accuracy, does not have unique identification to distinguish different users, and cannot avoid obstacles. The target following algorithm can also be used, and the target following algorithm combining CamShift algorithm and Kalman filter is a widely used real-time moving object tracking method. When the target is not blocked, the CamShift algorithm continuously follows the target. When the target is blocked, the Kalman filter predicts the position of the target according to the motion information of the object before the blocking until the CamShift algorithm continues to continuously follow the target when the target is no longer blocked. Since the above algorithm only focuses on color information, the performance of the above algorithm will be unsatisfactory when the background color is the same as the color of the user's upper garment.

[0005] In summary, the above-mentioned ways of using transmitter and receiver and using ultrasonic sensor to locate the user cannot distinguish different users and are difficult to achieve the obstacle avoidance function. The above-mentioned way of using the camera has limitations in accuracy when tracking specific color blocks, and the use of the target following algorithm requires manual selection of the object to be tracked in the initial camera image, and the mobile robot then tracks it. The operation is relatively cumbersome and cannot achieve automatic identification. The above three methods do not optimize the processing of the specific category of "people", and there is a large space for performance improvement.

[0006] Therefore, it is necessary to improve the prior art. SUMMARY

[0007] To solve the above technical problems, the present application provides a target following method and system to solve the existing problems.

[0008] In one aspect of the present application, a target following method is provided, which is applied to a device for following a target object, comprising:

[0009] generating at least one detection frame to identify the target object framed in the current frame of the acquired video and determine its category;

[0010] generating at least one prediction frame to predict the position and moving speed of the target object in the next frame of the video;

[0011] comparing the prediction frame with the detection frame of the next frame of the video based on the position and moving speed of the target object to verify the correctness of the category classification of the target object;

[0012] judging the similarity degree between the classified target object and a pre-set object to determine that the followed target object is the pre-set object.

[0013] In another aspect of the present application, a target following device is provided, which comprises an identification module, a prediction module, a matching module and a re-identification module;

[0014] The identification module is configured to generate at least one detection frame to identify the target object framed in the current frame of the acquired video and determine its category;

[0015] The prediction module is configured to generate at least one prediction frame to predict the position and moving speed of the target object in the next frame of the video;

[0016] The matching module is configured to compare the prediction frame with the detection frame of the next frame of the video based on the position and moving speed of the target object to verify the correctness of the category classification of the target object;

[0017] The re-identification module is configured to judge the similarity degree between the classified target object and a pre-set object to determine that the followed target object is the pre-set object.

[0018] In another aspect of the present application, an electronic device is provided, which comprises:

[0019] at least one processor;

[0020] a memory in communication connection with the at least one processor; and

[0021] a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method when executing the computer program.

[0022] In another aspect of the present application, a non-volatile computer readable storage medium is provided, having stored thereon a computer program, wherein the computer program is configured to cause the computer to perform the method.

[0023] In another aspect of the present application, a target following system is provided, comprising:

[0024] A movable vehicle having an image acquisition device;

[0025] Further having a controller maintaining automatic adjustment of a distance between the movable vehicle and a target object, so that the distance is maintained at a stable value;

[0026] The movable vehicle follows the target object according to the method.

[0027] The one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0028] By adopting the technical solutions of the present application, the functions of following and obstacle avoidance can be realized while using hardware assembly with lower cost; the accuracy of following and identifying the target object can be improved, and the performance of following and identifying the target object can be enhanced. In addition, the technical solutions of the present application are low in overall cost and wide in application scenarios, can effectively distinguish between users and other crowds in places with dense crowds, and are less affected by light and background color.

[0029] The above brief description is only a summary of the technical solutions of the present application, in order to enable the technical means of the present application to be more clearly understood, and to be implemented according to the content of the specification, and in order to enable the above and other purposes, features and advantages of the present application to be more obvious and easy to understand, the specific embodiments of the present application are described in detail below. BRIEF DESCRIPTION OF DRAWINGS

[0030] In the following description of exemplary embodiments with reference to the accompanying drawings, more details, features and advantages of the present application are disclosed, in which:

[0031] Figure 1 A flow chart of a target following method provided by the present application is shown;

[0032] Figure 2 A schematic diagram of a target following device provided by the present application is shown;

[0033] Figure 3 A schematic diagram of a target following system provided by the present application is shown. DETAILED DESCRIPTION

[0034] Embodiments of the present application will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present application are shown in the drawings, it is understood that the present application can be embodied in various forms and should not be interpreted in limitation to the embodiments set forth herein, but rather by the appended claims. It is to be understood that the drawings and embodiments are only for illustrative purposes and should not be used to limit the scope of the present application.

[0035] It should be understood that each of the steps in the method embodiments of the present application can be performed in a different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit performing the steps shown. The scope of the present application is not limited in this respect.

[0036] The term "comprising" and variations thereof as used herein are used inclusively, i.e., "comprising, but not limited to." The term "based on" is "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." Related terms are defined in the description that follows. It should be noted that reference to a "first," "second," etc. concept does not imply that the concepts so designated must be different, but rather that they can be the same or different.

[0037] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present application are used only for illustrative purposes and are not intended to limit the scope of the messages or information.

[0038] Embodiment One

[0039] In this embodiment, a target following method is provided, which is applied to a device for following a target object. The target can be a person, including but not limited to a person, an animal or other movable device, and in this embodiment, a person is taken as an example for illustration.

[0040] In this embodiment, the device can be any device that needs to follow a corresponding target, such as a wheeled / tracked robot, a leg-foot robot and a humanoid robot. More specifically, in an airport or a supermarket, the device can be a mobile device, such as a trolley, which can place articles to assist passengers or customers to move or carry articles.

[0041] The method provided in the embodiment creates at least one entity corresponding to the identified object, including collecting picture information of each identified object at different time stamps by the device and storing in at least one folder. The picture information of the object at different time stamps can include corresponding the same object with the same digital number in different frames of the video, so as to belong to the same entity. Wherein, for the person showing multiple postures in different frames of the video and / or the person affected by different environmental light, the device can also correspond to the same person. In this way, the device can realize continuous tracking of at least one specific person.

[0042] As shown in Figure 1 The method comprises the following steps:

[0043] Step S101: dividing the picture (picture) of at least one frame in the acquired video into at least one grid, and setting a detection frame in the grid. When there is an object in the grid (i.e. when there is an object in the video), the conditional probability of the detection frame to a certain category of object is calculated. The object category corresponding to the maximum conditional probability is determined as the object category corresponding to the detection frame by comparing the conditional probability, and the category of the object is marked;

[0044] Step S102: for the object framed by the detection frame, the position and moving speed information of the object in one or more previous frames of the video is calculated to predict the position and moving speed information of the next frame, and at least one prediction frame is generated;

[0045] Step S103: comparing the prediction frame with the detection frame of the next frame to determine the coincidence degree of the two, verifying the accuracy of the classification marked according to the object category;

[0046] Step S104: comparing the entity corresponding to the marked object with the entity corresponding to the object preset, judging the similarity degree of the feature vectors of the two entity pictures, to determine the degree of similarity of the two.

[0047] In an optional mode of the embodiment, the video frame (picture) is identified by using YOLO algorithm, which can divide the picture into 7*7 grid and calculate the tensor size of grid output. Using YOLO algorithm to identify the picture can frame the specific target and number it separately, but cannot distinguish different targets, nor can it identify the same target in different frames of the picture as the same target.

[0048] In another optional mode of the embodiment, the target object is predicted by using a Kalman filtering algorithm, and an object corresponding to the entity of the object is created, including position information, size information and speed information of the entity. All the above information includes an average value and a variance. The greater the variance, the greater the uncertainty of the average value predicted by the Kalman filtering algorithm.

[0049] In another optional mode of the embodiment, the target object (person re-identification) is determined by using an OSNet algorithm to solve the lightweight and size diversity problems of person re-identification. The lightweight means that the parameters of the neural network are as few as possible. The main reasons for needing lightweight include: (1) the data set related to the appearance of a person is relatively small, and a neural network with too many parameters may cause overfitting; (2) the application scene of the neural network has a relatively high requirement for operation real-time, and the computing power of the device following the corresponding target is usually not sufficient, and the lightweight network model has relatively fewer operation times, so it can meet the occasion with limited computing power and relatively high requirement for timeliness. The size diversity means that when identifying the whole body information of a person, the features of the person are mainly extracted through the style and color of the clothes and accessories, so when distinguishing different persons according to the whole body information of the person, the whole and local information are very important. In addition, since the whole body information of the person may be blocked in some frames of the video frames, the features of the person also need to be obtained according to the local information.

[0050] The OSNet algorithm used in the embodiment can meet the two requirements of lightweight and size diversity, and is designed as follows:

[0051] The basic component unit in the OSNet algorithm is a Lite convolution block containing a depth separable convolution, so that it can reduce the parameters, and the size and width of the convolution kernel are separated, which can significantly reduce the parameter amount. Each depth separable convolution layer is named Lite, which is used as a basic unit for building a complete network;

[0052] The bottleneck network design of the OSNet algorithm designs two 1*1 point-wise convolution layers at the beginning and the end of the network, which is similar to the operation of separating the depth and size of the convolution in the previous section, and can reduce the parameter amount while ensuring the complexity of the model.

[0053] In addition, since the identity mapping is also an important part of the fitting target function of the neural network, a residual is added before the ReLU activation function of the OSNet algorithm. The residual network directly adds the identity mapping part, which can make the deeper neural network have better training effect and better performance.

[0054] The OSNet algorithm is divided into four parallel paths, which are respectively formed by 1, 2, 3 and 4 layers of Lite modules in series. In the convolutional neural network, the receptive field is defined as the size of the region on the input picture to which each pixel point on the feature map after each layer of convolution is mapped, and the receptive field will increase after the convolution. When the convolution size of the Lite layer is 3*3, the receptive field of the Lite layer after one action is 3*3 pixels; the receptive field of the Lite layer after two actions is 5*5 pixels; the receptive field of the Lite layer after three actions is 7*7 pixels; and the receptive field of the Lite layer after four actions is 9*9 pixels, that is, the features of the receptive fields contained in the four paths are different, so that the overall information and local information can be considered. For obtaining different types of pictures (pictures) of a video, there are differences in attention to overall information or local information, so it is necessary to weight and sum the feature matrices obtained by the four paths, and the weights of different paths depend on the nature of the obtained video frames. The OSNet algorithm is designed with a multi-layer perceptron to assign weights to the four paths, and the multi-layer perceptron is trained to fix its parameters, which will not be changed in subsequent testing and reasoning. After superimposing and processing multiple bottleneck layers, the final overall architecture design of the OSNet algorithm is obtained.

[0055] The OSNet algorithm takes the bottleneck structure as the main framework and the Lite module as the microstructure, which is more lightweight to meet the requirements.

[0056] In another optional mode of the embodiment, the improvement of predicting the target object adopts the DeepSort algorithm, which can not only evaluate the positional relationship between the prediction box and the detection box (i.e. contains the motion information of the person), but also compare the entity features of the person with the features of the detection box (i.e. contains all the color information of the person), and the design is as follows: the DeepSort algorithm uses a simple convolutional neural network to extract the feature information, and the network is composed of two convolutional layers and six residual convolutional blocks, with a parameter amount of 2800864, and the demand for computing power is low (such as using an ordinary consumer-level GPU); in addition, for the extraction and recognition of the features of the person, the convolutional neural network can be pre-trained using a large-scale person re-identification dataset.

[0057] The step S101 further comprises:

[0058] Each prediction box contains coordinate information x and y of the center point of the prediction box, size information w (width) and h (height) of the prediction box, and confidence P (Object) information c of whether the prediction box contains an object, a total of five values.

[0059] For an image divided into a 7x7 grid, each grid contains two bounding boxes. Each bounding box has two coordinate information, two size information, and one confidence information, as well as probability information corresponding to all object categories within each grid. There are a total of 20 object categories. Therefore, each grid contains 2*(2+2+1)+20=30 numbers, meaning the final output tensor size for each grid is 7x7x30.

[0060] The YOLO algorithm gives the probability P(Classi|Object) that each grid cell in the image is classified into a certain object class. If there are N object classes, the object classes are denoted as C1, C2, C3…C… N The probability that a grid square corresponds to an N type of object is: P(C i |Box(1, 7)), 1≤i<N, represents the probability that the grid square covers the i-th type of object. The color of the grid square represents the object category to which different grid squares are most likely to belong. For example, the color of Box(1, 7) in the above formula can be set to red, representing the car object category.

[0061] Based on the calculation of the probability that grid squares cover the i-th type of object, assuming that the predicted box has selected an object, the conditional probability that the predicted box m covers all grid squares belonging to object category i is calculated, thus obtaining the conditional probability that the predicted box m has selected an object belonging to object category i. Comparing this with the conditional probability that the predicted box m has selected a person belonging to object category i, the object category i with the highest probability is taken as the category of the object most likely to be selected by the predicted box m. This process can be expressed as the following chain rule of conditional probabilities:

[0062] P(Class i|Bounding Box)=

[0063] ∑ i P(Object|Bounding Box)*P(Grid j|Bounding Box)*P(Class i|Grid j,Object)

[0064] For example, after comparing the conditional probabilities of an object selected by a prediction box belonging to a certain object category, it is found that the object category is dog with the highest probability. Since the color of the object category dog ​​is blue, among all prediction boxes that are close to the blue box in terms of location and size, the blue box has the highest probability of completely selecting an object. Therefore, the blue box is finally displayed and its corresponding category is marked as dog.

[0065] Step S102 further includes:

[0066] In the process of continuously tracking the target object, when the object is occluded, the speed and position information of the object in the next frame is predicted according to the speed and position information calculated from one or more frames before the video frame, and the object information corresponding to the predicted bounding box is determined.

[0067] Wherein, due to the deviation between the state transition matrix and the real situation, the prediction error of the Kalman filter algorithm is continuously accumulated in the process of calculating the speed and position information of the object in the next frame, which reduces the accuracy of the prediction value and increases the variance. Moreover, when the target object enters the screen at least one frame in advance, the uncertainty of the corresponding entity motion is large. However, after at least one frame of information collection, the uncertainty of the prediction of the possible motion trajectory of the corresponding entity in the next frame of the object will be smaller, so a more accurate prediction estimate of the motion of the corresponding entity can be obtained, and at least one predicted bounding box is generated and the corresponding object information is determined. However, when the screen is occluded, the predicted bounding box of the Kalman filter algorithm cannot continuously track the correct entity detection box, and with the passage of time, the uncertainty of the prediction result of the Kalman filter algorithm will become larger and larger. Even if the object is no longer occluded, it is also difficult to restore the relationship between the predicted bounding box and the corresponding entity, resulting in the loss of the continuously tracked target object.

[0068] The step S103 further comprises:

[0069] Step S301: In the process of continuously tracking the target object, when the object returns to the field of view after being occluded, the object framed by the detection box is determined again and the object category is identified.

[0070] Optionally, the state of the target object is measured again by using the YOLO algorithm, which can reduce the variance and improve the accuracy.

[0071] Step S302: The predicted bounding box is compared with the detection box in the next frame of the picture, and the degree to which the specified detection box in the next frame of the picture matches the corresponding entity is evaluated; if the coincidence degree of the object information determined by the predicted bounding box and the object identified by the detection box is high when the object is not occluded, the classification of all detection boxes in the next frame of the picture to the corresponding entity is completed.

[0072] Step S303: If the coincidence degree is low, the positional relationship between the predicted bounding box and the detection box (i.e. the motion information of the person) is evaluated, and the features of the entity are compared with the features identified by the detection box (i.e. the full color information of the person), and the closeness of the relationship between the predicted bounding box and the detection box is evaluated in combination with the motion information and full color information of the person, and the detection box is assigned to the existing entity according to the evaluation score.

[0073] Optionally, the DeepSort algorithm is used to improve the prediction of the target object in the case of low coincidence, which can reduce the demand for device computing power.

[0074] More specifically, when the DeepSort algorithm is not used, the |id label corresponding to the person (target person) in the video frame who best matches the characteristics of the target object will change frequently in a short time, although this does not affect the identification prediction, the person corresponding to the |id label is still the target person, but the person identified in the video frame does not have continuity and is not the same entity; and if a person who wears similar clothes to the target person appears in the video frame, the id label corresponding to the target person will not change throughout the video. In the above case, it is more likely to follow the wrong target person without using the DeepSort algorithm, and the use of the DeepSort algorithm plus the restriction of spatial variation continuity makes it impossible for the position of the target person to suddenly change from one side of the video frame to the other side, so the identification prediction based on the DeepSort algorithm has better robustness.

[0075] Step S304: If the evaluation score of the closeness of a detection box to all entities is below a certain threshold, it is considered that a new object appears in the video frame, and the object is identified by returning to step S101.

[0076] Step S305: If an entity does not have a detection box matched thereto after a certain time interval threshold A_max, it is considered that the object corresponding to the entity has disappeared from the acquired video, and the entity is deleted to reduce the memory burden. Preferably, the time interval threshold A_max is set to 30 frames.

[0077] The step S104 further comprises:

[0078] The video frame picture corresponding to the labeled and classified object is compared with the pre-shot object picture, and for the person re-identification in the picture, the OSNet algorithm is used to re-identify the person in the picture, and the OSNet algorithm outputs a feature vector with a length of 512, which implicitly represents the whole body feature information of the person; the more similar the feature vectors corresponding to the two pictures are, the greater the probability that the persons in the two pictures are the same person or have more similar appearance features.

[0079] More specifically, still taking the person (target person) in the video frame who best matches the characteristics of the target object as an example, the OSNet algorithm is used to re-identify and output a feature vector with a length of 512 of the target person, and the distance L2 between the feature vector of each frame picture and the feature vector of the target person is calculated, i.e.

[0080] L2(k)=∑ i (d 0，i -dk，i ) 2

[0081] wherein i corresponds to each item in the feature vector of the target person, d 0，i is the i-th item in the feature vector of the target person, d k，i is the i-th item in the feature vector of the k-th person in the frame of the video.

[0082] Take the item with the minimum L2 median, i.e., the person most similar to the target person.

[0083] Based on the same inventive concept, the present application also provides a device embodiment corresponding to the method of embodiment one, which is described in detail in embodiment two.

[0084] Embodiment two

[0085] In this embodiment, a target following device is provided, as shown in the figure, which comprises an identification module, a prediction module, a matching module and a re-identification module. Figure 2 The identification module divides the acquired picture (image) of at least one frame in the video into at least one grid, and a detection frame is arranged in the grid. When there is an object in the grid (i.e., when there is an object in the video), the conditional probability of the detection frame selecting a certain type of object is calculated. The object type corresponding to the maximum conditional probability is determined as the object type corresponding to the detection frame by comparing the conditional probabilities, and the object type is marked.

[0086] The prediction module is used to predict the position and moving speed information of the object in the next frame according to the object position and moving speed information calculated from one or more previous frames in the video, and generate at least one prediction frame.

[0087] The matching module is used to compare the prediction frame with the detection frame of the next frame to determine the coincidence degree of the two, and verify the accuracy of the classification marked according to the object type.

[0088] The re-identification module is used to compare the entity corresponding to the marked object with the entity corresponding to the pre-set object, judge the similarity degree of the feature vectors of the two entity pictures, and determine the degree of similarity of the two.

[0089] The re-identification module is used to compare the entity corresponding to the marked object with the entity corresponding to the pre-set object, judge the similarity degree of the feature vectors of the two entity pictures, and determine the degree of similarity of the two.

[0090] Since the device introduced in this embodiment is the device used to implement the method of embodiment one of the present application, the specific structure and modifications of the device can be understood by those skilled in the art based on the method introduced in embodiment one of the present application, and therefore will not be described here. Any device used to implement the method of embodiment one of the present application belongs to the scope of protection of the present application.

[0091] Based on the same inventive concept, the present application also provides an electronic device corresponding to the method of embodiment one, which is described in detail in embodiment three.

[0092] Embodiment three

[0093] In this embodiment, an electronic device is provided, which comprises:

[0094] at least one processor;

[0095] a memory in communication connection with the at least one processor; and

[0096] a computer program stored in the memory and executable on the processor, wherein the processor implements the method of embodiment one of the present application when executing the computer program.

[0097] Since the electronic device introduced in this embodiment is the device used to implement the method of embodiment one of the present application, the specific implementation of the electronic device and its various forms can be understood by those skilled in the art based on the method introduced in embodiment one of the present application, and therefore the electronic device how to implement the method of embodiment one of the present application will not be introduced in detail here. As long as the device used by those skilled in the art to implement the method of embodiment one of the present application belongs to the scope of the present application.

[0098] Based on the same inventive concept, the present application also provides a non-volatile computer readable storage medium corresponding to the method of embodiment one, which is described in detail in embodiment four.

[0099] Embodiment four

[0100] Those skilled in the art should understand that the embodiments of the present application can be provided as the method, device or computer program product, and the present application can adopt the form of complete hardware embodiment, complete software embodiment, or embodiment combining software and hardware aspects. Therefore, as described below, the present application can adopt the form of computer program product implemented on one or more computer readable storage media (including but not limited to optical storage, magnetic storage, non-volatile semiconductor memory) containing computer usable program code.

[0101] In this embodiment, a non-volatile computer readable storage medium is provided, and the non-volatile computer readable storage medium has a computer program stored thereon, wherein the computer program is used to make the computer execute the method of embodiment one of the present application.

[0102] Based on the same inventive concept, the present application also provides a target following system embodiment, which is described in detail in embodiment five.

[0103] Embodiment five

[0104] In the present embodiment, a target following system is provided, as shown in Figure 3 The system comprises:

[0105] a movable vehicle, having an image acquisition device;

[0106] and a controller capable of maintaining automatic adjustment of the distance between the movable vehicle and the target object, so that the distance remains stable;

[0107] The movable vehicle follows the target object according to the method of the first embodiment of the present application.

[0108] The image acquisition device includes but is not limited to laser radar, binocular camera, single camera. Because of the image acquisition device, the movable vehicle can realize obstacle avoidance function, including:

[0109] Through photographing, the movable vehicle is trained to use the scene, and a training data set and a test set are obtained, and the pictures are marked as blocked and free two labels. As the input of the trained deep learning convolutional neural network AlexNet, the network parameters are fine-tuned. The accuracy of the network AlexNet for two label classification of images is higher than 80%. Because the environment of the movable vehicle is relatively fixed, there is no overfitting and generalization problem faced by the training and testing of autonomous vehicles.

[0110] Among them, the network AlexNet can score each frame of picture in the acquired image, and two scores s1 and s2 respectively represent the possibility of being blocked and the possibility of being free. The softmax function is used for normalization processing to obtain the probability of the movable vehicle being blocked or free, and the sum of the probabilities is 1. The reason for using the softmax function is that it can widen the score gap, which is suitable for classification problems, and has the property of easy derivation, which is convenient for calculation in the training stage.

[0111] In an optional mode of the present embodiment, the threshold of the probability of being blocked or free is set to 0.8. When the probability of the movable vehicle being blocked is greater than 0.8, the movable vehicle is commanded to turn right until the probability of being blocked is less than 0.8, and then the target object is found and followed to achieve the purpose of obstacle avoidance.

[0112] The controller can be a PID controller. The PID algorithm adopted by the controller is divided into three parts, in which P represents proportion, I represents integral, and D represents differential. The mathematical expression form of the PID algorithm is:

[0113]

[0114] wherein e(t) is the difference between the actual value and the target value of the physical quantity; U(t) is the change amplitude of the action quantity used to control the target physical quantity; K c , T i and T d are adjustable parameters, respectively representing the proportional, integral and differential contribution size; K c e(t) corresponds to the proportional part of P, corresponds to the integral part of I, corresponds to the differential part of D.

[0115] For example, if the physical quantity to be controlled is the water temperature, U(t) can be the change of the power of the water heater relative to the power at the previous time; and if the distance between the movable vehicle and the target object is to be controlled, U(t) is the change amplitude of the driving power of the movable vehicle relative to the power at the previous time. When the actual value of the controlled physical quantity reaches the target value, e(t) = 0, at this time U(t) = 0, indicating that the value of the action quantity is appropriate and does not need to be changed.

[0116] In an optional manner of the embodiment, the ideal distance between the movable vehicle and the target object can be determined first, and the height of the bounding box of the target object in the camera of the movable vehicle in the ideal case is obtained as the target value of the automatic control. When the movable vehicle is following, the height of the bounding box of the target object in the camera is the actual value, and the difference between the actual value and the target value is e(t). U(t) is the change value of the output power of the movable vehicle, which is used to adjust the running speed of the movable vehicle. Similarly, the lateral position of the target user in the bounding box of the camera is taken as the target physical quantity, so that the movable vehicle can always face the user, and when the target object turns, the movable vehicle also turns following the target object, and the output powers of the two wheels of the movable vehicle are controlled respectively so that the movable vehicle can always align with the target object as much as possible.

[0117] Although the specific embodiments of the present application are described above, those skilled in the art should understand that the specific embodiments described in the present application are only illustrative, and are not intended to limit the scope of the present application, and equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present application should be covered within the scope of the claims of the present application.

Claims

1. A target following method, applied to a device for following a target object, comprising: Generate at least one detection box, identify the target object selected in the current frame of the acquired video, and determine its category; Generate at least one prediction box to predict the position and speed of the target object in the next frame of the video; Based on the position and speed of the target object, the predicted bounding box is compared with the detection bounding box in the next frame of the video to verify the correctness of the target object's category classification. Determine the degree of similarity between the target object being categorized and a pre-defined object, and identify the target object being followed as the pre-defined object.

2. The method according to claim 1, characterized in that, The generation of at least one detection box is within at least one grid divided in at least one frame of the acquired video.

3. The method according to claim 2, characterized in that, The identification also includes calculating the conditional probability that the detection box selects a certain type of object when there is an object in the grid, comparing the conditional probabilities, determining the object category corresponding to the highest conditional probability as the object category corresponding to the detection box, and labeling the category of the object.

4. The method according to claim 3, characterized in that, Determining the degree of similarity involves comparing the entity corresponding to the marked object with the entity corresponding to a pre-defined object, and judging the similarity of the feature vectors of the two entity images to determine the degree of similarity between them.

5. The method according to claim 1, characterized in that, The prediction also includes, for the object selected by the detection box, predicting its position and speed information in the next frame based on the object's position and speed information calculated from one or more previous frames in the video.

6. A target following device, the device comprising an identification module, a prediction module, a matching module, and a re-identification module; The recognition module is used to generate at least one detection box, identify the target object selected in the current frame of the acquired video, and determine its category; The prediction module is used to generate at least one prediction box to predict the position and speed of the target object in the next frame of the video. The matching module is used to compare the predicted bounding box with the detection bounding box in the next frame of the video based on the position and moving speed of the target object, and to verify the correctness of the target object category classification. The re-identification module is used to determine the similarity between the classified target object and a pre-defined object, and to determine that the following target object is a pre-defined object.

7. An electronic device, comprising: At least one processor; A memory that is communicatively connected to at least one of the processors; as well as A computer program stored on the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the method as described in any one of claims 1 to 5.

8. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is used to cause the computer to perform the method according to any one of claims 1 to 5.

9. A target following system, comprising: A mobile vehicle equipped with image acquisition devices; It also has a controller that maintains automatic adjustment of the distance between the movable vehicle and the target object, so that the distance remains at a stable value; The mobile vehicle follows the target object according to any one of claims 1 to 5.

10. The method according to claim 9, characterized in that, The probability that the movable vehicle is blocked in at least one frame of the acquired image is evaluated, and the movable vehicle is continuously moved until the probability of being blocked is less than a set threshold.

Images