A device and method for multi-target tracking and behavior determination of handheld objects

CN111967352BActive Publication Date: 2024-11-22BEIJING ZHIYUANDA TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010757549.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-07-31
Publication Date
2024-11-22
Estimated Expiration
2040-07-31

AI Technical Summary

Technical Problem

[0004]随着人工智能的快速发展,目标跟踪算法得到了快速发展,特别是在行人、车辆跟踪等多目标跟踪领域取得了很好的效果,但是对于其他目标的跟踪,例如专利201310568186.6这类的智能冰箱的使用,尤其是手持物品的多目标跟踪,由于物体被遮挡,跟踪时很容易被误判为另一个目标

Benefits of technology

[0024]通过采用上述技术方案,本装置和方法通过将手掌,手掌-手持物品,物品三种目标分情况检测,并通过检测逻辑,在不更改检测算法的前提下,提升识别率。在行为判定阶段,通过行为判定逻辑,剔除错误轨迹,提升正确率。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111967352B_ABST
    Figure CN111967352B_ABST
Patent Text Reader

Abstract

The present invention provides a device and method for multi-target tracking and behavior determination of handheld objects. The entire system is divided into an imaging module, a labeling module, a training module, a recognition module, a trajectory tracking module, and a behavior determination module. Images are acquired through a network camera, and image data is uploaded to a server. A program running in the server determines whether a person is holding an object, and finally determines whether to place an object or take an object. The device and method detect three types of targets, namely palm, palm-handheld object, and object, in different situations, and improves the recognition rate through detection logic without changing the detection algorithm. In the behavior determination stage, the behavior determination logic is used to eliminate wrong trajectories and improve the accuracy rate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a device and method for multi-target tracking and behavior determination of handheld objects. Background Art

[0002] Smart home (smart home, home automation) is based on the house as a platform, using integrated wiring technology, network communication technology, security technology, automatic control technology, audio and video technology to integrate facilities related to home life, build an efficient management system for residential facilities and family schedule affairs, improve home safety, convenience, comfort, artistry, and achieve an environmentally friendly and energy-saving living environment.

[0003] In recent years, with the popularization of smart homes, more and more household appliances have been upgraded to intelligent ones. However, the freezers, refrigerators, food cabinets or storage cabinets used to store food in existing smart homes only have some functions of the front end and lack the intelligent support of the back end. They are insufficiently intelligent and have fewer functions.

[0004] With the rapid development of artificial intelligence, target tracking algorithms have developed rapidly, especially in the field of multi-target tracking such as pedestrian and vehicle tracking, and have achieved good results. However, for the tracking of other targets, such as the use of smart refrigerators such as patent 201310568186.6, especially the multi-target tracking of handheld objects, due to the obstruction of the object, it is easy to be misjudged as another target during tracking. Summary of the invention

[0005] The purpose of the present invention is to provide a device and method for multi-target tracking and behavior determination of handheld objects in order to solve the above-mentioned problems, thereby reducing the possibility of misjudgment of targets.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] A device for multi-target tracking and behavior determination of handheld objects, comprising the following parts:

[0008] An imaging module, used to photograph and capture moving targets within the field of view;

[0009] The labeling module is used to label the palm, label the palm-held object, label the object, generate labeling image data, and provide training data to the training module;

[0010] A training module, used for training the image data annotated by the annotation module;

[0011] The recognition module is used to recognize the target object using the model trained by the training module and give the recognized target position;

[0012] A trajectory tracking module is used to determine whether the target object is a suspected target in a frame of image based on the target object trajectory;

[0013] The judgment module is used to limit the trajectory in the continuous frame images of the suspected target through the preset behavior judgment logic, eliminate some trajectories that obviously do not conform to the behavior, and finally give the behavior judgment of different trajectories.

[0014] A method for multi-target tracking and behavior determination of handheld objects, using a network camera to shoot and capture moving targets within a field of view, respectively marking the palm, marking the palm-handheld object, marking the object, generating labeled image data, training the image data, identifying the target object with the trained model, giving the identified target position, judging whether the target object is a suspected target in a frame image through the target object trajectory, limiting the trajectory in the continuous frame images of the suspected target through a pre-set behavior determination logic, eliminating some trajectories that obviously do not conform to the behavior, and finally giving behavior determinations for different trajectories.

[0015] Furthermore, the image data is trained using an rcnn algorithm or a yolo algorithm.

[0016] Furthermore, the trained model is used to identify the target objects, including palm A, palm-held object B, and object C, and the identified target position box, category class, and score score are given, which are respectively the target position box_A of the palm, the category class_A of the palm, the score score_A of the palm, the target position box_B of the palm-held object, the category class_B of the palm-handheld object, the score score_B of the palm-handheld object, the target position box_C of the object, the category class_C of the object, and the score score_C of the object.

[0017] Furthermore, through the target object trajectory, in a frame image, it is determined whether the target object is a suspected target and whether it contains class_A. If not, it is considered that there is no target in this frame and waits for the detection of the next frame. If class_A is contained, the relevant class_A, box_A, score_A data are saved, and class_B and class_C are checked. If they exist, the relevant class_B, box_B, score_B and class_C, box_C, score_C data are saved, and the intersection and union (IOU) of different class_A and class_C targets is used. A_C To determine which targets are integrated and eliminate the wrong targets, Multiple IOUs A_C The composition set E(IOU A_C ), by suppressing nms with non-maximum values, removing the set E(IOU A_C ) and repeated targets in the set E(class_B) of multiple class_B.

[0018] Furthermore, in the continuous frame images of the suspected target, the trajectory is limited by the pre-set behavior judgment logic, and some trajectories that obviously do not conform to the behavior are eliminated, and finally the behavior judgment of different trajectories is given. The trajectory has three states: take, put, and uncertain. now The center point and the target box of the previous frame pre The position difference confirms the state of the trajectory. Let the suspected target set of the current frame be E(object), the current trajectory set be E(track), and the trend determination be

[0019] ifbox now .x>box pre .xandbox now .y>box pre .y:take

[0020] ifbox now .x <box pre .xandbox now .y <box pre .y:put

[0021] if(box now .x-box pre .x)*(box now .y-box pre .y)<0:unsure

[0022] If the movement trend of the first object in E(object) is consistent with a track in E(track), the object belongs to the track and the track is updated. If they are inconsistent, select targets in E(object) in turn and compare them with the track of E(track). If there is no match, the current E(object) is listed as a suspected track and added to E(track). When there are multiple consecutive frames in E(track) with the same trend, the behavior is confirmed and the track is deleted.

[0023] The beneficial effects of the present invention are:

[0024] By adopting the above technical solution, the device and method detects three types of targets, palm, palm-held object, and object, and improves the recognition rate through detection logic without changing the detection algorithm. In the behavior determination stage, the behavior determination logic is used to eliminate wrong trajectories and improve the accuracy rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0026] Figure 1 It is an overall implementation flow chart of the multi-target tracking and behavior determination method of the handheld object of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0028] This example provides a device and method for multi-target tracking and behavior determination of handheld objects. Figure 1 The process is to obtain images through a network camera, upload the image data to a server, and then a program running on the server determines whether a person is holding an object, and finally determines whether to place the object or take it.

[0029] Specifically, the whole system is divided into imaging module, labeling module, training module, recognition module, trajectory tracking module, and behavior determination module. The imaging module is placed in a fixed position through an external network camera to shoot and capture moving targets within the field of view.

[0030] The labeling module provides training data to the training module. It is divided into three parts. The first part is to label the palm, the second part is to label the hand-held object, and the third part is to label different types of objects.

[0031] The training module can select different models for training the labeled images. For example, the two-step rcnn series model or the one-step yolo series. The training categories include three categories: palm, palm-held objects, and objects. The first two categories are fixed, and the last category of objects is selected according to actual needs. It can be one object or multiple objects.

[0032] The recognition module uses the trained model to recognize the image. The objects include palm A, palm-held object B, and held object C. It gives the recognized target location box, category class, and score. The reason for recognizing the three objects is that the handheld object itself is easily blocked and has a low recognition rate. The palm itself has a high recognition rate. Using the palm as an auxiliary judgment condition can easily reduce the misjudgment rate.

[0033] Trajectory tracking module, the specific steps are as follows:

[0034] 1. Target category and position determination. Determine whether the category class returned by the recognition module contains the palm class_A. If not, it is considered that there is no target in this frame and waits for the next frame detection. If it contains class_A, save the data of box_A and score_A, and continue to check class_B and class_C. If it exists, save the relevant class, box, score and other related data.

[0035] 2. Combination of targets. Since it is a multi-target detection, there may be multiple class_A, class_B, and class_C. You can use the IOU of different class_A and class_C targets to find the target. A_C To determine which targets are integrated, and thus eliminate the wrong targets. The formula is as follows:

[0036]

[0037] Multiple IOUs A_C The composition set E(IOU A_C ), set E(IOU A_C ) Internally press IOU A_C Sort by values ​​in ascending order.

[0038] 3. Deduplication. Since the set E(IOU AC ) and the set E(class_B) of multiple class_Bs all represent palm-handheld objects, so there may be duplication. By suppressing nms with non-maximum value, duplicate targets in both sets are removed. nms is generally used to remove overlapping boxes through score. In this case, the set E(class_B) contains score_B of category B, but the set E(class_B) contains IOU. A_C Cannot compare with score_B, can set Compare again.

[0039] The behavior determination module confirms the trajectory in consecutive frames. There are three states of trajectory: take, put, and uncertain. The state is confirmed by trend determination, that is, relying on the current target box. nowThe center point is compared with the target box in the previous frame pre Assuming that the set of suspected targets in the current frame is E(object) and the set of current tracks is E(track), the specific determination steps are as follows:

[0040] 1. Trend determination, ifbox now .x>box pre .xandbox now .y>box pre .y:take

[0041] ifbox now .x <box pre .xandbox now .y <box pre .y:put

[0042] if(box now .x-box pre .x)*(box now .y-box pre .y)<0:unsure

[0043] 2. Track update: If the movement trend of the first object obj in E(object) is consistent with a track track_x in E(track), then the object is considered to belong to the track and the track is updated. If they are inconsistent, then select the target in E(object) and compare it with the track of E(track). If there is no match, the current E(object) is listed as a suspected track and added to E(track).

[0044] 3. Track culling: When there are multiple consecutive frames in E(track) showing a trend, the behavior is determined and the track is deleted.

[0045] The above embodiments of the present invention are not intended to limit the protection scope of the present invention, and the implementation modes of the present invention are not limited thereto. All other modifications, replacements or changes made to the above structures of the present invention based on the above contents of the present invention, in accordance with common technical knowledge and customary means in the art, without departing from the above basic technical ideas of the present invention, should fall within the protection scope of the present invention.

Claims

1. A method for multi-target tracking and behavior determination of handheld objects, characterized by: Through the network camera, the moving target within the field of view is photographed and captured, and the palm, palm-held object and object are marked respectively to generate the marked image data, train the image data, identify the target object with the trained model, give the identified target position, and determine whether the target object is a suspected target in a frame image through the trajectory of the target object. In the continuous frame images of the suspected target, the trajectory is limited through the pre-set behavior judgment logic, and some trajectories that obviously do not conform to the behavior are eliminated, and finally the behavior judgment of different trajectories is given; Use the trained model to identify the target objects, including palm A, palm-held object B, and object C, and give the identified target position box, category class, and score score, which are respectively the target position box_A of the palm, the category class_A of the palm, the score score_A of the palm, the target position box_B of the palm-held object, the category class_B of the palm-held object, the score score_B of the palm-held object, the target position box_C of the object, the category class_C of the object, and the score score_C of the object; Through the trajectory of the target object, in one frame of image, determine whether the target object is a suspected target and whether it contains class_A. If not, it is considered that there is no target in this frame and wait for the detection of the next frame. If class_A is contained, the relevant class_A, box_A, score_A data are saved, and class_B and class_C are checked. If they exist, the relevant class_B, box_B, score_B and class_C, box_C, score_C data are saved. The intersection of different class_A and class_C targets IOU_(A_C) is used to determine which targets are integrated and eliminate wrong targets. IOU_(A_C) = (box_A∩box_C) / (box_A∪box_C). Multiple groups of IOU_(A_C) constitute the set E(IOU_(A_C)). Through non-maximum suppression nms, the repeated targets in the set E(IOU_(A_C)) and the set E(class_B) of multiple class_Bs are removed.

2. The method for multi-target tracking and behavior determination of handheld objects according to claim 1, characterized in that: in, Train the image data using the rcnn algorithm or the yolo algorithm.

3. The method for multi-target tracking and behavior determination of handheld objects according to claim 1, characterized in that: in, In the continuous frame images of the suspected target, the trajectory is limited by the pre-set behavior judgment logic, and some trajectories that obviously do not conform to the behavior are eliminated. Finally, the behavior judgment of different trajectories is given. The trajectory has three states: take, put, and uncertain. now The center point and the target box of the previous frame pre The position difference confirms the state of the trajectory. Let the suspected target set of the current frame be E(object), the current trajectory set be E(track), and the trend determination be ifbox now .x>box pre .x and box now .y>box pre .y:take ifbox now .x<box pre .x and box now .y<box pre .y:put if(box now .x-box pre .x)*(box now .y-box pre .y)<0:unsure If the movement trend of the first object in E(object) is consistent with a track in E(track), the object belongs to the track and the track is updated. If they are inconsistent, the targets in E(object) are selected in turn to compare with the tracks of E(track). If there is no match, the current E(object) is listed as a suspected track and added to E(track). When there are multiple consecutive frames in E(track) with the same trend, the behavior is confirmed and the track is deleted.