Cake throwing behavior analysis method based on time sequence image analysis

Abstract

The invention discloses a cake throwing behavior analysis method based on time sequence image analysis. The method comprises the following steps: S1, acquiring image data It at the current moment; s2, detecting the position of the cake; s3, tracking the position of the cake; s4, calculating the area change of the cake; s5, calculating a second derivative of the cake track; s6, judging the center coordinates of the cake; and S7, counting a tracking result. According to the method, the size change and the motion trail of a target cake are obtained through a deep learning detection method and a target tracking method, and the target behavior is obtained through analysis by using the mathematical principle on the basis of an existing image processing algorithm in combination with behavior time-space characteristics.

Description

Technical Field

[0001] The present invention relates to the technical field of behavior analysis, and in particular to a pancake-throwing behavior analysis method based on time-series image analysis. Background Art

[0002] Temporal image analysis is a technique that extracts information about image content and dynamics by analyzing changes in time series data. It can be used in many fields, such as video surveillance, medical image analysis, and motion tracking. It provides us with dynamic information that cannot be obtained from static images, helping us to understand and analyze image data more comprehensively.

[0003] There is currently no solution for the behavior analysis of pancake throwing, which can be solved through general methods, such as general behavior recognition methods. However, the current behavior recognition methods are not perfect enough and the effect of behavior analysis on pancake throwing is not good. Summary of the invention

[0004] The purpose of the present invention is to provide a pancake tossing behavior analysis method based on time series image analysis to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for analyzing pancake throwing behavior based on time series image analysis, comprising the following steps:

[0006] S1. Get the current image data I t ;

[0007] S2. Detect the position of the cake;

[0008] S3. Track the location of the pie;

[0009] S4. Calculate the change in the pie area;

[0010] S5. Calculate the second-order derivative of the pie trajectory;

[0011] S6. Determine the center coordinates of the pie;

[0012] S7. Statistical tracking results.

[0013] Preferably, in the process of detecting the position of the pie,

[0014] By sending the image to the detection network, we get the coordinates of the target pie.

[0015] l t (lx1 t ,ly1 t ,lx2 t ,ly2 t )=Detect(I t ).

[0016] Preferably, in the process of tracking the position of the pie,

[0017] According to the detection coordinates, the tracking coordinates of the previous moment are confirmed, and the tracking coordinate values ​​of the current moment are confirmed.

[0018] t t =Track(I t ,t t-1 ).

[0019] Preferably, in the process of calculating the change in the pie area,

[0020] Calculate the difference between the area of ​​the pie at the current moment and the area of ​​the pie at the previous moment, and store it in a queue with a maximum length of 50, AreaDiff.

[0021] area t =|(lx2 t -lx1 t )×(ly2 t -ly1 t )|

[0022] AreaDiff={(area1-area0),…,(area t -area t-1 ),…}.

[0023] Preferably, in the process of calculating the second-order derivative of the pie trajectory,

[0024] According to the tracking center coordinates of the target area at the current moment and the tracking center coordinates of the previous two moments, assuming that the x coordinate is the independent variable and the y coordinate is the dependent variable, the discrete second-order derivative is calculated and stored in a queue D with a maximum length of 50.

[0025]

[0026] D={d0,d1,…}.

[0027] Preferably, in the process of determining the center coordinates of the pie,

[0028] According to the table area range coordinates Table = {(x1, y1), ..., (x5, y5)}, determine the center coordinates of the detection target at the current moment (lx t ,ly t ), the center coordinates of the tracking target (tx t ,ty t ) is within the table range, and stores them in queues Pd and Pt of length 10 respectively.

[0029] Preferably, during the processing of statistical tracking results,

[0030] Count the number of targets inside and outside the boundary in queues Pd and Pt respectively, obtaining the number of times the detected target is inside the boundary CountDetIner, the number of times the detected target is outside the boundary CountDetOuter, the number of times the tracked target is inside the boundary CountTracIner, and the number of times the tracked target is outside the boundary CountTracOute. If CountDetIner > CountDetOuter and CountTracIner > CountTracIner, that is, both the detection result and the tracking result are inside the boundary, count the number of elements greater than 0 in queue AreaDiff; if the number is greater than 25, it is considered that the current behavior is pancake rolling; if CountDetIner < CountDetOuter and CountTracIner < CountTracIner, that is, both the detection result and the tracking result are outside the boundary, count the number of elements greater than 0 in queue D; if the number is greater than 25, it is considered that the current behavior is pancake tossing; if CountDetIner > CountDetOuter and CountTracIner < CountTracIner, that is, the detection result is inside the boundary and the tracking result is outside the boundary, it is considered that the previous pancake tossing cycle ends and a new pancake tossing behavior analysis starts.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] In the present invention, the size change and movement trajectory of the target pancake are obtained through the deep learning detection method and the target tracking method. Combining the spatio-temporal characteristics of the behavior, based on the existing image processing algorithms, mathematical principles are used for analysis to obtain the target behavior. Description of the Drawings

[0033] Figure 1 It is a flowchart of the pancake tossing behavior analysis method of the present invention. Detailed Embodiment

[0034] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0035] Please refer to Figure 1 , an embodiment provided by the present invention:

[0036] A pancake tossing behavior analysis method based on sequential image analysis, comprising the following steps:

[0037] S1. Get the current image data I t ;

[0038] The image monitoring system is used to monitor the pancake-throwing working environment, obtain image data taken under good lighting conditions, and clear objects in the shooting environment that are similar in shape to the pancakes, effectively avoiding subsequent misidentification.

[0039] S2. Detect the position of the cake;

[0040] The target pie position is detected based on the two-stage and one-stage target detection algorithms. In the two-stage, the region is generated first, which is called region proposal, and then the sample is classified through the convolutional neural network. The task flow is: feature extraction--generation of RP--classification / positioning regression. The common two-stage target detection algorithms include: R-CNN, SPP-Net, FastR-CNN, FasterR-CNN and R-FCN.

[0041] In addition, One stage can directly extract features on the network to predict object classification and location. Its task process is: feature extraction-classification / positioning regression. Common One stage target detection algorithms include: OverFeat, YOLOv1, YOLOv2, YOLOv3, SSD and RetinaNet.

[0042] S3. Track the location of the pie;

[0043] Given the size and position of the target (pie) in the initial frame of the video sequence, the size and position of the target in the subsequent frames are predicted. By inputting the initial target frame, a number of candidate frames are generated in the next frame, the features of such candidate frames are extracted, and then the candidate frames are evaluated to find the candidate frame with the highest score as the predicted target, or multiple prediction values ​​are fused to obtain a more optimized predicted target; among them, feature extraction (discriminative feature representation) is one of the keys to target tracking, and commonly used features are divided into two types: manually designed features and deep features. Commonly used manually designed features include grayscale features, oriented gradient histograms, Haar features, scale-invariant features, etc. Unlike manually designed features, deep features are features learned from a large number of training samples, which are more discriminative than manually designed features. Therefore, tracking methods using deep features can usually easily achieve good results.

[0044] S4. Calculate the change in the pie area;

[0045] S5. Calculate the second-order derivative of the pie trajectory;

[0046] S6. Determine the center coordinates of the pie;

[0047] S7. Statistical tracking results.

[0048] The label detection method and tracking method in this method can be replaced by various methods of the same type, the second-order derivative can also be replaced by other mathematical methods, and the constraints in this method can also be increased accordingly. For example, circularity and color can be added on the basis of area and coordinates in this technical solution.

[0049] In the process of detecting the position of the pie,

[0050] By sending the image to the detection network, we get the coordinates of the target pie.

[0051] l t (lx1 t ,ly1 t ,lx2 t ,ly2 t )=Detect(I t ).

[0052] In the process of tracking the position of the pie,

[0053] According to the detection coordinates, the tracking coordinates of the previous moment are confirmed, and the tracking coordinate values ​​of the current moment are confirmed.

[0054] t t =Track(I t ,t t-1 ).

[0055] In the process of calculating the change in the size of the pie area,

[0056] Calculate the difference between the area of ​​the pie at the current moment and the area of ​​the pie at the previous moment, and store it in a queue with a maximum length of 50, AreaDiff.

[0057] area t =|(lx2 t -lx1 t )×(ly2 t -ly1 t )|

[0058] AreaDiff={(area1-area0),…,(area t -area t-1 ),d}.

[0059] In the process of calculating the second derivative of the pie trajectory,

[0060] According to the tracking center coordinates of the target area at the current moment and the tracking center coordinates of the previous two moments, assuming that the x coordinate is the independent variable and the y coordinate is the dependent variable, calculate the discrete second derivative and store it in queue D with a maximum length of 50.

[0061]

[0062] D = {d0, d1, …}.

[0063] During the process of discriminating the center coordinates of the pie,

[0064] According to the table area range coordinates Table = {(x1, y1), …, (x5, y5)}, judge whether the center coordinates (lx t , ly t ) of the detected target at the current moment and the center coordinates (tx t , ty t ) of the tracked target are within the table range, and store them in queues Pd and Pt with a length of 10 respectively.

[0065] During the process of counting the tracking results,

[0066] Count the number of targets inside and outside the boundary in queues Pd and Pt respectively, to obtain the number of times the detected target is inside the boundary CountDetIner, the number of times the detected target is outside the boundary CountDetOuter, the number of times the tracked target is inside the boundary CountTracIner, and the number of times the tracked target is outside the boundary CountTracOute. If CountDetIner > CountDetOuter and CountTracIner > CountTracIner, that is, both the detection result and the tracking result are inside the boundary, count the number of elements greater than 0 in queue AreaDiff; if the number is greater than 25, it is considered that the current behavior is rolling the pie; if CountDetIner < CountDetOuter and CountTracIner < CountTracIner, that is, both the detection result and the tracking result are outside the boundary, count the number of elements greater than 0 in queue D; if the number is greater than 25, it is considered that the current behavior is throwing the pie; if CountDetIner > CountDetOuter and CountTracIner < CountTracIner, that is, the detection result is inside the boundary and the tracking result is outside the boundary, it is considered that the previous throwing pie cycle ends and a new throwing pie behavior analysis starts.

[0067] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0068] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for analyzing pancake throwing behavior based on time series image analysis, characterized in that: It includes the following steps: S1. Get the current image data I t ; S2. Detect the position of the pie; S3. Track the position of the pie; S4. Calculate the change in the area of the pie; S5. Calculate the second derivative of the pie trajectory; S6. Discriminate the center coordinates of the pie; S7. Statistically analyze the tracking results.

2. The method for analyzing pancake throwing behavior based on time series image analysis according to claim 1, characterized in that: During the process of detecting the position of the pie, The coordinates of the target pie are obtained by sending the image into the detection network. l t (lx1 t ,ly1 t ,lx2 t ,ly2 t )=Detect(I t )。 3. The method for analyzing pancake throwing behavior based on time series image analysis according to claim 1, characterized in that: During the process of tracking the position of the pie, Based on the detection coordinates and the tracking coordinates at the previous moment, confirm the tracking coordinate value at the current moment. t t =Track(I t ,t t-1 )。 4. The method for analyzing pancake throwing behavior based on time series image analysis according to claim 1, characterized in that: During the process of calculating the change in the area of the pie, Calculate the difference between the area of the pie at the current moment and the area of the pie at the previous moment, and store it in the queue AreaDiff with a maximum length of 50. area t =|(lx2 t -lx1 t )×(ly2 t -ly1 t )| AreaDiff={(area1-area0),…,(area t -area t-1 ),…}。 5. The method for analyzing pancake throwing behavior based on time series image analysis according to claim 1, characterized in that: During the process of calculating the second derivative of the pie trajectory, Based on the tracking center coordinates of the target area at the current moment and the tracking center coordinates at the previous two moments, assuming the x coordinate is the independent variable and the y coordinate is the dependent variable, calculate the discrete second derivative and store it in the queue D with a maximum length of 50. D = {d0, d1, …}.

6. The method for analyzing pancake throwing behavior based on time series image analysis according to claim 1, characterized in that: During the process of discriminating the center coordinates of the pie, According to the table area range coordinates Table = {(x1, y1), ..., (x5, y5)}, determine the center coordinates of the detection target at the current moment (lx t ,ly t ), the center coordinates of the tracking target (tx t ,ty t ) is within the table range, and stores them in queues Pd and Pt of length 10 respectively.

7. The method for analyzing pancake throwing behavior based on time series image analysis according to claim 1, characterized in that: During the process of statistically analyzing the tracking results, Respectively count the number of targets inside and outside the boundary in the queues Pd and Pt to obtain the number of times the detected target is inside the boundary CountDetIner, the number of times the detected target is outside the boundary CountDetOuter, the number of times the tracked target is inside the boundary CountTracIner, and the number of times the tracked target is outside the boundary CountTracOute. If CountDetIner > CountDetOuter and CountTracIner > CountTracIner, that is, both the detection result and the tracking result are inside the boundary, count the number of values greater than 0 in the queue AreaDiff; if the number is greater than 25, it is considered that the current behavior is rolling the pie; if CountDetIner < CountDetOuter and CountTracIner < CountTracIner, that is, both the detection result and the tracking result are outside the boundary, count the number of values greater than 0 in the queue D; if the number is greater than 25, it is considered that the current behavior is throwing the pie; if CountDetIner > CountDetOuter and CountTracIner < CountTracIner, that is, the detection result is inside the boundary and the tracking result is outside the boundary, it is considered that the previous pie-throwing cycle ends and a new pie-throwing behavior analysis starts.

Images