Method and system for continuous tracking of occluded targets based on video stream data

Abstract

The present invention discloses a method and system for continuously tracking an occluded target based on video stream data, including: S1: using pulse neurons to collect all video stream data in the same neighborhood space, and judging whether there is an occluded target at the pixel point position contained therein; S2: arranging the video stream data collected by the pulse neurons according to the pixel point index to obtain a two-dimensional image containing the occluded target; S3: taking the feature point of the two-dimensional image as the center, intercepting the window image, inputting the metric neural network model, and obtaining the type of the occluded target; S4: converting the pixel index of the occluded target to obtain the actual position of the occluded target; S5: arranging the actual position of the occluded target according to the time series to obtain the target point trace, and performing correction and interpolation operations on the point trace to obtain the processed target point trace. The present invention uses pulse neurons to construct a two-dimensional image to realize accurate identification of the occluded target; and uses coordinate conversion and point trace correction to realize accurate positioning and tracking of the occluded target.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular to a method and system for continuously tracking an occluded target based on video stream data. Background Art

[0002] Video target tracking refers to the use of computer vision technology to accurately track the target of interest. Video target tracking has a wide range of applications in autonomous driving, human-computer interaction, and video surveillance. Existing video target tracking methods focus on using deep learning methods to identify video images to capture the target of interest and then achieve stable tracking of the target.

[0003] However, due to the limitations of video acquisition conditions and the transmission process, the video stream data has problems of loss and disorder, and the resulting video images have a certain degree of pixel loss and occlusion of the target of interest. Due to the limited number of pixels, the identity of the occluded target cannot be accurately identified. In the process of tracking the occluded target, there is a certain probability of target loss, target trajectory interruption, and significant deviation of the target trajectory. The current method for tracking occluded targets is mainly to improve the deep learning method to improve the recognition accuracy of occluded targets, but the effect is limited. Summary of the invention

[0004] The present invention provides a method and system for continuously tracking an occluded target based on video stream data. Based on the video stream data of each frame, a two-dimensional image containing the occluded target is collected and constructed through pulse neurons, and the video stream data of each frame is processed to achieve accurate target classification and continuous and precise positioning, thereby achieving continuous target tracking.

[0005] To achieve the above object, the present invention is implemented through the following technical solutions:

[0006] A method for continuously tracking an occluded target based on video stream data comprises the following steps:

[0007] S1: Use pulse neurons to collect all video stream data in the same neighborhood space and determine whether there is an occluded target at the location of the included pixel points;

[0008] S2: Arrange the video stream data collected by the pulse neuron according to the pixel index to obtain a two-dimensional image containing the occluded target;

[0009] S3: Taking the feature points of the two-dimensional image as the center, intercept the window image, input the metric neural network model, and obtain the occluded target type;

[0010] S4: converting the pixel index of the occluded target to obtain the actual position of the occluded target;

[0011] S5: Arrange the actual positions of the occluded targets in time series to obtain target point traces, and perform correction and interpolation operations on the target point traces to obtain processed target point traces.

[0012] As a further improvement method of the present invention:

[0013] Optionally, in step S1, using pulse neurons to collect all video stream data in the same neighborhood space, and determining whether there is an occlusion target at the position of the included pixel points, includes:

[0014] S101: Obtaining video stream data ,in Indicates the video stream index;

[0015] The video stream data will be obtained Input spike neuron ;

[0016] S102: Determine video stream data And the video stream data to be retrieved Whether they belong to the same neighborhood space:

[0017] ;

[0018] in, Indicates the video stream data to be retrieved. Indicates the index of the video stream to be retrieved. and Respectively and Contains the pixel index value, and Respectively and The frame index of represents the neighborhood size, express and belong to the same neighborhood space, express and They do not belong to the same neighborhood space;

[0019] S103: If the video stream data and the video stream data to be retrieved belong to the same neighborhood space, Input spike neuron ;

[0020] S104: repeating steps S102 and S103 to traverse all the video stream data to be retrieved;

[0021] If a spiking neuron The membrane potential Reaching the activation threshold , it means that in the video stream data Pixels There is an occluding target at the position.

[0022] Spiking Neurons Contains and retrieves video stream data All video stream data belong to the same neighborhood space.

[0023] Optionally, in step S104, it is characterized in that the pulse neuron The membrane potential It is expressed as:

[0024] ;

[0025] in, Indicates the change in membrane potential of a pulse neuron caused by a single video stream data. Indicates the number of video stream data contained in the spiking neuron.

[0026] Optionally, in step S101, the video stream data It is expressed as:

[0027]

[0028] in, Represents video stream data, Indicates the video stream index. Represents the frame index, Represents the pixel index, Represents the gray value of a pixel;

[0029] Video stream data The choice can be determined based on the initial positioning of the target or based on experience.

[0030] Optionally, in step S102, the neighborhood size Determination, including:

[0031] Neighborhood size It is related to the target size and is determined according to the specific scenario.

[0032] Optionally, in step S104, the activation threshold Determination, including:

[0033] Activation Threshold ,in, The minimum number of pixels required for occluded target recognition in the current scene;

[0034] Activation Threshold It is related to the minimum number of pixels required to identify the occluded object in the current scene;

[0035] When the number of video stream data stored in the spiking neuron exceeds , the membrane potential of the spiking neuron Reaching the activation threshold ;

[0036] Membrane potential only Reaching the activation threshold The pulse neurons traverse the video stream data and process it, which is conducive to filtering out the video stream data that does not meet the requirements and only processing the identifiable video stream data to improve efficiency.

[0037] Optionally, in step S3, taking the feature point of the two-dimensional image as the center, intercepting the window image, inputting the metric neural network model, and obtaining the occluded target type include:

[0038] S301: Construct a window with the feature points of the two-dimensional image as the center:

[0039] ;

[0040] in, represents the pixel index of the window, Indicates the index value of the pixel where the feature point is located. Indicates the size of the window;

[0041] S302: Capture the window image and input it into the trained metric neural network model, which classifies the image according to the cosine distance:

[0042] ;

[0043] in, Represents the cosine distance between the window image and the classification target, Represents the projection space obtained by inputting the window image into the neural network model, Represents the projection space of classification targets;

[0044] according to Determines the type of occlusion target.

[0045] Optionally, in step S4, converting the pixel index of the occluded target to obtain the actual position of the occluded target includes:

[0046] S401: Establishing a matching relationship between the pixel coordinate system and the Cartesian coordinate system:

[0047] ;

[0048] in, represents the pixel index of the calibration target, represents the coordinates of the calibration target in the Cartesian coordinate system, Represents the transformation from pixel coordinate system to image coordinate system, Represents the transformation from the image coordinate system to the camera coordinate system, Represents the transformation from the camera coordinate system to the Cartesian coordinate system;

[0049] S402: Establishing a relationship matrix between the occluded target and the calibration target in the pixel coordinate system:

[0050] ;

[0051] in, represents the pixel index of the occluded target, represents the scaling factor, represents the displacement factor;

[0052] S403: Determine the position of the occluding target in the Cartesian coordinate system:

[0053] ;

[0054] in, Represents the coordinates of the occluding target in the Cartesian coordinate system, Representation Matrix The inverse matrix of

[0055] The actual position of the occluded target is .

[0056] Optionally, in step S5, the actual positions of the blocked targets are arranged in time series to obtain target point traces, and the target point traces are corrected and interpolated to obtain processed target point traces, including:

[0057] S501: Arrange the actual positions of the blocked targets in time series to obtain target point traces;

[0058] S502: Determine the target preset route according to the target point trace and the type of the occluded target obtained in step S3; bind the target point trace to the target preset route, correct the target point trace that obviously deviates from the route, and perform interpolation operations on the interrupted target point trace to obtain the processed target point trace.

[0059] The present invention also provides a system for continuously tracking an occluded target based on video stream data, comprising:

[0060] Video stream data collection module: video stream data judgment in the same neighborhood space, video stream data input into pulse neurons, and judge whether the membrane potential of the pulse neurons reaches the activation threshold;

[0061] Two-dimensional image generation module: Arrange the video stream data in the pulse neurons to generate a two-dimensional image containing the occluded target;

[0062] Occlusion target type identification module: captures the window image and identifies the occlusion target type;

[0063] Occlusion target coordinate conversion module: converts the pixel index of the occlusion target to determine the actual position of the occlusion target;

[0064] Target point trace construction module: obtain the target point trace, and perform correction and interpolation operations on the target point trace. Beneficial Effects

[0065] The present invention uses pulse neurons to collect video stream data to ensure that the number of pixel points of the video stream data meets the recognition requirements; constructs a two-dimensional image, captures a window image based on feature points, and accurately classifies occluded targets; converts the pixel index of the occluded target to improve positioning accuracy; and performs point trace correction and interpolation to improve the quality of target tracking.

[0066] The present invention defines the neighborhood space discrimination of video stream data, and accurately selects video stream data belonging to the same neighborhood space; uses pulse neurons to accurately collect video stream data required for identification; constructs a two-dimensional image only for video stream data that meets the identification requirements, thereby improving work efficiency and ensuring that video stream data will not be missed due to disordered transmission; captures a window image centered on a two-dimensional image feature point, which is beneficial for extracting significant features of the image; and uses a metric neural network model, which is beneficial for accurate image classification.

[0067] The present invention accurately establishes a matching relationship between a pixel coordinate system and a Cartesian coordinate system based on a calibration target; based on a relationship matrix between an occluded target and a calibration target in a pixel coordinate system, the position of the occluded target in the Cartesian coordinate system is derived, which is conducive to achieving accurate positioning of the occluded target.

[0068] The present invention binds the target point trace to the preset route, which is beneficial to the prior information of the preset route, corrects the point trace, and obtains high-quality point trace. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A schematic flow chart of a method for continuously tracking an occluded target based on video stream data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention belong to the protection scope of the present invention.

[0071] Embodiment 1:

[0072] A method for continuous tracking of occluded targets based on video stream data, such as Figure 1 As shown, the following steps are included:

[0073] S1: Use pulse neurons to collect all video stream data in the same neighborhood space and determine whether there is an occluded target at the location of the included pixel points;

[0074] S101: Obtaining video stream data ,in Indicates the video stream index;

[0075] The video stream data will be obtained Input spike neuron ;

[0076] S102: Determine video stream data And the video stream data to be retrieved Whether they belong to the same neighborhood space:

[0077] ;

[0078] in, Indicates the video stream data to be retrieved. Indicates the index of the video stream to be retrieved. and Respectively and Contains the pixel index value, and Respectively and The frame index of represents the neighborhood size, express and belong to the same neighborhood space, express and They do not belong to the same neighborhood space;

[0079] S103: If the video stream data and the video stream data to be retrieved belong to the same neighborhood space, Input spike neuron ;

[0080] S104: repeating steps S102 and S103 to traverse all the video stream data to be retrieved;

[0081] If a spiking neuron The membrane potential Reaching the activation threshold , it means that in the video stream data Pixels There is an occluding target at the position.

[0082] Spiking Neurons Contains and retrieves video stream data All video stream data belong to the same neighborhood space.

[0083] The spiking neuron The membrane potential It is expressed as:

[0084] ;

[0085] in, Indicates the change in membrane potential of a pulse neuron caused by a single video stream data. Indicates the number of video stream data contained in the spiking neuron.

[0086] The above video stream data It is expressed as:

[0087]

[0088] in, Represents video stream data, Indicates the video stream index. Represents the frame index, Represents the pixel index, Represents the gray value of a pixel;

[0089] Video stream data The choice can be determined based on the initial positioning of the target or based on experience.

[0090] Neighborhood size It is related to the target size and is determined according to the specific scenario.

[0091] Activation Threshold ,in, The minimum number of pixels required for occluded target recognition in the current scene;

[0092] Activation Threshold It is related to the minimum number of pixels required to identify the occluded object in the current scene;

[0093] When the number of video stream data stored in the spiking neuron exceeds , the membrane potential of the spiking neuron Reaching the activation threshold ;

[0094] Membrane potential only Reaching the activation threshold The pulse neurons traverse the video stream data and process it, which is conducive to filtering out the video stream data that does not meet the requirements and only processing the identifiable video stream data to improve efficiency.

[0095] S2: Arrange the video stream data collected by the pulse neuron according to the pixel index to obtain a two-dimensional image containing the occluded target;

[0096] Example: Spiking Neurons There are 5 video stream data in the video stream, and the pixel indexes are (1,1), (1,2), (2,2), (2,3), (3,2).

[0097] Arrange the pixels according to their indices to obtain a 3*3 image space, in which 5 pixel positions have values, and the remaining positions cannot obtain images due to occlusion;

[0098] This 3*3 image space is a two-dimensional image containing the occluded target;

[0099] Depending on the scenario, the spiking neurons The number of video stream data in is different;

[0100] S3: Taking the feature points of the two-dimensional image as the center, intercept the window image, input the metric neural network model, and obtain the occluded target type;

[0101] S301: Construct a window with the feature points of the two-dimensional image as the center:

[0102] ;

[0103] in, represents the pixel index of the window, Indicates the index value of the pixel where the feature point is located. Indicates the size of the window;

[0104] S302: Capture the window image and input it into the trained metric neural network model, which classifies the image according to the cosine distance:

[0105] ;

[0106] in, Represents the cosine distance between the window image and the classification target, Represents the projection space obtained by inputting the window image into the neural network model, Represents the projection space of classification targets;

[0107] according to Determines the type of occlusion target.

[0108] For example: Window size Determined based on the significant features around the feature points of the two-dimensional image, for aircraft targets, when the feature point is selected as a point on the nose, select the window size , ensure that the window image includes the entire aircraft nose component; when the feature point is selected as a point on the wing, select the window size , ensuring that the window image includes the entire aircraft wing component.

[0109] S4: converting the pixel index of the occluded target to obtain the actual position of the occluded target;

[0110] S401: Establishing a matching relationship between the pixel coordinate system and the Cartesian coordinate system:

[0111] ;

[0112] in, represents the pixel index of the calibration target, represents the coordinates of the calibration target in the Cartesian coordinate system, Represents the transformation from pixel coordinate system to image coordinate system, Represents the transformation from the image coordinate system to the camera coordinate system, Represents the transformation from the camera coordinate system to the Cartesian coordinate system;

[0113] S402: Establishing a relationship matrix between the occluded target and the calibration target in the pixel coordinate system:

[0114] ;

[0115] in, represents the pixel index of the occluded target, represents the scaling factor, represents the displacement factor;

[0116] S403: Determine the position of the occluding target in the Cartesian coordinate system:

[0117] ;

[0118] in, Represents the coordinates of the occluding target in the Cartesian coordinate system, Representation Matrix The inverse matrix of

[0119] The actual position of the occluded target is .

[0120] S5: arranging the actual positions of the blocked targets in time series to obtain target point traces, and performing correction and interpolation operations on the target point traces to obtain processed target point traces;

[0121] S501: Arrange the actual positions of the blocked targets in time series to obtain target point traces;

[0122] S502: Determine the target preset route according to the target point trace and the type of the occluded target obtained in step S3; bind the target point trace to the target preset route, correct the target point trace that obviously deviates from the route, and perform interpolation operations on the interrupted target point trace to obtain the processed target point trace.

[0123] For example, the target preset route is the prior information for target tracking; the target preset route is the trajectory of the target under ideal conditions, such as air routes, existing roads, and sea routes; in reality, due to obstacles, uncontrollable environmental factors, and human operations, the target point trace will deviate from the target preset route;

[0124] If the occluded target type is an aircraft, the flight route of the aircraft can be determined based on the spatial position corresponding to the video stream data and the existing air route information, and the target point trace obtained in step S501 is matched with the existing flight route to find the flight route with the highest similarity, and the target point trace is bound to the flight route; if the target point trace obviously deviates from the flight route, it is corrected; if the target point trace disappears within a certain period of time, the target running speed and acceleration are calculated according to the flight route and the target motion model, and the point trace is virtually interpolated;

[0125] If the occluded target type is a vehicle, the vehicle's driving route can be determined based on the spatial position corresponding to the video stream data and the existing road information, and the target point trace obtained in step S501 is matched with the existing driving route to find the driving route with the highest similarity, and the target point trace is bound to the driving route; if the target point trace obviously deviates from the driving route, it is corrected; if the target point trace disappears within a certain period of time, the target running speed and acceleration are calculated according to the driving route and the target motion model, and the point trace is virtually interpolated;

[0126] If the occluded target type is a ship, the ship's navigation route can be determined based on the spatial position corresponding to the video stream data and the existing sea route information, and the target point trace obtained in step S501 is matched with the existing navigation route to find the navigation route with the highest similarity, and the target point trace is bound to the navigation route; if the target point trace obviously deviates from the navigation route, it is corrected; if the target point trace disappears within a certain period of time, the target running speed and acceleration are calculated according to the navigation route and the target motion model, and the point trace is virtually interpolated.

[0127] Embodiment 2:

[0128] The present invention also provides a system for continuously tracking an occluded target based on video stream data, comprising the following five modules:

[0129] Video stream data collection module: video stream data judgment in the same neighborhood space, video stream data input into pulse neurons, and judge whether the membrane potential of the pulse neurons reaches the activation threshold;

[0130] Two-dimensional image generation module: Arrange the video stream data in the pulse neurons to generate a two-dimensional image containing the occluded target;

[0131] Occlusion target type identification module: captures the window image and identifies the occlusion target type;

[0132] Occlusion target coordinate conversion module: converts the pixel index of the occlusion target to determine the actual position of the occlusion target;

[0133] Target point trace construction module: obtain the target point trace, and perform correction and interpolation operations on the target point trace.

[0134] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0135] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0136] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for continuously tracking an occluded target based on video stream data, characterized in that: The method comprises: S1: Use pulse neurons to collect all video stream data in the same neighborhood space and determine whether there is an occluded target at the location of the included pixel points; S2: Arrange the video stream data collected by the pulse neuron according to the pixel index to obtain a two-dimensional image containing the occluded target; S3: Taking the feature points of the two-dimensional image as the center, intercept the window image, input the metric neural network model, and obtain the occluded target type; S4: converting the pixel index of the occluded target to obtain the actual position of the occluded target; S5: arranging the actual positions of the blocked targets in time series to obtain target point traces, and performing correction and interpolation operations on the target point traces to obtain processed target point traces; The step S1 comprises: S101: Obtaining video stream data ,in Indicates the video stream index; The video stream data will be obtained Input spike neuron ; S102: Determine video stream data And the video stream data to be retrieved Whether they belong to the same neighborhood space: ； in, Indicates the video stream data to be retrieved. Indicates the index of the video stream to be retrieved. and Respectively and Contains the pixel index value, and Respectively and The frame index of represents the neighborhood size, express and belong to the same neighborhood space, express and They do not belong to the same neighborhood space; S103: If the video stream data and the video stream data to be retrieved belong to the same neighborhood space, Input spike neuron ; S104: repeating steps S102 and S103 to traverse all the video stream data to be retrieved; If a spiking neuron The membrane potential Reaching the activation threshold , it means that in the video stream data Pixels There is an occluding target at the position.

2. The method for continuously tracking an occluded target based on video stream data according to claim 1, characterized in that: The spiking neuron The membrane potential It is expressed as: ； in, Indicates the change in membrane potential of a pulse neuron caused by a single video stream data. Indicates the number of video stream data contained in the spiking neuron.

3. The method for continuously tracking an occluded target based on video stream data according to claim 1, characterized in that: The step S3 comprises: S301: Construct a window with the feature points of the two-dimensional image as the center: ； in, represents the pixel index of the window, Indicates the index value of the pixel where the feature point is located. Indicates the size of the window; S302: Capture the window image and input it into the trained metric neural network model, which classifies the image according to the cosine distance: ； in, Represents the cosine distance between the window image and the classification target, Represents the projection space obtained by inputting the window image into the neural network model, Represents the projection space of classification targets; according to Determines the type of occlusion target.

4. The method for continuously tracking an occluded target based on video stream data according to claim 3, characterized in that: The step S4 comprises: S401: Establishing a matching relationship between the pixel coordinate system and the Cartesian coordinate system: ； in, represents the pixel index of the calibration target, represents the coordinates of the calibration target in the Cartesian coordinate system, Represents the transformation from pixel coordinate system to image coordinate system, Represents the transformation from the image coordinate system to the camera coordinate system, Represents the transformation from the camera coordinate system to the Cartesian coordinate system; S402: Establishing a relationship matrix between the occluded target and the calibration target in the pixel coordinate system: ； in, represents the pixel index of the occluded target, represents the scaling factor, represents the displacement factor; S403: Determine the position of the occluding target in the Cartesian coordinate system: ； in, Represents the coordinates of the occluding target in the Cartesian coordinate system, Representation Matrix The inverse matrix of The actual position of the occluded target is .

5. The method for continuously tracking an occluded target based on video stream data according to claim 4, characterized in that: The step S5 comprises: S501: Arrange the actual positions of the blocked targets in time series to obtain target point traces; S502: Determine the target preset route according to the target point trace and the type of the occluded target obtained in step S3; bind the target point trace to the target preset route, correct the target point trace that obviously deviates from the route, and perform interpolation operations on the interrupted target point trace to obtain the processed target point trace.

6. A continuous tracking system for occluded targets based on video stream data, characterized in that: include: Video stream data collection module: video stream data judgment in the same neighborhood space, video stream data input into pulse neurons, and judge whether the membrane potential of the pulse neurons reaches the activation threshold; Two-dimensional image generation module: Arrange the video stream data in the pulse neurons to generate a two-dimensional image containing the occluded target; Occlusion target type identification module: captures the window image and identifies the occlusion target type; Occlusion target coordinate conversion module: converts the pixel index of the occlusion target to determine the actual position of the occlusion target; Target point trace construction module: obtain the target point trace, perform deviation correction and interpolation operations on the target point trace; To realize the method for continuous tracking of occluded targets based on video stream data as described in any one of claims 1-5.

Images