Real-time tracking and positioning system and method for mobile targets based on multi-modality

Abstract

The present invention proposes a real-time tracking and positioning method and system for a mobile target based on multi-modality, including: obtaining multi-modal information of the mobile target, including image information of the mobile target collected in a motion scene, and acceleration of the mobile target in different directions collected by a collection device carried by the mobile target; obtaining an image of the mobile target with a detection frame based on the image information of the mobile target; obtaining the state quantity of the mobile target based on the multi-modal information of the mobile target; tracking the position of the mobile target and predicting the motion position based on the image of the mobile target with the detection frame and the state quantity of the mobile target. The present invention combines the multi-modal interaction mode of software and hardware to track and position players, which can reduce errors and achieve more accurate and stable effects.

Description

Technical Field

[0001] The present invention belongs to the field of tracking and positioning technology and computer vision technology, and in particular relates to a multi-modal mobile target real-time tracking and positioning system and method. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Target tracking is an important issue in the field of computer vision, mainly involving single target tracking and multi-target tracking. It is currently widely used in sports events, security monitoring, drones, unmanned vehicles, robots, as well as human bodies, faces, and gestures. Simply put, visual target tracking is to establish the position relationship of the object to be tracked in a continuous video sequence and obtain the complete motion trajectory of the object. During the movement, the target may show some changes in the image, such as changes in posture or shape, changes in scale, background occlusion or changes in light brightness, etc. The research on target tracking algorithms also revolves around solving these changes and specific applications.

[0004] Motion sensors are very useful for monitoring the movement of equipment, such as tilt, vibration, rotation, swing, etc. These movements usually directly reflect the user's input. With the development of micro-electromechanical system (MEMS) technology, both domestic and foreign countries have made the development of micro-accelerometers a priority for the commercialization of MEMS. Compared with ordinary accelerometers, micro-accelerometers have many advantages: small size, light weight, low cost, low power consumption, good reliability, etc. Accelerometers are used to measure the force of equipment and are good at detecting the movement of equipment relative to external reference objects (such as the ground). It can be widely used in aerospace, automotive industry, industrial automation and robotics and other fields, and has broad application prospects.

[0005] At present, there have been many studies on the tracking and positioning of moving targets through surveillance image analysis, but there are relatively few studies on methods for tracking and positioning in combination with hardware devices. Summary of the invention

[0006] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a real-time tracking and positioning method for mobile targets based on multi-modality, which solves the occlusion problem existing in the tracking process of visual sensors and achieves more accurate and stable positioning and tracking.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] In a first aspect, a multi-modal real-time tracking and positioning method for a mobile target is disclosed, comprising:

[0009] Obtain multiple modal information of the moving target, including image information of the moving target collected in the motion scene, and acceleration of the moving target in different directions collected by the collection device carried by the moving target;

[0010] Obtaining an image of the moving target with a detection frame based on the moving target image information;

[0011] Associating multiple modal information of a mobile target to obtain a state quantity of the mobile target;

[0012] The position of the moving target is tracked and the motion position is predicted based on the image of the moving target with a detection frame and the state quantity of the moving target.

[0013] According to a further technical solution, the accelerations of the moving target in different directions collected by a collection device carried by the moving target include accelerations along the x-axis and accelerations along the y-axis.

[0014] A further technical solution, when obtaining an image of a moving target with a detection frame based on the moving target image information, includes:

[0015] The image is preprocessed to adjust the pixel size, and a large number of candidate samples are generated through the pedestrian detection algorithm. Feature extraction is performed, and a classifier is trained to distinguish between the target and the background. The candidate sample with the highest confidence is selected as the prediction result, and the detection box is output.

[0016] According to a further technical solution, the detection frame is a rectangular area including the moving target, and the relative position coordinates of the moving target are obtained based on the detection frame.

[0017] In a further technical solution, the state quantity of the mobile target is obtained by associating multiple modal information of the mobile target, and the state quantity includes:

[0018] Relative position coordinates of moving target;

[0019] The acceleration of the moving target along the x-axis and y-axis;

[0020] The moving speed at time t is calculated based on the change in the moving target's moving position coordinates from time t-1 to time t and the change in moving time.

[0021] A further technical solution also includes: processing the state quantity as input for target tracking.

[0022] A further technical solution is to detect and track the moving target, specifically:

[0023] The target image with the detection frame is used as the initial image, and the center coordinates of the detection frame are used as the initial tracking position;

[0024] The target sample obtains all the sampled samples through the circulation matrix, where the target is the positive sample and the background is the negative sample;

[0025] Extract multi-channel features and introduce kernel functions to determine the similarity between each cycle sample and the base sample;

[0026] Train a discriminative classifier;

[0027] The input image determines whether the tracked object is the target or the surrounding background information, and the sample with the largest response value is taken as the position of this frame.

[0028] A further technical solution is to predict the movement of the mobile target, and when the tracking object is blocked during the tracking process, the target state of the mobile target is continuously updated, thereby updating the positioning information and the transmitted acceleration signal;

[0029] Then, based on the change time and the player's movement speed at this moment, predict the player's movement speed at the next moment;

[0030] According to the obtained speed and acceleration direction, the displacement that changes during this period of time is predicted, thereby predicting the moving position coordinates of the moving target at the next moment.

[0031] In the second aspect, a multi-modal real-time tracking and positioning system for a mobile target is disclosed, comprising:

[0032] Visual information acquisition module, used to collect image information of moving targets in motion scenes;

[0033] Acceleration information acquisition module, which collects the acceleration of the moving target in different directions;

[0034] The data communication module transmits the collected moving target image information and acceleration information to the data processing equipment in real time;

[0035] Data processing equipment performs real-time tracking and prediction of moving targets based on received data.

[0036] According to a further technical solution, the data processing device comprises:

[0037] The target detection module calculates the relative position coordinates of the moving target through the target detection algorithm;

[0038] The associated state quantity module associates the data to obtain the state quantity: the moving target position coordinates, motion acceleration and motion speed, which provide input for the target tracking module;

[0039] The target tracking module tracks the moving target in real time through the target tracking method;

[0040] The signal prediction module uses the position coordinates, acceleration and speed of the moving target to predict the moving state of the moving target in the next frame.

[0041] One or more of the above technical solutions have the following beneficial effects:

[0042] The multi-modal player real-time tracking and positioning system and method provided by the present invention combines the multi-modal interaction mode of software and hardware to track and position players, which can reduce errors and achieve more accurate and stable effects.

[0043] The present invention uses active positioning of visual sensor information, passive positioning of hardware sensor information, and the complementarity of multiple modal information and active and passive combination to complete real-time tracking and positioning of players. It can also use accelerometer sensor information combined with visual information to predict the player's movement speed and displacement, solve the occlusion problem in the visual sensor tracking process, and achieve more accurate and stable positioning and tracking.

[0044] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0046] Figure 1 A schematic diagram of a multi-modal player real-time tracking and positioning system according to an embodiment of the present invention;

[0047] Figure 2 This is a flow chart of a multi-modal player real-time tracking and positioning method according to an embodiment of the present invention;

[0048] Figure 3 This is an example diagram of a multi-modal player real-time tracking and positioning system and method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0050] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0051] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0052] Embodiment 1

[0053] This embodiment discloses a real-time tracking and positioning method for a mobile target based on multi-modality. The mobile target in this embodiment of the application is explained by taking a basketball player as an example, specifically taking a camera C that outputs RGB images, a sports bracelet B with an acceleration sensor, and a basketball player A as examples.

[0054] See attached Figure 2 , specifically including the following steps:

[0055] Step S0, collecting multiple modal information of player A in a sports scene;

[0056] The multimodal information includes: first, the RGB image I of the player A captured by camera C t , and secondly, player A carries sports bracelet B with an accelerometer sensor to collect acceleration a along the x-axis direction xt , acceleration a along the y-axis yt ;

[0057] Step S1: transmit the acceleration signal a via Bluetooth communication xt 、a yt Transfer to computer terminal;

[0058] Step S2, obtaining the relative position coordinates of the player through the YOLOv3 target detection algorithm module;

[0059] The image is preprocessed to adjust the pixel size, and a pedestrian detection algorithm (such as YOLO target detection, etc.) is used to generate many candidate samples, feature extraction, and a classifier is trained to distinguish between the target and the background. The candidate sample with the highest confidence is selected as the prediction result, and the detection box (Bounding Box), that is, the rectangular area containing the object, is output to obtain the return value x t ,y t , w, h. Among them, x t ,y t It refers to the predicted coordinates of the representative object position, that is, the relative position coordinates of the player. w and h are the width and height of the detection box (Bounding Box).

[0060] In the specific implementation, the image is first scaled to 416x416 pixels, and three pre-selected grids of different sizes are densely distributed on the image (the sizes of the three pre-selected grids are 8x8, 16x16, and 32x32 pixels, respectively). After the convolution layer and residual network feature extraction, three four-dimensional tensors are obtained, in which each element represents the output information of the network for a certain type of pre-selected grid. Taking the output tensor of a pre-selected grid of 32x32 pixels as an example, its output size is (batch_size, 13, 13, 18). The second and third dimensions indicate that 13 pre-selected grids are distributed in both dimensions of the image. The fourth dimension indicates the prior box information in each pre-selected grid: each pre-selected grid has 3 prior boxes, and each prior box has 6 parameters: the first parameter indicates the confidence of the network in judging that the prior box contains a person, and the remaining five parameters are x_offset: the distance (offset) from the upper left corner of the grid to the x-axis, y_offset: the distance (offset) from the upper left corner of the grid to the y-axis, w, h: the width and height of the Bounding Box, and confidence. The one with the largest confidence parameter value among the three prior boxes is taken as the output of the candidate grid;

[0061] Step S3, calculating the state quantity of the object based on the multimodal information of player A;

[0062] The state quantities include three: the above three state quantities are all used in step S5 as inputs for prediction;

[0063] Step S3-1. RGB image I t Through the target detection algorithm in step S2, the relative position coordinates (x_offset, y_offset) of the player are obtained;

[0064] Step S3-2. Obtain the acceleration a of the player along the x-axis and y-axis through the accelerometer sensor sports bracelet B xt 、a yt ;

[0065] Step S3-3. Using the above two state quantities, the change in the player's position coordinates from time t-1 to time t, Δx, Δy, and the change in movement time, Δt, calculate the movement speed v at time t t ,

[0066]

[0067] Normalize the state quantity as the input for target tracking;

[0068] Step S4, using the target tracking algorithm Kernel Correlation Filter (KCF) to track player A;

[0069] S4-1. Use the circulant matrix to sample the candidate grids outputted in step S2, taking the candidate grids containing the tracked object as positive samples and the grids without the tracked object as negative samples. Take the center of the candidate grid corresponding to each positive sample as the initial value of the tracking position of the sample;

[0070] S4-2. Extract HOG multi-channel features for each positive sample;

[0071] S4-3. Train a classification network that takes a pair of images as input and outputs a floating point number between 0 and 1. The higher the similarity of the targets in the two images, the closer the output of the network is to 1, and vice versa. Use all the positive and negative samples obtained by the circulant matrix in step S4-1 for training. Send all the positive samples generated by the circulant matrix of the current frame image into the trained classification network in pairs, and take the positive sample with the largest output value as the estimated value of the target tracking position.

[0072] The classification network is used to generate an estimate of the track's location.

[0073] The network is trained using the positive and negative samples generated in the previous frame, and the trained network is used for the positive samples of the current frame.

[0074] Step S5, predicting player A using the accelerometer signal collected by sports bracelet B;

[0075] Player movement prediction: When camera C encounters extreme situations such as object occlusion during the process of shooting target player A, it continuously updates the player target state and then updates the positioning information, using the acceleration signal a transmitted by the accelerometer sensor sports bracelet B xt 、a yt ,calculate

[0076]

[0077] Then according to the change time Δt and the player's movement speed v at this time t , predict the player's movement speed v at the next moment t+1 ,

[0078]

[0079] According to the obtained speed and the direction of acceleration, predict the displacement s that changes during this period of time,

[0080] v t+1 2 -v t 2 =2as

[0081] Thus predicting the player's movement position coordinates (x t+1 ,y t+1 ), reducing tracking errors through prediction, and achieving more accurate and stable positioning and tracking.

[0082] See attached Figure 3 As shown, the present invention uses multiple modes to work together, utilizes the real-time nature of the correlation filtering algorithm, can correct the delay problem of the sensor, can accurately perform initial positioning through the target detection algorithm, uses the signal of the hardware sensor as prediction information, can solve the occlusion problem in the visual sensor tracking process, integrates multiple modal information, reduces errors, and achieves more accurate and stable tracking and positioning.

[0083] The active positioning method using visual sensors to target players can obtain real-time information such as the players' positions and movement trajectories on the field without the need for active cooperation from the athletes. The passive positioning method using portable sensors to target players is to position in a real three-dimensional environment. Combining the passive positioning method with the active method can improve positioning accuracy.

[0084] This method is not only applicable to the tracking and positioning of a single target player, but also to multiple target players. It is necessary to bind the accelerometer sensor device to each player to determine a unique number.

[0085] Embodiment 2

[0086] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0087] Embodiment 3

[0088] The purpose of this embodiment is to provide a computer-readable storage medium.

[0089] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.

[0090] Embodiment 4

[0091] See attached Figure 1 As shown, the purpose of this embodiment is to provide a multi-modal player real-time tracking and positioning system, including: a visual information acquisition module, an acceleration information acquisition module, a data communication module, a target detection module, an associated state quantity module, a target tracking module, and a signal prediction module;

[0092] The visual information acquisition module is used to: provide a modal information, and the visual sensor acquires RGB images;

[0093] The acceleration information acquisition module is used to: provide another modal information, the acceleration sensor collects the player's motion acceleration a xt 、a yt ;

[0094] The data communication module is used to: transmit the acceleration signal collected by the accelerometer sensor to the computer terminal in real time using Bluetooth communication;

[0095] The target detection module is used to calculate the relative position coordinates x of the sports player through the target detection algorithm. t ,y t ;

[0096] The associated state module is used to provide input for the target tracking module, the player position coordinates (x t ,y t ), motion acceleration a xt 、a yt , and the movement speed v t ;

[0097] The target tracking module is used to: track the target of the sports player in real time by using the target tracking method;

[0098] The signal prediction module is used to: use the accelerometer sensor signal to predict the target's next frame of motion, solve the occlusion problem in the visual sensor tracking process, and achieve more stable positioning.

[0099] The modalities of the present invention are: one is an active positioning method for player targets based on visual sensors, using RGB images captured by surveillance cameras to detect players to obtain position coordinates and use KCF correlation filtering to track targets; the other is a passive positioning method for player targets based on hardware sensor accelerometers, using sensor devices (such as bracelets, stickers, wristbands, chips, etc.) carried by players to achieve device communication between players and the back end to obtain player motion acceleration. The two modalities are integrated, and position coordinates, player motion speed, and motion acceleration are used as the state quantity of each player, thereby achieving player positioning and motion trajectory tracking. By implementing the present invention, multi-modal information can complement each other, and the accelerometer sensor information can be combined with visual information to predict the player's motion speed and displacement, solving the occlusion problem in the visual sensor tracking process, so as to achieve more accurate and stable positioning and tracking.

[0100] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0101] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0102] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A multi-modal real-time tracking and positioning method for mobile targets, characterized in that: include: Obtain multiple modal information of the moving target, including image information of the moving target collected in the motion scene, and acceleration of the moving target in different directions collected by the collection device carried by the moving target; Obtaining an image of the moving target with a detection frame based on the moving target image information; Associating multiple modal information of a mobile target to obtain a state quantity of the mobile target; Tracking the position of a moving target and predicting its motion position based on an image of the moving target with a detection frame and a state quantity of the moving target; The associating multiple modal information of the mobile target to obtain the state quantity of the mobile target, the state quantity includes: The RGB image is passed through the target detection algorithm to obtain the relative position coordinates (x_offset, y_offset) of the player; The accelerometer sensor sports bracelet obtains the player's acceleration a along the x-axis and y-axis xt 、a yt ; Using the above two state quantities, the player's position coordinate change from time t-1 to time t, Δx, Δy, and the movement time change Δt, calculate the movement speed v at time t t , Process the state quantity as input for target tracking; When obtaining an image of a moving target with a detection frame based on the moving target image information, it includes: The image is preprocessed to adjust the pixel size, and a number of candidate samples are generated through the pedestrian detection algorithm. Feature extraction is performed, and a classifier is trained to distinguish between the target and the background. The candidate sample with the highest confidence is selected as the prediction result, and the detection box is output; Specifically, the image is scaled to 416x416 pixels, and three pre-selected grids of different sizes are densely distributed on the image. The sizes of the three pre-selected grids of different sizes are 8x8, 16x16 and 32x32 pixels respectively. After the convolution layer and residual network feature extraction, three four-dimensional tensors are obtained, in which each element represents the output information of the network for a certain type of pre-selected grid; the second and third dimensions represent the number of pre-selected grids distributed in two dimensions of the image, and the fourth dimension represents the prior box information in each pre-selected grid: each pre-selected grid has 3 prior boxes, and each prior box has 6 parameters: the first parameter represents the confidence of the network in judging that the prior box contains a person, and the remaining five parameters are x_offset: the distance of the upper left corner of the grid relative to the x-axis, y_offset: the distance of the upper left corner of the grid relative to the y-axis, w, h: the width and height of the BoundingBox, and the confidence; the one with the largest confidence parameter value in the three prior boxes is taken as the output of the candidate grid; The detection frame is a rectangular area containing the moving target, and the relative position coordinates of the moving target are obtained based on the detection frame; The mobile target position tracking and motion position prediction based on the image of the mobile target with the detection frame and the state of the mobile target, when the tracking object is blocked in the process of tracking the mobile target, the target state of the mobile target is continuously updated, and then the positioning information is updated, and the acceleration signal transmitted is specifically: the acceleration signal a transmitted by the accelerometer sensor sports bracelet B xt 、a yt ,calculate Then according to the change time Δt and the player's movement speed v at this time t Predict the player's movement speed v at the next moment t+1 , According to the obtained velocity and acceleration direction, predict the displacement s that changes during this period of time. in t+1 2 -v t 2 =2as; Thus, the moving target position coordinates (x t+1 ,y t+1 ).

2. The multi-modal real-time tracking and positioning method for a mobile target according to claim 1, characterized in that: The accelerations of the mobile target in different directions collected by the collection device carried by the mobile target include the acceleration along the x-axis direction and the acceleration along the y-axis direction.

3. The multi-modal real-time tracking and positioning method for a mobile target according to claim 1, characterized in that: Detect and track the moving target, specifically: The candidate grid obtains all sampling samples through the circulatory matrix, where the target is the positive sample and the background is the negative sample, and the center coordinate position of the positive sample is used as the initial tracking position; Extract multi-channel features; Train the discriminant classifier and take the positive sample with the largest response value as the estimated value of the target tracking position.

4. A real-time tracking and positioning system for mobile targets based on multi-modality, characterized in that: include: Visual information acquisition module, used to collect image information of moving targets in motion scenes; Acceleration information acquisition module, which collects the acceleration of the moving target in different directions; The data communication module transmits the collected moving target image information and acceleration information to the data processing equipment in real time; Data processing equipment to track and predict moving targets in real time based on received data; The state quantity of the mobile target is obtained by associating multiple modal information of the mobile target, and the state quantity includes: The RGB image is passed through the target detection algorithm to obtain the relative position coordinates (x_offset, y_offset) of the player; The accelerometer sensor sports bracelet obtains the player's acceleration a along the x-axis and y-axis xt 、a yt ; Using the above two state quantities, the player's position coordinate change from time t-1 to time t, Δx, Δy, and the movement time change Δt, calculate the movement speed v at time t t , Process the state quantity as input for target tracking; When obtaining an image of a moving target with a detection frame based on the moving target image information, it includes: The image is preprocessed to adjust the pixel size, and a number of candidate samples are generated through the pedestrian detection algorithm. Feature extraction is performed, and a classifier is trained to distinguish between the target and the background. The candidate sample with the highest confidence is selected as the prediction result, and the detection box is output; Specifically, the image is scaled to 416x416 pixels, and three pre-selected grids of different sizes are densely distributed on the image. The sizes of the three pre-selected grids of different sizes are 8x8, 16x16 and 32x32 pixels respectively. After the convolution layer and residual network feature extraction, three four-dimensional tensors are obtained, in which each element represents the output information of the network for a certain type of pre-selected grid; the second and third dimensions represent the number of pre-selected grids distributed in two dimensions of the image, and the fourth dimension represents the prior box information in each pre-selected grid: each pre-selected grid has 3 prior boxes, and each prior box has 6 parameters: the first parameter represents the confidence of the network in judging that the prior box contains a person, and the remaining five parameters are x_offset: the distance of the upper left corner of the grid relative to the x-axis, y_offset: the distance of the upper left corner of the grid relative to the y-axis, w, h: the width and height of the BoundingBox, and the confidence; the one with the largest confidence parameter value in the three prior boxes is taken as the output of the candidate grid; The detection frame is a rectangular area containing the moving target, and the relative position coordinates of the moving target are obtained based on the detection frame; Based on the image of the moving target with the detection frame and the state of the moving target, the moving target position is tracked and the motion position is predicted. When the tracking object is blocked in the process of tracking the moving target, the target state of the moving target is continuously updated, and then the positioning information is updated, and the acceleration signal transmitted is transmitted; specifically: the acceleration signal a transmitted by the accelerometer sensor sports bracelet B xt 、a yt ,calculate Then according to the change time Δt and the player's movement speed v at this time t Predict the player's movement speed v at the next moment t+1 , According to the obtained velocity and acceleration direction, predict the displacement s that changes during this period of time. in t+1 2 -v t 2 =2as; Thus, the moving target position coordinates (x t+1 ,y t+1 ).

5. The multi-modal real-time tracking and positioning system for mobile targets as claimed in claim 4, characterized in that: The data processing device comprises: The target detection module calculates the relative position coordinates of the moving target through the target detection algorithm; The state quantity calculation module calculates the data to obtain the state quantity: the moving target position coordinates, motion acceleration and motion speed, which provide input for the target tracking module; The target tracking module tracks the moving target in real time through the target tracking method; The signal prediction module uses the position coordinates, acceleration and speed of the moving target to predict the moving state of the moving target in the next frame.

Images