Nonlinear multi-target detection and tracking methods and related equipment

Abstract

This invention provides a nonlinear multi-target detection and tracking method and related equipment, comprising: acquiring image data to be detected and attitude data of a shooting motion platform synchronized with the image data to be detected; performing target detection processing on the image data to be detected to obtain at least one target detection result; performing coordinate system transformation processing on the at least one target detection result based on the attitude data to obtain observation information corresponding to the target detection result; performing prediction processing on the motion trajectory of the target based on the observation information to obtain predicted trajectory data of multiple targets; correcting the predicted trajectory data based on the observation information; and outputting target tracking path data corresponding to the target detection result when the predicted trajectory data converges. By introducing attitude information to perform coordinate system transformation on the detection result and predicting and correcting the target motion trajectory under a stable reference, the stability and accuracy of multi-target detection and tracking in motion platform scenarios are improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to a nonlinear multi-target detection and tracking method, apparatus, electronic device and its storage medium. Background Technology

[0002] With the development of intelligent sensing, intelligent monitoring, and automated systems, image-based target detection and tracking technologies are widely used in fields such as unmanned platforms, mobile carrier monitoring, and dynamic scene perception. In these applications, it is typically necessary to identify targets in continuously acquired image data and track their motion trajectories to obtain information on target position changes and movement trends.

[0003] Existing target detection and tracking methods mostly use image data as the main input, typically analyzing the target's position and predicting its trajectory directly in the image coordinate system. However, when the imaging platform is in motion, the target's position in the image is often affected by both the target's own motion and the imaging platform's attitude, leading to a discrepancy between the detection results and the actual target's motion.

[0004] Furthermore, in existing technologies, there is often a lack of effective observation feedback mechanism between target detection results and trajectory prediction process. The prediction results are difficult to correct in a timely manner based on new detection information, which can easily lead to problems such as trajectory drift, unstable prediction, or confusion in multi-target tracking, thereby affecting the reliability and accuracy of multi-target tracking.

[0005] Therefore, existing target detection and tracking methods have the problem of difficulty in effectively distinguishing between the target's own motion and the influence of the platform's motion when the shooting platform undergoes posture changes, resulting in insufficient stability and accuracy of target tracking results. Summary of the Invention

[0006] This invention provides a nonlinear multi-target detection and tracking method to solve the problem that existing target detection and tracking methods have difficulty in effectively distinguishing between the target's own motion and the influence of the platform's motion when the shooting platform undergoes attitude changes, resulting in insufficient stability and accuracy of target tracking results.

[0007] In a first aspect, the present invention provides a nonlinear multi-target detection and tracking method, the method comprising the following steps: Acquire the image data to be detected and the attitude data of the shooting motion platform synchronized with the image data to be detected; The image data to be detected is subjected to target detection processing to obtain at least one target detection result; Based on the attitude data, coordinate system transformation is performed on the at least one target detection result to obtain the observation information corresponding to the target detection result; Based on the observation information, the motion trajectory of the target is predicted to obtain predicted trajectory data of multiple targets. Based on the observation information, the predicted trajectory data is corrected, and when the predicted trajectory data converges, the target tracking path data corresponding to the target detection result is output.

[0008] Optionally, acquiring the image data to be detected and the attitude data of the shooting motion platform synchronized with the image data to be detected includes: Acquire the image data to be detected collected by the shooting motion platform at the same time reference; The attitude information of the shooting motion platform, which is time-synchronized with the image data to be detected, is obtained. The attitude information is used to determine the motion state of the shooting motion platform when acquiring the image data to be detected.

[0009] Optionally, the step of performing target detection processing on the image data to be detected to obtain at least one target detection result includes: Spatial position change feature processing is performed on the image data to be detected to obtain spatial position change feature information; Based on the spatial location change feature information, multiple target regions are identified in the image data to be detected; Target determination is performed in the multiple target regions to determine the position of the target in the image data to be detected; Based on the location of the target in the image data to be detected, at least one target detection result is output.

[0010] Optionally, the step of performing coordinate system transformation on the at least one target detection result based on the attitude data to obtain the observation information corresponding to the target detection result includes: Based on the posture data, the spatial posture state of the shooting motion platform when acquiring the image data to be detected is determined; Based on the spatial attitude state, determine the transformation relationship between the image coordinate system and the stable coordinate system; Based on the transformation relationship, the position information of the target detection result in the image coordinate system is converted into the position information in the stable coordinate system, and the corresponding observation information is output.

[0011] Optionally, determining the transformation relationship between the image coordinate system and the stable coordinate system based on the spatial pose state includes: Based on the spatial attitude state, the attitude parameters of the shooting motion platform relative to the stable coordinate system are determined; Based on the attitude parameters, determine the coordinate mapping relationship between the coordinate axis directions of the image coordinate system and the coordinate axis directions of the stable coordinate system; Based on the coordinate mapping relationship, a transformation relationship is established to map the position information in the image coordinate system to the position information in the stable coordinate system.

[0012] Optionally, based on the observation information, the prediction processing of the target's motion trajectory to obtain predicted trajectory data for multiple targets includes: Based on the observation information, target state information corresponding to the at least one target is determined, wherein the target state information includes target position and / or target velocity; Based on the target state information, the state of the target is predicted by a preset nonlinear motion model to obtain the predicted state information of each target. Based on the predicted state information, predicted trajectory data for multiple targets are generated.

[0013] Optionally, the step of correcting the predicted trajectory data based on the observation information, and outputting the target tracking path data corresponding to the target detection result when the predicted trajectory data converges, includes: Based on the observation information, determine the current observation position of the target in the stable coordinate system; The current observation position is compared with the predicted position at the corresponding time in the predicted trajectory data to obtain trajectory deviation information; Based on the trajectory deviation information, the predicted trajectory data is corrected and updated so that the corrected predicted trajectory data approaches the state corresponding to the observation information; When the corrected predicted trajectory data meets the preset convergence condition, the target tracking path data corresponding to the target detection result is output.

[0014] Secondly, the present invention also provides a nonlinear multi-target detection and tracking device, the nonlinear multi-target detection and tracking device comprising: The first acquisition module is used to acquire the image data to be detected and the attitude data of the shooting motion platform synchronized with the image data to be detected; The first processing module is used to perform target detection processing on the image data to be detected, and obtain at least one target detection result; The second processing module is used to perform coordinate system transformation processing on the at least one target detection result based on the attitude data to obtain the observation information corresponding to the target detection result; The first prediction module is used to predict the trajectory of the target based on the observation information to obtain predicted trajectory data of multiple targets. The first correction module is used to correct the predicted trajectory data based on the observation information, and output the target tracking path data corresponding to the target detection result when the predicted trajectory data converges.

[0015] Thirdly, the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the nonlinear multi-target detection and tracking method provided by the present invention.

[0016] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the nonlinear multi-target detection and tracking method provided by the invention.

[0017] This invention acquires image data to be detected and attitude data of a shooting platform synchronized with the image data; performs target detection processing on the image data to obtain at least one target detection result; based on the attitude data, performs coordinate system transformation processing on the at least one target detection result to obtain observation information corresponding to the target detection result; based on the observation information, performs prediction processing on the target's motion trajectory to obtain predicted trajectory data for multiple targets; based on the observation information, corrects the predicted trajectory data, and outputs target tracking path data corresponding to the target detection result when the predicted trajectory data converges. By introducing attitude information to perform coordinate system transformation on the detection results and predicting and correcting the target motion trajectory under a stable reference, the stability and accuracy of multi-target detection and tracking in motion platform scenarios are improved. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a nonlinear multi-target detection and tracking method provided in an embodiment of the present invention; Figure 2 This is a diagram of a target detection model architecture for multi-scale feature extraction and fusion provided in an embodiment of the present invention; Figure 3 This is a prediction model effect diagram provided by an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a nonlinear multi-target detection and tracking device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] like Figure 1 As shown, Figure 1 This is a flowchart of a nonlinear multi-target detection and tracking method provided in an embodiment of the present invention. The nonlinear multi-target detection and tracking method includes the following steps: 101. Acquire the image data to be detected and the attitude data of the shooting motion platform synchronized with the image data to be detected.

[0022] In this embodiment of the invention, the aforementioned nonlinear multi-target detection and tracking method can be applied to a nonlinear multi-target detection and tracking platform. This platform has functions such as multi-target detection and tracking data processing, multi-target detection and tracking data transmission and reception, and multi-target detection and tracking data memory storage. Furthermore, it can be built based on a server or server cluster, where the server or server cluster can be an electronic device with multi-target detection and tracking data processing capabilities. The aforementioned image data to be detected can refer to the image data acquired by the aforementioned nonlinear multi-target detection and tracking platform from the shooting motion platform during operation, and used for target detection and tracking processing. That is, the scene information within the current field of view of the aforementioned shooting motion platform, which can be in the form of single-frame image data or multiple frames of image data acquired continuously in time sequence.

[0023] The aforementioned shooting motion platform can refer to a moving carrier used to support the image acquisition device and to acquire images during operation. It is understood that the aforementioned shooting motion platform may undergo translation, turning or posture changes during operation, and its motion state will directly affect the representation of the target position in the image. Therefore, this embodiment can introduce posture data to eliminate the influence of platform motion.

[0024] The aforementioned attitude data can refer to the spatial attitude state of the aforementioned shooting motion platform when acquiring the aforementioned image data to be detected, including but not limited to attitude parameters used to describe the spatial orientation of the shooting motion platform, such as at least one of heading, pitch, and roll, and / or angular velocity parameters used to describe the trend of attitude change. Generally, the aforementioned attitude data is synchronized with the aforementioned image data to be detected in time, and is used to determine the motion state of the shooting motion platform during image acquisition in subsequent processing.

[0025] In one possible embodiment, the aforementioned nonlinear multi-target detection and tracking platform acquires image information and platform attitude information at the same time reference, laying a data foundation for subsequent elimination of the influence of platform motion.

[0026] 102. Perform target detection processing on the image data to be detected to obtain at least one target detection result.

[0027] In this embodiment of the invention, the aforementioned nonlinear multi-target detection and tracking platform can identify and determine the target location in the image based on the image data to be detected, thereby completing the target detection process. Specifically, by analyzing the spatial structure, regional differences, or positional change features in the image, regions that may contain targets can be identified in the image, and target determination can be completed in the aforementioned regions, thereby achieving the distinction between the target and the background.

[0028] The target detection result mentioned above can refer to the output result of the target detection process mentioned above, which is used to describe the target information detected in the image data to be detected. It includes at least the position information of the target in the image coordinate system, such as the target center position, the target region position, or equivalent position description parameters. When there are multiple targets, the target detection result mentioned above includes the position information of each of the multiple targets.

[0029] More specifically, it can be done through, for example Figure 2 The diagram illustrates a multi-scale feature extraction and fusion target detection model architecture. This model comprises a backbone network, a neck network, and an output structure connected sequentially. The backbone network extracts features from the noisy input infrared image at various levels, acquiring spatial feature information at different levels through multi-layer convolutional structures. The neck network, based on the output features of the backbone network, introduces a cross-scale feature fusion structure, performing skip connections and fusion of features at different resolutions. Attention or feature transformation modules are used to enhance the fused features, highlighting weak target features and suppressing background noise interference. The output branch, based on the fused multi-scale features, sets up multiple detection branches to detect targets at different scales and outputs the corresponding target location information.

[0030] Through the architecture of the target detection model based on multi-scale feature extraction and fusion, the nonlinear multi-target detection and tracking platform can achieve stable detection of targets at different scales under conditions of low signal-to-noise ratio and complex background, providing reliable target detection results for subsequent coordinate system transformation and target tracking processing.

[0031] 103. Based on the attitude data, perform coordinate system transformation on at least one target detection result to obtain the observation information corresponding to the target detection result.

[0032] In this embodiment of the invention, the nonlinear multi-target detection and tracking platform can transform the target detection result from the image coordinate system to the stable coordinate system based on the above posture data, thereby eliminating the influence of the shooting motion platform posture change on the target position estimation, and enabling the target position to be uniformly expressed under a stable reference system.

[0033] The aforementioned observation information may refer to the output results of the coordinate system transformation process, which is used to characterize the current observation state of the target in the stable coordinate system, including but not limited to the target's position information in the stable coordinate system. It is the common input information for target trajectory prediction and predicted trajectory correction.

[0034] The aforementioned stable coordinate system can be understood as the reference coordinate system introduced in the nonlinear multi-target detection and tracking platform to eliminate the influence of attitude changes in the shooting platform. Generally, this stable coordinate system remains stable relative to the external environment, providing a reference benchmark that does not change with the movement of the shooting platform during target position analysis and trajectory prediction. In other words, it carries the observation information after coordinate system transformation, allowing the target's position to be described within a unified and stable reference system. It is understood that by transforming the target detection results from the aforementioned image coordinate system to the aforementioned stable coordinate system, the nonlinear multi-target detection and tracking platform can effectively separate the influence of the target's own motion and the shooting platform's motion on target position estimation, thereby improving the stability and accuracy of target trajectory prediction and tracking results.

[0035] The aforementioned image coordinate system can be a coordinate system used to describe the positional relationship of the target in the aforementioned image data to be detected. A preset reference point in the image data to be detected can be used as the origin, and the row and column directions or pixel arrangement directions of the image can be used as the coordinate axis directions to quantitatively describe the spatial position of the target in the image.

[0036] In one possible embodiment, the aforementioned nonlinear multi-target detection and tracking platform first determines the spatial attitude state at the time of image frame acquisition based on attitude data, and determines the transformation relationship between the image coordinate system and the stable coordinate system accordingly; then, the position information of the target detection result in the image coordinate system is converted into the position information in the stable coordinate system through the aforementioned transformation relationship.

[0037] 104. Based on the observation information, the motion trajectory of the target is predicted to obtain the predicted trajectory data of multiple targets.

[0038] In this embodiment of the invention, the aforementioned nonlinear multi-target detection and tracking platform can estimate the motion state of the target at subsequent moments based on the aforementioned observation information. Specifically, the target state information can be determined based on the observation information, such as the target velocity or motion trend information calculated from the target position and historical observation sequences; then, the target state is extrapolated based on a preset nonlinear motion model to obtain the predicted state information of the target at subsequent moments.

[0039] The aforementioned preset nonlinear motion model can refer to a model used to describe the relationship between the target's motion state and time in a stable coordinate system. It is used to characterize the motion law in which the state change and time have a nonlinear relationship during the target's motion. It can be understood that the setting of the aforementioned preset nonlinear motion model can be based on the target's historical observation information, the target's possible motion characteristics, or the motion constraints under the application scenario. It is used to reflect the nonlinear motion behaviors that the target may have during actual motion, such as acceleration, turning, speed change, or path bending.

[0040] Specifically, in the aforementioned nonlinear multi-target detection and tracking platform, the pre-set nonlinear motion model is used in the prediction processing stage to extrapolate the target's state at subsequent time points based on the target's current state information, thereby obtaining the target's predicted state information and further generating predicted trajectory data. By introducing the aforementioned pre-set nonlinear motion model, the nonlinear multi-target detection and tracking platform can make more reasonable predictions of target motion trends when target motion behavior is complex or irregular, thus improving the accuracy and adaptability of multi-target trajectory prediction.

[0041] In one possible embodiment, the aforementioned preset nonlinear motion model can use an extended Kalman filter model to predict the target motion state, wherein the nonlinear state transition relationship is linearized to achieve a recursive estimation of the target motion state; however, the present invention is not limited to using an extended Kalman filter model, and other motion models based on the principle of nonlinear state estimation can also be used.

[0042] The aforementioned predicted trajectory data may refer to the output data obtained by the aforementioned nonlinear multi-target detection and tracking platform performing the aforementioned prediction processing. In a multi-target scenario, the aforementioned predicted trajectory data includes the predicted trajectories of multiple targets, which are used to maintain trajectory continuity when the observation information is not updated or fluctuates, and serve as the basic reference object for correction processing.

[0043] In another possible embodiment, the aforementioned nonlinear multi-target detection and tracking platform first determines the current motion state of each target based on observation information. The current motion state of the target includes at least its current position in a stable coordinate system, and can further incorporate historical observation information to infer the target's motion trend information. Subsequently, based on the current motion state of the targets, the platform invokes a preset nonlinear motion model to extrapolate the target's motion state at subsequent times, thereby obtaining the predicted state information of the targets at future times.

[0044] In multi-target scenarios, the aforementioned nonlinear multi-target detection and tracking platform performs the above-mentioned prediction processing on each target and generates prediction trajectory data for multiple targets based on the prediction state information corresponding to each target, so that each target corresponds to an independent prediction trajectory.

[0045] 105. Based on the observation information, the predicted trajectory data is corrected, and when the predicted trajectory data converges, the target tracking path data corresponding to the target detection result is output.

[0046] In this embodiment of the invention, the aforementioned nonlinear multi-target detection and tracking platform can update and adjust the aforementioned predicted trajectory data using the aforementioned observation information. Specifically, the nonlinear multi-target detection and tracking platform compares the current observed position of the target in the stable coordinate system with the predicted position at the corresponding time in the predicted trajectory data to obtain trajectory deviation information; then, it corrects and updates the predicted trajectory data based on the trajectory deviation information, so that the corrected predicted trajectory approaches the state corresponding to the observation information.

[0047] Understandably, the aforementioned nonlinear multi-target detection and tracking platform can determine the stability and reliability of the predicted trajectory by comparing whether the results converge during the continuous execution of prediction and correction processing. For example, the platform judges the corrected predicted trajectory data based on preset convergence conditions. These conditions may include trajectory deviation information being less than a preset threshold, trajectory update amount being less than a preset range within a certain number of consecutive time intervals, or equivalent stability criteria. When the preset convergence conditions are met, the predicted trajectory data is considered to have reached a convergence state.

[0048] The aforementioned target tracking path data is the final trajectory result output by the aforementioned nonlinear multi-target detection and tracking platform after the predicted trajectory data reaches a convergence state. Generally, the aforementioned nonlinear multi-target detection and tracking platform outputs the converged predicted trajectory data as target tracking path data to characterize the continuous motion path of the target in a stable coordinate system. In multi-target scenarios, the aforementioned target tracking path data includes the tracking paths of multiple targets.

[0049] In this embodiment of the invention, image data to be detected and attitude data of a shooting platform synchronized with the image data to be detected are acquired; target detection processing is performed on the image data to be detected to obtain at least one target detection result; based on the attitude data, coordinate system transformation processing is performed on the at least one target detection result to obtain observation information corresponding to the target detection result; based on the observation information, the motion trajectory of the target is predicted to obtain predicted trajectory data of multiple targets; based on the observation information, the predicted trajectory data is corrected, and when the predicted trajectory data converges, the target tracking path data corresponding to the target detection result is output. By introducing attitude information to perform coordinate system transformation on the detection results and predicting and correcting the target motion trajectory under a stable reference, the stability and accuracy of multi-target detection and tracking in motion platform scenarios are improved.

[0050] Optionally, in the steps of acquiring the image data to be detected and the attitude data of the shooting motion platform synchronized with the image data to be detected, the image data to be detected collected by the shooting motion platform under the same time reference can also be acquired; and the attitude information of the shooting motion platform synchronized with the image data to be detected can be acquired.

[0051] In this embodiment of the invention, the nonlinear multi-target detection and tracking platform can also acquire the image data to be detected collected by the shooting motion platform at the same time reference, and acquire the attitude information of the shooting motion platform that is synchronized with the image data to be detected in time.

[0052] The aforementioned time reference can be used to calibrate the time of image data and pose data. The aforementioned nonlinear multi-target detection and tracking platform uses the aforementioned time reference to attach a time identifier to each frame of image data to be detected and the corresponding pose information, so as to ensure the consistency of different types of data in the time dimension. The aforementioned time synchronization can refer to matching the posture information of the shooting motion platform when acquiring the image data to be detected with the image data in time, based on the aforementioned time reference, thereby ensuring that the posture information can truly reflect the motion state of the shooting motion platform at the moment of image acquisition.

[0053] Through the above methods and steps, the nonlinear multi-target detection and tracking platform can avoid spatial position errors caused by the time inconsistency between image data and attitude data, providing a reliable data foundation for subsequent coordinate system transformation and target trajectory prediction.

[0054] Optionally, the step of performing target detection processing on the image data to be detected to obtain at least one target detection result further includes performing spatial position change feature processing on the image data to be detected to obtain spatial position change feature information; determining multiple target regions in the image data to be detected based on the spatial position change feature information; performing target determination in the multiple target regions to determine the position of the target in the image data to be detected; and outputting at least one target detection result based on the position of the target in the image data to be detected.

[0055] In this embodiment of the invention, the aforementioned nonlinear multi-target detection and tracking platform can perform image analysis processing related to positional change detection on the image data to be detected, in order to extract features from the image that reflect the spatial relationship between different regions. Specifically, by analyzing the pixel distribution, regional structural differences, or local spatial changes in the image, spatial positional change feature information with obvious spatial positional change characteristics relative to the background can be identified, thereby providing a basis for the subsequent determination of the target region.

[0056] The aforementioned spatial position change feature information can refer to the intermediate result information obtained by the nonlinear multi-target detection and tracking platform after performing spatial position change feature processing. It is used to characterize the spatial position or structural change characteristics of different regions in the image data to be detected. Generally, it can exist in the form of feature maps, feature sets or equivalent data, and is used to indicate which locations in the image have a high probability of target presence.

[0057] The aforementioned target region can refer to the local region divided in the image data to be detected by the nonlinear multi-target detection and tracking platform based on the aforementioned spatial position change feature information. It can be a candidate region of the target, the range of which is smaller than the complete image region, and is used to limit the processing object for subsequent target determination.

[0058] In one possible embodiment, the aforementioned nonlinear multi-target detection and tracking platform can perform discrimination processing within the target area to determine whether a target to be detected exists within the target area. Generally, image features within the target area can be analyzed according to preset judgment conditions to confirm whether a target exists and determine the specific location of the target in the image.

[0059] The aforementioned preset judgment conditions are rules or standards pre-set by the nonlinear multi-target detection and tracking platform before performing target judgment processing. They are used to determine whether a target exists in the target area. For example, they are set according to the imaging characteristics of the target in the image data to be detected and the detection application scenario. The setting content may include spatial feature requirements, position change feature requirements, or equivalent discrimination criteria used to distinguish the target from the background.

[0060] The aforementioned target can refer to the object that needs to be detected and tracked. It can be represented in the above image data to be detected as an imaging entity with specific spatial distribution characteristics and positional change characteristics. Its position in the image can be determined through target detection processing.

[0061] Through the above methods and steps, the nonlinear multi-target detection and tracking platform can effectively reduce the target detection range under complex background conditions, improve the accuracy of target positioning and the reliability of detection results, and provide a stable initial input for subsequent coordinate system transformation and multi-target tracking processing.

[0062] Optionally, in the step of performing coordinate system transformation processing on at least one target detection result based on attitude data to obtain the observation information corresponding to the target detection result, the method further includes determining the spatial attitude state of the shooting motion platform when acquiring the image data to be detected based on the attitude data; determining the transformation relationship between the image coordinate system and the stable coordinate system based on the spatial attitude state; and converting the position information of the target detection result in the image coordinate system into the position information in the stable coordinate system based on the transformation relationship, and outputting the corresponding observation information.

[0063] In this embodiment of the invention, the aforementioned spatial attitude state refers to the spatial orientation and motion state of the shooting motion platform when acquiring the image data to be detected, as determined by the nonlinear multi-target detection and tracking platform based on the attitude data. This can generally be obtained through the nonlinear multi-target detection and tracking platform's analysis of the attitude data, and is used to characterize the attitude of the shooting motion platform relative to a stable reference direction at the moment of image acquisition.

[0064] The aforementioned transformation relationship refers to the mapping rule established by the nonlinear multi-target detection and tracking platform based on spatial attitude state, used to describe the correspondence between the image coordinate system and the stable coordinate system. This rule can be determined by the nonlinear multi-target detection and tracking platform according to the attitude parameters of the shooting motion platform relative to the stable coordinate system, and is used to characterize the correspondence between the image coordinate axis directions and the stable coordinate axis directions. This transformation relationship is used to map the target's position information in the image coordinate system to the stable coordinate system, thereby eliminating the influence of changes in the shooting motion platform's attitude on the target's position representation.

[0065] The aforementioned location information is data used to characterize the spatial position of the target in the coordinate system. This location information can be generated by the aforementioned nonlinear multi-target detection and tracking platform during the target detection processing stage, and is used to describe the target's spatial position in the image coordinate system. After coordinate system transformation, the aforementioned location information is converted into a positional expression in a stable coordinate system, reflecting the actual spatial position of the target in a stable reference system.

[0066] In one possible embodiment, the aforementioned nonlinear multi-target detection and tracking platform determines the spatial attitude state of the shooting motion platform when acquiring the image data to be detected based on the acquired attitude data. Then, it determines the transformation relationship between the image coordinate system and the stable coordinate system based on the spatial attitude state. According to this transformation relationship, the position information of the target detection result in the image coordinate system is converted into the position information in the stable coordinate system, and the corresponding observation information is output accordingly.

[0067] By using the above methods and steps, the influence of changes in the attitude of the shooting platform on the representation of the target position can be eliminated, so that the target position can be described in a unified and stable coordinate system, thereby providing an observation basis for the subsequent prediction and correction of the target's motion trajectory.

[0068] Optionally, the step of determining the transformation relationship between the image coordinate system and the stable coordinate system based on the spatial attitude state further includes determining the attitude parameters of the shooting motion platform relative to the stable coordinate system based on the spatial attitude state; determining the coordinate mapping relationship between the coordinate axis direction of the image coordinate system and the coordinate axis direction of the stable coordinate system based on the attitude parameters; and establishing a transformation relationship to map the position information in the image coordinate system to the position information in the stable coordinate system based on the coordinate mapping relationship.

[0069] In this embodiment of the invention, the aforementioned attitude parameters are data parameters determined by the nonlinear multi-target detection and tracking platform based on the spatial attitude state, used to quantitatively describe the spatial orientation of the shooting motion platform relative to the stable coordinate system. Generally, they can be obtained by the nonlinear multi-target detection and tracking platform through parsing the attitude data, and are used to reflect the spatial attitude characteristics of the shooting motion platform when acquiring the image data to be detected. For example, it can be angular parameters of the spatial orientation of the shooting motion platform, such as one or more of the heading angle, pitch angle, and roll angle; it can also include attitude change parameters used to describe attitude changes, such as angular velocity or attitude change rate.

[0070] The aforementioned coordinate mapping relationship refers to the rule determined by the nonlinear multi-target detection and tracking platform based on the attitude parameters, which describes the correspondence between the coordinate axis directions of the image coordinate system and the coordinate axis directions of the stable coordinate system. This rule can be generated by the nonlinear multi-target detection and tracking platform after analyzing the attitude parameters.

[0071] For example, the above coordinate mapping relationship can be reflected in the direction correspondence of the image coordinate axes relative to the stable coordinate axes, such as which direction axis in the stable coordinate system the horizontal axis and vertical axis in the image coordinate system correspond to respectively, or it can be reflected in the direction transformation relationship between different coordinate axes.

[0072] In one possible embodiment, the aforementioned nonlinear multi-target detection and tracking platform constructs and forms a relational model or mapping rule that can be used for subsequent processing based on the determined information. For example, the aforementioned nonlinear multi-target detection and tracking platform can generate mapping rules, transformation matrices, or equivalent mapping descriptions for coordinate transformation based on coordinate mapping relationships, so that the transformation relationship can be directly invoked in subsequent coordinate system transformation processing.

[0073] Optionally, the step of predicting the motion trajectory of a target based on observation information to obtain predicted trajectory data of multiple targets further includes determining the target state information corresponding to at least one target based on observation information; predicting the state of the target based on the target state information using a preset nonlinear motion model to obtain predicted state information of each target; and generating predicted trajectory data of multiple targets based on the predicted state information.

[0074] In this embodiment of the invention, the aforementioned target state information is a set of data determined by the nonlinear multi-target detection and tracking platform based on observation information, used to describe the current motion state of the target. This data includes, but is not limited to, target position and / or target velocity, obtained by parsing and organizing the observation information. It is used to quantitatively characterize the motion state data of the target in a stable coordinate system and may further include the target velocity calculated from continuous observation information. For example, the target state information may include the current position coordinates of the target in the stable coordinate system and the velocity information corresponding to the position change of the target between adjacent observation times.

[0075] The aforementioned preset nonlinear motion model can refer to the motion state description model pre-set by the aforementioned nonlinear multi-target detection and tracking platform before performing target trajectory prediction. It is used to characterize the motion characteristics of the target motion state that do not satisfy linear relationships during the time-varying process.

[0076] Specifically, the settings can be based on the target's historical observation information and the nonlinear motion behaviors that the target may exhibit during its actual movement, such as acceleration, turning, or speed change. For example, an extended Kalman filter model can be used to predict the target's motion state, or other nonlinear state estimation models can be used.

[0077] More specifically, it can be done through, for example Figure 3 The predicted model's effect is illustrated in the diagram. The actual trajectory of the target in the stable coordinate system exhibits nonlinear characteristics over time, with observed values ​​distributed around this true trajectory and accompanied by certain perturbations. The aforementioned nonlinear multi-target detection and tracking platform uses a pre-defined nonlinear motion model to describe the target's state changes over time based on continuous observation information, enabling the predicted trajectory to conform to the target's bending motion trend, thus forming a nonlinear predicted trajectory and its corresponding prediction confidence interval.

[0078] Compared to the prediction method based on linear assumptions in the comparison, the above-mentioned pre-defined nonlinear motion model can avoid the problem of the predicted trajectory deviating from the actual motion path when the target changes direction, speed, or trajectory bends, so that the prediction results are always updated around the actual motion trend of the target.

[0079] Through the above methods and steps, the nonlinear multi-target detection and tracking platform can obtain predicted trajectories that better match the actual motion behavior of targets in complex motion scenarios, providing a more stable and reliable prior basis for subsequent trajectory correction and convergence judgment, thereby improving the continuity and accuracy of multi-target detection and tracking.

[0080] In one possible embodiment, the aforementioned nonlinear multi-target detection and tracking platform can estimate the motion state of the target at subsequent times by using the aforementioned target state information and combining it with the aforementioned preset nonlinear motion model, thus completing the state prediction process. Generally, the state estimation result of the target at the prediction time can be obtained by extrapolating the current state of the target.

[0081] The aforementioned predicted state information may refer to the output result obtained by the aforementioned nonlinear multi-target detection and tracking platform after performing state prediction. It is used to describe the predicted motion state of the target at subsequent times and can be calculated by the aforementioned nonlinear multi-target detection and tracking platform according to a preset nonlinear motion model. It includes, but is not limited to, the predicted position of the target at the predicted time and the state parameters such as the predicted velocity corresponding to the predicted position.

[0082] In another possible embodiment, the aforementioned nonlinear multi-target detection and tracking platform constructs and outputs new data results based on existing calculation results. Specifically, the aforementioned nonlinear multi-target detection and tracking platform can form predicted trajectory data describing the changes of the target over time based on the predicted state information, for example, organizing the predicted state information corresponding to multiple predicted times into a continuous trajectory sequence.

[0083] Optionally, the steps of correcting the predicted trajectory data based on observation information and outputting the target tracking path data corresponding to the target detection result when the predicted trajectory data converges further include: determining the current observation position of the target in the stable coordinate system based on observation information; comparing the current observation position with the predicted position at the corresponding time in the predicted trajectory data to obtain trajectory deviation information; correcting and updating the predicted trajectory data according to the trajectory deviation information so that the corrected predicted trajectory data approaches the state corresponding to the observation information; and outputting the target tracking path data corresponding to the target detection result when the corrected predicted trajectory data meets the preset convergence conditions.

[0084] In this embodiment of the invention, the aforementioned current observation position is the target's current position data in a stable coordinate system, determined by the nonlinear multi-target detection and tracking platform based on observation information. For example, it could be the target's coordinate values ​​or equivalent position description parameters in the stable coordinate system.

[0085] The predicted position is the target position estimation result corresponding to the current time obtained by the nonlinear multi-target detection and tracking platform from the predicted trajectory data. It can be generated by the nonlinear multi-target detection and tracking platform in the prediction processing stage and recorded in the trajectory node at the corresponding time when generating the predicted trajectory data. For example, the predicted position can be the predicted coordinate value of the target at a certain predicted time in the stable coordinate system.

[0086] In this embodiment, the aforementioned nonlinear multi-target detection and tracking platform can perform comparative analysis on two types of position information to obtain differences. Specifically, the platform compares the current observed position and the predicted position in the same stable coordinate system, calculates the degree of difference between them, and thus provides a basis for subsequently generating trajectory deviation information, i.e., trajectory deviation information.

[0087] The aforementioned trajectory deviation information can refer to the data obtained by the aforementioned nonlinear multi-target detection and tracking platform based on the comparison results, which is used to characterize the difference between the current observation position and the predicted position. It can be generated by the aforementioned nonlinear multi-target detection and tracking platform after completing the comparison. Its content can include position difference, deviation magnitude or equivalent error characterization parameters, which are used to reflect the degree of deviation of the predicted trajectory at the current moment and to determine the direction and magnitude that the predicted trajectory should be adjusted.

[0088] In one possible embodiment, the aforementioned nonlinear multi-target detection and tracking platform can adjust the predicted trajectory data based on trajectory deviation information to form an updated trajectory. Specifically, the correction update incorporates trajectory deviation information into the update rules of the predicted trajectory data, corrects the predicted position, and can synchronously update subsequent nodes or related state variables of the trajectory, making the predicted trajectory more closely match actual observations.

[0089] The aforementioned preset convergence conditions can be the judgment rules set in advance by the nonlinear multi-target detection and tracking platform before outputting target tracking path data. These rules are used to determine whether the corrected predicted trajectory data has reached a stable and reliable state, including trajectory deviation information being less than a preset threshold, trajectory update amount being less than a preset range for a number of consecutive time periods, or trajectory state remaining stable within a preset time window.

[0090] like Figure 4 As shown, this embodiment of the invention also provides a nonlinear multi-target detection and tracking device 400, which includes: The first acquisition module 401 is used to acquire the image data to be detected and the attitude data of the shooting motion platform synchronized with the image data to be detected; The first processing module 402 is used to perform target detection processing on the image data to be detected to obtain at least one target detection result; The second processing module 403 is used to perform coordinate system transformation processing on the at least one target detection result based on the attitude data to obtain the observation information corresponding to the target detection result; The first prediction module 404 is used to predict the motion trajectory of the target based on the observation information to obtain predicted trajectory data of multiple targets. The first correction module 405 is used to correct the predicted trajectory data based on the observation information, and output the target tracking path data corresponding to the target detection result when the predicted trajectory data converges.

[0091] Optionally, the first acquisition module 401 mentioned above includes: The first acquisition submodule is used to acquire the image data to be detected collected by the shooting motion platform under the same time reference; The second acquisition submodule is used to acquire the posture information of the shooting motion platform that is time-synchronized with the image data to be detected. The posture information is used to determine the motion state of the shooting motion platform when acquiring the image data to be detected.

[0092] Optionally, the first processing module 402 mentioned above includes: The first processing submodule is used to perform spatial position change feature processing on the image data to be detected to obtain spatial position change feature information. The second processing submodule is used to determine multiple target regions in the image data to be detected based on the spatial position change feature information. The third processing submodule is used to perform target determination in the multiple target regions and determine the position of the target in the image data to be detected. The fourth processing submodule is used to output at least one target detection result based on the position of the target in the image data to be detected.

[0093] Optionally, the second processing module 403 mentioned above includes: The fifth processing submodule is used to determine the spatial attitude state of the shooting motion platform when acquiring the image data to be detected, based on the attitude data. The sixth processing submodule is used to determine the transformation relationship between the image coordinate system and the stable coordinate system based on the spatial attitude state; The seventh processing submodule is used to convert the position information of the target detection result in the image coordinate system into the position information in the stable coordinate system based on the transformation relationship, and output the corresponding observation information.

[0094] Optionally, the seventh processing submodule mentioned above includes: The first processing unit is used to determine the attitude parameters of the shooting motion platform relative to the stable coordinate system based on the spatial attitude state. The second processing unit is used to determine the coordinate mapping relationship between the coordinate axis direction of the image coordinate system and the coordinate axis direction of the stable coordinate system based on the attitude parameters. The third processing unit is used to establish a transformation relationship that maps the position information in the image coordinate system to the position information in the stable coordinate system based on the coordinate mapping relationship.

[0095] Optionally, the first prediction module 404 mentioned above includes: The first prediction submodule is used to determine the spatial attitude state of the shooting motion platform when acquiring the image data to be detected, based on the attitude data. The second prediction submodule is used to determine the transformation relationship between the image coordinate system and the stable coordinate system based on the spatial attitude state. The third prediction submodule is used to convert the position information of the target detection result in the image coordinate system into the position information in the stable coordinate system based on the transformation relationship, and output the corresponding observation information.

[0096] Optionally, the first correction module 405 mentioned above includes: The first correction submodule is used to determine the current observation position of the target in the stable coordinate system based on the observation information; The second correction submodule is used to compare the current observation position with the predicted position at the corresponding time in the predicted trajectory data to obtain trajectory deviation information; The third correction submodule is used to correct and update the predicted trajectory data according to the trajectory deviation information, so that the corrected predicted trajectory data approaches the state corresponding to the observation information. The fourth correction submodule is used to output the target tracking path data corresponding to the target detection result when the corrected predicted trajectory data meets the preset convergence conditions.

[0097] like Figure 5 As shown, this embodiment of the invention also provides an electronic device 500, including a processor, which can execute any of the above-described nonlinear multi-target detection and tracking methods.

[0098] Specifically, it includes a processor 501 and a memory 502, as well as a computer program stored in the memory 502 and capable of running on the processor 501, which executes a nonlinear multi-target detection and tracking method, wherein: The processor 501 executes the calculator program for the nonlinear multi-target detection and tracking method stored in the memory 502, and performs the following steps: Acquire the image data to be detected and the attitude data of the shooting motion platform synchronized with the image data to be detected; The image data to be detected is subjected to target detection processing to obtain at least one target detection result; Based on the attitude data, coordinate system transformation is performed on the at least one target detection result to obtain the observation information corresponding to the target detection result; Based on the observation information, the motion trajectory of the target is predicted to obtain predicted trajectory data of multiple targets. Based on the observation information, the predicted trajectory data is corrected, and when the predicted trajectory data converges, the target tracking path data corresponding to the target detection result is output.

[0099] Optionally, the processor 501 executes the acquisition of the image data to be detected and the attitude data of the shooting motion platform synchronized with the image data to be detected, including: Acquire the image data to be detected collected by the shooting motion platform at the same time reference; The attitude information of the shooting motion platform, which is time-synchronized with the image data to be detected, is obtained. The attitude information is used to determine the motion state of the shooting motion platform when acquiring the image data to be detected.

[0100] Optionally, the processor 501 performs the target detection processing on the image data to be detected to obtain at least one target detection result, including: Spatial position change feature processing is performed on the image data to be detected to obtain spatial position change feature information; Based on the spatial location change feature information, multiple target regions are identified in the image data to be detected; Target determination is performed in the multiple target regions to determine the position of the target in the image data to be detected; Based on the location of the target in the image data to be detected, at least one target detection result is output.

[0101] Optionally, the processor 501 performs coordinate system transformation processing on the at least one target detection result based on the attitude data to obtain observation information corresponding to the target detection result, including: Based on the posture data, the spatial posture state of the shooting motion platform when acquiring the image data to be detected is determined; Based on the spatial attitude state, determine the transformation relationship between the image coordinate system and the stable coordinate system; Based on the transformation relationship, the position information of the target detection result in the image coordinate system is converted into the position information in the stable coordinate system, and the corresponding observation information is output.

[0102] Optionally, the processor 501 performs the step of determining the transformation relationship between the image coordinate system and the stable coordinate system based on the spatial pose state, including: Based on the spatial attitude state, the attitude parameters of the shooting motion platform relative to the stable coordinate system are determined; Based on the attitude parameters, determine the coordinate mapping relationship between the coordinate axis directions of the image coordinate system and the coordinate axis directions of the stable coordinate system; Based on the coordinate mapping relationship, a transformation relationship is established to map the position information in the image coordinate system to the position information in the stable coordinate system.

[0103] Optionally, the processor 501 performs the prediction processing on the target's motion trajectory based on the observation information to obtain predicted trajectory data for multiple targets, including: Based on the observation information, target state information corresponding to the at least one target is determined, wherein the target state information includes target position and / or target velocity; Based on the target state information, the state of the target is predicted by a preset nonlinear motion model to obtain the predicted state information of each target. Based on the predicted state information, predicted trajectory data for multiple targets are generated.

[0104] Optionally, the processor 501 performs the step of correcting the predicted trajectory data based on the observation information, and outputs the target tracking path data corresponding to the target detection result when the predicted trajectory data converges, including: Based on the observation information, determine the current observation position of the target in the stable coordinate system; The current observation position is compared with the predicted position at the corresponding time in the predicted trajectory data to obtain trajectory deviation information; Based on the trajectory deviation information, the predicted trajectory data is corrected and updated so that the corrected predicted trajectory data approaches the state corresponding to the observation information; When the corrected predicted trajectory data meets the preset convergence condition, the target tracking path data corresponding to the target detection result is output.

[0105] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the nonlinear multi-target detection and tracking method or the application-side nonlinear multi-target detection and tracking method provided in this invention, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0106] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments can be done by a computer program instructing related hardware, and can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0107] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A nonlinear multi-target detection and tracking method, characterized in that, include: Acquire the image data to be detected and the attitude data of the shooting motion platform synchronized with the image data to be detected; The image data to be detected is subjected to target detection processing to obtain at least one target detection result; Based on the attitude data, coordinate system transformation is performed on the at least one target detection result to obtain the observation information corresponding to the target detection result; Based on the observation information, the motion trajectory of the target is predicted to obtain predicted trajectory data of multiple targets. Based on the observation information, the predicted trajectory data is corrected, and when the predicted trajectory data converges, the target tracking path data corresponding to the target detection result is output.

2. The nonlinear multi-target detection and tracking method as described in claim 1, characterized in that, The acquisition of the image data to be detected and the attitude data of the shooting motion platform synchronized with the image data to be detected includes: Acquire the image data to be detected collected by the shooting motion platform at the same time reference; The attitude information of the shooting motion platform, which is time-synchronized with the image data to be detected, is obtained. The attitude information is used to determine the motion state of the shooting motion platform when acquiring the image data to be detected.

3. The nonlinear multi-target detection and tracking method as described in claim 1, characterized in that, The step of performing target detection processing on the image data to be detected to obtain at least one target detection result includes: Spatial position change feature processing is performed on the image data to be detected to obtain spatial position change feature information; Based on the spatial location change feature information, multiple target regions are identified in the image data to be detected; Target determination is performed in the multiple target regions to determine the position of the target in the image data to be detected; Based on the location of the target in the image data to be detected, at least one target detection result is output.

4. The nonlinear multi-target detection and tracking method as described in claim 1, characterized in that, The step of performing coordinate system transformation on the at least one target detection result based on the attitude data to obtain the observation information corresponding to the target detection result includes: Based on the posture data, the spatial posture state of the shooting motion platform when acquiring the image data to be detected is determined; Based on the spatial attitude state, determine the transformation relationship between the image coordinate system and the stable coordinate system; Based on the transformation relationship, the position information of the target detection result in the image coordinate system is converted into the position information in the stable coordinate system, and the corresponding observation information is output.

5. The nonlinear multi-target detection and tracking method as described in claim 4, characterized in that, Determining the transformation relationship between the image coordinate system and the stable coordinate system based on the spatial pose state includes: Based on the spatial attitude state, the attitude parameters of the shooting motion platform relative to the stable coordinate system are determined; Based on the attitude parameters, determine the coordinate mapping relationship between the coordinate axis directions of the image coordinate system and the coordinate axis directions of the stable coordinate system; Based on the coordinate mapping relationship, a transformation relationship is established to map the position information in the image coordinate system to the position information in the stable coordinate system.

6. The nonlinear multi-target detection and tracking method as described in claim 1, characterized in that, Based on the observation information, the motion trajectory of the target is predicted to obtain predicted trajectory data for multiple targets, including: Based on the observation information, target state information corresponding to the at least one target is determined, wherein the target state information includes target position and / or target velocity; Based on the target state information, the state of the target is predicted by a preset nonlinear motion model to obtain the predicted state information of each target. Based on the predicted state information, predicted trajectory data for multiple targets are generated.

7. The nonlinear multi-target detection and tracking method as described in claim 1, characterized in that, The step of correcting the predicted trajectory data based on the observation information, and outputting the target tracking path data corresponding to the target detection result when the predicted trajectory data converges, includes: Based on the observation information, determine the current observation position of the target in the stable coordinate system; The current observation position is compared with the predicted position at the corresponding time in the predicted trajectory data to obtain trajectory deviation information; Based on the trajectory deviation information, the predicted trajectory data is corrected and updated so that the corrected predicted trajectory data approaches the state corresponding to the observation information; When the corrected predicted trajectory data meets the preset convergence condition, the target tracking path data corresponding to the target detection result is output.

8. A nonlinear multi-target detection and tracking device, characterized in that, include: The first acquisition module is used to acquire the image data to be detected and the attitude data of the shooting motion platform synchronized with the image data to be detected; The first processing module is used to perform target detection processing on the image data to be detected, and obtain at least one target detection result; The second processing module is used to perform coordinate system transformation processing on the at least one target detection result based on the attitude data to obtain the observation information corresponding to the target detection result; The first prediction module is used to predict the trajectory of the target based on the observation information to obtain predicted trajectory data of multiple targets. The first correction module is used to correct the predicted trajectory data based on the observation information, and output the target tracking path data corresponding to the target detection result when the predicted trajectory data converges.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the nonlinear multi-target detection and tracking method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the nonlinear multi-target detection and tracking method as described in any one of claims 1 to 7.

Images