A real-time video multi-target tracking scheduling method and system

Abstract

This invention relates to the field of visual recognition and intelligent control technology, and discloses a real-time video multi-target tracking and scheduling method and system. Based on the MTTrack framework, this solution collects multi-source state data of the target scene to generate a target detection set; it predicts the target motion state through Kalman filtering, combines IoU and ReID features to complete data association, and dynamically adjusts the matching cost weight according to the target evaluation value; it generates semantic event quadruples to connect the entire process, and calculates the comprehensive evaluation value from four dimensions: rules, time accumulation, multi-target propagation, and target-device coupling; it predicts future events and evaluation values ​​based on historical semantic event sequences and semantic reasoning graphs, generating optimal scheduling actions to achieve early closed-loop intervention; this invention solves the problems of tracking and decision-making disconnect, high misjudgment rate, delayed response, and easy loss of high-priority targets in traditional solutions, and is applicable to scenarios such as industrial security and smart homes.

Description

Technical Field

[0001] This invention relates to the field of visual recognition and intelligent control technology, specifically a real-time video multi-target tracking and scheduling method and system. Background Technology

[0002] Real-time video multi-object tracking is a key research direction in the field of computer vision, aiming to continuously detect, locate, and identify multiple objects in a video sequence. MTTrack, as a typical real-time multi-object tracking framework, typically involves the following core processes: acquiring target bounding boxes in each frame based on an object detector; predicting the target's position in the next frame using motion models such as Kalman filtering; and finally, calculating the intersection-over-union ratio (IoU) between the predicted and detected boxes and combining this with the target's appearance features to perform data association, thereby forming and maintaining the motion trajectories of each target. This technology has been widely applied in scenarios such as security monitoring, intelligent transportation, and human-computer interaction.

[0003] However, when applying such tracking technology to complex systems that require real-time decision-making and scheduling, existing solutions have significant drawbacks, mainly in the following three aspects: First, there is a serious disconnect between tracking and high-level decision-making. Traditional MTTrack systems primarily output low-level data such as target coordinates and trajectories, lacking high-level semantic information. Decision-making systems rely on additional, complex modules to parse and understand the tracking results, leading to a loose system architecture, fragmented information flow, and difficulty in forming a closed loop from perception to control.

[0004] Secondly, the assessment of targets is based on a single dimension, resulting in a high false alarm rate. Existing systems rely heavily on preset, isolated single rules when conducting risk assessments or prioritization, such as "entering a specific area will trigger an alarm." This approach fails to comprehensively consider multi-dimensional information such as the target's historical behavior, the interaction between targets, and the coupling relationship between the target and scene devices. It has poor adaptability in dynamic and complex real-world scenarios and is prone to false alarms and missed alarms.

[0005] Finally, the response model is passive and lagging, lacking foresight. Most systems adopt a reactive logic of "event occurrence - threshold trigger - response execution", taking action only after the anomaly or risk reaches a predetermined level. This "post-event handling" model has inherent delays. In scenarios that require rapid intervention, it often fails to effectively avoid risks or affect the user experience due to untimely response. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a real-time video multi-target tracking and scheduling method and system, which can effectively solve the problems mentioned in the background art.

[0007] To solve the above problems, the technical solution adopted by the present invention is: a real-time video multi-target tracking and scheduling method (MTTrack), comprising the following steps: S1: Collect multi-source state data of the target scene, including sensing data and device state data, and generate a target detection set; S2: MTTrack's dynamic multi-target tracking steps: Based on the target detection set, generate the state vector and attitude feature vector for each target, predict the target's motion state through Kalman filtering, and associate the target data by combining the Intersection over Union (IoU) and Re-identification (ReID) features; during the association process, dynamically adjust the weight of the matching cost based on the target's evaluation value (the evaluation value at the previous moment), prioritize the allocation of tracking resources to high-priority targets, and avoid the loss of tracking of high-priority targets; S3: Action Recognition and Semantic Event Generation Steps: Based on the target's state vector and posture feature vector, calculate the target's action probability, and abstract the action probability, target position, and context information into a semantic event quadruple; using semantic events as an intermediate carrier, connect the subsequent evaluation calculation, semantic reasoning, and scheduling processes to solve the problem of tracking and decision-making being disconnected; S4: Multi-dimensional evaluation fusion step: Based on the semantic events, calculate the comprehensive evaluation value of the target from four dimensions: rule evaluation item, time accumulation evaluation item, multi-target propagation evaluation item, and target-device coupling evaluation item, to achieve full-dimensional state evaluation and solve the problem of high misjudgment rate of single rule; S5: Semantic reasoning prediction step: Based on the target's historical semantic event sequence and the preset semantic reasoning graph, predict the semantic event at the next moment, and then generate a prediction evaluation value; S6: Advance scheduling intervention step: Based on the predicted evaluation value, generate the optimal scheduling action, perform closed-loop intervention on the target and equipment in advance, realize advance control of the state, and solve the problem of response lag.

[0008] Preferably, the prediction formula of the Kalman filter is: , ; This formula is the core prediction logic of the dynamic tracking step, providing a state prediction basis for subsequent target data association; in: : The current state vector of target i, containing state information such as position and velocity; F: State transition matrix, used to describe the motion model of the target; H: Observation matrix, used to map the state vector to the observation space; Process noise describes the uncertainty of the motion model; Observational noise describes the uncertainty of perceived data; : The observation vector of target i, which is the target observation data collected by the sensor.

[0009] Preferably, the semantic event quadruple is ,in, Target ID, used to uniquely identify the target being tracked; Action type: The current behavior of the identified target; Target spatial location, which is the target's position in the scene coordinate system; Contextual conditions, including global contextual information such as time and scene type; Based on semantic events and target state vector Calculate the overall evaluation value. The formula for calculating the overall evaluation is: ; Specifically: Rule Evaluation Substituting semantic events into the formula: ; in, The weight of the k-th rule (which can be set through training on historical data or expert experience to match the preset rule logic). The rule mapping function is a threshold-based judgment function or a pre-trained neural network model used to map semantic events to the evaluation probability of a single rule. The evaluation formula is as follows: Time assessment For rule evaluation within the time window Exponential weighted sum within: ; in, This is the attenuation coefficient, used to adjust the influence weight of historical behavior and achieve cumulative evaluation of historical behavior; Dissemination assessment items The calculation function is: ; in, Let be the interaction strength between target i and target j, used to measure the degree of influence between the two targets. The comprehensive evaluation value of target j in the neighborhood at the current time t is the comprehensive evaluation result calculated for target j using the above comprehensive evaluation formula, which is the core input for the propagation evaluation. Let i be the spatial distance between target i and target j. This is the distance attenuation factor; the greater the distance, the smaller the propagation effect. Target-equipment coupling assessment for: ; This function is used to evaluate the coupling state between the target and the device, describing the impact of the relative states of the target and the device. Where M represents the number of devices in the scene. The distance between the target and the equipment. The relative speed between the target and the equipment. This represents the priority weight level of device m, used to distinguish the importance of different devices; To prevent the coefficient from being divided by zero and to avoid calculation errors when the distance is 0; This is the speed weighting coefficient, used to adjust the degree of influence of relative speed.

[0010] Preferably, the prediction formula for semantic reasoning is: ; This formula takes a sequence of historical semantic events as input to predict the future behavior of a target, providing a basis for advance scheduling. Substitute the predicted events into the evaluation formula to generate a predictive evaluation. , ; in: The historical semantic event sequence of target i serves as the input sequence for reasoning. ; Semantic reasoning graph A collection of semantic event nodes. This is a set of edges connecting events, used to describe the patterns of behavior transition; the semantic reasoning graph The event nodes and transition probability edges are obtained by training on historical behavior data of the target scenario or by pre-setting event nodes and transition probability edges through custom scene behavior logic. The predicted semantic event at the next moment is the output of the reasoning. The transition probability of an event is used to measure the likelihood of a change in behavior.

[0011] Preferably, the optimization formula for the scheduling is: ; This formula is used to achieve the optimal balance between benefits and costs, providing the best decision-making results for advance scheduling; in: A is a set of optional scheduling actions, containing all executable device control actions; The benefit function of state intervention is used to measure the experience or safety benefits brought about by scheduling actions; Predicted and evaluated values ​​serve as input for scheduling. : The execution cost of action a, used to measure the energy or resource cost of scheduling actions.

[0012] A real-time video multi-target tracking scheduling system, used to implement the above-mentioned real-time video multi-target tracking scheduling method, includes: The data acquisition module is used to collect multi-source state data of the target scene and generate a target detection set; The MTTrack multi-target tracking module generates target state vectors based on the target detection set, predicts target states through Kalman filtering, and performs data association by combining IoU and ReID features. At the same time, it dynamically adjusts the matching cost weights based on the target's evaluation value and prioritizes the allocation of tracking resources to high-priority targets. The behavior recognition module is used to calculate the behavior probability based on state and posture features, generate semantic event quadruples, and connect the entire process with semantic events. The evaluation modeling module is used to calculate a comprehensive evaluation value based on semantic events from four dimensions: rules, time accumulation, multi-target propagation, and target-device coupling. The semantic reasoning module is used to predict the semantic event at the next moment based on the historical semantic event sequence and semantic reasoning graph, and input the predicted event into the evaluation modeling module to calculate the predicted evaluation value; The evaluation-driven scheduling module is used to generate optimal scheduling actions based on predicted evaluation values, enabling closed-loop intervention of targets and equipment in advance.

[0013] Preferably, the data acquisition module includes a visual sensor, a voice sensor, an environmental sensor, and a device status acquisition unit.

[0014] Preferably, the MTTrack multi-target tracking module includes a target detection unit, a Kalman filter prediction unit, a feature matching unit, and a dynamic weight adjustment unit.

[0015] Preferably, the behavior recognition module includes a posture recognition unit, a speech semantic parsing unit, and a semantic event generation unit; the semantic reasoning module includes a sequence encoding unit, a semantic graph reasoning unit, and a prediction and evaluation unit.

[0016] Preferably, the evaluation modeling module includes a rule matching unit, a time accumulation unit, a propagation calculation unit, and a coupling calculation unit; the evaluation-driven scheduling module includes an action optimization unit, a device control unit, and an early warning notification unit.

[0017] Compared with the prior art, the present invention provides a real-time video multi-target tracking and scheduling method and system, which has the following beneficial effects: 1. This invention uses structured semantic events as a unified intermediate carrier to connect the entire process from tracking, evaluation, reasoning to scheduling, thereby achieving seamless connection and closed-loop control between perceived information and decision-making instructions, solving the technical problems of rigid resource allocation and easy loss of high-priority targets in traditional multi-target tracking.

[0018] 2. This invention achieves a more comprehensive and accurate assessment of the target status by simultaneously integrating information from four dimensions: rules, time accumulation, multi-target propagation, and target-device coupling. This significantly reduces the false judgment rate and solves the problem of target tracking and behavior recognition, status assessment, and equipment scheduling being disconnected and unable to form a closed-loop decision-making process.

[0019] 3. This invention drives the system to schedule and intervene in advance by predicting the future behavior and state of the target, thereby realizing the transformation from passive response to proactive and predictive control, fundamentally improving the system's response timeliness and handling effect. Attached Figure Description

[0020] Figure 1 This is a flowchart of the real-time video multi-target tracking and scheduling method of the present invention; Figure 2 This is a block diagram of the real-time video multi-target tracking and scheduling system architecture of the present invention. Detailed Implementation

[0021] The present invention will now be described with reference to embodiments. It should be understood that the embodiments described herein are for the purpose of more clearly illustrating the invention, and not for limiting the scope of the invention.

[0022] The endpoints and any values ​​of the ranges disclosed herein are not limited to the precise ranges or values, and these ranges or values ​​should be understood to include values ​​close to these ranges or values. For numerical ranges, the endpoint values ​​of the various ranges, the endpoint values ​​of the various ranges and individual point values, and individual point values ​​can be combined with each other to obtain one or more new numerical ranges, which should be considered as specifically disclosed herein.

[0023] refer to Figure 1-2 ; As a specific embodiment 1 of the present invention: taking a stamping workshop as an example, it is necessary to prevent workers from accidentally entering the dangerous area of ​​the stamping equipment, and at the same time avoid the safety risks caused by the gathering of many people. The MTTrack framework is used to realize early warning and equipment intervention.

[0024] Visual sensors collect video data from the workshop to detect the position and posture of workers. Voice sensors collect voice commands and abnormal sounds from the workshop; Environmental sensors collect environmental status data from the workshop. The equipment status acquisition unit collects the operating status and switching signals of the stamping equipment; Execution process: Step 1: Multi-source data acquisition (t=0 time) The system collects current status data through the modules described above: Target 1: Worker A, detected position coordinates are (1.2, 3.5) (unit: m, with the stamping equipment as the origin), movement speed is (0.8, 0.2) m / s, posture characteristics are "walking forward, facing the stamping equipment". Target 2: Worker B, detected position coordinates are (2.5, 4.0), movement speed is (0, 0), currently operating at a normal workstation. Equipment: Stamping machine No. 1, location coordinates (0,0), current running status "automatic operation", equipment priority weight S_m=0.9 Context: 14:30 on a weekday, normal workshop production time Step 2: MTTrack Dynamic Multi-Target Tracking 2.1. Kalman Filter Prediction: Substituting into the Kalman formula, predict the state of worker A at the next moment: , ; The predicted location of worker A was calculated to be (0.4, 2.7), which provides a basis for subsequent correlation.

[0025] 2.2. Dynamic weight adjustment: Based on the evaluation value of worker A in the previous moment, the weight of the matching cost is dynamically adjusted to prioritize the allocation of tracking resources to worker A, ensuring that the tracking of high-priority targets does not lose frames and avoiding the loss of tracking of dangerous targets.

[0026] Step 3: Behavior Recognition and Semantic Event Generation The system identifies worker A's behavior and generates a semantic event quadruple: ; This semantic event serves as the unified carrier for subsequent calculations.

[0027] Step 4: Multi-dimensional evaluation and fusion calculation Substitute the values ​​into the comprehensive evaluation formula to calculate worker A's comprehensive evaluation score: ; The weighting coefficients are set to α=0.3, β=0.2, γ=0.1, and δ=0.4. The evaluation items for the four dimensions are calculated respectively. 4.1 Rule Evaluation Items: The rule mapping function matches the rule "worker is near the stamping equipment". Output 0.9 (representing the risk probability of this action), rule weight =0.8, substitute into the formula: ; 4.2 Time-cumulative assessment items: Time window =5s, attenuation coefficient =0.5, and the rule evaluation values ​​for worker A in the past 5 seconds were 0.1, 0.2, 0.4, 0.6, and 0.72 respectively. Substituting these values ​​into the formula: ; The calculation yielded: ; 4.3 Multi-target propagation assessment items: worker B in the neighborhood, interaction strength =0.3, Worker B's overall evaluation value =0.1, the distance between the two people =1.2m, distance attenuation factor =0.5, substitute into the formula: ; 4.4 Target-Equipment Coupling Evaluation Items: Distance between worker A and the stamping equipment =3.7m, relative speed =0.82m / s, zero-prevention coefficient =0.01, velocity weight =0.1, substitute into the formula: ; 4.5 Overall Evaluation Results: Substituting into the comprehensive evaluation formula, we obtain the current comprehensive evaluation value: ; Step 5: Semantic Reasoning Prediction Based on worker A's historical semantic event sequence and the workshop's semantic reasoning graph, the system predicts the semantic event for the next moment: ; The predicted event at time t+1 is "entering the danger zone of the stamping equipment". Substituting this into the evaluation formula, the predicted evaluation value is obtained: ; This value has exceeded the safety threshold of 0.7, indicating that a safety risk is imminent.

[0028] Step 6: Early intervention and scheduling Substituting into the scheduling optimization formula, calculate the optimal scheduling action: ; Action 1: Pause the stamping equipment and issue an early warning; Benefits =100 (Benefits from avoiding safety incidents), Costs =10 (loss due to equipment shutdown), Revenue - Cost = 90 Action 2: No intervention, benefit 0, cost 0, benefit - cost = 0 The system selected action 1, which stopped the operation of the stamping equipment in advance and issued an audible and visual warning to prevent workers from accidentally entering the dangerous area. This achieved early closed-loop intervention and real-time monitoring.

[0029] As a specific embodiment 2 of the present invention: taking whole-house smart home as the usage scenario, it is necessary to actively identify the user's needs and start devices such as water heaters and air conditioners in advance to achieve "seamless intelligence" without requiring manual operation by the user. Visual sensors (smart cameras) collect video data from the living room to detect the user's position and posture; The voice sensor collects the user's voice commands and sounds (microphone array); Environmental sensors (temperature and humidity sensors) collect temperature and humidity data in the bathroom and living room; The device status acquisition unit (multi-mode gateway) collects the operating status of devices such as water heaters and air conditioners; Execution process Step 1: Multi-source data acquisition (t=0 time) The system collects current status data through modules: Target 1: User, detected location coordinates are (5,0) (originating from the living room, at the doorway), movement speed is (1.0,0.5) m / s, just opened the door to return home. Device 1: Electric water heater, location coordinates (3,4) (bathroom location), current status is off, device priority S_m=0.8 Context: 6:30 PM, the user's off-get off work time, outdoor temperature 10 degrees Celsius. Step 2: MTTrack Dynamic Multi-Target Tracking 2.2 Kalman Filter Prediction: Substituting into the formula, the user's movement trajectory is predicted, and it is predicted that the user will arrive at the bathroom location in 30 seconds.

[0030] 2.2 Dynamic weight adjustment: Based on the user's evaluation value, priority is given to allocating tracking resources to the user to ensure that the user's tracking is not lost.

[0031] Step 3: Behavior Recognition and Semantic Event Generation The system identifies the user's behavior and generates a semantic event quadruple: ; Step 4: Multi-dimensional evaluation and fusion calculation Substituting into the comprehensive evaluation formula, calculate the user's comprehensive evaluation value, with weighting coefficients also set as α=0.3, β=0.2, γ=0.1, δ=0.4: 4.1 Rule Evaluation Items: Matching the user habit "After getting off work, users usually rest for a while before taking a shower," the rule mapping function outputs 0.8, indicating a rule weight. =0.7, substitute into the formula: ; 4.2 Time-cumulative assessment items: Time window =5s, the user's rule evaluation value in the past 5 seconds is 0, 0, 0.2, 0.4, 0.56, decay coefficient =0.5, substitute into the formula to calculate: ; 4.3 Multi-target propagation assessment items: There are no other objectives in the home setting, so this assessment item is 0.

[0032] 4.4 Target-Equipment Coupling Evaluation Items: Distance between user and water heater =4.12m, relative velocity =1.11m / s, substituting into the formula: ; 4.5 Overall Evaluation Results: ; Step 5: Semantic Reasoning Prediction Based on the user's historical behavior sequence and the family's semantic reasoning graph, the system predicts the user's semantic event in 30 seconds as "preparing to take a shower." Substituting this into the evaluation formula, the predicted evaluation value is obtained: ; This value exceeds the demand threshold of 0.7, indicating that the user will soon need hot water.

[0033] Step 6: Early intervention and scheduling Substitute the values ​​into the scheduling optimization formula to calculate the optimal action: Action 1: Turn on the water heater in advance to heat the water to 45 degrees Celsius. Benefits =50 (improved user experience), cost =5 (energy consumption for preheating), benefit - cost = 45; Action 2: Wait for user to manually operate, revenue 0, cost 0, revenue - cost = 0; The system selected action 1, which preheated the water heater. When the user entered the bathroom, the water heater had already preheated, and the user could use hot water directly without having to manually turn it on, achieving seamless and proactive intelligence. When the user left the bathroom, the system automatically turned off the water heater.

[0034] The technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0035] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made using the contents of the present invention under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A real-time video multi-target tracking scheduling method, based on MTTrack real-time multi-target tracking, characterized in that, Includes the following steps: S1: Collect multi-source state data of the target scene, including sensing data and device state data, and generate a target detection set; S2: MTTrack's dynamic multi-target tracking steps: Based on the target detection set, generate the state vector and attitude feature vector of each target, predict the target's motion state through Kalman filtering, and associate the target data by combining the intersection-over-union ratio (IoU) and ReID features; During the association process, the weight of the matching cost is dynamically adjusted based on the comprehensive evaluation value of the target at the previous moment, and tracking resources are allocated to high-priority targets first to avoid the loss of tracking of high-priority targets; S3: Action Recognition and Semantic Event Generation Steps: Based on the target's state vector and posture feature vector, calculate the target's action probability, and abstract the action probability, target position, and context information into a semantic event quadruple; using semantic events as an intermediate carrier, connect the subsequent evaluation calculation, semantic reasoning, and scheduling processes to solve the problem of tracking and decision-making being disconnected; S4: Multi-dimensional evaluation fusion step: Based on the semantic events, calculate the comprehensive evaluation value of the target from four dimensions: rule evaluation item, time accumulation evaluation item, multi-target propagation evaluation item, and target-device coupling evaluation item, to achieve full-dimensional state evaluation and solve the problem of high misjudgment rate of single rule; S5: Semantic reasoning prediction step: Based on the target's historical semantic event sequence and the preset semantic reasoning graph, predict the semantic event at the next moment, and then generate a prediction evaluation value; S6: Advance scheduling intervention step: Based on the predicted evaluation value, generate the optimal scheduling action, perform closed-loop intervention on the target and equipment in advance, realize advance control of the state, and solve the problem of response lag.

2. The real-time video multi-target tracking and scheduling method according to claim 1, characterized in that, The prediction formula for the Kalman filter is as follows: ， ； in: : The current state vector of target i, containing state information such as position and velocity; F: State transition matrix, used to describe the motion model of the target; H: Observation matrix, used to map the state vector to the observation space; Process noise describes the uncertainty of the motion model; Observational noise describes the uncertainty of perceived data; : The observation vector of target i, which is the target observation data collected by the sensor.

3. The real-time video multi-target tracking and scheduling method according to claim 2, characterized in that, The semantic event quadruple is ,in, Target ID, used to uniquely identify the target being tracked; Action type: The current behavior of the identified target; Target spatial location, which is the target's position in the scene coordinate system; Contextual conditions, including global contextual information such as time and scene type; Based on semantic events and target state vector Calculate the overall evaluation value. The formula for calculating the overall evaluation is: ； Specifically: Rule Evaluation Substituting semantic events into the formula: ； in, Let k be the weight of the k-th rule. The rule mapping function is a threshold-based judgment function or a pre-trained neural network model used to map semantic events to the evaluation probability of a single rule. The evaluation formula is as follows: Time assessment For rule evaluation within the time window Exponential weighted sum within: ； in, This is the attenuation coefficient, used to adjust the influence weight of historical behavior and achieve cumulative evaluation of historical behavior; Dissemination assessment items The calculation function is: ； in, Let be the interaction strength between target i and target j, used to measure the degree of influence between the two targets. The comprehensive evaluation value of target j in the neighborhood at the current time t is the comprehensive evaluation result calculated for target j using the above comprehensive evaluation formula, which is the core input for the propagation evaluation. Let i be the spatial distance between target i and target j. This is the distance attenuation factor; Target-equipment coupling assessment for: ； This function is used to evaluate the coupling state between the target and the device, describing the impact of the relative states of the target and the device. Where M represents the number of devices in the scene. The distance between the target and the equipment. The relative speed between the target and the equipment. This represents the priority weight level of device m, used to distinguish the importance of different devices; To prevent the coefficient from being divided by zero and to avoid calculation errors when the distance is 0; This is the speed weighting coefficient, used to adjust the degree of influence of relative speed.

4. The real-time video multi-target tracking and scheduling method according to claim 3, characterized in that, The prediction formula for semantic reasoning is: ； This formula takes a sequence of historical semantic events as input to predict the future behavior of a target, providing a basis for advance scheduling. Substitute the predicted events into the evaluation formula to generate a predictive evaluation. , ; in: The historical semantic event sequence of target i serves as the input sequence for reasoning. ; Semantic reasoning graph A collection of semantic event nodes. This is a set of edges connecting events, used to describe the patterns of behavior transition; the semantic reasoning graph The event nodes and transition probability edges are obtained by training on historical behavior data of the target scenario or by pre-setting event nodes and transition probability edges through custom scene behavior logic. The predicted semantic event at the next moment is the output of the reasoning. The transition probability of an event is used to measure the likelihood of a change in behavior.

5. The method according to claim 4, characterized in that, The optimization formula for the scheduling is: ； This formula is used to achieve the optimal balance between benefits and costs, providing the best decision-making results for advance scheduling; in: A is a set of optional scheduling actions, containing all executable device control actions; The benefit function of state intervention is used to measure the experience or safety benefits brought about by scheduling actions; Predicted and evaluated values ​​serve as input for scheduling. : The execution cost of action a, used to measure the energy or resource cost of scheduling actions.

6. A real-time video multi-target tracking and scheduling system, used to implement the real-time video multi-target tracking and scheduling method of any one of claims 1-5, characterized in that, include: The data acquisition module is used to collect multi-source state data of the target scene and generate a target detection set; The MTTrack multi-target tracking module generates target state vectors based on the target detection set, predicts target states through Kalman filtering, and performs data association by combining IoU and ReID features. At the same time, it dynamically adjusts the matching cost weights based on the target's evaluation value and prioritizes the allocation of tracking resources to high-priority targets. The behavior recognition module is used to calculate the behavior probability based on state and posture features, generate semantic event quadruples, and connect the entire process with semantic events. The evaluation modeling module is used to calculate a comprehensive evaluation value based on semantic events from four dimensions: rules, time accumulation, multi-target propagation, and target-device coupling. The semantic reasoning module is used to predict the semantic event at the next moment based on the historical semantic event sequence and semantic reasoning graph, and input the predicted event into the evaluation modeling module to calculate the predicted evaluation value; The evaluation-driven scheduling module is used to generate optimal scheduling actions based on predicted evaluation values, enabling closed-loop intervention of targets and equipment in advance.

7. A real-time video multi-target tracking and scheduling system according to claim 6, characterized in that, The data acquisition module includes a visual sensor, a voice sensor, an environmental sensor, and a device status acquisition unit.

8. A real-time video multi-target tracking and scheduling system according to claim 6, characterized in that, The MTTrack multi-target tracking module includes a target detection unit, a Kalman filter prediction unit, a feature matching unit, and a dynamic weight adjustment unit.

9. A real-time video multi-target tracking and scheduling system according to claim 6, characterized in that, The behavior recognition module includes a posture recognition unit, a speech semantic parsing unit, and a semantic event generation unit; the semantic reasoning module includes a sequence encoding unit, a semantic graph reasoning unit, and a prediction and evaluation unit.

10. A real-time video multi-target tracking and scheduling system according to claim 6, Its features are, The evaluation modeling module includes a rule matching unit, a time accumulation unit, a propagation calculation unit, and a coupling calculation unit; the evaluation-driven scheduling module includes an action optimization unit, a device control unit, and an early warning notification unit.

Images