Space-time joint multi-target association detection method and device, equipment and storage medium

Abstract

The application discloses a kind of spatio-temporal joint multi-target association detection method, device, equipment and storage medium, method includes: based on monitoring video stream, according to target detection network, identify the first element and second element in picture, obtain first element bounding box and second element bounding box;First element bounding box is tracked across frame, and first element is assigned unique identification;In each frame, the spatial overlap measure value of first element bounding box and second element bounding box is calculated, when spatial overlap measure value satisfies spatial constraint condition, the association relationship of first element and second element is established;Abnormal state cache pool is constructed, and the unmatched frame count of first element not associated to second element is calculated;When unmatched frame count reaches fault tolerance threshold, it is judged that corresponding first element unique identification is in violation and triggers alarm;When first element is successfully associated to second element, the unmatched frame count corresponding to first element unique identification is reset and removed from abnormal state cache pool.

Description

Technical Field

[0001] This application relates to a spatiotemporal joint multi-target correlation detection method, device, equipment, and storage medium for complex operation scenarios based on target detection and target tracking algorithms, belonging to the fields of computer vision and industrial safety monitoring technology. Background Technology

[0002] In the fields of special industries and large-scale equipment testing, computer vision-based monitoring of worker safety has become a core management tool for ensuring safe production. In typical open environments, traditional visual monitoring systems primarily use target detection algorithms to analyze video streams frame by frame, identifying workers and their safety equipment (such as helmets and safety harnesses). However, in special operational scenarios with large and complex physical structures, such as high-altitude simulation test benches for aero-engines (referred to as high-altitude test benches), the confined space, crisscrossing equipment pipelines, and frequent mechanical movement and personnel turnover easily lead to occlusion between workers and equipment, resulting in lost target identification. Relying solely on static image frame detection can cause serious problems such as missed target detection, identity changes, and ambiguity in the relationship between personnel and equipment, leading to numerous false alarms from the system. Summary of the Invention

[0003] To address the problems of high false alarm rate in single-frame detection, unclear relationship between people and equipment, and instantaneous missed detection in densely populated and frequently obstructed environments in existing technologies, this invention aims to propose a spatiotemporal joint multi-target association detection method, device, equipment, and storage medium based on target detection algorithms and target tracking technology for complex work scenarios. By constructing precise spatial topology constraints and a temporal state fault tolerance mechanism, the accuracy and robustness of identifying the relationship between people and safety equipment under complex working conditions are significantly improved while ensuring real-time performance.

[0004] The first aspect of this application discloses a spatiotemporal joint multi-target association detection method, the method comprising: acquiring a surveillance video stream; based on the surveillance video stream, identifying a first element and a second element in the image according to a target detection network, and obtaining the bounding boxes of the first element and the second element; performing cross-frame tracking on the bounding boxes of the first element and assigning a unique identifier to the first element; in each frame, calculating the spatial overlap metric between the bounding boxes of the first element and the second element, and establishing an association relationship between the first element and the second element when the spatial overlap metric satisfies a preset spatial constraint condition; constructing an abnormal state cache pool, calculating the unmatched frame count for the first element that is not associated with the second element; when the unmatched frame count reaches a preset fault tolerance threshold, determining that the unique identifier of the corresponding first element is in violation and triggering an alarm; when the first element is successfully associated with the second element, resetting the unmatched frame count corresponding to the unique identifier of the first element and removing it from the abnormal state cache pool.

[0005] Furthermore, the spatial overlap metric between the first feature bounding box and the second feature bounding box is calculated. When the spatial overlap metric satisfies the preset spatial constraint conditions, the association between the first feature and the second feature is established, including: traversing the second feature bounding boxes around the first feature bounding box and calculating the ratio of the intersection area to the union area between the two. When the ratio is greater than the preset empirical threshold, the association between the first feature and the second feature is established.

[0006] Furthermore, before traversing the second element bounding boxes surrounding the first element bounding box, the process includes: determining the coordinate center point of the first element bounding box; establishing an extended boundary range based on the coordinate center point, the extended boundary range enclosing the first element bounding box and having an area larger than the first element bounding box; and determining the second element bounding box as the object to be traversed in response to the intersection of the second element bounding box and the extended boundary range, where the intersection is defined as a non-empty intersection region between the second element bounding box and the extended boundary range.

[0007] Furthermore, the first element and the second element constitute a mandatory binding relationship. The method also includes: performing cross-frame tracking on the bounding box of the second element and assigning a unique identifier to the second element; when establishing the association relationship between the first element and the second element, simultaneously binding the unique identifier of the first element and the unique identifier of the second element; when the number of unmatched frames reaches a preset fault tolerance threshold, tracing the flow path of the unassociated second element backward and determining the flow status based on the current unassociated time point and the set traceback time. The flow status includes one of the following: the second element is left in a non-operational area, the second element is illegally mixed or lent, or the second element is abnormally lost or damaged.

[0008] Furthermore, the first element and the second element constitute a non-forced binding relationship. The method also includes: when it is detected that the unique identifier of the second element has not established an association relationship with any unique identifier of the first element for multiple consecutive frames, it is determined that the second element is in an idle state and removed from the active association list.

[0009] Furthermore, the monitoring video stream is the monitoring video stream from fixed or panoramic view cameras on the high-altitude platform; the unmatched frame count is the unmatched frame count of consecutive frames.

[0010] Furthermore, the first element is personnel, and the second element includes safety equipment; safety equipment includes safety helmets and safety ropes.

[0011] The second aspect of this application discloses a spatiotemporal joint multi-target association detection device, the device comprising: an acquisition module for acquiring a surveillance video stream; an identification module for identifying a first element and a second element in the image based on the surveillance video stream and a target detection network, thereby obtaining the bounding boxes of the first element and the second element; a tracking module for performing cross-frame tracking on the bounding boxes of the first element and assigning a unique identifier to the first element; an association module for calculating the spatial overlap metric between the bounding boxes of the first element and the second element in each frame, and establishing an association relationship between the first element and the second element when the spatial overlap metric satisfies a preset spatial constraint condition; a construction module for constructing an abnormal state cache pool and calculating the unmatched frame count for the first element that is not associated with the second element; an alarm module for determining that the unique identifier of the corresponding first element is in violation and triggering an alarm when the unmatched frame count reaches a preset fault tolerance threshold; and a reset module for resetting the unmatched frame count corresponding to the unique identifier of the first element and removing it from the abnormal state cache pool when the first element is successfully associated with the second element.

[0012] A third aspect of this application discloses a computer-readable storage medium comprising a stored program, wherein the program, when running, controls the execution of the spatiotemporal joint multi-target correlation detection method of the above embodiments in the processor of the device.

[0013] A fourth aspect of this application discloses a computer device, the computer device including a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed by the spatiotemporal joint multi-target association detection method of the above embodiments.

[0014] Compared with the prior art, the embodiments of this application have the following beneficial effects: (1) High robustness and low false alarm rate: Breaking through the limitations of traditional single-frame detection, the mechanism of cross-frame temporal constraints and continuous frame state counting is introduced to effectively overcome the occasional missed detection problem caused by frequent occlusion and significantly reduce the false alarm rate.

[0015] (2) Accurate attribution determination: The IoU (Intersection over Union) spatial matching logic is adopted to transform the spatial topological constraints of the target detection box into quantifiable indicators, clearly defining the correspondence between personnel and safety equipment (safety helmets, safety ropes), effectively solving the problem of ambiguous equipment attribution in densely populated scenarios.

[0016] (3) Balancing computational efficiency and anti-interference capability: An extended boundary range filtering mechanism is introduced before calculating IoU to pre-exclude interference targets in non-related areas at the far end of the screen. While ensuring the accuracy of association, the amount of invalid computation is greatly reduced, significantly improving the real-time processing capability of the algorithm in situations with high traffic, making it more suitable for the deployment needs of edge computing devices in complex industrial scenarios such as high-altitude platforms.

[0017] (4) Visualization and auditing of dynamic flow paths: Based on the bound dual ID relationship, this application supports reverse tracing of violations. When the alarm for not wearing the equipment is triggered, the historical bound equipment ID of the person can be automatically retrieved to trace the position and status of the equipment in the preceding frame. This provides a visualized chain of evidence for accident investigation and meets the stringent requirements for safety auditing in high-level industrial scenarios such as aviation and chemical industries. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0019] Figure 1 A flowchart of the spatiotemporal joint multi-target association detection method provided in the embodiments of this application.

[0020] Figure 2 A flowchart of the spatiotemporal joint multi-target association detection method provided in the embodiments of this application.

[0021] Figure 3 This is a structural diagram of the spatiotemporal joint multi-target correlation detection device provided in the embodiments of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Example 1: Figure 1 and Figure 2 This is a flowchart of the spatiotemporal joint multi-target association detection method provided in an embodiment of this application. Figure 1 and Figure 2 As shown, the method may include: Based on the surveillance video stream, S101 identifies the first and second elements in the image using a target detection network, and obtains the bounding boxes of the first and second elements.

[0025] Before this step, acquire the surveillance video stream.

[0026] This embodiment acquires real-time monitoring video streams from fixed or panoramic view cameras on the high-altitude test stand, which is referred to simply as the high-altitude test stand.

[0027] In this step, the object detection network includes, but is not limited to, any one or more combinations of the following: 1) a one-stage object detection network, selected from the YOLO series, SSD series, or RetinaNet series; 2) a two-stage object detection network, selected from the Faster R-CNN series; 3) a detection network based on the Transformer architecture, selected from the DETR series or the Swin Transformer series; 4) a lightweight detection network optimized for edge computing devices, selected from the MobileNet-SSD, YOLO-Lite, or NanoDet series.

[0028] The target detection network in this embodiment is the YOLOv5 network. In this embodiment, video frames are fed into the YOLOv5 network for inference to obtain the coordinate information of all detected targets in the current frame, including the personBox (first element bounding box) and the equipmentBox (second element bounding box).

[0029] In this step, the safety equipment includes at least one of the following: a) Personal protective equipment, including: safety helmet, safety rope, protective gloves, and safety glasses; b) Special operation equipment, including: fall arrestor for working at height, flame-retardant clothing, explosion-proof flashlight, and gas detector; c) Operation tools, including: wrench, hydraulic pliers, and screwdriver; d) Area control equipment, including: warning signs, fences, and fire-fighting equipment.

[0030] The safety equipment in this embodiment is a safety helmet and a safety rope.

[0031] S102 performs cross-frame tracking of the first feature's bounding box and assigns a unique identifier to the first feature.

[0032] During continuous video stream processing, the ByteTrack algorithm receives the set of personBoxes in the current frame and associates them with historical tracks. Regardless of the confidence level of the detection boxes, ByteTrack performs association calculations on these detection boxes, assigning or maintaining a unique trackId for each person in the frame, thereby establishing cross-frame association relationships.

[0033] In each frame of S103, the spatial overlap metric between the bounding box of the first feature and the bounding box of the second feature is calculated. When the spatial overlap metric satisfies the preset spatial constraint conditions, the association between the first feature and the second feature is established.

[0034] In some embodiments, a spatial overlap metric between the bounding boxes of the first and second features is calculated. When the spatial overlap metric satisfies a preset spatial constraint, an association relationship is established between the first and second features. This may include: S1031 determines the coordinate center point (Cx, Cy) of the first feature bounding box.

[0035] S1032 establishes an extended boundary range based on the coordinate center point. The extended boundary range surrounds the first feature's bounding box and has an area larger than the first feature's bounding box.

[0036] As an example, let the personBox have a width of W and a height of H. Using the center point (Cx, Cy) as a reference, expand outwards by k pixels in each direction. Typically, k = max(W, H) × 0.5.

[0037] S1033 responds to the intersection of the second feature bounding box and the extended boundary range, and determines the second feature bounding box as the object to be traversed.

[0038] In this embodiment, intersection is defined as: the second element bounding box and the extended boundary range have a non-empty intersection region in the planar coordinate system.

[0039] S1034 iterates through the bounding boxes of the second elements surrounding the bounding box of the first element, calculates the ratio of the intersection area to the union area between the two, and establishes the association between the first element and the second element when the ratio is greater than a preset empirical threshold.

[0040] As an example, for a personBox with a trackId identifier in the current frame, the equipmentBoxes surrounding its center point are extracted, and the ratio of their intersection area to their union area (denoted as iouValue) is calculated:

[0041] The calculated iouValue is compared with the system's preset empirical threshold iouThreshold. If iouValue is greater than iouThreshold, it is determined that the person is wearing the corresponding safety equipment in the current frame (indicating the establishment of an association); otherwise, it is determined that the person is not wearing the corresponding safety equipment in the single frame.

[0042] It should be noted that by adopting IoU spatial matching logic, the spatial topological constraints of the target detection box are transformed into quantifiable indicators, clearly defining the correspondence between personnel and safety equipment (safety helmets, safety ropes), effectively solving the problem of ambiguous equipment ownership in densely populated scenarios.

[0043] It is worth noting that an extended boundary range filtering mechanism is introduced before calculating IoU to preemptively exclude interfering targets in unrelated areas at the far end of the screen. While ensuring the accuracy of correlation, this significantly reduces the amount of unnecessary computation and greatly improves the algorithm's real-time processing capability in high-resolution video streams, making it more suitable for the deployment needs of edge computing devices in complex industrial scenarios such as aerial platforms.

[0044] S104 constructs an abnormal state cache pool, calculates the unmatched frame count for the first element that is not associated with the second element; when the unmatched frame count reaches the preset fault tolerance threshold, it determines that the unique identifier of the corresponding first element is in violation and triggers an alarm; when the first element is successfully associated with the second element, it resets the unmatched frame count corresponding to the unique identifier of the first element and removes it from the abnormal state cache pool.

[0045] In a preferred embodiment, the unmatched frame count is the unmatched frame count of consecutive frames.

[0046] Step S104 may include: S1041 maintains a list of abnormal states in memory, specifically for recording information about personnel who are not wearing the device in a single frame.

[0047] S1042 If step S103 determines that the person corresponding to a certain trackId is not wearing equipment, the system will first check whether the trackId already exists in the exception list: if it does not exist, a new record will be added and the unmatched count unmatchedCount will be initialized to 1; if it already exists, the count will be updated and unmatchedCount = unmatchedCount + 1 will be executed.

[0048] S1043 determines whether the current unmatchedCount of the trackId exceeds the preset alarm fault tolerance threshold (e.g., 5 consecutive frames): if it exceeds the threshold, it is determined that the person has committed a real violation and an alarm signal is immediately issued; if it does not exceed the threshold, no alarm is triggered and the next round of video frame recognition is entered.

[0049] If, in a subsequent frame, the trackId matches the security device (iouValue>iouThreshold), the abnormal state list will be actively cleared, the abnormal trackId and its corresponding unmatchedCount record in memory will be deleted, and the state will be reset, thereby eliminating the risk of false alarms caused by brief occlusion.

[0050] It is worth noting that this embodiment breaks through the limitations of traditional single-frame detection by introducing cross-frame temporal constraints and continuous frame state counting mechanisms, which effectively overcomes the problem of occasional missed detection caused by frequent occlusion and significantly reduces the false alarm rate.

[0051] In some embodiments, the method further includes: performing cross-frame tracking of the bounding box of the second element and assigning a unique identifier to the second element; when establishing an association between the first element and the second element, synchronously binding the unique identifier of the first element and the unique identifier of the second element; when the number of unmatched frames reaches a preset fault tolerance threshold, tracing the flow path of the unassociated second element backward and determining the flow status based on the current unassociated time point and the set traceback time.

[0052] In this embodiment, the first element and the second element form a mandatory binding relationship, that is, a worker must have a set of designated safety equipment, such as a dedicated safety helmet, during a specific work period.

[0053] Based on this mandatory binding relationship, when it is determined in step S104 that the number of unmatched frames has reached the preset fault tolerance threshold (i.e., it is determined that the personnel are violating regulations by not wearing equipment), the method also includes tracing the reverse flow path based on the equipment ID. A specific embodiment is as follows: Assuming worker A (trackId_01) is detected not wearing a safety helmet (hatId_005) for 5 consecutive frames, determine the current unassociated time point and record it in the post-binding relationship hatId_005-trackId_01. Based on the current unassociated time point, retrieve the cross-frame tracking trajectory of hatId_005 over the past T minutes (e.g., 5 minutes). Determine the workflow status: Scenario 1: If the trajectory of hatId_005 shows that its last stop was on the toolbox and it was stationary, the system determines that "equipment was left in a non-operational area" and will issue an alarm stating "Worker A is not wearing a safety helmet, which is currently located at the toolbox".

[0054] Scenario 2: If the trajectory of hatId_005 shows that it has established a relationship with another worker B (trackId_02) (i.e., a traceability binding relationship hatId_005-trackId_02 is generated), then the system determines that the traceability binding relationship hatId_005-trackId_02 is different from the subsequent binding relationship hatId_005-trackId_01, and judges it as "unauthorized mixing or lending of equipment", and prompts in the alarm "Worker A is not wearing a safety helmet, and the helmet has been taken away by worker B".

[0055] Scenario 3: If the trajectory of hatId_005 suddenly disappears (is obscured or moves out of the frame) and cannot be recaptured, the system determines it as "equipment abnormally lost or damaged".

[0056] Through the aforementioned traceability mechanism, this application not only enables real-time detection of personnel violations, but also achieves closed-loop management of equipment circulation status, solving the problem of "knowing only the violation, but not the reason" in traditional monitoring.

[0057] In some embodiments, the method further includes: when it is detected that the unique identifier of the second element has not been associated with any unique identifier of the first element for multiple consecutive frames, determining that the second element is in an idle state and removing it from the active association list.

[0058] In this embodiment, the first element and the second element form a non-mandatory binding relationship, that is, multiple workers can share the same set of safety equipment, such as a shared safety rope, during a specific work period.

[0059] In complex industrial operation scenarios, shared safety equipment exists in both active and idle states. This embodiment establishes an active association list in memory to enable focused monitoring and management of target association pairs currently performing operations. When a second element is determined to be in use, it is moved to the active association list; when a second element is determined to be idle, it is removed from the active association list.

[0060] Example 2: Figure 3 This is a structural diagram of the spatiotemporal joint multi-target correlation detection device provided in an embodiment of this application. Figure 3 As shown, the device may include: The acquisition module 201 is used to acquire the surveillance video stream.

[0061] The recognition module 202 is used to identify the first element and the second element in the image based on the surveillance video stream and according to the target detection network, and obtain the bounding box of the first element and the bounding box of the second element.

[0062] The tracking module 203 is used to perform cross-frame tracking of the bounding box of the first feature and assign a unique identifier to the first feature.

[0063] The association module 204 is used to calculate the spatial overlap metric between the bounding box of the first feature and the bounding box of the second feature in each frame. When the spatial overlap metric satisfies the preset spatial constraint conditions, the association relationship between the first feature and the second feature is established.

[0064] Module 205 is used to build an abnormal state cache pool and calculate the number of unmatched frames for first elements that are not associated with second elements.

[0065] The alarm module 206 is used to determine that the unique identifier of the corresponding first element is in violation and trigger an alarm when the number of unmatched frames reaches a preset fault tolerance threshold.

[0066] The reset module 207 is used to reset the count of unmatched frames corresponding to the unique identifier of the first element and remove them from the abnormal state cache pool when the first element is successfully associated with the second element.

[0067] Example 3: Embodiments of this application also provide a computer device, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of the present invention during runtime.

[0068] The aforementioned memory can refer to devices inside a computer used to store data and programs, including RAM, hard disks, etc. RAM can be used to temporarily store running programs and data, while hard disks can be used to store programs and data long-term. Memory enables the computer to read and write data and execute programs. The aforementioned processor is responsible for executing instructions in computer programs and performing data processing. It can also be responsible for controlling and executing various operations, including arithmetic operations, logical operations, and data transmission.

[0069] Example 4: Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of the present invention.

[0070] The aforementioned computer storage media can refer to the media used in computer memory to store certain discontinuous physical quantities. Computer storage media mainly include semiconductors, magnetic cores, magnetic drums, magnetic tapes, laser discs, etc. Computer-readable storage media include stored programs, which can be a set of instructions that a computer can recognize and execute, running on an electronic computer to meet certain information needs.

[0071] Example 5: Embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the methods of various embodiments of the present invention.

[0072] The aforementioned computer program products can refer to software programs that have been written, tested, and released, and can run on computers or other devices. Computer program products can include application programs, operating systems, utility software, etc., used to achieve specific functions or solve specific problems.

[0073] Example 6: Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the methods in various embodiments of the present invention.

[0074] The aforementioned non-volatile computer-readable storage medium can refer to a medium for storing data. Non-volatile computer-readable storage media can retain data without loss when power is off and can be used to store long-term data, such as operating systems, applications, and user files. Non-volatile storage media can include hard disk drives, solid-state drives, optical disks, and flash memory storage devices, etc.

[0075] Example 7: Embodiments of this application also provide a computer program that, when executed by a processor, implements the methods described in the various embodiments of the present invention.

[0076] The aforementioned computer program can refer to a set of instructions used to tell the computer to perform specific tasks or operations. Computer programs can be written by programmers using specific programming languages ​​and can include algorithms, data structures, logic, and control flow. Computer programs can be used for a variety of purposes, including application software, operating systems, etc.

[0077] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0078] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0079] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0080] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0081] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0082] In summary, to address the problems of high false alarm rate in single-frame detection, unclear attribution relationship between people and equipment, and instantaneous missed detection in densely populated and frequently obstructed environments, this invention aims to propose a spatiotemporal joint multi-target association detection method, device, equipment, and storage medium based on target detection algorithm and target tracking technology for complex work scenarios. By constructing a precise spatial topology constraint and temporal state fault tolerance mechanism, the accuracy and robustness of identifying the attribution relationship between people and safety equipment under complex working conditions are significantly improved while ensuring real-time performance.

[0083] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A spatiotemporal joint multi-target association detection method, characterized in that, include: Acquire surveillance video stream; Based on the surveillance video stream, and using the target detection network, the first element and the second element in the image are identified, and the bounding boxes of the first element and the second element are obtained. Perform cross-frame tracking on the bounding box of the first feature and assign a unique identifier to the first feature; In each frame, the spatial overlap metric between the bounding box of the first feature and the bounding box of the second feature is calculated. When the spatial overlap metric satisfies the preset spatial constraints, the association between the first feature and the second feature is established. Construct an abnormal state cache pool and calculate the number of unmatched frames for first elements that are not associated with second elements; When the number of unmatched frames reaches the preset fault tolerance threshold, the unique identifier of the corresponding first element is determined to be in violation and an alarm is triggered. When the first element is successfully associated with the second element, the count of unmatched frames corresponding to the unique identifier of the first element is reset and removed from the abnormal state cache pool.

2. The spatiotemporal joint multi-target correlation detection method according to claim 1, characterized in that, Calculate the spatial overlap metric between the bounding boxes of the first and second features. When the spatial overlap metric satisfies preset spatial constraints, establish the association between the first and second features, including: Traverse the bounding boxes of the second features surrounding the bounding box of the first feature, and calculate the ratio of the intersection area to the union area between them; When the ratio is greater than a preset empirical threshold, a correlation is established between the first element and the second element.

3. The spatiotemporal joint multi-target correlation detection method according to claim 2, characterized in that, Before traversing the second feature bounding box surrounding the first feature bounding box, include: Determine the coordinate center point of the first element's bounding box; Using the center point of the coordinate system as a reference, establish an extended boundary range that surrounds the bounding box of the first feature and has an area larger than the bounding box of the first feature. In response to the intersection of the second feature bounding box and the extended boundary range, the second feature bounding box is determined as the object to be traversed. The intersection is defined as the existence of a non-empty intersection region between the second feature bounding box and the extended boundary range.

4. The spatiotemporal joint multi-target correlation detection method according to claim 1, characterized in that, The first element and the second element constitute a mandatory binding relationship, and the method further includes: Perform cross-frame tracking on the bounding box of the second feature and assign a unique identifier to the second feature; When establishing a relationship between the first element and the second element, the unique identifiers of the first element and the second element are bound simultaneously. When the number of unmatched frames reaches the preset fault tolerance threshold, the flow path of the unmatched second element is traced back based on the current unassociated time point and the set trace time, and the flow status is determined. The flow status includes one of the following: the second element is left in a non-operation area, the second element is illegally mixed or lent, or the second element is abnormally lost or damaged.

5. The spatiotemporal joint multi-target correlation detection method according to claim 1, characterized in that, The first element and the second element constitute a non-mandatory binding relationship, and the method further includes: When it is detected that the unique identifier of the second element has not been associated with any unique identifier of the first element for multiple consecutive frames, the second element is determined to be in an idle state and removed from the active association list.

6. The spatiotemporal joint multi-target correlation detection method according to claim 1, characterized in that, The surveillance video stream is a surveillance video stream from fixed or panoramic cameras on a high-altitude platform; The unmatched frame count is the count of unmatched frames in consecutive frames.

7. The spatiotemporal joint multi-target correlation detection method according to claim 1, characterized in that, The first element is personnel, and the second element includes safety equipment; Safety equipment includes safety helmets and safety ropes.

8. A spatiotemporal joint multi-target correlation detection device, characterized in that, include: The acquisition module is used to acquire surveillance video streams; The recognition module is used to identify the first element and the second element in the image based on the surveillance video stream and the target detection network, and to obtain the bounding box of the first element and the bounding box of the second element. The tracking module is used to perform cross-frame tracking of the bounding box of the first feature and assign a unique identifier to the first feature. The association module is used to calculate the spatial overlap metric between the bounding box of the first feature and the bounding box of the second feature in each frame. When the spatial overlap metric satisfies the preset spatial constraints, the association relationship between the first feature and the second feature is established. The building module is used to build an abnormal state cache pool and calculate the number of unmatched frames for first elements that are not associated with second elements; The alarm module is used to determine that the unique identifier of the corresponding first element is in violation and trigger an alarm when the number of unmatched frames reaches a preset fault tolerance threshold. The reset module is used to reset the count of unmatched frames corresponding to the unique identifier of the first element and remove them from the abnormal state cache pool when the first element is successfully associated with the second element.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the spatiotemporal joint multi-target correlation detection method as described in any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the spatiotemporal joint multi-target correlation detection method as described in any one of claims 1 to 7.

Images