A tunnel entrance foreign body identification method and system based on improved DeepSORT and difference detection

Abstract

A method and system for identifying foreign objects at tunnel entrances that improves DeepSORT and difference detection, comprising: inputting a video stream, processing it frame by frame, and extracting foreground targets using scene difference detection; gradually performing object matching and cascade matching on the extracted foreground targets with historically stored foreign object targets, short-term stored foreign object targets, confirmed and unconfirmed targets tracked in the DeepSORT method. Ultimately, the matched targets and targets that meet the movement characteristics and are less than the DeepSORT model parameter max_age are regarded as the final confirmed foreign object targets and tracked targets; the foreground targets and untracked targets are regarded as newly tracked targets; and the targets that meet the movement characteristics and are greater than max_age are regarded as the final confirmed foreign objects. The present invention enhances the ability to identify foreign objects and reduces misidentification caused by background interference. Compared with conventional methods, the present invention has the characteristics of high accuracy, multiple recognition types, and a wide range of applications.

Description

Technical Field

[0001] The present invention belongs to the field of image recognition technology, and relates to foreign object recognition in a specific outdoor tunnel entrance scene, and specifically to a tunnel entrance foreign object recognition method and system based on improved DeepSORT and difference detection. Background Art

[0002] Many tunnel entrances are in open sections, passing under soil or mountains, and there may be intrusions of people, small animals, floating objects, etc., as well as tree collapses, landslides, etc., which have a great impact on the safety of train operation. At present, most tunnel entrances have been installed with surveillance cameras. Using cameras to check for foreign objects intrusion at tunnel entrances can effectively improve the efficiency of emergency response to potential risks and abnormal scenes. Manual inspection of camera videos for foreign objects has the problems of large workload and low work efficiency; the method of using conventional motion detection algorithms to identify foreign objects has the problems of high false alarm rate and low practicality; the method of identifying foreign object targets based on deep learning object recognition methods has the problem of being unable to exhaustively enumerate foreign object categories and unable to collect sample data. Therefore, the present invention proposes a difference detection and recognition method that can identify mobile targets. At the same time, based on the objectification of mobile targets, the DeepSORT algorithm idea is introduced, and the method is improved to eliminate background interference. Summary of the Invention

[0003] In view of the deficiencies in the prior art, the present invention provides a method and system for identifying foreign objects at tunnel entrances by improving DeepSORT and difference detection.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for identifying foreign objects at tunnel entrances based on improved DeepSORT and difference detection, characterized by comprising the following steps:

[0006] Step 1: Input the video stream containing foreign objects and extract them frame by frame. Use the scene difference detection model to extract the differentiated foreground objects in each frame of the video.

[0007] Step 2: Extract and match the differentiated foreground targets with the historical foreign object target dataset to obtain matched foreground targets and unmatched foreground targets;

[0008] Step 3: Extract and match the unmatched foreground targets in step 2 with the short-term foreign body target dataset to obtain matched foreground targets and unmatched foreground targets;

[0009] Step 4: Cascade matching is performed on the foreground targets matched in steps 2 and 3 and the foreground targets not matched in step 3, with the confirmed targets tracked in the DeepSORT method to obtain the matched foreground-tracked targets, the unmatched tracked targets, and the unmatched foreground targets;

[0010] Step 5: Extract and match the unmatched tracking targets and unmatched foreground targets in step 4 with the unconfirmed targets tracked in the DeepSORT method to obtain matched foreground-tracking targets, unmatched tracking targets, and unmatched foreground targets.

[0011] Step 6: Based on the foreground-tracking targets matched in steps 4 and 5, update the tracked targets, historical foreign object target dataset, and short-term foreign object target dataset;

[0012] Step 7: Take the foreground target that was not matched in step 5 as the new tracked target and update the short-term foreign object target dataset;

[0013] Step 8: For the tracking targets that were not matched in step 5, identify the movement characteristics, directly delete the targets that do not meet the movement characteristics, and compare the targets that meet the movement characteristics with the DeepSORT model parameter max_age. If it is greater than max_age, update the historical foreign object target dataset. If it is less than max_age, update the tracked target.

[0014] To optimize the above technical solutions, specific measures taken also include:

[0015] Furthermore, in step 1, the scene difference detection model adopts the transformer-based SCDNet model.

[0016] Furthermore, in step 2, the historical foreign body target dataset is a feature dataset of objects that have been historically tracked and confirmed as foreign bodies, and is used to store result images of objects that have been detected and confirmed as foreign bodies and feature result data extracted through features. Each tracked object stores feature result data three times, including feature result data of the start, end, and one frame taken according to feature differentiation during the tracking process.

[0017] Furthermore, in step 3, the short-term foreign body target data set is an object feature data set constructed based on the target being tracked within 5 to 10 seconds, and is used to store object feature data within 30 seconds.

[0018] Furthermore, in step 4, the foreground target matched in steps 2 and 3 indicates that a similar object has appeared before.

[0019] Furthermore, in step 6, the foreground-tracking target matched in steps 4 and 5 indicates that the current foreground target is the target object that has been tracked, and the target object is updated to the tracked target and updated to the tracking processing flow; at the same time, it is confirmed that the target object is a foreign target, and its target features are updated to the historical foreign target dataset and the short-term foreign target dataset.

[0020] Furthermore, in step 7, the foreground target not matched in step 5 represents a new target object that has not been tracked before. The target object is used as a new tracking target and updated to the tracking processing flow; at the same time, its target features are updated to the short-term foreign body target dataset.

[0021] Furthermore, in step 8, the identification of movement features adopts the calculation of the movement distance of the target trajectory, stores the target trajectory in each frame, calculates the distance between the initial position and the current position of the target, and when the distance is greater than 1 / 10 of the original image width, it is confirmed that the target meets the movement feature, otherwise it does not meet the movement feature.

[0022] Furthermore, it also includes using a loop traversal method to compare the object features of the target objects in the historical foreign body target data set one by one, and deleting the target objects with similar features.

[0023] The present invention also proposes a tunnel entrance foreign body identification system based on improved DeepSORT and difference detection, which is characterized by comprising:

[0024] The extraction module is used to extract the foreign objects from the input video stream frame by frame and use the scene difference detection model to extract the differentiated foreground objects in each frame of the video;

[0025] A first matching module is used to extract and match the object features of the differentiated foreground targets with the historical foreign object target dataset to obtain matched foreground targets and unmatched foreground targets;

[0026] A second matching module is used to extract and match the foreground targets that are not matched in the first matching module with the short-term foreign object target data set to obtain matched foreground targets and unmatched foreground targets;

[0027] a third matching module, configured to perform cascade matching on the foreground targets matched in the first matching module and the second matching module and the foreground targets not matched in the second matching module, with the confirmed targets tracked in the DeepSORT method, to obtain matched foreground-tracked targets, unmatched tracked targets, and unmatched foreground targets;

[0028] a fourth matching module, configured to extract and match object features of the unmatched tracking targets and unmatched foreground targets in the third matching module with the unconfirmed targets tracked in the DeepSORT method, thereby obtaining matched foreground-tracking targets, unmatched tracking targets, and unmatched foreground targets;

[0029] The update module updates the tracked targets, historical foreign object target datasets, and short-term foreign object target datasets based on the foreground-tracked targets matched in the third matching module and the fourth matching module; uses the foreground targets not matched in the fourth matching module as new tracked targets, and updates the short-term foreign object target dataset; for the tracked targets not matched in the fourth matching module, identifies the movement features, directly deletes the targets that do not meet the movement features, and compares the targets that meet the movement features with the DeepSORT model parameter max_age. When it is greater than max_age, the historical foreign object target dataset is updated; when it is less than max_age, the tracked targets are updated.

[0030] The beneficial effects of this invention are as follows: It proposes a foreign object detection method and system that integrates foreground difference detection with DeepSORT, effectively reducing false positives and improving accuracy; it introduces historical and short-term feature datasets of foreign object targets, enhancing foreign object recognition capabilities; and it introduces target object motion characteristics, further reducing false positives caused by background interference. Compared to other foreign object intrusion detection methods, this invention boasts high accuracy, low false positives, wide applicability, and high practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Flowchart of a method according to an embodiment of the present invention.

[0032] Figure 2a and Figure 2b They are schematic diagrams of the personnel intrusion scene at the tunnel entrance and the identification results of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0034] In one embodiment, if Figure 1 As shown, the present invention proposes a method for identifying foreign objects at tunnel entrances by improving DeepSORT and difference detection, comprising the following steps:

[0035] Step 1: Input the video stream containing foreign objects and extract them frame by frame. Use the scene difference detection model to extract the differentiated foreground targets in each frame of the video.

[0036] In step 1, a video stream containing foreign objects is input and each frame of the video is extracted. For each frame of the picture, a scene difference detection model is used. The difference detection model uses the transformer-based SCDNet model to extract the differentiated foreground target objects in each frame of the video.

[0037] Step 2: Extract and match the differentiated foreground with the historical foreign object target dataset to obtain the matched foreground targets and the unmatched foreground targets.

[0038] In step 2, this method constructs a historical foreign object target dataset for previously identified foreign objects. This dataset stores the result images of objects that have been detected and confirmed as foreign objects, as well as the feature result data extracted through feature extraction. Each tracked object stores three result data frames, including the initial and final result data, as well as at least one frame of target features during the tracking process, based on feature differentiation. The historical foreign object target dataset is a feature dataset of objects previously tracked and confirmed as foreign objects, retaining all foreign object types. Currently tracked objects are also included in this dataset, but any consecutive, similar features are deleted.

[0039] Furthermore, object features are extracted for the foreground target objects detected in each frame, and feature matching is performed with each object feature in the above-mentioned historical foreign object target dataset to detect whether the foreground target object has a matching historical object, and distinguish between matched foreground targets and unmatched foreground targets.

[0040] Step 3: For the foreground targets that were not matched in step 2, object features are extracted and matched with the short-term foreign object target dataset to obtain the matched foreground targets and the unmatched foreground targets.

[0041] In step 3, this method constructs a short-term foreign object target dataset for the currently tracked target within a certain period of time (typically set to 5-10 seconds). This dataset stores the currently tracked object feature data. Given the short-term, rapid changes in scene foreign objects, feature data within 30 seconds is stored. Both currently tracked, unconfirmed targets and currently tracked, confirmed targets are placed in the short-term foreign object target dataset. Continuous similarity target features are not deleted.

[0042] Furthermore, the foreground target features that are not matched in step 2 are feature matched with each object feature in the above-mentioned short-term foreign object target dataset, so as to detect whether the foreground target object has a matching tracked object and distinguish between matched foreground targets and unmatched foreground targets.

[0043] Step 4: For the foreground targets matched in steps 2 and 3, and the unmatched foreground targets in step 3, perform Matching Cascade matching with the confirmed tracking targets tracked in the DeepSORT method to obtain the matched foreground-tracking targets, unmatched tracking targets, and unmatched foreground targets.

[0044] In step 4, for the foreground target matched in steps 2 and 3, it means that a similar object has appeared before. The foreground target and the foreground target not matched in step 3 are matched with the confirmed tracking target tracked in the DeepSORT method to obtain the matched foreground-tracking target, the unmatched tracking target and the unmatched foreground target.

[0045] Step 5: For the unmatched tracking targets and unmatched foreground targets in step 4, object features are extracted and matched with the unconfirmed tracking targets tracked in the DeepSORT method to obtain matched foreground-tracking targets, unmatched tracking targets, and unmatched foreground targets.

[0046] In step 5, for the unmatched tracking targets, unmatched foreground targets and unconfirmed tracking targets tracked in the DeepSORT method in step 4, these three types of target objects are subjected to object feature extraction and matching to further obtain matched foreground-tracking targets, unmatched tracking targets and unmatched foreground targets.

[0047] Step 6: For the foreground-tracking targets matched in steps 4 and 5, update the tracked targets, the historical feature datasets confirmed as foreign targets (i.e., historical foreign target datasets), and the historical feature datasets confirmed as short-term foreign targets (i.e., short-term foreign target datasets).

[0048] In step 6, for the foreground-tracking target matched in steps 4 and 5, the current foreground target is identified as a tracked target object. This target object is updated to the tracked target, and the tracking process is further updated. Simultaneously, the current object is confirmed to be a foreign object, and its features are updated to the historical feature dataset for confirmed foreign objects. Furthermore, the target features are updated to the historical feature dataset for short-term foreign objects.

[0049] Step 7: Foreground targets that were not matched in step 5 are taken as new tracked targets, and the historical feature dataset confirmed as short-term foreign body targets is updated.

[0050] In step 7, for the foreground target that was not matched in step 5, it indicates that it has not been tracked before and is a new target object. As a newly tracked target, it is further updated to the tracking processing flow and the target feature is updated to the historical feature dataset of short-term foreign body targets.

[0051] Step 8: For the tracked targets not matched in Step 5, identify their motion characteristics. Targets that do not meet the motion characteristics are directly deleted. Targets that meet the motion characteristics and whose age is greater than max_age are identified as foreign objects in the historical feature dataset. If the age of targets that meet the motion characteristics is less than max_age, update the tracked targets. max_age is a Deep SORT model parameter, i.e., the maximum age, indicating the number of consecutive frames that meet the requirements. Here, max_age is the maximum age.

[0052] In step 8, for the tracking targets that were not matched in step 5, the motion features are identified. Since foreign objects in the scene do not have the characteristic of constant position, the moving distance of the target trajectory is calculated. The target trajectory in each frame is stored and the distance between the initial position and the current position of the target is calculated. When the distance is greater than 1 / 10 of the original image width, it is confirmed that the target meets the motion features. Otherwise, it does not meet the motion features. Targets that do not meet the motion features are directly deleted. Targets that meet the motion features are confirmed as foreign objects when they are greater than max_age. When the target that meets the motion features is less than max_age, the tracked target is updated.

[0053] Step 9: For the historical feature dataset of foreign objects, a periodic loop traversal method is used to delete target objects with similar features.

[0054] In step 9, for the historical feature data set of foreign objects, target objects with similar features are deleted, and a loop traversal method is used to compare the object features one by one, exclude similar target objects, and delete target objects with similar features.

[0055] Figure 2a and Figure 2b This is the result of identifying a human intrusion at a subway tunnel entrance using the above embodiment. It can be seen that the method proposed in this embodiment can effectively identify foreign objects at the tunnel entrance.

[0056] In another embodiment, the present invention provides a tunnel entrance foreign object recognition system corresponding to the tunnel entrance foreign object recognition method based on the improved DeepSORT and difference detection described in the above embodiment, comprising:

[0057] The extraction module is used to extract the foreign objects from the input video stream frame by frame and use the scene difference detection model to extract the differentiated foreground objects in each frame of the video;

[0058] A first matching module is used to extract and match the object features of the differentiated foreground targets with the historical foreign object target dataset to obtain matched foreground targets and unmatched foreground targets;

[0059] A second matching module is used to extract and match the foreground targets that are not matched in the first matching module with the short-term foreign object target data set to obtain matched foreground targets and unmatched foreground targets;

[0060] a third matching module, configured to perform cascade matching on the foreground targets matched in the first matching module and the second matching module and the foreground targets not matched in the second matching module, with the confirmed targets tracked in the DeepSORT method, to obtain matched foreground-tracked targets, unmatched tracked targets, and unmatched foreground targets;

[0061] a fourth matching module, configured to extract and match object features of the unmatched tracking targets and unmatched foreground targets in the third matching module with the unconfirmed targets tracked in the DeepSORT method, thereby obtaining matched foreground-tracking targets, unmatched tracking targets, and unmatched foreground targets;

[0062] The update module updates the tracked targets, historical foreign object target datasets, and short-term foreign object target datasets based on the foreground-tracking targets matched in the third matching module and the fourth matching module; uses the foreground targets not matched in the fourth matching module as new tracked targets, and updates the short-term foreign object target dataset; for the tracked targets not matched in the fourth matching module, identifies the movement features, directly deletes the targets that do not meet the movement features, and compares the targets that meet the movement features with the DeepSORT model parameter max_age. When it is greater than max_age, updates the historical foreign object target dataset; when it is less than max_age, updates the tracked targets.

[0063] In another embodiment, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the tunnel entrance foreign body identification method based on the improved DeepSORT and difference detection as described in the aforementioned embodiment.

[0064] In another embodiment, the present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for identifying foreign objects at the tunnel entrance using the improved DeepSORT and difference detection method as described in the aforementioned embodiment is implemented.

[0065] In the embodiments disclosed herein, computer storage media can be tangible media that can contain or store programs for use by or in conjunction with an instruction execution system, device, or apparatus. Computer storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0066] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0067] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A tunnel entrance foreign body identification method based on improved DeepSORT and difference detection, characterized in that: The steps include: Step 1: Input the video stream containing foreign objects and extract them frame by frame. Use the scene difference detection model to extract the differentiated foreground objects in each frame of the video. Step 2: Extract and match the differentiated foreground targets with the historical foreign object target dataset to obtain matched foreground targets and unmatched foreground targets; Step 3: Extract and match the unmatched foreground targets in step 2 with the short-term foreign object target dataset to obtain matched foreground targets and unmatched foreground targets; the short-term foreign object target dataset is an object feature dataset constructed based on the target being tracked within 5 to 10 seconds, and is used to store object feature data within 30 seconds; Step 4: Cascade matching is performed on the foreground targets matched in steps 2 and 3 and the foreground targets not matched in step 3, with the confirmed targets tracked in the DeepSORT method to obtain the matched foreground-tracked targets, the unmatched tracked targets, and the unmatched foreground targets; Step 5: Extract and match the unmatched tracking targets and unmatched foreground targets in step 4 with the unconfirmed targets tracked in the DeepSORT method to obtain matched foreground-tracking targets, unmatched tracking targets, and unmatched foreground targets. Step 6: Based on the foreground-tracking targets matched in steps 4 and 5, update the tracked targets, historical foreign object target dataset, and short-term foreign object target dataset; Step 7: Take the foreground target that was not matched in step 5 as the new tracked target and update the short-term foreign object target dataset; Step 8: For the tracking targets that were not matched in step 5, identify the movement characteristics, directly delete the targets that do not meet the movement characteristics, and compare the targets that meet the movement characteristics with the DeepSORT model parameter max_age. If it is greater than max_age, update the historical foreign object target dataset. If it is less than max_age, update the tracked target.

2. The method for identifying foreign objects at tunnel entrances using an improved DeepSORT and difference detection method as claimed in claim 1, characterized in that: In step 1, the scene difference detection model adopts the transformer-based SCDNet model.

3. The method for identifying foreign objects at tunnel entrances based on improved DeepSORT and difference detection as claimed in claim 1, characterized in that: In step 2, the historical foreign body target dataset is a feature dataset of objects that have been historically tracked and confirmed as foreign bodies, and is used to store the result images of objects that have been detected and confirmed as foreign bodies and the feature result data extracted by features. Each tracked object stores three feature result data, including the initial, final, and feature result data of one frame taken according to feature differentiation during the tracking process.

4. The method for identifying foreign objects at tunnel entrances using an improved DeepSORT and difference detection method as claimed in claim 1, characterized in that: In step 4, the foreground target matched in steps 2 and 3 indicates that a similar object has appeared before.

5. The method for identifying foreign objects at tunnel entrances using improved DeepSORT and difference detection as claimed in claim 1, characterized in that: In step 6, the foreground-tracking target matched in steps 4 and 5 indicates that the current foreground target is the tracked target object, and the target object is updated to the tracked target and the tracking process is updated; At the same time, the target object is confirmed to be a foreign object target, and its target features are updated to the historical foreign object target dataset and the short-term foreign object target dataset.

6. The method for identifying foreign objects at tunnel entrances by improving DeepSORT and difference detection as claimed in claim 1, characterized in that: In step 7, the foreground target not matched in step 5 represents a new target object that has not been tracked before, and the target object is used as a new tracking target and the tracking process is updated; At the same time, its target features are updated to the short-term foreign body target dataset.

7. The method for identifying foreign objects at tunnel entrances using improved DeepSORT and difference detection as claimed in claim 1, characterized in that: In step 8, the identification of movement features adopts the calculation of the movement distance of the target trajectory, stores the target trajectory in each frame, calculates the distance between the initial position and the current position of the target, and confirms that the target meets the movement feature when the distance is greater than 1 / 10 of the original image width, otherwise it does not meet the movement feature.

8. The method for identifying foreign objects at tunnel entrances based on improved DeepSORT and difference detection as claimed in claim 1, characterized in that: It also includes using a loop traversal method to compare the object features of the target objects in the historical foreign body target data set one by one, and deleting the target objects with similar features.

9. A tunnel entrance foreign body identification system based on improved DeepSORT and difference detection, characterized in that: include: The extraction module is used to extract the foreign objects from the input video stream frame by frame and use the scene difference detection model to extract the differentiated foreground objects in each frame of the video; A first matching module is used to extract and match the object features of the differentiated foreground targets with the historical foreign object target dataset to obtain matched foreground targets and unmatched foreground targets; a second matching module, configured to extract and match object features of foreground targets not matched in the first matching module with a short-term foreign object target dataset, thereby obtaining matched foreground targets and unmatched foreground targets; the short-term foreign object target dataset is an object feature dataset constructed based on the target being tracked within 5 to 10 seconds, and is configured to store object feature data within 30 seconds; a third matching module, configured to perform cascade matching on the foreground targets matched in the first matching module and the second matching module and the foreground targets not matched in the second matching module, with the confirmed targets tracked in the DeepSORT method, to obtain matched foreground-tracked targets, unmatched tracked targets, and unmatched foreground targets; a fourth matching module, configured to extract and match object features of the unmatched tracking targets and unmatched foreground targets in the third matching module with the unconfirmed targets tracked in the DeepSORT method, thereby obtaining matched foreground-tracking targets, unmatched tracking targets, and unmatched foreground targets; An updating module, which updates the tracked target, the historical foreign object target dataset, and the short-term foreign object target dataset based on the foreground-tracked target matched in the third matching module and the fourth matching module; The foreground targets that are not matched in the fourth matching module are taken as new tracked targets, and the short-term foreign object target dataset is updated; for the tracked targets that are not matched in the fourth matching module, the movement characteristics are identified, and the targets that do not meet the movement characteristics are directly deleted. The targets that meet the movement characteristics are compared with the DeepSORT model parameter max_age. When it is greater than max_age, the historical foreign object target dataset is updated. When it is less than max_age, the tracked targets are updated.

Images