Method, system and equipment for enhancing abnormal scene in safety limit of unmanned subway train and medium

Abstract

The invention discloses a method, a system and equipment for enhancing an abnormal scene in a safety limit of an unmanned subway train and a medium, and relates to the technical field of safe driving. The method comprises the following steps: firstly, constructing a pre-trained basic diffusion model, and generating initial subway abnormal scene video data according to an initial frame image and a text prompt; then, a texture enhancement module is used for reducing the texture contrast and improving the trueness of the video; a reference pool containing a real luggage case image is introduced, luggage features are extracted through an object enhancement module, cosine similarity with reference pool features is calculated to serve as a target loss function, and image details are enhanced. And finally, constructing an orbit segmentation algorithm and training the model to realize multi-dimensional enhancement of the abnormal scene. According to the method, an abnormal data scene is generated by means of a diffusion model, and the problem of insufficient training data in subway abnormal object detection is solved through data enhancement.

Description

Technical Field

[0001] The present invention relates to the technical field of safe driving, and particularly to a method, system, device and medium for enhancing abnormal scenarios within the safety clearance of an unmanned subway train. Background Art

[0002] In the current unmanned subway system that widely uses artificial intelligence technology, detecting abnormal objects within the safety clearance of the subway track provides necessary technical support for preventing accidents at public transportation hubs. The safety clearance of the subway track mainly includes the track and the areas around the front of the subway. Objects in these areas may pose a threat to the safe operation of the subway train. Detecting these objects early is crucial for minimizing accident risks and improving the reliability of subway train operation. For the task of detecting abnormal objects within the safety clearance of the subway track, obtaining a high-quality and large-scale dataset is the basis for training an efficient and accurate abnormal object detection model. Obtaining sufficient abnormal scenario data is crucial for promoting early detection and timely response, thereby preventing accidents caused by obstacles or other dangers that the subway train may encounter. However, due to the extremely low probability of abnormal situations occurring within the safety clearance area of the subway track, coupled with many restrictions of the real environment and data privacy, it is extremely difficult to obtain sufficient abnormal situation data. Therefore, how to obtain effective subway abnormal scenario data that can be used for training has become an urgent problem to be solved currently.

[0003] The current related research has the following deficiencies: 1) In the unmanned subway system, since real abnormal scenarios are difficult to capture and traditional data augmentation methods (such as rotation, flipping, color adjustment, etc.) cannot generate new subway abnormal scenarios or realistic complex environments, there is a lack of sufficient subway abnormal scenario data to train an efficient abnormal object detection model. 2) Although the abnormal scenario data generated according to the current technology helps to alleviate the problem of scarce training data for downstream detection tasks, the generated data may still have a gap with real-world data in terms of texture and details, affecting the training effect of the detection model. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, system, device and medium for enhancing abnormal scenarios within the safety clearance of an unmanned subway train, which can generate various abnormal data scenarios by means of a diffusion model and solve the problem of insufficient training data in subway abnormal object detection by performing data augmentation on these generated abnormal scenarios.

[0005] To achieve the above purpose, the present invention provides the following solutions:

[0006] A method for enhancing abnormal scenarios within the safety clearance of an unmanned subway train, comprising:

[0007] Construct a pre-trained basic diffusion model, and use the basic diffusion model to generate initial subway abnormal scene video data according to the initial frame image and text prompt; the basic diffusion model adopts a ConsistI2V model constructed based on an improved spatio-temporal attention mechanism; the improved spatio-temporal attention mechanism includes applying attention operations on the spatial and temporal layers respectively;

[0008] Use a texture enhancement module to reduce the texture contrast in the initial subway abnormal scene video data to obtain subway abnormal scene video data with enhanced authenticity;

[0009] Introduce a reference pool containing real suitcase images, use an object enhancement module to extract features from the luggage specific area in the subway abnormal scene video data with enhanced authenticity, and calculate the cosine similarity between the extracted features and the luggage features in the reference pool, and use the cosine similarity as the target loss function for image detail enhancement to obtain subway abnormal scene video data with enhanced details;

[0010] Construct an orbit segmentation algorithm, and use the orbit segmentation algorithm for model training to enable the trained model to achieve multi-dimensional enhancement of abnormal scenes; the orbit segmentation algorithm includes an initial mask and a combined mask; the initial mask is obtained from the trajectory area in the first frame of the input video; the combined mask is composed of the initial mask and an attention mask.

[0011] Optionally, the loss function of the basic diffusion model is to minimize the mean square error between the predicted noise and the target noise.

[0012] Optionally, the use of the texture enhancement module to reduce the texture contrast in the initial subway abnormal scene video data specifically includes:

[0013] Randomly crop the initial subway abnormal scene video data to obtain multiple image patches;

[0014] Apply multiple Gaussian filters to each of the image patches, sort them according to the texture diversity corresponding to the image patches, and perform image reconstruction according to the richness threshold to obtain a texture-rich image composed of texture-rich image patches and a texture-poor image composed of texture-poor image patches;

[0015] Calculate the texture contrast of each frame in the initial subway abnormal scene video data, and align the texture contrast with the real image contrast to obtain subway abnormal scene video data with enhanced authenticity; the texture contrast is the absolute value of the difference between the texture-rich image and the texture-poor image.

[0016] Optionally, the formula for calculating the texture diversity is:

[0017]

[0018] Among them, x i,j represents the position pixel value in the image block after multiple Gaussian filtrations, and M represents the number of pixels.

[0019] Optionally, the calculation formula of the track segmentation algorithm is:

[0020]

[0021] Among them, Q , K, and V represent the query, key, and value matrices of the text attention mechanism, D represents the mask of the trajectory, λ represents the intensity factor of the mask, and d k represents the dimension of the key matrix.

[0022] The present invention also provides an abnormal scenario enhancement system within the safety clearance of an unmanned subway train, including:

[0023] A pre-trained network construction unit for constructing a pre-trained basic diffusion model, and using the basic diffusion model to generate initial subway abnormal scenario video data according to the initial frame image and text prompt; the basic diffusion model adopts a ConsistI2V model constructed based on an improved spatio-temporal attention mechanism; the improved spatio-temporal attention mechanism includes applying attention operations on the spatial and temporal layers respectively;

[0024] A realism enhancement unit for using a texture enhancement module to reduce the texture contrast in the initial subway abnormal scenario video data to obtain realism-enhanced subway abnormal scenario video data;

[0025] A detail enhancement unit for introducing a reference pool containing real luggage images, using an object enhancement module to extract features from the luggage-specific area in the realism-enhanced subway abnormal scenario video data, calculating the cosine similarity between the extracted features and the luggage features in the reference pool, and using the cosine similarity as the target loss function for image detail enhancement to obtain detail-enhanced subway abnormal scenario video data;

[0026] An algorithm training unit for constructing a track segmentation algorithm and using the track segmentation algorithm for model training to enable the trained model to achieve multi-dimensional enhancement of abnormal scenarios; the track segmentation algorithm includes an initial mask and a combined mask; the initial mask is obtained from the trajectory area in the first frame of the input video; the combined mask is composed of the initial mask and an attention mask.

[0027] The present invention also provides an electronic device, including a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the method for enhancing abnormal scenarios within the safety clearance of an unmanned subway train as described above.

[0028] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method for enhancing abnormal scenarios within the safety clearance of an unmanned subway train as described above.

[0029] According to the specific embodiments provided by the present invention, the following technical effects are disclosed:

[0030] The present invention discloses a method, system, device and medium for enhancing abnormal scenarios within the safety clearance of an unmanned subway train. The method includes constructing a pre-trained basic diffusion model, and using the basic diffusion model to generate initial subway abnormal scenario video data according to an initial frame image and a text prompt; using a texture enhancement module to reduce the texture contrast in the initial subway abnormal scenario video data to obtain subway abnormal scenario video data with enhanced authenticity; introducing a reference pool containing real suitcase images, using an object enhancement module to extract features from the luggage specific area in the subway abnormal scenario video data with enhanced authenticity, and calculating the cosine similarity between the extracted features and the luggage features in the reference pool, and using the cosine similarity as a target loss function for image detail enhancement to obtain subway abnormal scenario video data with enhanced details; constructing an orbit segmentation algorithm, and using the orbit segmentation algorithm for model training to enable the trained model to achieve multi-dimensional enhancement of abnormal scenarios. The present invention can generate various abnormal data scenarios with the help of a diffusion model, and solve the problem of insufficient training data in subway abnormal object detection by performing data enhancement on these generated abnormal scenarios. Description of the Drawings

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0032] Figure 1 It is a flowchart of the method for enhancing the subway track safety clearance scenario in this embodiment;

[0033] Figure 2 It is a schematic diagram of the difference in texture contrast between real images and synthetic images in this embodiment;

[0034] Figure 3Schematic diagram of regional attention enhancement in this embodiment;

[0035] Figure 4 Visualization result diagram obtained by the subway track safety limit scenario enhancement method in this embodiment;

[0036] Figure 5 Schematic diagram of generation of other typical subway abnormal scenarios in this embodiment;

[0037] Figure 6 Schematic diagram of performance improvement in this embodiment. Detailed implementation manners

[0038] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0039] The purpose of the present invention is to provide an abnormal scenario enhancement method, system, device and medium within the safety limit of a driverless subway train, which can generate various abnormal data scenarios with the help of a diffusion model and enhance the training data in the detection of subway abnormal objects by enhancing these generated abnormal scenarios.

[0040] To make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0041] The present invention provides an abnormal scenario enhancement method within the safety limit of a driverless subway train, including:

[0042] Step 100: Construct a pre-trained basic diffusion model, and use the basic diffusion model to generate initial subway abnormal scenario video data according to an initial frame image and a text prompt; the basic diffusion model adopts a ConsistI2V model constructed based on an improved spatio-temporal attention mechanism; the improved spatio-temporal attention mechanism includes applying attention operations on the spatial and temporal layers respectively.

[0043] Step 200: Use a texture enhancement module to reduce the texture contrast in the initial subway abnormal scenario video data to obtain subway abnormal scenario video data with enhanced authenticity.

[0044] Step 300: Introduce a reference pool containing real luggage images. Use the object enhancement module to extract features from the luggage-specific regions in the subway anomaly scene video data with enhanced authenticity, and calculate the cosine similarity between the extracted features and the luggage features in the reference pool. Use the cosine similarity as the target loss function for image detail enhancement to obtain the subway anomaly scene video data with enhanced details.

[0045] Step 400: Construct an orbit segmentation algorithm and use the orbit segmentation algorithm for model training to enable the trained model to achieve multi-dimensional enhancement of the anomaly scene; the orbit segmentation algorithm includes an initial mask and a combined mask; the initial mask is obtained from the trajectory region in the first frame of the input video; the combined mask is composed of the initial mask and an attention mask.

[0046] As a specific implementation, each of the above steps is described in detail.

[0047] The subway track safety limit scene enhancement method based on the diffusion model includes three main implementation steps: First, use a pre-trained diffusion model to generate a preliminary anomaly scene. Then, use a texture enhancement module to ensure the authenticity and effectiveness of the generated video. After that, use an object enhancement module to improve the details of the generated video. In addition, we also use an object localization module to focus more attention on the track to ensure more accurate generation of abnormal objects (such as luggage or pedestrians) within the subway train track safety limit. The scene enhancement method consists of a pre-trained diffusion model, a texture enhancement module, an object generation and localization module, and an object enhancement module, and its specific structure is as Figure 1 shown.

[0048] Step 1: Use the ConsistI2V model as the pre-trained basic diffusion model, and use this model to generate the initial subway anomaly scene video data according to the initial frame image and text prompts. The ConsistI2V model uses an improved spatio-temporal attention mechanism, which enhances the connection between each frame and the initial frame by applying attention operations on the spatial and temporal layers respectively, ensuring the temporal smoothness and consistency between the generated video frames.

[0049] For the diffusion model, the loss function of the diffusion model hopes to minimize the mean square error between the predicted noise and the target noise, and its formula is:

[0050]

[0051] where, ∈ represents noise, ∈ θ (x t , c, t) represents the predicted value of the noise ∈ , x t represents the time step tThe video frames below, where c represents semantic features, represents mean squared error calculation.

[0052] Step 2: In real life, real videos captured by cameras contain various random noises and interferences, showing more randomness and diversity in texture. Since the diffusion model lacks these naturally occurring noises, the generated videos appear overly smooth or perfect in texture, showing unnatural consistency or regularity. For a real captured video, one frame is intercepted. The texture contrast between the texture-rich area and the texture-poor area in this frame is very small, while the texture contrast between the high-texture-rich area and the texture-poor area in the frame generated by the diffusion model is very large, as Figure 2 shown. Therefore, by aligning the texture between the video generated by the diffusion model and the real video, and reducing the contrast between the texture-rich area and the texture-poor area in each frame of the generated video, the abnormal scenes generated by the diffusion model can be made closer to the texture distribution of the real world, thus ensuring the authenticity of the generated abnormal scene data and being able to function as expanding the dataset for downstream anomaly detection tasks.

[0053] As Figure 2 shown, we first randomly crop the generated image frames into multiple image patches, apply multiple Gaussian filters to these image patches, and sort these image patches according to their texture diversity. This process suppresses the semantic information of the image, forcing the pre-trained video generation model to pay more attention to the alignment of the texture distribution between the real image and the generated image in the next optimization process, and reducing the attention to the alignment of semantic information. Then, we use these image patches to reconstruct two images respectively, one composed of texture-rich image patches and the other composed of texture-poor image patches.

[0054] The formula for texture diversity is as follows:

[0055]

[0056] where x i,j represents the position pixel value in the image patch after multiple Gaussian filters, and M represents the number of pixels.

[0057] Taking the absolute value of the difference in texture diversity between the texture-rich area and the texture-poor area in each frame of the video generated by the diffusion model as the texture contrast can help the diffusion model make the texture distribution of the generated frames consistent with the texture presented by the real video during the generation of abnormal scenes, playing a role in texture enhancement. The texture loss function of the texture enhancement module is:

[0058]

[0059] Among them, N represents the total number of video frames generated, and l div,rich represents the texture diversity of the image composed of texture-rich image patches, and l div,poor represents the texture diversity of the image composed of texture-poor image patches.

[0060] Step 3: To ensure better presentation of the details of the abnormal scene (mainly referring to generating a discarded suitcase in this embodiment). Therefore, we introduce a reference pool containing real suitcase images. By extracting a specific area of the luggage from the generated video (the selection of this specific area is obtained through the track segmentation model), using the encoder to extract features of this area, and then comparing the cosine similarity between the extracted features and the luggage features in the reference pool. Taking the obtained cosine similarity as the target loss function, this process can make the generated suitcase approximate to the appearance of the real suitcase in the reference library, making the details of the generated abnormal scene more perfect.

[0061]

[0062] Among them, X f is the area for generating a given abnormal object, C j is the reference object pool, cosSim(.) is the cosine similarity calculation, and T is the threshold.

[0063] The total loss function of the scene enhancement method is:

[0064]

[0065] Among them, is the loss function of the diffusion model, is the loss function of the texture loss module, is the loss function of the target enhancement module, and μ1 and μ2 are hyperparameters for balancing the three loss functions.

[0066] Step 4: Train the track segmentation algorithm to guide the model to focus on the safe driving area in front of the subway train. The initial mask is obtained from the track area in the first frame of the video through the track segmentation algorithm. Then this mask is combined with the attention mask, which is generated by the pre-trained diffusion model through text prompts and is used to guide the diffusion model to focus on the track area. This process helps to improve the model's ability to accurately identify and locate the track during the generation process, ensuring that abnormal objects (such as objects crossing the track) can be more accurately located at the generated positions in the video. Figure 3 Shows the effect of this mechanism, and the calculation formula is as follows:

[0067]

[0068] Among them, Q , K, and VThe query, key, and value matrices representing the text attention mechanism, D represents the mask of the trajectory, λ represents the intensity factor of the mask, and d k represents the dimension of the key matrix.

[0069] Based on the above technical solution, the following embodiments are provided.

[0070] The prior scenario data used in this solution comes from the subway track driving videos collected in the real world on Line 16 of the Shanghai Metro. The video resolution is 3840×2160 and was filmed by the data collection team of Shanghai Quannuo. The video content mainly includes the track safety limit area in front of Line 16 of the Shanghai Metro, and subway abnormal scenarios such as placing suitcases in the track area and pedestrians repeatedly crossing the track were constructed for subsequent detection training.

[0071] The abnormal object detection model uses the Yolov5n, Yolov5m, Yolov8n, Yolov8m models, which have an efficient network architecture, data augmentation technology, and multi-scale feature fusion method, and can significantly improve the detection accuracy while maintaining real-time detection speed. We use mAP50 as the evaluation index in the abnormal object detection task, and its calculation formula is as follows:

[0072]

[0073] Among them, AP i is the average precision for the i-th category when the IoU threshold is 0.5, and N is the number of categories, which are two categories of people and suitcases in this solution.

[0074] The test results using the subway track safety limit scenario enhancement method proposed in this solution are as Figure 4 and Figure 5 shown, Figure Six demonstrating the improvement in the performance of the anomaly detection model using synthetic abnormal scenario data as training data. Note that the image was detected by the Yolov5m as the anomaly detection model, and the synthetic abnormal scenario data accounts for 40% of the total data.

[0075] As Figure 6 shown, this solution trains the generated abnormal scenario data and the collected real data as the dataset of the detection model, and uses the collected real data as the test set and validation set to illustrate the role of the subway track safety limit scenario enhancement method based on the diffusion model proposed in this solution in improving the abnormal object detection ability.

[0076] Table 1 shows the performance improvement of the anomaly detection model by adding synthetic abnormal scenario data to the training dataset. The evaluation index of the model is mAP50. Note: The percentage in parentheses represents the proportion of the enhanced scenario data in the total training data.

[0077] Table 1 Performance improvement of the synthesized abnormal scenario data on the abnormal detection model

[0078]

[0079] Meanwhile, ablation experiments were conducted on the proposed texture enhancement module and target enhancement module in this solution. The experimental results shown in Table 2 verified their performance improvement effects on training the abnormal detection model. The evaluation index of the model was mAP50. Note that in all comparison models, the synthesized abnormal scenario data accounted for 40% of the total training data.

[0080] Table 2 Ablation experiment

[0081]

[0082]

[0083] Therefore, the research of this solution mainly focuses on generating high-quality subway abnormal scenario data in the case of lack of real data, providing a new solution for insufficient training data, and at the same time generating abnormal scenario data with high similarity to real data. Moreover, this solution provides a feasible abnormal scenario data enhancement solution for the problem of detecting abnormal objects in the safe clearance area of the track of the subway unmanned train, improving the performance by 3%-6% in terms of the mAP50 index, proving that this method can improve the detection ability of the detection model and is helpful for improving the safety of the subway operation and maintenance system.

[0084] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other.

[0085] Specific examples are used in this article to elaborate on the principles and implementation manners of the present invention. The descriptions of the above embodiments are only used to help understand the core idea of the present invention; at the same time, for those of ordinary skill in the art, there will be changes in the specific implementation manners and application scopes according to the idea of the present invention. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. An enhanced method for abnormal scenarios within the safety clearance of an unmanned subway train, characterized in that, Including: Construct a pre-trained basic diffusion model, and use the basic diffusion model to generate initial subway abnormal scene video data according to the initial frame image and text prompt; The basic diffusion model adopts a ConsistI2V model constructed based on an improved spatio-temporal attention mechanism; the improved spatio-temporal attention mechanism includes applying attention operations on the spatial and temporal layers respectively; Use a texture enhancement module to reduce the texture contrast in the initial subway abnormal scene video data to obtain subway abnormal scene video data with enhanced authenticity; Introduce a reference pool containing real suitcase images, use an object enhancement module to extract features from the luggage specific area in the subway abnormal scene video data with enhanced authenticity, and calculate the cosine similarity between the extracted features and the luggage features in the reference pool. Use the cosine similarity as the target loss function for image detail enhancement to obtain subway abnormal scene video data with enhanced details; Construct an orbit segmentation algorithm, and use the orbit segmentation algorithm for model training to enable the trained model to achieve multi-dimensional enhancement of abnormal scenes; the orbit segmentation algorithm includes an initial mask and a combined mask; the initial mask is obtained from the track area in the first frame of the input video; the combined mask is composed of the initial mask and an attention mask.

2. The method for enhancing abnormal scenarios within the safety clearance of an unmanned subway train according to claim 1, wherein The loss function of the basic diffusion model is to minimize the mean square error between the predicted noise and the target noise.

3. The method for enhancing abnormal scenarios within the safety clearance of an unmanned subway train according to claim 1, characterized in that, The using the texture enhancement module to reduce the texture contrast in the initial subway abnormal scene video data specifically includes: Randomly crop the initial subway abnormal scene video data to obtain multiple image blocks; Apply multiple Gaussian filters to each of the image blocks, sort them according to the texture diversity corresponding to the image blocks, and perform image reconstruction according to the richness threshold after sorting to obtain a texture-rich image composed of texture-rich image blocks and a texture-poor image composed of texture-poor image blocks; Calculate the texture contrast of each frame in the initial subway abnormal scene video data, and align the texture contrast with the real image contrast to obtain subway abnormal scene video data with enhanced authenticity; the texture contrast is the absolute value of the difference between the texture-rich image and the texture-poor image.

4. The method for enhancing abnormal scenarios within the safety clearance of an unmanned subway train according to claim 3, wherein, The calculation formula for the texture diversity is: Among them, x i,j represents the position pixel value in the image block after multiple Gaussian filtrations, and M represents the number of pixels.

5. The method for enhancing abnormal scenarios within the safety clearance of an unmanned subway train according to claim 1, wherein The calculation formula for the orbit segmentation algorithm is: Among them, Q , K, and V represent the query, key, and value matrices of the text attention mechanism, D represents the mask of the trajectory, λ represents the intensity factor of the mask, and d k represents the dimension of the key matrix.

6. An abnormal scene enhancement system within the safety clearance of an unmanned subway train, characterized in that, Including: A pre-trained network construction unit for constructing a pre-trained basic diffusion model and using the basic diffusion model to generate initial subway abnormal scene video data according to the initial frame image and text prompt; The basic diffusion model adopts a ConsistI2V model constructed based on an improved spatio-temporal attention mechanism; the improved spatio-temporal attention mechanism includes applying attention operations on the spatial and temporal layers respectively; An authenticity enhancement unit for using a texture enhancement module to reduce the texture contrast in the initial subway abnormal scene video data to obtain subway abnormal scene video data with enhanced authenticity; A detail enhancement unit for introducing a reference pool containing real luggage images, using an object enhancement module to extract features from the luggage-specific regions in the video data of the subway abnormal scene with enhanced authenticity, calculating the cosine similarity between the extracted features and the luggage features in the reference pool, and using the cosine similarity as the target loss function for image detail enhancement to obtain the video data of the subway abnormal scene with enhanced details; An algorithm training unit for constructing a track segmentation algorithm and using the track segmentation algorithm for model training to enable the trained model to achieve multi-dimensional enhancement of abnormal scenes; the track segmentation algorithm includes an initial mask and a combined mask; the initial mask is obtained from the trajectory region in the first frame of the input video; the combined mask is composed of the initial mask and an attention mask.

7. An electronic device, characterized in that, It includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the method for enhancing abnormal scenes within the safety limit of an unmanned subway train according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a computer program, and when the computer program is executed by a processor, it implements the method for enhancing abnormal scenes within the safety limit of an unmanned subway train according to any one of claims 1-5.

Images