Pedestrian detection method, system and terminal based on mask and sub-region feature fusion on subway escalator

Abstract

This invention discloses a method, system, and terminal for pedestrian detection on subway escalators based on mask and regional feature fusion, belonging to the field of subway escalator pedestrian detection technology. The method involves masking the image; dividing the image of interest into upper and lower regions; using convolution at different scales to extract features from the upper and lower regions; fusing the features of the upper and lower regions to obtain the final feature map; and then performing target detection processing to obtain the final target detection result. The regional feature extraction and fusion method of this invention uses different convolution kernels to extract features from targets in different regions, which can balance the feature information of targets of different sizes, thereby improving the detection effect of large / small targets. By removing irrelevant regions and only detecting the escalator region, the detection speed can be improved, the amount of irrelevant computation can be reduced, and the influence of pedestrian features in irrelevant regions can be shielded, which is beneficial for extracting features for model training of pedestrians on escalators.

Description

Technical Field

[0001] This invention relates to the field of pedestrian detection technology on subway escalators, and in particular to a method, system and terminal for pedestrian detection on subway escalators based on the fusion of mask and regional features. Background Technology

[0002] Object detection and recognition, particularly pedestrian detection, holds a crucial position and has been extensively studied in recent decades. In the era of deep learning, convolutional neural network (CNN) pedestrian detection algorithms have become mainstream, with keypoint-based CNN algorithms being a current research hotspot and yielding substantial results. However, current pedestrian detection often occurs in relatively stable environments such as roads, sidewalks, or indoor spaces, where pedestrians typically stand at similar heights. These datasets are relatively abundant, and existing algorithms have demonstrated good performance. However, for complex application scenarios, such as subway escalators, there are still shortcomings in terms of detection accuracy and speed. Escalator pedestrian detection, compared to ordinary pedestrian detection, requires more robust target feature extraction, viewpoint invariance, and occlusion handling capabilities to address the challenges posed by the special environment and dynamic nature of the terrain. Therefore, it typically requires specially designed algorithms and datasets.

[0003] In urban rail scenarios, subway stations are usually underground and may face insufficient lighting, which leads to poor image quality and increases the difficulty of detection. At the same time, the subway environment often has occlusion, dense crowds and complex and varied backgrounds, resulting in fewer effective pixels, fewer features, and less obvious features of occluded targets. Moreover, they are largely submerged in noise and background clutter, making it difficult to extract effective features and easily causing false detection, missed detection and duplicate detection, thus affecting the performance of the target detection model. In addition, as the number of network layers increases, target features are easily lost. Existing target detection algorithms have the following problems: (1) Because subway cameras are always at an upward or downward angle, pedestrians in the distance are small. Without sufficient context information, it is easy to miss or falsely detect small targets. (2) In the area captured by the camera, there are other areas that are not of interest in addition to escalators. Common target detection algorithms will also detect pedestrians in the areas that are not of interest, which increases the detection calculation time. At the same time, the model training also captures the features of pedestrians in the areas that are not of interest. Summary of the Invention

[0004] The purpose of this invention is to overcome the problems of the prior art and provide a method, system and terminal for pedestrian detection on subway escalators based on the fusion of Mask and regional features.

[0005] The objective of this invention is achieved through the following technical solution: a method for pedestrian detection on subway escalators based on the fusion of mask and regional features, the method comprising the following steps:

[0006] Mask the image to obtain an image of the region of interest that retains only the escalator area;

[0007] The image of the region of interest is divided into upper and lower regions. Convolution at different scales is used to extract features from the upper and lower regions. The convolution kernel for the lower region feature extraction is larger than that for the upper region feature extraction. The features from the upper and lower regions are then fused to obtain the final feature map.

[0008] The final feature map is processed for object detection to obtain the final object detection result.

[0009] In one example, the image masking process includes:

[0010] Select the initial escalator area;

[0011] The original image is converted to a grayscale image, and then smoothed, gradient calculated, non-maximum suppressed, edge tracked, and thinned to obtain edge information.

[0012] The edge information is subjected to Hough transform to generate a family of straight lines;

[0013] The straight lines of the escalator are fitted to obtain the left handrail line and the right handrail line;

[0014] Calculate the intersection points of the left and right handrail lines with the image edge to obtain the polygonal region formed by the intersection points. Calculate the intersection of the polygonal region with the initial escalator region to obtain the intersection region. Determine the Mask region based on the intersection region.

[0015] Specifically, the intersection points of the left and right handrail lines with the image edge are calculated to obtain the polygonal region formed by the intersection points. The intersection of the polygonal region and the initial escalator region is calculated to obtain the intersection region. The Mask region of the original image is determined based on the intersection region.

[0016] Perform a bitwise AND operation between the mask region and the pixel values ​​of the original image to remove irrelevant regions and obtain the image of the region of interest.

[0017] In one example, the fitting process of the escalator straight line to obtain the left handrail line and the right handrail line includes the following steps:

[0018] Calculate the inclination angle slope of the left and right handrails of the escalator based on the initial escalator area, and remove interfering straight lines based on the inclination angle slope and the mean value of the straight line cluster.

[0019] The remaining straight lines are fitted using least squares to obtain the left and right handrail lines.

[0020] In one example, the image to be masked is a five-channel image, and the five-channel image generation steps include:

[0021] Edge detection and Hough transform are performed on the original image to obtain an edge information map and an orientation gradient histogram.

[0022] The edge information map, the directional gradient histogram, and the original RGB image are combined to form a five-channel image.

[0023] In one example, dividing the region of interest image into two regions (upper and lower) is replaced by dividing the region of interest image into three regions (upper, middle, and lower) or several regions. When the region of interest image is divided into three regions (upper, middle, and lower), feature extraction includes:

[0024] Different scale convolutions are used to extract features from the upper, middle, and lower regions. The convolution kernel for the lower region feature extraction is greater than that for the middle region feature extraction, which is greater than that for the upper region feature extraction. The features from the upper, middle, and lower regions are then fused to obtain the final feature map.

[0025] In one example, the intermediate region comprises a portion of the upper region and a portion of the lower region, in which case the feature fusion is as follows:

[0026] The features obtained from the upper and middle regions are fused to obtain feature map M1, and the features obtained from the lower and middle regions are fused to obtain feature map M2. Then, feature map M1 and feature map M2 are fused to obtain the final feature map.

[0027] It should be further noted that the technical features corresponding to the above examples can be combined or replaced to form new technical solutions.

[0028] This invention also includes a pedestrian detection system for subway escalators based on the fusion of mask and regional features, which has the same inventive concept as the above-mentioned detection method. This system includes:

[0029] The preprocessing unit is used to mask the image to obtain an image of the region of interest that retains only the escalator area;

[0030] A labeling unit is used to divide the image of the region of interest into two regions (upper and lower), three regions (upper, middle, and lower), or several regions.

[0031] The feature extraction unit is used to extract features from the upper and lower regions using convolutions of different scales. The convolution kernel for feature extraction in the lower region is larger than that for feature extraction in the upper region. The features from the upper and lower regions are then fused to obtain the final feature map.

[0032] The detection unit is used to perform detection processing on the final feature map to obtain the final target detection result.

[0033] In one example, the feature extraction unit is the backbone network of the YOLO network, and the detection unit is the neck network and the detection head of the YOLO network;

[0034] When the image of interest is divided into three regions, upper, middle, and lower, the backbone network uses convolutions of different scales to extract features from the upper, middle, and lower regions. The convolution kernel for extracting features from the lower region is greater than that for the middle region, which is greater than that for the upper region. The features from the upper, middle, and lower regions are then fused to obtain the final feature map.

[0035] In one example, the middle region includes a portion of the upper region and a portion of the lower region. In this case, the backbone network uses convolutions of different scales to extract regional features from the upper, middle, and lower regions:

[0036] The backbone network fuses the features obtained from the upper and middle regions to obtain feature map M1, and fuses the features obtained from the lower and middle regions to obtain feature map M2. Then, feature map M1 and feature map M2 are fused to obtain the final feature map.

[0037] It should be further noted that the technical features corresponding to the above examples can be combined or substituted to form new technical solutions.

[0038] The present invention also includes a storage medium storing computer instructions, which, when executed, perform the steps of the described method for pedestrian detection on subway escalators based on the fusion of Mask and regional features.

[0039] The present invention also includes a terminal comprising a memory and a processor, wherein the memory stores computer instructions executable on the processor, and the processor executes the steps of the method for pedestrian detection on subway escalators based on the fusion of mask and regional features described in any one or more of the above examples when executing the computer instructions.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] This invention presents a region-based feature extraction and fusion method that uses different convolutional kernels to extract features from targets in different regions. This balances the feature information of targets of different sizes, thereby improving the detection performance of both large and small targets and enhancing the robustness and accuracy of detection. Simultaneously, by removing irrelevant regions and detecting only the escalator region, detection speed is improved, irrelevant computation is reduced, and the influence of pedestrian features in irrelevant regions is shielded, facilitating the extraction of features for model training of pedestrians on escalators. Furthermore, by focusing only on the region of interest and shielding irrelevant regions, the model can allocate more resolution and attention to the target region of interest, making it easier to detect and locate small targets. This allows the model to better capture the details and features of target objects, thus improving detection performance. Further, shielding irrelevant regions effectively reduces the number of background samples, resulting in a more balanced positive and negative sample count, which helps the model better learn the features of small targets. Attached Figure Description

[0042] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, which are used to provide a further understanding of the present application and constitute a part of the present application. The same reference numerals are used in these drawings to denote the same or similar parts. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application.

[0043] Figure 1 This is a flowchart of a method in an example of the present invention;

[0044] Figure 2 This is a block diagram of image mask processing in an example of the present invention;

[0045] Figure 3 This is a diagram of the detection network structure in an example of the present invention;

[0046] Figure 4 This is a structural diagram of the SFE module in an example of the present invention. Detailed Implementation

[0047] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] In the description of this invention, it should be noted that the directions or positional relationships indicated by terms such as "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer" are based on the directions or positional relationships shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the use of ordinal numbers (e.g., "first and second," "first to fourth," etc.) is for distinguishing objects and is not limited to this order, and should not be construed as indicating or implying relative importance.

[0049] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0050] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0051] In one example, such as Figure 1 As shown, a pedestrian detection method for subway escalators based on the fusion of mask and regional features is proposed. Figure 1 As shown, it includes the following steps:

[0052] S1: Apply a mask to the image to obtain an image of the region of interest that retains only the escalator area;

[0053] S2: Divide the image of the region of interest into two regions, upper and lower, and use convolution of different scales to extract features from the upper and lower regions. The convolution kernel for the feature extraction of the lower region is larger than that for the feature extraction of the upper region. Then, perform feature fusion processing on the features of the upper and lower regions to obtain the final feature map. Optionally, the image of the region of interest can be divided into three regions, or even more regions, such as upper, middle and lower.

[0054] S3: Perform target detection processing on the final feature map to obtain the final target detection result, that is, to realize the detection of pedestrians on subway escalators.

[0055] In step S1, masking is performed by using a special tensor to process elements in irrelevant regions, setting these elements to 0 so they do not need to participate in subsequent calculations, thus removing irrelevant regions. For the pedestrian detection task on escalators, the area captured by the camera includes other areas that are not of interest besides the escalator. Therefore, masking the irrelevant regions of the image before feeding the image into the feature extraction network can reduce the amount of irrelevant computation.

[0056] In step S2, subway escalator cameras typically use a top / bottom view, resulting in significant size differences between pedestrians closer to and farther from the camera. To better capture information about targets of different sizes, the image is divided into upper and lower regions. Smaller targets are processed using smaller convolutional kernels, while larger people are processed using larger kernels. Therefore, we introduce a Sub-Region Feature Extraction (SFE) module to obtain rich contextual information for feature enhancement. Features are extracted from the upper and lower regions using convolutions of the same type but different scales. The features from the two regions are then fused, and the fused features are combined with the overall image features, improving the network's multi-scale expressive capability.

[0057] In step S3, the network model for object detection can be any one of Faster R-CNN, YOLO, or SSD. In this example, YOLO is preferred. The final feature map is input into the Neck subnetwork, detection head, and NMS (non-maximum suppression) operation in the YOLO network to obtain the final detection result.

[0058] In one example, such as Figure 2 As shown, image masking involves first converting the image to grayscale, then blurring it to reduce noise. Next, an edge detection algorithm is used to detect edges, generating a binary image where white pixels represent edges and black pixels represent the background. The escalator area in the image is trapezoidal. Hough transform is used to detect straight lines, and filtering conditions are defined to select two straight lines corresponding to the handrail positions. These lines are then intersected with the initial escalator area to generate a valid mask region. Finally, invalid areas are removed from the original image to remove irrelevant regions. Specifically, this includes:

[0059] S11: Select the initial escalator area; optionally, set a trapezoidal region in the image as the initial escalator area, with the coordinates of the four points of the trapezoid arranged clockwise as follows: The initial escalator area selection is no longer required throughout the subsequent detection process; it is only performed once at the initial stage, reducing manual intervention. Optionally, for each camera, the escalator area is initially selected. However, since the cameras are located inside the subway station, external factors such as subway operation may cause the cameras to shift. Therefore, the initially manually selected area needs to be adjusted periodically using edge detection.

[0060] S12: Convert the original RGB image (width w, height h) to a grayscale image, and perform smoothing, gradient calculation, non-maximum suppression, edge tracking, and thinning to obtain edge information with obvious intensity changes. Smoothing includes, but is not limited to, interpolation, linear smoothing, and convolution methods to suppress noise, smooth image brightness, reduce abrupt gradients, and improve image quality. Gradient calculation calculates the rate of image change through differentiation. Non-maximum suppression is used to remove detection results that overlap significantly with maxima. Edge tracking connects discrete pixels into continuous edges by tracing the pixel paths of the boundaries. Thinning reduces the line width of the image from multiple pixels to a single pixel width.

[0061] S13: Perform Hough transform on the edge information to generate a line cluster; set the Hough transform parameters θ = π / 180, ρ = 1, set the minimum length of the detected line to h / 3, and the minimum distance between two points on the same line to h / 20, and obtain the line cluster (l) through Hough transform. 1 ,l 2 ,l 3 ,…,l n ), where l i =((x1) i ,y1 i ),(x2 i ,y2 i )).

[0062] S14: Fit the escalator straight lines to obtain the left and right handrail lines; optionally, before fitting the escalator straight lines, interference lines are also removed. Specifically, the slope of the inclination angle of the left and right handrails is calculated based on the initial escalator area, and interference lines are removed based on the slope of the inclination angle and the mean value of the line cluster. Combining the removal of interference lines, the calculation method for the left and right handrail lines is as follows:

[0063] Calculate the inclination angles and slopes of the left and right handrails of the escalator based on the initially selected area. Let the slope of the left handrail be... For the line cluster obtained in S13, based on whether the slope k>0 or k<0, the line cluster is divided into the left handrail preselection cluster (l 1 ,l 2 ,l 3 ,…,l L (1) For each straight line in the left handrail cluster, calculate the slope k. (1) Keep the straight line, otherwise discard it; (2) Calculate the mean. For each straight line l i Calculate diff = x1 i -mean, if diff is greater than the set threshold, it is discarded; (3) For the remaining straight lines, least squares fitting is performed to obtain the final left handrail line ll =((x1) l ,y1 l ),(x2 l ,y2 l (4) Perform the same processing on the pre-selected cluster of the right handrail to obtain the right handrail line l. r =((x1) r ,y1 r ),(x2 r ,y2 r )).

[0064] S15: Perform Boolean operations on the initial escalator area with the left and right handrail lines to obtain the Mask area. Specifically, calculate the intersection points of the left and right handrail lines with the image edges. The polygonal area formed by these intersection points intersects with the artificial area in S1. The resulting intersection area is denoted as (x1, y1), (x2, y2), (x3, y3), (x4, y4), ..., (xm, ym). Create an array with the same dimensions as the original image and initialize all variables to zero. Set the pixel values ​​of the intersection area to (255, 255, 255), finally obtaining the Mask area.

[0065] S16: Perform a bitwise AND operation between the Mask region and the pixel values ​​of the original image to remove irrelevant regions and obtain the image of the region of interest.

[0066] In one example, the process of masking an image also includes:

[0067] S01: Perform edge detection and Hough transform on the original image to obtain the edge information map and the orientation gradient histogram, respectively.

[0068] S02: The edge information map, histogram of oriented gradients, and the original RGB image are combined to form a five-channel image, which is then masked. The original RGB image is combined with the edge-extracted image and the Hough transform image to provide multi-source information and supplement the feature information of the occluded target. This is then fed into the feature extraction network of the deep learning model. The edge-extracted image and the Hough transform image are used to capture the edge and texture information of the image. Multi-channel input can improve the model's robustness to noise and interference, thereby improving the model's detection performance.

[0069] In one example, when the region of interest image is divided into three regions (top, middle, and bottom), feature extraction includes:

[0070] Different scale convolutions are used to extract features from the upper, middle, and lower regions. The convolution kernel for the lower region feature extraction is greater than that for the middle region feature extraction, which is greater than that for the upper region feature extraction. Then, the features from the upper, middle, and lower regions are fused to obtain the final feature map. Preferably, the feature fusion is weighted fusion to extract more pedestrian target features.

[0071] In one example, to address the issue that a target might simultaneously reside in two regions, the middle region contains parts of both the upper and lower regions. Feature extraction is then performed on these three regions using convolutions of the same type but different scales. Finally, the features from these three regions are fused, and the fused features are then combined with the overall features, improving the network's multi-scale expressive ability and achieving a balance in information extraction for targets of different sizes. In this example, the top half of the region of interest is region S1 (upper region), the middle one-third to two-thirds is region S2 (middle region), and the bottom half is region S3 (lower region). Region S2 simultaneously contains portions of regions S1 and S3. Convolution processing is performed on these three regions to obtain feature information. Region S1 has the smallest target and uses the smallest convolution kernel, while region S3 has the largest target and uses the largest convolution kernel. Then, the result obtained from the S1 convolution is weighted and the result obtained from the S2 convolution to obtain the fused feature map M1. At the same time, the result obtained from the S2 convolution is weighted and the result obtained from the S3 convolution to obtain the fused feature map M2. Finally, the feature maps M1 and M2 are weighted and fused to obtain the final feature map.

[0072] Combining the above examples, we obtain a preferred example of the present invention, in which the detection method includes the following steps:

[0073] S0': Input an RGB image, perform edge detection and Hough transform on the original image to obtain an edge information map and an orientation gradient histogram, respectively; combine the edge information map, orientation gradient histogram and the original RGB image to form a five-channel image;

[0074] S1': Mask the five-channel image to obtain an image of the region of interest that retains only the escalator area;

[0075] S2': Divide the image of interest into three regions: upper, middle, and lower. The middle region contains part of the upper region and part of the lower region. Use convolution of different scales to extract features from the upper, middle, and lower regions. The convolution kernel for the lower region feature extraction is greater than that for the middle region feature extraction, which is greater than that for the upper region feature extraction. The features obtained from the upper and middle regions are fused to obtain feature map M1. The features obtained from the lower and middle regions are fused to obtain feature map M2. Finally, feature map M1 and feature map M2 are fused to obtain the final feature map.

[0076] S3': Perform object detection processing on the final feature map to obtain the final object detection result.

[0077] This invention also includes a pedestrian detection system for subway escalators based on mask and regional feature fusion, which shares the same inventive concept as the aforementioned detection method. This system includes a preprocessing unit, a labeling unit, a feature extraction unit, and a detection unit. The preprocessing unit performs mask processing on the image to obtain an image of the region of interest, retaining only the escalator area. The labeling unit divides the region of interest image into two regions (upper and lower), three regions (upper, middle, and lower), or several regions. The feature extraction unit uses convolutions of different scales to extract features from the upper and lower regions, with the convolution kernel for the lower region feature extraction being larger than that for the upper region feature extraction. The features from the upper and lower regions are then fused to obtain the final feature map. The detection unit performs detection processing on the final feature map to obtain the final target detection result. The feature extraction unit and the detection unit can be replaced with neural networks that have feature extraction and detection functions, such as Faster R-CNN, YOLO, and SSD. In this example, YOLO is preferred.

[0078] In one example, the feature extraction unit is the backbone network (BACKONE) in the YOLO network, and the detection unit is the neck network (Neck) and the detection head in the YOLO network;

[0079] When the image of interest is divided into three regions (upper, middle, and lower), the backbone network uses convolutions of different scales to extract features from each region. The convolution kernel for the lower region is greater than that for the middle region, which is greater than that for the upper region. The features from the upper, middle, and lower regions are then fused to obtain the final feature map. Preferably, the middle region includes part of the upper region and part of the lower region. In this case, after the backbone network extracts features from each region using convolutions of different scales:

[0080] The backbone network fuses the features obtained from the upper and middle regions to obtain feature map M1, and fuses the features obtained from the lower and middle regions to obtain feature map M2. Then, feature map M1 and feature map M2 are fused to obtain the final feature map.

[0081] like Figure 3 As shown, the feature extraction network of this invention includes a first convolutional layer, a second convolutional layer, a first C3 module, a third convolutional layer, a second C3 module, a fourth convolutional layer, a third C3 module, a fifth convolutional layer, a fourth C3 module, and an SPPF module connected in sequence; the detection network includes a neck sub-network and a detection head; the neck sub-network includes a sixth convolutional layer, a first upsampling layer, a first concatenation layer, a fifth C3 module, a seventh convolutional layer, a second upsampling layer, a second concatenation layer, a sixth C3 module, an eighth convolutional layer, a third concatenation layer, a seventh C3 module, a ninth convolutional layer, a fourth concatenation layer, and an eighth C3 module connected in sequence; wherein, the sixth convolutional layer is connected to the fourth concatenation layer, the seventh convolutional layer is connected to the third concatenation layer, the second concatenation layer is connected to the sixth C3 module, the sixth C3 module is connected to a detection layer Detect0 via a convolutional layer, the seventh C3 module is connected to a detection layer Detect1 via a convolutional layer, and the eighth C3 module is connected to a detection layer Detect2 via a convolutional layer. Furthermore, the C3 module and SPPF module are based on the existing YOLO network structure, and will not be described in detail here.

[0082] Furthermore, the interconnection between feature extraction and the neck sub-network is explained. Specifically, the second C3 module is directly connected to the second concatenation layer in one path, and connected to the second concatenation layer via an SFE module (regional feature extraction module) in another path; the third C3 module is directly connected to the first concatenation layer in one path, and connected to the first concatenation layer via an SFE module in another path; the SPPF module is directly connected to the sixth convolutional layer in one path, and connected to the sixth convolutional layer via an SFE module in another path; the SFE module includes at least two convolutional layers, and feature fusion processing is performed between adjacent convolutional layers. When the number of convolutional layers is greater than two, after feature fusion processing between adjacent convolutional layers, the adjacent feature fusion results are subjected to feature fusion processing again until a unique final feature map is obtained. Figure 4 As shown, the SFE module includes three independent convolutional layers. Feature fusion is performed between adjacent convolutional layers to obtain fused feature maps M1 and M2. Finally, feature maps M1 and M2 are fused to obtain the final feature map. The feature fusion described in this invention is preferably weighted feature fusion.

[0083] This invention innovatively proposes an SFE module structure and integrates the SFE module into existing detection networks. It also proposes a regional feature extraction and fusion approach, using different convolutional kernels to extract features from targets in different regions, thereby balancing the feature information of targets of different sizes and improving the detection performance of large / small targets.

[0084] This embodiment provides a storage medium that has the same inventive concept as the subway escalator pedestrian detection method based on the fusion of Mask and regional features formed by any or more of the above examples. The medium stores computer instructions, which, when executed, perform the steps of the subway escalator pedestrian detection method based on the fusion of Mask and regional features formed by any or more of the above examples.

[0085] Based on this understanding, the technical solution of this embodiment, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] This application also includes a terminal having the same inventive concept as any or a combination of examples corresponding to the above-described method for pedestrian detection on subway escalators based on the fusion of mask and regional features. The terminal includes a memory and a processor, wherein the memory stores computer instructions executable on the processor. When the processor executes the computer instructions, it performs the steps of the above-described method for pedestrian detection on subway escalators based on the fusion of mask and regional features. The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement this invention.

[0087] In one example, the terminal, i.e., the electronic device, is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit (processor) mentioned above, at least one storage unit mentioned above, and a bus connecting different system components (including storage units and processing units).

[0088] The storage unit stores program code that can be executed by the processing unit, causing the processing unit to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit can execute the above-described method for pedestrian detection on subway escalators based on the fusion of mask and regional features.

[0089] The storage unit may include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 3201 and / or a cache storage unit, and may further include a read-only memory (ROM).

[0090] The storage unit may also include a program / utility having a set (at least one) of program modules, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0091] A bus can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus that uses any of the various bus structures.

[0092] The electronic device can also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication can be performed via input / output (I / O) interfaces. Furthermore, the electronic device can communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter communicates with other modules of the electronic device via a bus. It should be understood that other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0093] Through the above description, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to this exemplary embodiment can be embodied in the form of a software product, which can be stored on a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method of the exemplary embodiment of this application.

[0094] The above detailed embodiments are a description of the present invention. It should not be considered that the specific embodiments of the present invention are limited to these descriptions. For those skilled in the art, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the protection scope of the present invention.

Claims

1. A pedestrian detection method on subway escalators based on the fusion of mask and regional features, characterized in that: Includes the following steps: Mask the image to obtain an image of the region of interest that retains only the escalator area; The image of the region of interest is divided into upper and lower regions. Convolution at different scales is used to extract features from the upper and lower regions. The convolution kernel for the lower region feature extraction is larger than that for the upper region feature extraction. The features from the upper and lower regions are then fused to obtain the final feature map. The final feature map is then processed for object detection to obtain the final object detection result. The image masking process includes: Select the initial escalator area; The original image is converted to a grayscale image, and then smoothed, gradient calculated, non-maximum suppressed, edge tracked, and thinned to obtain edge information. The edge information is subjected to Hough transform to generate a family of straight lines; The straight lines of the escalator are fitted to obtain the left handrail line and the right handrail line; Calculate the intersection points of the left and right handrail lines with the image edge to obtain the polygonal region formed by the intersection points. Calculate the intersection of the polygonal region with the initial escalator region to obtain the intersection region. Determine the Mask region based on the intersection region. Perform a bitwise AND operation between the Mask region and the pixel values ​​of the original image to remove irrelevant regions and obtain the image of the region of interest. The process of fitting the escalator straight lines to obtain the left and right handrail lines includes the following steps: Calculate the inclination angle slope of the left and right handrails of the escalator based on the initial escalator area, and remove interfering straight lines based on the inclination angle slope and the mean value of the straight line cluster. The remaining straight lines are fitted using least squares to obtain the left and right handrail lines.

2. The method for pedestrian detection on subway escalators based on the fusion of Mask and regional features as described in claim 1, characterized in that: The image to be masked is a five-channel image. The steps for generating a five-channel image include: Edge detection and Hough transform are performed on the original image to obtain an edge information map and an orientation gradient histogram. The edge information map, the directional gradient histogram, and the original RGB image are combined to form a five-channel image.

3. The method for pedestrian detection on subway escalators based on the fusion of Mask and regional features as described in claim 1, characterized in that: The division of the image of interest into two regions (upper and lower) is replaced by dividing the image of interest into three regions (upper, middle, and lower). Feature extraction includes: Different scale convolutions are used to extract features from the upper, middle, and lower regions. The convolution kernel for the lower region feature extraction is greater than that for the middle region feature extraction, which is greater than that for the upper region feature extraction. The features from the upper, middle, and lower regions are then fused to obtain the final feature map.

4. The method for pedestrian detection on subway escalators based on the fusion of mask and regional features according to claim 3, characterized in that: The intermediate region comprises a portion of the upper region and a portion of the lower region; at this point, the features are fused as follows: The features obtained from the upper and middle regions are fused to obtain feature map M1, and the features obtained from the lower and middle regions are fused to obtain feature map M2. Then, feature map M1 and feature map M2 are fused to obtain the final feature map.

5. A subway escalator pedestrian detection system based on Mask and regional feature fusion, used to implement the subway escalator pedestrian detection method based on Mask and regional feature fusion as described in any one of claims 1-4, characterized in that: It includes: The preprocessing unit is used to mask the image to obtain an image of the region of interest that retains only the escalator area; A labeling unit is used to divide the image of the region of interest into two regions (upper and lower) or three regions (upper, middle, and lower). The feature extraction unit is used to extract features from the upper and lower regions using convolutions of different scales. The convolution kernel for feature extraction in the lower region is larger than that for feature extraction in the upper region. The features from the upper and lower regions are then fused to obtain the final feature map. The detection unit is used to perform detection processing on the final feature map to obtain the final target detection result.

6. The subway escalator pedestrian detection system based on Mask and regional feature fusion as described in claim 5, characterized in that: The feature extraction unit is the backbone network in the YOLO network, and the detection unit is the neck network and detection head in the YOLO network. When the image of interest is divided into three regions, upper, middle, and lower, the backbone network uses convolutions of different scales to extract features from the upper, middle, and lower regions. The convolution kernel for extracting features from the lower region is greater than that for the middle region, which is greater than that for the upper region. The features from the upper, middle, and lower regions are then fused to obtain the final feature map.

7. The subway escalator pedestrian detection system based on Mask and regional feature fusion as described in claim 5, characterized in that: The intermediate region includes a portion of the upper region and a portion of the lower region. The backbone network then uses convolutions of different scales to extract regional features from the upper, middle, and lower regions: The backbone network fuses the features obtained from the upper and middle regions to obtain feature map M1, and fuses the features obtained from the lower and middle regions to obtain feature map M2. Then, feature map M1 and feature map M2 are fused to obtain the final feature map.

8. A terminal comprising a memory and a processor, wherein the memory stores computer instructions executable on the processor, characterized in that: When the processor executes the computer instructions, it performs the steps of the pedestrian detection method on subway escalators based on the fusion of Mask and regional features as described in any one of claims 1-4.

Images