A real-time track multi-target segmentation method and system based on deep learning

Abstract

The present invention discloses a real-time track multi-target segmentation method and system based on deep learning, comprising the following steps: S1: acquiring a track video data stream and performing dual-branch spatiotemporal feature extraction and fusion to obtain a fused image; S2: enhancing the fused image based on an adaptive attention mechanism to obtain an enhanced image; S3: employing a multi-scale convolutional layer to perform feature analysis and reorganization on the enhanced feature map, constructing a multi-scale feature representation, and obtaining a multi-scale feature map; S4: designing a dynamic target classification branch network based on the multi-scale feature map to recognize targets of different categories and obtain a feature map with category information; S5: utilizing the target feature map with category information to complete the real-time segmentation of multiple targets in the track scene. The present invention can achieve real-time and accurate segmentation of multiple targets in a marathon event scene, providing effective technical support for event video analysis and processing.

Description

Technical Field

[0001] The present invention relates to the technical field of real-time segmentation of multiple targets on a racetrack, and in particular to a method and system for real-time segmentation of multiple targets on a racetrack based on deep learning. Background Art

[0002] With the rapid development of sports live broadcasting and analysis technologies, the requirements for object segmentation in large-scale events such as marathons are becoming increasingly stringent. Accurately identifying and segmenting athletes and staff on the track is crucial for applications such as live broadcasting, security monitoring, and trajectory analysis. While computer vision and deep learning technologies have made significant progress in object segmentation, they still face numerous challenges in addressing the unique challenges of marathon scenarios.

[0003] Existing technologies mainly use semantic segmentation methods based on deep learning. For example, patent CN118941796A discloses a deep asymmetric bottleneck real-time semantic segmentation method that integrates multi-scale information. This method uses a self-correcting DAB module and a CCSP-ASPP module to achieve segmentation of targets of different scales. Although this method has achieved certain results in static image segmentation, it has the following main shortcomings: First, the method only considers the spatial features of a single frame image and ignores the rich temporal information in the video sequence, resulting in limited performance when processing moving targets; second, the feature extraction method used is relatively simple and fails to fully consider the target motion characteristics and scene complexity, making it difficult to cope with complex situations such as dense crowds and overlapping occlusions in marathon events.

[0004] In addition, other existing segmentation methods also have some common problems: first, insufficient feature expression capabilities make it difficult to accurately distinguish different categories of targets with similar visual features, such as athletes and staff; second, they lack effective attention mechanisms, making it impossible to adaptively highlight important features and suppress background interference; third, they fail to design specialized target classification strategies based on the characteristics of the event scene, resulting in suboptimal classification accuracy. These problems have severely restricted the practical application of segmentation technology in marathon events.

[0005] Therefore, there is an urgent need to develop an object segmentation method that can effectively utilize temporal information, possesses powerful feature expression capabilities, and adapts to the characteristics of the event scene. This paper addresses these technical issues and proposes an object segmentation method based on dual-branch spatiotemporal feature extraction and an adaptive attention mechanism to achieve real-time and accurate segmentation of multiple objects in marathon event scenes. Summary of the Invention

[0006] In view of this, the present invention provides a real-time track multi-target segmentation method based on deep learning, aiming to achieve real-time and accurate segmentation of multiple targets in marathon event scenes, and provide effective technical support for event video analysis and processing.

[0007] To achieve the above objectives, the present invention provides a method for real-time segmentation of multiple targets on a track based on deep learning, comprising the following steps:

[0008] S1: Obtain the track video data stream and perform dual-branch spatiotemporal feature extraction and fusion to obtain the fused image;

[0009] S2: Enhance the fused image based on the adaptive attention mechanism to obtain an enhanced image;

[0010] S3: Use multi-scale convolutional layers to perform feature analysis and reorganization on the enhanced feature map, construct multi-scale feature representation, and obtain multi-scale feature maps;

[0011] S4: Based on the multi-scale feature map, a dynamic target classification branch network is designed to realize the recognition of targets of different categories and obtain feature maps with category information;

[0012] S5: Use target feature maps with category information to complete real-time segmentation of multiple targets in the track scene.

[0013] As a further improvement method of the present invention:

[0014] Optionally, in step S1, obtaining the track video data stream and performing dual-branch spatiotemporal feature extraction and fusion to obtain a fused image includes:

[0015] S11: Obtain single-frame images and optical flow image sequences from the track video data stream, specifically:

[0016] Get a single frame image I in the track video data stream at time t t , and simultaneously obtain the optical flow image sequence O of the preceding and following frames including the frame O={O t-r ,…,O t ,…,O t+r},in:

[0017]

[0018] O t =Φ(I t ,I t+1 ),t∈[tr,t+r];

[0019] Where t is the time index of the current frame; is the rounding down operation; T is the timing window size, which is an odd number; r is the timing radius; I t and I t+1 are the images at time t and t+1 respectively; O t is the optical flow image at time t; Φ is the optical flow estimation function;

[0020] S12: Input single frame image I t Perform spatial feature extraction, specifically:

[0021]

[0022] Among them, F spt (x,y) is the spatial feature map F spt The eigenvalue at the pixel coordinate (x, y); W spt (P,q) is the spatial convolution kernel weight matrix W spt The value at the row and column number (p,q) of the weight matrix; I t (x+p,y+q) is a single frame image I at time t t The pixel value at pixel coordinates (x+p,y+q);

[0023] S13: Extract temporal features from the optical flow image sequence of consecutive T frames, specifically:

[0024]

[0025] Among them, F tmp (x,y) is the time series feature graph F tmp The eigenvalue at the pixel coordinate (x, y); W tmp (m,n) is the temporal convolution kernel weight matrix W tmp The value at the row and column number (m, n) of the weight matrix; O k’ (x+m, y+n) is the value of the k'th frame optical flow image at the pixel coordinate (x+m, y+n); γ is the normalization function; D k’ (x, y) is the motion displacement vector of the k'th frame at pixel coordinates (x, y);

[0026] S14: Fusing spatial features and temporal features to obtain a fused image H, specifically:

[0027] H(x,y)=F spt (x,y)⊙σ(F tmp (x,y)+F tmp (x,y)⊙σ(F spt (x,y)));

[0028] Where H(x,y) is the eigenvalue of the fused image H at the pixel coordinate (x,y); σ is the sigmoid function; ⊙ is the element-by-element multiplication.

[0029] Optionally, in step S2, enhancing the fused image based on an adaptive attention mechanism to obtain an enhanced image includes:

[0030] S21: Construct a channel attention module to calculate the importance weight of each channel of the fused image, specifically:

[0031]

[0032] Among them, H c (x, y) is the pixel value of the c-th channel of the fused image H at the pixel coordinate (x, y); Z c is the global average pooling value of the c-th channel; and are the width and length of the fused image H respectively; ω c is the attention weight of the cth channel; N is the number of channels; e is a natural constant; Z d is the global average pooling value of the d-th channel, d is the dimension index, and its value range is a positive integer from 1 to N;

[0033] S22: Construct a spatial attention module to calculate the attention map of each spatial position of the fused image H, specifically:

[0034]

[0035] Where Q(x,y) is the gradient response intensity at the pixel coordinate (x,y); is the spatial gradient of the cth channel of the fused image H at the pixel coordinate (x, y); R(x, y) is the spatial attention value at the pixel coordinate (x, y);

[0036] S23: Fuse channel attention and spatial attention to obtain the enhanced image E, specifically:

[0037]

[0038] Among them, E c (x, y) is the pixel value of the c-th channel of the enhanced image E at the pixel coordinate (x, y); σ′ is the ReLU function; The square root of N is obtained by dividing Standardization can avoid numerical instability problems.

[0039] Optionally, in step S3, a multi-scale convolutional layer is used to perform feature analysis and reorganization on the enhanced image, construct a multi-scale feature representation, and obtain a multi-scale feature map, including:

[0040] S31: Generate decomposition diagrams of different scales, specifically:

[0041] P k =Conv k (E),k∈{1,2,3};

[0042]

[0043] Among them, P k is a decomposition diagram with scale k, P k (x,y) is the decomposition graph P with scale k k The pixel value at the pixel coordinate (x, y); Conv k is a convolution operation of scale k; and are the width and length of the decomposition graph with scale k; α k is the global representation of the decomposition graph of scale k;

[0044] S32: Generate a multi-scale feature map, specifically:

[0045]

[0046] in, is a feature map of scale k; Conv re Reconstruct the convolution operation; Res k (E) is the feature map obtained by performing a residual connection operation on the enhanced image E when the scale is k.

[0047] Optionally, in step S4, a dynamic target classification branch network is designed based on the multi-scale feature map to realize the recognition of targets of different categories and obtain a feature map with category information, including:

[0048] S41: Construct a category-aware attention module to extract category-related features, specifically:

[0049]

[0050]

[0051]

[0052] Among them, μ k is a feature map of scale k Category-aware convolution cls The characteristics of the latter; For the Category feature representation of class targets, through category-specific convolution Extraction; Pool is a global pooling operation; For the Attention weight of class target; is the number of target categories; and For category features and The feature vector obtained by the fully connected layer transformation can map any range of input to the range of 0 to 1 and maintain the relative importance of different categories through a combination of exponential function and normalization.

[0053] S42: Output the target feature map with category information, specifically:

[0054]

[0055] in, For the The final feature representation of the class target at the pixel coordinate (x, y); Conv map is the feature map convolution; Ω(x,y) is the feature set containing all category information at the pixel coordinate (x,y).

[0056] Optionally, in step S5, the target feature map with category information is used to complete the real-time segmentation of multiple targets in the track scene, including:

[0057] S51: Construct a feature enhancement module to fuse category information and location information, specifically:

[0058] Λ(x,y)=Conv enhance (Ω(x,y))+PE(x,y);

[0059] Among them, Λ(x,y) is the fusion feature at the pixel coordinate (x,y); Conv enhance is enhanced convolution; PE(x,y) is the position encoding of the pixel coordinate (x,y);

[0060] S52: Generate mask prediction and obtain segmentation results, specifically:

[0061]

[0062] where ξ(x,y) is the instance mask prediction at pixel coordinate (x,y), To predict the pixel at the pixel coordinate (x, y) belongs to The probability of the class target; Conv mask Convolution for mask prediction;

[0063] The segmentation results are:

[0064]

[0065] Where τ(x,y) is the category label at the pixel coordinate (x,y).

[0066] The present invention also discloses a track multi-object real-time segmentation system based on deep learning, comprising:

[0067] Fusion module: obtains the track video data stream and performs dual-branch spatiotemporal feature extraction and fusion;

[0068] Enhancement module: enhances the fused image based on the adaptive attention mechanism;

[0069] Multi-scale module: uses multi-scale convolutional layers to perform feature analysis and reorganization on the enhanced feature maps to construct multi-scale feature representations;

[0070] Target feature extraction module: Based on multi-scale feature maps, a dynamic target classification branch network is designed to realize the recognition of targets of different categories;

[0071] Segmentation module: Uses target feature maps with category information to complete real-time segmentation of multiple targets in the track scene.

[0072] Compared with the prior art, the present invention has at least the following beneficial effects:

[0073] This paper proposes an object segmentation method based on dual-branch spatiotemporal feature extraction and an adaptive attention mechanism. By designing a dual-branch architecture to extract spatial and temporal features separately, and employing an innovative feature fusion strategy, this method enables accurate modeling of dynamic objects. This method fully utilizes temporal information when processing video data.

[0074] This paper innovatively designs a multi-level attention mechanism, including a channel attention module and a spatial attention module, and proposes an adaptive feature enhancement strategy. This multi-level attention mechanism can adaptively focus on important features, effectively suppress background interference, and improve the discriminability of feature expression.

[0075] This paper utilizes an innovative architecture combining multi-scale feature representation and a dynamic object classification branch network. By parsing and reorganizing features through multi-scale convolutional layers, it effectively addresses the detection of objects of varying scales. This method accurately captures the multi-scale features of an object and extracts category-related features through a category-aware attention module, ultimately achieving precise object classification and segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a flowchart of a method for real-time segmentation of multiple targets on a track based on deep learning according to an embodiment of the present invention;

[0077] Figure 2 This is a diagram of the multi-target segmentation result according to an embodiment of the present invention, (a) the image to be segmented; (b) a schematic diagram of the segmentation result;

[0078] Figure 3 This is a deep learning network training diagram according to an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention fall within the scope of protection of the present invention.

[0080] Example 1: A real-time segmentation method for multiple targets on a track based on deep learning, such as Figure 1 As shown, the following steps are included:

[0081] S1: Obtain the track video data stream and perform dual-branch spatiotemporal feature extraction and fusion to obtain the fused image:

[0082] S11: Obtain single-frame images and optical flow image sequences from the track video data stream, specifically:

[0083] Get a single frame image I in the track video data stream at time t t , and simultaneously obtain the optical flow image sequence O of the preceding and following frames including the frame O={O t-r ,…,O t ,…,O t+r},in:

[0084]

[0085] O t =Φ(I t ,I t+1 ),t∈[tr,t+r];

[0086] Where t is the time index of the current frame; is a rounding down operation; T is the timing window size, which is an odd number, 7 in this embodiment; r is the timing radius; I t and I t+1 are the images at time t and t+1 respectively; O t is the optical flow image at time t; Φ is the optical flow estimation function;

[0087] S12: Input single frame image I t Perform spatial feature extraction, specifically:

[0088]

[0089] Among them, F spt (x,y) is the spatial feature map F spt The eigenvalue at the pixel coordinate (x, y); W spt (p,q) is the spatial convolution kernel weight matrix W spt The value at the row and column number (p,q) of the weight matrix; I t (x+p,y+q) is a single frame image I at time t tThe pixel value at the pixel coordinate (x+p, y+q); the spatial convolution kernel weight matrix W spt Specifically:

[0090]

[0091] Among them, e is a natural constant; λ spt is the spatial scale parameter, which is 0.9 in this embodiment; η spt is the directional enhancement coefficient, which is 0.5 in this embodiment; θ spt (p,q) is the local direction angle at the (p,q) position, specifically:

[0092]

[0093] Among them, G x (x+p,y+q) and G y (x+p, y+q) are the single frame images I at time t t The horizontal and vertical gradients at the pixel coordinate (x+p,y+q);

[0094] S13: Extract temporal features from the optical flow image sequence of consecutive T frames, specifically:

[0095]

[0096] Among them, F tmp (x,y) is the time series feature graph F tmp The eigenvalue at the pixel coordinate (x, y); W tmp (m,n) is the temporal convolution kernel weight matrix W tmp The value at the row and column number (m, n) of the weight matrix; O k’ (x+m, y+n) is the value of the k'th frame optical flow image at the pixel coordinate (x+m, y+n); γ is the normalization function; D k’ (x, y) is the motion displacement vector of the k'th frame at the pixel coordinate (x, y); the temporal convolution kernel weight matrix W tmp Specifically:

[0097]

[0098] Among them, λ tmp is the time series scale parameter, which is 1.5 in this embodiment; η tmp is the motion enhancement coefficient, which is 0.3 in this embodiment; M k′ (m,n) is the motion amplitude of the k′th frame at the (m,n) position, specifically:

[0099]

[0100] in, and are the gradients of the k′th frame optical flow image in the m and n directions respectively;

[0101] S14: Fusing spatial features and temporal features to obtain a fused image H, specifically:

[0102] H(x,y)=F spt (x,y)⊙σ(F tmp (x,y)+F tmp (x,y)⊙σ(F spt (x,y)));

[0103] Where H(x,y) is the eigenvalue of the fused image H at the pixel coordinate (x,y); σ is the sigmoid function; ⊙ is the element-by-element multiplication.

[0104] This step effectively addresses the critical challenges of object detection and segmentation in racetrack scenarios by designing a dual-branch spatiotemporal feature extraction and fusion architecture. In the spatial feature extraction branch, a directionally aware adaptive convolution kernel is used to extract features, highlighting key edges and contours in racetrack scenes. In the temporal feature extraction branch, motion information from multi-frame optical flow sequences is modeled to effectively capture the temporal dynamic characteristics of moving objects in the scene.

[0105] S2: Enhance the fused image based on the adaptive attention mechanism to obtain the enhanced image:

[0106] S21: Construct a channel attention module to calculate the importance weight of each channel of the fused image, specifically:

[0107]

[0108]

[0109] Among them, H c (x, y) is the pixel value of the c-th channel of the fused image H at the pixel coordinate (x, y); Z c is the global average pooling value of the c-th channel; and are the width and length of the fused image H respectively; ω c is the attention weight of the cth channel; N is the number of channels, which is 3 in this embodiment; Z d is the global average pooling value of the d-th channel, d is the dimension index, and its value range is a positive integer from 1 to N;

[0110] S22: Construct a spatial attention module to calculate the attention map of each spatial position of the fused image H, specifically:

[0111]

[0112] Where Q(x,y) is the gradient response intensity at the pixel coordinate x(,y); is the spatial gradient of the cth channel of the fused image H at the pixel coordinate (x, y), which is calculated using the Sobel operator in this embodiment; R(x, y) is the spatial attention value at the pixel coordinate (x, y);

[0113] S23: Fuse channel attention and spatial attention to obtain the enhanced image E, specifically:

[0114]

[0115] Among them, E c (x, y) is the pixel value of the c-th channel of the enhanced image E at the pixel coordinate (x, y); σ′ is the ReLU function; The square root of N is obtained by dividing Standardization can avoid numerical instability problems.

[0116] This step adaptively enhances features by designing a dual attention mechanism, effectively improving their expressiveness and discriminability. In the channel dimension, global average pooling is used to calculate the importance weight of each channel, enabling the system to automatically identify and enhance feature channels that are more important for object detection and segmentation tasks, while suppressing interference from irrelevant channels. In the spatial dimension, a spatial attention map is constructed by calculating the gradient response strength, enabling the system to highlight feature expressions in key areas.

[0117] S3: Use multi-scale convolutional layers to perform feature analysis and reorganization on the enhanced feature map, construct multi-scale feature representation, and obtain multi-scale feature maps:

[0118] S31: Generate decomposition diagrams of different scales, specifically:

[0119] P k =Conv k (E),k∈{1,2,3};

[0120]

[0121] Among them, P k is a decomposition diagram with scale k, P k (x,y) is the decomposition graph P with scale k k The pixel value at the pixel coordinate (x, y); Conv kFor the convolution operation with scale k, when k=1, a 3×3 convolution kernel is used and 128 channels are output. When k=2, a 5×5 convolution kernel is used and 128 channels are output. When k=3, a 7×7 convolution kernel is used and 128 channels are output. All Conv k All operations use a step size of 1; and are the width and length of the decomposition graph with scale k; α k is the global representation of the decomposition graph of scale k;

[0122] S32: Generate a multi-scale feature map, specifically:

[0123]

[0124] in, is a feature map of scale k; Conv re To reconstruct the convolution operation, in this embodiment, Conv re It is a 1×1 convolution operation, the number of output channels is 3, and the stride is 1; Res k is the residual connection operation when the scale is k. In this embodiment, Res k It consists of two 1×1 convolutional layers connected in series. The number of output channels of the first 1×1 convolutional layer is 64, and the number of output channels of the second 1×1 convolutional layer is 3. A ReLU activation function is added between the two convolutional layers, and the step size is 1. k (E) is the feature map obtained by performing a residual connection operation on the enhanced image E when the scale is k.

[0125] This step achieves multi-level feature extraction and effective fusion of feature maps by designing a multi-scale convolutional layer network structure. By using convolution kernels of different sizes for feature decomposition, the system is able to simultaneously capture local detail features and large-scale contextual information in the marathon track scene. Small-sized convolution kernels focus on extracting the fine texture and edge features of the target, medium-sized convolution kernels can capture the local structural information of the target, and large-sized convolution kernels help to understand the overall shape of the target and the scene context.

[0126] S4: Based on the multi-scale feature map, a dynamic target classification branch network is designed to realize the recognition of targets of different categories and obtain feature maps with category information:

[0127] S41: Construct a category-aware attention module to extract category-related features, specifically:

[0128]

[0129] Among them, μ k is a feature map of scale k Category-aware convolutioncls After the feature, in this embodiment, Conv cls It is a 3×3 convolution, the number of output channels is 128, and the stride is 1; For the Category feature representation of class targets, through category-specific convolution Extraction, in this embodiment It is a 1×1 convolution, the number of output channels is 64, and the stride is 1; Pool is a global pooling operation; For the Attention weight of class target; is the number of target categories, which is 3 in this embodiment, corresponding to background, athletes, and staff respectively; FC is the fully connected layer, which has an input dimension of 64 and an output dimension of 1 in this embodiment; and For category features and The feature vector obtained by the fully connected layer transformation can map any range of input to the range of 0 to 1 and maintain the relative importance of different categories through a combination of exponential function and normalization.

[0130] S42: Output the target feature map with category information, specifically:

[0131]

[0132] in, For the The final feature representation of the class target at the pixel coordinate (x, y); Conv map Convolution is the feature map. In this embodiment, Conv map It is a feature map convolution with a 1×1 convolution structure, 32 output channels, and a step size of 1. Ω(x,y) is the feature set containing all category information at the pixel coordinate (x,y).

[0133] This step achieves accurate recognition of multiple dynamic targets in marathon scenarios by designing a dynamic target classification branch network. The introduced category-aware attention module adaptively extracts and enhances feature representations related to specific target categories, effectively distinguishing between different target categories, such as athletes, staff, and spectators. By integrating multi-scale features and performing category-specific convolution operations, this step simultaneously processes feature representations of targets at different scales, capturing the overall characteristics of large-scale targets while preserving the detailed information of small-scale targets.

[0134] S5: Using target feature maps with category information to complete real-time segmentation of multiple targets in the track scene:

[0135] S51: Construct a feature enhancement module to fuse category information and location information, specifically:

[0136] Λ(x,y)=Conv enhance (Ω(x,y))+PE(x,y);

[0137] Among them, Λ(x,y) is the fusion feature at the pixel coordinate (x,y); Conv enhance To enhance convolution, in this embodiment, Conv enhance A two-layer 3×3 convolutional structure is used. The number of output channels of the first layer is 256, and the number of output channels is 128. Each layer is followed by BatchNorm and ReLU activation functions, and the step size is 1. PE(x, y) is the position encoding of the pixel coordinate (x, y). In this embodiment, sine-cosine position encoding is used, and the encoding dimension is 128.

[0138] S52: Generate mask prediction and obtain segmentation results, specifically:

[0139] ξ(x,y))=Softmax(Conv mask (Λ(x,y)));

[0140] where ξ(x,y) is the instance mask prediction at pixel coordinate (x,y), To predict the pixel at the pixel coordinate (x, y) belongs to The probability of the class target; Conv mask For mask prediction convolution, a 1×1 convolution structure is used in this embodiment, and the number of output channels is The step size is 1;

[0141] The segmentation results are:

[0142]

[0143] Where τ(x,y) is the category label at the pixel coordinate (x,y);

[0144] In this embodiment, network training is performed in an end-to-end manner, and the training parameters include all convolution parameters in steps S3 to S5. During the training process, the cross entropy loss function is used to calculate the difference between the predicted mask and the true label, and the L2 regularization is used to constrain the model parameters. The Adam optimizer is used for parameter optimization, with an initial learning rate of 0.001. The learning rate is reduced to the original 0.1 after every 50 training rounds. The batch size is set to 16, and the total number of training rounds is 200 rounds. The training process is as follows: Figure 3 shown.

[0145] In order to verify the effectiveness of the present invention, a complete experimental scheme was designed, and a progressive approach was used to verify the effectiveness of each module. As shown in Table 1, starting from the baseline model, the spatiotemporal feature fusion module, attention mechanism and multi-scale features were gradually added to finally form a complete scheme. The experimental results show that the spatiotemporal feature fusion module improves the mAP from 81.2% to 85.6%, which proves that the dual-branch spatiotemporal feature extraction architecture can effectively capture the temporal information of dynamic targets. The introduction of the attention mechanism further improves the performance to 88.9%, indicating that the adaptive attention module can effectively enhance the expression of key features. The integration of multi-scale features brings the performance to 91.5%, indicating that the multi-scale feature representation can better handle the detection and segmentation of targets of different scales. The final mAP of the complete scheme reaches 93.2%, and the real-time performance of the system reaches 22.3FPS, which fully meets the needs of practical applications.

[0146] Table 1 Performance comparison of different modules

[0147]

[0148] As shown in Table 2, the background category achieved the highest F1 score of 96.5%, thanks to the relative stability of background region features and the feature enhancement mechanism proposed in this paper. The athlete category achieved an F1 score of 94.2%, primarily due to the effective modeling of motion information by the spatiotemporal feature fusion module. The staff category achieved an F1 score of 90.8%, demonstrating that the multi-scale feature representation and dynamic target classification branch network proposed in this paper are capable of effectively identifying and segmenting this type of target.

[0149] Table 2 Segmentation accuracy of each category

[0150] Target category Accuracy Recall F1 score background 97.1 95.9 96.5 Athletes 94.8 93.6 94.2 staff 91.5 90.1 90.8

[0151] Example 2: The present invention also discloses a track multi-object real-time segmentation system based on deep learning, which includes the following five modules:

[0152] The present invention also discloses a track multi-object real-time segmentation system based on deep learning, comprising:

[0153] Fusion module: obtains the track video data stream and performs dual-branch spatiotemporal feature extraction and fusion;

[0154] Enhancement module: enhances the fused image based on the adaptive attention mechanism;

[0155] Multi-scale module: uses multi-scale convolutional layers to perform feature analysis and reorganization on the enhanced feature maps to construct multi-scale feature representations;

[0156] Target feature extraction module: Based on multi-scale feature maps, a dynamic target classification branch network is designed to realize the recognition of targets of different categories;

[0157] Segmentation module: Uses target feature maps with category information to complete real-time segmentation of multiple targets in the track scene.

[0158] It should be noted that the serial numbers of the above-mentioned embodiments of the present invention are for descriptive purposes only and do not represent the advantages or disadvantages of the embodiments. In addition, the terms "including", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method comprising the element.

[0159] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0160] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A real-time track multi-target segmentation method based on deep learning, characterized by: The following steps are involved: S1: Obtain the track video data stream and perform dual-branch spatiotemporal feature extraction and fusion to obtain the fused image; S2: Enhance the fused image based on the adaptive attention mechanism to obtain an enhanced image; S3: Use multi-scale convolutional layers to perform feature analysis and reorganization on the enhanced feature map, construct multi-scale feature representation, and obtain multi-scale feature maps; S4: Based on the multi-scale feature map, a dynamic target classification branch network is designed to realize the recognition of targets of different categories and obtain feature maps with category information: First, a category-aware convolution operation is performed on the feature map of each scale to obtain the initial features. Then, a specific convolution and global pooling operation are performed on each target category to obtain the category feature representation. Finally, the attention weight of each category is obtained through a fully connected layer and softmax operation. Based on the category feature representation and attention weight, a target feature map with category information is output: the category attention weight is weighted with the category feature after convolution of the feature map to obtain the final feature representation of each category, and the feature representations of all categories are combined to form a complete feature set; S5: Use target feature maps with category information to complete real-time segmentation of multiple targets in the track scene.

2. The track multi-object real-time segmentation method based on deep learning according to claim 1 is characterized in that: The step S1 comprises: S11: Obtain single-frame images and optical flow image sequences from the track video data stream, specifically: Get a single frame image I in the track video data stream at time t t , and simultaneously obtain the optical flow image sequence O of the preceding and following frames including the frame O={O t-T ,…,O t ,…,O t+r },in: O t =Φ(I t ,I t+1 ),t∈[t-r,t+r]; Where t is the time index of the current frame; is the rounding down operation; T is the timing window size, which is an odd number; r is the timing radius; I t and I t+1 are the images at time t and t+1 respectively; O t is the optical flow image at time t; Φ is the optical flow estimation function; S12: Input single frame image I t Perform spatial feature extraction, specifically: Among them, F spt (x,y) is the spatial feature map F spt The eigenvalue at the pixel coordinate (x, y); W spt (p,q) is the spatial convolution kernel weight matrix W spt The value at the row and column number (p,q) of the weight matrix; I t (x+p,y+q) is a single frame image I at time t t The pixel value at pixel coordinates (x+p,y+q); S13: Extract temporal features from the optical flow image sequence of consecutive T frames, specifically: Among them, F tmp (x,y) is the time series feature graph F tmp The eigenvalue at the pixel coordinate (x, y); W tmp (m,n) is the temporal convolution kernel weight matrix W tmp The value at the row and column number (m, n) of the weight matrix; O k’ (x+m, y+n) is the value of the k'th frame optical flow image at the pixel coordinate (x+m, y+n); γ is the normalization function; D k’ (x, y) is the motion displacement vector of the k'th frame at pixel coordinates (x, y); S14: Fusing spatial features and temporal features to obtain a fused image G, specifically: H(x,y)=F spt (x,y)⊙σ(F tmp (x,y)+F tmp (x,y)⊙σ(F spt (x,y))); Where H(x,y) is the eigenvalue of the fused image H at the pixel coordinate (x,y); σ is the sigmoid function; ⊙ is the element-by-element multiplication.

3. The track multi-target real-time segmentation method based on deep learning according to claim 2 is characterized in that: The step S2 comprises: S21: Construct a channel attention module to calculate the importance weight of each channel of the fused image, specifically: Among them, H c (x, y) is the pixel value of the cth channel of the fused image H at the pixel coordinate (x, y); Z c is the global average pooling value of the c-th channel; and are the width and length of the fused image H respectively; ω c is the attention weight of the cth channel; N is the number of channels; e is a natural constant; Z d is the global average pooling value of the d-th channel, d is the dimension index, and its value range is a positive integer from 1 to N; S22: Construct a spatial attention module to calculate the attention map of each spatial position of the fused image H, specifically: Where Q(x,y) is the gradient response intensity at the pixel coordinate (x,y); is the spatial gradient of the cth channel of the fused image H at the pixel coordinate (x, y); R(x, y) is the spatial attention value at the pixel coordinate (x, y); S23: Fuse channel attention and spatial attention to obtain the enhanced image E, specifically: Among them, E c (x, y) is the pixel value of the c-th channel of the enhanced image E at the pixel coordinate (x, y); σ′ is the ReLU function; is the arithmetic square root of N.

4. The method for real-time segmentation of multiple targets on a track based on deep learning according to claim 3, characterized in that: Said S3 comprises the following steps: S31: Generate decomposition diagrams of different scales, specifically: P k =Conv k (E),k∈{1,2,3}; Among them, P k is a decomposition diagram with scale k, P k (x,y) is the decomposition graph P with scale k k The pixel value at the pixel coordinate (x, y); Conv k is a convolution operation of scale k; and are the width and length of the decomposition graph with scale k; α k is the global representation of the decomposition graph with scale k; S32: Generate a multi-scale feature map, specifically: in, is a feature map of scale k; Conv re Reconstruct the convolution operation; Res k (E) is the feature map obtained by performing a residual connection operation on the enhanced image E when the scale is k.

5. The method for real-time segmentation of multiple targets on a track based on deep learning according to claim 4, characterized in that: Said S4 comprises the following steps: S41: Construct a category-aware attention module to extract category-related features, specifically: Among them, μ k is a feature map of scale k Category-aware convolution cls The characteristics of the latter; For the Category feature representation of class targets, through category-specific convolution Extraction; Pool is a global pooling operation; For the Attention weight of class target; is the number of target categories; and For category features and The feature vector obtained by performing the fully connected layer transformation can map any range of input to the range of 0 to 1 through a combination of exponential function and normalization while maintaining the relative importance between different categories; S42: Output the target feature map with category information, specifically: in, For the The final feature representation of the class target at the pixel coordinate (x, y); Conv map is the feature map convolution; Ω(x,y) is the feature set containing all category information at the pixel coordinate (x,y).

6. The method for real-time segmentation of multiple targets on a track based on deep learning according to claim 5, characterized in that: Said S5 comprises the following steps: S51: Construct a feature enhancement module to fuse category information and location information, specifically: Λ(x,y)=Conv enhance (Ω(x,y))+PE(x,y); Among them, Λ(x,y) is the fusion feature at the pixel coordinate (x,y); Conv enhance is enhanced convolution; PE(x,y) is the position encoding of the pixel coordinate (x,y); S52: Generate mask prediction and obtain segmentation results, specifically: ξ(x,y)=Softmax(Conv mask (Λ(x,y))); where ξ(x,y) is the instance mask prediction at pixel coordinate (x,y), To predict the pixel at the pixel coordinate (x, y) belongs to The probability of the class target; Conv mask Convolution for mask prediction; The segmentation results are: Where τ(x,y) is the category label at the pixel coordinate (x,y).

7. A track multi-object real-time segmentation system based on deep learning, characterized by: include: Fusion module: obtains the track video data stream and performs dual-branch spatiotemporal feature extraction and fusion; Enhancement module: enhances the fused image based on the adaptive attention mechanism; Multi-scale module: uses multi-scale convolutional layers to perform feature analysis and reorganization on the enhanced feature maps to construct multi-scale feature representations; Target feature extraction module: Based on multi-scale feature maps, a dynamic target classification branch network is designed to realize the recognition of targets of different categories; Segmentation module: Uses target feature maps with category information to complete real-time segmentation of multiple targets in the track scene; To realize a real-time track multi-object segmentation method based on deep learning as described in any one of claims 1-6.

Images