Visual information processing method in railway shunting operation scene

Abstract

The present invention discloses a method for processing visual information in a railway shunting operation scene, and belongs to the field of computer vision. In order to solve the problem of efficiently and quickly understanding and identifying tracks and railway targets, the present invention implements an anchor line generator, a track detection head and a target detection head on the basis of a shared encoder, and finally efficiently realizes track detection tasks and railway target detection tasks. The present invention has low requirements for computing resources, can realize real-time railway target detection and track detection tasks, and can track and identify tracks in complex track scenes, thereby improving system reliability.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for processing visual information in a railway shunting operation scene. Background Art

[0002] At present, intelligent visual information processing in the field of autonomous driving is an extremely popular research direction. In the field of autonomous driving, multi-task detection networks mainly focus on three detection tasks: lane segmentation, drivable area segmentation, and traffic object target detection. This is because the road structure is relatively complex, including multiple lanes, intersections, pedestrian crossings and other variable factors. Especially when high-definition maps and precise positioning are not available, the perception of these three tasks is crucial for autonomous driving.

[0003] In the railway shunting operation scenario, since the railway tracks are relatively regular, the technical focus should be on (a) railway target detection tasks; (b) track detection tasks, that is, accurately identifying and monitoring fixed track paths.

[0004] The basis of the current railway video car leading system is to use a variety of technical means such as video acquisition technology, deep learning technology, and visual ranging technology to obtain the road conditions in front of the first car pushed in the shunting operation and the position information of the parked car.

[0005] With the rapid development of deep learning, convolutional neural networks have been widely used in the fields of track detection and railway target detection. However, existing detection methods usually consider track detection tasks and railway target detection tasks separately. In the system for pushing shunting operation video lead-in, due to limited computing resources, if two independent detection models are used on the equipment installed on the first vehicle pushed, it will not only increase the hardware requirements but also reduce the real-time performance.

[0006] (a) For the railway target detection task, the main purpose is to identify the retained vehicles and / or coupler targets in a specific area, and indicate the location of the object through a bounding box, so as to locate the object and determine the object type. In the field of target detection, detection methods can be divided into two-stage target detection methods and one-stage target detection methods.

[0007] The two-stage object detection method consists of two steps: first, selecting candidate regions from the entire image, and second, finding objects in the candidate regions, classifying the found objects, and generating the bounding box of the final object. Although the two-stage object detection method has a high accuracy rate, it consumes a lot of time and resources, which is not conducive to real-time application of the network.

[0008] The one-stage target detection method is an improvement on the two-stage target detection method. It regards positioning and classification as a regression problem, realizes end-to-end detection, and has a fast detection speed. First, the detection box selection is integrated with the target positioning classification, and the target is selected when the anchor box is arranged for the entire image, so as to realize the region proposal selection and target positioning. Each anchor box is divided into classes and the confidence is calculated to complete the classification of each anchor box, and finally the target is positioned and classified. The one-stage network performs better in terms of speed and accuracy, so it has been well applied in academia and industry.

[0009] (b) For track detection tasks, traditional track detection methods divide track line areas through edge detection and filtering, and then perform track line detection in combination with Hough transform, Ransac method, etc. However, traditional methods require manual adjustment of parameters according to the characteristics of the application scenario, which is greatly limited in application. Therefore, methods based on deep learning are highly favored. Methods based on deep learning can be roughly divided into three categories: one is based on semantic segmentation, the second is based on line classification, and the third is based on anchor line.

[0010] The track line detection method based on semantic segmentation only focuses on the track within the current track area of ​​the train. Due to the slender shape of the track, the target to be detected occupies very few pixels in the scene. This method of segmenting small targets is not only difficult to segment the intended target, but also wastes computing resources and does not meet the requirements of real-time detection. The row classification-based method is a row classification detection method based on the grid division of the input image. For each row, the model predicts the cell that is most likely to contain part of the track, and then fits the track line through these key points. This method is faster than the segmentation method, but the detection effect for occluded scenes may be poor. The anchor line-based method can directly predict the geometric structure of the track line, has a fast execution speed, meets the real-time processing requirements, and because it focuses on the structure of the line rather than the color or texture information at the pixel level, it also has good robustness for occluded scenes.

[0011] For railway shunting operation scenarios, there is no public technical solution that can complete railway target detection tasks and track detection tasks in real time and simultaneously, and occupy less computing resources.

[0012] The present invention is based on the railway video lead system and focuses on the track detection tasks and railway target detection tasks in the railway shunting operation scenario. The present invention proposes a new multi-task detection network that can be installed in the equipment for pushing the first car with limited computing resources and can achieve end-to-end detection of multiple targets. Summary of the invention

[0013] In order to alleviate or partially alleviate the above technical problems, the solution of the present invention is as follows:

[0014] A method for processing visual information in a railway shunting operation scene, inputting a picture obtained by collecting visual information into an encoder, wherein the encoder includes a convolutional neural network and a feature pyramid network, the convolutional neural network includes multiple segments, and extracting the feature maps of the last layer in several segments of the convolutional neural network to construct a feature pyramid network including a first feature map, a second feature map and a third feature map; the second feature map is sequentially input into a first prediction branch, a second prediction branch, a third prediction branch and a fourth prediction branch, and is used to predict a starting point, a starting point offset, an angle and a dynamic kernel parameter about an anchor line respectively; a plurality of dynamic convolution kernel parameters are generated for each predicted starting point through a loop instance module; and a plurality of dynamic convolution kernel parameters are generated from the third feature map. Extracting features of the region of interest about the anchor line, and using the multiple dynamic convolution kernel parameters to perform conditional convolution operations on the features of the region of interest, and using them as the input of the fully connected layer to obtain the horizontal offset between the track and the anchor line, and obtaining the predicted track information based on the starting point of the anchor line, the starting point offset and angle, and the horizontal offset between the track and the anchor line; and, according to the feature pyramid network, downsampling the next layer of features and fusing them with the previous layer of features to generate a path aggregation network; assigning multiple prior anchor frames with different aspect ratios to each grid in the path aggregation network, and predicting the position offset of the railway target, the scaling of the height and width of the prior anchor frame, and the probability that the railway target belongs to each category through the target detection head.

[0015] Furthermore, the sizes of the first characteristic map, the second characteristic map and the third characteristic map increase sequentially.

[0016] Furthermore, the first feature map in the feature pyramid network is obtained by extracting the feature map of the last layer in the last section of the convolutional neural network and performing a multi-layer maximum pooling operation of the spatial pyramid pooling module; the second feature map in the feature pyramid network is obtained by extracting the feature map of the last layer in the second-to-last section of the convolutional neural network and performing a 1×1 convolution and adding the upsampled first feature map; the third feature map in the feature pyramid network is obtained by extracting the feature map of the last layer in the third-to-last section of the convolutional neural network and performing a 1×1 convolution and adding the upsampled second feature map.

[0017] Furthermore, when predicting the starting point, starting point offset, angle and dynamic kernel parameters about the anchor line, a starting point heat map, a starting point offset map, an angle map and a parameter map are generated respectively.

[0018] Furthermore, multiple dynamic convolution kernel parameters are generated for each anchor line starting point through a loop instance module.

[0019] Furthermore, during the training phase, the weighted track detection loss and railway object detection loss are used as the total loss of the multi-task network that simultaneously processes the track detection task and the railway object detection task.

[0020] Furthermore, within each segment of the convolutional neural network, the size of the feature map is not changed.

[0021] Furthermore, the railway object is a retained car and / or a coupler.

[0022] Furthermore, the size of the starting point heat map, the size of the starting point offset map, and the size of the angle map are all the same as the size of the second feature map.

[0023] Furthermore, during the training phase, the track detection loss includes angle prediction loss, starting point offset prediction loss, starting point prediction loss and cross entropy loss, where the cross entropy loss is a constraint on the state output result of the loop instance module.

[0024] The technical solution of the present invention has one or more of the following beneficial technical effects:

[0025] (1) This paper focuses on the specific needs of the video-guided vehicle system and takes the lead in building an encoder-decoder multi-task network architecture to achieve an end-to-end training mode. This architecture can detect and process tracks and targets simultaneously through a neural network in real time, significantly reducing the number of parameters that need to be trained and stored, reducing the demand for computing resources, and thus improving the operating efficiency of the system.

[0026] (2) Compared with the prior art that realizes track detection and railway target detection tasks through two independent networks, the present invention uses a single neural network to handle multiple tasks, which helps to improve the processing speed, enabling the system to run in real time in an environment with limited resources and meet the needs of real-time monitoring and processing.

[0027] (3) Given the difficulty in accurately distinguishing and tracking each track line in the existing single anchor line generation method, the present invention can ensure that each rail is tracked independently and accurately, thereby improving accuracy and robustness in complex turnout conditions. This adaptability is particularly critical in complex or changing track environments and can effectively improve the overall performance and reliability of the system.

[0028] (4) By predicting the starting point, starting point offset, and angle, information about the anchor line can be efficiently output, reducing the difficulty of the track detection task.

[0029] In addition, other beneficial effects of the present invention will be mentioned in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the overall architecture of the multi-task detection network;

[0031] Figure 2 This is a schematic diagram of the architecture of the multi-task detection network for track detection tasks;

[0032] Figure 3 It is a schematic diagram of the anchor line;

[0033] Figure 4 This is a schematic diagram of the architecture of the multi-task detection network for railway target detection tasks. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order.

[0036] Figure 1 Schematic diagram of the overall architecture of the multi-task detection network. Different from the conventional technology, for given tasks 1, 2 and 3, the multi-task detection network of the present invention comprises an encoder and a decoder 1 for processing task 1, a decoder 2 for processing task 2 and a decoder 3 for processing task 3 respectively.

[0037] The overall architecture of the multi-task detection network is divided into an encoder and a decoder. The encoder is an encoder based on a hard parameter sharing mechanism. The encoder shares task features in the encoding stage and is a shared underlying network in the architecture, located after the input layer.

[0038] On top of the encoder is a pyramid network for multiple tasks, and the decoder pyramid is built on the output of the shared underlying network, which is the output of the encoder. Decoder 1, Decoder 2, and Decoder 3 are a set of independent task-specific heads that directly predict the output of all tasks from the same input in one processing cycle, such as the output of task 1, the output of task 2, and the output of task 3.

[0039] For example, the encoder of the present invention is a backbone network composed of a series of convolutional layers, which is used to extract features from input. The input referred to in the present invention is an image acquired or processed by an image acquisition device.

[0040] Figure 2 The multi-task detection network of the present invention is a schematic diagram of a track detection task architecture. The multi-task detection network of the present invention performs a track detection task on the one hand, and performs a railway target detection task on the other hand.

[0041] As one of the technical features of the present invention, the present invention simultaneously completes the track detection task and the railway target detection task in one network, which reduces the hardware resource requirements and improves the real-time performance of information processing.

[0042] The encoder receives a picture captured by a visual information acquisition device, such as a camera, as input information.

[0043] In the encoder, a convolutional neural network is included. The convolutional neural network includes multiple segments or stages, exemplarily a total of 5 segments of the neural network, wherein the feature map size in each segment of the neural network is the same, in other words, the size of the feature map is not changed in each segment of the convolutional neural network. As the depth increases, the size of the convolutional neural network is getting smaller.

[0044] The feature graphs of the last layer of several segments of the convolutional neural network are extracted and applied to construct feature pyramid networks (FPN). Since the deep network layer has stronger semantic features but weaker position information representation, and the shallow network layer has stronger position information representation but weaker semantic features, the feature pyramid network can be used to fuse feature graphs of different scales and enhance the representation ability of feature graphs.

[0045] The encoder includes a feature pyramid network. Through the upsampling method of interpolation values, the size of the extracted feature map is enlarged, and the size of the enlarged feature map is the same as the size of the next layer of feature map extracted. This method can transfer the semantic features in the deep network to the shallow network, thereby enhancing the semantic expression at multiple scales.

[0046] Furthermore, the upsampling refers to inserting new pixel values ​​between different pixel values ​​through an interpolation algorithm based on the original feature map pixel values, thereby expanding the size of the original feature map. For example, the upsampling ratio is 2 times.

[0047] The feature map generated from bottom to top is subjected to 1×1 convolution, and then fused with the enlarged feature map obtained by upsampling by addition, so as to obtain the feature maps of different layers, namely, the second feature map P4 and the third feature map P3. The present invention fuses the feature maps of different layers to provide them with richer information.

[0048] In other words, by extracting the feature map of the last layer in the penultimate segment of the convolutional neural network, performing a 1×1 convolution, and adding the upsampled first feature map P5, the second feature map P4 in the feature pyramid network is obtained.

[0049] The third feature map P3 in the feature pyramid network is obtained by extracting the feature map of the last layer in the third-to-last segment of the convolutional neural network, performing a 1×1 convolution, and adding the upsampled second feature map P4.

[0050] In order to solve the problem of image distortion caused by cropping and scaling operations, and to adapt to the problem of input images of different sizes, the feature map of the last layer in the last segment of the convolutional neural network is extracted, and a multi-layer maximum pooling operation is performed by the spatial pyramid pooling (SPP) module to generate fixed-size outputs for feature maps of different sizes, thereby enhancing the feature extraction capability of the network and obtaining the first feature map P5 in the feature pyramid network.

[0051] In the present invention, the feature pyramid network includes a first feature map P5, a second feature map P4 and a third feature map P3, and the sizes of the first feature map P5, the second feature map P4 and the third feature map P3 increase sequentially.

[0052] In the present invention, the specific implementation details of the spatial pyramid pooling technology can refer to the prior art 1: He, Kaiming et al. “Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition.” IEEE Transactions on Pattern Analysis and Machine Intelligence 37 (2014): 1904-1916. The present invention introduces all the above technical contents into this application by reference.

[0053] The number of decoders designed in the present invention is 3, and they perform three tasks respectively: (1) anchor line generation; (2) track detection; (3) railway target detection. Among them, the anchor line generation task is completed by the anchor line generator, and the anchor line track provides prior information about the position. On this basis, the track detection head completes the track detection task.

[0054] For example, the railway targets in the railway target detection task are retained cars and / or couplers.

[0055] The anchor line can be regarded as a ray starting from the image boundary with a certain slope, so it can be parameterized as ,in Indicates the starting point coordinates, Indicates an angle or slope.

[0056] Furthermore, the second feature map P4 is input into three parallel prediction branches, namely the first prediction branch, the second prediction branch and the third prediction branch, and respectively predicts the starting point, starting point offset and angle of the anchor line, and dynamically constructs the anchor line for each track instance based on the predicted results.

[0057] Furthermore, the second feature map P4 is also input into the fourth prediction branch to predict dynamic kernel parameters and form a parameter map.

[0058] In the present invention, the constructed anchor line is used as reference information for the track detection head, and finally the track is positioned by the predicted horizontal offset.

[0059] In the present invention, the track P can be It is represented by a sequence of two-dimensional coordinate point sets: P = The y coordinate in the coordinate point set is preset and is uniformly sampled along the vertical axis of the image. The starting index of the track on the vertical axis is recorded as s , the length is l .

[0060] In the present invention, the first heat map is a starting point heat map, the second heat map is a starting point offset map, and the third heat map is an angle map.

[0061] (1) For the starting point, the first prediction branch can use two convolutional layers to estimate the starting point position. The input of the two convolutional layers is the second feature map P4, and the output is the first heat map , to represent the probability of each pixel belonging to the starting point, where H 4 and W 4 is the size of the second feature map P4.

[0062] In the training phase, the starting point of the track is used to generate the first heat map of the ground truth, where i, j ) The true label at the position for:

[0063] = ;

[0064] in is the spatial index of the live heat map, H 4 and W 4 is the size of the fourth feature map P4, is the standard deviation related to the input image size, and For a given set of track starting point coordinates, k is the index number of the track.

[0065] In order to solve the imbalance problem between the starting point area and the non-starting point area, a focal loss is applied to the first heat map during the prediction process, assuming and In the first heat map, they are i, j ) The predicted label and the true label at the position, about the starting point prediction loss It can be defined as:

[0066] = ;

[0067] in, and is a tunable hyperparameter, is the number of tracks in the image, H 4 and W 4 is the size of the fourth feature map P4, and In the first heat map, they are i, j ) The predicted label and the true label at the position.

[0068] (2) For the starting point offset, the peak in the first heat map provides a rough estimate of the starting point, which is as accurate as the step size in the second feature map P4. Related.

[0069] Second prediction branch prediction offset map of the present invention ,in H 4 and W 4 is the size of the second heatmap.

[0070] In the training phase, a square area within the distance r from the actual starting point is selected as the effective area, and the smooth L1 loss function commonly used in the field is applied in this area (.) to constrain the predicted offset map, about the starting point offset prediction loss It can be defined as:

[0071] = ;

[0072] in, is the number of tracks in the image,r is half the length of the side of the square, and is the effective position relative to The offset along the x-axis and y-axis, is the offset map, is the step size in the second feature map P4, Is the starting point p The position points obtained after downsampling.

[0073] (3) For angles, in order to predict the angle map , H 4 and W 4 is the size of the fourth feature map P4. For a given track , the average slope of all lines connecting the non-starting point and the starting point on the track is taken as the ground truth slope or angle :

[0074] ;

[0075] in, arctan2 (.) is the four-quadrant inverse tangent function, the orbital cutoff index e=s+l -1, s is the starting index, l is the track length, ( x i , y i ) is the track L The point in , and ( x s , y s ) is the track L The starting point in .

[0076] During the training phase, the same valid training area is selected as the offset estimate and the angle prediction loss is used. Predict with constrained angles:

[0077] ;

[0078] in, is the ground truth slope or angle, k is the index number of the track, is the predicted angle graph, L 1 Loss (.) is the L1 loss function, r is half the length of the side of the square, and is the effective position relative to The offset along the x-axis and y-axis, is the number of tracks in the image, It is the position point obtained after downsampling of the starting point p.

[0079] Figure 3 is the anchor line diagram. Finally, the anchor line generated by the anchor line generator is: ,in is the starting point of the prediction In the offset diagram The predicted starting point offset in is the starting point of the prediction In the angle diagram The predicted angle, (.) represents mapping the coordinate values ​​to the final scale of the model output, ensuring that the predicted coordinates can correctly adapt to the location in the original image or target space.

[0080] In the above scheme, each starting point corresponds to a track instance. However, at the fork of the track, multiple rails may often share the same starting point.

[0081] In order to deal with this special situation, the present invention adopts a recurrent instance module (RIM) to generate dynamic kernel parameters through a recursive neural network and applies them to the conditional convolution processing process. This method can generate multiple sets of parameters for multiple tracks with the same starting point, ensuring that each rail is tracked and identified independently and accurately, thereby improving the accuracy and robustness of recognition in complex track scenarios.

[0082] The recurrent instance module is based on the Long Short-Term Memory (LSTM) module. The RIM receives an input feature vector and recurrently predicts a state vector and a kernel parameter vector.

[0083] In the present invention, regarding the loop instance module technology, reference may be made to prior art 2: Liu, Lizhe et al. “CondLaneNet: a Top-to-down Lane Detection Framework Based on ConditionalConvolution.” 2021 IEEE / CVF International Conference on Computer Vision (ICCV) (2021): 3753-3762. The present invention introduces all the above technical contents into this application by reference.

[0084] For each starting point predicted in the anchor line generator, RIM is added. During the inference phase, RIM repeatedly predicts the kernel parameter vector corresponding to the track bound to the same starting point until the state indicates to stop, thereby generating multiple dynamic convolution kernel parameters for each anchor line starting point through the feature maps of the starting point and the surrounding starting point.

[0085] The multiple dynamic convolution kernel parameters can be regarded as multiple sub-proposals of the starting point, and each sub-proposal corresponds to a candidate instance. Therefore, each starting point can guide the shape prediction of multiple track instances.

[0086] For example, we can use the cross entropy loss Constrain the status output results of RIM:

[0087] ;

[0088] in, s i is the i-th state softmax The output after operation is y i is the ground truth label of the ith state, N s is the total number of state outputs in the batch.

[0089] In the track detection head, the anchor line generated by the anchor line generator is used as a reference to ultimately accurately locate the track.

[0090] First, in the RoI feature extraction module, the region of interest (RoI) features about the anchor line are extracted from the third feature map P3. , aligned uniform sampling N s points, and then use bilinear interpolation to calculate the features of each sampling point, and concatenate the features of all sampling points into RoI features, where ( x start , y start ) are the coordinates of the starting point, is the angle.

[0091] Then, the RIM module is used to generate dynamic convolution kernel parameters, and the RoI features are used as input. Conditional convolution operations are performed on the RoI features according to the dynamic convolution kernel parameters. The generated features will be more refined and can adapt to the features of different instances.

[0092] Finally, in the prediction adjustment module, the RoI features processed by the dynamic convolution kernel parameters are used as the input of the fully connected layer to generate a Npts Horizontal offset The track length is l , the starting index is s .

[0093] For each fixed vertical coordinate y i , the horizontal axis coordinate of each corresponding prediction for:

[0094] ;

[0095] in, is the angle, y i is the vertical axis coordinate, ( x start , y start ) are the coordinates of the starting point, is the i-th horizontal offset, is the predicted horizontal axis coordinate. The predicted track information can be obtained based on the starting point, starting point offset and angle information about the anchor line, as well as the horizontal offset between the track and the anchor line.

[0096] During the training phase, the track detection loss It can be defined as follows:

[0097] ;

[0098] in α, β, γ and η is a hyperparameter, is the aforementioned cross entropy loss, is the angle prediction loss, is the starting point offset prediction loss, is the starting point prediction loss.

[0099] Figure 4 This is a schematic diagram of the architecture of the multi-task detection network for railway target detection tasks. The core of the target detection task is to identify the target object in the image and accurately determine its location.

[0100] The present invention adopts a target detection scheme based on a multi-scale anchor frame mechanism. In order to extract features of different scales in an image, it is necessary to obtain feature maps of different sizes of the image through sampling, and the collection of these feature maps of different sizes constitutes an image pyramid.

[0101] Directly using multiple scale feature maps to predict the target position will result in a large amount of calculation and occupy a large amount of memory. In this regard, the present invention adopts the FPN structure, extracts features from the bottom up and fuses different levels of features from the top down, which not only avoids a large amount of calculation, but also fully retains the original image features.

[0102] However, the FPN structure only transmits semantic information from top to bottom and does not contain positioning information. To this end, the present invention uses another feature pyramid to specifically transmit positioning information, namely the Path Aggregation Network (PAN) structure. The PAN structure transmits multi-scale positioning information from bottom to top. FPN transmits semantic features from top to bottom, and PAN transmits positioning features from bottom to top. The combination of the two can achieve a better feature fusion effect.

[0103] For example, according to the feature pyramid network, the next layer of features or low-level features are downsampled and fused with the previous layer of features or high-level features to generate a path aggregation network. The path aggregation network includes feature graphs N5, N4, and N3, wherein feature graph N5 is located at the top layer and feature graph N3 is located at the bottom layer. In other words, high-level features are generated from low-level features, and the high-level and low-level are relative concepts.

[0104] Subsequently, each grid of the multi-scale fusion feature map in PAN will be assigned three prior anchor boxes with different aspect ratios, and the object detection head will predict the position offset of the object, the scaling of the height and width of the prior anchor boxes, and the probability of the railway object belonging to each category.

[0105] The present invention proposes the following anchor box matching criteria: first determine the grid position where the center point of the true value target is located, and then calculate the aspect ratios of multiple anchor boxes of different sizes generated corresponding to this grid and the bounding box of the true value target. If the ratio is less than the ratio threshold, the anchor box is determined to match the true value target and marked as a positive sample; otherwise, it is marked as a negative sample.

[0106] According to the above anchor box matching criteria, after obtaining the predicted value of the network output, the target detection loss It can be calculated as follows:

[0107] ;

[0108] in is the bounding box regression loss, is the target confidence loss, is the classification loss.

[0109] Finally, the total loss of the multi-task network that simultaneously completes the railway target detection task and the track detection task is is the track detection loss With object detection loss The weighted sum of:

[0110] ;

[0111] in, and are the track detection task weight and the target detection task weight, which are two hyperparameters; is the track detection loss, is the object detection loss.

[0112] The multi-task network is trained by the back-propagation algorithm and the total loss of the multi-task network to obtain network parameters that minimize or approach the minimum total loss of the multi-task network and use them as parameters for multi-task network reasoning.

[0113] In order to better illustrate the present invention, numerous specific details are provided in the above specific embodiments. It should be understood by those skilled in the art that the present invention can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present invention.

[0114] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A visual information processing method in a railway shunting operation scene, characterized in that: Inputting the image obtained by collecting visual information into an encoder, wherein the encoder includes a convolutional neural network and a feature pyramid network, the convolutional neural network includes a plurality of segments, and extracting feature maps of the last layer in a plurality of segments in the convolutional neural network to construct a feature pyramid network including a first feature map, a second feature map, and a third feature map; The second feature map is sequentially input into the first prediction branch, the second prediction branch, the third prediction branch and the fourth prediction branch, and is respectively used to predict the starting point, the starting point offset, the angle and the dynamic kernel parameter of the anchor line, and the anchor line is dynamically constructed for each track instance according to the predicted starting point, the starting point offset and the angle of the anchor line; and, When predicting the starting point, starting point offset, angle and dynamic kernel parameters about the anchor line, a starting point heat map, a starting point offset map, an angle map and a parameter map are generated respectively; Generate multiple dynamic convolution kernel parameters for each predicted starting point and generate multiple dynamic convolution kernel parameters for each anchor line starting point through a loop instance module; Extracting a region of interest feature about the anchor line from the third feature map, performing a conditional convolution operation on the region of interest feature using the multiple dynamic convolution kernel parameters, and using the feature as an input of the fully connected layer to obtain a horizontal offset between the track and the anchor line, and obtaining predicted track information according to a starting point, a starting point offset and an angle of the anchor line, and a horizontal offset between the track and the anchor line; as well as, According to the feature pyramid network, the next layer of features is downsampled and fused with the previous layer of features to generate a path aggregation network; Assign multiple prior anchor boxes with different aspect ratios to each grid in the path aggregation network, and predict the position offset of the railway target, the height and width scaling of the prior anchor box, and the probability that the railway target belongs to each category through the object detection head; and The first feature map in the feature pyramid network is obtained by extracting the feature map of the last layer in the last first segment of the convolutional neural network and performing multi-layer maximum pooling operations in the spatial pyramid pooling module; The second feature map in the feature pyramid network is obtained by extracting the feature map of the last layer in the penultimate section of the convolutional neural network, performing a 1×1 convolution, and adding the upsampled first feature map. The third feature map in the feature pyramid network is obtained by extracting the feature map of the last layer in the third-to-last segment of the convolutional neural network, performing a 1×1 convolution, and adding the upsampled second feature map.

2. The visual information processing method in the railway shunting operation scene according to claim 1 is characterized in that: The sizes of the first characteristic map, the second characteristic map and the third characteristic map increase sequentially.

3. The visual information processing method in the railway shunting operation scene according to claim 1 or 2, characterized in that: During the training phase, the weighted track detection loss and railway object detection loss are used as the total loss of the multi-task network that simultaneously processes the track detection task and the railway object detection task.

4. The visual information processing method in the railway shunting operation scene according to claim 3 is characterized in that: In each segment of the convolutional neural network, the size of the feature map is not changed.

5. The visual information processing method in the railway shunting operation scene according to claim 1 or 4, characterized in that: The railway objects are retained cars and / or couplers.

6. The visual information processing method in the railway shunting operation scene according to claim 5 is characterized by: The size of the starting point heat map, the size of the starting point offset map, and the size of the angle map are all the same as the size of the second feature map.

7. The visual information processing method in the railway shunting operation scene according to claim 1 is characterized by: During the training phase, the track detection loss includes angle prediction loss, starting point offset prediction loss, starting point prediction loss and cross entropy loss, where the cross entropy loss is a constraint on the state output results of the loop instance module.

Images