Floating object identification method, device, computer program product and electronic equipment

Abstract

This application discloses a floating object identification method, device, computer program product, and electronic device. This method relates to the fields of artificial intelligence and big data. The method comprises: acquiring railway images captured by monitoring equipment at each acquisition moment within a target time period to obtain a target railway image; determining meteorological data at each acquisition moment to obtain target meteorological data corresponding to each frame of the target railway image; and inputting the target railway image and target meteorological data into a target model to obtain floating object identification results at each acquisition moment. The target model is trained using multiple sets of training samples, each set of training samples comprising historical railway images, historical meteorological data corresponding to the historical railway images, and historical floating object identification results. This application addresses the problem of poor floating object identification accuracy in related technologies.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and big data, and more specifically, to a floating object identification method, device, computer program product, and electronic device. Background Art

[0002] With the rapid expansion of railway coverage, train safety has become a key concern. The catenary system provides a stable power supply for trains. Damage to the catenary system caused by shear, stretching, and friction, among other forces, can threaten train safety. Floating debris is a major cause of damage to the catenary system. If not promptly removed, floating debris, under the influence of wind, can significantly impact the operation of the catenary system. This can cause minor faults and power disruptions, while more serious consequences include disconnection, impacting train safety, and even causing derailments and overturning.

[0003] In related technologies, single-modal railway images are used as input for artificial intelligence algorithms. When performing target type recognition, they are easily affected by the training process. If the amount of training data is insufficient, it will affect the accuracy of the algorithm, resulting in poor accuracy in floating object recognition.

[0004] Currently, no effective solution has been proposed to address the problem of poor accuracy in identifying floating objects in related technologies. Summary of the Invention

[0005] The main purpose of this application is to provide a floating object identification method, device, computer program product and electronic device to solve the problem of poor accuracy in floating object identification in related technologies.

[0006] To achieve the above objectives, according to one aspect of the present application, a floating object identification method is provided. The method comprises: acquiring railway images captured by monitoring equipment at each acquisition moment within a target time period to obtain a target railway image; determining meteorological data at each acquisition moment to obtain target meteorological data corresponding to each frame of the target railway image; and inputting the target railway image and target meteorological data into a target model to obtain a floating object identification result at each acquisition moment. The target model is trained using multiple sets of training samples, each set of training samples comprising historical railway images, historical meteorological data corresponding to the historical railway images, and historical floating object identification results.

[0007] Optionally, the target model is obtained by: determining an initial convolution processing layer, an initial encoder, and an initial decoder; obtaining a first training sample set, training the initial convolution processing layer based on the first training sample set, and obtaining a trained convolution processing layer, wherein the first training sample set includes historical railway images and historical floating object recognition results; obtaining a second training sample set, training the initial encoder and the initial decoder based on the second training sample set, and obtaining a trained encoder and a trained decoder, wherein the second training sample set includes historical superimposed feature maps, historical coding features, and historical floating object recognition results; obtaining multiple groups of training samples, and jointly training the trained convolution processing layer, the trained encoder, and the trained decoder based on the multiple groups of training samples to obtain the target model.

[0008] Optionally, the target railway image and target meteorological data are input into the target model to obtain the floating object recognition result at each acquisition moment, including: extracting the three primary color pixel values ​​of the target railway image to obtain a three primary color pixel matrix; inputting the three primary color pixel matrix into the convolution processing layer of the target model for processing to obtain multiple feature maps; extracting keywords from the target meteorological data to obtain meteorological keywords, and converting the meteorological keywords into meteorological word vectors through a word vector conversion tool, wherein the meteorological word vectors have the same dimension as the feature map; for each feature map, superimposing the meteorological word vector and the feature map through the fusion parameters of the target model to obtain a superimposed feature map; encoding the superimposed feature map through the encoder of the target model to obtain encoded features, and decoding the encoded features through the decoder of the target model to obtain the floating object recognition result.

[0009] Optionally, the fusion parameters include meteorological word vector coefficients and feature map coefficients. The meteorological word vector and the feature map are superimposed by the fusion parameters of the target model to obtain the superimposed feature map, including: determining each first element in the meteorological word vector, multiplying the meteorological word vector coefficient by each first element to obtain an intermediate word vector; determining each second element in the feature map, multiplying the feature map coefficient by each second element to obtain an intermediate feature map; and summing the intermediate word vector and the intermediate feature map to obtain a superimposed feature map.

[0010] Optionally, obtaining the railway image captured by the monitoring device at each capture moment within the target time period, and obtaining the target railway image includes: receiving the railway image captured by the monitoring device at the capture moment; comparing the railway image with a preset background image to obtain a difference pixel area, wherein the background image is an image without floating objects captured by the monitoring device at the same shooting angle as the railway image; and determining the image corresponding to the difference pixel area as the target railway image.

[0011] Optionally, after obtaining the floating object identification result at each acquisition moment, the method further includes: when the floating object identification result indicates the presence of floating objects in the railway image, determining the railway image as the initial image in which the floating objects appear; determining multiple frames of railway images within a preset time period acquired after the initial image, and determining the movement trajectory of the floating objects based on the multiple frames of railway images, and determining the predicted movement trajectory of the floating objects within the target time period based on the movement trajectory of the floating objects; determining the position information of the railway power grid coverage area, and judging whether the floating objects fall into the railway power grid coverage area based on the position information and the predicted movement trajectory; and when it is determined that the floating objects will fall into the railway power grid coverage area, controlling the blowing device to adjust the movement trajectory of the floating objects.

[0012] Optionally, controlling the blowing device to adjust the motion trajectory of the floating object includes: determining the position information of the blowing device, and determining the opening time and blowing parameters of the blowing device based on the position information of the blowing device and the predicted motion trajectory, wherein the blowing parameters include at least one of the following: wind speed level and wind direction; controlling the blowing device to operate according to the blowing parameters at the opening time to adjust the motion trajectory of the floating object.

[0013] To achieve the above objectives, according to another aspect of the present application, a floating object identification device is provided. The device comprises: an acquisition unit configured to acquire railway images captured by monitoring equipment at each acquisition moment within a target time period to obtain a target railway image; a first determination unit configured to determine meteorological data at each acquisition moment to obtain target meteorological data corresponding to each frame of the target railway image; and an input unit configured to input the target railway image and target meteorological data into a target model to obtain a floating object identification result at each acquisition moment. The target model is trained using multiple sets of training samples, each set of training samples comprising historical railway images, historical meteorological data corresponding to the historical railway images, and historical floating object identification results.

[0014] In order to achieve the above-mentioned object, according to another aspect of the present application, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the floating object identification method described in each embodiment of the present application.

[0015] The present application employs the following steps: obtaining railway images captured by monitoring equipment at each acquisition moment within a target time period to obtain a target railway image; determining meteorological data at each acquisition moment to obtain target meteorological data corresponding to each frame of the target railway image; and inputting the target railway image and target meteorological data into a target model to obtain a floating object recognition result at each acquisition moment. The target model is trained using multiple sets of training samples, each set of training samples including historical railway images, historical meteorological data corresponding to the historical railway images, and historical floating object recognition results. This solves the problem of poor floating object recognition accuracy in related technologies. Because floating objects are significantly affected by weather, by combining railway images and meteorological data, multimodal data is input into the target model for floating object recognition. This ensures recognition accuracy even when the amount of training data is insufficient, thereby improving the accuracy of floating object recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0017] Figure 1 is a flow chart of a floating object identification method provided in an embodiment of the present application;

[0018] Figure 2 is a flowchart of a target model training method provided in an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of an optional floating object identification method provided according to an embodiment of the present application;

[0020] Figure 4 1 is a schematic diagram of the structure of the recognition algorithm used by the floating object recognition device provided in an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of a floating object identification device provided according to an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] The present invention will be described below in conjunction with preferred implementation steps. Figure 1 is a flow chart of a floating object identification method according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0027] Step S101 : acquiring railway images collected by a monitoring device at each collection moment within a target time period to obtain a target railway image.

[0028] In step S101, a monitoring device, such as a camera, is installed near the railway. The camera is in communication with the floating object identification device. The camera captures railway images and transmits each frame of the railway image and the time of capture of each frame to the floating object identification device. The floating object identification device stores each frame of the railway image and the time of capture of each frame.

[0029] Step S102 : determining the meteorological data at each acquisition moment, and obtaining target meteorological data corresponding to each frame of the target railway image.

[0030] In step S102, the floating object identification device also communicates with the meteorological system. The device receives meteorological data from the meteorological system, counts the acquisition times of multiple frames of railway imagery, generates a target time period, encodes the target time period using a preset communication protocol, generates a data request, and sends the data request to the meteorological system. Upon receiving the data request, the meteorological system decodes the request, obtains the target time period, retrieves meteorological data for the target time period locally, and sends this data to the floating object identification device, thereby obtaining the target meteorological data corresponding to each frame of the target railway image.

[0031] In step S103, the target railway image and target meteorological data are input into a target model to obtain the floating object recognition result at each acquisition moment. The target model is trained by multiple sets of training samples, each set of training samples including historical railway images, historical meteorological data corresponding to the historical railway images, and historical floating object recognition results.

[0032] In step S103, the floating object recognition result may include the probability of the target railway image containing floating objects and the probability of the floating object belonging to a specific type of floating object. Floating object types may include leaves, plastic film, branches, straw, weeds, etc. The target model may be a convolutional neural network model. The convolutional neural network model is trained by collecting multiple sets of training samples. Each set of training samples may include historical railway images that may or may not contain floating objects. Historical meteorological data, which is weather information corresponding to the time the historical railway images were captured, includes but is not limited to wind speed, wind direction, and temperature. The historical floating object recognition result is a label, annotated by experts or previous models, indicating whether floating objects are present in the image. This can be a binary classification (presence or absence) or a multi-class classification (for example, identifying the specific type of floating object). The convolutional neural network model is trained using multiple sets of training samples to obtain a trained target model. The target railway image and target meteorological data are then input into the target model, which then outputs the floating object recognition result for the target railway image.

[0033] The floating object identification method provided in an embodiment of the present application obtains a target railway image by acquiring railway images collected by monitoring equipment at each acquisition moment within a target time period; determines meteorological data at each acquisition moment to obtain target meteorological data corresponding to each frame of the target railway image; and inputs the target railway image and target meteorological data into a target model to obtain a floating object identification result at each acquisition moment. The target model is trained using multiple sets of training samples, each set of training samples including historical railway images, historical meteorological data corresponding to the historical railway images, and historical floating object identification results. This method addresses the problem of poor floating object identification accuracy in related technologies. By combining railway images and meteorological data, multimodal data is input into the target model for floating object identification. This ensures identification accuracy even when the amount of training data is insufficient, thereby improving the accuracy of floating object identification.

[0034] In order to improve the accuracy of floating object identification, it is necessary to train a target model. Optionally, in the floating object identification method provided in the embodiment of the present application, the target model is obtained by: determining an initial convolution processing layer, an initial encoder and an initial decoder; obtaining a first training sample set, training the initial convolution processing layer based on the first training sample set to obtain a trained convolution processing layer, wherein the first training sample set includes historical railway images and historical floating object identification results; obtaining a second training sample set, training the initial encoder and the initial decoder based on the second training sample set to obtain a trained encoder and a trained decoder, wherein the second training sample set includes historical superimposed feature maps, historical coding features and historical floating object identification results; obtaining multiple groups of training samples, and jointly training the trained convolution processing layer, the trained encoder and the trained decoder based on the multiple groups of training samples to obtain the target model.

[0035] In some embodiments, the target model may include multiple layers of convolutional processing layers, encoders, and decoders. Figure 2 is a flow chart of a target model training method according to an embodiment of the present application, such as Figure 2 As shown, the training method includes:

[0036] First, a first training sample set is obtained, and the convolution processing layer is trained using the first training sample set to optimize the parameters in the convolution processing layer to obtain a trained convolution processing layer.

[0037] Specifically, a convolutional neural network is constructed based on the convolutional processing layer by adding a fully connected layer. A first training sample set is obtained, comprising historical railway images and historical floating object recognition results. The convolutional neural network is trained using the first training sample set, and the parameters of the convolutional processing layer and the fully connected layer are optimized to obtain a trained convolutional neural network. The number of convolutional processing layers and the parameters of each convolutional processing layer are obtained from the trained convolutional neural network, thereby obtaining a trained multi-layer convolutional processing layer.

[0038] By adding a fully connected layer to the convolutional processing layer, building a complete convolutional neural network, and training this network, the convolutional processing layer can learn effective representations of image features in an independent environment. This independent training method allows the convolutional processing layer to focus on extracting image features without being affected by other components (such as the encoder and decoder), thereby improving the accuracy and efficiency of feature extraction.

[0039] Then, a second training sample set is obtained, and the encoder and decoder are trained using the second training sample set to optimize parameters in the encoder and decoder to obtain a trained encoder and a trained decoder.

[0040] Specifically, a trained multi-layer convolutional processing layer is used as the input layer of the encoder. The image input layer is used to process historical railway images to obtain first training input features. This first training input feature is then fed into the trained multi-layer convolutional processing layer, outputting multiple training feature maps. Multiple training feature maps are concatenated to generate a second training input feature, namely a historical overlay feature map. A multi-stage encoder is used to encode the historical overlay feature map to obtain historical encoded features. A multi-stage decoder is used to decode the historical encoded features and output the probability of whether the historical railway image contains floating objects. A loss function is calculated based on the probability of whether the historical railway image contains floating objects and the historical floating object recognition results. This loss function is then used to optimize the parameters of the encoder and decoder. By continuously optimizing the parameters of the encoder and decoder, a trained encoder and decoder are obtained.

[0041] After obtaining the trained convolutional layer, it is directly used as the encoder input layer for subsequent training, eliminating the need for additional training of the encoder input layer. This effectively avoids repeated training, reduces training time, and improves overall training efficiency. Using the trained convolutional layer directly as the encoder input layer ensures the continuity of feature representation during training. This continuity helps the encoder better understand and utilize the features extracted by the convolutional layer, thereby improving the recognition performance of the entire model. Furthermore, the consistent interface between the convolutional layer and the encoder reduces the complexity of the model during training and deployment.

[0042] Finally, the trained convolutional layer, the trained encoder, and the trained decoder are jointly trained through multiple sets of training samples, and the fusion parameters are optimized according to the loss function to obtain the trained target model.

[0043] Specifically, the image input layer processes the historical railway images in each training sample to obtain the first training input features. This first training input feature is then fed into a trained multi-layer convolutional processing layer, outputting multiple training feature maps. Keyword extraction is performed on the historical meteorological data to obtain training meteorological keywords. A word vector conversion tool is used to convert the training meteorological keywords into training meteorological word vectors, ensuring that the dimensions of the training meteorological word vectors match those of the training feature maps. For each training feature map, the training meteorological word vectors and the training feature maps are superimposed using fusion parameters to obtain multiple superimposed training feature maps. These superimposed training feature maps are then concatenated to obtain the second training input features. The second training input features are then encoded using a trained multi-stage encoder to obtain encoded features. These encoded features are then decoded using a trained multi-stage decoder to output the probability of whether the historical railway images contain floating objects. A loss function is calculated based on the probability of whether the historical railway images contain floating objects and the historical floating object recognition results. The fusion parameters are then optimized using this loss function, and a trained target model is obtained through continuous optimization.

[0044] During the joint training phase, since the convolutional processing layer has been fully trained, the encoder and decoder can adapt more quickly and learn how to work with the convolutional processing layer. This collaborative work not only improves the recognition accuracy of the model, but also accelerates the convergence of the training process, further improving training efficiency.

[0045] This embodiment optimizes the fusion parameters by first training the convolution processing layer, encoder and decoder separately, and then jointly training the trained convolution layer, trained encoder and trained decoder. Compared with training the parameters of the convolution processing layer, encoder parameters, decoder parameters and fusion parameters at the same time, the step-by-step training method can reduce the number of parameters trained at the same time, reduce the model training time, simplify the training process, and reduce the complexity of model training. Since the step-by-step training strategy decomposes the training process into multiple stages, each stage only focuses on the optimization of some parameters, which makes the training objectives of each stage clearer and the training process more efficient. In the preliminary training stage, by training the convolution processing layer, encoder and decoder separately, these components can have good initial performance before joint training, thereby accelerating the convergence speed of the subsequent joint training stage and improving the overall training efficiency.

[0046] The step-by-step training strategy trains each component separately, enabling each component to acquire a certain level of feature representation capability when trained individually. During the joint training phase, these components can work together better, collectively improving the model's generalization capabilities. By optimizing fusion parameters, the meteorological word vectors and feature maps are better integrated during the overlay process, thereby improving the accuracy of floating object recognition. Faced with varying weather conditions and railway imagery, the model can more accurately identify floating objects, reducing false positives and missed negatives. The step-by-step training strategy makes the model training process more modular and flexible. This makes it easier to adjust and optimize when adding new components or improving existing ones. Furthermore, because each component is individually trained and tested, the model's maintainability is enhanced, making it easier to identify and resolve potential issues.

[0047] After the target model is trained, the target railway image and target meteorological data are input into the target model to obtain a floating object recognition result. Optionally, in the floating object recognition method provided in the embodiment of the present application, the target railway image and target meteorological data are input into the target model to obtain the floating object recognition result at each acquisition moment, including: extracting the three primary color pixel values ​​of the target railway image to obtain a three primary color pixel matrix; inputting the three primary color pixel matrix into the convolution processing layer of the target model for processing to obtain multiple feature maps; extracting keywords from the target meteorological data to obtain meteorological keywords, and converting the meteorological keywords into meteorological word vectors through a word vector conversion tool, wherein the meteorological word vectors have the same dimension as the feature maps; for each feature map, superimposing the meteorological word vectors and the feature maps through the fusion parameters of the target model to obtain a superimposed feature map; encoding the superimposed feature map through the encoder of the target model to obtain encoded features, and decoding the encoded features through the decoder of the target model to obtain floating object recognition results.

[0048] In some embodiments, a ternary color pixel matrix is ​​obtained by extracting the ternary color pixel values ​​of the target railway image. The ternary color pixel matrix is ​​used as the first input feature. The ternary color pixel matrix is ​​input into the convolution processing layer of the target model for processing. Each convolution processing layer includes a convolution layer and a pooling layer connected to the output end of the convolution layer. The convolution layer performs convolution processing on the input ternary color pixel matrix, and the pooling layer downsamples the convolution-processed data and outputs a feature map. Each convolution processing layer processes the feature map output by the previous convolution processing layer and outputs a feature map. Each convolution processing layer has multiple convolution kernels, and each convolution kernel processes the feature map output by the previous convolution processing layer, so that multiple features can be output. If the dimensions of the feature maps are different, the multiple feature maps can be unified to the same dimension through linear transformation.

[0049] During the image processing phase, the present embodiment utilizes multiple convolutional layers to process railway images and output multiple feature maps. These layers extract local features from the image, such as edges, corners, and textures, through convolution operations. The introduction of these layers enables the floating object recognition device to gradually extract richer and more abstract feature representations, further enhancing its feature extraction capabilities. This results in greater sensitivity and specificity for floating object features, thereby improving recognition accuracy.

[0050] The floating object identification device uses a preset keyword extraction method to extract keywords from the target meteorological data to obtain meteorological keywords. The meteorological keywords are then converted and processed using a word vector conversion tool to output meteorological word vectors. When setting the parameters of the word vector conversion tool, the parameters can be set based on the size of the feature map to ensure that the dimension of the output meteorological word vector is the same as the dimension of the feature map. The meteorological word vector and the feature map are matrices of the same dimension. After processing the meteorological word vector and each feature map using the fusion parameter, the processed feature map and the processed meteorological word vector are matrix-added to obtain a superimposed feature map. The superimposed feature maps are then connected to obtain the second input feature.

[0051] The present embodiment extracts keywords and converts word vectors from meteorological data to obtain meteorological word vectors, which are then fused with feature maps. The introduction of meteorological word vectors enables the floating object identification device to consider the impact of meteorological factors on floating object identification, thereby improving the robustness of identification. Especially in complex and changeable meteorological conditions, such as strong winds, rain, and snow, the fusion of meteorological word vectors with feature maps can significantly enhance the floating object identification device's ability to identify floating objects, reducing false positives and missed positives.

[0052] The second input feature is input to the multi-stage encoder of the target model, and the encoder encodes the input data and outputs it. Among them, the first-stage encoder encodes the second input feature, the second-stage encoder encodes the encoded data output by the first-stage encoder, the third-stage encoder encodes the encoded data output by the second-stage encoder, and so on. Each encoder includes a first multi-head attention layer, a first residual normalization layer, a first fully connected layer, and a second residual normalization layer. The input and output of the first multi-head attention layer are both connected to the first residual normalization layer, the output of the first residual normalization layer is connected to the input of the first fully connected layer, and the output and input of the first fully connected layer are both connected to the second residual normalization layer.

[0053] For the first encoder layer, the first multi-head attention layer is used to process the second input features to obtain a first multi-head attention matrix. The first residual normalization layer includes a residual layer and a normalization layer. The first residual normalization layer is used to process the second input features and the first multi-head attention matrix to output a first residual normalization matrix. The input end of the first fully connected layer receives the first residual normalization matrix, and the first fully connected layer processes the first residual normalization matrix to output the first fully connected layer matrix. The input end of the second residual normalization layer receives the first residual normalization matrix and the first fully connected matrix, and processes the first residual normalization matrix and the first fully connected matrix to output the first encoding matrix.

[0054] For other encoders, the first multi-head attention layer processes the encoding matrix output by the previous encoder to obtain a second multi-head attention matrix. The first residual normalization layer processes the encoding matrix output by the previous encoder and the second multi-head attention matrix to output a second residual normalization matrix. The first fully connected layer receives the second residual normalization matrix, processes the second residual normalization matrix, and outputs a second fully connected matrix. The second residual normalization layer receives the second residual normalization matrix and the second fully connected matrix, processes the second residual normalization matrix and the second fully connected matrix, and outputs an encoding matrix.

[0055] Because the identification of floating objects on railways is highly dependent on meteorological conditions (such as wind speed and direction), the encoder utilizes a multi-head attention mechanism to more effectively integrate meteorological data with image features, thereby improving recognition accuracy. This feature fusion capability enables the model to make more accurate judgments in complex and changing railway environments. The encoder consists of a first multi-head attention layer, a first residual normalization layer, a first fully connected layer, and a second residual normalization layer. By incorporating meteorological data into the second input feature, the first multi-head attention layer focuses on its impact on each feature map. The first fully connected layer then processes the feature maps, integrating them and taking into account both local and global features, preserving information integrity. During the encoding process, the introduction of the first and second residual normalization layers effectively prevents information loss. By adding residual connections, the residual normalization layer ensures that each layer's output incorporates information from the previous layer, helping the model retain important features of the original input during deep encoding. Maintaining this information integrity improves floating object recognition accuracy. The use of residual normalization layers not only helps preserve information but also improves model training efficiency. By introducing residual connections, the model can more easily find the optimal solution during training, thereby accelerating convergence, significantly reducing training time and lowering computational costs.

[0056] Each decoder decodes the encoded data from the last encoder and the previous decoder, ultimately outputting the floating object recognition result. The first-stage decoder decodes the encoded data from the last encoder, and the second-stage decoder decodes the decoded data from the first-stage decoder based on the encoded data from the last encoder. This continues in this order until the last decoder decodes the decoded data from the penultimate decoder based on the encoded data from the last encoder. Furthermore, the decoders are connected to a Softmax (activation) function to output the probability of whether the target railway image contains floating objects.

[0057] For example, each decoder includes a second multi-head attention layer, a third residual normalization layer, a second fully connected layer, and a fourth residual normalization layer. The second multi-head attention layer receives the last encoding matrix, and the second multi-head attention layer receives the previous decoding matrix. The output of the second multi-head attention layer is connected to the input of the third residual normalization layer, and the input of the third residual normalization layer is also connected to the output of the previous decoder. The output of the third residual normalization layer is connected to the input of the second fully connected layer, and the output and input of the second fully connected layer are connected to the input of the fourth residual normalization layer.

[0058] The second multi-head attention layer receives the last encoding matrix and the decoding matrix output by the previous decoder, processes the last encoding matrix and the decoding matrix output by the previous decoder, and outputs a second multi-head attention matrix. The third residual normalization layer receives the decoding matrix output by the previous decoder and the second multi-head attention matrix, processes the third residual normalization matrix and the second multi-head attention matrix, and outputs a third residual normalization matrix. If the decoder is a first-level decoder, the second multi-head attention layer receives the last encoding matrix, processes the last encoding matrix, and outputs a second multi-head attention matrix. The second fully connected layer receives the third residual normalization matrix, processes the third residual normalization matrix, and outputs a third fully connected matrix. The fourth residual normalization layer receives the third fully connected matrix and the third residual normalization matrix, processes the third fully connected matrix and the third residual normalization matrix, and outputs a decoding matrix.

[0059] The decoder includes a second multi-head attention layer, a third residual normalization layer, a second fully connected layer, and a fourth residual normalization layer. The second multi-head attention layer processes the previous decoding matrix and the final encoding matrix, effectively preserving information during decoding and improving decoding accuracy. The second multi-head attention layer in the decoder simultaneously considers the decryption matrix output by the previous decoder and the final encoding matrix. This design enables more effective information preservation during the decoding process. The multi-head attention mechanism captures dependencies between different positions, allowing for more accurate restoration of the original information during decoding. For railway floating object identification, this improved information preservation helps ensure the accuracy of decoding results and reduce false positives and missed detections. The introduction of the third and fourth residual normalization layers, by adding residual connections, effectively prevents information loss during the decoding process. Residual connections ensure that each decoder layer output incorporates information from the previous layer, helping to maintain information integrity during decoding.

[0060] The use of the residual normalization layer not only helps preserve information but also improves the decoder's computational efficiency. By introducing residual connections, the decoder can more quickly find the optimal solution during decoding, thereby accelerating the computational process. This is particularly important for processing large-scale railway image data and meteorological data, significantly reducing decoding time and improving the overall system's responsiveness. The second fully connected layer in the decoder is responsible for mapping the third residual matrix to the output space, achieving nonlinear transformation of features. This fully connected layer design helps the model capture complex nonlinear relationships and improves its robustness. For railway floating object identification, the shape and position of floating objects may vary depending on meteorological conditions and environmental factors. Through the processing of the fully connected layer, the decoder can better adapt to such changes and achieve accurate decoding.

[0061] During the feature encoding and decoding stages, the embodiments of the present application utilize a multi-level codec to process the fused features. The encoder encodes the input features through a multi-level attention mechanism, extracting more advanced and abstract feature representations; the decoder decodes the encoded features through a multi-level attention mechanism and outputs the floating object recognition results. The introduction of multi-level encoders and decoders enables the floating object recognition device to better handle complex feature representations, improving recognition accuracy and efficiency. At the same time, by training the codec and optimizing its parameters and fusion parameters, the floating object recognition device can maintain stable recognition performance in different scenarios.

[0062] This embodiment of the application effectively overcomes the shortcomings of single-modal data by integrating meteorological data with railway imagery through multimodal fusion. Meteorological data, such as wind speed, direction, and temperature, significantly influences the motion and morphology of floating objects. By integrating meteorological data, the floating object identification device can more comprehensively understand the characteristics and state of floating objects, thereby maintaining high recognition accuracy even when model training data is insufficient.

[0063] In order to more accurately identify whether the railway image contains floating objects, the meteorological word vector and the feature map are superimposed by fusion parameters. Optionally, in the floating object identification method provided in the embodiment of the present application, the fusion parameters include meteorological word vector coefficients and feature map coefficients. The meteorological word vector and the feature map are superimposed by the fusion parameters of the target model to obtain the superimposed feature map, including: determining each first element in the meteorological word vector, multiplying the meteorological word vector coefficient by each first element to obtain an intermediate word vector; determining each second element in the feature map, multiplying the feature map coefficient by each second element to obtain an intermediate feature map; and summing the intermediate word vector and the intermediate feature map to obtain the superimposed feature map.

[0064] In some embodiments, the meteorological word vector and the feature map are matrices of the same dimension, and the fusion parameters may include the meteorological word vector coefficient Kq and the feature map coefficient Kt. The meteorological word vector coefficient is multiplied by each first element in the meteorological word vector to obtain an intermediate word vector. The meteorological word vector is a matrix Q of size m×n, qij represents any element in the meteorological word vector, and the intermediate word vector Q1 is Kq×qij, 1≤i≤m, 1≤j≤n. i, j, m and n are all positive integers. The intermediate feature map is obtained by multiplying each element in the feature map by the feature map coefficient. The feature map is a matrix T of size m×n, tij represents any element in the feature map, and the intermediate feature map T1 is Kt×tij. The sum of the intermediate word vector and the intermediate feature map is calculated according to the following formula to obtain the superimposed feature map T2:

[0065] t2ij=Kt×tij+Kq×qij;

[0066] Among them, t2ij is the element in the i-th row and j-th column of the superimposed feature map T2.

[0067] For example, there is a 5×4 meteorological word vector and 10 5×4 feature maps. For each 5×4 feature map, the 5×4 meteorological word vector and the 5×4 feature map are superimposed using the fusion parameter to obtain 10 5×4 superimposed feature maps. By connecting the 10 5×4 superimposed feature maps, a 50×4 second input feature map can be obtained. The second input feature can be represented as:

[0068] ;

[0069] Among them, T20 is the first 5×4 superimposed feature map matrix, T21 is the second 5×4 superimposed feature map matrix, T28 is the ninth 5×4 superimposed feature map matrix, and T29 is the tenth 5×4 superimposed feature map matrix.

[0070] In this embodiment, the meteorological word vector coefficient is multiplied by each element in the meteorological word vector, so that the meteorological word vectors share a coefficient and the information in the meteorological word vector is completely retained. Correspondingly, the feature map coefficient is multiplied by each element in the feature map, so that the feature map shares a coefficient and the information in the feature map is completely retained. The intermediate word vector and the intermediate feature map are then summed to obtain the superimposed feature map, thereby adding meteorological data to each feature map. In addition, when fusing the meteorological word vector and the feature map, only two parameters are required, which can reduce the model training process, thereby reducing the complexity of model training and improving training efficiency. The embodiment of the present application achieves an effective fusion of the meteorological word vector and the feature map by introducing the meteorological word vector coefficient and the feature map coefficient. This fusion method not only retains the information of the meteorological word vector and the feature map, but also, through the adjustment of the coefficients, enables the two to complement each other during the superposition process, thereby jointly improving the accuracy of recognition. Since meteorological data has a significant impact on the morphology and motion state of floating objects, integrating meteorological data into the recognition process can significantly improve the robustness and adaptability of recognition. Especially under complex and changeable meteorological conditions, such as strong winds, rain and snow, the fusion of meteorological word vectors and feature maps can enable floating object recognition devices to better respond to environmental changes, reduce false alarms and missed alarms, and improve recognition accuracy.

[0071] In order to improve image processing efficiency, the difference pixel area can be extracted before the image is input into the target model. Optionally, in the floating object identification method provided in the embodiment of the present application, the railway image collected by the monitoring device at each collection moment within the target time period is obtained, and the target railway image is obtained, including: receiving the railway image collected by the monitoring device at the collection moment; comparing the railway image with a preset background image to obtain the difference pixel area, wherein the background image is an image without floating objects collected by the monitoring device at the same shooting angle as the railway image; and determining the image corresponding to the difference pixel area as the target railway image.

[0072] In some embodiments, in railway images captured by cameras installed near the railway, most objects are background. By comparing a reference image with the railway image, objects in newly entered images can be extracted and identified, reducing data processing. The background image can be a pre-captured railway image captured from the same angle as the railway image, facilitating comparison. Each pixel in the railway image is compared with the corresponding pixel in the background image. If they are identical, the next pixel is selected for comparison. If they are different, the pixel position is saved. By traversing each pixel in the railway image, a region of differing pixels is obtained.

[0073] Based on the location information of the difference pixel area, the pixel values ​​of the difference pixel area are extracted from the railway image. The pixel areas outside the difference pixel area are filled with the supplementary pixel values ​​to obtain the target railway image. For example, if the railway image is 5×5 in size, the four vertices of the rectangular area are the pixels in the first row and first column, the pixels in the third row and first column, the pixels in the first row and third column, and the pixels in the third row and third column as the four vertices of the difference pixel area. The pixel values ​​within this rectangular area are extracted, and the pixels in the fourth and fifth rows are filled with the supplementary pixel values, and the pixels in the fourth and fifth columns are filled with the supplementary pixel values ​​to obtain the target railway image.

[0074] This embodiment significantly reduces the amount of data that needs to be processed by comparing the railway image with the background image and extracting only the difference pixel areas for processing. This makes the subsequent image feature extraction and recognition process more efficient, improving overall processing efficiency. The difference pixel area contains objects that have newly entered the railway image. These objects are key information that the floating object recognition device needs to pay attention to. By comparing with the background image, this key information can be accurately located and extracted, avoiding interference from the background area. This enables the floating object recognition device to more accurately identify target objects such as floating objects in the railway image, improving the accuracy and reliability of recognition. The environment along the railway is complex and changeable, and the background image may change due to factors such as weather and lighting. By comparing the difference pixel areas, the floating object recognition device can automatically adapt to these changes and only focus on objects that have newly entered the image. This preprocessing strategy enhances the robustness of the system, enabling it to maintain stable recognition performance in complex and changing environments.

[0075] After obtaining the floating object identification result, the floating object can be adjusted based on the floating object identification result. Optionally, in the floating object identification method provided in the embodiment of the present application, after obtaining the floating object identification result at each acquisition moment, the method further includes: when the floating object identification result indicates that there is a floating object in the railway image, determining the railway image as the initial image in which the floating object appears; determining multiple frames of railway images within a preset time period acquired after the initial image, and determining the movement trajectory of the floating object based on the multiple frames of railway images, and determining the predicted movement trajectory of the floating object within the target time period based on the movement trajectory of the floating object; determining the location information of the railway power grid coverage area, and judging whether the floating object falls into the coverage area of ​​the railway power grid based on the location information and the predicted movement trajectory; and when it is determined that the floating object will fall into the coverage area of ​​the railway power grid, controlling the blowing device to adjust the movement trajectory of the floating object.

[0076] In some embodiments, after determining that there are floating objects in the railway image, the floating object identification device obtains the identification results of multiple frames of railway images collected within a preset time period after the initial image, and obtains the actual position of the floating object at each collection moment based on the identification results of each frame of railway image, thereby determining the movement trajectory of the floating object.

[0077] For example, if the decoder outputs a decoding result indicating the presence of a floating object in a particular railway image frame, the floating object identification device obtains the identification results of multiple railway image frames captured after the previous frame and extracts the floating object's position within the image from the identification results of each frame. The floating object's position within the image is then converted using the camera's internal and external parameters to determine the object's actual position at each capture moment. Based on the object's actual position at each capture moment, the device calculates the object's velocity at each capture moment. The velocity at each capture moment is used to predict the object's velocity at future times. This velocity is then used to determine the object's position at future times, thereby determining the object's predicted trajectory within the target time period.

[0078] A blower is installed near the railway power grid. Based on the grid's installation data, the system determines the location of the grid's coverage area. Based on this location information and the predicted trajectory of the floating object, it determines whether the object's trajectory overlaps with the grid's coverage area. If so, the object is likely to land on the grid. When the object passes through the blower, the system controls its operation to alter the object's trajectory and prevent it from landing on the grid.

[0079] This embodiment, after identifying a floating object in railway imagery, controls a camera to continuously capture multiple frames of railway imagery, extracting the object's location within each frame. Based on the object's location and the time each frame was captured, the object's trajectory is predicted to determine whether the object will land on the railway power grid. If the object is likely to land on the power grid, the air blower is controlled to alter the object's trajectory to prevent it from landing on the grid and potentially damaging railway facilities. Based on the recognition results and capture time information from multiple frames of railway imagery, the trajectory of the floating object is predicted in a data-driven manner. This allows for more accurate assessment of the object's movement trends and potential impact range. This embodiment can eliminate hidden dangers while railway facilities are operating normally, ensuring robust operation. By predicting the trajectory of floating objects in real time, potential hazards can be identified before the object approaches the power grid, providing a valuable window of opportunity for subsequent proactive intervention. This embodiment, through automated processing, reduces the need for manual intervention and improves processing efficiency. Furthermore, by implementing intervention at the earliest possible stage of a potential hazard, it prevents equipment damage or the escalation of safety hazards caused by objects remaining attached to the power grid for extended periods.

[0080] After determining that the floating object will fall into the coverage area of ​​the railway power grid, the movement trajectory of the floating object is changed by a blowing device. Optionally, in the floating object identification method provided in the embodiment of the present application, controlling the blowing device to adjust the movement trajectory of the floating object includes: determining the position information of the blowing device, and determining the opening time and blowing parameters of the blowing device based on the position information of the blowing device and the predicted movement trajectory, wherein the blowing parameters include at least one of the following: wind speed level and wind direction; controlling the blowing device to operate according to the blowing parameters at the opening time to adjust the movement trajectory of the floating object.

[0081] In some embodiments, if it is determined that the floating object will fall onto the railway power grid, the start-up time and blowing parameters of the blowing device are determined based on the trajectory of the floating object and the installation position of the blowing device; the blowing parameters include wind force and wind direction. The distance between each trajectory point of the floating object and the installation position of the blowing device is calculated, and the time corresponding to the trajectory point with the shortest distance is selected as the start-up time of the blowing device. The wind force and wind direction are calculated based on the speed of the floating object at the trajectory point with the shortest distance and the distance between the trajectory point and the railway power grid, so that the floating object will not fall onto the railway power grid. For example: the blowing device provides upward buoyancy to allow the floating object to pass over the railway power grid. The size of the upward buoyancy is related to the wind speed of the blowing device. The wind speed of the blowing device can be determined based on the speed of the floating object in the vertical direction.

[0082] When this embodiment determines that a floating object is likely to fall onto the railway power grid, it estimates the optimal activation time and optimal blowing parameters based on the object's trajectory, meteorological data, and the installation location of the blower. This ensures that the object's trajectory is altered with minimal energy consumption, reducing the operating costs of the blower and saving energy. After determining that a floating object is at risk of falling onto the railway power grid, this embodiment can determine the optimal activation time for the blower. The blower can be activated at a critical moment before the floating object approaches the power grid, effectively changing its trajectory and ensuring the safe operation of railway facilities. This prevents potential damage to the railway power grid caused by floating objects and ensures the normal operation of trains.

[0083] Meteorological data plays a crucial role in the calculation of air blowing parameters. By factoring in real-time weather conditions, this embodiment enables more flexible control of the air blowing device, adapting to varying environmental conditions. For example, in strong winds, the air blowing parameters can be automatically adjusted to offset the effects of external wind forces, ensuring accurate trajectory tracking of floating objects. This adaptability enables the system to maintain efficient operation in a variety of complex environments.

[0084] According to another embodiment of the present application, an optional floating object identification method is also provided. Figure 3 FIG. 1 is a schematic diagram of an optional floating object identification method provided in accordance with an embodiment of the present application. Figure 3 As shown, the method includes:

[0085] S301. The floating object recognition device obtains a railway image captured by a camera and the time at which the railway image was captured.

[0086] The camera is set up near the railway to capture railway images and transmit each railway image frame and the acquisition time of each railway image frame to the floating object recognition device. The floating object recognition device stores each railway image frame and the acquisition time of each railway image frame.

[0087] S302: The floating object identification device generates a data request according to the acquisition time of the railway image, and sends the data request to the meteorological system, so that the meteorological system sends the meteorological data at the acquisition time to the floating object identification device.

[0088] The floating object identification device counts the acquisition times of multiple frames of railway images to generate an observation time period. It then encodes the observation time period using a preset communication protocol, generates a data request, and sends the data request to the meteorological system. Upon receiving the data request, the meteorological system decodes it, obtains the observation time period, retrieves meteorological data for that period locally, and transmits the data to the floating object identification device.

[0089] S303. The floating object recognition device uses the image input layer to process the railway image to obtain a first input feature, inputs the first input feature to a multi-layer convolution processing layer, and outputs multiple feature maps.

[0090] In this process, a ternary color pixel matrix is ​​obtained by extracting the ternary color pixel values ​​of the railway image. The ternary color pixel matrix is ​​used as the first input feature. Each convolution processing layer includes a convolution layer and a pooling layer connected to the output end of the convolution layer. The convolution layer performs convolution processing on the input matrix, and the pooling layer downsamples the convolution-processed data and outputs a feature map. Each convolution processing layer processes the feature map output by the previous convolution processing layer and outputs a feature map. Each convolution processing layer has multiple convolution kernels, and each convolution kernel processes the feature map output by the previous convolution processing layer, so that multiple features can be output. If the dimensions of the feature maps are different, multiple feature maps can be unified to the same dimension through linear transformation.

[0091] For example, Figure 4 Schematic diagram of the structure of the recognition algorithm used by the floating object recognition device according to the embodiment of the present application. Figure 4 As shown in the figure, the recognition algorithm includes: multi-layer convolutional processing layers, word embedding tools, encoders, and decoders. The multi-layer convolutional processing layers process railway images to generate feature maps, and the word embedding tools process meteorological data to generate meteorological word embeddings. The feature maps and meteorological word embeddings are superimposed using fusion parameters to generate superimposed features. The encoder encodes these superimposed features to generate encoded features, and the decoder decodes the encoded features to generate decoded features. Finally, the Softmax function transforms the decoded features into the probability of whether the object is a floating object.

[0092] S304. The floating object identification device extracts keywords from the meteorological data to obtain meteorological keywords, and uses a word vector conversion tool to convert the meteorological keywords into meteorological word vectors so that the dimension of the meteorological word vector is the same as the dimension of the feature map.

[0093] The floating object identification device uses existing keyword extraction methods to extract keywords from meteorological data, obtaining meteorological keywords. It then uses existing word vector conversion tools to convert these meteorological keywords and output meteorological word vectors. When setting parameters for the word vector conversion tool, these parameters can be based on the feature map size, ensuring that the dimensions of the output meteorological word vectors match those of the feature map.

[0094] S305: For each feature map, use the fusion parameter to superimpose the meteorological word vector and the feature map to obtain multiple superimposed feature maps, and connect the multiple superimposed feature maps to obtain the second input feature.

[0095] The meteorological word vector and feature map are matrices of the same dimension. After processing the meteorological word vector and each feature map using the fusion parameter, the processed feature map and the processed meteorological word vector are matrix-added to obtain a superimposed feature map. The superimposed feature maps are then concatenated to obtain the second input feature. For example, if there is a 5×4 meteorological word vector and 10 5×4 feature maps, the fusion parameter is used to superimpose the 5×4 meteorological word vector and the 5×4 feature map for each 5×4 feature map to obtain 10 superimposed 5×4 feature maps. By concatenating these 10 superimposed 5×4 feature maps, a 50×4 second input feature map is obtained.

[0096] The second input feature representation is:

[0097] ;

[0098] Among them, T20 is the first 5×4 superimposed feature map matrix, T21 is the second 5×4 superimposed feature map matrix, T28 is the ninth 5×4 superimposed feature map matrix, and T29 is the tenth 5×4 superimposed feature map matrix.

[0099] S306: Use a multi-stage encoder to encode the second input feature to obtain an encoded feature, use a multi-stage decoder to decode the encoded feature, and output whether the target is a light object type.

[0100] After obtaining the second input feature through the above method, the second input feature is input into a multi-stage encoder, which encodes the input data and outputs it. More specifically, the first-stage encoder encodes the second input feature, the second-stage encoder encodes the encoded data output by the first-stage encoder, the third-stage encoder encodes the encoded data output by the second-stage encoder, and so on. Each decoder decodes the encoded data output by the last encoder and the output data of the previous decoder, and outputs it, ultimately outputting the target object type. More specifically, the first-stage decoder decodes the encoded data output by the last encoder, the second-stage decoder decodes the decoded data output by the first-stage decoder based on the encoded data output by the last encoder, and so on, until the last decoder decodes the decoded data output by the penultimate decoder based on the encoded data output by the last encoder. Furthermore, the decoder needs to be connected to a Softmax function to output the probability of whether the target is a light object.

[0101] In this embodiment, a floating object recognition device receives railway images sent by a camera. After receiving the railway images, a data request is sent to a meteorological system to obtain meteorological data issued by the meteorological system. The floating object recognition device processes the railway images using multiple convolutional layers to obtain multiple feature maps, processes the meteorological data using keyword extraction and word vector conversion to obtain meteorological word vectors, processes the meteorological word vectors and feature maps using fusion parameters, realizes the combination of meteorological word vectors and various feature maps, and then uses a multi-attention mechanism for encoding and decoding, so that the attention mechanism can consider the impact of meteorological data on the recognition of feature maps, realizes floating object recognition based on multimodal input of meteorological data and railway images, and improves the accuracy of floating object recognition.

[0102] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0103] The embodiment of the present application further provides a floating object identification device. It should be noted that the floating object identification device of the embodiment of the present application can be used to execute the floating object identification method provided in the embodiment of the present application. The floating object identification device provided in the embodiment of the present application is introduced below.

[0104] Figure 5 Schematic diagram of a floating object identification device according to an embodiment of the present application. Figure 5 As shown, the device includes:

[0105] An acquisition unit 501 is configured to acquire a railway image acquired by a monitoring device at each acquisition moment within a target time period to obtain a target railway image;

[0106] The first determining unit 502 is configured to determine the meteorological data at each acquisition moment and obtain target meteorological data corresponding to each frame of the target railway image;

[0107] Input unit 503 is used to input the target railway image and target meteorological data into the target model to obtain the floating object recognition result at each acquisition time. The target model is trained by multiple sets of training samples, each set of training samples including historical railway images, historical meteorological data corresponding to the historical railway images, and historical floating object recognition results.

[0108] The floating object identification device provided in the embodiment of the present application obtains railway images collected by the monitoring equipment at each collection moment within a target time period through an acquisition unit 501 to obtain a target railway image. A first determination unit 502 determines the meteorological data at each collection moment to obtain target meteorological data corresponding to each frame of the target railway image. An input unit 503 inputs the target railway image and the target meteorological data into a target model to obtain a floating object identification result at each collection moment. The target model is trained using multiple sets of training samples, each set of training samples including historical railway images, historical meteorological data corresponding to the historical railway images, and historical floating object identification results. This solves the problem of poor floating object identification accuracy in related technologies. By combining railway images and meteorological data, multimodal data is input into the target model for floating object identification. Even when the amount of training data is insufficient, identification accuracy can still be guaranteed, thereby achieving the effect of improving the accuracy of floating object identification.

[0109] Optionally, in the floating object identification device provided in the embodiment of the present application, the device further includes: a second determination unit, used to determine an initial convolution processing layer, an initial encoder, and an initial decoder; a first training unit, used to obtain a first training sample set, and train the initial convolution processing layer based on the first training sample set to obtain a trained convolution processing layer, wherein the first training sample set includes historical railway images and historical floating object identification results; a second training unit, used to obtain a second training sample set, and train the initial encoder and the initial decoder based on the second training sample set to obtain a trained encoder and a trained decoder, wherein the second training sample set includes historical superimposed feature maps, historical coding features, and historical floating object identification results; a third training unit, used to obtain multiple groups of training samples, and jointly train the trained convolution processing layer, the trained encoder, and the trained decoder based on the multiple groups of training samples to obtain a target model.

[0110] Optionally, in the floating object identification device provided in the embodiment of the present application, the input unit 503 includes: a first extraction module, used to extract the three primary color pixel values ​​of the target railway image to obtain a three primary color pixel matrix; an input module, used to input the three primary color pixel matrix into the convolution processing layer of the target model for processing to obtain multiple feature maps; a second extraction module, used to extract keywords from the target meteorological data to obtain meteorological keywords, and convert the meteorological keywords into meteorological word vectors through a word vector conversion tool, wherein the meteorological word vector has the same dimension as the feature map; a superposition module, used to superimpose the meteorological word vector and the feature map for each feature map through the fusion parameters of the target model to obtain a superimposed feature map; an encoding and decoding module, used to encode the superimposed feature map through the encoder of the target model to obtain an encoded feature, and decode the encoded feature through the decoder of the target model to obtain a floating object identification result.

[0111] Optionally, in the floating object identification device provided in the embodiment of the present application, the superposition module includes: a first determination submodule, used to determine each first element in the meteorological word vector, multiply the meteorological word vector coefficient by each first element, and obtain an intermediate word vector; a second determination submodule, used to determine each second element in the feature map, multiply the feature map coefficient by each second element, and obtain an intermediate feature map; a summation submodule, used to sum the intermediate word vector and the intermediate feature map to obtain a superimposed feature map.

[0112] Optionally, in the floating object identification device provided in the embodiment of the present application, the acquisition unit 501 includes: a receiving module, used to receive the railway image collected by the monitoring equipment at the collection time; a comparison module, used to compare the railway image with a preset background image to obtain a difference pixel area, wherein the background image is an image without floating objects collected by the monitoring equipment at the same shooting angle as the railway image; a first determination module, used to determine the image corresponding to the difference pixel area as the target railway image.

[0113] Optionally, in the floating object identification device provided in the embodiment of the present application, the device further includes: a third determination unit, for determining the railway image as the initial image in which the floating object appears when the floating object identification result indicates that the floating object exists in the railway image; a fourth determination unit, for determining multiple frames of railway images within a preset time period collected after the initial image, and determining the movement trajectory of the floating object based on the multiple frames of railway images, and determining the predicted movement trajectory of the floating object within the target time period based on the movement trajectory of the floating object; a fifth determination unit, for determining the location information of the railway power grid coverage area, and judging whether the floating object falls into the coverage area of ​​the railway power grid based on the location information and the predicted movement trajectory; and a control unit, for controlling the blowing device to adjust the movement trajectory of the floating object when it is determined that the floating object will fall into the coverage area of ​​the railway power grid.

[0114] Optionally, in the floating object identification device provided in the embodiment of the present application, the control unit includes: a second determination module, used to determine the position information of the blowing device, and determine the opening time and blowing parameters of the blowing device based on the position information of the blowing device and the predicted motion trajectory, wherein the blowing parameters include at least one of the following: wind speed level and wind direction; a control module, used to control the blowing device to operate according to the blowing parameters at the opening time to adjust the motion trajectory of the floating object.

[0115] The floating object identification device includes a processor and a memory. The acquisition unit 501, the first determination unit 502 and the input unit 503 are all stored in the memory as program units. The processor executes the program units stored in the memory to implement corresponding functions.

[0116] The processor contains a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the accuracy of floating object recognition can be improved by adjusting the kernel parameters.

[0117] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0118] An embodiment of the present invention provides a computer-readable storage medium storing a program, which implements a floating object identification method when executed by a processor.

[0119] An embodiment of the present invention provides a processor, which is used to run a program, wherein the program executes a floating object recognition method when running.

[0120] Figure 6 Schematic diagram of an electronic device according to an embodiment of the present application. Figure 6 As shown, electronic device 601 includes a processor, memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are performed: obtaining railway images captured by the monitoring device at each acquisition time within a target time period to obtain a target railway image; determining meteorological data at each acquisition time to obtain target meteorological data corresponding to each frame of the target railway image; and inputting the target railway image and target meteorological data into a target model to obtain a floating object recognition result at each acquisition time. The target model is trained using multiple sets of training samples, each set of training samples including historical railway images, historical meteorological data corresponding to the historical railway images, and historical floating object recognition results. The device herein may be a server, a PC, a PAD, a mobile phone, or the like.

[0121] The present application also provides a computer program product which, when executed on a data processing device, is adapted to execute a program that initializes the following method steps: obtaining railway images collected by a monitoring device at each collection moment within a target time period to obtain a target railway image; determining meteorological data at each collection moment to obtain target meteorological data corresponding to each frame of the target railway image; and inputting the target railway image and the target meteorological data into a target model to obtain a floating object recognition result at each collection moment, wherein the target model is trained using multiple sets of training samples, each set of training samples including historical railway images, historical meteorological data corresponding to the historical railway images, and historical floating object recognition results.

[0122] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0126] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0127] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0128] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0129] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0130] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A floating object identification method, characterized in that: include: Acquire railway images collected by the monitoring equipment at each collection moment within a target time period to obtain a target railway image; Determining the meteorological data at each acquisition moment to obtain target meteorological data corresponding to each frame of the target railway image; Inputting the target railway image and the target meteorological data into a target model to obtain a floating object recognition result at each acquisition moment, wherein the target model is trained by multiple sets of training samples, each set of training samples including a historical railway image, historical meteorological data corresponding to the historical railway image, and historical floating object recognition results; The method of inputting the target railway image and the target meteorological data into a target model to obtain a floating object recognition result at each acquisition moment includes: extracting the three primary color pixel values ​​of the target railway image to obtain a three primary color pixel matrix; inputting the three primary color pixel matrix into the convolution processing layer of the target model for processing to obtain multiple feature maps; extracting keywords from the target meteorological data to obtain meteorological keywords, and converting the meteorological keywords into meteorological word vectors using a word vector conversion tool, wherein the meteorological word vectors have the same dimension as the feature maps; for each feature map, superimposing the meteorological word vector and the feature map using the fusion parameters of the target model to obtain a superimposed feature map; encoding the superimposed feature map using the encoder of the target model to obtain encoded features, and decoding the encoded features using the decoder of the target model to obtain the floating object recognition result.

2. The method according to claim 1, characterized in that The target model is obtained in the following way: Determine the initial convolutional processing layer, initial encoder and initial decoder; Obtaining a first training sample set, and training the initial convolution processing layer based on the first training sample set to obtain a trained convolution processing layer, wherein the first training sample set includes historical railway images and historical floating object recognition results; Obtaining a second training sample set, and training the initial encoder and the initial decoder based on the second training sample set to obtain a trained encoder and a trained decoder, wherein the second training sample set includes historical superimposed feature maps, historical coding features, and historical floating object recognition results; The multiple groups of training samples are obtained, and the trained convolution processing layer, the trained encoder, and the trained decoder are jointly trained based on the multiple groups of training samples to obtain the target model.

3. The method according to claim 1, characterized in that The fusion parameters include meteorological word vector coefficients and feature map coefficients. The meteorological word vector and the feature map are superimposed using the fusion parameters of the target model to obtain a superimposed feature map including: Determine each first element in the meteorological word vector, and multiply each first element by the meteorological word vector coefficient to obtain an intermediate word vector; Determine each second element in the feature map, and multiply each second element by the feature map coefficient to obtain an intermediate feature map; The intermediate word vector and the intermediate feature map are summed to obtain the superimposed feature map.

4. The method according to claim 1, wherein Obtain railway images collected by the monitoring equipment at each collection time within the target time period. The target railway images include: receiving the railway image captured by the monitoring device at the capture time; Comparing the railway image with a preset background image to obtain a difference pixel area, wherein the background image is an image without floating objects captured by the monitoring device at the same shooting angle as the railway image; The image corresponding to the difference pixel area is determined as the target railway image.

5. The method according to claim 1, characterized in that After obtaining the floating object identification result at each collection moment, the method further includes: If the floating object recognition result indicates that the floating object exists in the railway image, determining the railway image as an initial image in which the floating object appears; determining a plurality of railway image frames acquired within a preset time period after the initial image, determining a motion trajectory of the floating object based on the plurality of railway image frames, and determining a predicted motion trajectory of the floating object within a target time period based on the motion trajectory of the floating object; determining location information of an area covered by a railway power grid, and determining whether the floating object falls within the area covered by the railway power grid based on the location information and the predicted motion trajectory; When it is determined that the floating object will fall into the coverage area of ​​the railway power grid, the blowing device is controlled to adjust the movement trajectory of the floating object.

6. The method according to claim 5, characterized in that Controlling the air blowing device to adjust the motion trajectory of the floating object includes: Determining position information of a blower device, and determining a start time and blower parameters of the blower device based on the position information of the blower device and the predicted motion trajectory, wherein the blower parameters include at least one of the following: wind speed level and wind direction; The blowing device is controlled to operate according to the blowing parameters at the start time to adjust the movement trajectory of the floating object.

7. A floating object identification device, characterized in that: include: an acquisition unit, configured to acquire the railway image collected by the monitoring device at each collection moment within a target time period to obtain a target railway image; a first determining unit, configured to determine the meteorological data at each acquisition moment, and obtain target meteorological data corresponding to each frame of the target railway image; an input unit, configured to input the target railway image and the target meteorological data into a target model to obtain a floating object recognition result at each acquisition moment, wherein the target model is trained by multiple sets of training samples, each set of training samples including a historical railway image, historical meteorological data corresponding to the historical railway image, and historical floating object recognition results; Among them, the input unit includes: a first extraction module, used to extract the three-primary color pixel values ​​of the target railway image to obtain a three-primary color pixel matrix; an input module, used to input the three-primary color pixel matrix into the convolution processing layer of the target model for processing to obtain multiple feature maps; a second extraction module, used to extract keywords from the target meteorological data to obtain meteorological keywords, and convert the meteorological keywords into meteorological word vectors through a word vector conversion tool, wherein the meteorological word vectors have the same dimension as the feature maps; a superposition module, used to superimpose the meteorological word vectors and the feature maps for each feature map through the fusion parameters of the target model to obtain a superimposed feature map; an encoding and decoding module, used to encode the superimposed feature maps through the encoder of the target model to obtain encoding features, and decode the encoding features through the decoder of the target model to obtain the floating object recognition results.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the floating object identification method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: The device comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the floating object identification method according to any one of claims 1 to 6.

Images