Railway image data processing method and device based on large model

Abstract

The application provides a railway image data processing method and device based on a large model. First image data containing railway elements is obtained, the first image data including a first picture with railway elements; first input data is generated based on the first image data and a first template, the first input data is input into a predetermined large model to obtain first output data; the first template is used to indicate output requirements; the first output data includes feature labels and / or descriptions; the first output data is matched with a predetermined multi-level category label to determine a first category label sequence to which the first image data belongs; and the first image data is stored in a folder corresponding to the first category label sequence. In this way, the image data containing railway elements can be classified and managed by means of the capacity of the large model, thereby improving the efficiency of classification and management and the usability of data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of railway image management, in particular, to a railway image data processing method and device based on a large model. BACKGROUND

[0002] Railway image and video resource intelligent analysis technology originates from the important position of railway transportation in economic and social development. As an important part of the comprehensive transportation system, the operation and maintenance of railways are crucial. In recent years, railway transportation has achieved remarkable achievements, but the complex transportation network and equipment structure have also brought challenges.

[0003] Traditional image analysis methods mainly rely on manual analysis, which leads to the following problems: on the one hand, the huge detection data brings heavy workload to the identification personnel, resulting in a large consumption of manpower and material resources; on the other hand, the accuracy and coverage of image analysis are directly related to the ability and experience of personnel, making it difficult to ensure the quality of analysis work; in addition, since manual analysis is generally carried out after image acquisition, it cannot ensure the timeliness of analysis. With the development of artificial intelligence technology, deep learning algorithms have made breakthroughs in image analysis. Image intelligent analysis aims to automatically analyze the category, position, state, and other information of key targets in the image through machine algorithms.

[0004] Traditional image analysis methods based on rules and feature engineering have been difficult to meet the needs of railway systems for large-scale data processing and intelligent analysis. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a railway image data processing method and device based on a large model, to alleviate the technical problems of high management cost of image data containing railway elements and low availability of data.

[0006] In a first aspect, the present application provides a railway image data processing method based on a large model, comprising:

[0007] Obtaining first image data containing railway elements, the first image data comprising a first picture with railway elements;

[0008] Based on the first image data and a first template, generating first input data, inputting the first input data into a pre-determined large model to obtain first output data; the first template is used to indicate output requirements; the first output data includes feature labels and / or descriptions;

[0009] Matching the first output data with a pre-determined multi-level category label to determine a first category label sequence to which the first image data belongs;

[0010] store the first image data in a folder corresponding to the first category label sequence.

[0011] In an optional implementation, the first output data corresponding to the first image data is stored in a folder corresponding to the first category label sequence; the method further comprises:

[0012] receiving an image data query request, the image data query request including second image data containing a railway element, the second image data including a second picture with a railway element;

[0013] generating second input data based on the second image data and the first template, inputting the second input data into a predetermined large model to obtain second output data; the second output data includes a label and / or a description;

[0014] matching the second output data with a predetermined multi-level category label to determine a second category label sequence to which the second image data belongs;

[0015] obtaining all third image data stored in a folder corresponding to the second category label sequence and third output data corresponding to the third image data, wherein third input data is generated based on the third image data and the first template, and the third input data is input into a predetermined large model to obtain third output data;

[0016] determining one or more third output data most matching the second output data; taking third image data corresponding to the most matching third output data as a query result of the image data query request and returning.

[0017] In an optional implementation, before obtaining all third image data stored in a folder corresponding to the second category label sequence, the method further comprises:

[0018] determining whether the number of the third image data is greater than a first threshold value;

[0019] If not, all the third image data is taken as a query result of the image data query request;

[0020] If yes, the step of obtaining all third image data stored in a folder corresponding to the second category label sequence is executed.

[0021] In an optional implementation, determining one or more third output data most matching the second output data comprises:

[0022] determining a matching degree of each of the third output data with the second output data, and sorting the third output data according to the matching degree when the number of the third image data is greater than a first threshold value;

[0023] determining one or more third output data that are most matched with the second output data based on the sorting;

[0024] After the third image data corresponding to the third output data that are most matched is taken as a query result of the image data query request and returned, the method further comprises:

[0025] receiving a continuous query request for the second image data, determining one or more third output data that are second matched with the second output data based on the sorting, and taking the third image data corresponding to the third output data that are second matched as a query result of the continuous query request and returning.

[0026] In an optional implementation, the multi-level category label comprises a basic category label and a dynamic category label, wherein the basic category label is maintained by configuration, and the dynamic category label is dynamically maintained in the process of classifying image data containing railway elements.

[0027] In an optional implementation, the method further comprises:

[0028] judging whether the number of image data in the folder corresponding to the first category label sequence exceeds a second threshold value;

[0029] When the number exceeds, clustering the image data in the folder corresponding to the first category label sequence, dividing into at least two clusters, and determining a key feature label of each cluster;

[0030] determining a new dynamic category label of each cluster based on the key feature label, and taking the new dynamic category label as a child node of a leaf node of the first category label sequence.

[0031] In an optional implementation, the image data comprises a picture or a video; wherein a first large model is used to determine the output data for the picture, and a second large model is used for the video, the first large model being obtained by tuning a multi-modal large model pre-trained on a public data set on a railway track transportation professional picture data set; and the second large model being obtained by tuning a multi-modal large model pre-trained on a public data set on a railway track transportation professional video data set.

[0032] In a second aspect, the application provides a railway image data processing device based on a large model, comprising:

[0033] An acquisition module is configured to acquire first image data containing a railway element, the first image data including a first picture with the railway element;

[0034] A generation module is configured to generate first input data based on the first image data and a first template, input the first input data into a predetermined large model, and obtain first output data; the first template is used to indicate output requirements; and the first output data includes a feature label and / or a description.

[0035] A matching module is configured to match the first output data with a plurality of predetermined class labels of multiple levels to determine a first class label sequence to which the first image data belongs.

[0036] A storage module is configured to store the first image data in a folder corresponding to the first class label sequence.

[0037] In an optional implementation, the first output data and the first image data are stored in the folder corresponding to the first class label sequence; and the apparatus further includes:

[0038] A receiving module is configured to receive an image data query request, the image data query request including second image data containing a railway element, the second image data including a second picture with the railway element.

[0039] The generation module is further configured to generate second input data based on the second image data and the first template, input the second input data into the predetermined large model, and obtain second output data; and the second output data includes a label and / or a description.

[0040] The matching module is further configured to match the second output data with the plurality of predetermined class labels of multiple levels to determine a second class label sequence to which the second image data belongs.

[0041] A returning module is configured to acquire all third image data stored in a folder corresponding to the second class label sequence and third output data corresponding to the third image data, generate third input data based on the third image data and the first template, input the third input data into the predetermined large model, and obtain third output data; determine one or more third output data most matched with the second output data; and return third image data corresponding to the most matched third output data as a query result of the image data query request.

[0042] In an optional implementation, the apparatus further includes:

[0043] The first judging module is configured to judge whether the quantity of the third image data is greater than a first threshold value;

[0044] The returning module is specifically configured to, if not, return all the third image data as the query result of the image data query request; and if yes, execute the operation of acquiring all the third image data stored in the folder corresponding to the second category label sequence.

[0045] In an optional implementation, the returning module is specifically configured to, when the quantity of the third image data is greater than the first threshold value, determine the matching degree of each third output data with the second output data, and sort the third output data according to the matching degree.

[0046] The one or more third output data most matched with the second output data are determined based on the sorting.

[0047] After the third image data corresponding to the third output data most matched is returned as the query result of the image data query request, the method further comprises:

[0048] receiving a continuous query request for the second image data, determining the one or more third output data second matched with the second output data based on the sorting, and returning the third image data corresponding to the second matched third output data as the query result of the continuous query request.

[0049] In an optional implementation, the multi-level category label comprises a basic category label and a dynamic category label, wherein the basic category label is maintained by configuration, and the dynamic category label is dynamically maintained in the process of classifying the image data containing the railway element.

[0050] In an optional implementation, the apparatus further comprises:

[0051] The second judging module is configured to judge whether the quantity of the image data in the folder corresponding to the first category label sequence exceeds a second threshold value.

[0052] The maintaining module is configured to, when the quantity exceeds the second threshold value, cluster the image data in the folder corresponding to the first category label sequence, divide the image data into at least two clusters, and determine a key feature label of each cluster; determine a new dynamic category label of each cluster based on the key feature label, and take the new dynamic category label as a child node of a leaf node of the first category label sequence.

[0053] In an optional implementation, the image data comprises a picture or a video; wherein the output data is determined by using a first large model for the picture, and a second large model is used for the video, the first large model is obtained by adjusting a multi-modal large model pre-trained on a public data set on a railway track transportation professional picture data set; and the second large model is obtained by adjusting a multi-modal large model pre-trained on a public data set on a railway track transportation professional video data set.

[0054] In a third aspect, the present application provides a server, comprising a memory and a processor, the memory stores a computer program executable on the processor, and the processor executes the computer program to implement the method of any one of the preceding embodiments.

[0055] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer executable instructions, when the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to execute the method of any one of the preceding embodiments.

[0056] The present application provides a railway image data processing method and device based on a large model. By obtaining first image data containing railway elements, the first image data includes a first picture with railway elements; based on the first image data and a first template, first input data is generated, the first input data is input into a pre-determined large model to obtain first output data; the first template is used to indicate output requirements; the first output data includes feature labels and / or descriptions; the first output data is matched with a pre-determined multi-level category label to determine a first category label sequence to which the first image data belongs; and the first image data is stored in a folder corresponding to the first category label sequence. In this way, the image data containing railway elements can be classified and managed by means of the capacity of the large model, improving the efficiency of classification and management and improving the usability of data. BRIEF DESCRIPTION OF DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0058] Figure 1 A railway image data processing method based on a large model is provided in the embodiments of the present application, and a flowchart of the method is shown in the figure;

[0059] Figure 2 Another large model-based railway image data processing method flowchart provided by an embodiment of the present application;

[0060] Figure 3 A multi-level category label structure diagram provided by an embodiment of the present application;

[0061] Figure 4 Another large model-based railway image data processing method flowchart provided by an embodiment of the present application;

[0062] Figure 5 A model structure diagram provided by an embodiment of the present application;

[0063] Figure 6 Another model structure diagram provided by an embodiment of the present application;

[0064] Figure 7 Another model structure diagram provided by an embodiment of the present application;

[0065] Figure 8 A large model-based railway image data processing device structure diagram provided by an embodiment of the present application;

[0066] Figure 9 A large model-based railway image data processing method flowchart provided by an embodiment of the present application. DETAILED DESCRIPTION

[0067] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0068] Figure 1 A large model-based railway image data processing method flowchart provided by an embodiment of the present application. As shown in Figure 1 The method comprises:

[0069] S110, acquiring first image data containing railway elements, the first image data comprising a first picture, the first picture containing railway elements.

[0070] The image data comprises a picture or a video; the first picture can be an image frame in a video.

[0071] The embodiments of the present application can batch classify image data, and can also realize classification of single image data into an already classified image database. Based on this, the first image data can be single image data or batch multiple image data.

[0072] For example, the user can first determine the image data to be classified, and then trigger a classification request for the to-be-classified data. After receiving the to-be-classified request, the first image data containing the railway element is obtained.

[0073] The method can be implemented in a cloud server or a local terminal.

[0074] In S120, first input data is generated based on the first image data and a first template, and the first input data is input into a pre-determined large model to obtain first output data; the first template is used to indicate output requirements; and the first output data includes feature labels and / or descriptions.

[0075] Here, the module can define a plurality of templates in advance, which are mainly used to indicate the expected output results of the large model. The template can be a text string with a preset format, which includes a selectable or to-be-input text reserved paragraph. For example, the template can be "for the following

picture / video

label output / description output

attachment

picture / video

label output / description output

attachment

picture

label output

first image data

[0076] In some embodiments, as shown in Figure 4 The present application can provide a template configuration interface, which can include input boxes for each reserved paragraph, and each input box can be used for input.

[0077] As an example, a first large model is used to determine the output data for pictures, and a second large model is used for videos. The first large model is obtained by adjusting a pre-trained multi-modal large model on a public data set through a railway track transportation professional picture data set. The second large model is obtained by adjusting a pre-trained multi-modal large model on a public data set through a railway track transportation professional video data set.

[0078] The process of adjusting the large model can be implemented by the following steps:

[0079] For railway image data, first, a label table (which can be the basic category label in the multi-level category label below) needs to be created, including possible target objects such as tracks, bullet trains, and contact nets, covering all possible labels as much as possible. Then, use the annotation tool to manually annotate the image, and organize the label information into a unified description template: "There are {label} in the scene". This way, a corresponding description file can be generated for each image. If a deep learning method based on graph convolutional neural network, CNN-RNN, etc. is used, data preprocessing is required. This includes data cleaning and denoising, image dehazing, data augmentation, sample balancing, etc. These preprocessing operations can improve the performance and effectiveness of intelligent analysis tasks for railway image and video resources, making them better adapt to the complex transportation environment and task requirements of the railway system. However, these operations may increase the complexity and training time of model inference. Therefore, after considering the above factors, it may be more appropriate to use a multi-modal large model method to complete the intelligent analysis task. This method does not require complex data preprocessing, only the input test and training data needs to be unified to a standard size, and the model training and inference process can be completed.

[0080] The advantage of multi-modal large models is that their structure includes specialized encoder modules for processing images and text. These modules can utilize pre-trained parameter files, thereby avoiding the large amount of time and computational resources required to train a model from scratch. Compared to other deep learning methods, this pre-trained model can achieve faster convergence and generally has better performance. By utilizing pre-trained image encoders and text encoders, multi-modal large models can better understand and represent input image and text information, thereby improving the model's ability to model the relevance between images and text. The use of this model not only saves training time and resource costs, but also provides a more efficient and accurate solution for semantic understanding and information extraction tasks between images and text.

[0081] Parameter optimization refers to the process of adjusting the parameters of the model during the training of the multi-modal large model to improve the model's performance and generalization ability. This includes hyperparameter adjustment, learning rate scheduling, regularization, optimizer selection, etc. By optimizing these parameters, the model's performance and generalization ability can be improved, making it better adapt to professional application scenarios and achieving better results.

[0082] Multimodal large models have achieved good results on public datasets, such as image classification, text classification, and image annotation tasks. Therefore, after adjusting the parameters of the pre-trained multimodal large model, it can be applied to the railway track transportation professional dataset to achieve good results. By fine-tuning on railway images and related text data, the model can better adapt to the characteristics and needs of the railway field, achieving more accurate and efficient railway image intelligent analysis. Such transfer learning methods can fully utilize the knowledge learned by pre-trained models on large-scale data, avoiding the time and resource consumption of training models from scratch, while also quickly achieving good results in the railway track transportation field.

[0083] Since the output of the multimodal large model is usually in the form of image descriptions, keyword extraction algorithms can be used to process these descriptions to obtain a set of image labels. Keyword extraction algorithms can identify and extract representative and important keywords from text, which can effectively express the content and features of the image. By converting image descriptions into keyword sets, the information expressed by the image can be more intuitively understood, and it also provides an important reference for subsequent image classification, retrieval, and other tasks. Such processing methods not only make full use of the output of the multimodal large model, but also extract image labels with more information and description, thereby improving the effectiveness and accuracy of railway image intelligent analysis.

[0084] For the adopted multimodal large model architecture, considering that railway image and video data have different modal characteristics, it is necessary to fully utilize the characteristics of different modal data, so that each model can focus on processing specific types of data and maximize its advantages. Therefore, two model structures are adopted to complete the multi-label output of railway images.

[0085] For the multimodal characteristics of video data, a model structure is adopted that can effectively process image, text, and speech data of three different modalities and achieve multi-label output of railway videos. This model structure design fully considers the diversity and complexity of video data, integrates information from different modal data, and improves the analysis and understanding ability of railway videos. By using this model structure, useful information in video data can be more comprehensively mined.

[0086] As an example, the structure of the first large model or the second large model can be a multimodal large model as shown in Figure 5 The multimodal large model adopts three main components: an image feature extraction network, a Transformer encoder, and a SwinTransformer module and linear layer.

[0087] The image feature extraction network adopts the Vision Transformer (ViT) structure, which is an advanced deep learning model based on the Transformer architecture. Specifically, the workflow of the ViT model is as follows: First, the first picture is divided into fixed-size blocks (patches), with a common size of 16x16 pixels. These image blocks are then flattened into one-dimensional vectors and embedded through a linear layer, converting them into the format required by the Transformer. This embedding step allows each image block to be represented as a fixed-size vector, which can be processed in subsequent network layers. ViT introduces position encoding after the embedding layer. This step ensures that the spatial information of the pixels in the image is preserved, which is crucial for image understanding tasks. After embedding and position encoding, the image code size becomes (batch_size, sequence_length, hidden_size), where batch_size is the size of the batch processing, sequence_length is the number of image blocks, and hidden_size is the dimension of the embedding vector of each image block.

[0088] This design not only improves the model's understanding of image content, but also has significant advantages in computational efficiency. The combination of flattened image blocks and position encoding allows data to be directly input into the Transformer encoder, facilitating subsequent image feature processing and classification tasks. In this way, ViT can efficiently learn image features and lay a solid foundation for further model processing.

[0089] Each Transformer encoder of the multi-modal large model includes two sub-layer connection structures: the first sub-layer connection structure includes a self-attention layer (Bi Sef-attention) and a normalization layer (Add&Normalize) and a residual connection; the second sub-layer connection structure includes a feed-forward fully connected sub-layer (FFN) and a normalization layer and a residual connection.

[0090] The Transformer encoder first passes the linearized feature map through the self-attention layer for fine feature interaction, and then reduces the feature dimension to D through a linear layer. This adjusted feature sequence is then sent to the cross-attention module of each decoder layer. This model does not introduce learnable or fixed position encoding in the encoder features.

[0091] The Swin Transformer module is constructed by replacing the standard multi-head self-attention (MSA) module in the Transformer with a shift window-based self-attention module (W-MSA) while maintaining the original layer structure. The module is composed of a specially designed shift window multi-head self-attention module and two subsequent multi-layer perceptron (MLP) layers, where the MLP contains a GELU non-linear activation function. Before each self-attention module and MLP, a layer normalization (LN) layer is applied, and a residual connection is added after each module to optimize information flow and enhance the robustness of the model.

[0092] The Swin Transformer module calculates self-attention within non-overlapping local windows. These windows uniformly divide the image, reducing the computational complexity to be linearly related to the window size and image size, significantly improving computational efficiency. The Swin Transformer module can adapt to images of different resolutions, and by adjusting the size and number of windows, the model can efficiently process images of various sizes without wasting computational resources due to changes in resolution.

[0093] Finally, after a linear layer, the image multi-label classification task can be completed. Based on this architecture, excellent performance can be achieved on the railway professional image label classification task, achieving a better classification effect.

[0094] As another example, the structure of the first large model or the second large model can be a multi-modal large model as shown in Figure 6 The entire structure is divided into two main parts: one part is the image feature extraction branch, and the other part is the related label processing branch.

[0095] The image feature extraction branch is constructed based on the convolutional neural network ConvNeXt v2. ConvNeXt v2 runs in a fully convolutional manner, and its learning signal is generated by randomly masking the original input visual information and using a high masking ratio. The model then predicts the masked part based on the remaining context information. ConvNeXt v2 introduces a global response normalization (GRN) layer, which can enhance the expression and generalization ability of the model. After simple image preprocessing, the image is sent to the ConvNeXt v2 pre-trained model to extract features.

[0096] The related label processing branch combines the multi-head attention mechanism and the graph neural network, aiming to learn the mutual relationship between the labels. First, the embedding vectors of the image-related labels, i.e., Label Embeddings, such as Z1, Z2, …, Zn, are obtained, and the cosine similarity between these embedding vectors is calculated. Based on the similarity and a preset threshold and weight parameter, an adjacency matrix is constructed. Using the adjacency matrix, the values of the query (Q), key (K), and value (V) can be obtained through different linear layers. After the attention mechanism processing and the softmax layer, different subgraphs can be obtained. These subgraphs are connected with each other to form a new graph structure, which is represented in matrix form. Then, the newly constructed matrix is input into the graph attention network (GAN).

[0097] The matrix multiplication operation of the image feature vectors obtained by the two branches and the label representation learned by the graph neural network can obtain the corresponding prediction scores, as shown on the far right side of the model diagram. On this basis, by setting a proper threshold, the corresponding label of the image can be accurately determined.

[0098] As another example, the structure of the pre-determined large model can be as shown in Figure 7 The model can include a first model, a second model, and a large language model, and the second model can include a first sub-model and a second sub-model. The first model can be a model as shown in Figure 5 The second model can be a model as shown in Figure 6 The first sub-model can be an image feature extraction branch, and the second sub-model can be a related label processing branch. By fusing the results of the first model and the second model, the initial label of the first picture can be obtained. The initial label and the first template are input into the pre-trained large language model to obtain the first output data. The large language model used here refers to those deep learning models with a large number of parameters, which are usually trained on large-scale text data to capture the complex structure and pattern of language. The large language model used in the embodiments of the present application can include BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer) series, T5 (Text-to-Text Transfer Transformer), ALBERT (A Lite BERT), RoBERTa (Robustly Optimized BERT Pretraining Approach), and RoBERTa, etc.

[0099] In some embodiments, the step of fusing the results of the first model and the second model can include a plurality of weighted fusions. For example, for weighted fusion, the weights can be determined in advance, specifically according to the recognition accuracy of the model in the training process.

[0100] In some embodiments, before data analysis and modeling, data preprocessing is a crucial step, which involves a series of preparations such as cleaning, integrating, transforming and reducing the original data, aiming to improve the quality and usability of the data. For image data sets, preprocessing is particularly important, as it is directly related to the efficiency and final performance of model learning. For image data, the preprocessing process usually includes the following steps: uniform image size: adjust all images to a fixed size to provide a uniform input format for the model and simplify subsequent image processing steps. Implement data augmentation: through random flipping, adjusting color attributes (such as contrast and saturation), and applying affine transformation, etc., to enhance the diversity of the data set and improve the model's ability to adapt to image transformation. Standardize image data: standardize the image by subtracting the mean value of each channel and dividing by the standard deviation to make the image data distribution more uniform, which helps to speed up the model convergence and improve the stability of training. After the above preprocessing steps, the obtained image tensor will be sent as input into different models for training and testing, so as to ensure that the model can learn effective features from high-quality data, thereby improving the accuracy and stability of prediction.

[0101] Since the training data set used is a railway track transportation data set, the images used need to be labeled in advance. In the training process of the railway track transportation data set, image preprocessing and labeling are key steps. For models such as Figure 5 As shown in the model, the core feature extraction module uses a pre-trained Vision Transformer (ViT) model as the backbone. This pre-trained model has learned general feature representations on large-scale data sets, so before starting training, the corresponding pre-trained parameters need to be downloaded.

[0102] Using a pre-trained ViT model can significantly save training time, reduce computational complexity, and improve the performance of the model on downstream tasks. In addition to the backbone, the rest of the model as shown in Figure 5 does not use pre-trained parameters during training.

[0103] During the training phase, multiple parameters will be configured, including the number of epochs, batch size, and image path. During the training process, the training progress will be recorded and displayed for each epoch, including the loss and F1 value for each batch. At the end of each epoch, the system will calculate and report the average precision, recall, and F1 value for that epoch, allowing us to monitor the training effect and performance of the model.

[0104] To optimize the training efficiency and reduce the complexity of the model as shown in Figure 6 , the backbone of the model uses a pre-trained ConvNext v2 model. The ConvNext v2 model is pre-trained on a large-scale dataset, which can extract high-level features of images, helping to speed up the training and improve the performance of the model.

[0105] In the model as shown in Figure 6 , the generation of label embedding vectors is also crucial. For this purpose, advanced natural language processing architectures such as Bert or Roberta are used to encode the features of the labels. These models can learn deep semantic representations of labels, thereby improving the consistency between generated images and labels. For datasets like COCO and VOC, pre-trained model parameters are usually provided in the form of pkl files. The model can directly read these files and use the parameters for training or testing, greatly simplifying the data preparation process.

[0106] The model as shown in Figure 5 is similar to the model as shown in Figure 6 . During training, a series of parameters need to be set, such as the number of epochs (epoch), batch size (batch_size), etc. When evaluating the performance of the model, precision, recall, and F1 value are used as evaluation indicators.

[0107] For both models, regular testing will be performed during the training process to evaluate the performance of the model at different epochs. The model will select the best-performing epoch and save the corresponding model architecture and weight parameters, as well as the parameter configuration during training, to facilitate subsequent use and further analysis.

[0108] To develop a high-performance multi-label classification model for railway track transportation data sets, this paper provides options for adjusting model parameters and allows for customized modifications to certain parts of the model architecture. This process aims to explore different model configurations to find the best parameter combination and architecture design.

[0109] During the model adjustment and optimization process, the model will be trained under multiple configurations, and its performance on the training set and test set will be compared in detail. Evaluation indicators will include classification accuracy, recall rate, F1 score, and other key indicators.

[0110] The model configuration with the best performance on the evaluation indicators will be selected. This selected model will be considered the best multi-label classification model for the railway track transportation dataset, as it not only accurately identifies and classifies multiple labels in the image, but also has good generalization ability and robustness. In this way, the practicality and application value of the model in the professional field are ensured.

[0111] The input of the model receives the image and the corresponding label, while the output generates the model's predicted label. By comparing and analyzing the predicted label with the true label, the performance of the model is monitored and evaluated in real time, which helps to ensure that the model maintains high accuracy and reliable stability in actual application.

[0112] In terms of model deployment, a local deployment method can be chosen, which deploys the complete architecture, parameters, and related support files of the model to a designated device. This deployment mode facilitates local training and debugging of the model, and also simplifies the use and maintenance process of the model.

[0113] S130, match the first output data with the pre-determined multi-level category labels to determine the first category label sequence to which the first image data belongs.

[0114] The first output data is a text, which can be a label or a description. The matching degree of the label or description with each branch of the multi-level category label can be determined. Each branch is composed of all category labels from the root node to the leaf node, and each branch constitutes a category label sequence.

[0115] The matching degree can be calculated by a text similarity algorithm. Text similarity matching algorithm is an important application in the field of natural language processing, used to measure the similarity between two texts. The following algorithms can be used to calculate the most similar category label sequence:

[0116] String similarity: Calculate the similarity by comparing the number of identical characters in two texts. This method is simple and easy to implement, but it ignores the semantic information of the text.

[0117] ‌Simhash: This is a local sensitive hashing algorithm suitable for handling massive amounts of text data, capable of quickly processing web text and removing duplicates.

[0118] ‌Word vector: Represent text as a low-dimensional real vector through training. This method takes into account the semantic information of the text and can better reflect the meaning of the text.

[0119] Cosine similarity: This measure of text similarity is achieved by calculating the cosine of the angle between two vectors. This method is widely used in search engines, automated question answering, and other fields.

[0120] TF-IDF: This is a statistical model that assesses document importance by calculating the weights of words. TF-IDF can be combined with cosine similarity to calculate the similarity between documents.

[0121] Deep learning algorithms, such as DSSM, ConvNet, and Siamese LSTM, can learn more complex text representations, thereby improving the accuracy of similarity calculations.

[0122] Other algorithms, such as Jaccard similarity and Levenshtein edit distance, can also be used to calculate the similarity between texts.

[0123] In practical applications, choosing the right algorithm requires considering the characteristics of the data, the limitations of computing resources, and the expected results. For example, Simhash is favored for its efficient processing capabilities for large-scale text data; while deep learning algorithms may be more suitable for scenarios requiring precise semantic understanding.

[0124] S140, store the first image data in the folder corresponding to the first category label sequence.

[0125] In some embodiments, the folder name may be based on a first category label sequence.

[0126] In some embodiments, such as Figure 3 As shown, the multi-level category labels include basic category labels and dynamic category labels. The basic category labels are maintained through configuration, while the dynamic category labels are dynamically maintained during the classification of image data containing railway elements. When a folder contains too many images, it hinders data utilization; therefore, the folder can be split based on the maintenance principles of the dynamic category labels. Based on this, the method further includes: determining whether the number of image data in the folder corresponding to the first category label sequence exceeds a second threshold; if so, clustering the image data in the folder corresponding to the first category label sequence into at least two clusters and determining the key feature labels for each cluster; determining a new dynamic category label for each cluster based on the key feature labels, and using the new dynamic category label as a child node of the leaf node of the first category label sequence.

[0127] For the image search task, the target image needs to be input, that is, a picture is input to search for related pictures or information, which is a common image search technology. Image search searches for pictures with similar visual features by analyzing the visual features of the input picture using a similarity matching algorithm. Using a convolutional neural network, image search can more deeply understand the content of the picture, including object, scene, emotion, etc. However, compared with multi-modal large models, this deep learning method requires manual annotation of data and manual adjustment of parameters, consuming human and time costs, and the generalization effect is poor. Therefore, the following model structure will be used to complete:

[0128] In some embodiments, the first output data corresponding to the first image data is stored in the folder corresponding to the first category label sequence; the method can further include the following steps, as shown in Figure 2

[0129] S210, receiving an image data query request, the image data query request including second image data containing railway elements, the second image data including a second picture with railway elements;

[0130] S220, generating second input data based on the second image data and the first template, inputting the second input data into the pre-determined large model to obtain second output data; the second output data includes labels and / or descriptions;

[0131] S230, matching the second output data with the pre-determined multi-level category label to determine the second category label sequence to which the second image data belongs;

[0132] S240, obtaining all third image data stored in the folder corresponding to the second category label sequence and third output data corresponding to the third image data, wherein the third input data is generated based on the third image data and the first template, and the third input data is input into the pre-determined large model to obtain the third output data;

[0133] S250, determining one or more third output data most matching the second output data; taking the third image data corresponding to the most matching third output data as the query result of the image data query request and returning.

[0134] Since the display space is usually limited, the amount of matched image data can be large, so when displaying, only the most relevant image data can be displayed.

[0135] ​Additionally, before acquiring all third image data stored in the folder corresponding to the second category label sequence, the method further includes: determining whether the number of third image data is greater than a first threshold; if not, all third image data is taken as the query result of the image data query request; if yes, acquiring all third image data stored in the folder corresponding to the second category label sequence is performed.

[0136] Based on this, the step of determining one or more third output data most matched with the second output data in the foregoing S250 can be implemented by the following steps: when the number of third image data is greater than the first threshold, determining the matching degree of each third output data with the second output data, and sorting the third output data according to the matching degree; determining one or more third output data most matched with the second output data based on the sorting.

[0137] After the third image data corresponding to the most matched third output data is taken as the query result of the image data query request and returned, the method further includes: receiving a continuous query request for the second image data, determining one or more third output data second matched with the second output data based on the sorting, and taking the third image data corresponding to the second matched third output data as the query result of the continuous query request and returning.

[0138] Figure 8 A structure schematic diagram of a railway image data processing device based on a large model provided for the embodiment is shown in FIG. 1, which can include: Figure 8

[0139] The acquisition module 501 is configured to acquire first image data containing a railway element, the first image data including a first picture with the railway element.

[0140] The generation module 502 is configured to generate first input data based on the first image data and a first template, input the first input data into a pre-determined large model, and obtain first output data; the first template is used to indicate output requirements; and the first output data includes a feature label and / or a description.

[0141] The matching module 503 is configured to match the first output data with a pre-determined multi-level category label, and determine a first category label sequence to which the first image data belongs.

[0142] The storage module 504 is configured to store the first image data in a folder corresponding to the first category label sequence.

[0143] In some embodiments, the first output data corresponding to the first image data is stored in the folder corresponding to the first category label sequence; and the device further includes:

[0144] ​The receiving module is configured to receive an image data query request, the image data query request comprising second image data containing a railway element, the second image data comprising a second picture with the railway element;

[0145] The generating module is further configured to generate second input data based on the second image data and the first template, input the second input data into the pre-determined large model to obtain second output data, and the second output data comprising a label and / or a description.

[0146] The matching module is further configured to match the second output data with the pre-determined multi-level category labels to determine a second category label sequence to which the second image data belongs.

[0147] The returning module is configured to obtain all third image data stored in a folder corresponding to the second category label sequence and third output data corresponding to the third image data, generate third input data based on the third image data and the first template, input the third input data into the pre-determined large model to obtain the third output data, determine one or more third output data most matching the second output data, and return third image data corresponding to the most matching third output data as a query result of the image data query request.

[0148] In some embodiments, the apparatus further comprises:

[0149] The first judging module is configured to judge whether the number of the third image data is greater than a first threshold value.

[0150] The returning module is specifically configured to, if not, return all the third image data as the query result of the image data query request, and if yes, execute the obtaining of all the third image data stored in the folder corresponding to the second category label sequence.

[0151] In some embodiments, the returning module is specifically configured to,

[0152] When the number of the third image data is greater than the first threshold value, determine a matching degree of each third output data with the second output data, and sort the third output data according to the matching degree.

[0153] Determine one or more third output data most matching the second output data based on the sorting.

[0154] After the third image data corresponding to the most matching third output data is returned as the query result of the image data query request, the method further comprises:

[0155] The second image data is received, and one or more third output data that is second matched with the second output data is determined based on the sorting, and the third image data corresponding to the second matched third output data is returned as the query result of the continue query request.

[0156] In some embodiments, the multi-level category label includes a basic category label and a dynamic category label, wherein the basic category label is maintained by configuration, and the dynamic category label is dynamically maintained in the process of classifying image data containing railway elements.

[0157] In some embodiments, the device further comprises:

[0158] The second determining module is configured to determine whether the number of image data in the folder corresponding to the first category label sequence exceeds a second threshold value.

[0159] The maintaining module is configured to, when the number of image data in the folder corresponding to the first category label sequence exceeds the second threshold value, cluster the image data in the folder corresponding to the first category label sequence into at least two clusters, determine a key feature label of each cluster, determine a new dynamic category label of each cluster based on the key feature label, and take the new dynamic category label as a child node of a leaf node of the first category label sequence.

[0160] In some embodiments, the image data includes a picture or a video; wherein the output data is determined by using a first large model for the picture, and a second large model is used for the video, the first large model is obtained by adjusting a multi-modal large model pre-trained on a public data set through a railway track transportation professional picture data set, and the second large model is obtained by adjusting a multi-modal large model pre-trained on a public data set through a railway track transportation professional video data set.

[0161] Referring to Figure 9 As shown in the accompanying drawings, the server 600 provided by the embodiments of the present application at least includes: a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601, and the processor 601 executes the computer program to implement the railway image data processing method based on a large model provided by the embodiments of the present application.

[0162] The server 600 provided by the embodiments of the present application can further include a bus 603 connecting different components (including the processor 601 and the memory 602). Among them, the bus 603 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, etc.

[0163] The memory 602 can include a readable storage medium in the form of volatile memory, such as a random access memory (RAM) 6021 and / or cache memory 6022, and can further include a read-only memory (ROM) 6023. The memory 602 can also include program tools 6025 having a set of (at least one) program modules 6024, including but not limited to operating systems, one or more applications, other program modules, and program data, each of which or some combination of which can include the implementation of a network environment.

[0164] The processor 601 can be one processing element or a collective term for a plurality of processing elements. For example, the processor 601 can be a central processing unit (CPU), or one or more integrated circuits configured to implement the method of processing railway image data based on a large model provided by the embodiments of the present application. Specifically, the processor 601 can be a general-purpose processor, including but not limited to a CPU, an application specific integrated circuit (ASIC), a ready-to-program gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0165] The server 600 can communicate with one or more external devices 604 (such as a keyboard, a remote control, etc.), and can also communicate with one or more devices that enable a user to interact with the server 600 (such as a mobile phone, a computer, etc.), and / or with any device that enables the server 600 to communicate with one or more other servers 600 (such as a router, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 605. Furthermore, the server 600 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 606. As Figure 6 shown, the network adapter 606 communicates with other modules of the server 600 through the bus 603. It should be understood that although Figure 6Other hardware and / or software modules that can be used in conjunction with server 600, which are not illustrated, include but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data archival storage subsystems, etc.

[0166] It should be noted that, Figure 9 The server 600 shown is only one example. It should not be understood to limit the applicability and functionality of embodiments of the present application.

[0167] A computer readable storage medium provided by an embodiment of the present application is described below. The computer readable storage medium provided by an embodiment of the present application stores computer instructions. The computer instructions are executed by a processor to implement a railway image data processing method based on a large model provided by an embodiment of the present application. Specifically, the computer instructions can be built-in or installed in the processor. In this way, the processor can implement the railway image data processing method based on the large model provided by an embodiment of the present application by executing the built-in or installed computer instructions.

[0168] In addition, the railway image data processing method based on the large model provided by an embodiment of the present application can also be implemented as a computer program product. The computer program product includes program codes. The program codes are run on a processor to implement the railway image data processing method based on the large model provided by an embodiment of the present application.

[0169] The computer program product provided by an embodiment of the present application can use one or more computer readable storage media. The computer readable storage media can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any suitable combination of the above. Specifically, more specific examples (non-exhaustive list) of the computer readable storage media include: an electrical connection with one or more wires, a portable disc, a hard disk, a RAM, a ROM, an Erasable Programmable Read Only Memory (EPROM), an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0170] The computer program product provided by the embodiments of the present application can adopt a CD-ROM and include program codes, and can also run on a server such as a road management device. However, the computer program product provided by the embodiments of the present application is not limited to this. In the embodiments of the present application, the computer readable storage medium can be any tangible medium containing or storing program codes, which can be used by or in combination with an instruction execution system, device or apparatus.

[0171] It should be noted that, although several units or sub-units of the apparatus are mentioned in the foregoing detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units for embodiment.

[0172] In addition, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all of the illustrated operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, combined into one step, and / or split into multiple steps.

[0173] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to cover all changes and modifications falling within the scope of the present application.

[0174] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. A large model-based railway image data processing method, characterized in that, The method comprises: acquiring first image data containing railway elements, the first image data comprising a first picture with railway elements; generating first input data based on the first image data and a first template, inputting the first input data into a predetermined large model to obtain first output data; the first template is used to indicate output requirements; the first output data comprises feature labels and / or descriptions; wherein the predetermined large model comprises a first model, a second model, and a large language model, the second model comprises a first sub-model and a second sub-model, the first sub-model is an image feature extraction branch, and the second sub-model is a related label processing branch, the initial label of the first picture is obtained by fusing the results of the first model and the second model; the initial label and the first template are input into a pre-trained large language model to obtain the first output data; the first model comprises an image feature extraction network, a Transformer encoder, a SwinTransformer module, and a linear layer; matching the first output data with a predetermined multi-level category label to determine a first category label sequence to which the first image data belongs; storing the first image data in a folder corresponding to the first category label sequence; wherein the multi-level category label comprises a basic category label and a dynamic category label, wherein the basic category label is maintained by configuration, and the dynamic category label is dynamically maintained in the process of classifying image data containing railway elements; The method further comprises: determining whether the number of image data in the folder corresponding to the first category label sequence exceeds a second threshold value; when the number exceeds, clustering the image data in the folder corresponding to the first category label sequence into at least two clusters, and determining the key feature label of each cluster; determining a new dynamic category label for each cluster based on the key feature label, and taking the new dynamic category label as a child node of a leaf node of the first category label sequence.

2. The method of claim 1, wherein, The first output data and the first image data are stored in the folder corresponding to the first category label sequence; the method further comprises: receiving an image data query request, the image data query request comprising second image data containing railway elements, the second image data comprising a second picture with railway elements; generating second input data based on the second image data and the first template, inputting the second input data into a predetermined large model to obtain second output data; the second output data comprises labels and / or descriptions; matching the second output data with a predetermined multi-level category label to determine a second category label sequence to which the second image data belongs; obtaining all third image data stored in the folder corresponding to the second category label sequence, and third output data corresponding to the third image data, wherein third input data is generated based on the third image data and the first template, the third input data is input into a predetermined large model to obtain third output data; determining one or more third output data that best matches the second output data; the third image data corresponding to the best matching third output data is taken as the query result of the image data query request and returned.

3. The method of claim 2, wherein, Before obtaining all third image data stored in the folder corresponding to the second category label sequence, the method further comprises: determining whether the number of third image data is greater than a first threshold value; if not, all third image data is taken as the query result of the image data query request; if yes, obtaining all third image data stored in the folder corresponding to the second category label sequence.

4. The method of claim 3, wherein, Determining one or more third output data that best matches the second output data comprises: when the number of third image data is greater than the first threshold value, determining the matching degree of each third output data with the second output data, and sorting the third output data according to the matching degree; determining one or more third output data that best matches the second output data based on the sorting; After taking the third image data corresponding to the best matching third output data as the query result of the image data query request and returning, the method further comprises: receiving a continue query request for the second image data, determining one or more third output data that second matches the second output data based on the sorting, and taking the third image data corresponding to the second matching third output data as the query result of the continue query request and returning.

5. The method according to any one of claims 1 to 4, characterized in that, The image data includes pictures or videos; wherein the first large model is used to determine the output data for the pictures, and the second large model is used for videos, the first large model is obtained by adjusting the multi-modal large model pre-trained on the public data set through the railway track transportation professional picture data set; the second large model is obtained by adjusting the multi-modal large model pre-trained on the public data set through the railway track transportation professional video data set. 6.A large model-based railway image data processing apparatus, characterized by comprising: comprises: an obtaining module for obtaining first image data containing railway elements, the first image data including a first picture with railway elements; a generating module for generating first input data based on the first image data and a first template, inputting the first input data into a predetermined large model to obtain first output data; The first template is used to indicate output requirements; the first output data includes feature labels and / or descriptions; wherein the pre-determined large model includes a first model, a second model, and a large language model, the second model includes a first sub-model and a second sub-model, the first sub-model is an image feature extraction branch, and the second sub-model is a related label processing branch, and an initial label of the first picture is obtained by fusing the results of the first model and the second model; the initial label and the first template are input into the pre-trained large language model to obtain the first output data; the first model includes an image feature extraction network, a Transformer encoder, a SwinTransformer module, and a linear layer; The matching module is configured to match the first output data with pre-determined multi-level category labels to determine a first category label sequence to which the first image data belongs; The storage module is configured to store the first image data in a folder corresponding to the first category label sequence; Wherein, the multi-level category labels include basic category labels and dynamic category labels, wherein the basic category labels are maintained by configuration, and the dynamic category labels are dynamically maintained in the process of classifying image data containing railway elements; Further comprising: The second judgment module is configured to determine whether the number of image data in the folder corresponding to the first category label sequence exceeds a second threshold value; The maintenance module is configured to, when the number exceeds the second threshold value, cluster the image data in the folder corresponding to the first category label sequence into at least two clusters, determine a key feature label for each cluster, determine a new dynamic category label for each cluster based on the key feature label, and take the new dynamic category label as a child node of a leaf node of the first category label sequence.

7. A server comprising a memory, a processor, the memory having stored therein a computer program executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and when the computer executable instructions are called and executed by the processor, the computer executable instructions cause the processor to execute the method of any one of claims 1-5.

Images