Railway image data processing method and device based on large model

Abstract

The invention provides a railway image data processing method and device based on a large model. The method comprises the steps that first image data containing railway elements are acquired, the first image data comprise a first picture, and the first picture contains the railway elements; generating first input data based on the first image data and a first template, and inputting the first input data into a predetermined large model to obtain first output data; the first template is used for indicating an output requirement; the first output data comprises feature tags and / or descriptions; matching the first output data with predetermined multi-level category labels, and determining a first category label sequence to which the first image data belongs; and storing the first image data in a folder corresponding to the first category label sequence. In this way, the image data containing the railway elements can be classified and managed by means of the capacity of the large model, and the usability of the data is improved while the efficiency of classification management is improved.

Description

Technical Field

[0001] The present application relates to the field of railway image management, and in particular, to a railway image data processing method and device based on a large model. Background Art

[0002] Railway image and video resource intelligent analysis technology stems from the important position of railway transportation in economic and social development. As an important part of the comprehensive transportation system, the operation and maintenance of railways are of vital importance. In recent years, railway transportation has achieved remarkable achievements, however, its complex transportation network and equipment structure also bring challenges.

[0003] Traditional image analysis methods mainly rely on manual analysis, which leads to the following problems: on the one hand, the huge amount of detection data brings heavy workload to identification personnel, resulting in a large consumption of manpower and material resources; on the other hand, the accuracy and coverage of image analysis are directly related to the ability and experience of personnel, making it difficult to ensure the quality of analysis; in addition, since manual analysis is generally performed after image acquisition, the timeliness of analysis cannot be ensured. Currently, with the development of artificial intelligence technology, deep learning algorithms have made breakthrough progress in the field of image analysis. Image intelligent analysis aims to automatically analyze the category, location, status and other information of key targets in images through machine algorithms.

[0004] Traditional image analysis methods based on rules and feature engineering can no longer meet the railway system's needs for large-scale data processing and intelligent analysis. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a railway image data processing method and device based on a large model, so as to alleviate the technical problems of high management cost of image data containing railway elements and low data availability.

[0006] In a first aspect, the present invention provides a railway image data processing method based on a large model, comprising: Acquire first image data containing railway elements, wherein the first image data includes a first picture, and the first picture contains railway elements; Based on the first image data and the first template, first input data is generated, and the first input data is input into a predetermined large model to obtain first output data; the first template is used to indicate output requirements; the first output data includes feature labels and / or descriptions; Matching the first output data with a predetermined multi-level category label to determine a first category label sequence to which the first image data belongs; The first image data is stored in a folder corresponding to the first category label sequence.

[0007] In an optional implementation, the first output data and the first image data are stored in a folder corresponding to the first category label sequence; the method further includes: receiving an image data query request, wherein the image data query request includes second image data containing railway elements, wherein the second image data includes a second picture, and the second picture contains railway elements; Based on the second image data and the first template, second input data is generated, and the second input data is input into a predetermined macro model to obtain second output data; the second output data includes a label and / or a description; Matching the second output data with a predetermined multi-level category label to determine a second category label sequence to which the second image data belongs; Acquire all third image data stored in the folder corresponding to the second category label sequence, and third output data corresponding to the third image data, wherein third input data is generated based on the third image data and the first template, and the third input data is input into a predetermined large model to obtain third output data; Determine one or more third output data that best matches the second output data; and return the third image data corresponding to the best matching third output data as the query result of the image data query request.

[0008] In an optional implementation, before acquiring all third image data stored in the folder corresponding to the second category label sequence, the method further includes: Determining whether the amount of the third image data is greater than a first threshold; If not, taking all the third image data as the query result of the image data query request; If yes, then acquiring all third image data stored in the folder corresponding to the second category label sequence is performed.

[0009] In an optional implementation manner, determining one or more third output data that best matches the second output data includes: When the amount of the third image data is greater than a first threshold, determining a matching degree between each of the third output data and the second output data, and sorting the third output data according to the matching degree; determining one or more third output data that best matches the second output data based on the ranking; After returning the third image data corresponding to the most matched third output data as the query result of the image data query request, the method further includes: A continued query request for the second image data is received, one or more third output data that second matches the second output data are determined based on the sorting, and the third image data corresponding to the second matching third output data is returned as a query result of the continued query request.

[0010] In an optional implementation, the multi-level category labels include basic category labels and dynamic category labels, wherein the basic category labels are maintained by configuration, and the dynamic category labels are dynamically maintained during the process of classifying image data containing railway elements.

[0011] In an optional embodiment, the method further comprises: Determine whether the amount of image data in the folder corresponding to the first category label sequence exceeds a second threshold; When the number exceeds the limit, clustering the image data in the folder corresponding to the first category label sequence into at least two clusters, and determining the key feature label of each cluster; A new dynamic category label of each cluster is determined based on the key feature label, and the new dynamic category label is used as a child node of a leaf node of the first category label sequence.

[0012] In an optional embodiment, the image data includes pictures or videos; wherein, a first large model is used to determine output data for pictures, and a second large model is used for videos, wherein the first large model is obtained by adjusting the parameters of a multimodal large model pre-trained on a public data set on a professional picture data set for railway transit; and the second large model is obtained by adjusting the parameters of a multimodal large model pre-trained on a public data set on a professional video data set for railway transit.

[0013] In a second aspect, the present invention provides a railway image data processing device based on a large model, comprising: An acquisition module, used for acquiring first image data containing railway elements, wherein the first image data includes a first picture, and the first picture contains railway elements; A generating module, configured to generate first input data based on the first image data and a first template, and input the first input data into a predetermined large model to obtain first output data; the first template is used to indicate output requirements; the first output data includes feature labels and / or descriptions; a matching module, configured to match the first output data with a predetermined multi-level category label to determine a first category label sequence to which the first image data belongs; A storage module is used to store the first image data in a folder corresponding to the first category label sequence.

[0014] In an optional implementation, the first output data and the first image data are stored in a folder corresponding to the first category label sequence; the device further includes: A receiving module, configured to receive an image data query request, wherein the image data query request includes second image data containing railway elements, wherein the second image data includes a second picture, and the second picture contains railway elements; The generating module is further used to generate second input data based on the second image data and the first template, and input the second input data into a predetermined macro model to obtain second output data; the second output data includes a label and / or a description; The matching module is further used to match the second output data with a predetermined multi-level category label to determine a second category label sequence to which the second image data belongs; A return module is used to obtain all third image data stored in the folder corresponding to the second category label sequence, and third output data corresponding to the third image data, wherein, based on the third image data and the first template, third input data is generated, and the third input data is input into a predetermined large model to obtain third output data; one or more third output data that best matches the second output data is determined; and the third image data corresponding to the best matching third output data is used as the query result of the image data query request and returned.

[0015] In an optional embodiment, the device further comprises: A first judging module, configured to judge whether the amount of the third image data is greater than a first threshold; The returning module is specifically configured to, if not, use all the third image data as the query result of the image data query request; if yes, execute acquisition of all the third image data stored in the folder corresponding to the second category label sequence.

[0016] In an optional implementation manner, the returning module is specifically configured to, when the amount of the third image data is greater than a first threshold, determine a matching degree between each of the third output data and the second output data, and sort the third output data according to the matching degree; determining one or more third output data that best matches the second output data based on the ranking; After returning the third image data corresponding to the most matched third output data as the query result of the image data query request, the method further includes: A continued query request for the second image data is received, one or more third output data that second matches the second output data are determined based on the sorting, and the third image data corresponding to the second matching third output data is returned as a query result of the continued query request.

[0017] In an optional implementation, the multi-level category labels include basic category labels and dynamic category labels, wherein the basic category labels are maintained by configuration, and the dynamic category labels are dynamically maintained during the process of classifying image data containing railway elements.

[0018] In an optional embodiment, the device further comprises: A second judgment module is used to judge whether the amount of image data in the folder corresponding to the first category label sequence exceeds a second threshold; A maintenance module is used to cluster the image data in the folder corresponding to the first category label sequence, divide it into at least two clusters, and determine the key feature label of each cluster when it exceeds the limit; determine a new dynamic category label for each cluster based on the key feature label, and use the new dynamic category label as a child node of the leaf node of the first category label sequence.

[0019] In an optional embodiment, the image data includes pictures or videos; wherein, a first large model is used to determine output data for pictures, and a second large model is used for videos, wherein the first large model is obtained by adjusting the parameters of a multimodal large model pre-trained on a public data set on a professional picture data set for railway transit; and the second large model is obtained by adjusting the parameters of a multimodal large model pre-trained on a public data set on a professional video data set for railway transit.

[0020] In a third aspect, the present invention provides a server, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the method described in any one of the aforementioned implementation modes when executing the computer program.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to execute the method described in any one of the aforementioned implementations.

[0022] The present invention provides a railway image data processing method and device based on a large model. By acquiring first image data containing railway elements, the first image data includes a first picture, and the first picture contains railway elements; based on the first image data and a first template, first input data is generated, and the first input data is input into a predetermined large model to obtain first output data; the first template is used to indicate output requirements; the first output data includes feature labels and / or descriptions; the first output data is matched with a predetermined multi-level category label to determine the first category label sequence to which the first image data belongs; the first image data is stored in a folder corresponding to the first category label sequence. In this way, the image data containing railway elements can be classified and managed with the help of the ability of the large model, which improves the efficiency of classification management while improving the availability of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 A schematic flow chart of a railway image data processing method based on a large model provided in an embodiment of the present application; Figure 2 A schematic flow chart of another railway image data processing method based on a large model provided in an embodiment of the present application; Figure 3 A schematic diagram of a multi-level category label structure provided in an embodiment of the present application; Figure 4 A schematic flow chart of another railway image data processing method based on a large model provided in an embodiment of the present application; Figure 5 A schematic diagram of a model structure provided in an embodiment of the present application; Figure 6 Another schematic diagram of a model structure provided in an embodiment of the present application; Figure 7 Another schematic diagram of a model structure provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of a railway image data processing device based on a large model provided in an embodiment of the present application; Fig. 9 A schematic flow chart of a railway image data processing method based on a large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0026] Figure 1 A schematic diagram of a railway image data processing method based on a large model provided in an embodiment of the present application. Figure 1 As shown, the method includes: S110, obtaining first image data containing railway elements, where the first image data includes a first picture, and the first picture contains railway elements.

[0027] The image data includes a picture or a video; the first picture may be an image frame in a video.

[0028] In the embodiment of the present application, the image data can be classified in batches, or a single image data can be classified into an image database that has been classified. Based on this, the first image data can be a single image data or a batch of multiple image data.

[0029] For example, the user may first determine the image data to be classified, and then trigger a classification request for the data to be classified. After receiving the classification request, the first image data containing the railway element is obtained.

[0030] The method can be implemented through a cloud server or a local terminal.

[0031] S120, based on the first image data and the first template, generate first input data, input the first input data into a predetermined large model, and obtain first output data; the first template is used to indicate output requirements; the first output data includes feature labels and / or descriptions.

[0032] The module here can be a pre-defined multiple template, which is mainly used to indicate the expected output result of the large model output. The template can be a text string with a preset format, which includes a reserved paragraph for selectable or to-be-entered text. For example, the template can be "For the following [pictures / videos], perform [label output / description output] [attachment]", where [pictures / videos] and [label output / description output] are optional reserved paragraphs, and one can be selected to enter; the [attachment] can be used to input the first image data, which is also a reserved paragraph. Finally, the first input data can be "For the following [pictures], perform [label output] [first image data].

[0033] In some embodiments, Figure 4 As shown, the present application may provide a template configuration interface, which may include input boxes for each reserved paragraph, and input will be made in each input box.

[0034] As an example, the first largest model is used to determine the output data for pictures, and the second largest model is used for videos. The first largest model is obtained by adjusting the parameters of the multimodal large model pre-trained on the public dataset on the railway track transit professional image dataset; the second largest model is obtained by adjusting the parameters of the multimodal large model pre-trained on the public dataset on the railway track transit professional video dataset.

[0035] The process of adjusting the parameters of a large model can be achieved through the following steps: For railway image data, we first need to create a label table (the label can be the basic category label in the multi-level category label in the following text), which includes possible target objects such as rails, trains, and contact lines, and covers all possible labels as much as possible. Then, use the labeling tool to manually label the image and organize the label information into a unified description template: "There are {label} in the scene". In this way, a corresponding description file can be generated for each image. If a deep learning method based on a graph convolutional neural network, CNN-RNN and other structures is used, the data needs to be preprocessed. This includes steps such as data cleaning and denoising, image defogging, data enhancement, and sample balancing. These preprocessing operations can improve the performance and effect of intelligent analysis tasks of railway images and video resources, so that they can better adapt to the complex transportation environment and task requirements of the railway system. However, these operations may increase the complexity of model reasoning and the length of training. Therefore, under comprehensive consideration, it may be more appropriate to use a multimodal large model method to complete the intelligent analysis task. This method does not require complex data preprocessing. It only needs to unify the input test and training data to a standard size to complete the model training and reasoning process.

[0036] The advantage of the multimodal large model is that its structure contains encoder modules specifically for processing images and text. These modules can make use of pre-trained parameter files, thus avoiding the large amount of time and computing resources required to train the model from scratch. Compared with other deep learning methods, this pre-trained model can achieve convergence faster and generally has better performance. By utilizing pre-trained image encoders and text encoders, the multimodal large model can better understand and represent the input image and text information, thereby improving the ability of the entire model in modeling the association between images and text. The use of this model can not only save training time and resource costs, but also provide a more efficient and accurate solution for semantic understanding and information extraction tasks between images and text.

[0037] Parameter optimization refers to the process of adjusting the parameters of a model to improve its performance and generalization ability when training a large multimodal model. It mainly includes hyperparameter adjustment, learning rate scheduling, regularization, optimizer selection, etc. By optimizing parameters in these aspects, the performance and generalization ability of the model can be improved, so that it can better adapt to professional application scenarios and achieve better results.

[0038] Multimodal large models have achieved good results on public datasets, such as image classification, text classification, and image annotation. Therefore, after adjusting the parameters of the pre-trained multimodal large model, it can be applied to the railway and track transportation professional dataset to achieve good results. By fine-tuning the railway images and related text data, the model can better adapt to the characteristics and needs of the railway field, thereby achieving more accurate and efficient intelligent analysis of railway images. Such a transfer learning method can make full use of the knowledge learned by the pre-trained model on large-scale data, avoiding the time and resource consumption of training the model from scratch, and can also quickly achieve good results in the field of railway and track transportation.

[0039] Since the output of the multimodal large model is usually in the form of image description, the keyword extraction algorithm can be used to process these descriptions to obtain a set of image labels. The keyword extraction algorithm can identify and extract representative and important keywords from the text, which can usually effectively express the content and characteristics of the image. By converting the image description into a set of keywords, the information expressed by the image can be understood more intuitively, and it also provides an important reference for subsequent image classification, retrieval and other tasks. This processing method can not only make full use of the output of the multimodal large model, but also extract more informative and descriptive image labels, thereby improving the effect and accuracy of intelligent analysis of railway images.

[0040] Regarding the multimodal large model architecture adopted, considering that railway images and video data have different modal characteristics, it is necessary to make full use of the characteristics of different modal data so that each model can focus on processing a specific type of data and maximize its advantages. Therefore, two model structures are used to complete the multi-label output for railway images.

[0041] In view of the multimodal characteristics of video data, a model that can effectively process three different modal data, image, text and voice, is used to achieve multi-label output for railway videos. The design of this model structure fully considers the diversity and complexity of video data, and improves the ability to analyze and understand railway videos by integrating information from different modal data. This model structure can more comprehensively mine useful information in video data.

[0042] As an example, the structure of the first large model or the second large model can be as follows Figure 5 The multimodal large model shown in Figure 1 uses three main components: image feature extraction network, Transformer encoder, and SwinTransformer module and linear layer.

[0043] The image feature extraction network adopts the Vision Transformer (ViT) structure, which is an advanced deep learning model based on the Transformer architecture. Specifically, the workflow of the ViT model is as follows: First, the first image is divided into fixed-size patches (patches), with a common size of 16x16 pixels. These image patches are then flattened into one-dimensional vectors and embedded through a linear layer to convert them into the format required by the Transformer. This embedding operation enables each image patch to be represented as a fixed-size vector so that it can be processed in subsequent network layers. ViT introduces position encoding after the embedding layer. This step ensures that the spatial information of the pixels in the image is preserved, which is crucial for image understanding tasks. After embedding and position encoding, the size of the image encoding becomes (batch_size, sequence_length, hidden_size), where batch_size is the batch size, sequence_length is the number of image patches, and hidden_size is the dimension of the embedding vector for each image patch.

[0044] This design not only improves the model's ability to understand image content, but also has significant advantages in computational efficiency. The combination of flattened image blocks and position encoding allows data to be directly input into the Transformer encoder, facilitating subsequent image feature processing and classification tasks. In this way, ViT can efficiently learn image features and lay a solid foundation for further model processing.

[0045] Each Transformer encoder of this multimodal large model consists of two sub-layer connection structures: the first sub-layer connection structure includes a self-attention layer (Bi Sef-attetion) and a normalization layer (Add&Normalize) and a residual connection; the second sub-layer connection structure includes a feedforward fully connected sub-layer (FFN) and a normalization layer and a residual connection.

[0046] The linearized feature map of the Transformer encoder first passes through a self-attention layer for fine feature interactions, and then passes through a linear layer to reduce the feature dimension to D. This adjusted feature sequence is then fed into the cross-attention module of each decoder layer. This model does not introduce learnable or fixed positional encodings in the encoder features.

[0047] The Swin Transformer module is constructed by replacing the standard multi-head self-attention (MSA) module in the Transformer with a shifted window-based self-attention module (W-MSA), while keeping the structure of the original layer unchanged. The module consists of a specially designed shifted window multi-head self-attention module followed by two layers of multi-layer perceptrons (MLPs), where the MLP contains a GELU nonlinear activation function. Before each self-attention module and MLP, a layer normalization (LN) layer is applied, and a residual connection is added after each module to optimize the information flow and enhance the robustness of the model.

[0048] The Swin Transformer module calculates self-attention in non-overlapping local windows. These windows evenly divide the image, reducing the computational complexity to a linear relationship between the window size and the image size, significantly improving computational efficiency. The Swin Transformer module can flexibly adapt to images of different resolutions. By adjusting the size and number of windows, the model can efficiently process images of various sizes without wasting computing resources due to changes in resolution.

[0049] Finally, after a linear layer, the image multi-label classification task can be completed. Based on this architecture, it can show excellent performance in the railway professional image label classification task and achieve better classification results.

[0050] As another example, the structure of the first large model or the second large model can be as follows: Figure 6 The multimodal large model shown in the figure has a whole structure divided into two main parts: one is the image feature extraction branch, and the other is the related label processing branch.

[0051] The image feature extraction branch is built on the convolutional neural network ConvNeXt v2. ConvNeXt v2 operates in a fully convolutional manner, and its learning signal is generated by randomly masking the original input visual information and using a high masking ratio. The model then predicts the masked part based on the remaining contextual information. ConvNeXt v2 introduces a global response normalization (GRN) layer to enhance the model's expressiveness and generalization capabilities. After simple image preprocessing, the image is fed into the ConvNeXt v2 pre-trained model to extract features.

[0052] The related label processing branch combines the multi-head attention mechanism and the graph neural network to learn the relationship between labels. First, the embedding vectors of the image-related labels, namely Label Embeddings, such as Z1, Z2, ..., Zn, are obtained, and the cosine similarity between these embedding vectors is calculated. Based on the similarity and the preset threshold and weight parameters, an adjacency matrix is ​​constructed. Using the adjacency matrix, the values ​​of the query (Q), key (K), and value (V) can be obtained respectively through different linear layers. After being processed by the attention mechanism and passing through the softmax layer, different subgraphs can be obtained. These subgraphs are connected to each other to form a new graph structure and represented in matrix form. Then, this newly constructed matrix is ​​input into the Graph Attention Networks (GAN).

[0053] By performing matrix multiplication on the image feature vectors obtained from the two branches and the label representation learned by the graph neural network, we can get the corresponding prediction score, as shown on the far right of the model diagram. On this basis, by setting an appropriate threshold, we can accurately determine the label corresponding to the image.

[0054] As another example, the structure of the predetermined large model can be as follows Figure 7 As shown, the model may include a first model, a second model and a large language model, and the second model may include a first sub-model and a second sub-model. The first model may be as follows Figure 5 The model shown in FIG. 1 , the second model can be as follows Figure 6The model shown, wherein the first sub-model can be an image feature extraction branch, and the second sub-model can be a related label processing branch, and the initial label of the first picture can be obtained by fusing the results of the first model and the second model. The initial label and the first template are input into a pre-trained large language model to obtain the first output data. The large language model here. Among them, large language models refer to those deep learning models with a large number of parameters, which are usually trained on large-scale text data to capture the complex structure and pattern of language. The large language models used in the embodiments of the present application may include: BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer) series, T5 (Text-to-Text Transfer Transformer), ALBERT (A Lite BERT), RoBERTa (Robustly Optimized BERT Pretraining Approach) and RoBERTa, etc.

[0055] The step of fusing the results of the first model and the second model may include multiple steps, and weighted fusion may be performed. For example, for weighted fusion, the weight may be predetermined, and may be determined specifically according to the recognition accuracy of the model during the training process.

[0056] In some embodiments, data preprocessing is a crucial step before data analysis and modeling. It involves a series of preparations such as cleaning, integration, transformation and reduction of raw data, aiming to improve the quality and availability of data. For image datasets, preprocessing is particularly important because it is directly related to the efficiency of model learning and the final performance. For image data, the preprocessing process usually includes the following steps: Unify image size: adjust all images to a fixed size to provide a unified input format for the model and simplify subsequent image processing steps. Implement data enhancement: enhance images by random flipping, adjusting color attributes (such as contrast and saturation), and applying affine transformations to expand the diversity of the dataset and improve the model's adaptability to image transformations. Standardize image data: standardize the image by subtracting the mean of each channel and dividing by the standard deviation to make the image data more evenly distributed, which helps to speed up the convergence of the model and improve the stability of training. After the above preprocessing steps, the image tensor obtained will be fed into different models as input for training and testing, thereby ensuring that the model can learn effective features from high-quality data, thereby improving the accuracy and stability of prediction.

[0057] Since the training dataset used is a railway transportation dataset, the images used need to be annotated in advance. In the training process of the railway transportation dataset, image preprocessing and annotation are key steps. Figure 5 For the model shown in the figure, its core feature extraction module uses the pre-trained Vision Transformer (ViT) model as the backbone. This pre-trained model has learned common feature representations on large-scale datasets, so before starting training, you need to download the corresponding pre-trained parameters.

[0058] Using the pre-trained ViT model can significantly save training time, reduce computational complexity, and improve the performance of the model on downstream tasks. Figure 5 The rest of the model shown is trained without pre-trained parameters.

[0059] During the training phase, multiple parameters will be configured, including training cycle (epoch), batch size (batch_size), and image path (image_path). During the training process, each epoch will record and display the training progress, including the loss and F1 value of each batch. At the same time, after each epoch, the system will calculate and report the average precision, recall, and F1 value of the epoch, so that we can monitor the training effect and performance of the model.

[0060] In order to optimize Figure 6 The model shown in the figure improves training efficiency and reduces training complexity. The backbone of the model uses a pre-trained ConvNext v2 model. The ConvNext v2 model is pre-trained on a large-scale dataset and can extract high-level features of images, which helps speed up training and improve model performance.

[0061] In such Figure 6 In the model shown, the generation of label embedding vectors is also crucial. To this end, advanced natural language processing architectures such as Bert or Roberta are used to encode the features of the labels. These models are able to learn deep semantic representations of labels, thereby improving the consistency of generated images with labels. For datasets such as COCO and VOC, trained model parameters are usually provided, which are usually saved in the form of pkl files. The model can directly read these files and use the parameters in them for training or testing, which greatly simplifies the data preparation process.

[0062] like Figure 5 The model shown is similar to Figure 6The model shown also needs to set a series of parameters during training, such as training cycle (epoch), batch size (batch_size), etc. When evaluating the performance of the model, precision, recall and F1 value are used as evaluation indicators.

[0063] For both models, tests are performed regularly during the training process to evaluate the performance of the model at different epochs. The model will select the epoch with the best performance and save the model architecture and weight parameters corresponding to that epoch, as well as the parameter configuration during training, for subsequent use and further analysis.

[0064] In order to develop a high-performance multi-label classification model for the railway transit dataset, this paper provides the option of adjusting the model parameters and allows for customized modifications to certain parts of the model architecture. This process aims to explore different model configurations to find the best combination of parameters and architecture design.

[0065] During the model tuning and optimization process, the model will be trained under multiple configurations and its performance on the training set and test set will be compared in detail. The evaluation indicators will include key indicators such as classification accuracy, recall rate, F1 score, etc.

[0066] The model configuration that performs best on the evaluation indicators will be selected. This selected model will be considered the best multi-label classification model for the railway track transportation dataset, which can not only accurately identify and classify multiple labels in the image, but also has good generalization ability and robustness. In this way, the practicality of the model and its application value in the professional field are ensured.

[0067] The input of the model receives images and corresponding labels, and the output generates labels predicted by the model. By comparing and analyzing the predicted labels with the actual labels, the performance of the model is monitored and evaluated in real time, which helps ensure that the model can maintain high accuracy and reliable stability in practical applications.

[0068] In terms of model deployment, you can choose local deployment, that is, deploy the complete architecture, parameters and related supporting files of the model to the specified device. This deployment mode facilitates local training and debugging of the model, and also simplifies the use and maintenance process of the model.

[0069] S130, matching the first output data with a predetermined multi-level category label to determine a first category label sequence to which the first image data belongs.

[0070] The first output data is text, which can be a label or description; the matching degree of the label or description with each branch of the multi-level category label can be determined. Each branch is composed of all category labels from the root node to the leaf node, and each branch constitutes a category label sequence.

[0071] The matching degree can be calculated by a text similarity algorithm. The text similarity matching algorithm is an important application in the field of natural language processing, which is used to measure the similarity between two texts. The following algorithm can be used for calculation to determine the most similar category label sequence: String similarity: Calculate the similarity by comparing the number of identical characters in two texts. This method is simple and easy to implement, but it ignores the semantic information of the text.

[0072] ‌Simhash: This is a local sensitive hashing algorithm that is suitable for processing massive text data. It can quickly process web page text and deduplicate it.

[0073] ‌Word vector: A low-dimensional real number vector is obtained through training to represent the text. This method takes into account the semantic information of the text and can better reflect the meaning of the text.

[0074] ‌Cosine similarity: measures the similarity between texts by calculating the cosine value of the angle between two vectors. This method is widely used in search engines, automatic question answering and other fields.

[0075] ‌TF-IDF: This is a statistical model that evaluates the importance of a document by calculating the weight of words. TF-IDF can be used in conjunction with cosine similarity to calculate the similarity between documents.

[0076] Deep learning algorithms: such as ‌DSSM, ‌ConvNet, and ‌Siamese LSTM, which can learn more complex text representations and thus improve the accuracy of similarity calculations.

[0077] Other algorithms: such as ‌Jaccard similarity, ‌Levenshtein edit distance, etc. These algorithms can also be used to calculate the similarity between texts.

[0078] In practical applications, choosing an appropriate algorithm requires considering the characteristics of the data, the limitations of computing resources, and the expected results. For example, for large-scale text data, Simhash is favored due to its efficient processing capabilities; while for scenarios that require precise semantic understanding, deep learning algorithms may be more suitable.

[0079] S140: Store the first image data in a folder corresponding to the first category label sequence.

[0080] In some embodiments, the folder may be named based on the first category tag sequence.

[0081] Wherein, in some embodiments, Figure 3 As shown, the multi-level category labels include basic category labels and dynamic category labels, wherein the basic category labels are maintained by configuration, and the dynamic category labels are dynamically maintained in the process of classifying image data containing railway elements. When there are too many images in a folder, it is not conducive to the use of data. Based on this, the folder can be split according to the maintenance principle of dynamic category labels. Based on this, the method also includes: judging whether the number of image data in the folder corresponding to the first category label sequence exceeds the second threshold; when it exceeds, clustering the image data in the folder corresponding to the first category label sequence, dividing it into at least two clusters, and determining the key feature label of each cluster; determining a new dynamic category label for each cluster based on the key feature label, and using the new dynamic category label as a child node of the leaf node of the first category label sequence.

[0082] For the image search task, it is necessary to input the target image, that is, to search for related images or information by inputting an image. It is a common image search technology. Image search analyzes the visual features of the input image and uses a similarity matching algorithm to search for images with similar visual features. Using convolutional neural networks, image search can have a deeper understanding of the content of the image, including objects, scenes, emotions and other information. However, compared with large multimodal models, this deep learning method requires manual annotation of data and manual adjustment of parameters, which consumes manpower and time costs, and has poor generalization effects. Therefore, the following model structure will be used to complete it: In some embodiments, the first output data and the first image data are stored in a folder corresponding to the first category label sequence; the method may also include the following steps: Figure 2 As shown: S210, receiving an image data query request, the image data query request including second image data containing railway elements, the second image data including a second picture, and the second picture containing railway elements; S220, generating second input data based on the second image data and the first template, and inputting the second input data into a predetermined large model to obtain second output data; the second output data includes a label and / or a description; S230, matching the second output data with a predetermined multi-level category label to determine a second category label sequence to which the second image data belongs; S240, acquiring all third image data stored in the folder corresponding to the second category label sequence, and third output data corresponding to the third image data, wherein third input data is generated based on the third image data and the first template, and the third input data is input into a predetermined large model to obtain third output data; S250, determining one or more third output data that best matches the second output data; and returning the third image data corresponding to the best matching third output data as the query result of the image data query request.

[0083] Since the display space is usually limited and the amount of matching image data may be large, only the most relevant image data may be displayed during the display.

[0084] In addition, before obtaining all the third image data stored in the folder corresponding to the second category label sequence, the method also includes: determining whether the amount of the third image data is greater than a first threshold; if not, taking all the third image data as the query result of the image data query request; if yes, executing to obtain all the third image data stored in the folder corresponding to the second category label sequence.

[0085] Based on this, the step of determining one or more third output data that best matches the second output data in the aforementioned S250 can be implemented by the following steps: when the number of the third image data is greater than the first threshold, determining the matching degree of each third output data with the second output data, and sorting the third output data according to the matching degree; determining one or more third output data that best matches the second output data based on the sorting; After returning the third image data corresponding to the most matching third output data as the query result of the image data query request, the method also includes: receiving a continued query request for the second image data, determining one or more third output data that second matches the second output data based on the sorting, and returning the third image data corresponding to the second matching third output data as the query result of the continued query request.

[0086] Figure 8 A schematic diagram of the structure of a railway image data processing device based on a large model provided in this embodiment is shown in FIG. Figure 8 As shown, the device may include: An acquisition module 501 is used to acquire first image data containing railway elements, where the first image data includes a first picture, and the first picture contains railway elements; A generating module 502 is used to generate first input data based on the first image data and the first template, and input the first input data into a predetermined large model to obtain first output data; the first template is used to indicate output requirements; the first output data includes feature labels and / or descriptions; A matching module 503 is used to match the first output data with a predetermined multi-level category label to determine a first category label sequence to which the first image data belongs; The storage module 504 is configured to store the first image data in a folder corresponding to the first category label sequence.

[0087] In some embodiments, the first output data and the first image data are stored in a folder corresponding to the first category label sequence; the device further comprises: A receiving module, configured to receive an image data query request, wherein the image data query request includes second image data containing railway elements, wherein the second image data includes a second picture, and the second picture contains railway elements; The generating module is further used to generate second input data based on the second image data and the first template, and input the second input data into a predetermined macro model to obtain second output data; the second output data includes a label and / or a description; The matching module is also used to match the second output data with a predetermined multi-level category label to determine a second category label sequence to which the second image data belongs; A return module is used to obtain all third image data stored in a folder corresponding to a second category label sequence, and third output data corresponding to the third image data, wherein, based on the third image data and the first template, third input data is generated, and the third input data is input into a predetermined large model to obtain third output data; one or more third output data that best matches the second output data is determined; and the third image data corresponding to the best matching third output data is used as a query result of the image data query request and returned.

[0088] In some embodiments, the apparatus further comprises: A first judging module, used for judging whether the amount of the third image data is greater than a first threshold; The returning module is specifically used for, if not, taking all the third image data as the query result of the image data query request; if yes, executing to obtain all the third image data stored in the folder corresponding to the second category label sequence.

[0089] In some embodiments, the return module is specifically used to: When the amount of the third image data is greater than the first threshold, determining a matching degree between each third output data and the second output data, and sorting the third output data according to the matching degree; determining one or more third output data that best matches the second output data based on the ranking; After returning the third image data corresponding to the most matched third output data as the query result of the image data query request, the method further includes: A continued query request for the second image data is received, one or more third output data that second matches the second output data are determined based on the ranking, and the third image data corresponding to the second matching third output data is returned as a query result of the continued query request.

[0090] In some embodiments, the multi-level category labels include basic category labels and dynamic category labels, wherein the basic category labels are maintained by configuration and the dynamic category labels are dynamically maintained during the process of classifying image data containing railway elements.

[0091] In some embodiments, the apparatus further comprises: A second judgment module is used to judge whether the amount of image data in the folder corresponding to the first category label sequence exceeds a second threshold; The maintenance module is used to cluster the image data in the folder corresponding to the first category label sequence, divide it into at least two clusters, and determine the key feature label of each cluster; determine the new dynamic category label of each cluster based on the key feature label, and use the new dynamic category label as the child node of the leaf node of the first category label sequence.

[0092] In some embodiments, the image data includes pictures or videos; wherein, a first large model is used to determine output data for pictures, and a second large model is used for videos, the first large model is obtained by adjusting the parameters of a multimodal large model pre-trained on a public data set on a professional picture data set for railway rail transit; the second large model is obtained by adjusting the parameters of a multimodal large model pre-trained on a public data set on a professional video data set for railway rail transit.

[0093] See also Fig. 9 As shown, the server 600 provided in the embodiment of the present application includes at least: a processor 601, a memory 602, and a computer program stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program, the railway image data processing method based on the large model provided in the embodiment of the present application is implemented.

[0094] The server 600 provided in the embodiment of the present application may also include a bus 603 connecting different components (including the processor 601 and the memory 602). The bus 603 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0095] The memory 602 may include a readable storage medium in the form of a volatile memory, such as a random access memory (RAM) 6021 and / or a cache memory 6022, and may further include a read-only memory (ROM) 6023. The memory 602 may also include a program tool 6025 having a group (at least one) of program modules 6024, and the program modules 6024 include but are not limited to an operating subsystem, one or more application programs, other program modules, and program data, each of which or some combination may include the implementation of a network environment.

[0096] The processor 601 may be a processing element or a collective term for multiple processing elements. For example, the processor 601 may be a central processing unit (CPU) or one or more integrated circuits configured to implement the railway image data processing method based on a large model provided in the embodiment of the present application. Specifically, the processor 601 may be a general-purpose processor, including but not limited to a CPU, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.

[0097] The server 600 can communicate with one or more external devices 604 (e.g., keyboards, remote controls, etc.), and can also communicate with one or more devices that enable users to interact with the server 600 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the server 600 to communicate with one or more other servers 600 (e.g., routers, modems, etc.). Such communication can be performed through an input / output (I / O) interface 605. In addition, the server 600 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 606. Figure 6 As shown, the network adapter 606 communicates with other modules of the server 600 via the bus 603. It should be understood that although Figure 6Not shown, other hardware and / or software modules may be used in conjunction with server 600, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0098] It should be noted that Fig. 9 The server 600 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0099] The computer-readable storage medium provided in the embodiment of the present application is introduced below. The computer-readable storage medium provided in the embodiment of the present application stores computer instructions, and when the computer instructions are executed by the processor, the railway image data processing method based on the large model provided in the embodiment of the present application is implemented. Specifically, the computer instructions can be built-in or installed in the processor, so that the processor can implement the railway image data processing method based on the large model provided in the embodiment of the present application by executing the built-in or installed computer instructions.

[0100] In addition, the railway image data processing method based on a large model provided in the embodiment of the present application can also be implemented as a computer program product, which includes a program code. When the program code is run on a processor, it implements the railway image data processing method based on a large model provided in the embodiment of the present application.

[0101] The computer program product provided in the embodiments of the present application may adopt one or more computer-readable storage media, and the computer-readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any suitable combination of the above. Specifically, more specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0102] The computer program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on a server such as a road management device. However, the computer program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the computer-readable storage medium may be any tangible medium containing or storing program code, and the program code may be used by or in combination with an instruction execution system, apparatus, or device.

[0103] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0104] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0105] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0106] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A railway image data processing method based on a large model, characterized in that: include: Acquire first image data containing railway elements, wherein the first image data includes a first picture, and the first picture contains railway elements; Generate first input data based on the first image data and the first template, and input the first input data into a predetermined large model to obtain first output data; The first template is used to indicate output requirements; the first output data includes feature tags and / or descriptions; Matching the first output data with a predetermined multi-level category label to determine a first category label sequence to which the first image data belongs; The first image data is stored in a folder corresponding to the first category label sequence.

2. The method according to claim 1, characterized in that The first output data and the first image data are stored in a folder corresponding to the first category label sequence; the method further includes: receiving an image data query request, wherein the image data query request includes second image data containing railway elements, wherein the second image data includes a second picture, and the second picture contains railway elements; Based on the second image data and the first template, second input data is generated, and the second input data is input into a predetermined macro model to obtain second output data; the second output data includes a label and / or a description; Matching the second output data with a predetermined multi-level category label to determine a second category label sequence to which the second image data belongs; Acquire all third image data stored in the folder corresponding to the second category label sequence, and third output data corresponding to the third image data, wherein third input data is generated based on the third image data and the first template, and the third input data is input into a predetermined large model to obtain third output data; Determine one or more third output data that best matches the second output data; and return the third image data corresponding to the best matching third output data as the query result of the image data query request.

3. The method according to claim 2, characterized in that Before acquiring all third image data stored in the folder corresponding to the second category label sequence, the method further includes: Determining whether the amount of the third image data is greater than a first threshold; If not, taking all the third image data as the query result of the image data query request; If yes, then acquiring all third image data stored in the folder corresponding to the second category label sequence is performed.

4. The method according to claim 3, characterized in that Determining one or more third output data that best matches the second output data includes: When the amount of the third image data is greater than a first threshold, determining a matching degree between each of the third output data and the second output data, and sorting the third output data according to the matching degree; determining one or more third output data that best matches the second output data based on the ranking; After returning the third image data corresponding to the most matched third output data as the query result of the image data query request, the method further includes: A continued query request for the second image data is received, one or more third output data that second matches the second output data are determined based on the sorting, and the third image data corresponding to the second matching third output data is returned as a query result of the continued query request.

5. The method according to claim 1, characterized in that The multi-level category labels include basic category labels and dynamic category labels, wherein the basic category labels are maintained by configuration, and the dynamic category labels are dynamically maintained during the process of classifying image data containing railway elements.

6. The method according to claim 5, characterized in that The method further comprises: Determine whether the amount of image data in the folder corresponding to the first category label sequence exceeds a second threshold; When the number exceeds the limit, clustering the image data in the folder corresponding to the first category label sequence into at least two clusters, and determining the key feature label of each cluster; A new dynamic category label of each cluster is determined based on the key feature label, and the new dynamic category label is used as a child node of a leaf node of the first category label sequence.

7. The method according to any one of claims 1 to 6, characterized in that: The image data includes pictures or videos; wherein, a first large model is used to determine output data for pictures, and a second large model is used for videos, wherein the first large model is obtained by adjusting the parameters of a multimodal large model pre-trained on a public data set on a railway track transit professional picture data set; the second large model is obtained by adjusting the parameters of a multimodal large model pre-trained on a public data set on a railway track transit professional video data set.

8. A railway image data processing device based on a large model, characterized in that: include: An acquisition module, used for acquiring first image data containing railway elements, wherein the first image data includes a first picture, and the first picture contains railway elements; A generating module, configured to generate first input data based on the first image data and the first template, and input the first input data into a predetermined large model to obtain first output data; The first template is used to indicate output requirements; the first output data includes feature tags and / or descriptions; a matching module, configured to match the first output data with a predetermined multi-level category label to determine a first category label sequence to which the first image data belongs; A storage module is used to store the first image data in a folder corresponding to the first category label sequence.

9. A server comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the method according to any one of claims 1 to 7.

Images