Cross-modal railway data retrieval method and device, equipment and storage medium

Abstract

The invention provides a cross-modal railway data retrieval method and device, equipment and a storage medium, and relates to the technical field of data processing, the method comprises the following steps: obtaining a retrieval word of cross-modal railway data; performing feature extraction on at least one piece of first railway data in the original railway database to obtain a first feature vector corresponding to each piece of first railway data; the type of the first railway data is any one of the following items: text data, picture data without texts and picture data containing texts; utilizing an intelligent retrieval model to determine cosine similarity between each first feature vector and the retrieval word; and determining target railway data according to the cosine similarity between each first feature vector and the search word. According to the method, feature extraction is carried out on the first railway data to obtain the first feature vector, and then image and text feature matching is carried out on the first feature vector and the feature vector converted from the retrieval word, so that cross-modal intelligent data retrieval between image-text typical modals is effectively realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a cross-modal railway data retrieval method, device, equipment and storage medium. Background Art

[0002] With the continuous development of information technology, the informatization and intelligentization of various industries are constantly advancing. The dissemination of data is no longer limited to text as a carrier. More carrier forms such as images, sounds, and videos continue to appear. Massive multimodal data are stored in different business systems in various industries. Traditional single-modal methods can only handle the retrieval and annotation of single-modal data, which is difficult to meet the needs of multimodal data retrieval. Therefore, related industries have successively carried out cross-modal retrieval and joint analysis technology research on multimodal data. With the development of informatization and digitalization, the data types generated by the railway system are becoming increasingly rich and complex, including text, images, videos, sensor data and other modalities, which brings challenges to data processing and analysis. The traditional railway data management and analysis model can no longer meet the needs of modern railway data services.

[0003] Cross-modal retrieval provides an effective method to solve the differences in cross-modal data and realize joint retrieval and query of data in different forms, so as to obtain more complete and accurate information about business objects. However, there are still two major difficulties. One is the difference in multimodal expression. There are large differences between the data expressions of different modalities, and it is difficult to measure the correlation between different modal data; the other is the semantic gap. The difficulty of feature analysis between different modalities lies in the gap between the underlying expression and the high-level semantics, and it is difficult to mine the high-level semantics of data between different modalities.

[0004] Therefore, how to effectively realize the retrieval of cross-modal railway data has become an urgent problem that needs to be solved in the industry. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a cross-modal railway data retrieval method, device, equipment and storage medium, which effectively realizes cross-modal intelligent data retrieval between typical modes of graphics and text.

[0006] In a first aspect, the present invention provides a method for retrieving cross-modal railway data, the method comprising the following steps: to obtain search terms for cross-modal railway data; Performing feature extraction on at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data; wherein the type of the first railway data is any one of the following: text data, image data without text, and image data containing text; Determine the cosine similarity between each of the first feature vectors and the search term using an intelligent retrieval model; Target railway data is determined according to the cosine similarity between each of the first feature vectors and the search term.

[0007] According to a cross-modal railway data retrieval method provided by the present invention, the use of an intelligent retrieval model to determine the cosine similarity between each of the first feature vectors and the search term includes: Using an intelligent retrieval model, extract key information from the search term to obtain a second feature vector; The cosine similarity between each of the first feature vectors and the second feature vectors is determined by using an intelligent retrieval model.

[0008] According to a cross-modal railway data retrieval method provided by the present invention, the target railway data is determined according to the cosine similarity between each of the first feature vectors and the search term, including: Calculating the cosine similarity between each of the first feature vectors and the search term; The first railway data corresponding to the maximum value of each of the cosine similarities is determined as the target railway data.

[0009] According to a cross-modal railway data retrieval method provided by the present invention, the feature extraction of at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data includes: When the first railway data is text data, using a bidirectional long short-term memory network model to extract key information from the text data to obtain a first text feature vector; The first text feature vector is determined as the first feature vector corresponding to the first railway data.

[0010] According to a cross-modal railway data retrieval method provided by the present invention, the bidirectional long short-term memory network model includes a word embedding layer, a bidirectional long short-term memory network layer and a conditional random field model layer; the bidirectional long short-term memory network model is used to extract key information from the text data to obtain a text feature vector, including: Input the text data into the word embedding layer to obtain a word vector matrix corresponding to the text data, wherein the word vector matrix is ​​a continuous vector representation; Inputting the word vector matrix corresponding to the text data into the bidirectional long short-term memory network layer to obtain a context feature sequence of at least two time steps; Using the conditional random field model layer, the context feature sequence of each time step is labeled, and the output score of each labeled context feature sequence relative to the label prediction sequence is calculated, and the output score is used to characterize the credibility corresponding to the context feature sequence of the time step; The conditional random field model layer is used to determine the text feature vector according to the output scores of each annotated context feature sequence relative to the label prediction sequence.

[0011] According to a cross-modal railway data retrieval method provided by the present invention, the feature extraction of at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data includes: In the case where the first railway data is picture data without text, inputting the picture data with text into a face recognition network to extract key information of the picture data with text; According to the key information of the picture data of the text, the face in the picture data of the text is framed by using the yolov5 model to obtain a face picture; Using a deep convolutional neural network to extract features from the face image, and obtain a feature vector corresponding to the face image; The feature vector corresponding to the face image is determined as the first feature vector corresponding to the first railway data.

[0012] According to a cross-modal railway data retrieval method provided by the present invention, the feature extraction of at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data includes: In the case where the first railway data is picture data containing text, the picture data containing text is input into a text detection model to identify a text area in the picture data containing text; According to the text region in the picture data containing text, using a text direction classifier to detect and obtain the text direction of the text instance in the text region; Based on the text direction, the text instance is rotated to a parallel direction to obtain a rotated text instance; Inputting the rotated text instance into a text recognition model to obtain a second text feature vector; The second text feature vector is determined as the first feature vector corresponding to the first railway data.

[0013] In a second aspect, the present invention further provides a cross-modal railway data retrieval device, the device comprising the following modules: An acquisition module, used to obtain search terms for cross-modal railway data; A retrieval module is used to extract features from at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data; wherein the type of the first railway data is any one of the following: text data, image data without text, and image data containing text; Determine the cosine similarity between each of the first feature vectors and the search term using an intelligent retrieval model; Target railway data is determined according to the cosine similarity between each of the first feature vectors and the search term.

[0014] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for retrieving cross-modal railway data as described above is implemented.

[0015] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described cross-modal railway data retrieval methods.

[0016] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned cross-modal railway data retrieval methods.

[0017] The cross-modal railway data retrieval method, device, equipment and storage medium provided by the present invention obtain the retrieval term of the cross-modal railway data; then, feature extraction is performed on at least one first railway data in the original railway database to obtain the first feature vector corresponding to each first railway data; wherein the type of the first railway data is any one of the following: text data, image data without text, and image data containing text; then, the cosine similarity between each first feature vector and the retrieval term is determined by using an intelligent retrieval model; and the target railway data is determined according to the cosine similarity between each first feature vector and the retrieval term.

[0018] In the present invention, the search term is text, and the original railway database contains pictures. By extracting features from each first railway data, a first feature vector corresponding to each first railway data is obtained, and then the first feature vector is matched with the feature vector converted from the search term for image and text features, thereby effectively realizing cross-modal intelligent data retrieval between typical modes of pictures and texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1This is one of the flow charts of the cross-modal railway data retrieval method provided by the present invention.

[0021] Figure 2 It is a schematic diagram of the principle of the bidirectional long short-term memory network model provided by the present invention.

[0022] Figure 3 It is a schematic diagram of the principle of face recognition network image feature extraction provided by the present invention.

[0023] Figure 4 It is a schematic diagram of the principle of the distinguishable binary text detection algorithm provided by the present invention.

[0024] Figure 5 This is the second flow chart of the cross-modal railway data retrieval method provided by the present invention.

[0025] Figure 6 It is a schematic diagram of the structure of the cross-modal railway data retrieval device provided by the present invention.

[0026] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally a class, and the number of objects is not limited. For example, the first node can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally represents that the front and back associated objects are in an "or" relationship.

[0029] Combine the following Figure 1-Figure 7 The invention describes a cross-modal railway data retrieval method, device, equipment and storage medium.

[0030] Figure 1 is one of the flow charts of the cross-modal railway data retrieval method provided by the present invention, such as Figure 1As shown, the method comprises the following steps: Step 101, obtaining search terms for cross-modal railway data; Specifically, it should be noted that the execution subject of the present invention is an electronic device, and the present invention is used to execute the following cross-modal railway data retrieval method, thereby effectively realizing the intelligent retrieval of cross-modal railway data.

[0031] First, a search term for cross-modal railway data is obtained. The search term is used to match the corresponding target railway data in the original railway database. Usually, it is text data, and in a few cases, there may also be image data. In this embodiment, multi-modal data retrieval can be implemented. The following takes the search term as text data as an example.

[0032] Step 102: extracting features from at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data; wherein the type of the first railway data is any one of the following: text data, image data without text, and image data containing text; Specifically, it can be understood that the original railway database contains multiple railway data, and the data type is multiple modes. For example, the type of the first railway data is any one of the following: text data (plain text), picture data without text (plain picture), picture data containing text (picture containing text information).

[0033] Before using the search terms for intelligent matching and retrieval, at least one first railway data in the original railway database may be preprocessed, such as extracting features from the first railway data to obtain first feature vectors corresponding to each first railway data.

[0034] Step 103: using the intelligent retrieval model, determining the cosine similarity between each first feature vector and the search term; Specifically, after obtaining the first feature vectors corresponding to each first railway data, the cosine similarity calculation may be performed between the second feature vectors corresponding to the search terms and the first feature vectors corresponding to each first railway data, so as to obtain the most matching target railway data.

[0035] Step 104: Determine target railway data according to the cosine similarity between each first feature vector and the search term.

[0036] Specifically, it can be understood that the larger the value of the cosine similarity between the first feature vector and the second feature vector is, the higher the semantic association between the two is.

[0037] After obtaining the cosine similarity between each first feature vector and the search term, the target railway data may be determined according to the cosine similarity between each first feature vector and the search term. For example, the first railway data corresponding to the maximum cosine similarity may be determined as the target railway data.

[0038] The method provided in this embodiment obtains search terms for cross-modal railway data; then, extracts features from at least one first railway data in the original railway database to obtain first feature vectors corresponding to each first railway data; wherein the type of the first railway data is any one of the following: text data, image data without text, and image data containing text; then, using an intelligent retrieval model, determines the cosine similarity between each first feature vector and the search term; and determines the target railway data according to the cosine similarity between each first feature vector and the search term.

[0039] In the present invention, the search term is text, and the original railway database contains pictures. By extracting features from each first railway data, a first feature vector corresponding to each first railway data is obtained, and then the first feature vector is matched with the feature vector converted from the search term for image and text features, thereby effectively realizing cross-modal intelligent data retrieval between typical modes of pictures and texts.

[0040] According to a cross-modal railway data retrieval method provided by the present invention, the cosine similarity between each first feature vector and the search term is determined by using an intelligent retrieval model, including: Using the intelligent retrieval model, key information is extracted from the search terms to obtain the second feature vector; The intelligent retrieval model is used to determine the cosine similarity between each first eigenvector and the second eigenvector.

[0041] Specifically, in some embodiments, the process of using the intelligent retrieval model to determine the cosine similarity between each first feature vector and the search term in step 103 includes the following steps: First, the intelligent retrieval model is used to extract key information from the search terms to obtain the second feature vector. The key information extraction of the search terms can be performed using the text key information extraction technology based on BiLSTM-CRF, which can effectively and accurately identify and extract key information from these unstructured texts. BiLSTM (bidirectional long short-term memory network) can capture the front-to-back dependency of time series in text data, and CRF (conditional random field) can consider the context constraint relationship in sequence labeling problems. The combination of the two greatly improves the accuracy and reliability of key information. BiLSTM-CRF consists of a word embedding layer, a BiLSTM layer, and a CRF layer.

[0042] Furthermore, the cosine similarity between each first feature vector and the second feature vector is determined by using the intelligent retrieval model. After extracting the second feature vector of the search term, the second feature vector and the feature vector corresponding to the first railway data are semantically associated by calculating the cosine similarity.

[0043] The method provided in this embodiment uses an intelligent retrieval model and an image-text semantic association algorithm based on cosine similarity measurement to achieve intelligent retrieval of cross-modal railway data.

[0044] According to a cross-modal railway data retrieval method provided by the present invention, target railway data is determined according to the cosine similarity between each first feature vector and the search term, including: Calculate the cosine similarity between each first eigenvector and the search term; The first railway data corresponding to the maximum value among the cosine similarities is determined as the target railway data.

[0045] Specifically, in some embodiments, the process of determining the target railway data according to the cosine similarity between each first feature vector and the search term in step 104 can be implemented by the following steps, including: First, the cosine similarity between each first feature vector and the search term is calculated.

[0046] Regardless of the type of the first railway data, images and texts are represented as their feature vectors, and then the cosine similarity between them is calculated to measure the similarity between them. Specifically, for images and texts, feature representations are first extracted based on the key information feature technology for images and texts, and then the extracted feature vectors are normalized to unit vectors (i.e., they are divided by their norms) in order to calculate the cosine similarity between them and the search terms.

[0047] The image-text semantic association based on cosine similarity measurement is trained using the idea of ​​generative adversarial networks. By calculating the cosine similarity between image features and text features, the semantic similarity between them is measured, thereby performing association analysis. The cosine similarity value ranges from -1 to 1. When the cosine similarity is close to 1, it means that the directions pointed by the two vectors are very similar; when the cosine similarity is close to -1, it means that the directions pointed by the two vectors are exactly opposite; when the cosine similarity is close to 0, it means that the two vectors are orthogonal, that is, there is no association.

[0048]

[0049] Among them, A and B represent two vectors to be compared, such as the image feature vector corresponding to the first railway data and the feature vector corresponding to the search term, · represents the dot product of the vectors, and ||A|| and ||B|| represent the norms (lengths) of A and B, respectively. The image-text semantic association method based on cosine similarity measurement determines the semantic relevance between image feature vectors and text feature vectors by calculating the cosine similarity between them, which can help understand the semantic connection between images and texts.

[0050] Furthermore, the first railway data corresponding to the maximum value of each cosine similarity is determined as the target railway data, thereby realizing the retrieval of cross-modal railway data.

[0051] The method provided in this embodiment uses an image-text semantic association algorithm based on cosine similarity measurement to achieve intelligent retrieval of cross-modal railway data.

[0052] According to a cross-modal railway data retrieval method provided by the present invention, feature extraction is performed on at least one first railway data in an original railway database to obtain a first feature vector corresponding to each first railway data, including: When the first railway data is text data, a bidirectional long short-term memory network model is used to extract key information from the text data to obtain a first text feature vector; The first text feature vector is determined as the first feature vector corresponding to the first railway data.

[0053] Specifically, in some embodiments, step 102 extracts features from at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data, including the following steps: When the first railway data is text data, a bidirectional long short-term memory network model is used to extract key information from the text data to obtain a first text feature vector.

[0054] Among them, the bidirectional long short-term memory network model is implemented based on the BiLSTM-CRF text key information extraction technology. BiLSTM (bidirectional long short-term memory network) can capture the front-end dependency of time series in text data, and CRF (conditional random field) can consider the context constraint relationship in sequence labeling problems. The combination of the two greatly improves the accuracy and reliability of key information. BiLSTM-CRF consists of a word embedding layer, a BiLSTM layer, and a CRF layer.

[0055] The word embedding layer converts discrete characters into distributed representations. LSTM is the core of the BiLSTM layer, which can complete the task of learning long-term dependency information, and a gating is designed to avoid long-term dependency problems, which remembers the information for transmission. The CRF layer confirms the final required prediction label sequence.

[0056] Furthermore, the first text feature vector is determined as the first feature vector corresponding to the first railway data, thereby achieving accurate feature extraction of the text data.

[0057] The method provided in this embodiment, when the first railway data is text data, uses a bidirectional long short-term memory network model to extract key information from the text data to obtain a first text feature vector, and then determines the first text feature vector as the first feature vector corresponding to the first railway data. The extraction of text key information of railway data based on BiLSTM-CRF is realized, which facilitates the subsequent semantic association of cross-modal data.

[0058] According to a cross-modal railway data retrieval method provided by the present invention, a bidirectional long short-term memory network model includes a word embedding layer, a bidirectional long short-term memory network layer and a conditional random field model layer; the bidirectional long short-term memory network model is used to extract key information from text data to obtain a text feature vector, including: Input the text data into the word embedding layer to obtain the word vector matrix corresponding to the text data. The word vector matrix is ​​a continuous vector representation. Input the word vector matrix corresponding to the text data into the bidirectional long short-term memory network layer to obtain a context feature sequence of at least two time steps; Using the conditional random field model layer, the context feature sequence of each time step is annotated, and the output score of each annotated context feature sequence relative to the label prediction sequence is calculated. The output score is used to represent the credibility of the context feature sequence of the time step; The conditional random field model layer is used to determine the text feature vector based on the output score of each annotated context feature sequence relative to the label prediction sequence.

[0059] Specifically, in some embodiments, the bidirectional long short-term memory network model includes a word embedding layer, a bidirectional long short-term memory network BiLSTM layer and a conditional random field CRF model layer. Figure 2 It is a schematic diagram of the principle of the bidirectional long short-term memory network model provided by the present invention.

[0060] Correspondingly, the process of extracting key information from text data using a bidirectional long short-term memory network model to obtain a text feature vector can be achieved through the following steps, including: First, the text data is input into the word embedding layer to obtain the word vector matrix corresponding to the text data. The word vector matrix is ​​a continuous vector representation.

[0061] Here, the word embedding layer converts discrete characters into distributed representations, which means using continuous values ​​in the vector space to represent features (such as characters, words, or other discrete objects); in the word embedding layer, the input is the index of discrete characters or words (which may contain contextual information), and the output is the continuous vector representation of these indexes mapped to the dense vector space. First, preprocess the annotated corpus files, usually text files such as CSV, JSON, etc. The annotated information includes part of speech, syntactic structure, entity recognition, etc. The main purpose of preprocessing is to convert the raw data into a format suitable for model training to ensure the quality and consistency of the data. Next, initialize a random matrix for input into the model. Random matrix initialization helps prevent model overfitting. The dimension of the random matrix is ​​n*d, where n represents the dictionary size (that is, the number of different words in the vocabulary) and d represents the attribute vector dimension of each character (that is, the dimension of the word vector). Finally, the word embedding layer outputs the vector representation of each word in the text and forms a matrix.

[0062] Furthermore, the word vector matrix corresponding to the text data is input into the bidirectional long short-term memory network layer to obtain a context feature sequence of at least two time steps.

[0063] Among them, LSTM is the core of the BiLSTM layer, which can complete the task of learning long-term dependency information, and a gate is designed to avoid long-term dependency problems, which will remember the information for transmission. The word vector matrix generated from the word embedding layer is input into the BiLSTM bidirectional long short-term memory network, where BiLSTM consists of two LSTMs, one from the beginning to the end of the sequence (forward LSTM) and the other from the end to the beginning (reverse LSTM). This structure can consider the contextual information of the previous and following words at the same time, making the model's understanding of the sequence more comprehensive. The core calculation formula of the long short-term memory network (LSTM) is as follows:

[0064] The parameters are defined as follows: : The hidden state (or output) of the previous moment, indicating the model at time memories and information.

[0065] : The input vector at the current time t, which can be the word embedding representation, features, etc. of the text.

[0066] , , , : Weight matrix, used for linear transformation.

[0067] : Weight of the forget gate.

[0068] : Weight of the input gate.

[0069] : The weight of the candidate memory unit.

[0070] : Weight of the output gate.

[0071] , , , : Bias vector, added to the calculation of each gate to help the model learn.

[0072] : Bias of forget gate.

[0073] : Bias of input gate.

[0074] : Bias of candidate memory cell.

[0075] : Bias of output gate.

[0076] : Sigmoid activation function, the output value is between 0 and 1.

[0077] : Hyperbolic tangent activation function, used to normalize and scale the contents of the candidate memory unit, with an output range of [-1, 1].

[0078] : The state of the memory unit at the previous moment, storing the long-term memory passed from the previous time step.

[0079] : The current state of the memory cell is updated by combining the output of the input gate and the forget gate.

[0080] : The activation value of the output gate determines the content of the output at the current moment.

[0081] : The hidden state at the current moment, which is also the output of LSTM, combines the memory cell state at the current moment and the decision of the output gate.

[0082] After the above calculations, the hidden state sequence can be generated BiLSTM passes information forward and backward through two LSTMs respectively, and concatenates the generated hidden state sequences, thereby obtaining contextual information at the same time, effectively improving the performance of the model, and finally outputting the context feature representation of each time step, that is, the context feature sequence of at least two time steps.

[0083] Furthermore, the conditional random field model layer is used to annotate the context feature sequence of each time step, and the output score of each annotated context feature sequence relative to the label prediction sequence is calculated. The output score is used to characterize the credibility corresponding to the context feature sequence of the time step; the text feature vector is determined according to the output score of each annotated context feature sequence relative to the label prediction sequence.

[0084] The output of the BiLSTM model can be connected to the CRF model. The CRF model decodes the label sequence of the BiLSTM model at the sentence level (rather than decoding it individually) and annotates the sequence. The annotated information includes named entities, parts of speech, semantics, keywords, etc., covering a variety of natural language processing tasks. In the loss function of the CRF module, the label prediction sequence is the sequence with the highest output score, that is, the sequence with the highest credibility or probability of each label sequence calculated by the CRF model. Given a sequence X, let the sequence annotation result be y, and its definition score is:

[0085] Where P is the initial score matrix and A is the transformed score matrix. ij represents the probability of label j being the successor of label i, P ij Representation word W i The probability of mapping to label j. Calculate the score of the output label sequence y corresponding to the input sequence X, and regard the sequence with the highest score as the predicted label sequence required by the model.

[0086] After being processed by the BiLSTM-CRF model, we can finally obtain the feature vector corresponding to the key information in the text, that is, the text feature vector. This information includes the extracted named entities, keywords, etc.

[0087] The method provided in this embodiment first inputs text data into a character embedding layer to obtain a word vector matrix corresponding to the text data, where the word vector matrix is ​​a continuous vector representation; then, the word vector matrix corresponding to the text data is input into a bidirectional long short-term memory network layer to obtain a context feature sequence of at least two time steps; then, a conditional random field model layer is used to label the context feature sequence of each time step, and an output score of each labeled context feature sequence relative to a label prediction sequence is calculated, where the output score is used to characterize the credibility corresponding to the context feature sequence of the time step; then, a conditional random field model layer is used to determine a text feature vector according to the output score of each labeled context feature sequence relative to a label prediction sequence.

[0088] When the first railway data is text data, the present invention uses a bidirectional long short-term memory network model to extract key information from the text data to obtain a first text feature vector, thereby realizing the extraction of text key information from railway data based on BiLSTM-CRF, which is convenient for subsequent semantic association of cross-modal data.

[0089] According to a cross-modal railway data retrieval method provided by the present invention, feature extraction is performed on at least one first railway data in an original railway database to obtain a first feature vector corresponding to each first railway data, including: In the case where the first railway data is picture data without text, the picture data with text is input into a face recognition network to extract key information of the picture data with text; According to the key information of the text image data, the yolov5 model is used to frame the face in the text image data to obtain the face image; Use deep convolutional neural network to extract features from face images and obtain feature vectors corresponding to face images; The feature vector corresponding to the face image is determined as the first feature vector corresponding to the first railway data.

[0090] Specifically, in some embodiments, the process of extracting features from the first railway data in the original railway database to obtain first feature vectors corresponding to each first railway data may be implemented by the following steps, including: In the case where the first railway data is picture data without text, first, in the case where the first railway data is picture data without text, the picture data with text is input into the face recognition network to extract key information of the picture data with text. Figure 3It is a schematic diagram of the principle of face recognition network image feature extraction provided by the present invention. Further, according to the key information of the picture data of the text, the face in the picture data of the text is framed by the yolov5 model to obtain a face picture; further, the face picture is feature extracted by using a deep convolutional neural network, for example, the key features in the face picture can be extracted, including the contour, eyes, nose and other features of the face, to obtain a feature vector corresponding to the face picture; further, the feature vector corresponding to the face picture is determined as the first feature vector corresponding to the first railway data.

[0091] Here, the face recognition network FaceNet maps face images to Euclidean space, directly reflecting the similarity of face images in spatial distance. The principle is that the distance between face images sampled from the same individual in Euclidean space is smaller, while the distance between face images from different individuals is larger. Through Facenet, the key information in the image is extracted, and the celeba data set is trained using the yolov5 network model to frame the face in the image. In this experiment, the framed face is taken out and put into Facenet to generate facial features. The facial features are compared with those in the existing database through SVM to calculate the similarity.

[0092] The structure of FaceNet includes a deep convolutional neural network, which is used to convert face images into fixed-length embeddings. During the training process, batch processing and a deep convolutional neural network structure Deep Architecture after removing softmax are used, and L2 regularization is used to ensure the stability of the embedding vector in Euclidean space. Finally, the triplet loss function is used to optimize the embedding, so that the embedding distance of the same face is closer and the embedding distance of different faces is farther.

[0093] 1) Use a deep convolutional neural network (DNN) to extract facial features and map the face image to an embedding vector of fixed length. The deep architecture used can capture the complex features in the image and improve recognition accuracy.

[0094] 2) FaceNet introduced a triplet loss function to enhance the model's discriminative ability. This loss function uses a triplet (Anchor, Positive, Negative) to optimize the network, making the face embedding vectors (Anchor and Positive) of the same person closer in Euclidean space, while the face embedding vectors (Anchor and Negative) of different people are farther apart.

[0095] The learning process of the network is to continuously adjust the Euclidean distance between Anchor, Positive, and Negative. During learning, the distance between Anchor and Negative is increased while the distance between Anchor and Positive is reduced, and finally the distance between Anchor and Negative is greater than the distance between Anchor and Positive. Let x be the input image, is a eigenvector, then its ternary function is expressed as: Then the expression of its ternary function is:

[0096] in is a constant, so the final loss function is:

[0097] The working principle of FaceNet is to realize the recognition of face information in images by training deep learning models. Its workflow is as follows: 1) Input a face picture: Input a picture containing a face into the FaceNet network and use the yolov5 model to frame the face.

[0098] 2) Feature extraction through deep convolutional network: FaceNet uses deep convolutional neural network (CNN) to extract features from input face images. After training, the deep convolutional network can extract key features from face images, including facial contours, eyes, nose and other features.

[0099] 3) L2 normalization: In the process of feature extraction, in order to enhance the stability and comparability of features, FaceNet uses the L2 normalization method. This method can normalize the length of feature vectors, reduce the differences between different features, and improve the comparability of feature vectors.

[0100] 4) Obtain a feature vector of length 128: After processing and normalizing the features extracted by FaceNet through the deep convolutional network, a feature vector of length 128 is obtained. This feature vector can uniquely represent the face in the input image, has good feature distinction ability, and can be used for tasks such as face recognition and face verification.

[0101] The method provided in this embodiment realizes the recognition of facial information in an image by training a deep learning model, and finally converts it into a feature vector to facilitate the subsequent cross-modal intelligent retrieval of images and texts.

[0102] According to a cross-modal railway data retrieval method provided by the present invention, feature extraction is performed on at least one first railway data in an original railway database to obtain a first feature vector corresponding to each first railway data, including: In the case where the first railway data is picture data containing text, the picture data containing text is input into a text detection model to identify a text region in the picture data containing text; According to the text region in the image data containing the text, the text direction of the text instance in the text region is detected by using a text direction classifier; Based on the text direction, the text instance is rotated to a parallel direction to obtain a rotated text instance; Inputting the rotated text instance into a text recognition model to obtain a second text feature vector; The second text feature vector is determined as the first feature vector corresponding to the first railway data.

[0103] Specifically, in some embodiments, the implementation process of extracting features from at least one first railway data in the original railway database in step 102 to obtain a first feature vector corresponding to each first railway data includes the following steps: if the first railway data is image data, further determine whether there is text information in the image data. In the case where the first railway data is image data containing text, this embodiment identifies the text information to obtain a text feature vector, including steps such as image preprocessing, character segmentation, feature extraction and classification. The technology includes three modules: a text detection model, a text direction classifier and a text recognition model. The implementation process example is as follows: Figure 4 Schematic diagram of the principle of the distinguishable binary text detection algorithm provided by the present invention, such as Figure 4 As shown, first, the image data containing text is input into the text detection model to identify the text area in the image data containing text. Among them, the main task of the text detection model is to detect the text area from the image, identify the text position in the image, and locate it as independent text areas. The Differentiable Binarization (DB) model is used to implement this function. The DB model adopts a segmentation-based text detection method, which converts the text detection problem into a semantic segmentation task. A differentiable binarization function is used to make the entire detection network end-to-end trainable. During the training process, the appropriate binarization threshold and sensitivity can be automatically learned to better match the real data ground truth.

[0104] The core of the DB model is a differentiable binarization function that accepts an input feature map and outputs a probability map between [0, 1]. The probability map is generated by the segmentation network and then converted into a binary map. Then, through this probability map, the DB model can obtain the binarized text area mask. The binarization process is:

[0105] Where 𝑡 is a predefined threshold, (𝑖, 𝑗) is the coordinate point on the graph, : The value of the 𝑖th row and 𝑗th column of the input feature map is the feature value or score calculated by the model before. Represents the binarized value of the 𝑖th row and 𝑗th column of the input feature map.

[0106] The differentiable binarization that can be optimized synchronously with the segmentation network during the training phase is defined as:

[0107] in, : The probability value of the pixel at the 𝑖th row and the 𝑗th column indicates the possibility that the pixel is a text area. K is a constant that adjusts the "slope" or "sensitivity" of the sigmoid function. The larger its value, the steeper the function changes, and the more effective it is in distinguishing a specific range of probability values.

[0108] This formula takes the eigenvalue and threshold Combined with the smoothness of the sigmoid function, a probability value in the interval [0,1] is generated, which enables the model to output continuous judgments. and , the sensitivity and feature selection ability of the model can be fine-tuned, thereby affecting the final text detection effect.

[0109] Therefore, it is concluded that when the value in the probability map is less than the threshold of the threshold map, it means that this is not a text area, so it is set to 0, otherwise it is a text area. In actual processing applications, the judgment of a single pixel alone is not enough to effectively recognize text. Therefore, a method of combining regional validity after processing is proposed, requiring a group of adjacent pixels to meet the condition of being higher than the threshold to form a coherent text area, which can be used to eliminate isolated noise points and ensure the reliability of detection results.

[0110] Furthermore, after detecting the text area, the text direction can be further classified. For example, based on the text area in the image data containing the text, a text direction classifier is used to detect the text direction of the text instance in the text area. Furthermore, based on the text direction, the text instance is rotated to a parallel direction to obtain a rotated text instance.

[0111] It should be noted that the task of the text direction classifier is to classify the direction of the text instance detected by the text, rotate the text to 0 degrees, and then send it to the subsequent text recognizer. The MobileNetV3 model is used to implement this function. MobileNetV3 introduces the Squeeze-and-Excitation module of channel attention and the NAS search method to improve network performance. The overall architecture adopts a lightweight depth-separable convolution and residual block structure.

[0112] MobileNetv3 converts the input image into a compact feature vector to represent the direction information of the text image. The direction information is output as a probability distribution vector, and each element represents the possibility that the input image is judged to be in the corresponding direction. For example, the output of [0.1, 0.7, 0.1, 0.1] may indicate that there is a 70% probability that the image is in a 90-degree direction. The features extracted by MobileNetv3 are input into a fully connected layer to achieve fast and efficient direction classification. While maintaining high classification accuracy, MobileNetv3 greatly reduces the number of model parameters and inference time, providing strong support for high-performance direction classification applications.

[0113] Furthermore, the rotated text instance is input into a text recognition model to obtain a second text feature vector.

[0114] It should be noted that the text recognition model is based on a sequence-to-sequence architecture of deep learning, and can accurately identify text content from the detected text area. The present invention uses a convolutional recurrent neural network (CRNN) model to achieve this function. The CRNN model efficiently captures the spatial and semantic information of the text. The CNN part is responsible for extracting the visual features of the text area, while the RNN part models the contextual relationship of the text and finally outputs the text recognition result. It has excellent performance in terms of accuracy, generalization, and real-time performance, and can be widely used in various text recognition scenarios. It mainly includes three parts: a sequence of convolutional layers for extracting image features, a sequence of recursive layers for sequence modeling, and a transcription layer, which decodes the output of the recursive layer into a text sequence.

[0115] The convolutional layer is responsible for extracting features of the image, which are then flattened and input to the recurrent layer. In the recurrent layer, the RNN processes the input sequence and is able to capture contextual information at each time step of the sequence. This is especially important for understanding the meaning of text, because specific characters of the text often require knowledge of the previous and following context. Finally, the transcription layer (usually using CTC, or Connectionist Temporal Classification) decodes the output of the RNN to generate the final text sequence.

[0116] The first part of CRNN is a series of convolutional layers, which are used to extract visual features from the input image. Let the input image be I, and the feature map obtained after L layers of convolution operations is:

[0117] Among them, W L and b L Represent the convolution weight and bias of the Lth layer respectively. The feature F extracted by the convolution operation L It will be sent to the subsequent recursive layer for further processing.

[0118] The second part is the recursive layer, which models the feature sequence and captures the time dependency in the sequence. It outputs a hidden state vector at each time step, which represents the feature representation of the current time step and contains the current input information and the state information of the previous time step.

[0119] The third part is the transcription layer, which is responsible for mapping the output of the recursive layer to the final sequence label. Transcription is usually done through (Connectionist Temporal Classification, CTC), which uses the principles of probability theory to solve the problem of sequence learning without aligned data, and its goal is to maximize the conditional probability .in represents a path that maps to the final label sequence l by removing duplicate labels and blank labels. The objective function of CTC is defined as follows, which is a function for all possible mappings to the label sequence Path The probability sum of .

[0120]

[0121] The input sequence : is the feature sequence generated by the recursive layer. Label sequence : The final output label sequence that you want the model to predict.

[0122] The objective function of CTC is to map all possible labels to a given sequence. The algorithm sums the path probabilities of the input sequence and flexibly handles the alignment problem between the input sequence and the output label sequence. Its mechanism allows to effectively “transcribe” clear outputs from sparse and unaligned inputs in sequence learning.

[0123] Then, the second text feature vector is determined as the first feature vector corresponding to the first railway data, so as to realize feature extraction of the image data containing text information, so as to facilitate subsequent intelligent retrieval and matching.

[0124] The method provided in this embodiment, when the first railway data is picture data containing text, inputs the picture data containing text into the text detection model to identify the text area in the picture data containing text; according to the text area in the picture data containing text, uses the text direction classifier to detect the text direction of the text instance in the text area; based on the text direction, rotates the text instance to a parallel direction to obtain the rotated text instance; inputs the rotated text instance into the text recognition model to obtain the second text feature vector; and determines the second text feature vector as the first feature vector corresponding to the first railway data. Thus, the image is converted into editable text, and finally the image-text semantic association algorithm based on the cosine similarity measurement is integrated, and the cross-modal intelligent retrieval of railway data is realized.

[0125] Figure 5 FIG. 2 is a flow chart of the cross-modal railway data retrieval method provided by the present invention. Figure 5 As shown, the cross-modal railway data retrieval method includes the following steps: First, the original railway scientific research management data is input and preprocessed. The preprocessing includes the following steps: For text data: Use the word embedding layer, BiLSTM layer and CRF layer to represent and extract key information features of text data and obtain the text feature vector corresponding to the text data; For image data: Determine whether the image data has text information; If the image data has text information, perform text area detection, text direction classification and text recognition to obtain the text feature vector corresponding to the image data with text information; If the image data has no text information, perform intelligent data annotation and feature extraction to obtain the image feature vector corresponding to the image data; Then, the search terms are input, and the intelligent retrieval model is used to calculate the cosine similarity between the search terms and the feature vectors corresponding to the original railway scientific research management data, and the results with high similarity are output.

[0126] The following is a description of a cross-modal railway data retrieval device provided by the present invention. The cross-modal railway data retrieval device described below and the cross-modal railway data retrieval method described above can be referred to each other.

[0127] Figure 6 is a schematic diagram of the structure of the cross-modal railway data retrieval device provided by the present invention, such as Figure 6 As shown, the cross-modal railway data retrieval device 600 includes an acquisition module 610 and a retrieval module 620: An acquisition module 610 is used to acquire search terms for cross-modal railway data; The retrieval module 620 is used to extract features from at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data; wherein the type of the first railway data is any one of the following: text data, image data without text, and image data containing text; Determine the cosine similarity between each of the first feature vectors and the search term using an intelligent retrieval model; Target railway data is determined according to the cosine similarity between each of the first feature vectors and the search term.

[0128] The device provided in this embodiment includes an acquisition module 610 and a retrieval module 620. The acquisition module 610 acquires the search terms of the cross-modal railway data. Then, the retrieval module 620 performs feature extraction on at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data. The type of the first railway data is any one of the following: text data, image data without text, and image data containing text. Then, the cosine similarity between each first feature vector and the search term is determined by using an intelligent retrieval model. According to the cosine similarity between each first feature vector and the search term, the target railway data is determined.

[0129] In the present invention, the search term is text, and the original railway database contains pictures. By extracting features from each first railway data, a first feature vector corresponding to each first railway data is obtained, and then the first feature vector is matched with the feature vector converted from the search term for image and text features, thereby effectively realizing cross-modal intelligent data retrieval between typical modes of pictures and texts.

[0130] According to a cross-modal railway data retrieval device 600 provided by the present invention, the retrieval module 620 is specifically used for: Using an intelligent retrieval model, extract key information from the search term to obtain a second feature vector; The cosine similarity between each of the first feature vectors and the second feature vectors is determined by using an intelligent retrieval model.

[0131] According to a cross-modal railway data retrieval device 600 provided by the present invention, the retrieval module 620 is further used for: Calculating the cosine similarity between each of the first feature vectors and the search term; The first railway data corresponding to the maximum value of each of the cosine similarities is determined as the target railway data.

[0132] According to a cross-modal railway data retrieval device 600 provided by the present invention, the retrieval module 620 is further used for: When the first railway data is text data, using a bidirectional long short-term memory network model to extract key information from the text data to obtain a first text feature vector; The first text feature vector is determined as the first feature vector corresponding to the first railway data.

[0133] According to a cross-modal railway data retrieval device 600 provided by the present invention, the bidirectional long short-term memory network model includes a word embedding layer, a bidirectional long short-term memory network layer and a conditional random field model layer; The retrieval module 620 is further used for: Input the text data into the word embedding layer to obtain a word vector matrix corresponding to the text data, wherein the word vector matrix is ​​a continuous vector representation; Inputting the word vector matrix corresponding to the text data into the bidirectional long short-term memory network layer to obtain a context feature sequence of at least two time steps; Using the conditional random field model layer, the context feature sequence of each time step is labeled, and the output score of each labeled context feature sequence relative to the label prediction sequence is calculated, and the output score is used to characterize the credibility corresponding to the context feature sequence of the time step; The conditional random field model layer is used to determine the text feature vector according to the output scores of each annotated context feature sequence relative to the label prediction sequence.

[0134] According to a cross-modal railway data retrieval device 600 provided by the present invention, the retrieval module 620 is further used for: In the case where the first railway data is picture data without text, inputting the picture data with text into a face recognition network to extract key information of the picture data with text; According to the key information of the picture data of the text, the face in the picture data of the text is framed by using the yolov5 model to obtain a face picture; Using a deep convolutional neural network to extract features from the face image, and obtain a feature vector corresponding to the face image; The feature vector corresponding to the face image is determined as the first feature vector corresponding to the first railway data.

[0135] According to a cross-modal railway data retrieval device 600 provided by the present invention, the retrieval module 620 is further used for: In the case where the first railway data is picture data containing text, the picture data containing text is input into a text detection model to identify a text area in the picture data containing text; According to the text region in the picture data containing text, using a text direction classifier to detect and obtain the text direction of the text instance in the text region; Based on the text direction, the text instance is rotated to a parallel direction to obtain a rotated text instance; Inputting the rotated text instance into a text recognition model to obtain a second text feature vector; The second text feature vector is determined as the first feature vector corresponding to the first railway data.

[0136] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communications interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the cross-modal railway data retrieval method, which includes: to obtain search terms for cross-modal railway data; Performing feature extraction on at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data; wherein the type of the first railway data is any one of the following: text data, image data without text, and image data containing text; Determine the cosine similarity between each of the first feature vectors and the search term using an intelligent retrieval model; Target railway data is determined according to the cosine similarity between each of the first feature vectors and the search term.

[0137] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0138] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the cross-modal railway data retrieval method provided by the above methods, the method comprising: to obtain search terms for cross-modal railway data; Performing feature extraction on at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data; wherein the type of the first railway data is any one of the following: text data, image data without text, and image data containing text; Determine the cosine similarity between each of the first feature vectors and the search term using an intelligent retrieval model; Target railway data is determined according to the cosine similarity between each of the first feature vectors and the search term.

[0139] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for retrieving cross-modal railway data provided by the above methods is implemented, and the method comprises: to obtain search terms for cross-modal railway data; Performing feature extraction on at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data; wherein the type of the first railway data is any one of the following: text data, image data without text, and image data containing text; Determine the cosine similarity between each of the first feature vectors and the search term using an intelligent retrieval model; Target railway data is determined according to the cosine similarity between each of the first feature vectors and the search term.

[0140] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal railway data retrieval method, characterized in that: include: to obtain search terms for cross-modal railway data; Performing feature extraction on at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data; wherein the type of the first railway data is any one of the following: text data, image data without text, and image data containing text; Determine the cosine similarity between each of the first feature vectors and the search term using an intelligent retrieval model; Target railway data is determined according to the cosine similarity between each of the first feature vectors and the search term.

2. The cross-modal railway data retrieval method according to claim 1, characterized in that: The using of the intelligent retrieval model to determine the cosine similarity between each of the first feature vectors and the search term includes: Using an intelligent retrieval model, extract key information from the search term to obtain a second feature vector; The cosine similarity between each of the first feature vectors and the second feature vectors is determined by using an intelligent retrieval model.

3. The cross-modal railway data retrieval method according to claim 1, characterized in that: The step of determining target railway data according to the cosine similarity between each of the first feature vectors and the search term comprises: Calculating the cosine similarity between each of the first feature vectors and the search term; The first railway data corresponding to the maximum value of each of the cosine similarities is determined as the target railway data.

4. The cross-modal railway data retrieval method according to claim 1, characterized in that: The step of extracting features from at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data includes: When the first railway data is text data, using a bidirectional long short-term memory network model to extract key information from the text data to obtain a first text feature vector; The first text feature vector is determined as the first feature vector corresponding to the first railway data.

5. The cross-modal railway data retrieval method according to claim 4, characterized in that: The bidirectional long short-term memory network model includes a word embedding layer, a bidirectional long short-term memory network layer and a conditional random field model layer; the bidirectional long short-term memory network model is used to extract key information from the text data to obtain a text feature vector, including: Input the text data into the word embedding layer to obtain a word vector matrix corresponding to the text data, wherein the word vector matrix is ​​a continuous vector representation; Inputting the word vector matrix corresponding to the text data into the bidirectional long short-term memory network layer to obtain a context feature sequence of at least two time steps; Using the conditional random field model layer, the context feature sequence of each time step is labeled, and the output score of each labeled context feature sequence relative to the label prediction sequence is calculated, and the output score is used to characterize the credibility corresponding to the context feature sequence of the time step; The conditional random field model layer is used to determine the text feature vector according to the output scores of each annotated context feature sequence relative to the label prediction sequence.

6. The cross-modal railway data retrieval method according to claim 1, characterized in that: The step of extracting features from at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data includes: In the case where the first railway data is picture data without text, inputting the picture data with text into a face recognition network to extract key information of the picture data with text; According to the key information of the picture data of the text, the face in the picture data of the text is framed by using the yolov5 model to obtain a face picture; Using a deep convolutional neural network to extract features from the face image, and obtain a feature vector corresponding to the face image; The feature vector corresponding to the face image is determined as the first feature vector corresponding to the first railway data.

7. The cross-modal railway data retrieval method according to claim 1, characterized in that: The step of extracting features from at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data includes: In the case where the first railway data is picture data containing text, the picture data containing text is input into a text detection model to identify a text area in the picture data containing text; According to the text region in the picture data containing text, using a text direction classifier to detect and obtain the text direction of the text instance in the text region; Based on the text direction, the text instance is rotated to a parallel direction to obtain a rotated text instance; Inputting the rotated text instance into a text recognition model to obtain a second text feature vector; The second text feature vector is determined as the first feature vector corresponding to the first railway data.

8. A cross-modal railway data retrieval device, characterized in that: include: An acquisition module, used to obtain search terms for cross-modal railway data; A retrieval module is used to extract features from at least one first railway data in the original railway database to obtain a first feature vector corresponding to each first railway data; wherein the type of the first railway data is any one of the following: text data, image data without text, and image data containing text; Determine the cosine similarity between each of the first feature vectors and the search term using an intelligent retrieval model; Target railway data is determined according to the cosine similarity between each of the first feature vectors and the search term.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for retrieving cross-modal railway data according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for retrieving cross-modal railway data as claimed in any one of claims 1 to 7 is implemented.

Images