A data collection and processing method and system for cross-modal retrieval

Abstract

The present invention discloses a data acquisition and processing method and system for cross-modal retrieval, comprising: performing distributed parallel acquisition of multi-modal data on a target open source data network; cleaning text modal data for special characters and invisible characters, and then storing the cleaned text modal data and image modal data in different message queues; utilizing a feature extraction model to extract features from each text and each image in the message queue to obtain text features and image features, matching and screening the text features and image features to obtain image-text combinations based on similarity, and storing the image features and text features of the image-text combinations as indexes in a database; during retrieval, screening and returning matching image-text combinations as retrieval results for the uploaded data based on the similarity between the uploaded data and the image-text combinations in the database. This method and system can realize high-quality cross-modal retrieval of image-text data.

Description

Technical Field

[0001] The present invention belongs to the field of cross-modal information retrieval + technology, and specifically relates to a data acquisition and processing method and system for cross-modal retrieval. Background Art

[0002] With the widespread advancement of artificial intelligence technologies, particularly deep learning, the AI ​​industry's demand for data in various modalities, such as images and text, has increased dramatically. The internet offers a vast amount of valuable multimodal open-source data that can be used to train deep learning models. For example, relevant image and text data from social networking platforms can be used to train multimodal deep learning models. The demand for image and text data is particularly high in cross-modal search scenarios.

[0003] To achieve this goal, it is crucial to construct data collection and processing channels for both modalities (images and text). Current data processing technologies for images and text primarily focus on efficient storage. However, in practice, the original text data may contain some invisible characters and display control characters, which can lead to inaccurate indexing of text data in subsequent cross-modal retrieval. Furthermore, the original image and text pairs may be semantically inconsistent, which also requires screening during data processing.

[0004] Patent document CN108877948A discloses a multimodal data processing method and system, comprising: a data acquisition network collects multimodal data corresponding to coronary heart disease cases; the data acquisition network processes the multimodal data based on the predetermined type to which the collected multimodal data belongs, in combination with a data cleaning model corresponding to the predetermined type; the data acquisition network sends the processed multimodal data to a data server; the data server performs data preprocessing on the received multimodal data to obtain fused data; and the data server performs association rule mining on the fused data. This process does not process invisible characters and display control characters in the collected data, which may result in inaccurate multimodal data processing results.

[0005] Patent document CN 112256786 discloses a multimodal data processing method and apparatus, comprising: a terminal acquiring multimodal data; the terminal extracting features from the multimodal data using a feature extraction algorithm to obtain data features of the multimodal data; the terminal converting the data features using a first conversion algorithm to obtain first data features, wherein the first conversion algorithm is used to map the multimodal data to a specific space; the terminal transmitting the data features, data labels, and terminal ID of the multimodal data to a server; the server converting the first data features using a second conversion algorithm corresponding to the terminal ID to obtain second data features, wherein the second conversion algorithm is used to map data in different specific spaces to the same space; and the server performing multimodal representation learning using the second data features as input and the data labels as output to train the multimodal representation learning algorithm. This process does not process invisible characters and display control characters in the collected data, which may cause the second data features obtained during the multimodal data conversion process to affect the modality learning results. Summary of the Invention

[0006] In view of the above, the purpose of the present invention is to provide a data collection and processing method and system for cross-modal retrieval, which can collect and deeply process open source image and text data, thereby ultimately achieving high-quality cross-modal retrieval of image-text data.

[0007] To achieve the above-mentioned object of the invention, one embodiment provides a data collection and processing method for cross-modal retrieval, comprising:

[0008] Perform distributed parallel collection of multimodal data on the target open source data network, where the multimodal data includes text modality data and image modality data;

[0009] After cleaning the text modal data for special characters and invisible characters, the cleaned text modal data and image modal data are stored in different message queues;

[0010] The feature extraction model is used to extract features from each text and each image in the message queue to obtain text features and image features. The image-text combination is obtained by matching and screening based on the similarity of the text features and image features. The image features and text features of the image-text combination are used as indexes for database storage.

[0011] During retrieval, matching image-text combinations are screened based on the similarity between the uploaded data and the image-text combinations in the database and returned as retrieval results for the uploaded data, wherein the uploaded data includes text data and image data.

[0012] In one embodiment, cleaning invisible characters from text modal data includes:

[0013] Invisible characters in text modal data are filtered out, where invisible characters include zero-length spaces, zero-length connectors, and zero-length non-connectors.

[0014] In one embodiment, special character cleaning of text modal data includes:

[0015] When the special character is a pictographic glyph, the pictographic glyph is replaced with the original character according to the mapping relationship between the pictographic glyph and the original character presented in the pictographic character table;

[0016] When the special character is a deletion character, different cleaning methods are used according to different deletion characters, including: when the deletion character is a backspace character, the backspace character and the character before the backspace character are deleted at the same time; when the deletion character is a delete character, the delete character and the character after the delete character are deleted at the same time; when the deletion character is a carriage return character, all characters after the carriage return character are overwritten from the start of the paragraph.

[0017] In one embodiment, special character cleaning of text modal data includes:

[0018] When the special character is a display order character, a bottom-up recursive recovery is performed according to a sequence of control-changing display order characters to ultimately restore the original text, wherein the display order characters include: PDF character, LRE character, RLE character, LRO character, RLO character, PDI character, LRI character, and RLI character, and the sequence of control-changing display order characters is [LRO, LRI, RLO, LRI, string 1, PDI, LRI, string 2, PDI, PDF, PDI, PDF].

[0019] Use bottom-up recursive recovery to restore the original text, including:

[0020] (a) Using a non-greedy matching algorithm, match a string sequence of the form [LRO, LRI, RLO, LRI, string 1, PDI, LRI, string 2, PDI, PDF, PDI, PDF] and replace the string sequence with a sequence of the form [string 2, string 1];

[0021] (b) repeating step (a) until the character string sequence no longer exists in the text modality data;

[0022] (c) If display sequence characters still exist in the remaining text modal data, all display sequence characters are deleted.

[0023] In one embodiment, the method of obtaining the image-text combination by matching and screening based on the similarity between text features and image features includes:

[0024] Calculate the similarity between text features and all image features, and select the image features with the greatest similarity to match with text features to form an image-text combination.

[0025] In one embodiment, the step of screening matching image-text combinations as retrieval results of the uploaded data based on similarity between the uploaded data and the image-text combinations in the database includes:

[0026] When the uploaded data is text data, the feature extraction model is used to extract the text features of the text data, and the similarity between the text features and the text features in the database is calculated. The text feature with the greatest similarity in the database is used as the matching text feature, and the image-text combination to which the matching text feature belongs is used as the search result;

[0027] When the uploaded data is image data, the feature extraction model is used to extract the image features of the image data, and the similarity between the image features and the image features in the database is calculated. The image feature with the greatest similarity in the database is used as the matching image feature, and the image-text combination to which the matching image feature belongs is used as the retrieval result.

[0028] In one embodiment, the feature extraction module uses a comparative text-image pre-training model.

[0029] In one embodiment, the similarity between text features and image features is cosine similarity or L2 distance; the similarity between text features is cosine similarity or L2 distance; and the similarity between image features is cosine similarity or L2 distance.

[0030] To achieve the above-mentioned purpose of the invention, an embodiment further provides a data acquisition and processing system for cross-modal retrieval, comprising:

[0031] An acquisition module, configured to perform distributed parallel acquisition of multimodal data from a target open source data network, wherein the multimodal data includes text modal data and image modal data;

[0032] A cleaning module is used to clean the text modal data of special characters and invisible characters, and then store the cleaned text modal data and image modal data in different message queues;

[0033] A feature extraction module is used to extract features from each text and each image in the message queue using a feature extraction model to obtain text features and image features;

[0034] A matching module is used to obtain image-text combinations based on the similarity matching of text features and image features, and store the image features and text features of the image-text combinations as indexes in a database;

[0035] The retrieval module is used to screen and return matching picture-text combinations as retrieval results of the uploaded data based on the similarity between the uploaded data and the picture-text combinations in the database, wherein the uploaded data includes text data and image data.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] After collecting multimodal data in distributed parallel manner, deep data cleaning is achieved by processing special characters and invisible characters in text modal data, avoiding the influence of special characters and invisible characters on semantics, which leads to inaccurate indexing. The text features corresponding to the text modal data and the image features of the image model data are matched and associated by similarity. The resulting image-text combination cleverly introduces the semantic relevance of text and image. The database storage based on the image features and text features of the image-text combination as indexes omits additional indexes, making data storage simpler and more direct, and convenient for retrieval applications. In short, this method can quickly obtain multimodal data, effectively build a cross-modal retrieval data warehouse, realize automatic mining of large-scale multimodal data, save manual analysis costs, and is conducive to large-scale deployment and implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 is a flow chart of a data collection and processing method for cross-modal retrieval provided by an embodiment;

[0040] Figure 2 is another flow chart of the data collection and processing method for cross-modal retrieval provided by the embodiment;

[0041] Figure 3 3 is a structural diagram of a data acquisition and processing system for cross-modal retrieval provided in an embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0043] Figure 1 This is a flowchart of a data collection and processing method for cross-modal retrieval provided in an embodiment. Figure 2 This is another flow chart of the data collection and processing method for cross-modal retrieval provided by the embodiment. Figure 1 and Figure 2 As shown, the data collection and processing method for cross-modal retrieval provided in the embodiment includes the following steps:

[0044] Step 1: Perform distributed parallel collection of multimodal data on the target open source data network.

[0045] In an embodiment, multimodal data includes text modal data and image modal data. Multimodal data is collected through a distributed system, which can realize parallel collection of multimodal data by multiple processes in multiple nodes, and can also defend against interference attacks added at the text content level. Each process can collect text modal data or image modal data separately, and of course, can collect text modal data and image modal data simultaneously.

[0046] In the application, the distributed system provides different collection options according to the different modal data that need to be collected. For example, if the data to be collected is text modal data, the text modal data collection function is selected. After that, the distributed system can allocate corresponding collection tasks according to the corresponding nodes it owns, and hand over the corresponding collection requirements to specific nodes for execution.

[0047] Step 2: Preprocess the multimodal data and store it in the message queue.

[0048] In the embodiment, a data processing engine is used to pre-process the multimodal data, and when it is determined that the multimodal data includes text modal data, the text modal data is deep cleansed. The deep cleaning mainly includes cleaning the text modal data of special characters and invisible characters.

[0049] The specific cleaning process includes:

[0050] (2-1) Check the text modal data, traverse each text character, and observe whether it may be a special character or an invisible character;

[0051] (2-2) If there are special characters or invisible characters, the character category is determined by the character type;

[0052] (2-3) When the character category is determined to be an invisible character, a method of filtering the invisible characters is adopted for cleaning.

[0053] Invisible characters include the following: zero-length space (Unicode code U+200B), zero-length joiner (Unicode code U+200D), and zero-length non-joiner (Unicode code U+200C). These invisible characters are directly deleted from the text modal data to clean the text modal data of invisible characters.

[0054] (2-4) When the character category is determined to be a special character, different cleaning methods are used according to different special character categories.

[0055] In this embodiment, when a special character is a pictographic glyph, the pictographic glyph is replaced with the original character based on the mapping relationship between the pictographic glyph and the original character presented in the pictographic glyph table to achieve pictographic glyph cleaning. It should be noted that the pictographic glyph table is a predefined dictionary of visually similar characters, which stores the mapping relationship between the original character and visually similar characters (pictographic glyphs).

[0056] When special characters are deletion characters, different cleaning methods are used depending on the meaning of the deletion characters. Deletion characters include the following: Backspace character, whose Unicode code is U+8; Delete character, whose Unicode code is U+7F; and Carriage Return character, whose Unicode code is U+D.

[0057] When the deletion character is a backspace character, the backspace character and the character before it will be deleted at the same time; when the deletion character is a delete character, the delete character and the character after it will be deleted at the same time; when the deletion character is a carriage return character, all characters after the carriage return character will be overwritten from the beginning of the paragraph.

[0058] When the special character is a display sequence character, the sequence of the display sequence characters is changed according to the control, and a bottom-up recursive recovery is performed to finally restore the original text.

[0059] Among them, the display order characters include the following: PDF character, whose Unicode code is U+202C; LRE character, whose Unicode code is U+202A; RLE character, whose Unicode code is U+202B; LRO character, whose Unicode code is U+202D; RLO character, whose Unicode code is U+202E; PDI character, whose Unicode code is U+2069; LRI character, whose Unicode code is U+2066; RLI character, whose Unicode code is U+2067.

[0060] The sequence of characters that control changing the display order is [LRO, LRI, RLO, LRI, string1, PDI, LRI, string2, PDI, PDF, PDI, PDF].

[0061] Use bottom-up recursive recovery to restore the original text, including:

[0062] (a) Using a non-greedy matching algorithm, match a string sequence of the form [LRO, LRI, RLO, LRI, string 1, PDI, LRI, string 2, PDI, PDF, PDI, PDF] and replace the string sequence with a sequence of the form [string 2, string 1];

[0063] (b) repeating step (a) until the character string sequence no longer exists in the text modality data;

[0064] (c) If display sequence characters still exist in the remaining text modal data, all display sequence characters are deleted.

[0065] The cleaned text modality data and image modality data are stored in two message queues respectively to be read by the subsequent feature extraction application. It should be noted that in order to reduce the amount of data storage, only the link in the network is stored for the image modality data.

[0066] Step 3: Use the feature extraction model to extract features from the text and images in the message queue.

[0067] In the embodiment, the feature extraction model is constructed based on a deep learning model, specifically using a contrastive language-image pre-training (CLIP) model that simultaneously encodes text and images. The CLIP model consists of a text encoder and an image encoder. Its training method uses a large number of interrelated images and texts existing on the Internet to compare and learn these interrelated images and texts. Therefore, the CLIP model can well mine the relevant information between images and texts and can provide a relatively accurate cross-modal index. Therefore, the embodiment only needs to perform feature encoding through the encoder in the CLIP model.

[0068] In one embodiment, a data processing engine is used to implement feature encoding by calling a CLIP model through multiple threads. Specifically, each text read from a message queue is input into the CLIP model's text encoder to calculate text features, and each image read from the message queue is input into the CLIP model's image encoder to calculate image features.

[0069] Step 4: Match the similarity between text features and image features to construct and store image-text combinations.

[0070] In the embodiment, similarity matching is also implemented by a data processing engine, specifically including obtaining a picture-text combination based on similarity matching of text features and image features, and storing the picture-text combination in a database using the image features and text features of the picture-text combination as indexes. Specifically, during similarity matching screening, the similarity between the text features and all image features is calculated, and the image features with the greatest similarity are screened and matched with the text features to form a picture-text combination. It should be noted that the similarity can be cosine similarity or L2 distance. The database can be an elastic search database.

[0071] Step 5: Upload data to the retrieval application.

[0072] When searching the application, the uploaded data is received, wherein the uploaded data includes text data and image data, and then the matching image-text combinations are screened and returned as the search results of the uploaded data based on the similarity between the uploaded data and the image-text combinations in the database.

[0073] The retrieval application is implemented through a client with a data retrieval interface. For the user's query, the client first determines the requested query data modality, and then calls different encoding parts of the CLIP model for index encoding according to the different query data modalities.

[0074] The data retrieval interface is divided into two parts: image retrieval and text retrieval. In text retrieval, that is, when the uploaded data is text data, the text encoder of the feature extraction model is used to extract the text features of the text data, and the similarity between the text features and the text features in the database is calculated. The text feature with the greatest similarity in the database is used as the matching text feature, and the image-text combination to which the matching text feature belongs is used as the retrieval result. It should be noted that the similarity between text features is cosine similarity or L2 distance.

[0075] In image retrieval, that is, when the uploaded data is image data, the image encoder of the feature extraction model is used to extract the image features of the image data, and the similarity between the image features and the image features in the database is calculated. The image feature with the greatest similarity in the database is used as the matching image feature, and the image-text combination to which the matching image feature belongs is used as the retrieval result. It should be noted that the similarity between image features is cosine similarity or L2 distance.

[0076] The data collection and processing method for cross-modal retrieval provided in the above embodiment can collect and deeply process open source image and text data, thereby ultimately achieving high-quality cross-modal retrieval of image-text data.

[0077] Figure 3Schematic diagram of the structure of the data acquisition and processing system for cross-modal retrieval provided by the embodiment. Figure 3 As shown, the data acquisition and processing system provided in the embodiment includes:

[0078] An acquisition module, configured to perform distributed parallel acquisition of multimodal data from a target open source data network, wherein the multimodal data includes text modal data and image modal data;

[0079] A cleaning module is used to clean the text modal data of special characters and invisible characters, and then store the cleaned text modal data and image modal data in separate message queues;

[0080] A feature extraction module is used to extract features from each text and each image in the message queue using a feature extraction model to obtain text features and image features;

[0081] A matching module is used to obtain image-text combinations based on the similarity matching of text features and image features, and store the image features and text features of the image-text combinations as indexes in a database;

[0082] The retrieval module is used to screen and return matching picture-text combinations as retrieval results of the uploaded data based on the similarity between the uploaded data and the picture-text combinations in the database, wherein the uploaded data includes text data and image data.

[0083] The acquisition module collects data through a distributed system. Based on acquisition control commands, it performs distributed and parallel multimodal data collection on the target open source data network. This multimodal data is then transmitted back to the distributed system as a data stream. The cleaning module, feature extraction module, and matching module are implemented through a data processing engine. This data processing engine uses streaming methods to clean text modality data from different channels in the message queue, invoke feature extraction models, and perform feature matching. The resulting image-text combinations are then stored in a database.

[0084] It should be noted that in the data processing engine, users can select the data modality to be processed (e.g., image, text) and the processing method (single-modal processing method, multi-modal association processing) in the engine, thereby selecting three functions according to actual needs: feature extraction for images, feature extraction for text, and association analysis between multi-modal features of images and text. It should also be noted that after extracting multi-modal features such as text features and image features, the data processing engine also locates the multi-modal features, that is, locates the position of the text data and image data corresponding to the text features and image features on the web page.

[0085] The retrieval module is implemented through an interactive data retrieval interface. A comment request is sent through the data retrieval interface. Based on the request, the corresponding text features and image features are retrieved from the database according to the data index to perform similarity matching calculations. The image-text combination with the greatest similarity is selected and returned as the search response.

[0086] The data collection and processing of the above-mentioned system for cross-modal retrieval can quickly obtain multimodal data, effectively build a cross-modal retrieval data warehouse, and realize automatic mining of large-scale multimodal data, saving manual analysis costs and facilitating large-scale deployment and implementation.

[0087] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data collection and processing method for cross-modal retrieval, characterized in that: include: Perform distributed parallel collection of multimodal data on the target open source data network, where the multimodal data includes text modality data and image modality data; After cleaning the text modal data for special characters and invisible characters, the cleaned text modal data and image modal data are stored in different message queues; The feature extraction model is used to extract features from each text and each image in the message queue to obtain text features and image features. The image-text combination is obtained by matching and screening based on the similarity of the text features and image features. The image features and text features of the image-text combination are used as indexes for database storage. During retrieval, matching picture-text combinations are screened based on the similarity between the uploaded data and the picture-text combinations in the database and returned as retrieval results for the uploaded data, wherein the uploaded data includes text data or image data.

2. The data collection and processing method for cross-modal retrieval according to claim 1, characterized in that: Clean invisible characters from text modal data, including: Invisible characters in text modal data are filtered out, where invisible characters include zero-length spaces, zero-length connectors, and zero-length non-connectors.

3. The data collection and processing method for cross-modal retrieval according to claim 1 is characterized in that: Clean special characters from text modal data, including: When the special character is a pictographic glyph, the pictographic glyph is replaced with the original character according to the mapping relationship between the pictographic glyph and the original character presented in the pictographic character table; When the special character is a deletion character, different cleaning methods are used according to different deletion characters, including: when the deletion character is a backspace character, the backspace character and the character before the backspace character are deleted at the same time; when the deletion character is a delete character, the delete character and the character after the delete character are deleted at the same time; when the deletion character is a carriage return character, all characters after the carriage return character are overwritten from the start of the paragraph.

4. The data collection and processing method for cross-modal retrieval according to claim 1, characterized in that: Clean special characters from text modal data, including: When the special character is a display order character, a bottom-up recursive recovery is performed according to a sequence of control-changing display order characters to ultimately restore the original text, wherein the display order characters include: PDF character, LRE character, RLE character, LRO character, RLO character, PDI character, LRI character, and RLI character, and the sequence of control-changing display order characters is [LRO, LRI, RLO, LRI, string 1, PDI, LRI, string 2, PDI, PDF, PDI, PDF]. Use bottom-up recursive recovery to restore the original text, including: (a) Using a non-greedy matching algorithm, match a string sequence of the form [LRO, LRI, RLO, LRI, string1, PDI, LRI, string2, PDI, PDF, PDI, PDF] and replace the string sequence with a sequence of the form [string2, string1]; (b) repeating step (a) until the character string sequence no longer exists in the text modality data; (c) If display sequence characters still exist in the remaining text modal data, all display sequence characters are deleted.

5. The data collection and processing method for cross-modal retrieval according to claim 1, characterized in that: The image-text combination is obtained by matching and screening based on the similarity between text features and image features, including: Calculate the similarity between text features and all image features, and select the image features with the greatest similarity to match with text features to form an image-text combination.

6. The data collection and processing method for cross-modal retrieval according to claim 1, characterized in that: The method of screening matching picture-text combinations based on the similarity between the uploaded data and the picture-text combinations in the database as the retrieval results of the uploaded data includes: When the uploaded data is text data, the feature extraction model is used to extract the text features of the text data, and the similarity between the text features and the text features in the database is calculated. The text feature with the greatest similarity in the database is used as the matching text feature, and the image-text combination to which the matching text feature belongs is used as the search result; When the uploaded data is image data, the feature extraction model is used to extract the image features of the image data, and the similarity between the image features and the image features in the database is calculated. The image feature with the greatest similarity in the database is used as the matching image feature, and the image-text combination to which the matching image feature belongs is used as the retrieval result.

7. The data collection and processing method for cross-modal retrieval according to claim 1 or 6, characterized in that: The feature extraction module uses a comparative text-image pre-training model.

8. The data collection and processing method for cross-modal retrieval according to claim 1, 5 or 6, characterized in that: The similarity between text features and image features is cosine similarity or L2 distance; the similarity between text features and text features is cosine similarity or L2 distance; the similarity between image features and image features is cosine similarity or L2 distance.

9. A data acquisition and processing system for cross-modal retrieval, characterized in that: include: An acquisition module, configured to perform distributed parallel acquisition of multimodal data from a target open source data network, wherein the multimodal data includes text modal data and image modal data; A cleaning module is used to clean the text modal data of special characters and invisible characters, and then store the cleaned text modal data and image modal data in different message queues; A feature extraction module is used to extract features from each text and each image in the message queue using a feature extraction model to obtain text features and image features; A matching module is used to obtain image-text combinations based on the similarity matching of text features and image features, and store the image features and text features of the image-text combinations as indexes in a database; The retrieval module is used to screen and return matching picture-text combinations as retrieval results of the uploaded data based on similarities between the uploaded data and the picture-text combinations in the database, wherein the uploaded data includes text data or image data.

Images