Multi-modal data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-modal data processing method and device, electronic equipment and a storage medium, and relates to the technical field of data processing.The multi-modal data processing method comprises the steps that structured text content of text data is extracted, and structured picture content of picture data is extracted; on the basis of at least one of the semantic matching degree, the target matching degree and the space-time proximity between the structured text content and the structured picture content, calculating the correlation degree between each piece of text data and each piece of picture data; establishing a cross-modal association relationship between each piece of text data and each piece of picture data based on the association degree; and respectively carrying out associated storage on the cross-modal association relationship and the corresponding text data and picture data. According to the invention, efficient and accurate association of multi-modal data can be realized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multimodal data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of information technology, various industries have accumulated massive amounts of multimodal heterogeneous data, including text, images, videos, and other formats. This data is typically stored in silos across different systems and platforms, with varying data structures and interfaces, lacking inherent logical connections. This fragmentation in storage and management directly leads to difficulties in comprehensive data utilization, low query efficiency, and an inability to fully unlock the underlying data value.

[0003] In existing technologies, some methods attempt to perform cross-modal association, but most are limited to surface alignment based on simple metadata such as filenames and timestamps, or to post-association using manual rules or statistical methods after single-modal analysis (such as completing text keyword extraction and image object recognition separately). Essentially, this remains a mechanical splicing of multiple independent processing flows. These methods struggle to automatically uncover and establish deep relationships between different modalities, their processing flows are lengthy and inflexible, and their effectiveness heavily relies on prior knowledge and manual design, failing to meet the needs of intelligent querying and analysis of the intrinsic relationships between cross-modal data in complex scenarios.

[0004] Therefore, how to achieve efficient and accurate correlation of multimodal data has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] This invention provides a multimodal data processing method, apparatus, electronic device, and storage medium, which can achieve efficient and accurate correlation of multimodal data.

[0006] This invention provides a multimodal data processing method, comprising: Extract structured text content from text data and structured image content from image data; Based on at least one of the semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content, calculate the correlation degree between each text data and each image data; Based on the aforementioned correlation, a cross-modal correlation relationship is established between each text data and each image data; The cross-modal association relationships are then associated and stored with the corresponding text data and image data, respectively.

[0007] According to a multimodal data processing method provided by the present invention, before calculating the correlation degree between each text data and each image data based on at least one of semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content, the method further includes: The structured text content is stored in the first index of the distributed search engine, and the structured image content is stored in the second index of the distributed search engine; The calculation of the correlation between each text data and each image data based on at least one of the semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content includes: Using the first index and the second index, cross-modal retrieval is performed based on at least one of the semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content; Based on the cross-modal retrieval results, the correlation between each text data and each image data is calculated.

[0008] According to a multimodal data processing method provided by the present invention, the step of calculating the correlation degree between each text data and each image data based on at least one of semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content includes: Based on the keywords in the structured text content and the visual features in the structured image content, the semantic matching degree is calculated; Calculate the target matching degree based on the named entities in the structured text content and the target objects in the structured image content; Based on the named entities and / or text meta-information in the structured text content, and the image meta-information in the structured image content, the spatiotemporal proximity is calculated; The semantic matching degree, the target matching degree, and the spatiotemporal proximity are weighted and summed to obtain the correlation degree between each text data and each image data.

[0009] According to a multimodal data processing method provided by the present invention, the step of associating and storing the cross-modal association relationship with corresponding text data and image data respectively includes: The cross-modal association relationship is stored in the first cross-modal association field of the corresponding text data and in the second cross-modal association field of the corresponding image data; The first cross-modal association field and the second cross-modal association field are configured as nested object types.

[0010] According to a multimodal data processing method provided by the present invention, the multimodal data processing method further includes: Obtain the lineage information of the text data and / or the image data, wherein the lineage information includes at least one of data source, derivative relationship and version relationship; The blood relationship information is stored in the blood relationship field of the corresponding data.

[0011] According to a multimodal data processing method provided by the present invention, the structured text content includes at least one of keywords, named entities, and categorized topics, and the multimodal data processing method further includes: Based on the keywords, calculate the first similarity between each text data, and establish a first association relationship based on the first similarity; Based on the named entity, a first target text data including the same named entity is determined, and a second association relationship is established based on the first target text data; Based on the classification topic, determine the second target text data belonging to the same classification topic, and establish a third association based on the second text data; Store at least one of the first association, the second association, and the third association into the text association field of the corresponding text data.

[0012] According to a multimodal data processing method provided by the present invention, the structured image content includes at least one of visual features and target objects, and the multimodal data processing method further includes: Based on the visual features, a second similarity is calculated between each image data, and a fourth association relationship is established based on the second similarity; Based on the target object, target image data including the same target object are determined, and a fifth association relationship is established based on the target image data. Store at least one of the fourth and fifth association relationships into the image association field of the corresponding image data.

[0013] According to a multimodal data processing method provided by the present invention, before extracting the structured text content of text data and the structured image content of image data, the method further includes: The text data and the image data are preprocessed; The preprocessed text data is subjected to text quality detection, and corresponding text correction processing is performed based on the text quality detection results; the text quality detection includes at least one of accuracy detection, syntax consistency detection and duplicate data detection. The preprocessed image data is subjected to image quality detection, and corresponding image correction processing is performed based on the image quality detection results; the image quality detection includes at least one of integrity detection, noise detection, and blur detection.

[0014] The present invention also provides a multimodal data processing apparatus, comprising: The structured content extraction module is used to extract structured text content from text data and structured image content from image data. The correlation calculation module is used to calculate the correlation between each text data and each image data based on at least one of the semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content. A cross-modal association establishment module is used to establish cross-modal association relationships between each text data and each image data based on the association degree. The cross-modal association storage module is used to associate and store the cross-modal association relationship with the corresponding text data and image data, respectively.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal data processing method described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multimodal data processing method as described above.

[0017] The multimodal data processing method, apparatus, electronic device, and storage medium provided by this invention extracts structured text content from text data and structured image content from image data, transforming multi-source heterogeneous unstructured text and image data into structured content rich in semantic and content features, laying the foundation for deep cross-modal data association. Then, based on at least one of semantic matching degree, target matching degree, and spatiotemporal proximity between structured text content and structured image content, the association degree between each text data and each image data is calculated. Based on the association degree, cross-modal association relationships between each text data and each image data are established, realizing the automatic and quantitative mining of deep cross-modal data association relationships from a multi-dimensional content level, significantly improving the accuracy and processing efficiency of association. Finally, the cross-modal association relationships are associated and stored with the corresponding text and image data respectively, making the association relationships structured data that can be directly indexed and queried. This provides efficient and reliable data support for intelligent cross-modal retrieval and analysis in complex scenarios, meeting practical application needs. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts of the multimodal data processing method provided by the present invention.

[0020] Figure 2 This is the second flowchart of the multimodal data processing method provided by the present invention.

[0021] Figure 3 This is the third flowchart of the multimodal data processing method provided by the present invention.

[0022] Figure 4 This is the fourth flowchart of the multimodal data processing method provided by the present invention.

[0023] Figure 5 This is the fifth flowchart of the multimodal data processing method provided by the present invention.

[0024] Figure 6 This is a schematic diagram of the structure of the multimodal data processing device provided by the present invention.

[0025] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] This invention proposes a multimodal data processing method, apparatus, electronic device, and storage medium, which are described below in conjunction with... Figures 1-7 Describe it.

[0028] Figure 1 This is one of the flowcharts illustrating the multimodal data processing method provided by the present invention, such as... Figure 1 As shown, the multimodal data processing method includes steps S110, S120, S130 and S140.

[0029] Step S110: Extract the structured text content from the text data and extract the structured image content from the image data.

[0030] Text data refers to digitized text containing textual information, which may include, but is not limited to, text documents, Word documents, PDF documents, etc.

[0031] Image data refers to digital images containing visual information, which may include, but are not limited to, photographs, scans, charts, screenshots, etc.

[0032] Structured text content includes at least one of keywords, named entities, and category topics, and may also include text meta-information. This text meta-information may include, but is not limited to, document format, page number, title, and upload time.

[0033] The extraction process for keywords, named entities, and category topics can be referred to in the following embodiment, and will not be elaborated here. The extraction process for text metadata is as follows: Meta-information is extracted from the text data, and the extracted metadata is recorded as text metadata. Specifically, firstly, the document format of the text data is identified by its file extension, such as txt, doc, docx, PDF, etc.; then, for text data of different formats, the corresponding parser is used to extract the text content and page number. Simultaneously, attribute information of the text data, such as upload time, is obtained; preliminary analysis is performed on the extracted text content, identifying the title of the text from the first line or a specific position.

[0034] Structured image content includes at least one of visual features and target objects, and may also include: image metadata and image text information. Image metadata may include, but is not limited to: resolution, image format, color space, shooting time, and geographic location information.

[0035] The process of extracting visual features and target objects can be referred to in the following embodiments, which will not be elaborated here.

[0036] The process of extracting image metadata is as follows: For basic file attributes, they are directly obtained from the file header information, such as resolution and image format. For images that include EXIF ​​(Exchangeable Image File Format) information, the Python Pillow library (an image processing library) can be used to extract EXIF ​​metadata, which is referred to as image metadata.

[0037] The process of extracting text information from images is as follows: the text content in the image data is extracted using OCR (Optical Character Recognition), and the extracted text is recorded as the image text information.

[0038] By transforming unstructured text and image data into structured content rich in semantic and content features, a unified and computable data foundation is provided for subsequent association calculations.

[0039] Step S120: Calculate the correlation between each text data and each image data based on at least one of the semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content.

[0040] The correlation between each text data and each image data is calculated based on at least one of the semantic matching degree, target matching degree, and spatiotemporal proximity between structured text content and structured image content.

[0041] In one embodiment, semantic matching degree is calculated based on keywords in structured text content and visual features in structured image content; target matching degree is calculated based on named entities in structured text content and target objects in structured image content; spatiotemporal proximity is calculated based on named entities and / or text meta-information in structured text content and image meta-information in structured image content; any one of semantic matching degree, target matching degree, and spatiotemporal proximity is used as the correlation degree between each text data and each image data.

[0042] In another implementation, semantic matching degree is calculated based on keywords in structured text content and visual features in structured image content; target matching degree is calculated based on named entities in structured text content and target objects in structured image content; spatiotemporal proximity is calculated based on named entities and / or text metadata in structured text content and image metadata in structured image content; at least two of the semantic matching degree, target matching degree, and spatiotemporal proximity are weighted and summed to obtain the correlation degree between each text data and each image data. The specific execution process can be referred to the following embodiments, which will not be elaborated here.

[0043] Through multi-dimensional and quantifiable computation, deep connections between text data and image data based on semantics, content (the objects contained), spatiotemporal context, and other factors are automatically discovered, thereby constructing cross-modal associations.

[0044] Step S130: Based on the correlation degree, establish cross-modal correlation relationships between each text data and each image data.

[0045] Based on the correlation degree calculated above, text data-image data pairs that meet the preset correlation conditions are identified as having cross-modal correlation, thereby establishing cross-modal correlation relationships between each text data and each image data.

[0046] The preset association conditions can be set according to the calculation method of the association degree and actual needs, and are not limited here. For example, if any one of semantic matching degree, target matching degree, and spatiotemporal proximity is used as the association degree, different thresholds can be set according to different dimensions. If it is greater than the corresponding threshold, cross-modal association is determined to exist. Similarly, if semantic matching degree, target matching degree, and spatiotemporal proximity are all used as the association degree, different thresholds can be set according to different dimensions. If semantic matching degree, target matching degree, and spatiotemporal proximity are all greater than their corresponding thresholds, cross-modal association is determined to exist. Furthermore, if the weighted sum of semantic matching degree, target matching degree, and spatiotemporal proximity is used as the association degree, a threshold can be set. If it is greater than this threshold, cross-modal association is determined to exist.

[0047] Step S140: The cross-modal association relationship is associated and stored with the corresponding text data and image data respectively.

[0048] In one implementation, a traditional relational database intermediate table strategy is used for relational storage. Specifically, text data and image data are stored in different tables, such as the `text_documents` table and the `image_documents` table. The structured text content of the text data and the structured image content of the image data are stored in their respective tables. Simultaneously, an intermediate table for relational relationships is created, such as the `cross_modal_relations` table, containing fields such as: `id` (auto-incrementing primary key), `text_id` (foreign key, related to the text table), `image_id` (foreign key, related to the image table), `relation_strength` (relation strength), `relation_type` (relation type), and `relation_created_time` (creation time). The aforementioned cross-modal relationships (including the associated text ID and image ID, relation strength, and relation type) are inserted into the `cross_modal_relations` table.

[0049] In another implementation, a pre-computation + indexing strategy is used for associated storage. Specifically, structured text content is stored in a first index of a distributed search engine, and structured image content is stored in a second index of the same distributed search engine, preferably Elasticsearch. Cross-modal relationships are stored in a first cross-modal relationship field of the corresponding text data and a second cross-modal relationship field of the corresponding image data. For example, a `related_image` field is added to the first index of the text data. For image data associated with the text data, a nested object is stored in this field, which may include, but is not limited to, `image_id`, `relation_strength`, `relation_type`, `relation_created_time`, and `relation_confidence`. Simultaneously, a `related_texts` field is added to the second index of the image data. For text data associated with the image data, a nested object is stored in this field, which may include, but is not limited to, `text_id`, `relation_strength`, `relation_type`, `relation_created_time`, and `relation_confidence`.

[0050] It should be noted that the associated storage method in the second embodiment above has significant advantages over the associated storage method in the first embodiment in several key dimensions such as query performance, scalability, development efficiency and business support, especially in business scenarios that process massive amounts of data, require real-time response and support complex queries.

[0051] By using the above method, the dynamically calculated cross-modal relationships are persisted and stored in a structured manner, enabling them to be efficiently retrieved, filtered, and aggregated like ordinary data fields, facilitating subsequent cross-modal data association queries.

[0052] The multimodal data processing method provided in this invention extracts structured text content from text data and structured image content from image data, transforming multi-source heterogeneous unstructured text and image data into structured content rich in semantic and content features, laying the foundation for deep cross-modal data association. Then, based on at least one of semantic matching degree, target matching degree, and spatiotemporal proximity between structured text content and structured image content, the association degree between each text data and each image data is calculated. Based on the association degree, cross-modal association relationships are established between each text data and each image data, enabling automatic and quantitative mining of deep cross-modal data association relationships from a multi-dimensional content level, significantly improving the accuracy and processing efficiency of the association. Finally, the cross-modal association relationships are associated and stored with the corresponding text and image data, making the association relationships directly indexable and queried structured data. This provides efficient and reliable data support for intelligent cross-modal retrieval and analysis in complex scenarios, meeting practical application needs.

[0053] Based on any of the above embodiments Figure 2 This is the second flowchart of the multimodal data processing method provided by the present invention, as shown below. Figure 2 As shown, before step S120, the procedure further includes: Step S150: Store the structured text content in the first index of the distributed search engine, and store the structured image content in the second index of the distributed search engine.

[0054] The structured text content of the text data is stored in the first index of the distributed search engine, for example, named text_documents. At the same time, the structured image content of the image data is stored in the second index of the distributed search engine, for example, named image_documents.

[0055] It should be understood that each piece of text data is assigned a unique identifier, text_id, and each piece of image data is assigned a unique identifier, image_id.

[0056] Furthermore, the distributed search engine is preferably Elasticsearch, which has advanced features such as inverted indexes, vector retrieval, and nested queries, making it easy to retrieve and query efficiently.

[0057] Specifically, when storing structured text content in Elasticsearch, an index (denoted as the first index) is first created for the text data. The structure is defined by setting the field types through mapping. A mapping definition can be as follows: PUT / text_documents { "mappings": { "properties": { "document_id": { "type": "keyword"}, "format": { "type": "keyword"}, "pages": { "type": "integer"}, "title": { "type": "text"}, "upload_time": { "type": "date"}, "classification": { "type": "keyword"}, "keywords": { "type": "text"}, "topic": { "type": "text"}, "entities": { "type": "nested", "properties": { "entity": { "type": "text"}, "type": { "type": "keyword"} } } } } } For each piece of text data, the extracted structured text content is organized into a JSON (JavaScript Object Notation) document format that conforms to the index mapping, and the JSON document is stored in the first index of Elasticsearch.

[0058] Similarly, for each image data entry, the extracted structured image content is organized into a JSON document format that conforms to the index mapping, and the JSON document is stored in the second index of Elasticsearch. For example, the JSON document corresponding to a certain image data entry is as follows: { "document_id": "image12345", "type": "image", format: "jpg", resolution: 1920x1080 "color_space": "RGB", "capture_time": "2025-01-22T14:30:00Z", "location": { "lat": 40.7128, "lon": -74.0060}, "extracted_text": "Example extracted text from image", "detected_objects": [ { "object": "person", "coordinates": { "x_min": 50, "y_min": 120, "x_max": 150, "y_max": 300}}, { "object": "car", "coordinates": { "x_min": 200, "y_min": 50, "x_max": 400, "y_max": 250}} ], "visual_features": { "color_histogram": [0.1, 0.05, ...], "texture_features": [0.02, 0.03, ...], }, "image_url": "https: / / minio.example.com / image12345.jpg" } It should be noted that text data can also be stored in Elasticsearch, while image data needs to be stored in a dedicated object storage system, such as MinIO (Mini Object Storage) or Amazon S3 (Amazon Simple Storage Service), and accessed through a URL (Uniform Resource Locator) or a unique identifier.

[0059] At this time, step S120 includes: step S121 and step S122.

[0060] Step S121: Using the first index and the second index, perform cross-modal retrieval based on at least one of the semantic matching degree, target matching degree and spatiotemporal proximity between the structured text content and the structured image content.

[0061] In the first index (i.e., the text_documents index), an inverted index is built for one or more of the following fields: keywords, entities.entity, topic, and upload_time.

[0062] Meanwhile, in the second index (i.e., the image_documents index), inverted indexes are built for one or more of the following fields: detected_objects (named entities), capture_time (capture time), and location (geographical information), and vector indexes are built for the visual_features field, such as the HNSW (HierarchicalNavigable Small World) index, to support KNN (K-Nearest Neighbors) similarity queries.

[0063] The first index is used to retrieve keywords in the structured text content, and the second index's vector index is used to retrieve visual features in the structured image content, which are then used to calculate the semantic matching degree.

[0064] The first index is used to retrieve named entities in the structured text content, and the second index is used to retrieve target objects in the structured image content, in order to calculate the target matching degree.

[0065] The first index is used to retrieve named entities and / or text meta-information (specifically, upload time) from the structured text content, and the second index is used to retrieve image meta-information (specifically, shooting time and geographic location information) from the structured image content, in order to calculate spatiotemporal proximity.

[0066] Step S122: Based on the cross-modal retrieval results, calculate the correlation between each text data and each image data.

[0067] Based on the cross-modal retrieval results, the correlation degree between each text data and each image data is calculated. Specific calculation methods can be found in relevant implementation examples, and will not be elaborated here.

[0068] The multimodal data processing method provided in this invention can greatly improve retrieval efficiency by storing structured text content and structured image content in different indexes in a distributed search engine, thereby supporting efficient cross-modal correlation calculation.

[0069] Based on any of the above embodiments Figure 3 This is the third flowchart of the multimodal data processing method provided by the present invention, as shown below. Figure 3 As shown, step S120 further includes steps S123, S124, S125 and S126.

[0070] Step S123: Calculate the semantic matching degree based on the keywords in the structured text content and the visual features in the structured image content.

[0071] Keywords are extracted from structured text content and converted into keyword vectors using a word vector model. A weighted average of these keyword vectors yields the text semantic vector. Simultaneously, visual features are extracted from structured image content and mapped to the same space as the text semantic vector using a pre-trained projection matrix, resulting in an image semantic vector. The cosine similarity between the text and image semantic vectors is then calculated as the semantic matching score.

[0072] Step S124: Calculate the target matching degree based on the named entities in the structured text content and the target objects in the structured image content.

[0073] The target matching degree is calculated as follows: The number of matching pairs is determined based on named entities in structured text content and target objects in structured image content. Specifically, for each named entity in the text data, a search is conducted to determine if a corresponding target object exists in the image data, thus obtaining the number of matching pairs. Then, based on the number of matching pairs, the total number of named entities, and the total number of target objects, the target matching degree is calculated.

[0074] In one implementation, the target matching degree can be calculated as follows: Target matching degree = Number of matching pairs / (Total number of named entities + Total number of target objects). For example, if text data A corresponds to 3 named entities and image data B corresponds to 2 target objects, and the number of matching pairs is 1, then the target matching degree = 1 / (2 + 3) = 0.2.

[0075] It should be understood that the calculation method for the target matching degree can be set according to actual needs. The rule is: the more matching pairs there are, the greater the target matching degree.

[0076] Step S125: Calculate the spatiotemporal proximity based on the named entities and / or text meta-information in the structured text content and the image meta-information in the structured image content.

[0077] The spatiotemporal proximity is calculated as follows: Retrieve named entities and / or text metadata from structured text content, specifically, retrieve the upload time from the text metadata; simultaneously, retrieve image metadata from structured image content, specifically, retrieve the shooting time and geographic location information from the image metadata.

[0078] Spatial proximity is calculated based on named entities and geographic location information. Specifically, the spatial proximity is calculated as follows: The geographic location information of the text is determined based on the named entity. For example, if the named entity is "Forbidden City," the geographic location information of the text can be determined as the location of the Forbidden City. Then, the geographic distance Δd between the geographic location information of the text and the geographic location information of the image is calculated. Based on this geographic distance, the spatial proximity is calculated using the following formula: ; in, represents spatial proximity; D represents distance attenuation coefficient, which can be preset according to actual needs, for example, set to 10 kilometers, without specific limitation here.

[0079] The time proximity is calculated based on the upload time and the capture time. Specifically, the time proximity is calculated as follows: the difference between the upload time and the image capture time is denoted as the time difference Δt. Based on this time difference, the time proximity is calculated using the following formula: ; in, represents time proximity; T represents time decay coefficient, which can be preset according to actual needs, for example, set to 24 hours, without specific limitation here.

[0080] Finally, the spatial proximity and temporal proximity are weighted and summed to obtain the spatiotemporal proximity. The specific formula is as follows: .

[0081] in, Indicates spatiotemporal proximity; Indicates time weight, Spatial weights can be used to determine temporal weights based on the application scenario. and spatial weights .

[0082] It should be noted that when only time information is available, When only spatial information is available, .

[0083] Step S126: The semantic matching degree, the target matching degree, and the spatiotemporal proximity are weighted and summed to obtain the correlation degree between each text data and each image data.

[0084] The correlation between each text data and each image data is obtained by weighted summation of semantic matching degree, target matching degree and spatiotemporal proximity.

[0085] The multimodal data processing method provided in this invention establishes deep associations between text and images from multiple dimensions, including semantics, content, and spatiotemporal context. The associations are of high quality and highly practical.

[0086] Based on any of the above embodiments, step S140 includes: step S141.

[0087] Step S141: Store the cross-modal association relationship in the first cross-modal association field of the corresponding text data, and store it in the second cross-modal association field of the corresponding image data.

[0088] The first cross-modal association field and the second cross-modal association field are configured as nested object types.

[0089] Nested object structures are a special object type in Elasticsearch that allows each object in an array of objects to be indexed and queried independently. In Elasticsearch mappings, cross-modal join fields are defined as nested types.

[0090] Cross-modal relationships are stored in the corresponding cross-modal relationship field of the text data (denoted as the first cross-modal relationship field). For example, a `related_image` field of type `nested` is added to the first index of the text data. For image data associated with this text data, a nested object is stored in this field. The attributes of the nested object may include, but are not limited to: `image_id`, `relation_strength`, `relation_type`, `relation_created_time`, and `relation_confidence`.

[0091] Simultaneously, cross-modal relationships are stored in the corresponding cross-modal relationship field of the image data (denoted as the second cross-modal relationship field). For example, a `related_texts` field of type `nested` is added to the second index of the image data. For text data associated with this image data, a nested object is stored in this field. The attributes of the nested object may include, but are not limited to: `text_id`, `relation_strength`, `relation_type`, `relation_created_time`, and `relation_confidence`.

[0092] It should be understood that each cross-modal association field of text or image data can store multiple nested objects. The nested objects are independent of each other, and there will be no problem of flattening and merging of field values.

[0093] The multimodal data processing method provided in this invention stores each cross-modal relationship as an independent nested object, ensuring they do not interfere with each other and avoiding the flattening problem of traditional object arrays. Furthermore, this storage method allows cross-modal relationships to become part of text or image data, and enables efficient querying using Elasticsearch's inverted index on nested fields.

[0094] Based on any of the above embodiments, the multimodal data processing method further includes: step S161 and step S162.

[0095] Step S161: Obtain the lineage information of the text data and / or the image data, wherein the lineage information includes at least one of data source, derivative relationship and version relationship.

[0096] In key processing stages such as the creation, processing, modification, or derivation of text or image data, blood relationship information is collected and generated, which includes at least one of the following: data source, derived relationship, and relationship.

[0097] Data source refers to the original data or parent data that generated the current data.

[0098] Derivation relationships refer to the processing chain and operational semantics that record data from source data to current data. For example, the image IMG_001 is derived into IMG_001_thumb through a cropping operation, and the document DOC_A is derived into DOC_A_EN through a machine translation operation.

[0099] Version relationships are a special case of derivative relationships, specifically used to record the state sequence of the same data at different points in time, such as document v1 to document v2.

[0100] Step S162: Store the blood relationship information in the blood relationship field of the corresponding data.

[0101] Bloodline information is stored in the corresponding bloodline field of the data, such as the lineage field. This bloodline field includes multiple attributes, including but not limited to: Parent data (parent_ids) is used to store the IDs of all parent data. Child data (child_ids) is used to store the IDs of all child data (i.e., derived data); The derivation history is a nested array of objects that records detailed derivation relationships. Each object contains subfields such as operation (operation type), operator (operator), parameters (operation parameters), and timestamp (operation time). Version information (version_info) is used to record version relationships, such as current_version (current version number), previous_version_id (previous version ID), and version_changes (version change description).

[0102] The multimodal data processing method provided in this invention, by acquiring the pedigree information of data and storing it in a pedigree field, supports data pedigree tracing, making it easier for users to understand the source and evolution of the data. In particular, for cross-modal data association, it enhances the interpretability and reliability of the association.

[0103] Based on any of the above embodiments Figure 4 This is the fourth flowchart of the multimodal data processing method provided by the present invention, as shown below. Figure 4 As shown, the structured text content includes at least one of keywords, named entities, and category topics, and the multimodal data processing method further includes steps S171, S172, S173, and S174.

[0104] Step S171: Calculate the first similarity between the text data based on the keywords, and establish a first association relationship based on the first similarity.

[0105] Structured text content includes at least one of keywords, named entities, and category topics.

[0106] The keyword extraction process involves using a keyword extraction algorithm to extract keywords from the text data. These algorithms may include, but are not limited to, TextRank and TF-IDF (Term Frequency-Inverse Document Frequency).

[0107] In one implementation, based on the TextRank algorithm, popular keywords and weight values ​​are extracted from the text data, and the five words with the highest weight values ​​are obtained as text keywords.

[0108] Furthermore, to improve the accuracy of keyword extraction, part-of-speech tagging and stop word filtering can be performed on the text data before keyword extraction.

[0109] Furthermore, words with specific parts of speech (such as nouns) can be prioritized as text keywords.

[0110] Using the inverted index mechanism of a distributed search engine, an index is built for the keyword field of all text data. When it is necessary to establish a connection for text data A, the inverted index mechanism is used to retrieve other text data containing any keywords of text data A. Then, the similarity between the keyword set of text data A and the keyword sets of these other text data is calculated and denoted as the first similarity. Text data with a first similarity exceeding a first preset threshold are identified as associated text data of text data A, forming a keyword-based association relationship, denoted as the first association relationship.

[0111] Furthermore, when calculating the first similarity, the Jaccard similarity algorithm can be used to calculate the degree of overlap of keywords between text data.

[0112] Step S172: Based on the named entity, determine the first target text data that includes the same named entity, and establish a second association relationship based on the first target text data.

[0113] The process of extracting named entities is as follows: perform named entity recognition on the text data to obtain the named entity recognition result. The named entity recognition result includes at least the named entity, and may also include at least one of entity type and entity location.

[0114] When performing named entity recognition, the BERT-based Named Entity Recognition (NER) model can be used. This model is a pre-trained model based on the BERT architecture specifically designed for named entity recognition tasks. Entity types include basic entity types and extended entity types. Basic entity types cover person names, place names, organization names, time and numeric quantifiers; extended entity types can be dynamically configured according to the text domain, such as disease names and drug names in the medical field, and stock codes and currency units in the financial field.

[0115] When it is necessary to establish a relationship for text data A, other text data including all named entities of text data A are retrieved through the inverted index mechanism. This is denoted as the first target text data and used as the associated text data of text data A to form a relationship based on named entities, which is denoted as the second relationship.

[0116] Step S173: Based on the classification topic, determine the second target text data belonging to the same classification topic, and establish a third association relationship based on the second target text data.

[0117] The process of extracting categorized topics is as follows: the text data is categorized by topic classification model to obtain topic classification results. The topic classification results include at least categorized topics and may also include confidence scores.

[0118] A topic classification model is pre-trained. Text data is input into the topic classification model to perform topic classification, and the topic classification results output by the topic classification model are obtained. The initial model used for the topic classification model includes, but is not limited to, BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (A Robustly Optimized BERT Pretraining Approach), and FastText. The initial model is fine-tuned using sample text data to obtain the topic classification model.

[0119] Furthermore, when classifying topics, a hierarchical classification strategy can be adopted. First, a coarse-grained primary classification can be carried out, such as dividing into major categories like technical documents, business reports, and academic papers. Then, a fine-grained secondary classification can be carried out based on the primary classification, such as further dividing the technical document category into subcategories like software instructions, hardware manuals, and protocol specifications.

[0120] When it is necessary to establish a relationship for text data A, other text data with the same category theme as text data A are retrieved through the inverted index mechanism and denoted as the second target text data. These text data are then used as the associated text data of text data A to form a relationship based on the category theme, which is denoted as the third relationship.

[0121] Step S174: Store at least one of the first association, the second association, and the third association into the text association field of the corresponding text data.

[0122] Store at least one of the first association, the second association, and the third association into the text association field of the corresponding text data.

[0123] For the first association, the ID of the filtered associated text data is stored in the first association field of the text data, such as the related_documents field.

[0124] For the second association, the filtered associated text data IDs and their details (such as the associated named entities and entity types) are stored in the second association field of the text data, such as the related_entities field. Furthermore, the related_entities field is configured as a nested object type to independently store and manage the association records of multiple named entities.

[0125] For the third association, the filtered associated text data IDs and their details (such as the associated category topics) are stored in the third association field of the text data, such as the related_topics field, completing the logical association at the category topic level. Furthermore, the related_topics field is configured as a nested object type to maintain data independence.

[0126] By storing different related dimensions in different fields, great flexibility is provided for subsequent queries.

[0127] The text association field includes at least one of the first association field, the second association field, and the third association field.

[0128] The multimodal data processing method provided in this invention, through cross-associations of keywords, named entities, and category topics, can capture complex relationships between text data at different granularities. This multi-level, multi-granularity association reveals the relationships between text data more comprehensively and deeply, far exceeding single-dimensional associations. Furthermore, based on the constructed association network, it supports efficient querying of relevant text data from different dimensions.

[0129] Based on any of the above embodiments Figure 5This is the fifth flowchart of the multimodal data processing method provided by the present invention, as shown below. Figure 5 As shown, the structured image content includes at least one of visual features and target objects, and the multimodal data processing method further includes steps S181, S182 and S183.

[0130] Step S181: Based on the visual features, calculate the second similarity between each image data, and establish a fourth association relationship based on the second similarity.

[0131] Visual features include at least one of color features and texture features. Color features are used to describe the global or local distribution of colors in an image, and extraction methods include color histograms, color moments, etc. Texture features are used to describe the variation of gray levels on the image surface, and extraction methods include Local Binary Patterns (LBP) and Gray-Level Co-occurrence Matrix (GLCM), etc.

[0132] In one embodiment, the similarity between color features of each image data is calculated, or the texture similarity between texture features of each image data is calculated, as the similarity between the image data (denoted as the second similarity). In another embodiment, the similarity between the color features of each image data is calculated and denoted as color similarity. At the same time, the similarity between the texture features of each image data is calculated and denoted as texture similarity. Then, the color similarity and texture similarity are weighted and summed to obtain the second similarity between the image data.

[0133] After calculating the second similarity, if the second similarity exceeds the second preset threshold, the corresponding image data is identified as associated image data, forming an association relationship based on visual features, which is denoted as the fourth association relationship.

[0134] Step S182: Based on the target object, determine target image data that includes the same target object, and establish a fifth association relationship based on the target image data.

[0135] The target object detection process is as follows: a deep learning-based target detection model, such as YOLO (You Only Look Once, target recognition model) or its improved version, is used to detect targets in the image data to obtain various target objects and their location coordinates in the image data. Target objects include, but are not limited to: people, faces, vehicles, animals, furniture, buildings, natural scenery, etc.

[0136] In one embodiment, image data containing the same target object is retrieved through an inverted index mechanism and used as associated image data to form an association relationship based on the target object, which is denoted as the fifth association relationship.

[0137] In another implementation, an inverted index mechanism is used to retrieve image data containing the same target object. Then, based on location coordinates, related image data is filtered from these retrieved images to form a target object-based association, denoted as the fifth association. During filtering, for target objects of the same category, the intersection-over-union (IoU) ratio of the bounding boxes can be calculated based on their location coordinates. If the IoU ratio exceeds a third preset threshold, the corresponding image data is determined as related image data.

[0138] Step S183: Store at least one of the fourth association relationship and the fifth association relationship into the image association field of the corresponding image data.

[0139] Store at least one of the fourth and fifth association relationships in the image association field of the corresponding image data.

[0140] For the fourth association, the filtered associated text data IDs are stored in the fourth association field of the image data, such as the related_images field.

[0141] For the fifth association, the filtered associated text data IDs and their details (such as the associated target object and location coordinates) are stored in the fifth association field of the text data, such as the related_objects field. Furthermore, the fifth association field related_objects is configured as a nested object type to independently store and manage the association records of multiple target objects.

[0142] The multimodal data processing method provided in this invention captures the complex relationships between image data through two dimensions: visual features and target objects, enabling a more comprehensive and in-depth revelation of these relationships. Furthermore, based on the constructed association network, it supports efficient querying of relevant image data from different dimensions.

[0143] Based on any of the above embodiments, before step S110, the method further includes steps S101, S102, and S103.

[0144] Step S101: Preprocess the text data and the image data.

[0145] In this embodiment of the invention, a data integration platform supporting multiple data types is constructed for multi-source heterogeneous data environments. The data integration platform includes various unstructured data access modules, supporting the collection of text files (such as txt and doc) and image files (such as jpg and png) in different formats. Supported interface types include web service interfaces, HTTP interfaces, and socket interfaces, thereby enabling the acquisition of text and image data from different data sources through different interfaces.

[0146] After acquiring text and image data, the data integration platform provides a configurable preprocessing pipeline to support preprocessing of the text and image data. Preprocessing methods include, but are not limited to, cleaning, transformation, and standardization. Cleaning includes, but is not limited to, removing noisy data, duplicate data, and format correction; transformation includes, but is not limited to, format conversion, encoding conversion, and data structuring; and standardization includes, but is not limited to, data normalization, time standardization, and unit unification.

[0147] Of course, the data integration platform also supports user-defined preprocessing rules. Specifically, it receives user-defined cleaning, transformation, and standardization rules, and then preprocesses the text and image data based on these rules.

[0148] Furthermore, the preprocessed text data is used to support full-text search and complex queries; image data is stored in the distributed object storage MinIO to achieve highly available and low-cost file management.

[0149] In addition, the data integration platform maintains an open architecture and can be adapted to other big data storage components such as HDFS (Hadoop Distributed File System), HBase (Hadoop Database, a distributed, column-oriented NoSQL database), Hive (a data warehouse tool on Hadoop), and Solr (an open-source enterprise search server built on Lucene) to meet the storage and analysis needs of different business scenarios.

[0150] Step S102: Perform text quality detection on the preprocessed text data, and perform corresponding text correction processing based on the text quality detection results; the text quality detection includes at least one of accuracy detection, grammatical consistency detection and duplicate data detection.

[0151] In this embodiment of the invention, a data quality platform is constructed to address data quality issues, providing the capability to inspect the quality of multi-source, heterogeneous, and multimodal data. Specifically, it can be used for text quality detection and correction of text data, and image quality detection and correction of image data.

[0152] The text quality detection and corresponding text correction process are as follows: (1) Accuracy detection and correction: Spelling errors in text data are detected using the Natural Language Processing (NLP) tool Spacy and spell checking libraries (such as pyspellchecker and Hunspell). For misspelled words, spelling correction tools are used for automatic correction, or correction is performed according to dictionary rules, or correction is performed through manual review.

[0153] (2) Syntactic consistency detection and correction: A deep learning model is used to check whether the syntactic structure in the text data conforms to standard rules, mark syntactic errors, and provide the error type and location. For places with syntactic errors, automatic correction can be performed according to preset rules, or correction can be performed through manual review.

[0154] (3) Duplicate Data Detection and Correction: Duplicate items in text data are detected by calculating hash values ​​or comparing text content. For example, the hash value of each text is calculated, and the hash values ​​of different texts are compared to identify duplicate text and mark duplicate data. For duplicate text, duplicates can be automatically removed, and only one text is retained.

[0155] Step S103: Perform image quality detection on the preprocessed image data, and perform corresponding image correction processing based on the image quality detection results; the image quality detection includes at least one of integrity detection, noise detection, and blur detection.

[0156] The image quality inspection and corresponding image correction process is as follows: (1) Integrity detection and correction: The hash value of each image is calculated and compared with the pre-stored hash value to ensure that the image has not been damaged or tampered with and can be loaded and displayed completely and normally. If a hash value mismatch is detected, the image is marked as damaged or incomplete and can be repaired or re-acquired.

[0157] (2) Noise detection and correction: Check if there are too many noise points or interfering elements in the image. Specifically, the image sharpness detection algorithm can be used to evaluate the noise level of the image. The evaluation index can be selected as PSNR (Peak signal-to-noise ratio). After detecting background interference, image blur, and other noise effects and marking them, noise removal algorithms such as mean filtering are used to remove the noise.

[0158] (3) Blur detection and correction: The image sharpness is evaluated by calculating the gradient value of the Laplacian transform or analyzing the proportion of high-frequency components, and blurry images are identified and marked. For images with moderate blur, image enhancement algorithms (such as sharpening and contrast adjustment) are used to improve the image sharpness. If the blur is too high, the image is marked as a substandard image and is re-acquired.

[0159] The multimodal data processing method provided in this invention provides a high-quality, standardized, and reliable data foundation for downstream storage, retrieval, and analysis applications by preprocessing and quality checking text and image data.

[0160] The multimodal data processing apparatus provided in the embodiments of the present invention will be described below. The multimodal data processing apparatus described below and the multimodal data processing method described above can be referred to in correspondence with each other.

[0161] Figure 6 This is a schematic diagram of the structure of the multimodal data processing device provided by the present invention, as shown below. Figure 6 As shown, the device includes a structured content extraction module 610, a correlation calculation module 620, a cross-modal correlation establishment module 630, and a cross-modal correlation storage module 640; wherein: The structured content extraction module 610 is used to extract structured text content from text data and structured image content from image data. The correlation calculation module 620 is used to calculate the correlation between each text data and each image data based on at least one of the semantic matching degree, target matching degree and spatiotemporal proximity between the structured text content and the structured image content. The cross-modal association establishment module 630 is used to establish cross-modal association relationships between each text data and each image data based on the association degree. The cross-modal association storage module 640 is used to associate and store the cross-modal association relationship with the corresponding text data and image data respectively.

[0162] The multimodal data processing apparatus provided in this invention extracts structured text content from text data and structured image content from image data, transforming multi-source heterogeneous unstructured text and image data into structured content rich in semantic and content features, laying the foundation for deep cross-modal data association. Then, based on at least one of semantic matching degree, target matching degree, and spatiotemporal proximity between structured text content and structured image content, the association degree between each text data and each image data is calculated. Based on the association degree, cross-modal association relationships are established between each text data and each image data, enabling automatic and quantitative mining of deep cross-modal data association relationships from a multi-dimensional content level, significantly improving the accuracy and processing efficiency of the association. Finally, the cross-modal association relationships are associated and stored with the corresponding text and image data, making the association relationships directly indexable and queried structured data. This provides efficient and reliable data support for intelligent cross-modal retrieval and analysis in complex scenarios, meeting practical application needs.

[0163] According to a multimodal data processing apparatus provided by the present invention, the multimodal data processing apparatus further includes: An information storage module is used to store the structured text content in a first index of a distributed search engine, and to store the structured image content in a second index of a distributed search engine; The correlation calculation module 620 is specifically used for: Using the first index and the second index, cross-modal retrieval is performed based on at least one of the semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content; Based on the cross-modal retrieval results, the correlation between each text data and each image data is calculated.

[0164] According to a multimodal data processing apparatus provided by the present invention, the correlation calculation module 620 is further specifically used for: Based on the keywords in the structured text content and the visual features in the structured image content, the semantic matching degree is calculated; Calculate the target matching degree based on the named entities in the structured text content and the target objects in the structured image content; Based on the named entities and / or text meta-information in the structured text content, and the image meta-information in the structured image content, the spatiotemporal proximity is calculated; The semantic matching degree, the target matching degree, and the spatiotemporal proximity are weighted and summed to obtain the correlation degree between each text data and each image data.

[0165] According to the multimodal data processing apparatus provided by the present invention, the cross-modal association storage module 640 is specifically used for: The cross-modal association relationship is stored in the first cross-modal association field of the corresponding text data and in the second cross-modal association field of the corresponding image data; The first cross-modal association field and the second cross-modal association field are configured as nested object types.

[0166] According to a multimodal data processing apparatus provided by the present invention, the multimodal data processing apparatus further includes: A bloodline information acquisition module is used to acquire bloodline relationship information of the text data and / or the image data, wherein the bloodline relationship information includes at least one of data source, derivative relationship and version relationship; The bloodline information storage module is used to store the bloodline information into the bloodline field of the corresponding data.

[0167] According to a multimodal data processing apparatus provided by the present invention, the structured text content includes at least one of keywords, named entities, and category topics, and the multimodal data processing apparatus further includes: The first association establishment module is used to calculate the first similarity between each text data based on the keywords, and establish a first association relationship based on the first similarity; The second association establishment module is used to determine, based on the named entity, a first target text data including the same named entity, and to establish a second association relationship based on the first target text data; The third association establishment module is used to determine second target text data belonging to the same category topic based on the category topic, and to establish a third association relationship based on the second target text data; The text association storage module is used to store at least one of the first association, the second association, and the third association into the text association field of the corresponding text data.

[0168] According to a multimodal data processing apparatus provided by the present invention, the structured image content includes at least one of visual features and target objects, and the multimodal data processing apparatus further includes: The fourth association establishment module is used to calculate the second similarity between each image data based on the visual features, and establish a fourth association relationship based on the second similarity. The fifth association establishment module is used to determine target image data that includes the same target object based on the target object, and to establish a fifth association relationship based on the target image data; The image association storage module is used to store at least one of the fourth association relationship and the fifth association relationship into the image association field of the corresponding image data.

[0169] According to a multimodal data processing apparatus provided by the present invention, the multimodal data processing apparatus further includes: A data preprocessing module is used to preprocess the text data and the image data; The text quality detection module is used to perform text quality detection on the preprocessed text data and execute corresponding text correction processing based on the text quality detection results; the text quality detection includes at least one of accuracy detection, syntax consistency detection and duplicate data detection. The image quality detection module is used to perform image quality detection on the preprocessed image data and execute corresponding image correction processing based on the image quality detection results; the image quality detection includes at least one of integrity detection, noise detection and blur detection.

[0170] It should be noted that the multimodal data processing apparatus provided in this embodiment of the invention can implement all the method steps implemented in the above multimodal data processing method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0171] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute the multimodal data processing methods provided in the above embodiments.

[0172] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0173] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the multimodal data processing methods provided in the above embodiments.

[0174] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0175] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal data processing method, characterized in that, include: Extract structured text content from text data and structured image content from image data; Based on at least one of the semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content, calculate the correlation degree between each text data and each image data; Based on the aforementioned correlation, a cross-modal correlation relationship is established between each text data and each image data; The cross-modal association relationships are then associated and stored with the corresponding text data and image data, respectively.

2. The multimodal data processing method according to claim 1, characterized in that, Before calculating the correlation between each text data and each image data based on at least one of the semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content, the method further includes: The structured text content is stored in the first index of the distributed search engine, and the structured image content is stored in the second index of the distributed search engine; The calculation of the correlation between each text data and each image data based on at least one of the semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content includes: Using the first index and the second index, cross-modal retrieval is performed based on at least one of the semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content; Based on the cross-modal retrieval results, the correlation between each text data and each image data is calculated.

3. The multimodal data processing method according to claim 1, characterized in that, The calculation of the correlation between each text data and each image data based on at least one of the semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content includes: Based on the keywords in the structured text content and the visual features in the structured image content, the semantic matching degree is calculated; Calculate the target matching degree based on the named entities in the structured text content and the target objects in the structured image content; Based on the named entities and / or text meta-information in the structured text content, and the image meta-information in the structured image content, the spatiotemporal proximity is calculated; The semantic matching degree, the target matching degree, and the spatiotemporal proximity are weighted and summed to obtain the correlation degree between each text data and each image data.

4. The multimodal data processing method according to claim 1, characterized in that, The step of associating and storing the cross-modal association relationship with the corresponding text data and image data includes: The cross-modal association relationship is stored in the first cross-modal association field of the corresponding text data and in the second cross-modal association field of the corresponding image data; The first cross-modal association field and the second cross-modal association field are configured as nested object types.

5. The multimodal data processing method according to any one of claims 1 to 4, characterized in that, The multimodal data processing method further includes: Obtain the lineage information of the text data and / or the image data, wherein the lineage information includes at least one of data source, derivative relationship and version relationship; The blood relationship information is stored in the blood relationship field of the corresponding data.

6. The multimodal data processing method according to any one of claims 1 to 4, characterized in that, The structured text content includes at least one of keywords, named entities, and categorized topics, and the multimodal data processing method further includes: Based on the keywords, calculate the first similarity between each text data, and establish a first association relationship based on the first similarity; Based on the named entity, a first target text data including the same named entity is determined, and a second association relationship is established based on the first target text data; Based on the classification topic, determine the second target text data belonging to the same classification topic, and establish a third association based on the second text data; Store at least one of the first association, the second association, and the third association into the text association field of the corresponding text data.

7. The multimodal data processing method according to any one of claims 1 to 4, characterized in that, The structured image content includes at least one of visual features and target objects, and the multimodal data processing method further includes: Based on the visual features, a second similarity is calculated between each image data, and a fourth association relationship is established based on the second similarity; Based on the target object, target image data including the same target object are determined, and a fifth association relationship is established based on the target image data. Store at least one of the fourth and fifth association relationships into the image association field of the corresponding image data.

8. The multimodal data processing method according to any one of claims 1 to 4, characterized in that, Before extracting the structured text content from the text data and the structured image content from the image data, the process also includes: The text data and the image data are preprocessed; The preprocessed text data is subjected to text quality detection, and corresponding text correction processing is performed based on the text quality detection results; the text quality detection includes at least one of accuracy detection, syntax consistency detection and duplicate data detection. The preprocessed image data is subjected to image quality detection, and corresponding image correction processing is performed based on the image quality detection results; the image quality detection includes at least one of integrity detection, noise detection, and blur detection.

9. A multimodal data processing device, characterized in that, include: The structured content extraction module is used to extract structured text content from text data and structured image content from image data. The cross-modal association establishment module is used to establish cross-modal association relationships between each text data and each image data based on the structured text content and the structured image content; The correlation calculation module is used to calculate the correlation between each text data and each image data based on at least one of the semantic matching degree, target matching degree, and spatiotemporal proximity between the structured text content and the structured image content. A cross-modal association establishment module is used to establish cross-modal association relationships between each text data and each image data based on the association degree. The cross-modal association storage module is used to associate and store the cross-modal association relationship with the corresponding text data and image data, respectively.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal data processing method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multimodal data processing method as described in any one of claims 1 to 8.

Images