Data processing method and system based on integrated multi-modal model

Abstract

The invention provides a data processing method and system based on an integrated multi-modal model, and relates to the technical field of multi-modal data processing, and the method comprises the following steps: S1, obtaining a multi-modal file uploaded by a user through an interactive interface; s2, analyzing the multi-modal file to obtain corresponding user information and file information; s3, identifying a file type of the file information through the multi-modal model, selecting different processing methods according to the file type, and processing to obtain metadata and a standardized vector corresponding to the metadata; s4, a preset file template in the structured database is obtained, matching is conducted according to the preset file template, the metadata and the standardized vectors corresponding to the metadata, and the metadata and the standardized vectors corresponding to the metadata are jointly stored in the structured database in a structured mode; and S5, obtaining an initial user file directory graph displayed on the interactive interface through the user information, and updating the initial user file directory graph according to the successfully matched preset file template.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and in particular to a data processing method and system based on an integrated multimodal model. Background Art

[0002] With the development of the internet and social media, people are increasingly accustomed to recording information about different events in a variety of ways, including images, tables, and text. From a company management perspective, manually recording file information in multiple ways can lead to information confusion. Therefore, company management systems often use multimodal models to efficiently and intelligently aggregate and display different data sources and project files in different formats. In such management systems, different user permissions are generally pre-set based on the responsibilities of different employees, preventing employees from adding or deleting project files beyond their authorized permissions. Although this method ensures data confidentiality to a certain extent, it is extremely inconvenient because backend personnel are required to modify pre-set user permissions when employee responsibilities change or when projects overlap between employees with different responsibilities. Summary of the Invention

[0003] The purpose of the present invention is to provide a data processing method and system based on an integrated multimodal model. When a multimodal model is used to process and store project files uploaded by users, the user's permissions can be automatically corrected according to the structured form of the data in the data repository, which is more convenient.

[0004] In order to solve the above technical problems, the present invention adopts the following solutions:

[0005] A data processing method based on an integrated multimodal model comprises the following steps:

[0006] S1. Obtain a multimodal file uploaded by a user through an interactive interface;

[0007] S2. Parse the multimodal file to obtain corresponding user information and file information;

[0008] S3. Identify the file type of the file information using a multimodal model, select different processing methods based on the file type, and obtain metadata and its corresponding normalized vector through processing;

[0009] S4. Constructing a structured database, obtaining a file template preset in the structured database, matching the preset file template with the metadata and its corresponding normalized vector, and jointly storing the metadata and its corresponding normalized vector in a structured form in the structured database;

[0010] S5. Obtain an initial user file directory map displayed on the interactive interface through the user information, and update the initial user file directory map according to the preset file template that is successfully matched.

[0011] Furthermore, the multimodal file is file information with user information, and the file types of the file information include image files, table files, and text files. The user information includes a user account. After the user enters the user account on the interactive interface to log in, the file information is uploaded through the interactive interface.

[0012] Furthermore, in S3, the process of selecting different processing methods based on file type is as follows:

[0013] When the file type is an image file, the image in the image file is identified through the image recognition model, its image features are extracted, and image metadata and its corresponding normalized vector are generated based on the image features;

[0014] When the file type is a table file, the table in the table file is identified through the table recognition model, its table content features are extracted and the table metadata and its corresponding normalized vector are generated;

[0015] When the file type is a text file, the text in the text file is subjected to keyword recognition through the text recognition model, its keyword features are extracted, and text metadata and its corresponding normalized vector are generated.

[0016] Furthermore, in S4, the process of building a structured database is as follows:

[0017] Obtain a file template preset for the project file, perform a first classification on the preset file template according to the type of the project file, use the type of the project file as the first node, and store the relevant preset file template under the corresponding first node; and perform a second classification on the preset file template under the first node according to the name of the project file, use the name of the project file as the second node, and store the relevant file template under the corresponding second node.

[0018] Furthermore, in S4, the process of matching the metadata and its corresponding normalized vector according to the preset file template is as follows: first, the metadata and its corresponding normalized vector are matched with the type of the project file. After matching the corresponding project file type, the project file type is matched downward to the corresponding preset file template, so that the metadata and its corresponding normalized vector are stored in a structured database.

[0019] Furthermore, the user information includes the initial authority for viewing the structured database. In S5, the process of obtaining the initial user file directory map displayed on the interactive interface through the user information is as follows:

[0020] According to the initial permissions, the relevant preset file templates are located in the structured database, the nodes are filtered according to the node position of the relevant preset file templates and their corresponding upper and lower structures, and the filtered nodes are labeled to form the initial user file directory map and display it on the interactive interface.

[0021] Furthermore, the nodes are screened according to the node positions of the relevant preset file templates and their corresponding upper and lower structures, and the process of labeling the screened nodes is as follows:

[0022] From the node position where the relevant preset file template is located, find the corresponding second node and first node in sequence, generate labels according to the type of project files and the name of the project files corresponding to the first node and the second node, and form the initial user file directory diagram with the connection relationship between the corresponding labels.

[0023] Furthermore, in S5, the process of updating the initial user file directory map according to the successfully matched preset file template is as follows: obtaining the currently successfully matched preset file template, locating the node position of the currently successfully matched preset file template in the structured database, finding the corresponding second node and first node from the node position upward, and obtaining the current labels of the corresponding second node and first node;

[0024] Determine whether there is a current tag in the initial user file directory map. If so, generate a project key field using the currently matched metadata and its corresponding normalized vector. Use the project key field as the third node, and connect the third node to the corresponding second node to update the initial user file directory map. If not, update the initial user file directory map using the connection relationship between the current tag and the tag.

[0025] A data processing system based on an integrated multimodal model, applying the data processing method based on an integrated multimodal model, comprises:

[0026] Data acquisition module: obtains multimodal files uploaded by users through the interactive interface;

[0027] Data parsing module: parses multimodal files to obtain corresponding user information and file information;

[0028] Multimodal model processing module: identifies the file type of file information through a multimodal model, selects different processing methods based on the file type, and obtains metadata and its corresponding standardized vector after processing;

[0029] Structured database storage module: Builds a structured database, obtains a file template preset in the structured database, matches the preset file template with metadata and its corresponding normalized vector, and stores the metadata and its corresponding normalized vector in a structured form in the structured database;

[0030] User file directory map update module: obtains the initial user file directory map displayed on the interactive interface through user information, and updates the initial user file directory map according to the preset file template that is successfully matched.

[0031] Beneficial effects of the present invention:

[0032] The present invention provides a data processing method and system based on an integrated multimodal model. The method mainly adds a user file directory map update module to the system that uses a multimodal model to manage project files in the existing technology. The user file directory map update module can obtain data obtained after data processing of project files uploaded by users through the multimodal model from the data repository. Based on the structured form of the data in the structured database, the user's permissions can be automatically corrected without the need for correction by back-end personnel, which is more convenient. In addition, a user file directory map is generated on the interactive interface of the system. The user file directory map is displayed in a structured form and project key fields. By updating the user file directory map, the correction process of user permissions can be dynamically displayed and provided to users for viewing, so that the system can increase its user interactivity while keeping the project files confidential. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of a flow chart of a data processing method based on an integrated multimodal model in Example 1 of the present invention;

[0034] Figure 2 Schematic diagram of data processing of the multimodal model in Example 1 of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0036] Unless otherwise specifically stated, the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention.

[0037] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0038] Additionally, descriptions of well-known structures, functions, and configurations may be omitted for clarity and conciseness. Those skilled in the art will recognize that various changes and modifications can be made to the examples described herein without departing from the spirit and scope of the present disclosure.

[0039] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods and equipment should be considered part of the authorization specification.

[0040] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0041] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments:

[0042] Example 1

[0043] Because the company's management system often uses a multimodal model to efficiently and intelligently aggregate and display different data sources and project files in different forms, different user permissions are generally pre-set in this management system based on the responsibilities of different employees, so that employees cannot add or delete project files outside of their permissions. Although this method ensures data confidentiality to a certain extent, when employee responsibilities change or when projects overlap between employees with different responsibilities, back-end personnel are required to change the pre-set user permissions, which is extremely inconvenient.

[0044] Therefore, in this embodiment, a data processing method based on an integrated multimodal model is provided. In order to solve the above problems, the data processing method mainly realizes automatic correction of user permissions according to the structured form of data in the data repository during the process of using a multimodal model to process and store project files uploaded by users. This method does not require correction by back-end personnel and is more convenient.

[0045] like Figure 1 As shown, a data processing method based on an integrated multimodal model includes the following steps:

[0046] S1. Obtain a multimodal file uploaded by a user through an interactive interface;

[0047] S2. Parse the multimodal file to obtain corresponding user information and file information;

[0048] S3. Identify the file type of the file information using a multimodal model, select different processing methods based on the file type, and obtain metadata and its corresponding normalized vector through processing;

[0049] S4. Constructing a structured database, obtaining a file template preset in the structured database, matching the preset file template with the metadata and its corresponding normalized vector, and jointly storing the metadata and its corresponding normalized vector in a structured form in the structured database;

[0050] S5. Obtain an initial user file directory map displayed on the interactive interface through the user information, and update the initial user file directory map according to the preset file template that is successfully matched.

[0051] In one embodiment, the multimodal file is file information with user information, and the file type of the file information includes image files, table files, text files, etc.

[0052] When the file type is an image file, the image in the image file is identified through the image recognition model, its image features are extracted, and image metadata and its corresponding normalized vector are generated based on the image features;

[0053] When the file type is a table file, the table in the table file is identified through the table recognition model, its table content features are extracted and the table metadata and its corresponding normalized vector are generated;

[0054] When the file type is a text file, the text in the text file is subjected to keyword recognition through the text recognition model, its keyword features are extracted, and text metadata and its corresponding normalized vector are generated.

[0055] In one embodiment, the user information includes a user account. After the user enters the user account on the interactive interface to log in, the user uploads the file information through the interactive interface.

[0056] In one embodiment, if Figure 2As shown, after the user uploads a file, it can automatically identify whether the file is a multimodal file. If the file is a multimodal file, it is parsed by the format parser to extract the metadata corresponding to the file. Specifically, different extraction methods can be selected according to different file types, including table extraction, image extraction, formula extraction and other modal file extraction, so that the file is processed by the multimodal model to obtain a model recognition result. At the same time, the text content is extracted by the format parser, and the result of the text content extraction is merged with the model recognition result to obtain a merged content. The merged content is then segmented by the file slicing adapter, and the segmented result is embedded in the model for processing. The vector obtained after processing can be stored by the vector database adapter and stored in the vector database, so that the user can perform a mixed search through the vector and keyword. At this time, the keyword is the metadata mentioned in this embodiment.

[0057] In one embodiment, the integrated extracted content is stored, and the metadata and the extracted content of its corresponding standardized vector are jointly stored in a structured form in a structured database. The data can be stored in a structured manner through the structured database. The purpose of structured storage is to automatically correct the user's permissions through the structured form of the data in the structured database.

[0058] In one embodiment, in S4, the process of constructing a structured database is: obtaining a file template preset for a project file, performing a first classification on the preset file template according to the type of the project file, using the type of the project file as the first node, and storing the relevant preset file template under the corresponding first node; and performing a second classification on the preset file template under the first node according to the name of the project file, using the name of the project file as the second node, and storing the relevant file template under the corresponding second node.

[0059] In one embodiment, in S4, the process of matching the metadata and its corresponding normalized vector according to the preset file template is as follows: first, the metadata and its corresponding normalized vector are matched with the type of the project file. After matching the corresponding project file type, the project file type is matched downward to the corresponding preset file template, so that the metadata and its corresponding normalized vector are stored in a structured database.

[0060] In one embodiment, the user information includes initial permissions for viewing a structured database. In S5, the process of obtaining an initial user file directory diagram displayed on an interactive interface through the user information is as follows: locating relevant preset file templates in the structured database according to the initial permissions, filtering nodes according to the node positions of the relevant preset file templates and their corresponding upper and lower structures, labeling the filtered nodes, forming an initial user file directory diagram and displaying it on the interactive interface.

[0061] In one embodiment, nodes are screened according to the node positions of relevant preset file templates and their corresponding upper and lower structures, and the process of labeling the screened nodes is as follows: from the node positions of the relevant preset file templates upward, the corresponding second node and first node are found in sequence, and labels are generated according to the types of project files corresponding to the first node and the second node and the names of the project files, and the initial user file directory diagram is formed with the connection relationship between the corresponding labels and the labels.

[0062] In one embodiment, in S5, the process of updating the initial user file directory map according to the successfully matched preset file template is as follows:

[0063] Obtain the currently matched preset file template, locate the node position of the currently matched preset file template in the structured database, find the corresponding second node and first node from the node position upward, and obtain the current labels of the corresponding second node and first node;

[0064] Determine whether there is a current tag in the initial user file directory map. If so, generate a project key field using the currently matched metadata and its corresponding normalized vector. Use the project key field as the third node, and connect the third node to the corresponding second node to update the initial user file directory map. If not, update the initial user file directory map using the connection relationship between the current tag and the tag.

[0065] Example 2

[0066] A data processing system based on an integrated multimodal model, applying the data processing method based on an integrated multimodal model, comprises:

[0067] Data acquisition module: obtains multimodal files uploaded by users through the interactive interface;

[0068] Data parsing module: parses multimodal files to obtain corresponding user information and file information;

[0069] Multimodal model processing module: identifies the file type of file information through a multimodal model, selects different processing methods based on the file type, and obtains metadata and its corresponding standardized vector after processing;

[0070] Structured database storage module: Builds a structured database, obtains a file template preset in the structured database, matches the preset file template with metadata and its corresponding normalized vector, and stores the metadata and its corresponding normalized vector in a structured form in the structured database;

[0071] User file directory map update module: obtains the initial user file directory map displayed on the interactive interface through user information, and updates the initial user file directory map according to the preset file template that is successfully matched.

[0072] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Based on the technical essence of the present invention and within the spirit and principles of the present invention, any simple modification, equivalent replacement and improvement of the above embodiment shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A data processing method based on an integrated multimodal model, characterized in that: The data processing method comprises the following steps: S1. Obtain a multimodal file uploaded by a user through an interactive interface; S2. Parse the multimodal file to obtain corresponding user information and file information; S3. Identify the file type of the file information using a multimodal model, select different processing methods based on the file type, and obtain metadata and its corresponding normalized vector through processing; S4. Constructing a structured database, obtaining a file template preset in the structured database, matching the preset file template with the metadata and its corresponding normalized vector, and jointly storing the metadata and its corresponding normalized vector in a structured form in the structured database; S5. Obtain an initial user file directory map displayed on the interactive interface through the user information, and update the initial user file directory map according to the preset file template that is successfully matched.

2. The data processing method based on the integrated multimodal model according to claim 1, characterized in that: The multimodal file is file information with user information. The file types of the file information include image files, table files, and text files. The user information includes a user account. After the user enters the user account on the interactive interface to log in, the file information is uploaded through the interactive interface.

3. The data processing method based on the integrated multimodal model according to claim 2, characterized in that: In S3, the process of selecting different processing methods based on file type is as follows: When the file type is an image file, the image in the image file is identified through the image recognition model, its image features are extracted, and image metadata and its corresponding normalized vector are generated based on the image features; When the file type is a table file, the table in the table file is identified through the table recognition model, its table content features are extracted and the table metadata and its corresponding normalized vector are generated; When the file type is a text file, the text in the text file is subjected to keyword recognition through the text recognition model, its keyword features are extracted, and text metadata and its corresponding normalized vector are generated.

4. The data processing method based on an integrated multimodal model according to claim 1, characterized in that: In S4, the process of building a structured database is: Obtaining a preset file template for a project file, first classifying the preset file template according to the type of the project file, using the type of the project file as a first node, and storing the relevant preset file template under the corresponding first node; The file templates preset under the first node are subjected to a second classification based on the name of the project file, the name of the project file is used as the second node, and the related file templates are stored under the corresponding second node.

5. The data processing method based on the integrated multimodal model according to claim 4, characterized in that: In S4, the process of matching the metadata and its corresponding normalized vector according to the preset file template is as follows: first, the metadata and its corresponding normalized vector are matched with the type of the project file. After matching the corresponding project file type, the project file type is matched downward to the corresponding preset file template, so that the metadata and its corresponding normalized vector are stored in a structured database.

6. The data processing method based on the integrated multimodal model according to claim 4, characterized in that: The user information includes the initial authority for viewing the structured database. In S5, the process of obtaining the initial user file directory map displayed on the interactive interface through the user information is as follows: According to the initial permissions, the relevant preset file templates are located in the structured database, the nodes are filtered according to the node position of the relevant preset file templates and their corresponding upper and lower structures, and the filtered nodes are labeled to form the initial user file directory map and display it on the interactive interface.

7. The data processing method based on the integrated multimodal model according to claim 6, characterized in that: The nodes are filtered according to the node positions of the relevant preset file templates and their corresponding upper and lower structures, and the process of labeling the filtered nodes is as follows: From the node position where the relevant preset file template is located, find the corresponding second node and first node in sequence, generate labels according to the type of project files and the name of the project files corresponding to the first node and the second node, and form the initial user file directory diagram with the connection relationship between the corresponding labels.

8. The data processing method based on the integrated multimodal model according to claim 7, characterized in that: In S5, the process of updating the initial user file directory map according to the successfully matched preset file template is as follows: obtaining the currently successfully matched preset file template, locating the node position of the currently successfully matched preset file template in the structured database, finding the corresponding second node and first node from the node position upward, and obtaining the current labels of the corresponding second node and first node; Determine whether there is a current tag in the initial user file directory map. If so, generate a project key field using the currently matched metadata and its corresponding normalized vector. Use the project key field as the third node, and connect the third node to the corresponding second node to update the initial user file directory map. If not, update the initial user file directory map using the connection relationship between the current tag and the tag.

9. A data processing system based on an integrated multimodal model, characterized in that: Applying a data processing method based on an integrated multimodal model as described in any one of claims 1 to 8, comprising: Data acquisition module: obtains multimodal files uploaded by users through the interactive interface; Data parsing module: parses multimodal files to obtain corresponding user information and file information; Multimodal model processing module: identifies the file type of file information through a multimodal model, selects different processing methods based on the file type, and obtains metadata and its corresponding standardized vector after processing; Structured database storage module: Builds a structured database, obtains a file template preset in the structured database, matches the preset file template with metadata and its corresponding normalized vector, and stores the metadata and its corresponding normalized vector in a structured form in the structured database; User file directory map update module: obtains the initial user file directory map displayed on the interactive interface through user information, and updates the initial user file directory map according to the preset file template that is successfully matched.

Images