A data processing method and system based on an integrated multi-modal model

Abstract

The application provides a data processing method and system based on an integrated multi-modal model, and relates to the technical field of multi-modal data processing, and comprises the following steps: S1, obtaining a multi-modal file uploaded by a user through an interactive interface; S2, analyzing the multi-modal file to obtain corresponding user information and file information; S3, identifying the file type of the file information through a multi-modal model, selecting different processing methods according to the file type, and obtaining metadata and a corresponding standardized vector after processing; S4, obtaining a preset file template in a structured database, and matching the preset file template with the metadata and the corresponding standardized vector, so that the metadata and the corresponding standardized vector are jointly stored in the structured database in a structured form; and S5, obtaining an initial user file directory graph displayed on the interactive interface through the user information, and updating the initial user file directory graph according to the preset file template matched successfully.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal data processing, in particular to a data processing method and system based on an integrated multi-modal model. BACKGROUND

[0002] With the development of the Internet and social media, people are increasingly accustomed to recording information of different events through various ways, including pictures, tables, texts, etc. From the perspective of company management, the multi-modal model is often used in the management system of the company to efficiently and intelligently aggregate and display project files of different data sources and different forms. Different user permissions are generally set in advance in the management system according to the responsibilities of different employees, so that the employees cannot add or delete project files beyond the permissions. Although this method can ensure the security of data to a certain extent, it is extremely inconvenient when the responsibilities of employees change or when project cross occurs between employees with different responsibilities, and the background personnel need to change the pre-set user permissions. SUMMARY

[0003] The purpose of the present application is to provide a data processing method and system based on an integrated multi-modal model, which can automatically correct the user's permissions according to the structured form of data in the data storage library when the multi-modal model is used to process and store project files uploaded by users, and is more convenient.

[0004] To solve the above technical problems, the present application adopts the following scheme:

[0005] A data processing method based on an integrated multi-modal model, comprising the following steps:

[0006] S1, obtaining multi-modal files uploaded by a user through an interactive interface;

[0007] S2, analyzing the multi-modal files to obtain corresponding user information and file information;

[0008] S3, identifying the file type of the file information through a multi-modal model, selecting different processing methods according to the file type, and obtaining metadata and its corresponding standardized vector after processing;

[0009] S4, constructing a structured database, obtaining a pre-set file template in the structured database, and matching the pre-set file template with the metadata and its corresponding standardized vector, so that the metadata and its corresponding standardized vector are jointly stored in the structured database in a structured form;

[0010] S5, obtaining an initial user file directory graph displayed on the interactive interface through the user information, and updating the initial user file directory graph according to the matched preset file template.

[0011] Further, the multi-modal file is a file information with user information, the file type of the file information includes a picture file, a table file and a text file, and the user information includes a user account.

[0012] Further, in S3, the process of selecting different processing methods according to the file type is:

[0013] When the file type is a picture file, an image recognition model is used to recognize the picture in the picture file, extract the image features, generate image metadata and the corresponding standardized vector according to the image features;

[0014] When the file type is a table file, a table recognition model is used to recognize the content of the table in the table file, extract the table content features and generate table metadata and the corresponding standardized vector;

[0015] When the file type is a text file, a text recognition model is used to recognize the keywords in the text file, extract the keyword features and generate text metadata and the corresponding standardized vector.

[0016] Further, in S4, the process of constructing the structured database is:

[0017] A preset file template for the project file is obtained, the preset file template is first classified according to the type of the project file, the type of the project file is taken as a first node, and the related preset file template is stored under the corresponding first node; and the preset file template under the first node is second classified according to the name of the project file, the name of the project file is taken as a second node, and the related file template is stored under the corresponding second node.

[0018] Further, in S4, the process of matching the preset file template with the metadata and the corresponding standardized vector is: first, match the metadata and the corresponding standardized vector with the type of the project file, then match the corresponding preset file template from the type of the project file, and store the metadata and the corresponding standardized vector in the structured database.

[0019] Further, the user information includes an initial permission for viewing the structured database, and in S5, the process of obtaining the initial user file directory graph displayed on the interactive interface through the user information is:

[0020] According to the initial authority, the related preset file template in the structured database is located, the nodes are screened according to the node position where the related preset file template is located and the corresponding upper and lower structures thereof, the screened nodes are labeled, the initial user file directory graph is formed and displayed on the interactive interface.

[0021] Further, the process of screening the nodes according to the node position where the related preset file template is located and the corresponding upper and lower structures thereof and labeling the screened nodes is as follows:

[0022] The corresponding second node and first node are sequentially found from the node position where the related preset file template is located, the labels are generated according to the type and name of the project file corresponding to the first node and second node, and the initial user file directory graph is formed with the corresponding labels and the connection relationship between the labels.

[0023] Further, in S5, the process of updating the initial user file directory graph according to the matched preset file template is as follows: obtaining the currently matched preset file template, locating the node position where the currently matched preset file template is located in the structured database, sequentially finding the corresponding second node and first node from the node position, and obtaining the current labels of the corresponding second node and first node;

[0024] It is judged whether the initial user file directory graph has the current label, if yes, the project key field is generated with the currently matched metadata and the corresponding standardized vector, the project key field is taken as the third node, the third node is connected with the corresponding second node, and the update of the initial user file directory graph is realized; if not, the initial user file directory graph is updated with the connection relationship between the current label and the label.

[0025] A data processing system based on an integrated multi-modal model, applying the data processing method based on the integrated multi-modal model, comprising:

[0026] A data acquisition module: obtaining multi-modal files uploaded by a user through an interactive interface;

[0027] A data analysis module: analyzing the multi-modal files to obtain corresponding user information and file information;

[0028] A multi-modal model processing module: identifying the file type of the file information through a multi-modal model, selecting different processing methods according to the file type, and obtaining metadata and the corresponding standardized vector through processing;

[0029] The structured database storage module: construct a structured database, obtain a preset file template in the structured database, match the preset file template with the metadata and the corresponding standardized vector, and store the metadata and the corresponding standardized vector in the structured database in a structured form;

[0030] The user file directory graph updating module: obtain an initial user file directory graph displayed on the interactive interface through user information, and update the initial user file directory graph according to the matched preset file template.

[0031] The beneficial effects of the present application are as follows:

[0032] The present application provides a data processing method and system based on an integrated multi-modal model, which mainly adds a user file directory graph updating module to the system for data management of project files by using a multi-modal model. The user file directory graph updating module can obtain data obtained by processing user-uploaded project files by using the multi-modal model from a data storage library. The user's authority can be automatically corrected based on the structured form of the data in the structured database, without the need for correction by background personnel, which is more convenient. In addition, a user file directory graph is generated on the interactive interface of the system. The user file directory graph is displayed through a structured form and a project key field. The correction process of the user's authority can be dynamically displayed by updating the user file directory graph, which provides a user for viewing, so that the system can increase the user interactivity while keeping the project files confidential. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The flowchart of the data processing method based on the integrated multi-modal model in the embodiment 1 of the present application is shown.

[0034] Figure 2 The data processing schematic diagram of the multi-modal model in the embodiment 1 of the present application is shown. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. The description of the at least one exemplary embodiment is actually only illustrative, but not as any limitation on the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0036] The relative arrangement of parts and steps, numerical expressions, and values set forth in these embodiments are not intended to limit the scope of the present disclosure unless otherwise specifically stated.

[0037] It should be understood, however, that the dimensions of the various parts illustrated in the accompanying drawings are chosen for purposes of convenience and clarity of presentation, and are not intended to limit the scope of the disclosure.

[0038] In addition, descriptions of known structures, functions, and configurations can be omitted for clarity and conciseness.

[0039] The techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered part of the enabling description under appropriate circumstances.

[0040] In all of the examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Thus, other examples of the example embodiments can have different values.

[0041] The present disclosure will be described in detail below with reference to the attached drawings and embodiments:

[0042] Embodiment 1

[0043] Since the management system of a company often uses a multi-modal model to efficiently and intelligently aggregate and display project files of different data sources and different forms, different user permissions are generally set in advance in such a management system according to the responsibilities of different employees, so that employees cannot add or delete project files beyond their permissions. Although this approach can secure data to some extent, it is extremely inconvenient when the responsibilities of employees change or when project cross occurs between employees with different responsibilities, as the background personnel need to change the pre-set user permissions.

[0044] Therefore, in the present embodiment, a data processing method based on an integrated multi-modal model is provided to solve the above problems. The data processing method mainly realizes automatic correction of user permissions according to the structured form of data in the data storage library during the process of using a multi-modal model to process and store project files uploaded by users, without the need for correction by background personnel, which is more convenient.

[0045] As shown in Figure 1 A data processing method based on an integrated multi-modal model includes the following steps:

[0046] S1, obtaining a multi-modal file uploaded by a user through an interactive interface;

[0047] S2, the multi-modal file is parsed to obtain corresponding user information and file information;

[0048] S3, the file type of the file information is identified by the multi-modal model, different processing methods are selected according to the file type, and metadata and its corresponding standardized vector are obtained after processing;

[0049] S4, a structured database is constructed, a preset file template in the structured database is obtained, and the preset file template is matched with the metadata and its corresponding standardized vector, so that the metadata and its corresponding standardized vector are stored in the structured database in a structured form;

[0050] S5, the initial user file directory graph displayed on the interactive interface is obtained by the user information, and the initial user file directory graph is updated according to the preset file template matched successfully.

[0051] In an embodiment, the multi-modal file is a file information with user information, and the file type of the file information includes picture files, table files, text files and the like.

[0052] When the file type is a picture file, the picture in the picture file is identified by an image recognition model, the image features are extracted, and the image metadata and its corresponding standardized vector are generated according to the image features;

[0053] When the file type is a table file, the table content in the table file is recognized by a table recognition model, the table content features are extracted, and the table metadata and its corresponding standardized vector are generated;

[0054] When the file type is a text file, the text in the text file is recognized by a text recognition model, the keyword features are extracted, and the text metadata and its corresponding standardized vector are generated.

[0055] In an embodiment, the user information includes a user account, and after the user inputs the user account to realize account login on the interactive interface, the file information is uploaded through the interactive interface.

[0056] In an embodiment, as Figure 2As shown, after the user uploads the file, it can be automatically identified whether the file is a multi-modal file. If the file is a multi-modal file, it is parsed by the format parser to extract the metadata corresponding to the file. Specifically, different extraction methods can be selected according to different file types, including table extraction, picture extraction, formula extraction, and other modal file extraction. After the file is processed by the multi-modal model, the model recognition result is obtained. At the same time, the text content is extracted by the format parser, and the extracted text content and the model recognition result are combined to obtain the combined content. Then, the combined content is divided by the file slicing adapter, and the divided result is embedded in the model for processing. The vector database adapter can store the processed vector in the vector database, so that the user can perform mixed retrieval through the vector and the keyword. At this time, the keyword is the metadata mentioned in this embodiment.

[0057] In an embodiment, the extracted content is stored, and the metadata and the extracted content of the corresponding standardized vector are stored in a structured form in a structured database. The structured database can store data in a structured manner. The purpose of structured storage is to automatically correct the user's permissions through the structured form of data in the structured database.

[0058] In an embodiment, in S4, the process of constructing the structured database is: obtaining a preset file template for the project file, first classifying the preset file template by the type of the project file, taking the type of the project file as a first node, and storing the related preset file template under the corresponding first node; and second classifying the preset file template under the first node by the name of the project file, taking the name of the project file as a second node, and storing the related file template under the corresponding second node.

[0059] In an embodiment, in S4, the process of matching the preset file template with the metadata and the corresponding standardized vector is: first matching the metadata and the corresponding standardized vector with the type of the project file, then matching the corresponding preset file template from the type of the project file, and storing the metadata and the corresponding standardized vector in the structured database.

[0060] In an embodiment, the user information includes initial permissions for viewing the structured database, and in S5, the process of obtaining the initial user file directory graph displayed on the interactive interface through the user information is: locating the relevant preset file template in the structured database according to the initial permissions, screening the nodes according to the node positions of the relevant preset file template and the corresponding hierarchical structure thereof, performing label processing on the screened nodes, forming the initial user file directory graph, and displaying the initial user file directory graph on the interactive interface.

[0061] In an embodiment, the process of screening the nodes according to the node positions of the relevant preset file template and the corresponding hierarchical structure thereof, and performing label processing on the screened nodes is: sequentially finding the corresponding second node and first node from the node position of the relevant preset file template upwards, generating labels according to the types and names of the project files corresponding to the first node and second node, and forming the initial user file directory graph with the corresponding labels and the connection relationship between the labels.

[0062] In an embodiment, in S5, the process of updating the initial user file directory graph according to the preset file template that matches successfully is:

[0063] obtaining the preset file template that matches successfully at present, locating the node position of the preset file template that matches successfully at present in the structured database, sequentially finding the corresponding second node and first node from the node position upwards, and obtaining the current labels of the corresponding second node and first node;

[0064] determining whether the initial user file directory graph has the current label, if yes, generating a project key field with the current metadata that matches successfully and the corresponding standardized vector, taking the project key field as a third node, connecting the third node with the corresponding second node, and updating the initial user file directory graph; and if no, updating the initial user file directory graph with the connection relationship between the current label and the label.

[0065] Embodiment 2

[0066] A data processing system based on an integrated multi-modal model, which applies the data processing method based on the integrated multi-modal model, comprising:

[0067] a data acquisition module: obtaining multi-modal files uploaded by a user through an interactive interface;

[0068] a data analysis module: analyzing the multi-modal files to obtain corresponding user information and file information;

[0069] The multi-modal model processing module: identifying the file type of the file information through the multi-modal model, selecting different processing methods according to the file type, and obtaining the metadata and the corresponding standardized vector after processing;

[0070] The structured database storage module: constructing a structured database, obtaining a preset file template in the structured database, and matching the preset file template with the metadata and the corresponding standardized vector, so that the metadata and the corresponding standardized vector are jointly stored in the structured database in a structured form;

[0071] The user file directory graph updating module: obtaining the initial user file directory graph displayed on the interactive interface through the user information, and updating the initial user file directory graph according to the matched preset file template.

[0072] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. According to the technical essence of the present application, any simple modification, equivalent replacement and improvement of the above embodiment within the spirit and principles of the present application are still within the protection scope of the technical solution of the present application.

Claims

1. A data processing method based on an integrated multimodal model, characterized in that, The data processing method includes the following steps: S1. Obtain the multimodal file uploaded by the user through the interactive interface; S2. Parse the multimodal file to obtain the corresponding user information and file information; S3. Identify the file type of file information through a multimodal model, select different processing methods according to the file type, and obtain metadata and its corresponding standardized vector after processing; S4. Construct a structured database, obtain the preset file template in the structured database, match the preset file template with the metadata and its corresponding standardized vector, and store the metadata and its corresponding standardized vector in a structured form in the structured database. In S4, the process of building a structured database is as follows: obtain the preset file templates for project files, classify the preset file templates by the type of project file as the first category, use the type of project file as the first node, and store the relevant preset file templates under the corresponding first node; and classify the preset file templates under the first node by the name of project file as the second category, use the name of project file as the second node, and store the relevant file templates under the corresponding second node. In S4, the process of matching the preset file template with the metadata and its corresponding normalized vector is as follows: first, the metadata and its corresponding normalized vector are matched with the type of the project file. When the corresponding type of the project file is matched, the matching is then performed from the type of the project file down to the corresponding preset file template, so that the metadata and its corresponding normalized vector are stored in the structured database. S5. Obtain the initial user file directory diagram displayed on the interactive interface through user information, and update the initial user file directory diagram according to the preset file template that has been successfully matched.

2. The data processing method based on an integrated multimodal model according to claim 1, characterized in that, The multimodal file is a file containing user information. The file types of the file information include image files, table files, and text files. The user information includes a user account. After logging in by entering the user account on the interactive interface, the user can upload the file information through the interactive interface.

3. The data processing method based on an integrated multimodal model according to claim 2, characterized in that, In S3, the process of selecting different processing methods based on file type is as follows: When the file type is an image file, the image recognition model is used to identify the images in the image file, extract their image features, and generate image metadata and its corresponding standardized vector based on the image features. When the file type is a table file, the table recognition model is used to identify the content of the tables in the table file, extract the table content features, and generate table metadata and its corresponding standardized vector. When the file type is a text file, the text recognition model is used to identify keywords in the text file, extract its keyword features, and generate text metadata and its corresponding standardized vector.

4. The data processing method based on an integrated multimodal model according to claim 1, characterized in that, The user information includes initial permissions for viewing the structured database. In S5, the process of obtaining the initial user file directory diagram displayed on the interactive interface based on the user information is as follows: Based on the initial permissions, the system locates the relevant preset file templates in the structured database, filters the nodes according to the node position of the preset file templates and their corresponding superordinate structure, tags the filtered nodes, forms the initial user file directory diagram, and displays it on the interactive interface.

5. The data processing method based on an integrated multimodal model according to claim 4, characterized in that, The process of filtering nodes based on their positions within the pre-defined file templates and their corresponding superordinate and superordinate structures, and then tagging the filtered nodes, is as follows: Starting from the node position of the relevant preset file template, find the corresponding second node and first node in sequence. Generate tags according to the type and name of the project file corresponding to the first node and the second node, and form an initial user file directory diagram based on the corresponding tags and the connection relationship between the tags.

6. The data processing method based on an integrated multimodal model according to claim 5, characterized in that, In S5, the process of updating the initial user file directory graph based on the successfully matched preset file template is as follows: obtain the currently successfully matched preset file template, locate the node position of the currently successfully matched preset file template in the structured database, find the corresponding second node and first node from the node position upwards in sequence, and obtain the current labels of the corresponding second node and first node. If the current tag exists in the initial user file directory graph, generate a project key field using the currently matched metadata and its corresponding normalized vector. Use the project key field as the third node and connect the third node with the corresponding second node to update the initial user file directory graph. If not, update the initial user file directory graph using the current tag and the connection relationships between tags.

7. A data processing system based on an integrated multimodal model, characterized in that, The data processing method based on an integrated multimodal model as described in any one of claims 1-6 includes: Data acquisition module: Acquires multimodal files uploaded by users through an interactive interface; Data parsing module: Parses multimodal files to obtain corresponding user and file information; Multimodal model processing module: Identifies file types of file information through multimodal models, selects different processing methods according to file types, and obtains metadata and its corresponding standardized vectors after processing; Structured database storage module: Constructs a structured database, obtains a preset file template in the structured database, matches the preset file template with metadata and its corresponding standardized vector, and stores the metadata and its corresponding standardized vector in a structured form in the structured database. User file directory map update module: Obtains the initial user file directory map displayed on the interactive interface based on user information, and updates the initial user file directory map according to the successfully matched preset file templates.

Images