A data processing method, device and storage medium

Abstract

Embodiments of the present application relate to the technical field of data processing, and provide a data processing method and device and a storage medium. The method comprises: obtaining data, a relationship of the data and metadata of the data; constructing a data graph according to the relationship of the structured data, wherein the relationship of the structured data comprises a relationship between tables and tasks, a relationship between tables and user operation behaviors, a relationship between tables and tables, etc.; performing semantic analysis and feature extraction on unstructured data to obtain a semantic vector of the unstructured data; fusing the data graph of the structured data and the semantic vector of the unstructured data to obtain a relationship network between the structured data and the unstructured data; determining the weight of the data and the importance of the data according to the relationship network and the label of the field; and displaying the data after the importance is determined. According to the actual use of the data, the importance of the structured data and the unstructured data is calculated together, the data is systematically measured, and the data is better managed.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, and more particularly to a data processing method, device and storage medium. BACKGROUND

[0002] With the continuous rise of the position of data elements in enterprise digital transformation, data as a core resource of enterprises, its management and evaluation ability is increasingly important. Traditional data processing is mostly focused on data sorting, classification, recording or setting permission control level, static tags are marked on data and data is evaluated manually, lacking systematic measurement and dynamic management mechanism of data value. Especially in the context of large number of data, various types and complex use behaviors, the existing technology is rough and single in data processing, which cannot handle complex different types of data. SUMMARY

[0003] Embodiments of the present application provide a data processing method, device and storage medium, which can systematically measure data and better manage data.

[0004] In a first aspect, embodiments of the present application provide a data processing method, which comprises: acquiring data, relationship of the data and metadata of the data, the data comprising structured data and unstructured data; constructing a data graph according to the relationship of the structured data, the relationship of the structured data comprising the relationship between tables and fields, the relationship between tables and tasks, the relationship between tables and user operation behaviors and the relationship between tables; performing semantic analysis and feature extraction on the unstructured data to obtain a semantic vector of the unstructured data; fusing the data graph of the structured data and the semantic vector of the unstructured data to obtain a relationship network between the structured data and the unstructured data; determining the weight of the data and determining the importance of the data according to the relationship network between the structured data and the unstructured data and the label of the field; and displaying the data after determining the importance.

[0005] In some possible implementations, the fusing the data graph of the structured data and the semantic vector of the unstructured data to obtain a relationship network between the structured data and the unstructured data comprises: analyzing the data graph of the structured data and the semantic vector of the unstructured data, and fusing the unstructured data into the data graph of the structured data according to the relationship of the data; the relationship of the data comprises: multiple items in the relationship of the structured data and the relationship between the unstructured data and the table, the relationship between the unstructured data and the task, the relationship between the unstructured data and the user operation behavior, the semantic similarity relationship between the unstructured data and the unstructured data, and the coupling relationship between the data.

[0006] In some possible implementations, determining the weight and importance of the data based on the relationship network and field labels of the structured and unstructured data includes: determining the importance of the structured data based on one or more indicators among the access volume, field sensitivity, or task level of the structured data; and determining the importance of the unstructured data based on one or more indicators among the semantic content quality, usage breadth, number of citations, or time activity of the unstructured data.

[0007] In some possible implementations, the access volume of the structured data is determined by the relationships between the tables established in the first time period, the relationships between the tables and tasks, and the relationships between the tables and user operation behaviors; the field sensitivity is determined by the relationships between the tables and fields and the labels of the fields; and the task level is determined by the number of downstream tasks that depend on it.

[0008] In some possible implementations, the semantic content quality of the unstructured data includes one or more of the following indicators: topic clarity, summary coverage, keyword richness, entity extraction accuracy, and language quality; the usage breadth is determined by the relationship between the unstructured data and the task and / or the relationship between the unstructured data and user actions; the number of citations is determined by the relationship between the unstructured data and the table, the semantic similarity relationship between unstructured data and unstructured data, or the coupling relationship between data; the time activity is determined by the number of citations in the third time period and the proportion of the usage breadth in the fourth time period.

[0009] In some possible implementations, the importance of the data includes important, general, and unimportant, and the display of the data after its importance has been determined includes: displaying data whose importance is important, a data weight path graph, and a data weight radar graph.

[0010] Secondly, embodiments of this application provide a data processing apparatus having functions corresponding to the data processing method provided in the first aspect above. These functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions, and these modules can be software and / or hardware.

[0011] In one embodiment, the data processing device includes: a data acquisition module for acquiring data, data relationships, and data metadata, wherein the data includes structured data and unstructured data; a structured data processing module for constructing a data graph based on the relationships of the structured data, wherein the relationships of the structured data include relationships between tables and fields, relationships between tables and tasks, relationships between tables and user actions, and relationships between tables; an unstructured data processing module for performing semantic parsing and feature extraction on the unstructured data to obtain semantic vectors of the unstructured data; a data fusion module for fusing the data graph of the structured data and the semantic vectors of the unstructured data to obtain a relationship network between the structured data and the unstructured data; an importance calculation module for determining the weight of the data and the importance of the data based on the labels of the fields in the relationship network between the structured data and the unstructured data; and a data display module for displaying the data after its importance has been determined.

[0012] In some possible implementations, the data fusion module is further configured to parse the data graph of structured data and the semantic vector of unstructured data, and to integrate unstructured data into the data graph of structured data according to the relationships between the data; the relationships between the data include multiple relationships among: relationships between structured data, relationships between unstructured data and tables, relationships between unstructured data and tasks, relationships between unstructured data and user operation behaviors, semantic similarity relationships between unstructured data and unstructured data, and coupling relationships between data.

[0013] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the data processing method as described in the first aspect.

[0014] Fourthly, embodiments of this application provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data processing method described in the first aspect.

[0015] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to execute the data processing method provided in the first aspect.

[0016] Compared to existing technologies, in this embodiment, structured and unstructured data are associated based on the relationship between them. The importance of both structured and unstructured data is calculated based on their actual usage, enabling a systematic measurement of data and better data management. Attached Figure Description

[0017] The objectives, features, and advantages of the embodiments of this application will become readily understood by referring to the accompanying drawings and the detailed description of the embodiments. Wherein:

[0018] Figure 1 This is a schematic diagram illustrating the implementation of unstructured data as a document in an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of the relationship network between the nodes of the data processing method in the embodiments of this application;

[0020] Figure 3 This is a schematic diagram of a page display module for the data processing method of this application embodiment;

[0021] Figure 4 This is a flowchart illustrating the data processing method according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the structure of the data processing apparatus according to an embodiment of this application;

[0023] Figure 6 This is another schematic diagram of the data processing apparatus according to an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of a server structure in one embodiment of this application.

[0025] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0026] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects (e.g., first unstructured data and second unstructured data represent different unstructured data, and so on), and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The division of modules appearing in the embodiments of this application is merely a logical division; in actual applications, there may be other division methods. For example, multiple modules may be combined into or integrated into another system, or some features may be ignored or not performed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through interfaces, indirect couplings between modules, or electrical or other similar forms of communication connections, none of which are limited in the embodiments of this application. Moreover, the modules or sub-modules described as separate components may or may not be physically separate, may or may not be physical modules, or may be distributed across multiple circuit modules. Some or all of these modules can be selected according to actual needs to achieve the purpose of the embodiments of this application.

[0027] The solutions provided in this application involve technologies such as Artificial Intelligence (AI), Computer Vision (CV), and Machine Learning (ML), which are specifically illustrated through the following embodiments:

[0028] AI, or Artificial Intelligence, refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, Artificial Intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine capable of reacting in a manner similar to human intelligence. Artificial Intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0029] AI technology is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0030] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes for tasks such as target recognition, tracking, and measurement, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include adversarial perturbation generation, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and biometric recognition technologies.

[0031] This application provides a data processing method, apparatus, and storage medium that associates structured and unstructured data based on the relationship between them. It also performs importance calculations on both structured and unstructured data based on their actual usage, achieving a systematic measurement of data and better data management.

[0032] Furthermore, the data processing method, apparatus, and storage medium of this application embodiment can also display data with higher weight, enabling users to intuitively obtain more valuable information.

[0033] The data processing method can be applied to a data processing apparatus capable of managing structured and unstructured data. This data processing apparatus may include a data acquisition module, a structured data processing module, an unstructured data processing module, a data fusion module, an importance calculation module, and a data visualization module. In some embodiments, the data processing apparatus further includes a model training and dynamic update module.

[0034] The data acquisition module is used to acquire data, data relationships, and data metadata.

[0035] The data includes both structured and unstructured data.

[0036] For example, the data acquisition module connects to structured and unstructured data sources via an API interface. It acquires tables from structured data sources and documents (such as logs), images, or videos from unstructured data sources. Structured data sources can be implemented as databases (such as MySQL, Oracle, Hive, etc.), data lakes, or data warehouses. Unstructured data sources can be implemented as knowledge document repositories, report management systems, image or video databases, etc.

[0037] Data relationships include one or more of the following: relationships between tables and fields, relationships between tables and tasks, relationships between tables and user actions, relationships between tables, relationships between unstructured data and tables, relationships between unstructured data and tasks, relationships between unstructured data and user actions, semantic similarity relationships between unstructured data and unstructured data, and coupling relationships between data.

[0038] The relationships in structured data include: relationships between tables and fields, relationships between tables and tasks, relationships between tables and user actions, and relationships between tables.

[0039] For example, the relationship between tables and fields can be determined by querying the fields in the table. For instance, if table 1 includes field 1 and field 2, then table 1 has a relationship with field 1 and field 2. If table 2 includes field 3 and field 4, then table 2 has a relationship with field 3 and field 4.

[0040] Tasks can be Extract-Transform-Load (ETL) tasks, such as data cleaning, field mapping, format conversion, dimensional modeling, task scheduling, and dependency management. Tasks can also be implemented as views, stored procedures, functions, offline synchronization tasks, offline development tasks, real-time synchronization tasks, or real-time computing tasks. Offline synchronization tasks store user-provided data from data sources (such as business databases, log files, third-party systems, etc.) into the data processing device's database at regular or different intervals. Real-time synchronization tasks store data from data sources into the data processing device's database with low latency and near real-time performance. Offline development tasks perform development operations on the database in the data processing device, such as inserting data from Table 1 into Table 2. The database type can be various, such as relational databases, key-value databases, document databases, or graph databases. Real-time computing tasks transform user-provided data in real time, such as transforming user-provided Kafka data into the data processing device's database in real time to form persistent assets.

[0041] For example, the relationship between tables and tasks, and the relationship between unstructured data and tasks, can be determined through task scheduling information. Task scheduling information includes records of task-to-scheduled data. For instance, the task scheduling information for offline development task 1 records that field 1 from table 3 of the data processing device's database is inserted into table 4. The relationship between table 3 and field 1 indicates that offline development task 1 and table 3 have a relationship. Offline development task 1 and table 4 also have a relationship. As another example, if a task is a view, and the view's task scheduling information includes references to tables 1 and 2, then both tables 1 and 2 are queried by the view, or tables 1 and 2 are joined by the view. Therefore, there is a relationship between tables 1, 2, and the view. For example, if the view `vw_order_user` references the order table and the user table during definition, then the view has a relationship with the order table and the user table. For instance, the definition of view `vw_order_user` can be implemented as follows: create a view named `vw_order_user`, select the order identity document (ID) and username, and join the user table from the order table; then the view has a relationship with the order table and the user table. The join condition is that the user ID in the order table is equal to the user ID in the user table.

[0042] The relationship between tables and user actions, and the relationship between unstructured data and user actions, can be determined by user actions obtained from user behavior logs. User actions may include data queries, table structure modifications, permission requests, task configuration (such as changing the task scheduling cycle from daily to weekly), log access, API calls, data downloads and sharing, etc.

[0043] For example, if user 1 requests permission to access tables 1 and 2, then user 1 has permission to access both tables 1 and 2, and there is a relationship between the user's action (user 1's permission request action) and table 1. Similarly, there is also a relationship between user 1 and table 2.

[0044] For another example, if User 1 accesses Table 1 within a preset time period, then there is a relationship between the user's action (User 1's data query behavior) and Table 1. The preset time period can be the time prior to establishing the relationship between the graph nodes, such as 1 day, 1 week, or 1 month prior to establishing the relationship between the graph nodes. Determining the access relationship between user actions and tables by knowing how users access tables within a preset time period makes the relationship between user actions and tables a better indicator of the table's importance, thus making it more beneficial for the importance calculation module to calculate the importance of the table.

[0045] For example, let's take Table 1 and Table 2 as examples to illustrate the relationships between tables. For instance, if a field in Table 1 references a field in Table 2 via a foreign key, then Table 1 and Table 2 have a relationship. Similarly, the user ID field in the order table might be used as a foreign key to reference the user ID field in the user table. Furthermore, the ETL task `etl_orders_summary` aggregates data from the order table and writes it to the order summary table; that is, the data in the order summary table originates from the order table, thus establishing a relationship between Table 1 and Table 2.

[0046] For example, if the content of unstructured data includes the content of a table, then there is a relationship between the unstructured data and the included table. See also Figure 1 For example, if unstructured data is implemented as document 1, and document 1 contains structured data table 1, then document 1 and table 1 are related. Similarly, if offline development task 2 converts Excel document 1 (e.g., test.xlsx) into a table (e.g., table1), then test.xlsx and table1 are related. The relationship between test.xlsx and table1 is that test.xlsx points to table1.

[0047] For example, if the content of the first unstructured data is similar to the content of the second unstructured data, then the first unstructured data and the second unstructured data have a semantic similarity relationship. For instance, when unstructured data is implemented as documents, if the content of document 1 is similar to the content of document 2, document 1 and document 2 have a semantic similarity relationship. For example, if unstructured data meet one or more of the following conditions, then these structured data have a semantic similarity relationship: having the same topic, having a first or more identical keywords, belonging to the same sentiment type, or belonging to the same industry.

[0048] In some examples, the first quantity is 2, 3, or 5. In other examples, the first quantity is a preset percentage of the average number of keywords across multiple unstructured data sets, such as 20%, 30%, or 50%. For example, if document 1's semantic vector contains 8 keywords and document 2's semantic vector contains 12 keywords, then the average number of keywords for document 1 and document 2 is 10, and the first quantity is 2, 3, or 5.

[0049] For example, if Task 1 and Task 2 perform the same type of operation on the same data, then the generated data are coupled. It's understood that "same type" refers to the type of operation, such as conversion or copying, not that the content of the operation is the same. For instance, converting the first page of Document 1 into Table 1 and the second page of Document 1 into Table 2 creates a coupling relationship between Table 1, Table 2, and Document 1. It is not necessary to convert the same content of Document 1 into Table 1 and Table 2 for Table 1 and Table 2 to have a coupling relationship.

[0050] Data metadata includes metadata for structured data and metadata for unstructured data. Taking a table as an example, the metadata for structured data includes the table name, the name of the data source containing the table (e.g., MySQL1), the data source (e.g., MySQL or Oracle), and the data source type (e.g., database or data lake). Metadata for unstructured data includes: the author of the unstructured data, creation time, number of accesses, and number of citations.

[0051] For example, the metadata of data can be metadata of one or more dimensions of business metadata, technical metadata, and management metadata.

[0052] For example, business metadata includes, but is not limited to, one or more of the following:

[0053] Business areas include: such as customer management, transaction clearing, risk control, and marketing.

[0054] Business manager or product manager: such as Zhang San (customer operations).

[0055] Business terms or business meanings: such as customer_id means "unique customer identifier"; order_amount means "order amount".

[0056] Main use cases: such as marketing profiling, anti-fraud analysis, and end-of-day reconciliation.

[0057] Business priority (or business importance): such as core business, important, ordinary.

[0058] Exemplary technical metadata includes, but is not limited to, one or more of the following:

[0059] Database type / storage location: such as Hive, MySQL, Postgres, object storage (s3: / / bucket / ...).

[0060] Field list and types: such as user_id: bigint, name: varchar(256), create_time: timestamp.

[0061] Partition keys: such as dt, year_month.

[0062] Number of rows / table size: e.g., rows=120,000,000, size=45GB.

[0063] Update frequency: such as real-time, hourly, daily.

[0064] Access methods: such as CDC, batch import, API push.

[0065] Last updated time: e.g., 2025-08-30T02:10:00Z.

[0066] Index / Primary Key Information: For example, primary key = user_id, secondary index = create_time.

[0067] For example, managing metadata includes, but is not limited to, one or more of the following:

[0068] Sensitivity / Confidentiality level: such as public, internal, sensitive, top secret.

[0069] Compliance / regulatory labels: such as PIPL, GDPR.

[0070] Retention period: such as 7 years, 3 years, or permanent.

[0071] Access control / authorization policies: such as RD group only, approval required, public API.

[0072] Data quality weight: such as 0.98 (0~1), or graded as A / B / C.

[0073] Lifecycle states: such as active, deprecated, archived.

[0074] Data governance responsible person: such as Li Si (data platform).

[0075] Scope of sharing: such as sharing within the enterprise, sharing across departments, and sharing with partners, requires approval.

[0076] The data acquisition module collects metadata through a metadata collector. This metadata collector can be a log collector. For example, it can collect logs through a metadata collector or collect task scheduling information through an API interface.

[0077] In some embodiments, the data acquisition module can also convert the acquired data according to a preset format of the device, and store the data after unifying the format. This allows for easier retrieval of the data and display of the data or its metadata in subsequent processes.

[0078] In some embodiments, the data acquisition module can also be used to acquire metadata for fields and tasks. Field metadata includes field name, type, length, whether it is nullable, and whether it is a primary key. Task metadata includes the task's Chinese name, English name, and task type (e.g., offline synchronization task, offline development task, etc.). This allows the display of metadata for tasks and fields related to the data during subsequent data presentation, providing users with more information associated with the data and facilitating their assessment of its importance.

[0079] The structured data processing module is used to construct data graphs based on the relationships within structured data.

[0080] The structured data processing module abstracts structured data, fields, tasks, and user actions into graph nodes, and builds a data graph based on the relationships between these graph nodes.

[0081] Data graphs can be stored and queried using graph databases (such as Neo4j) or graph engines (such as Apache AGE). By constructing a data graph, other modules can easily perform graph traversal in scenarios such as data influence analysis. The weight propagation process can be obtained by understanding the actual use of data, the dependencies between data points, and the upstream and downstream relationships of tasks. This allows for the calculation of weights on structured data, resulting in the process of determining the node's final weight (or influence distribution) after considering the network structure's impact. Weight propagation refers to the process of transferring and spreading the initial weight calculated based on the characteristics of a node in the data graph along the relational edges of the data graph to adjacent nodes according to certain rules.

[0082] Reference Figure 2 , Figure 2 A schematic diagram of the network relationships between the nodes is shown. Figure 2 The data shown includes Table 1, Table 2, Document 1, Document 2, Field 1, Field 2, Task 1, and User Action 1. Ellipses represent nodes, and lines connecting ellipses indicate relationships between nodes. It can be seen that there are relationships between Table 1 and Field 1 / Field 2; relationships between Table 1 / Table 2 and Task; a relationship between Table 1 and User Action 1; a referencing relationship between Table 1 and Table 2 / Document 1; a relationship between Table 2 and User Action 1; and a semantic similarity relationship between Document 2 and Document 1.

[0083] Understandable Figure 2 The data in the data can also have one or more of the relationships mentioned above with other data. Figure 2 The ellipsis indicates this. Figure 2 This is only an example of the relationships between some of the data and does not represent the relationships between all the data acquired by the device.

[0084] The unstructured data processing module is used to perform semantic parsing and feature extraction on unstructured data to obtain semantic vectors of the unstructured data.

[0085] In some embodiments, there are multiple semantic vectors of unstructured data, and the unstructured data processing module is also used to fuse the multiple semantic vectors of unstructured data into a single semantic vector through concatenation or attention mechanisms.

[0086] For example, for unstructured data of document type, the unstructured data processing module can perform operations such as topic extraction, summary generation, keyword extraction, sentiment judgment, and industry classification on the document content by calling a large language model to obtain the document's semantic vector. The semantic vector of a document can represent one or more of the document's content, topic, summary, keywords, and category. Category classification methods can include sentiment type, industry, legal attributes, etc. For example, according to legal attributes, documents can be classified as invoices, contracts, etc.

[0087] For another example, for unstructured data such as images or videos, the unstructured data processing module can use multimodal models (such as CLIP and BLIP) to perform image recognition on the images or videos to obtain the text in the images. Then, it can use a large language model to perform tasks such as topic extraction, summary generation, keyword extraction, and classification on the recognized text.

[0088] For example, a large language model first performs visual encoding and optical character recognition (OCR) on images or videos to obtain one or more of the following: visual features of the image, text content within the image, sentiment category of the image or video, and industry tags. For instance, if the image is of an apple, the visual features might be: apple, red, fruit, etc., and the text content might include phrases like "a big, red apple...". Visual features are generated as vectors using a multimodal encoder. Text content is converted into vectors using a text embedder. Sentiment category and industry tags can be vectorized using relevant techniques. These semantic vectors are then projected, normalized, and fused into a single semantic vector through concatenation or attention mechanisms, and stored in a vector library. This semantic vector can be used for similarity retrieval, semantic matching, or weight evaluation, enabling the semantic information of images or videos to be measured in a unified manner with documents, tables, and other data.

[0089] It should be noted that each piece of unstructured data can include one or more categories. For example, if unstructured data 1 is a video containing both invoice and contract content, then after parsing the video content, the large language model will classify the video as including invoices and contracts.

[0090] By obtaining the semantic vector of unstructured data, other modules can process the unstructured data, thereby determining the relationship between the unstructured data and structured data based on the content of the unstructured data, obtaining the value of the unstructured data, and thus obtaining the weight of the unstructured data.

[0091] In some embodiments, the unstructured data processing module is also used to store the semantic vectors of the unstructured data into a vector library. For example, the document category is stored as an invoice or contract, and the content includes amount, date, personnel, etc. When querying "contract" in the vector library later, the current document can be retrieved through the vectorized representation of the document.

[0092] The data fusion module is used to fuse the data graph of structured data and the semantic vector of unstructured data to obtain the relationship network between structured and unstructured data.

[0093] This includes data graphs that integrate structured data and semantic vectors that integrate unstructured data. The latter involves parsing the structured data into the data graph and then integrating the unstructured data into the structured data's semantic vectors based on the relationships between the data (e.g., ...). Figure 2 The data graph shown constructs a relationship network between structured and unstructured data. The importance calculation module can then use this network to determine the relationships between structured and unstructured data, thereby enabling the calculation of data importance.

[0094] The importance calculation module is used to determine the weight of data and the importance of data based on the relationship network between structured and unstructured data and the labels of fields.

[0095] For example, data importance can be categorized as important, average, and unimportant. Important indicates that the data is core data, average indicates that the data is relatively important, and unimportant indicates that the data is ordinary data. By determining the importance of data, it can be displayed or made available to external systems accordingly. For example, core data can be recommended to users in a user interface or fed back to external systems.

[0096] Determining the weight and importance of data based on relational networks and field labels can be achieved by determining the importance of data based on one or more indicators in the structured data access volume, field sensitivity, or task level.

[0097] For example, the access volume of structured data can be determined by the relationships between tables, between tables and tasks, and between tables and user actions established within a first time period. For instance, the first time period is 7 days. If the relationships between tables and tasks, and between tables and user actions established within 7 days, are greater than a first threshold, it indicates high access volume for the data; if they are not greater than the first threshold but greater than a second threshold, it indicates moderate access volume; and if they are not greater than the second threshold, it indicates low access volume. The first threshold is greater than the second threshold; for example, the first threshold might be 1000, and the second threshold might be 100.

[0098] For example, the access volume of structured data can also be determined by the number of relationships between tables, relationships between tables and tasks, and / or relationships between tables and user actions. For instance, if a table has relationships with three types of tasks and two types of user actions, then the number of relationships between tables and tasks, and between tables and user actions, is 5. If the number of relationships between tables and tasks, and between tables and user actions, is greater than a third threshold, it indicates high access volume for the data; if it is no greater than the third threshold but greater than a fourth threshold, it indicates moderate access volume; and if it is no greater than the fourth threshold, it indicates low access volume. The third threshold is greater than the fourth threshold; for example, the third threshold is 30, and the fourth threshold is 10.

[0099] For example, field sensitivity is used to indicate the sensitivity of content in a table, and can be determined by the relationship between the table and fields and the field labels. For example, if fields in a table that meet or exceed a fifth threshold have sensitive labels, then the field sensitivity of that table is high. If the number of fields with sensitive labels in a table is less than the fifth threshold but not less than a sixth threshold, then the field sensitivity of that table is low. For example, the fifth threshold might be 5 fields and the sixth threshold might be 3 fields. As another example, the fifth and sixth thresholds can be determined based on the proportion of the total number of fields in the table, with the fifth threshold being 50% and the sixth threshold being 20%. For example, if the total number of fields is 20, then the fifth threshold is 10 and the sixth threshold is 4. For example, sensitive labels can be pre-stored.

[0100] For example, the task level is determined by the number of downstream tasks it depends on. For instance, if a task is depended on by the second number of downstream tasks, the task level is high; if a task is depended on by fewer than the second number, but the third number or more, the task level is medium; and if a task is directly depended on by fewer than the third number of downstream tasks, the task level is low. For example, the second number is 5 and the third number is 1. (See reference...) Figure 2Table 1 points to Task 1, and Task 1 points to Table 2, indicating that Table 1 is upstream data of Table 2, and Table 2 is downstream data of Table 1, or that part of the content of Table 2 comes from Table 1. If Table 2 has no downstream data, then Table 1 is dependent on by one downstream task, and the task level is low. If Table 1, in addition to pointing to Task 1, also points to Task 2, Task 2 points to Table 3, Table 3 points to Task 3, and Task 4 points to Table 4, then a downstream link of Table 1 has at most two tasks, and Table 1 is dependent on by two downstream tasks, and the task level is medium.

[0101] Determining the weight and importance of data based on relational networks and field labels can be achieved by determining the importance of unstructured data based on one or more of the following indicators: semantic content quality, usage breadth, number of citations, or time activity.

[0102] For example, the semantic content quality of unstructured data is used to measure the completeness, specificity, and business relevance of information in unstructured data, including one or more indicators such as topic clarity, summary coverage, keyword richness, entity extraction accuracy, and language quality. For instance, a large language model can extract semantic vectors from unstructured data and compare them with a pre-stored content dictionary to score the content quality of the semantic vectors. The content dictionary includes core domain concepts, standard keywords, summary examples, or grammatical style guidelines. Core domain concepts are highly relevant professional terms or entity types (such as product names, technical component names, business process names, etc.) related to specific business operations. Standard keywords are a list of keywords and phrases used to measure the topic relevance of documents. Summary examples are used as a reference to evaluate the coverage and information density of the document summary under test. Grammatical style guidelines are rules used to evaluate language quality, such as sentence complexity and common error patterns.

[0103] The large language model compares the semantic vectors with a pre-stored content dictionary, scoring the content quality by evaluating multiple aspects, including topic clarity, summary coverage, keyword richness, and language quality. Topic clarity scoring involves matching the model-identified topics and extracted keywords with the "domain core concepts" and "standard keywords" in the content dictionary. Higher matching (i.e., more relevant the text content to the business domain) results in a higher score. Summary coverage scoring compares the semantic similarity of the model-generated summary with "summary examples" in the content dictionary, checking if it covers the key entities and core arguments of the original text. Keyword richness scoring assesses whether the extracted keywords comprehensively cover relevant concepts in the dictionary and whether they are distinctive; the more relevant concepts covered, the higher the score. Language quality scoring determines the presence of grammatical errors and unclear expressions based on the dictionary's "grammatical style guidelines," scoring these issues accordingly; fewer errors and unclear expressions result in a higher score. A semantic content quality score above the seventh threshold indicates high semantic content quality. If the semantic content quality score is no higher than the seventh threshold but higher than the eighth threshold, the semantic content quality is considered moderate. If the semantic content quality score is no higher than the eighth threshold, the semantic content quality is considered low. The seventh threshold is higher than the eighth threshold. For example, the seventh threshold is 0.85, and the eighth threshold is 0.6.

[0104] For example, the breadth of use of unstructured data, used to measure how many different users consume the unstructured asset, can be determined by the relationship between the unstructured data and tasks and / or the relationship between the unstructured data and user actions. For instance, if the unstructured data is accessed by a fourth or more users, the breadth of use is high. If the unstructured data is referenced by fewer than a fourth but more than a fifth of independent assets / projects, the breadth of use is moderate. If the unstructured data is referenced by fewer than a fifth of independent assets / projects, the breadth of use is low. Here, the fourth number is greater than the fifth number. For example, the fourth number is 100 and the fifth number is 50.

[0105] For example, the citation count of unstructured data refers to the cumulative number of times it is explicitly referenced by other assets. This can be determined through the relationship between unstructured data and tables, the semantic similarity between unstructured and unstructured data, or the coupling relationship between data. The citation count can be counted according to document paragraphs, SQL scripts, or reference entries in reports. For instance, within a second time period, if the citation count of unstructured data is greater than the ninth threshold, the citation count is considered high; if it is not greater than the ninth threshold but greater than the tenth threshold, the citation count is considered moderate; and if it is not greater than the tenth threshold, the citation count is considered low. The ninth threshold is greater than the tenth threshold. For example, if the second time period is 30 days, the ninth threshold is 50, and the tenth threshold is 10. The citation count of unstructured data can reflect the degree to which the data is relied upon.

[0106] In some embodiments, the second time period is the same as the first time period.

[0107] For example, the time activity level of unstructured data is used to measure the activity of the data over time, reflecting whether the data is still in use. Time activity level can be determined by the percentage of citations and usage breadth within a third time period compared to a fourth time period. For instance, if the percentage of citations and usage breadth within the most recent third time period compared to the most recent fourth time period is greater than the eleventh threshold, then the time activity level is high; if the percentage is not greater than the eleventh threshold but greater than the twelfth threshold, then the time activity level is moderate; if the number of accesses within the most recent third time period is not greater than the twelfth threshold, then the time activity level is low. For example, if the third time period is 7 days, the fourth time period is 11 days, the eleventh threshold is 50%, and the twelfth threshold is 20%.

[0108] In some examples, if a data point has a weight greater than the first weight, then that data point is considered important; if a data point has a weight no greater than the first weight but greater than the second weight, then that data point is considered moderately important; and if a data point has a weight no greater than the second weight, then that data point is considered unimportant. Here, the first weight is greater than the second weight.

[0109] Importance calculation models can calculate the importance of data and obtain the weight of the data.

[0110] For example, the importance calculation model uses a weighted formula (such as a weighted model). For structured data, the weight is calculated as: 0.5 * number of visits + 0.3 * field sensitivity + 0.2 * task level. For unstructured data, the weight is calculated as: 0.2 * semantic content quality + 0.2 * usage breadth + 0.4 * number of citations + 0.2 * time activity. The weight of each metric can be determined based on its level. For example, a high number of visits would have a weight of 10, a moderate number of visits would have a weight of 7, and a low number of visits would have a weight of 3. After calculating each weight, a weighted sum is applied, and a threshold is mapped to a high, medium, or low level.

[0111] In other examples, the importance calculation model is a machine learning model (such as XGBoost or LGBM). Exemplary input features include: access volume of structured data, field sensitivity or task level, semantic content quality of unstructured data, usage breadth, number of citations, or temporal activity. Training data can be derived from expert calculations or historical asset valuation results; the model predicts rankings through supervised learning, automatically determining the importance of data.

[0112] In other examples, the importance calculation model is a graph neural network propagation model. For instance, a graph neural network is used, with data, tasks, user actions, and fields as nodes, and relationships between data, data and fields, tasks, and user actions as edges. The training objective is to predict the level labels of the nodes. The training samples can come from partially manually labeled data assets.

[0113] The data display module is used to present data whose importance has been determined.

[0114] In some embodiments, the data display module is used to display the weight ranking of the data.

[0115] In some examples, the weighted ranking of data is implemented by displaying the data in the ranking corresponding to the control when the control is triggered. The controls include "Advanced Data," "Medium Data," and "Low Data" controls. "Advanced Data" controls correspond to data of high importance, "Medium Data" controls to data of moderate importance, and "Low Data" controls to data of low importance.

[0116] by Figure 2 Taking the data shown as an example, the data includes Document 1, Document 2, Table 1, and Table 2. Assume the weights of Document 1, Document 2, Table 1, and Table 2 are 91, 88, 94, and 80 respectively. Where the first weight is 90 and the second weight is 85, then Document 1 and Table 1 have weights greater than 90, corresponding to important importance. Correspondingly, Document 2 has a weight no greater than 90 but greater than 85, corresponding to moderate importance. Table 2 has a weight no greater than 85, corresponding to unimportant importance. (Refer to...) Figure 3 , Figure 3 This diagram illustrates the page layout displayed by the data display module. When the "Advanced Data" control is triggered, the page displays the contents of Table 1 and Document 1. Correspondingly, when the "Intermediate Data" control is triggered, the page displays the contents of Document 2. When the "Basic Data" control is triggered, the page displays the contents of Table 2 and Table 3.

[0117] In some embodiments, the data display module may also display metadata instead of the data content. The data display module then shows the data content when the user clicks the control to display metadata, thus allowing more data to be displayed to the user on a limited screen because the metadata content is relatively small.

[0118] In other examples, the data weighting is implemented by displaying only data with high importance. By showing data with high importance, the system can recommend the most useful information to users, making it easier for them to use the information.

[0119] In other embodiments, the data display module is also used to display a data weight path diagram. The data weight path diagram includes the basis for the data weights. For example, the data weight path diagram can be implemented as follows: Figure 2 The data relationship path diagram is shown.

[0120] In other embodiments, the data display module is also used to display a data weight radar chart. The data weight radar chart includes multiple data controls; if a data control is triggered, the relationship path of that data is displayed. For example, the data weight radar chart displays data including Document 1, Document 2, and Tables 1 to 3, and the relationships between the various data are as follows: Figure 2 As shown. If the control in Document 1 is triggered, Document 1, Document 2, and the semantic similarity relationship between Document 1 and Document 2 will be displayed.

[0121] The model training and dynamic update module is used to optimize the relational network of the data based on user feedback or by using self-supervised learning, thereby achieving optimization of the relational network.

[0122] For example, changes in task content, table fields, or user actions can periodically retrain the importance calculation model when the model training and dynamic update modules change. For instance, if a company adds an online e-commerce channel to its existing retail business, the access data expands from the offline sales table to the online order table—meaning the table content changes, which alters the importance of the data assets. Another example is changing a daily synchronization task to an hourly real-time synchronization task, altering the scheduling frequency of the data tables, which directly affects the table's importance calculation.

[0123] The model training and dynamic update module can also use user feedback as a reinforcement learning signal, dynamically adjusting the importance calculation criteria based on user feedback. This makes the page content displayed to users more aligned with their needs. For example, when the importance calculation model is a machine learning model (such as XGBoost or LGBM), user feedback can be input into the machine learning model, making the model's judgment of data weights more consistent with user needs.

[0124] The model training and dynamic update module can also automatically update the data map based on events such as the addition, deletion, and modification of data.

[0125] The data processing device provided in this application calculates and displays the importance of data, enabling enterprise management departments to prioritize critical data, such as monitoring data and asset management, based on the device's display. Through a data weight radar chart, when users perform operations such as deleting or modifying data, the device can assess the risk and impact of data deletion or modification based on the data's importance, thereby assisting users in maintaining data.

[0126] The data processing method of this application embodiment is described below. It is understood that, in addition to the content described below, the explanation, examples, and acquisition methods of the terms in the data processing method of this application embodiment can also be referred to the above-described data processing device embodiment, and will not be repeated here.

[0127] Reference Figure 4 The method is executed by a data processing device and includes steps 101 to 106.

[0128] Step 101: Obtain the data, the relationships between the data, and the metadata of the data.

[0129] The data includes both structured and unstructured data.

[0130] Step 102: Construct a data graph based on the relationships in the structured data.

[0131] Relationships in structured data include those between tables and fields, between tables and tasks, between tables and user actions, and between tables themselves.

[0132] Step 103: Perform semantic parsing and feature extraction on the unstructured data to obtain the semantic vector of the unstructured data.

[0133] In some embodiments, there are multiple semantic vectors for unstructured data, and step 103 can be implemented as step 1031.

[0134] Step 1031: Multiple semantic vectors of unstructured data are fused into a single semantic vector through concatenation or attention mechanisms.

[0135] For example, for unstructured data of document type, the unstructured data processing module can perform operations such as topic extraction, summary generation, keyword extraction, sentiment judgment, and industry classification on the document content by calling a large language model to obtain the document's semantic vector. The semantic vector of a document can represent one or more of the document's content, topic, summary, keywords, and category. Category classification methods can include sentiment type, industry, legal attributes, etc. For example, according to legal attributes, documents can be classified as invoices, contracts, etc.

[0136] For another example, for unstructured data such as images or videos, the unstructured data processing module can use multimodal models (such as CLIP and BLIP) to perform image recognition on the images or videos to obtain the text in the images. Then, it can use a large language model to perform tasks such as topic extraction, summary generation, keyword extraction, and classification on the recognized text.

[0137] It should be noted that each piece of unstructured data can include one or more categories. For example, if unstructured data 1 is a video containing both invoice and contract content, then after parsing the video content, the large language model will classify the video as including invoices and contracts.

[0138] Step 104: Merge the data graph of structured data and the semantic vector of unstructured data to obtain the relationship network between structured data and unstructured data.

[0139] Step 105: Determine the weights and importance of the data based on the relationship network between structured and unstructured data and the labels of the fields.

[0140] Step 106: Display the data after determining its importance.

[0141] In some embodiments, step 104 can be implemented as step 1041.

[0142] Step 1041: Parse the data graph of structured data and the semantic vector of unstructured data, and integrate the unstructured data into the data graph of structured data according to the relationships between the data.

[0143] The relationships between data include one or more of the following: relationships between structured data and between unstructured data and tables; relationships between unstructured data and tasks; relationships between unstructured data and user actions; semantic similarity relationships between unstructured and unstructured data; and coupling relationships between data. For example, a large language model first performs visual encoding and optical character recognition (OCR) on an image or video to obtain one or more of the following: visual features of the image, text content in the image, sentiment category of the image or video, and industry tags.

[0144] Relationships in structured data include: relationships between tables and fields, relationships between tables and tasks, relationships between tables and user actions, and relationships between tables.

[0145] Tasks can be data extraction, transformation, and loading (ETL) tasks, such as data cleaning, field mapping, format conversion, dimensional modeling, task scheduling, and dependency management. Tasks can also be implemented as views, stored procedures, functions, offline synchronization tasks, offline development tasks, real-time synchronization tasks, or real-time computing tasks.

[0146] For example, the relationship between tables and tasks, and the relationship between unstructured data and tasks, can be determined through task scheduling information. Task scheduling information includes records of task-to-schedule data.

[0147] The relationship between tables and user actions, and the relationship between unstructured data and user actions, can be determined by user actions obtained from user behavior logs. User actions may include data queries, table structure modifications, permission requests, task configuration (such as changing the task scheduling cycle from daily to weekly), log access, API calls, data downloads and sharing, etc.

[0148] If a field in Table 1 references a field in Table 2 via a foreign key, then Table 1 and Table 2 have a relationship. If the content of unstructured data includes content from a table, then there is a relationship between the unstructured data and the included table. If the content of the first unstructured data is similar to the content of the second unstructured data, then there is a semantic similarity relationship between the first and second unstructured data.

[0149] Structured data are semantically similar if they meet one or more of the following conditions: they have the same topic, they have more than one number of the same keywords, they belong to the same sentiment type, or they belong to the same industry.

[0150] If Task 1 and Task 2 perform the same type of operation on the same data, then the generated data are coupled together.

[0151] Data metadata includes metadata for structured data and metadata for unstructured data. Taking a table as an example, the metadata for structured data includes the table name, the name of the data source containing the table (e.g., MySQL1), the data source (e.g., MySQL or Oracle), and the data source type (e.g., database or data lake). Metadata for unstructured data includes: the author of the unstructured data, creation time, number of accesses, and number of citations.

[0152] For example, the metadata of data can be metadata of one or more dimensions of business metadata, technical metadata, and management metadata.

[0153] In some embodiments, step 201 is included before step 106, so that data can be obtained more conveniently in the future, and the data or data metadata can be displayed.

[0154] Step 201: Convert the collected data according to the preset format of the device, and store the data after unifying the format.

[0155] In some embodiments, step 202 is included before step 106.

[0156] Step 202: Obtain the metadata of the fields and the metadata of the task.

[0157] The metadata for a field includes its name, type, length, whether it is nullable, and whether it is a primary key. The metadata for a task includes its Chinese name, English name, and type (e.g., offline synchronization task, offline development task). This allows for the display of metadata related to the data, showing users more relevant information and facilitating their assessment of the data's importance.

[0158] In some embodiments, step 102 can be implemented as step 1021.

[0159] Step 1021: Abstract structured data, fields, tasks, and user operation behaviors into graph nodes, and establish a data graph based on the relationships between graph nodes.

[0160] Data graphs can be stored and queried using graph databases (such as Neo4j) or graph engines (such as Apache AGE). By constructing a data graph, other modules can easily perform graph traversal in scenarios such as data influence analysis. The weight propagation process can be obtained by understanding the actual use of data, the dependencies between data points, and the upstream and downstream relationships of tasks. This allows for the calculation of weights on structured data, resulting in the process of determining the node's final weight (or influence distribution) after considering the network structure's impact. Weight propagation refers to the process of transferring and spreading the initial weight calculated based on the characteristics of a node in the data graph along the relational edges of the data graph to adjacent nodes according to certain rules.

[0161] In some embodiments, step 105 can be implemented as steps 1051 and 1052.

[0162] Step 1051: Determine the importance of the structured data based on one or more metrics, such as the access volume of the structured data, field sensitivity, or task level.

[0163] Step 1052: Determine the importance of unstructured data based on one or more of the following metrics: semantic content quality, usage breadth, number of citations, or time activity.

[0164] In some embodiments, the access volume of structured data is determined by the tables and relationships between tables established in a first time period, the relationships between tables and tasks, and the relationships between tables and user actions. Field sensitivity is determined by the relationships between tables and fields and the field labels. Task hierarchy is determined by the number of downstream tasks that depend on it.

[0165] In some embodiments, the semantic content quality of unstructured data includes one or more of the following indicators: topic clarity, summary coverage, keyword richness, entity extraction accuracy, and language quality.

[0166] In some embodiments, breadth of use is determined by the relationship between unstructured data and tasks and / or the relationship between unstructured data and user actions.

[0167] In some embodiments, the number of references is determined by the relationship between unstructured data and tables, the semantic similarity between unstructured data and unstructured data, or the coupling relationship between data.

[0168] In some embodiments, time activity is determined by the percentage of citations and usage breadth during the third time period compared to the fourth time period.

[0169] In some embodiments, the importance of the data includes important, general, and unimportant, and step 106 is implemented as step 1061.

[0170] Step 1061 shows the importance of the data as important data, a data weight path diagram, and a data weight radar chart.

[0171] The importance of the data being presented can include the content of the data and its metadata. The data weight path diagram can include other data, tasks, and fields related to the data. The metadata of tasks and fields can be displayed within the data weight path diagram.

[0172] The functions implemented by the data processing device can be achieved through hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned functions, and these modules can be software and / or hardware.

[0173] The data processing apparatus in the embodiments of this application has been described above from the perspective of modular functional entities. The data processing apparatus in the embodiments of this application will be described below from the perspective of hardware processing.

[0174] like Figure 5 As shown, the data processing device 60 may include an input / output module 601 and a processing module 602. The processing module 602 can implement the functions of the aforementioned data acquisition module and data display module. The input / output module 601 can implement the functions of the aforementioned structured data processing module, unstructured data processing module, data fusion module, and importance calculation module, which will not be elaborated here. The processing module 602 can be used to control the sending, receiving, and acquisition operations of the input / output module 601.

[0175] Figure 5 The physical device corresponding to the input / output module 601 shown can be a transceiver, radio frequency circuit, communication module, and input / output (I / O) interface, etc., and the physical device corresponding to the processing module 602 can be a processor.

[0176] Figure 5 The devices shown can all have the following characteristics: Figure 6 The structure shown, when Figure 5 The data processing device 60 shown has, for example Figure 6 When the structure shown is used, Figure 6 The processor and transceiver in the device can perform the same or similar functions as the processing module 602 and input / output module 601 provided in the aforementioned device embodiments. Figure 6 The memory stores the computer programs that the processor needs to call when executing the above data processing methods.

[0177] It should be noted that the apparatus involved in the embodiments of this application may be an enterprise big data platform, a data middleware, a data catalog system, or a master data processing system, etc., and the system involved in the embodiments of this application may be deployed on a server.

[0178] The server involved in the embodiments of this application can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0179] See Figure 7 , Figure 7 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1100 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1122 (e.g., one or more processors) and memory 1132, and one or more storage media 1130 (e.g., one or more mass storage devices) for storing application programs 1142 or data 1144. The memory 1132 and storage media 1130 may be temporary or persistent storage. The program stored in the storage media 1130 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the CPU 1122 may be configured to communicate with the storage media 1130 and execute the series of instruction operations in the storage media 1130 on the server 1100.

[0180] Server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input / output interfaces 1158, and / or one or more operating systems 1141, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0181] The steps performed by the server in the above embodiments can be based on this Figure 7 The structure of server 1100 is shown. For example, in the above embodiment, it consists of... Figure 5 The steps performed by the input / output module 601 and the processing module 602 shown can be based on this Figure 7 The server structure is shown. For example, the central processing unit 1122 performs the following operations by calling instructions from memory 1132:

[0182] Data, data relationships, and data metadata are obtained through input / output interface 1158, and the data is displayed after its importance has been determined.

[0183] The above data processing methods can also be used to perform input / output related operations through the input / output interface.

[0184] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0185] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0186] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0187] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0189] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0190] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0191] The technical solutions provided in the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. A data processing method, characterized in that, The method includes: Acquire data, the relationships between data, and the metadata of the data, wherein the data includes structured data and unstructured data; A data graph is constructed based on the relationships of structured data, including relationships between tables and fields, relationships between tables and tasks, relationships between tables and user actions, and relationships between tables; wherein, the relationship between tables and tasks is determined through task scheduling information. Semantic parsing and feature extraction are performed on the unstructured data to obtain the semantic vector of the unstructured data; By fusing the data graph of the structured data and the semantic vector of the unstructured data, a relationship network between the structured and unstructured data is obtained; The weight and importance of the data are determined based on the relationship network between the structured and unstructured data and the labels of the fields. Present the data after its importance has been determined; The step of determining the weight and importance of the data based on the relationship network between the structured and unstructured data and the labels of the fields includes: The importance of the structured data is determined based on one or more of the following metrics: access volume, field sensitivity, or task level. The importance of the unstructured data is determined based on one or more of the following metrics: semantic content quality, usage breadth, number of citations, or temporal activity.

2. The method according to claim 1, characterized in that, The process of fusing the data graph of the structured data and the semantic vector of the unstructured data to obtain the relationship network between the structured data and the unstructured data includes: Analyze the data graph of structured data and the semantic vector of unstructured data, and integrate the unstructured data into the data graph of structured data according to the relationships between the data; The relationships between the data include: relationships between structured data and relationships between unstructured data and tables; relationships between unstructured data and tasks; relationships between unstructured data and user actions; semantic similarity relationships between unstructured data and unstructured data; and multiple relationships among the coupling relationships between data.

3. The method according to claim 1, characterized in that, The access volume of the structured data is determined by the relationships between the tables established in the first time period, the relationships between the tables and tasks, and the relationships between the tables and user operation behaviors. The field sensitivity is determined by the relationship between the table and the fields and the labels of the fields; The task hierarchy is determined by the number of tasks that depend on it.

4. The method according to claim 2, characterized in that, The semantic content quality of the unstructured data includes one or more of the following indicators: topic clarity, summary coverage, keyword richness, entity extraction accuracy, and language quality. The breadth of use is determined by the relationship between the unstructured data and the tasks and / or the relationship between the unstructured data and user actions. The number of references is determined by the relationship between the unstructured data and the table, the semantic similarity between the unstructured data and the unstructured data, or the coupling relationship between the data. The time activity level is determined by the number of times the data is cited in the third time period and the proportion of the usage breadth in the fourth time period.

5. The method according to claim 1, characterized in that, The importance of the data includes important, average, and unimportant. The display of the data after its importance has been determined includes: data showing that its importance is important, a data weight path graph, and a data weight radar graph.

6. A data processing apparatus, characterized in that, include: The data acquisition module is used to acquire data, data relationships, and data metadata, including structured data and unstructured data. The structured data processing module is used to construct a data graph based on the relationships of structured data. These relationships include those between tables and fields, between tables and tasks, between tables and user actions, and between tables themselves. The relationships between tables and tasks are determined through task scheduling information. The unstructured data processing module is used to perform semantic parsing and feature extraction on unstructured data to obtain semantic vectors of the unstructured data. The data fusion module is used to fuse the data graph of structured data and the semantic vector of unstructured data to obtain the relationship network between structured and unstructured data; The importance calculation module is used to determine the weight of the data and the importance of the data based on the labels of the network fields relating the structured and unstructured data. The data visualization module is used to display data after its importance has been determined. The step of determining the weight and importance of the data based on the relationship network between the structured and unstructured data and the labels of the fields includes: The importance of the structured data is determined based on one or more of the following metrics: access volume, field sensitivity, or task level. The importance of the unstructured data is determined based on one or more of the following metrics: semantic content quality, usage breadth, number of citations, or temporal activity.

7. The apparatus according to claim 6, characterized in that, The data fusion module is also used to parse the data graph of structured data and the semantic vector of unstructured data, and to integrate the unstructured data into the data graph of structured data according to the relationship between the data. The relationships between the data include: relationships between structured data and relationships between unstructured data and tables; relationships between unstructured data and tasks; relationships between unstructured data and user actions; semantic similarity relationships between unstructured data and unstructured data; and multiple relationships among the coupling relationships between data.

8. A computer-readable storage medium, characterized in that, It includes instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1-5.

9. A computer program product comprising computer instructions, characterized in that, When executed by a processor, the computer instructions implement the method of any one of claims 1-5.

Images