Data processing method, device and equipment

Abstract

The embodiment of the invention provides a data processing method, device and equipment. The method comprises the following steps: generating to-be-written data; determining a structure type of the to-be-written data, wherein the structure type is a structured type or an unstructured type; under the condition that the structure type is an unstructured type, performing multi-modal feature analysis on the to-be-written data, and determining feature information of the to-be-written data; the feature information is used for indicating the content attribute, the security level and the use scene of the to-be-written data; and storing the to-be-written data in a target node according to the feature information, the target node being a computing node in a data cluster managed by the server. The data processing mode provided by the invention is suitable for read-write operation of the unstructured data.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, and in particular, to a data processing method, device and equipment. BACKGROUND

[0002] In a high-performance computing scenario, a high-performance computing (HPC) cluster is a distributed data processing system for storage and computing, which is composed of multiple heterogeneous storage nodes.

[0003] A large amount of data is generated in the process of an HPC job, and a server needs to perform read and write operations on the data in the HPC cluster. In related technologies, the data in the HPC cluster can be read and written in the following manner: for structured data, the data in the HPC cluster can be classified and stored according to a preset rule. Then, the server determines a storage location of the to-be-written data according to a data type of the to-be-written data, and writes the to-be-written data into the storage location; the server can determine a storage location of the to-be-read data according to a data type of the to-be-read data, and reads the to-be-read data in the storage location. In the above process, the read and write operations are mainly performed on structured data, and cannot be applied to read and write operations on unstructured data. SUMMARY

[0004] Embodiments of the present application provide a data processing method, device and equipment, to solve the problem that unstructured data in an HPC cluster cannot be read and written.

[0005] In a first aspect, embodiments of the present application provide a data processing method applied to a server, comprising:

[0006] generating to-be-written data;

[0007] determining a structure type of the to-be-written data, the structure type being a structured type or an unstructured type;

[0008] in a case where the structure type is the unstructured type, performing multi-modal feature analysis on the to-be-written data to determine feature information of the to-be-written data; the feature information being used to indicate a content attribute, a security level and a use scenario of the to-be-written data;

[0009] storing the to-be-written data in a target node according to the feature information, the target node being a computing node in a data cluster managed by the server.

[0010] The embodiment of the application realizes fine and intelligent management of unstructured data writing, guarantees the security and scene adaptability of data storage, and improves the overall efficiency and standardization of unstructured data writing of the server in a cluster environment.

[0011] In a possible implementation, the multi-modal feature analysis is performed on the to-be-written data to determine the feature information of the to-be-written data, including:

[0012] The data format of the to-be-written data is determined, and the data format is used to indicate the carrier type of the to-be-written data.

[0013] According to the mapping relationship between the format and the model and the data format, the feature analysis model of the to-be-written data is determined.

[0014] According to the feature analysis model and the to-be-written data, the feature information is determined.

[0015] The embodiment of the application first determines the carrier format of the unstructured to-be-written data, and then matches the exclusive feature analysis model based on the preset mapping relationship between the format and the model to extract the feature information, thereby realizing accurate adaptation of feature analysis of unstructured data of different carrier types and improving the accuracy of determining the feature information.

[0016] In a possible implementation, according to the feature analysis model and the to-be-written data, the feature information is determined, including:

[0017] The to-be-written data is input into the feature analysis model to obtain the feature information.

[0018] The feature analysis model is obtained by training based on at least one sample data and at least one feature label, and the at least one sample data and the at least one feature label are one-to-one corresponding.

[0019] The embodiment of the application determines the feature information of the to-be-written data by the feature analysis model obtained by training, thereby improving the efficiency of determining the feature information.

[0020] In a possible implementation, according to the feature information, the to-be-written data is stored in the target node, including:

[0021] According to the feature information, the data category of the to-be-written data is determined.

[0022] According to the mapping relationship between the data category and the node and the data category, the target node is determined.

[0023] According to the mapping relationship between the feature information and the storage path and the feature information, the storage path of the to-be-written data is determined.

[0024] According to the storage path and the target node, the data to be written is stored in the target node.

[0025] The embodiment of the application determines the data category according to the feature information, selects the target node relying on the mapping relationship between the category and the cluster, and completes the data storage in combination with the feature information matching the exclusive storage path, realizes the precise adaptation of the unstructured data storage cluster and the storage path, improves the intelligentization and standardization of data storage, and lays a foundation for subsequent fast retrieval and efficient management of data.

[0026] In a possible implementation, according to the storage path and the target node, the data to be written is stored in the target node, including:

[0027] Determine the data storage protocol of the target node; the data storage protocol is used to indicate the data storage specification of the target node;

[0028] According to the data storage protocol, the data format of the data to be written is converted to obtain the converted data to be written;

[0029] According to the storage path, the converted data to be written is stored in the target node.

[0030] The embodiment of the application first determines the data storage protocol of the target node, then converts the data format according to the protocol, and finally stores the converted data in the corresponding path, solves the storage protocol compatibility problem of the HPC heterogeneous cluster, guarantees the adaptability and compliance of the unstructured data target node, ensures that the data can be successfully completed persistent storage, and improves the reliability of data storage.

[0031] In a second aspect, the application provides a data processing method applied to a server, including:

[0032] Receive a data reading request, the data reading request is used to request to read the data to be read in the data cluster managed by the server; the data cluster includes at least one computing node;

[0033] Determine the structure type of the data to be read, the structure type is a structured type or an unstructured type;

[0034] In the case where the structure type is an unstructured type, determine the feature information of the data to be read; the feature information is used to indicate the content attribute, the security level, and the use scenario of the data to be read;

[0035] According to the feature information, read the data to be read in the at least one computing node.

[0036] The embodiments of the present application determine the structure type of the data to be read in the data reading scene of the server cluster environment, perform multi-modal feature analysis on unstructured data to obtain core attribute information, and then accurately locate and read the data in the cluster according to the feature information, thereby realizing the reading operation of unstructured data and ensuring the compliance of data reading in combination with features such as security level, and improving the reading efficiency and security of unstructured data in the cluster environment.

[0037] In a possible implementation, reading the data to be read in the at least one computing node according to the feature information comprises:

[0038] According to the feature information, determining the data category of the data to be read;

[0039] According to the mapping relationship between the data category and the node and the data category, determining the reading node of the data to be read in the at least one computing node;

[0040] Determining the data reading protocol of the reading node; the data reading protocol is used to indicate the data reading specification of the reading node;

[0041] Reading the data to be read in the reading node according to the data reading protocol.

[0042] The embodiments of the present application determine the data category of the data to be read according to the feature information and match the corresponding reading cluster, and then adapt the data reading protocol of the cluster to complete the reading according to the specification, thereby realizing the accurate positioning of unstructured data in the multi-cluster, solving the reading protocol compatibility problem of the heterogeneous cluster, and ensuring the compliance and efficiency of data reading.

[0043] In a third aspect, the embodiments of the present application provide a data processing apparatus, and the apparatus comprises:

[0044] A generating module is configured to generate data to be written;

[0045] A first determining module is configured to determine the structure type of the data to be written, wherein the structure type is a structured type or an unstructured type;

[0046] A first processing module is configured to, in the case where the structure type is the unstructured type, perform multi-modal feature analysis on the data to be written to determine feature information of the data to be written; the feature information is used to indicate the content attribute, security level, and usage scenario of the data to be written;

[0047] A writing module is configured to store the data to be written in a target node according to the feature information, wherein the target node is a computing node in a data cluster managed by a server.

[0048] In a possible implementation, the first processing module is specifically configured to:

[0049] determine a data format of the data to be written, the data format being used to indicate a carrier type of the data to be written;

[0050] determine a feature analysis model of the data to be written according to the mapping relationship between the format and the model and the data format;

[0051] determine the feature information according to the feature analysis model and the data to be written.

[0052] In a possible implementation, the first processing module is specifically configured to:

[0053] input the data to be written into the feature analysis model to obtain the feature information;

[0054] The feature analysis model is obtained based on at least one sample data and at least one feature label, and the at least one sample data and the at least one feature label are one-to-one corresponding.

[0055] In a possible implementation, the writing module is specifically configured to:

[0056] determine a data category of the data to be written according to the feature information;

[0057] determine a target node according to a mapping relationship between the data category and the node and the data category;

[0058] determine a storage path of the data to be written according to a mapping relationship between the feature information and the storage path and the feature information;

[0059] store the data to be written in the target node according to the storage path and the target node.

[0060] In a possible implementation, the writing module is specifically configured to:

[0061] determine a data storage protocol of the target node, the data storage protocol being used to indicate a data storage specification of the target node;

[0062] perform conversion processing on the data format of the data to be written according to the data storage protocol to obtain converted data to be written;

[0063] store the converted data to be written into the target node according to the storage path.

[0064] In a fourth aspect, the present application provides a data processing apparatus, the apparatus comprising:

[0065] a receiving module configured to receive a data reading request, the data reading request being used to request reading data to be read in a data cluster managed by a server, the data cluster comprising at least one computing node;

[0066] The second determining module is configured to determine a structure type of the to-be-read data, the structure type being a structured type or an unstructured type.

[0067] The second processing module is configured to determine feature information of the to-be-read data when the structure type is the unstructured type, the feature information being used to indicate a content attribute, a security level, and a use scenario of the to-be-read data.

[0068] The reading module is configured to read the to-be-read data in the at least one computing node according to the feature information.

[0069] In a possible implementation, the reading module is specifically configured to:

[0070] determine a data category of the to-be-read data according to the feature information;

[0071] determine a reading node of the to-be-read data in the at least one computing node according to a mapping relationship between the data category and the node and the data category;

[0072] determine a data reading protocol of the reading node, the data reading protocol being used to indicate a data reading specification of the reading node;

[0073] read the to-be-read data in the reading node according to the data reading protocol.

[0074] In a fifth aspect, an embodiment of the present application provides a data processing device, comprising:

[0075] at least one processor; and

[0076] a memory in communication connection with the at least one processor; wherein

[0077] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method in any one of the first aspect and the method in any one of the second aspect.

[0078] In a sixth aspect, an embodiment of the present application provides a non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method in any one of the first aspect and the method in any one of the second aspect.

[0079] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method in any one of the first aspect and the method in any one of the second aspect.

[0080] The data processing method, device and equipment provided by the embodiments of the present application determine the structure type of the to-be-written data, then perform multi-modal feature analysis on unstructured data to obtain core feature information including content attributes, security levels and use scenarios, and finally store the data to the computing nodes of the server management cluster according to the feature information, which not only realizes fine attribute mining and intelligent storage adaptation of unstructured data, but also realizes the near connection of data storage and subsequent computing tasks by relying on the computing power advantage of the computing nodes, reduces the cross-node data transmission overhead, and improves the standardization, efficiency and security of data writing in the server cluster environment. BRIEF DESCRIPTION OF DRAWINGS

[0081] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0082] Figure 1 A schematic diagram of an application scenario provided by the embodiments of the present application;

[0083] Figure 2 A flowchart of a data processing method provided by the embodiments of the present application;

[0084] Figure 3 A flowchart of determining feature information of to-be-written data provided by the embodiments of the present application;

[0085] Figure 4 A flowchart of storing to-be-written data in a target node provided by the embodiments of the present application;

[0086] Figure 5 A flowchart of another data processing method provided by the embodiments of the present application;

[0087] Figure 6 A structural diagram of a data processing device provided by the embodiments of the present application;

[0088] Figure 7 A structural diagram of another data processing device provided by the embodiments of the present application;

[0089] Figure 8 A structural diagram of a data processing device provided by the embodiments of the present application.

[0090] Through the above drawings, the specific embodiments of the present application have been shown, and there will be more detailed descriptions in the following. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application by any means, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0091] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0092] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0093] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0094] To facilitate understanding, the following will be combined with... Figure 1 The application scenarios applicable to the embodiments of this application will be described.

[0095] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application. Please refer to [link / reference]. Figure 1 It includes server 101 and client 102. Server 101 is equipped with an intelligent agent module and can be the management server for at least one cluster.

[0096] In practical applications, client 102 can send a generation question to server 101. Upon receiving the generation question, server 101 generates corresponding response data through its intelligent agent module, sends the response data to client 101, and stores the response data as data to be written in at least one cluster. Client 102 can also send a data read request to server 101, requesting to read data to be read from at least one cluster. Upon receiving the data read request, server 101 reads the target data from at least one cluster and sends the target data to client 102.

[0097] In the high-performance computing (HPC) industry, with the rapid development of big data and artificial intelligence technologies, the demand for computing resources in fields such as scientific research, industrial simulation, and medical image processing is growing exponentially. HPC clusters are distributed collaborative computing systems built to meet the needs of large-scale, high-intensity computing and massive data processing. HPC clusters typically consist of multiple heterogeneous storage nodes, each of which may use different file systems, hardware architectures, or network protocols.

[0098] In practical applications, HPC jobs generate massive amounts of data during execution, such as analysis reports from scientific research experiments, intermediate data from industrial simulations, and annotation files for medical images. In some scenarios, it is necessary to read and write these data within HPC.

[0099] In related technologies, data read and write operations in HPC clusters rely on manual read / write operations or pre-defined rules. Specifically, for structured data, data in the HPC cluster can be classified and stored according to pre-defined rules. Then, the server determines the storage location of the data to be written based on its data type and writes the data to that location; similarly, the server determines the storage location of the data to be read based on its data type and reads the data from that location.

[0100] However, the aforementioned data processing methods are primarily designed for reading and writing structured data and are not suitable for reading and writing unstructured data dynamically generated by agents. Furthermore, due to the large size and complex data storage structure of HPC clusters, the efficiency of reading and writing data using these methods is low.

[0101] In this embodiment, by determining the structure type of the data to be written, multimodal feature analysis is performed on the unstructured data to obtain core feature information including content attributes, security level, and usage scenario. Finally, the data is stored in the computing nodes of the server management cluster based on the feature information. This not only realizes the refined attribute mining and intelligent storage adaptation of unstructured data, but also leverages the computing power advantage of the computing nodes to achieve close connection between data storage and subsequent computing tasks, reducing cross-node data transmission overhead and improving the standardization, efficiency, and security of data writing in the server cluster environment.

[0102] The method described in this application will now be illustrated through specific embodiments. It should be noted that the following embodiments may exist independently or in combination with each other; identical or similar content will not be repeated in different embodiments.

[0103] Figure 2This is a flowchart illustrating a data processing method provided in an embodiment of this application. Please refer to [link / reference]. Figure 2 The method may include:

[0104] S201. Generate data to be written.

[0105] The execution entity in this application embodiment can be a data processing device or a data processing apparatus installed within a data processing device. The data processing apparatus can be implemented through software or a combination of software and hardware. The data processing device can be a server.

[0106] The data to be written is the data to be stored in the cluster.

[0107] A cluster is a distributed collaborative computing system built to meet the needs of large-scale, high-intensity computing and massive data processing. A cluster includes at least one node. Each node in the at least one cluster can be a server or a virtual device.

[0108] The server is the cluster's management server, responsible for overall scheduling, status monitoring, and permission allocation of the cluster and at least one computing node within it, to ensure the cluster can operate in an orderly manner.

[0109] Optionally, an intelligent agent module is deployed on the server to interact with the client. After receiving a generation question from the client, the intelligent agent module analyzes the question and generates response data for that question. For example, assuming the generation question is to analyze experimental data, after receiving the question, the server uses the intelligent agent module to analyze the experimental data and generate an analysis report file.

[0110] The server can send response data to the client and identify that response data as data to be written.

[0111] Optionally, the server can also receive write requests from clients. Specifically, the server can receive write requests from clients to request that the data to be written be stored in the cluster.

[0112] S202. Determine the structure type of the data to be written, which can be either structured or unstructured.

[0113] The structure type indicates the data format of the data to be written, and is used to distinguish whether the data has fixed and standardized structural characteristics.

[0114] Structured data refers to data that has a fixed organizational format, clearly defined fields, and standardized logical relationships, and can be directly parsed, stored, and stored by the server according to preset rules. Examples of structured data types include JavaScript Object Notation (JSON) data.

[0115] Unstructured data refers to data that lacks a fixed organizational format, field definitions, and logical constraints, and cannot be directly parsed by a server according to preset rules. Examples of unstructured data include image data, audio data, and text data.

[0116] Optionally, a data management module can also be deployed in the server. This module stores and manages data, including data classification, indexing, and storage location management. After generating the data to be written, the agent module sends the data to the data management module. Upon receiving the data, the data management module determines the structure type of the data.

[0117] S203. When the structure type is unstructured, perform multimodal feature analysis on the data to be written to determine the feature information of the data to be written; the feature information is used to indicate the content attributes, security level, and usage scenario of the data to be written.

[0118] Multimodal feature analysis refers to the process of extracting and integrating features from different modal data such as images, audio, and text by combining the characteristics of different data carriers and calling corresponding analysis models.

[0119] Feature information refers to the core attribute data of unstructured data determined after performing multimodal analysis on the unstructured data.

[0120] Optionally, when the structure type is unstructured, multimodal feature analysis can be performed on the data to be written, and the data to be written can be analyzed in a targeted manner according to the type of the data to be written to determine the feature information of the data to be written.

[0121] Optionally, if the structure type is a structured type, the storage location of the data to be written can be determined according to preset rules, and the data to be written can be stored in that storage location.

[0122] S204. Based on the feature information, the data to be written is stored in the target node, which is a computing node in the data cluster managed by the server.

[0123] A data cluster is a distributed data processing and storage system formed by interconnecting multiple nodes with different functions through a high-speed network. It is the core carrier for servers to carry out large-scale data management. A data cluster includes at least one compute node, which can be a server or a virtual machine. For example, a data cluster can be an HPC cluster.

[0124] Optionally, the server also includes a transmission control module, which is responsible for controlling data transmission between the agent module and the file management system module, as well as between different nodes and the file management system.

[0125] After determining the characteristics of the data to be written, the file management module can send the data and its characteristics to the transmission control module. Upon receiving the data and its characteristics, the transmission control module determines the target node in at least one cluster based on the characteristics and stores the data to be written in the target node.

[0126] For example, suppose the data to be written received by the transmission control module is data A, and the characteristic information of data A is that data A is experimental data of project X. The transmission control module determines the target node as node S that stores project X based on the characteristic information of data A, and stores data A in node S.

[0127] The data processing method provided in this application determines the structure type of the data to be written, then performs multimodal feature analysis on the unstructured data to obtain core feature information including content attributes, security level, and usage scenario. Finally, based on the feature information, the data is stored in the computing nodes of the server management cluster. This not only achieves refined attribute mining and intelligent storage adaptation of unstructured data, but also leverages the computing power advantage of the computing nodes to achieve close connection between data storage and subsequent computing tasks, reducing cross-node data transmission overhead and improving the standardization, efficiency, and security of data writing in a server cluster environment.

[0128] Based on the above embodiments, the following, in conjunction with Figure 3 The detailed process of determining the characteristic information of the data to be written is explained.

[0129] Figure 3 This is a schematic diagram illustrating a process for determining feature information of data to be written, provided in an embodiment of this application. Please refer to... Figure 3 The process includes:

[0130] S301. Determine the data format of the data to be written. The data format is used to indicate the carrier type of the data to be written.

[0131] Data format refers to the specific encoding standards and organizational forms followed by the data to be written during storage and transmission; it is the concrete technical representation of the data carrier. In unstructured data scenarios, data format is specifically manifested as the proprietary format of various types of files, such as image formats, audio formats, and text formats.

[0132] Optionally, the server can determine the data format of the data to be written by using a preset format recognition algorithm (such as file header identifier parsing, format feature matching, etc.).

[0133] S302. Based on the mapping relationship between the format and the model, and the data format, determine the feature analysis model of the data to be written.

[0134] Optionally, the server stores a mapping relationship between formats and models. The server can perform matching processing on the data format according to the mapping relationship between the format and the model to determine the feature analysis model of the data to be written.

[0135] For example, assume the mapping relationship between the format and the model is shown in Table 1:

[0136] Table 1

[0137]

[0138] Assuming the data format is text, the feature analysis model for the data to be written is determined to be a text processing model.

[0139] S303. Determine the feature information based on the feature analysis model and the data to be written.

[0140] Optionally, the feature information can be determined based on the feature analysis model and the data to be written in the following way: input the data to be written into the feature analysis model to obtain the feature information; wherein, the feature analysis model is trained based on at least one sample data and at least one feature label, and at least one sample data and at least one feature label are in one-to-one correspondence.

[0141] Feature analysis models can be, for example, neural network models or large language models.

[0142] Optionally, the feature analysis model is trained based on at least one sample data and at least one feature label. During one training iteration of the feature analysis model, the server can input at least one sample data into the model, process it, and obtain the predicted feature information for each of the at least one sample data output by the model. Then, based on the predicted feature information of each of the at least one sample data and at least one feature label, a loss value is calculated, and the model parameters of the feature analysis model are adjusted according to the calculated loss value, thus completing one round of training.

[0143] The server can train the feature analysis model in one or more rounds until the training termination condition is met, at which point the training process stops, and the trained feature analysis model is obtained. The training termination condition can be set according to actual needs; for example, it can be set that the loss value is less than or equal to a preset loss value, or that the training iterations reach a preset number, etc. The trained feature analysis model has the ability to process the data to be written and determine the feature information of the data to be written.

[0144] In this embodiment, the server determines the format of the data to be written and matches it with the corresponding feature analysis model, achieving accurate adaptation between the model and the data carrier type and avoiding feature extraction deviations caused by model mismatch. At the same time, the feature analysis model is trained based on one-to-one corresponding sample data and feature labels, and has the ability to accurately identify and map data features. Then, the data to be written is input into the model to obtain feature information. A rigorous feature extraction link is constructed from the two core links of model adaptation and model training, which improves the accuracy of determining the feature information of unstructured data.

[0145] Based on the above embodiments, the following, in conjunction with Figure 4 This section describes the process of storing the data to be written into the target node.

[0146] Figure 4 This is a schematic diagram illustrating a process for storing data to be written in a target node, as provided in an embodiment of this application. Figure 4 As shown, the process includes:

[0147] S401. Determine the data category of the data to be written based on the feature information.

[0148] Data categories are macro-level classifications of the data to be written, used to indicate the business domain of the data. For example, a data category could be the project to which the data belongs.

[0149] Optionally, the server can parse and match the feature information according to preset rules to determine the data category of the data to be written.

[0150] S402. Determine the target node based on the mapping relationship between data categories and nodes, and the data category.

[0151] The server stores a mapping relationship between data categories and nodes. Based on this mapping, the server can match data categories to determine the target node.

[0152] For example, suppose at least one node includes node 1, node 2, and node 3, and the mapping relationship between data categories and nodes is shown in Table 2:

[0153] Table 2

[0154]

[0155] Assuming the data category is item Y, then the target node is determined to be node 2.

[0156] S403. Determine the storage path of the data to be written based on the mapping relationship between feature information and storage path, as well as the feature information.

[0157] In some embodiments, after determining the target node, the server also needs to determine the storage path of the data to be written on the target node.

[0158] Optionally, the server stores a mapping relationship between feature information and storage path. The feature information can be matched and processed through the mapping relationship between feature information and storage path to determine the storage path of the data to be written.

[0159] S404. Based on the storage path and target node, store the data to be written in the target node.

[0160] Optionally, depending on the storage path and the target node, the method for storing the data to be written in the target node can be as follows: determine the data storage protocol of the target node; the data storage protocol is used to indicate the data storage specifications of the target node; convert the data format of the data to be written according to the data storage protocol to obtain the converted data to be written; and store the converted data to be written in the target node according to the storage path.

[0161] Data storage protocols are standardized technical agreements pre-defined by target nodes to regulate data access, storage, and retrieval; they are the core set of rules for node data management. The content of data storage protocols typically covers data format requirements, encoding standards, encryption standards, and data verification mechanisms, such as specific compression protocols for image data and character encoding protocols for text data. These protocols serve as the core reference for servers to adapt data to the node storage environment.

[0162] The converted data to be written refers to the data to be written that has undergone format conversion processing to meet the data storage requirements of the target node.

[0163] Since at least one node in the data cluster is heterogeneous, the data storage protocols of each node may differ. Therefore, the data storage protocol of the target node can be determined. Then, the data format of the data to be written is converted according to the target node's data storage protocol, ensuring that the converted data fully conforms to the target node's storage specifications. For example, format encoding conversion and character encoding adjustment can be performed on the data to be written.

[0164] Finally, the converted data to be written is stored in the target node according to the storage path.

[0165] It should be noted that data migration between different nodes can also be achieved using the methods described above.

[0166] In this embodiment, the category of the data to be written is first determined based on feature information. Then, the target node is accurately selected based on the mapping relationship between the data category and the cluster. Simultaneously, the specific storage path is clarified by combining the mapping relationship between feature information and storage path, thus constructing a precise matching link from data attributes to storage location. Subsequently, by determining the data storage protocol of the target node and performing targeted conversion of the data format, the storage protocol compatibility problem of heterogeneous cluster nodes is solved. Finally, the converted data is stored in the designated path. The above method realizes intelligent management and control of the entire link of unstructured data from classification and definition to node adaptation and compliant storage, ensuring the accuracy, standardization, and compatibility of data storage.

[0167] The above embodiments mainly describe the storage process of the data to be written. Below, in conjunction with... Figure 5 The process of reading data is explained in detail.

[0168] Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of this application. Figure 5 As shown, the process includes:

[0169] S501, Receive a data read request. The data read request is used to request the reading of data to be read from the data cluster managed by the server. The data cluster includes at least one computing node.

[0170] A data read request is used to request the reading of data from a server-managed cluster. For example, a data read request might be used to request the reading of business data for project A from a server-managed data cluster. The data to be read is stored in the server-managed data cluster and is the data the client needs to read.

[0171] A data cluster is a distributed data processing and storage system formed by interconnecting multiple nodes with different functions through a high-speed network. It is the core carrier for servers to carry out large-scale data management. A data cluster includes at least one compute node, which can be a server or a virtual machine. For example, a data cluster can be an HPC cluster.

[0172] The server is the management server for the data cluster, responsible for overall scheduling, status monitoring, and permission allocation of the data cluster and at least one computing node in the data cluster, so as to ensure that the data cluster can operate in an orderly manner.

[0173] Optionally, when a client needs to read data, it can send a data read request to the server.

[0174] S502. Determine the structure type of the data to be read, which can be either structured or unstructured.

[0175] The structure type indicates the data format of the data to be read, and is used to distinguish whether the data has fixed and standardized structural characteristics.

[0176] Structured data refers to data that has a fixed organizational format, clearly defined fields, and standardized logical relationships, and can be directly parsed, stored, and processed by the server according to preset rules. Examples of structured data include JavaScript Object Notation (JSON) data.

[0177] Unstructured data refers to data that lacks a fixed organizational format, field definitions, and logical constraints, and cannot be directly parsed by the server according to preset rules. Examples of unstructured data include image data, audio data, and text data.

[0178] Optionally, the data read request may include the structure type of the data to be read. The server can parse the data read request to determine the structure type of the data to be read in the data read request.

[0179] S503. When the structure type is unstructured, determine the feature information of the data to be read; the feature information is used to indicate the content attributes, security level, and usage scenario of the data to be read.

[0180] Feature information refers to the core attribute data of unstructured data.

[0181] Optionally, the data read request may include characteristic information of the data to be read. The server can parse the data read request to determine the characteristic information of the data to be read.

[0182] S504. Based on the feature information, read the data to be read in at least one computing node.

[0183] Based on the feature information, the method for reading data to be read in at least one cluster can be as follows: Based on the feature information, determine the data category of the data to be read; based on the mapping relationship between the data category and the node, and the data category, determine the reading node of the data to be read in at least one computing node; determine the data reading protocol of the reading node; the data reading protocol is used to indicate the data reading specifications of the reading node; and read the data to be read in the reading node according to the data reading protocol.

[0184] Data categories are macro-level classifications of the data to be read, used to indicate the business domain of the data. For example, a data category could be the project to which the data belongs.

[0185] Optionally, the server can parse and match the feature information according to preset rules to determine the data category of the data to be read.

[0186] The server stores a mapping relationship between data categories and nodes. Based on this mapping, the server can match data categories and determine the appropriate nodes to read.

[0187] The data reading protocol is a standardized technical agreement between the reading node and the data requester in the data reading process. It is a core technical specification to ensure that data reading operations are carried out in an orderly, safe and efficient manner in a cluster environment. It is suitable for batch retrieval of structured data as well as accurate reading of unstructured image, audio and text data.

[0188] The server can obtain the target data from the reading node according to the data reading protocol. The target data is the data to be read stored in the reading node.

[0189] The data processing procedure provided in the embodiments of this application

[0190] Figure 6 This is a schematic diagram of the data processing apparatus provided in an embodiment of this application. Please refer to... Figure 6 The data processing device 10 may include:

[0191] Generation module 11 is used to generate data to be written;

[0192] The first determining module 12 is used to determine the structure type of the data to be written, which is either a structured type or an unstructured type.

[0193] The first processing module 13 is used to perform multimodal feature analysis on the data to be written when the structure type is unstructured, and to determine the feature information of the data to be written; the feature information is used to indicate the content attributes, security level, and usage scenario of the data to be written.

[0194] The writing module 14 is used to store the data to be written in the target node according to the feature information. The target node is a computing node in the data cluster managed by the server.

[0195] In one possible implementation, the first processing module 13 is specifically used for:

[0196] Determine the data format of the data to be written. The data format is used to indicate the carrier type of the data to be written.

[0197] Based on the mapping relationship between the format and the model, and the data format, determine the feature analysis model of the data to be written;

[0198] Based on the feature analysis model and the data to be written, the feature information is determined.

[0199] In one possible implementation, the first processing module 13 is specifically used for:

[0200] The data to be written is input into the feature analysis model to obtain feature information;

[0201] The feature analysis model is trained based on at least one sample data and at least one feature label, and there is a one-to-one correspondence between the at least one sample data and at least one feature label.

[0202] In one possible implementation, the writing module 14 is specifically used for:

[0203] Based on the feature information, determine the data category of the data to be written;

[0204] Based on the mapping relationship between data categories and nodes, and the data categories themselves, determine the target node;

[0205] Based on the mapping relationship between feature information and storage path, and the feature information itself, determine the storage path of the data to be written.

[0206] Based on the storage path and target node, the data to be written is stored in the target node.

[0207] In one possible implementation, the writing module 14 is specifically used for:

[0208] Determine the data storage protocol of the target node; the data storage protocol is used to indicate the data storage specifications of the target node;

[0209] The data format of the data to be written is converted according to the data storage protocol to obtain the converted data to be written.

[0210] Based on the storage path, the converted data to be written is stored in the target node.

[0211] The data processing device 10 provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0212] Figure 7 This is a schematic diagram of another data processing apparatus provided in an embodiment of this application. Please refer to... Figure 7 The data processing device 20 may include:

[0213] Receiving module 21 is used to receive data read requests, which are requests to read data to be read from a data cluster managed by the server; the data cluster includes at least one computing node;

[0214] The second determining module 22 is used to determine the structure type of the data to be read, which is either a structured type or an unstructured type.

[0215] The second processing module 23 is used to determine the feature information of the data to be read when the structure type is unstructured; the feature information is used to indicate the content attributes, security level, and usage scenario of the data to be read.

[0216] The reading module 24 is used to read the data to be read in at least one computing node based on the feature information.

[0217] In one possible implementation, the reading module 24 is specifically used for:

[0218] Based on the feature information, determine the data category of the data to be read;

[0219] Based on the mapping relationship between data categories and nodes, and the data category, determine the reading node for the data to be read in at least one computing node;

[0220] The data reading protocol of the read node is determined; the data reading protocol is used to indicate the data reading specifications of the read node.

[0221] The data to be read is read from the reading node according to the data reading protocol.

[0222] The data processing device 20 provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0223] Figure 8This is a schematic diagram of the data processing device provided in an embodiment of this application. Please refer to... Figure 8 The data processing device 30 may include a memory 31 and a processor 32. Exemplarily, the memory 31 and the processor 32 are interconnected via a bus 33.

[0224] Memory 31 is used to store program instructions;

[0225] The processor 32 is used to execute the program instructions stored in the memory, so that the data processing device 30 performs the method shown in the above method embodiment.

[0226] The data processing device 30 provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0227] This application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above-described method when executed by a processor.

[0228] This application embodiment may also provide a computer program product, including a computer program that, when executed by a processor, can implement the above-described method.

[0229] All or part of the steps in the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above-described method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), random access memory (RAM), flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disc, and any combination thereof.

[0230] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0231] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0232] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0233] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0234] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

Claims

1. A data processing method, characterized in that, Applied to servers, including: Generate data to be written; Determine the structure type of the data to be written, wherein the structure type is either structured or unstructured; When the structure type is the unstructured type, multimodal feature analysis is performed on the data to be written to determine the feature information of the data to be written; the feature information is used to indicate the content attributes, security level, and usage scenario of the data to be written. Based on the feature information, the data to be written is stored in the target node, which is a computing node in the data cluster managed by the server.

2. The method according to claim 1, characterized in that, Multimodal feature analysis is performed on the data to be written to determine the feature information of the data to be written, including: Determine the data format of the data to be written, wherein the data format is used to indicate the carrier type of the data to be written; Based on the mapping relationship between the format and the model, and the data format, determine the feature analysis model of the data to be written; The feature information is determined based on the feature analysis model and the data to be written.

3. The method according to claim 2, characterized in that, Based on the feature analysis model and the data to be written, the feature information is determined, including: The data to be written is input into the feature analysis model to obtain the feature information; The feature analysis model is trained based on at least one sample data and at least one feature label, and the at least one sample data and the at least one feature label are in one-to-one correspondence.

4. The method according to any one of claims 1-3, characterized in that, Based on the aforementioned feature information, storing the data to be written in the target node includes: Based on the feature information, determine the data category of the data to be written; The target node is determined based on the mapping relationship between data categories and nodes, and the data categories themselves. Based on the mapping relationship between feature information and storage path, and the feature information, the storage path of the data to be written is determined; Based on the storage path and the target node, the data to be written is stored in the target node.

5. The method according to claim 4, characterized in that, Based on the storage path and the target node, the data to be written is stored in the target node, including: Determine the data storage protocol of the target node; the data storage protocol is used to indicate the data storage specifications of the target node; The data format of the data to be written is converted according to the data storage protocol to obtain the converted data to be written. According to the storage path, the converted data to be written is stored in the target node.

6. A data processing method, characterized in that, Applied to a server, the method includes: The server receives a data read request, which requests the reading of data to be read from the data cluster managed by the server; the data cluster includes at least one computing node. Determine the structure type of the data to be read, wherein the structure type is either structured or unstructured; When the structure type is the unstructured type, the feature information of the data to be read is determined; the feature information is used to indicate the content attributes, security level, and usage scenario of the data to be read. Based on the feature information, the data to be read is read in the at least one computing node.

7. The method according to claim 6, characterized in that, Based on the feature information, reading the data to be read in the at least one computing node includes: Based on the feature information, the data category of the data to be read is determined; Based on the mapping relationship between data categories and nodes, and the data categories, determine the reading node for the data to be read in the at least one computing node; The data reading protocol of the reading node is determined; the data reading protocol is used to indicate the data reading specifications of the reading node; The data to be read is read in the reading node according to the data reading protocol.

8. A data processing apparatus, characterized in that, The device includes: The generation module is used to generate the data to be written. The first determining module is used to determine the structure type of the data to be written, wherein the structure type is a structured type or an unstructured type; The first processing module is used to perform multimodal feature analysis on the data to be written when the structure type is the unstructured type, and determine the feature information of the data to be written; the feature information is used to indicate the content attributes, security level, and usage scenario of the data to be written. The writing module is used to store the data to be written in the target node according to the feature information, wherein the target node is a computing node in the data cluster managed by the server.

9. A data processing apparatus, characterized in that, The device includes: A receiving module is configured to receive a data read request, the data read request being used to request the reading of data to be read from the data cluster managed by the server; the data cluster includes at least one computing node; The second determining module is used to determine the structure type of the data to be read, wherein the structure type is a structured type or an unstructured type; The second processing module is used to determine the feature information of the data to be read when the structure type is the unstructured type; the feature information is used to indicate the content attributes, security level, and usage scenario of the data to be read. A reading module is used to read the data to be read in the at least one computing node based on the feature information.

10. A data processing device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

Images