Data storage method, computer device, and computer program product

Abstract

This application is applicable to the field of computer technology, and provides a data storage method, a computer device, and a computer program product, including: obtaining original data from different data sources, where the original data of each data source is used to describe the same specified object or the same data usage purpose; defining a general tensor class based on the specified object or the data usage purpose, and generating tensor instances of the original data of each data source based on the general tensor class; encoding each of the tensor instances based on a preset common identifier to obtain the encoding results of each of the tensor instances; and storing the encoding results of each tensor instance in a specified storage space. This application can achieve the persistent storage of data in tensor form, reduce data storage redundancy, and improve data storage efficiency.

Description

Technical Field

[0001] This application belongs to the field of data storage, and particularly relates to a data storage method, a computer device, and a computer program product. Background Art

[0002] Tensors are the common language in the field of artificial intelligence. In the prior art, tensors have never been persistently stored in any form. Instead, they are processed through intermediate representations specific to the application domain, usually existing in synthetic, abstract mathematical expressions rather than representing raw data with practical significance. This approach means that tensors of raw data are usually temporary, dynamically created and destroyed during the calculation process without being stored for a long time. In this way, the raw data is usually stored in its original format, resulting in problems such as data redundancy, limited random access, low compression efficiency, and difficulty in interoperating across multiple application domains, especially in the fields of time series data (such as InfluxDB) and genomic data. Therefore, there is an urgent need for an efficient data storage method that can achieve persistent storage of tensors. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data storage method, a computer device, and a computer program product, which can achieve persistent storage of raw data in the form of tensors, reduce data storage redundancy, and improve data storage efficiency.

[0004] The first aspect of the embodiments of this application provides a data storage method, including:

[0005] Obtain raw data from different data sources, where the raw data of each data source is used to describe the same specified object or the same data usage purpose;

[0006] Define a general tensor class based on the specified object or the data usage purpose, and generate tensor instances of the raw data of each data source based on the general tensor class;

[0007] Encode each tensor instance based on a preset common identifier to obtain the encoding result of each tensor instance;

[0008] Store the encoding results of each tensor instance in a specified storage space.

[0009] In one implementation manner of the first aspect, the generating tensor instances of the raw data of each data source based on the general tensor class includes:

[0010] Create and obtain the collation information of the general tensor class of the specified object, where the collation information is used to describe the format and semantics of each field in the general tensor class;

[0011] Based on the collated information and the original data of each of the data sources, instantiation is performed for the general tensor class to obtain tensor instances of the original data of each of the data sources.

[0012] In one implementation manner of the first aspect, encoding the tensor instances based on a preset common identifier to obtain encoding results of the tensor instances includes:

[0013] Obtain a string table matching the tensor type of the general tensor class, where the string table is used to describe the correspondence between common strings and common identifiers;

[0014] Read the target identifier of the common string associated with the tensor instance from the string table;

[0015] Encode the tensor instances based on the target identifiers to obtain encoding results of the tensor instances.

[0016] In one implementation manner of the first aspect, encoding the tensor instances based on the target identifiers to obtain encoding results of the tensor instances includes:

[0017] If the format type of the original data is a key-value pair type, obtain the target identifier of each key or value in the original data;

[0018] Encode the tensor instances based on the target identifiers of the keys or the values in a preset order to obtain encoding results of the tensor instances.

[0019] In one implementation manner of the first aspect, if the format type of the original data is a variable-length string type, obtain the string length of each variable string in the original data;

[0020] Encode the tensor instances based on the target identifiers and the string lengths of the variable strings to obtain encoding results of the tensor instances.

[0021] In one implementation manner of the first aspect, before storing the encoding results of the tensor instances in a specified storage space, the method further includes:

[0022] For each tensor instance, determine the distance between the targeted tensor instance and other tensor instances;

[0023] Based on the distance, perform tensor deduplication to obtain the tensor instances to be stored.

[0024] In one implementation manner of the first aspect, for each tensor instance, determining the distance between the targeted tensor instance and other tensor instances includes:

[0025] For each of the tensor instances, determine the common columns between the targeted tensor instance and other tensor instances;

[0026] Normalize each of the common columns respectively to obtain the normalization results of each of the common columns;

[0027] Based on a preset distance determination method and the normalization results of each of the common columns, determine the similarity distance between the targeted tensor instance and the other tensor instances.

[0028] In one implementation manner of the first aspect, the determining, for each of the tensor instances, the distance between the targeted tensor instance and other tensor instances includes:

[0029] For each of the tensor instances, respectively determine the unique columns of the targeted tensor instance and other tensor instances;

[0030] Normalize each of the unique columns respectively to obtain the normalization results of each of the unique columns;

[0031] Based on the normalization results of each of the unique columns, determine the difference degree distance between the targeted tensor instance and the other tensor instances.

[0032] The second aspect of the embodiments of the present application provides a data storage device, including:

[0033] An acquisition module, configured to acquire raw data from different data sources, and the raw data of each data source is used to describe the same specified object or the same data usage purpose;

[0034] A generation module, configured to define a general tensor class based on the specified object or the data usage purpose, and generate tensor instances of the raw data of each data source based on the general tensor class;

[0035] An encoding module, configured to encode each of the tensor instances based on a preset common identifier to obtain the encoding results of each of the tensor instances;

[0036] A storage module, configured to store the encoding results of each of the tensor instances in a specified storage space.

[0037] The third aspect of the embodiments of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the computer device implements the method as described in the first aspect above.

[0038] The fourth aspect of the embodiments of the present application provides a computer-readable storage medium storing a computer program, which when executed by a processor implements the method described in the first aspect above.

[0039] The fifth aspect of the embodiments of the present application provides a computer program product including a computer program, which when run causes the method described in the first aspect above to be executed.

[0040] In the first aspect of the embodiments of the present application, by obtaining raw data describing the same specified object or the same data usage purpose from different data sources, generating a standardized tensor instance based on the general tensor class of the specified object, and then encoding and storing the tensor instance, the consistency, efficiency, and interoperability of data processing are significantly improved. Specifically, it converts the raw data describing the same specified object or the same data usage purpose into a tensor instance with a unified format and semantic constraints, eliminating data format differences; encodes the tensor instance using a preset common identifier, enhancing data consistency and parsability and facilitating cross-platform exchange; and stores the encoded instance in a specified space to achieve persistent storage of data in tensor form, reducing data storage redundancy and improving data storage efficiency.

[0041] It can be understood that the beneficial effects of the second to fifth aspects above can be referred to the relevant descriptions in the first aspect above, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0043] Figure 1 is a schematic flowchart of the implementation of the data storage method provided by the embodiments of the present application;

[0044] Figure 2 is an example diagram of the tensor instance provided by the embodiments of the present application;

[0045] Figure 3 is a schematic flowchart of the implementation of the instantiation method of the general tensor class provided by the embodiments of the present application;

[0046] Figure 4 is a schematic flowchart of the implementation of the encoding method of the tensor instance provided by the embodiments of the present application;

[0047] Figure 5 is an example diagram of the encoding result of the tensor instance provided by the embodiments of the present application;

[0048] Figure 6 It is a schematic diagram of the style example of FASTQ file data provided by an embodiment of the present application;

[0049] Figure 7 It is another schematic diagram of the encoding result of the tensor instance provided by an embodiment of the present application;

[0050] Figure 8 It is a flowchart of the implementation of the tensor deduplication method provided by an embodiment of the present application;

[0051] Figure 9 It is a flowchart of the implementation of the method for determining the similarity distance provided by an embodiment of the present application;

[0052] Figure 10 It is a schematic diagram of the similarity distance provided by an embodiment of the present application;

[0053] Figure 11 It is a flowchart of the implementation of the method for determining the difference distance provided by an embodiment of the present application;

[0054] Figure 12 It is a schematic diagram of the difference distance provided by an embodiment of the present application;

[0055] Figure 13 It is a schematic diagram of the data storage device provided by an embodiment of the present application;

[0056] Figure 14 It is a schematic diagram of the computer device provided by an embodiment of the present application. Detailed implementation manners

[0057] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system architectures and technologies are presented to thoroughly understand the embodiments of the present application. However, those skilled in the art should clearly understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from interfering with the description of the present application.

[0058] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0059] It should also be understood that the term "and / or" used in the specification of the present application and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0060] As used in the specification and claims of this application, the term "if" may be construed, depending on the context, as "when", "once", "in response to determining", or "in response to detecting". Similarly, the phrases "if determined" or "if [the described condition or event] is detected" may be construed, depending on the context, as meaning "once determined", "in response to determining", "once [the described condition or event] is detected", or "in response to detecting [the described condition or event]".

[0061] In addition, in the description of the specification and claims of this application, the terms "first", "second", "third", etc. are used only for distinguishing descriptions and cannot be construed as indicating or implying relative importance.

[0062] Reference to "one embodiment" or "some embodiments" or the like described in the specification of this application means that a specific feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of this application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0063] Currently, tensors, as the main data structure in artificial intelligence, are usually not persistently stored but are processed through intermediate representations specific to the application domain. This leads to problems such as data redundancy, limited random access, low compression efficiency, and difficulty in interoperating across multiple application domains. Especially in the fields of time-series data (such as data stored in InfluxDB) and genomic data (such as FASTQ, BAM, SAM, or BAI files), genomic data is a type of biological sequence data. Specifically, InfluxDB stores data in JSON format, resulting in large file sizes and limited random access; while BAI-type files of genomic data are constructed retrospectively, using LZ variants for compression, without considering the data efficiency issues between FP32 (32-bit floating-point type) and FP8 (8-bit floating-point type), and there is a read amplification phenomenon. In addition, the prior art lacks generality, cannot solve the problems of time-series and genomic data simultaneously, and fails to fully implement the distance-based tensor deduplication strategy, wasting storage space and computing resources. The difficulty in integrating multi-source heterogeneous data, poor dynamic adaptability, and lack of effective metadata management further limit the flexibility and efficiency of the system.

[0064] Based on this, an embodiment of the present application provides a data storage method, which can persistently store the original data from different data sources in the form of tensors with a unified format, reduce data storage redundancy, and improve data storage efficiency.

[0065] In one embodiment, as Figure 1 shown, a data storage method is provided. Taking the application of this method to a computer device as an example, it includes the following steps S101 to S104:

[0066] Step S101, obtain the original data from different data sources, and the original data of each of the data sources is used to describe the same specified object or the same data usage purpose.

[0067] In applications, original data from multiple different sources is used to describe the same specified object or the same data usage purpose. For example, the specified object can be the abnormal information generated by each node in the data center cluster, and the specified object can also be the meteorological information generated by multiple sensors in a weather station, etc. In these scenarios, for the specified object, the original data generated by multiple data sources can be associated. Taking the specified object as meteorological data as an example, there are multiple models of weather station sensors, and each sensor can generate abnormal data. Different sensors are different sources of the abnormal data. In any weather station, the meteorological data of the weather station is characterized by the original data collected by multiple different sensors. The original data generated by different data sources is used to describe this specified object.

[0068] Step S102, define a general tensor class based on the specified object or the data usage purpose, and generate tensor instances of the original data of each of the data sources based on the general tensor class.

[0069] In an application, for the same specified object or the same data usage purpose, a general tensor class is predefined. The general tensor class is used to describe the specified object or data usage purpose and includes a description format for the data that constrains the specified object or data usage purpose. In this way, the specified object or data usage purpose can be described in a unified format. Since the specified object is characterized by the raw data from multiple different data sources and there are differences between the raw data of each data source, in order to ensure that the raw data from multiple different data sources conforms to a unified format, the computer device defines a general tensor class for each specified object to standardize the raw data from different data sources associated with the specified object, obtaining target data in a unified format. The general tensor class determines the basic structure and usage of the tensor used to describe the specified object. In the general tensor class, tensor fields for describing the specified object are defined, including information such as name, type, and unit. A tensor instance is always contained in a node. A node is a container for tensor values and represents a specific physical or logical entity (such as a computer, storage element, or genome scanning device). Each node can hold and process one or more tensor instances, and the interpretation of the tensor instance header depends on the node (e.g., through the _ntyp field). In addition, a tensor class can coexist in multiple nodes, and there may be differences between these instances, specifically specified by the instance header. The tensor type is not restricted and can be very diverse. The node to which a tensor instance belongs can be distinguished through the header information of the tensor instance. For the same tensor class, tensor instances on different nodes may be different, and these differences are specified through the header information of the tensor instance.

[0070] In an application, for the raw data from different data sources of a specified object or the same data usage purpose, the computer device can standardize the raw data from different data sources based on the general tensor class of the specified object or the same data usage purpose, that is, perform an instantiation operation on the raw data from different data sources for the general tensor class to obtain tensor instances of the raw data from each data source relative to the general tensor class. A tensor instance is a specific implementation of a tensor class and contains actual data values and metadata. Each tensor instance always includes the general tensor class to which it belongs, ensuring the consistency and relevance between the instance and its definition.

[0071] In the general tensor class, internal elements and optional elements can be distinguished by different markers. Among them, internal elements: elements starting with "_" are regarded as internal elements and are usually used to represent metadata or other auxiliary information. These elements are part of the tensor definition and can be interpreted and used by AI programs. Optional elements: elements not starting with "_" are considered optional elements. They may contain additional information or annotations but will not be directly interpreted by AI programs. The presence or absence of these optional elements does not affect the core function of the tensor.

[0072] Step S103: Encode each of the tensor instances based on a preset common identifier to obtain the encoding results of each of the tensor instances.

[0073] In applications, the common identifier is used to identify common strings in the original data. The original data from different data sources has the characteristics of large data volume and large storage resource occupation. To achieve low storage space storage in tensor form, the common identifier is used to encode the common strings in the tensor instances, and the storage space required for the encoded tensor instances is lower than that of the tensor instances before encoding.

[0074] Exemplarily, as Figure 2 shown, a tensor instance containing data of multiple weather stations. Among them, all temperature readings are converted to degrees Celsius, wind_speed_kph readings are converted to kilometers per hour, and other metrics are also standardized accordingly. Fields with the prefix "_" such as _timestamp and _sensor_id are internal elements, indicating that they are metadata or key identifiers. Fields without the prefix "_" such as temperature and humidity are optional elements, indicating that they are actual measurement values for further analysis. Among them, the common identifier "1002" is used to represent the common strings "temperature" or "temperature"; the common identifier "1004" is used to represent the common strings "wind speed" or "wind_speed_kph". Therefore, at least the "temperature" and "wind speed" in the tensor instance can be encoded through the common identifier "1002" and the common identifier "1004" to obtain an encoding result with a lower required storage space.

[0075] Step S104: Store the encoding results of each of the tensor instances in a specified storage space.

[0076] In applications, the encoding results of the tensor instances of each piece of original data are stored in a specified storage space to achieve the persistent storage of the original data in tensor form.

[0077] In this embodiment, by obtaining the original data from different data sources and generating standardized tensor instances based on the general tensor class of the specified object, and then encoding and storing these tensor instances, the consistency, efficiency, and interoperability of data processing can be significantly improved. First, it is ensured that all the original data describing the same specified object, regardless of its source, is converted into tensor instances that follow a unified format and semantic constraints, thus eliminating the obstacles caused by data format differences. Second, each tensor instance is encoded using a preset identifier, further enhancing the consistency and parsability of data representation, making data exchange across platforms and systems smoother. Finally, the encoded tensor instances are stored in a specified storage space, enabling the persistent storage of the original data in the form of tensors, reducing data storage redundancy, and improving data storage efficiency.

[0078] In one embodiment, as Figure 3 shown, the implementation process of step 102 includes the following steps S201 to step S202:

[0079] Step S201, create and obtain the collation information of the general tensor class of the specified object, where the collation information is used to describe the format and semantics of each field in the general tensor class.

[0080] In applications, the fields in the general tensor class are associated with corresponding collation information, and the collation information is used to describe the clear semantics and format requirements of the fields. The collation information is stored in JSON format to ensure its easy parsing and use. JSON is a lightweight data exchange format widely used between various programming languages and platforms. Each general tensor class has its specific collation information in JSON format, which ensures that the collation information between the general tensor classes of different specified objects will not be confused and can be customized according to the different requirements of the classes.

[0081] Step S202, based on the collation information and the original data of each data source, perform instantiation for the general tensor class to obtain tensor instances of the original data of each data source.

[0082] In applications, based on the general tensor class, select the actual value of each field from the original data of each data source, and combine the collation information of the field to perform standardization processing on the actual value of the field to obtain the target value in the standard format, and finally obtain the tensor instances of the original data of each data source with respect to the general tensor class.

[0083] In this embodiment, by generating the original data tensor instances of each data source based on the general tensor class of the specified object, the consistency and efficiency of data processing can be significantly improved. Specifically, first, the collation information describing the format and semantics of each field in the tensor class is obtained to ensure a clear and unified understanding of the data structure. Then, using this collation information and combining with the original data of each data source, the instantiation operation for the general tensor class is performed to generate standardized tensor instances. This not only simplifies the integration process of multi-source heterogeneous data, ensures the consistency of data from different sources in format and semantics, but also enhances the flexibility and reliability of data processing. With a unified data representation form, it can more efficiently support cross-platform data exchange and analysis tasks, and reduce parsing errors caused by data format or semantic differences.

[0084] In one embodiment, as Figure 4 shown, the implementation process of step 103 includes the following steps S301 to S303:

[0085] Step S301, obtain a string table that matches the tensor type of the general tensor class, where the string table is used to describe the correspondence between common strings and common identifiers.

[0086] In applications, in order to reduce the storage space occupied by tensor instances, the computer device can compress each tensor instance of the general tensor class through a string table adapted to the general tensor class to reduce the storage space requirements for tensor instances. The string table is used to describe the correspondence between strings and identifiers. By mapping common string values to unique identifiers (usually integer IDs), the string table avoids storing these strings repeatedly in each tensor instance. Thus, when storing tensor instances, the strings in the tensor instances can be replaced with the corresponding identifiers, which can effectively reduce the storage space requirements. The working principle of the string table is as follows: when writing a tensor instance, the computer device checks whether the string in the tensor instance already exists in the string table. If it exists, only the identifier ID corresponding to the string is stored; if it does not exist, the new string is added to the string table and a new identifier ID is assigned. In this way, the storage space is significantly reduced, especially when there are many repeated strings. The query efficiency is improved because the corresponding string can be quickly found through the ID.

[0087] Step S302, read the target identifier of the common string associated with the tensor instance from the string table.

[0088] In applications, for each tensor instance of the specified object, the target identifier corresponding to each string in the tensor instance is read from the string table corresponding to the general tensor class of the specified object.

[0089] Step S303: Encode each of the tensor instances based on the respective target identifiers to obtain the encoding results of the tensor instances.

[0090] In an application, based on the target identifier corresponding to each string, replace the string in the tensor instance and perform encoding in a preset order to generate a target tensor instance.

[0091] In this embodiment, by encoding the tensor instances based on a preset common identifier, more efficient and consistent data processing can be achieved. Specifically, first, obtain a string table that matches the tensor type, and this string table details the correspondence between strings and identifiers. Then, read the target identifier of the string in each tensor instance from this string table. Finally, encode each tensor instance based on these target identifiers to obtain the encoded tensor instances. This method not only simplifies the data processing flow but also ensures the semantic consistency and accuracy of data from different sources. For example, when processing data involving multiple units (such as currency units USD, EUR or temperature units Celsius, Fahrenheit), it can ensure that the specific meanings of fields (such as counter, process ID, user ID, etc.) are correctly parsed and uniformly represented. This helps improve the efficiency of data processing, reduce errors caused by inconsistent units or semantics, and thereby enhance the reliability and maintainability of the system.

[0092] In one embodiment, the specific implementation process of step S303 is as follows: If the format type of the original data is a key-value pair type, obtain the target identifier of each key or value in the original data; based on the target identifiers of the keys or the values, encode each of the tensor instances in a preset order to obtain the encoding results of the tensor instances.

[0093] Exemplarily, such as Figure 5As shown, the original data is stored in the Influx database. The original data is stored in JSON format, and the format type of the original data is a key-value pair type. The keys or values in the original data usually correspond to the fields in the tensor instance. In the Influx model, the keys in the original data are often repeated, resulting in inherent data redundancy, including the following information: {"temperature": 81, "humidity": 43}. The key information is "temperature" and "humidity". The following preset string table is constructed for the key information or value information: "temperature" -> 1002, "humidity" -> 1003. Then, the target identifiers of each key or value are read from the preset character table, and the tensor instance corresponding to the original data is encoded in the preset order. The encoding result of the tensor instance is as follows: {1, 1002, 2, 81, 3, 1003, 4, 43}, where 1 in "1, 1002, 2" is the serial number in the preset order. The encoding result of the tensor instance is a column-oriented tensor. Compared with the original data, storing with a column-oriented tensor can achieve 50% compression and data auditing.

[0094] In this embodiment, when the format type of the original data is a key-value pair type, by obtaining the target identifier of each key or value and encoding based on the preset order, the consistency and efficiency of data processing can be significantly improved. Specifically, first, the identifier of each field in the tensor instance is extracted to ensure that each field has a clear and unique representation form. Then, these field identifiers are encoded in the preset order to generate a standardized encoding result. This not only simplifies the data structure for subsequent processing and analysis but also ensures a high degree of consistency in the comparison and operation between different tensor instances.

[0095] In one embodiment, if the format type of the original data is a variable-length string type, the string length of each variable string in the original data is obtained; based on the target identifier and string length of each variable string, each tensor instance is encoded to obtain the encoding result of each tensor instance.

[0096] In applications, the original data of variable-length string type can be genomic data. Genomic data is the binary encoding of gene scan data and the index of the scan encoding. Genomic data is usually in files such as FASTQ, BAM, SAM, BAI, etc. BAI files are constructed later. It does not consider the data efficiency issues between FP32 and FP8 and has a relatively large read amplification. Exemplarily, Figure 6 as shown in the style of FASTQ file data.

[0097] Exemplarily, Figure 7As shown, taking genomic data as an example of the original data, the original data is {"TAGAATACGCCAGGT \n", "ACGTATGGC\n", "TAGAATACGCCAGGT \n"}, the common string is "TAGAATACGCCAGGT", and the common identifier corresponding to the common string "TAGAATACGCCAGGT" stored in the string table is 2001. Then, based on the common strings in the string table, the tensor instances corresponding to the original data are encoded, and the encoded tensor instances are {1,15,2001,2,9,ACGTATGGC,3,15,2001}. {1, 15, 2001} is the first string, with a string length of 15 and an identifier of 2001.

[0098] In this embodiment, by encoding tensor instances based on respective target identifiers, especially for the case where the original data is of variable-length string type, this method significantly improves the efficiency and consistency of data processing. Specifically, first, the string lengths of the variable strings in each tensor instance are obtained, and then the tensor instances are encoded by combining the target identifiers and length information of these strings, thereby obtaining a standardized encoding result. This method not only ensures the effective representation of strings of different lengths but also improves the compactness of data storage and transmission and reduces redundancy.

[0099] In one embodiment, as Figure 8 shown, before storing the encoding results of each tensor instance into a specified storage space, the computer device can also perform tensor deduplication operations. The specific implementation process includes the following steps S401 to step S402:

[0100] Step S401, for each of the tensor instances, determine the distance between the targeted tensor instance and other tensor instances.

[0101] In an application, when storing tensor instances, a computer device can perform tensor deduplication based on a preset distance metric so that duplicate tensor instances only need to be stored once. The preset distance metrics include similarity distance and difference distance. Among them, the similarity distance is used to measure the similarity degree between two tensor instances, and the similarity between tensor instances is determined by identifying the number or proportion of common columns between the two tensor instances to be compared. If two tensor instances have the same data in multiple columns, then the similarity distance between them will be smaller, indicating that they have a high similarity in these columns. The similarity distance can be used to measure the common points of two datasets in specific features. The difference distance, on the other hand, focuses on the unique columns (i.e., exclusive columns) in the two tensor instances to be compared, and the uniqueness of each tensor can be measured by calculating its exclusive columns. A larger difference distance means that the two tensors differ in multiple columns, that is, they exhibit different data features in these columns. Regarding the similarity distance, first determine the common columns between two tensor instances to be compared; normalize the data of these common columns to a range from -1.0 to 1.0 to ensure that data of different scales can be compared using the same standard; apply a selected distance formula (such as Euclidean distance, Manhattan distance, etc.) to calculate the similarity distance.

[0102] Regarding the difference distance, determine the exclusive columns (unique columns) in each tensor instance; normalize the data of these exclusive columns to a range from -1.0 to 1.0; since the exclusive columns may have a greater impact on the overall distance, their weights can be adjusted according to the actual situation (weight adjustment).

[0103] In an application, a computer device determines the distance between targeted tensor instances based on a preset distance metric.

[0104] Step S402: Based on the distance, perform tensor deduplication to obtain the tensor instances to be stored.

[0105] In an application, a computer device compares the size relationship between the distance and a preset distance threshold. If the comparison result shows that the obtained distance reaches the distance threshold, it indicates that the two tensor instances associated with this distance are equivalent. Therefore, only one tensor instance needs to be retained as the tensor instance to be stored.

[0106] In this embodiment, by introducing the concepts of similarity distance and dissimilarity distance, the relationship between column-oriented tensors can be more comprehensively understood and analyzed. It can not only capture common features but also identify unique attributes, thus providing more detailed data insights.

[0107] In one embodiment, if the preset distance metric includes a similarity distance, such as Figure 9 shown, the implementation process of step S401 includes the following steps S501 to step S503:

[0108] Step S501, for each of the tensor instances, determine the common columns between the targeted tensor instance and other tensor instances.

[0109] Among them, the similarity distance measures the similarity degree between the common columns (i.e., the common columns) of two tensor instances to be compared, which helps to identify the common features between different data sets. The computer device determines the common columns between two tensor instances to be compared.

[0110] Step S502, normalize each of the common columns respectively to obtain the normalization results of each of the common columns.

[0111] In applications, in order to perform effective distance calculations within the same range, the data of the common columns is normalized, such as normalizing the data of the common columns to data between [-1.0, 1.0].

[0112] In applications, the normalization operation can ensure that data of different scales can be compared within the same range, avoiding certain columns dominating the distance calculation due to overly large numerical ranges. Common methods include min-max normalization, that is, scaling all values to the range [-1.0, 1.0].

[0113] Step S503, based on the preset distance determination method and the normalization results of each of the common columns, determine the similarity distance between the targeted tensor instance and the other tensor instances.

[0114] In applications, the selected distance formula (such as Euclidean distance, Manhattan distance, etc.) is applied in combination with the normalization results of the common columns to determine the similarity distance between two tensor instances.

[0115] Exemplarily, as Figure 10 shown, there are two tensor instances to be compared, Tensor A and Tensor B, which have common columns "temperature_c" and "humidity". The data of the common columns are normalized respectively to obtain the corresponding normalization results. Finally, the Euclidean distance formula is applied to calculate the similarity distance:

[0116]

[0117] In this embodiment, by determining the similarity distance between each tensor instance and other tensor instances, the accuracy and efficiency of data analysis and machine learning tasks can be significantly improved. Specifically, first, the common columns between each tensor instance and other tensor instances are determined to ensure that the same features or attributes are compared; then, these common columns are normalized to eliminate the influence of different scales or units, ensuring the fairness and accuracy of distance calculation. The normalized result can more truly reflect the relative differences between data. Finally, based on a preset distance determination method (such as Euclidean distance, cosine similarity, etc.), combined with the normalized common column results, the similarity distance between the targeted tensor instance and other tensor instances is accurately calculated. This method not only improves the scientificity and rationality of distance measurement but also enhances the model's adaptability to multi-source heterogeneous data, reduces the bias caused by data characteristic differences, and thus supports more complex data analysis tasks and application scenarios.

[0118] In one embodiment, if the preset distance metric includes a dissimilarity distance, such as Figure 11 shown, the implementation process of step S502 includes the following steps S601 to step S603:

[0119] Step S601, for each of the tensor instances, respectively determine the unique columns of the targeted tensor instance and other tensor instances.

[0120] In applications, for the tensor instances to be compared, the computer device determines the unique columns of the tensor instance. The unique columns are columns that do not exist in another tensor instance, which helps to identify the unique features of each tensor instance.

[0121] Step S602, respectively normalize each of the unique columns to obtain the normalized results of each of the unique columns.

[0122] In applications, the data of these unique columns are normalized to -1.0 to 1.0; since the unique columns may have a greater impact on the overall distance, their weights can be adjusted according to the actual situation (weight adjustment).

[0123] Step S603, based on the normalized results of each of the unique columns, determine the dissimilarity distance between the targeted tensor instance and the other tensor instances.

[0124] In an application, the dissimilarity distance can be calculated by simply adding the normalization results of these unique columns or applying other suitable distance formulas. For example, the dissimilarity distance can be determined by calculating the number of unique columns, i.e., the dissimilarity distance can be the number of unique columns in each tensor instance; the dissimilarity distance can also be determined by calculating the proportion of unique columns, i.e., the dissimilarity distance can be the number of unique columns in each tensor instance divided by the total number of all columns. In practical applications, it may be necessary to select a suitable calculation method according to specific circumstances. For example, if the number of columns in tensor instances varies greatly, using the proportion may be more appropriate. If the number of columns is similar, using the quantity may be more intuitive. In addition, according to specific application scenarios, other factors such as the importance or weight of columns may also need to be considered.

[0125] Exemplarily, as Figure 12 shown, there are two tensor instances, Tensor A and Tensor B, each with some unique columns. The unique column of Tensor A is "wind_speed_kph", and the unique column of Tensor B is "precipitation_mm". The data of the unique columns are normalized from -1.0 to 1.0 respectively. The dissimilarity distance determined by the number of unique columns is 0.7 (0.3 + 0.4).

[0126] In this embodiment, by determining the dissimilarity distance between each tensor instance and other tensor instances, the feature discrimination ability in data analysis and machine learning tasks can be significantly enhanced. The specific process includes: first, identifying the unique columns of each tensor instance respectively, which reflect the unique attributes or features of each tensor instance; then normalizing these unique columns to eliminate the influence of different scales or units and ensure the fairness and accuracy of distance calculation. Based on the normalized results of the unique columns, the dissimilarity distance between the targeted tensor instance and other tensor instances is accurately calculated. This method can not only effectively capture the unique differences between instances, avoid the problem that common features mask important individual features, but also enhance the model's understanding depth of complex data sets and improve the effect of data mining. In addition, by focusing on the distance metric of unique columns, it helps to discover subtle differences in data, supports applications such as more accurate data classification, anomaly detection, and personalized recommendation, thereby improving the overall performance and reliability of the system.

[0127] It should be noted that for data in the form of tensors stored persistently, the application can add new tensor instances to the running system in real time without restarting or interrupting the existing services. This flexibility is crucial for adapting to changing data requirements.

[0128] The embodiment of the present application also provides a data storage device for performing the steps in the above data storage method embodiment. AsFigure 13 As shown in Figure 13 , the data storage device 1300 provided by the embodiment of the present application includes:

[0129] An acquisition module 1310, configured to acquire raw data from different data sources, and the raw data of each data source is used to describe the same specified object or the same data usage purpose.

[0130] A generation module 1320, configured to define a general tensor class based on the specified object or the data usage purpose, and generate tensor instances of the raw data of each data source based on the general tensor class.

[0131] An encoding module 1330, configured to encode each tensor instance based on a preset common identifier to obtain an encoding result of each tensor instance.

[0132] A storage module 1340, configured to store the encoding results of each tensor instance in a specified storage space.

[0133] In one embodiment, the generation module is further configured to create and obtain proofreading information of the general tensor class of the specified object, where the proofreading information is used to describe the format and semantics of each field in the general tensor class; and perform instantiation of the general tensor class based on the proofreading information and the raw data of each data source to obtain tensor instances of the raw data of each data source.

[0134] In one embodiment, the encoding module is further configured to obtain a string table matching the tensor type of the general tensor class, where the string table is used to describe the correspondence between common strings and common identifiers; read the target identifier of the common string associated with the tensor instance from the string table; and encode each tensor instance based on each target identifier to obtain an encoded tensor instance.

[0135] In one embodiment, the encoding module is further configured to, if the format type of the raw data is a key-value pair type, obtain the target identifier of each key or value in the raw data; and encode each tensor instance based on the target identifiers of each key or each value in a preset order to obtain an encoding result of each tensor instance.

[0136] In one embodiment, the encoding module is further configured to, if the format type of the raw data is a variable-length string type, obtain the string length of each variable string in the tensor instance; and encode each tensor instance based on the target identifier and the string length of each variable string to obtain an encoding result of each tensor instance.

[0137] In one embodiment, the storage module is further configured to, for each of the tensor instances, determine the distance between the targeted tensor instance and other tensor instances; and perform tensor deduplication based on the distance to obtain the tensor instances to be stored.

[0138] In one embodiment, the storage module is further configured to, for each of the tensor instances, determine the common columns between the targeted tensor instance and other tensor instances; perform normalization processing on each of the common columns respectively to obtain the normalization results of each of the common columns; and determine the similarity distance between the targeted tensor instance and the other tensor instances based on a preset distance determination method and the normalization results of each of the common columns.

[0139] In one embodiment, the storage module is further configured to, for each of the tensor instances, respectively determine the unique columns of the targeted tensor instance and other tensor instances; perform normalization processing on each of the unique columns respectively to obtain the normalization results of each of the unique columns; and determine the difference degree distance between the targeted tensor instance and the other tensor instances based on the normalization results of each of the unique columns.

[0140] In applications, each module in the data storage device can be a software program module, can also be implemented by different logic circuits integrated in a processor, or can also be implemented by multiple distributed processors.

[0141] Figure 14 The figure is a schematic structural diagram of a computer device for implementing data storage provided by an embodiment of the present application. As Figure 14 shown, the computer device 1400 of this embodiment includes: at least one processor 1410 ( Figure 14 only one is shown in the figure), a processor, a memory 1420, and a computer program 1430 stored in the memory 1420 and executable on the at least one processor 1410. When the processor 1410 executes the computer program 1430, the steps in any of the above-mentioned robot control method embodiments are implemented.

[0142] The computer device 1400 can be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer device 1400 may include, but is not limited to, a processor 1410 and a memory 1420. Those skilled in the art can understand that Figure 14 merely examples of the computer device 1400, which do not constitute a limitation on the computer device 1400, and may include more or fewer components than those shown in the figure, or combine some components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0143] The so-called processor 1410 may be a Central Processing Unit (CPU), and this processor 1410 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0144] The memory 1420 may be an internal storage unit of the computer device 1400 in some embodiments, such as the hard disk or memory of the computer device 1400. The memory 1420 may also be an external storage device of the computer device 1400 in other embodiments, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. equipped on the computer device 1400. Further, the memory 1420 may also include both the internal storage unit and the external storage device of the computer device 1400. The memory 1420 is used to store an operating system, application programs, a BootLoader, data, and other programs, such as the program code of the computer program, etc. The memory 1420 may also be used to temporarily store data that has been output or will be output.

[0145] It should be noted that, regarding the information interaction, execution process, etc. between the above-mentioned devices / units, since they are based on the same concept as the method embodiments of this application, for their specific functions and the technical effects brought, please refer to the method embodiment section for details, and will not be elaborated here.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0147] An embodiment of this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented.

[0148] An embodiment of this application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal is caused to execute the steps in the foregoing method embodiments.

[0149] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above method embodiments of this application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the device / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0150] In the above embodiments, the descriptions of the various embodiments have their respective emphases. For parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0151] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0152] In the embodiments provided in this application, it should be understood that the disclosed device / network device and method can be implemented in other ways. For example, the device / network device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] The above embodiments are only used to illustrate the technical solutions of this application, not to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. A data storage method, characterized in that, Including: Obtain raw data from different data sources, where the raw data of each data source is used to describe the same specified object or the same data usage purpose; Based on the specified object or the data usage purpose, define a general tensor class, and generate tensor instances of the raw data of each data source based on the general tensor class; Encode each tensor instance based on a preset common identifier to obtain the encoding result of each tensor instance; Store the encoding results of each tensor instance in a specified storage space; Before storing the encoding results of each tensor instance in the specified storage space, the method further includes: for each tensor instance, determine the distance between the targeted tensor instance and other tensor instances; based on the distance, perform tensor deduplication to obtain the tensor instances to be stored; The determining the distance between the targeted tensor instance and other tensor instances for each tensor instance includes: for each tensor instance, determine the common columns between the targeted tensor instance and other tensor instances; perform normalization processing on each common column respectively to obtain the normalization results of each common column; based on a preset distance determination method and the normalization results of each common column, determine the similarity distance between the targeted tensor instance and the other tensor instances; or, for each tensor instance, respectively determine the unique columns of the targeted tensor instance and other tensor instances; perform normalization processing on each unique column respectively to obtain the normalization results of each unique column; based on the normalization results of each unique column, determine the difference distance between the targeted tensor instance and the other tensor instances.

2. The data storage method according to claim 1, wherein The generating tensor instances of the raw data of each data source based on the general tensor class includes: Create and obtain the collation information of the general tensor class of the specified object, where the collation information is used to describe the format and semantics of each field in the general tensor class; Based on the collation information and the raw data of each data source, perform instantiation for the general tensor class to obtain the tensor instances of the raw data of each data source.

3. The data storage method according to claim 1, characterized in that The encoding each tensor instance based on a preset common identifier to obtain the encoding result of each tensor instance includes: Obtain a string table matching the tensor type of the general tensor class, where the string table is used to describe the correspondence between common strings and common identifiers; Read the target identifier of the common string associated with the tensor instance from the string table; Encode each tensor instance based on each target identifier to obtain the encoding result of each tensor instance.

4. The data storage method according to claim 3, wherein The encoding each tensor instance based on each target identifier to obtain the encoding result of each tensor instance includes: If the format type of the raw data is a key-value pair type, obtain the target identifier of each key or value in the raw data; Encode each tensor instance in a preset order based on the target identifiers of each key or each value to obtain the encoding result of each tensor instance.

5. The data storage method according to claim 3, wherein Encoding each of the tensor instances based on the respective target identifiers to obtain an encoding result for each of the tensor instances, includes: If the format type of the original data is a variable-length string type, obtain the string length of each variable string in the original data; Encode each of the tensor instances based on the target identifier and the string length of each variable string to obtain an encoding result for each of the tensor instances.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the method described in any one of claims 1 to 5 is implemented.

7. A computer program product, characterized in that, Including a computer program, when the computer program is run, the method described in any one of claims 1 - 5 is executed.

Images