Data storage method, computer device and computer program product

Abstract

The invention is applicable to the technical field of computers, and provides a data storage method, computer equipment and a computer program product, comprising: acquiring original data from different data sources, the original data of each data source being used for describing the same specified object or the same data usage purpose; defining a general tensor class based on the specified object or the data use purpose, and generating a tensor instance of original data of each data source based on the general tensor class; on the basis of a preset public identifier, encoding each tensor instance to obtain an encoding result of each tensor instance; and storing the coding result of each tensor instance to a specified storage space. According to the method and the device, persistent storage of data in a tensor form can be realized, data storage redundancy is reduced, and data storage efficiency is improved.

Description

Technical Field

[0001] The present application belongs to the field of data storage, and in particular, relates to a data storage method, a computer device, and a computer program product. Background Art

[0002] Tensors are the lingua franca of artificial intelligence. In the prior art, tensors have never been stored persistently in any form. Instead, they are processed through intermediate representations specific to the application domain, usually in the form of synthetic, abstract mathematical expressions rather than representing meaningful raw data. This practice means that tensors of raw data are usually temporary, dynamically created and destroyed during the computation process, and are not saved for a long time. As a result, raw data is usually stored in its original format, resulting in data redundancy, limited random access, low compression efficiency, and difficulty in interoperability across multiple application domains, especially in the fields of time series data (such as InfluxDB) and genomic data. Therefore, there is an urgent need for an efficient data storage method that can achieve persistent storage of tensors. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a data storage method, a computer device, and a computer program product, which can realize persistent storage of original data in the form of tensors, reduce data storage redundancy, and improve data storage efficiency.

[0004] A first aspect of an embodiment of the present application provides a data storage method, including: Acquiring raw data from different data sources, where the raw data from each of the data sources is used to describe the same specified object or the same data usage purpose; Based on the specified object or the data usage purpose, a general tensor class is defined, and a tensor instance of the original data of each of the data sources is generated based on the general tensor class; Based on a preset public identifier, encoding each of the tensor instances to obtain encoding results of each of the tensor instances; The encoding result of each tensor instance is stored in a specified storage space.

[0005] In an implementation of the first aspect, generating a tensor instance of the original data of each of the data sources based on the general tensor class includes: Create and obtain collation information of a general tensor class of the specified object, where the collation information is used to describe the format and semantics of each field in the general tensor class; Based on the collation information and the original data of each of the data sources, the general tensor class is instantiated to obtain a tensor instance of the original data of each of the data sources.

[0006] In an implementation of the first aspect, encoding each of the tensor instances based on a preset public identifier to obtain an encoding result of each of the tensor instances includes: Obtain a string table matching a tensor type of a general tensor class, where the string table is used to describe a correspondence between a public string and a public identifier; Reading a target identifier of a public string associated with the tensor instance from the string table; Based on each of the target identifiers, each of the tensor instances is encoded to obtain an encoding result of each of the tensor instances.

[0007] In an implementation of the first aspect, encoding each of the tensor instances based on each of the target identifiers to obtain an encoding result of each of the tensor instances includes: If the format type of the original data is a key-value pair type, obtaining a target identifier for each key or value in the original data; Based on the target identifier of each of the keys or each of the values, each of the tensor instances is encoded in a preset order to obtain an encoding result of each of the tensor instances.

[0008] In an implementation of the first aspect, if the format type of the original data is a variable-length string type, obtaining the string length of each variable string in the original data; Based on the target identifier and the string length of each variable string, each tensor instance is encoded to obtain an encoding result of each tensor instance.

[0009] In an implementation of the first aspect, before storing the encoding result of each tensor instance in the designated storage space, the method further includes: For each of the tensor instances, determine the distance between the tensor instance and other tensor instances; Based on the distance, tensor deduplication is performed to obtain tensor instances to be stored.

[0010] In an implementation of the first aspect, determining, for each of the tensor instances, a distance between the tensor instance and other tensor instances includes: For each of the tensor instances, determine the common columns between the tensor instance and other tensor instances; Normalizing each of the common columns respectively to obtain normalized results of each of the common columns; Based on a preset distance determination method and the normalization results of each of the common columns, a similarity distance between the targeted tensor instance and the other tensor instances is determined.

[0011] In an implementation of the first aspect, determining, for each of the tensor instances, a distance between the tensor instance and other tensor instances includes: For each of the tensor instances, determining the unique columns of the tensor instance and other tensor instances respectively; Normalizing each of the unique columns respectively to obtain normalized results of each of the unique columns; Based on the normalized results of each of the unique columns, the difference distance between the targeted tensor instance and the other tensor instances is determined.

[0012] A second aspect of an embodiment of the present application provides a data storage device, including: An acquisition module, used to acquire raw data from different data sources, where the raw data from each data source is used to describe the same specified object or the same data usage purpose; A generation module, used to define a general tensor class based on the specified object or the data usage purpose, and generate a tensor instance of the original data of each of the data sources based on the general tensor class; An encoding module, used for encoding each of the tensor instances based on a preset public identifier to obtain an encoding result of each of the tensor instances; A storage module is used to store the encoding results of each tensor instance into a specified storage space.

[0013] A third aspect of an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the computer device implements the method described in the first aspect above.

[0014] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0015] A fifth aspect of the embodiments of the present application provides a computer program product, including a computer program, and when the computer program is executed, the method described in the first aspect above is executed.

[0016] The first aspect of the embodiment of the present application obtains the original data describing the same specified object or the same data usage purpose from different data sources, generates a standardized tensor instance based on the general tensor class of the specified object, and then encodes and stores the tensor instance, which significantly improves the consistency, efficiency and interoperability of data processing. Specifically, it converts the original data describing the same specified object or the same data usage purpose into a tensor instance with a unified format and semantic constraints, eliminating data format differences; uses a preset public identifier to encode the tensor instance, which enhances the consistency and parsability of the data and facilitates cross-platform exchange; and stores the encoded instance in a specified space to achieve persistent storage of data in tensor form, reduce data storage redundancy, and improve data storage efficiency.

[0017] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a schematic diagram of the implementation process of the data storage method provided in the embodiment of the present application; Figure 2 is an example diagram of a tensor instance provided in an embodiment of the present application; Figure 3 is a flowchart of an implementation method of an instantiation method of a general tensor class provided in an embodiment of the present application; Figure 4 is a flowchart of an implementation method of encoding a tensor instance provided in an embodiment of the present application; Figure 5 is an example diagram of the encoding result of the tensor instance provided in the embodiment of the present application; Figure 6 This is an example diagram of the style of FASTQ file data provided in an embodiment of the present application; Figure 7 is another example diagram of the encoding result of the tensor instance provided in the embodiment of the present application; Figure 8 is a flowchart of the implementation of the tensor deduplication method provided in the embodiment of the present application; Fig. 9 is a flowchart of the implementation of the method for determining the similarity distance provided in the embodiment of the present application; Fig.10is an example diagram of similarity distance provided in an embodiment of the present application; Fig.11 is a flowchart of the method for determining the difference distance provided in the embodiment of the present application; Fig.12 is an example diagram of the difference distance provided in the embodiment of the present application; Fig.13 is a schematic diagram of a data storage device provided in an embodiment of the present application; Fig.14 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0021] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0022] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0023] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0024] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0025] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0026] Currently, tensors, as the main data structure of artificial intelligence, are usually not stored persistently, but processed through intermediate representations specific to the application domain. This leads to problems such as data redundancy, limited random access, low compression efficiency, and difficulty in interoperability across multiple application domains. This is especially true in the fields of time series data (such as data stored in InfluxDB) and genomic data (such as FASTQ, BAM, SAM, or BAI files), which is a type of biological sequence data. Specifically, InfluxDB uses the JSON format to store data, resulting in large file sizes and limited random access; while the BAI files of genomic data are constructed after the fact, compressed using LZ variants, without considering the data efficiency issues of FP32 (32-bit floating point type) and FP8 (8-bit floating point type), and there is a phenomenon of read amplification. In addition, the existing technology lacks versatility, cannot solve the problems of time series and genomic data at the same time, and fails to fully implement the distance-based tensor deduplication strategy, wasting storage space and computing resources. The difficulty in integrating multi-source heterogeneous data, poor dynamic adaptability, and lack of effective metadata management further limit the flexibility and efficiency of the system.

[0027] Based on this, an embodiment of the present application provides a data storage method that can persistently store raw data from different data sources in the form of tensors in a unified format, reduce data storage redundancy, and improve data storage efficiency.

[0028] In one embodiment, Figure 1 As shown, a data storage method is provided, which is described by taking the method applied to a computer device as an example, and includes the following steps S101 to S104: Step S101 : acquiring original data from different data sources, wherein the original data from each data source is used to describe the same designated object or the same data usage purpose.

[0029] In applications, raw data from multiple different sources are used to describe the same specified object or the same data usage purpose. For example, the specified object can be the abnormal information generated by each node in the data center cluster, and the specified object can also be the meteorological information generated by multiple sensors in the weather station. In these scenarios, for the specified object, the raw data generated by multiple data sources can be associated. Taking the specified object as meteorological data as an example, there are multiple models of meteorological station sensors, each of which can generate abnormal data, and different sensors are different sources of abnormal data. In any meteorological station, the meteorological data of the meteorological station is represented by the raw data collected by multiple different sensors. The raw data generated by different data sources are all used to describe the specified object.

[0030] Step S102: Based on the designated object or the data usage purpose, a general tensor class is defined, and based on the general tensor class, a tensor instance of the original data of each of the data sources is generated.

[0031] In the application, for the same designated object or the same data usage purpose, a general tensor class is predefined, which is used to describe the designated object or data usage purpose and includes a description format of data for constraining the designated object or data usage purpose. In this way, the designated object or data usage purpose can be described in a unified format. Since the designated object is represented by raw data from multiple different data sources, there are differences between the raw data of each data source. In order to ensure that the raw data of multiple different data sources conform to a unified format, the computer device defines a general tensor class for each designated object to standardize the raw data of different data sources associated with the designated object to obtain target data in a unified format. The general tensor class determines the basic structure and purpose of the tensor used to describe the designated object. The general tensor class defines tensor fields that describe the designated object, including information such as name, type, and unit. A tensor instance is always contained in a node. A node is a container for tensor values, representing a specific physical or logical entity (such as a computer, storage element, or genome scanning device). Each node can hold and process one or more tensor instances and the interpretation of the tensor instance header depends on the node (for example, through the _ntyp field). In addition, a tensor class can coexist in multiple nodes, and there may be differences between these instances, which are specified by the instance header. Tensor types are not restricted and can be very diverse. The header information of the tensor instance can be used to distinguish the node to which the tensor instance belongs. Tensor instances of the same tensor class on different nodes may be different, and these differences are specified by the header information of the tensor instance.

[0032] In the application, for the raw data of different data sources for a specified object or the same data usage purpose, the computer device can standardize the raw data of different data sources based on the general tensor class for the specified object or the same data usage purpose, that is, use the raw data of different data sources to perform instantiation operations on the general tensor class to obtain the tensor instance of the raw data of each data source relative to the general tensor class. The tensor instance is a specific implementation of the tensor class, which contains the actual data value and metadata. Each tensor instance always includes the general tensor class to which it belongs, ensuring the consistency and relevance of the instance with its definition.

[0033] In the general tensor class, internal elements and optional elements can be distinguished by different tags. Among them, internal elements: Elements starting with "_" are considered internal elements, which are usually used to represent metadata or other auxiliary information. These elements are part of the tensor definition and can be interpreted and used by AI programs. Optional elements: Elements that do not start with "_" are considered optional elements. They may contain additional information or comments, but will not be directly interpreted by AI programs. The presence or absence of these optional elements does not affect the core functionality of the tensor.

[0034] Step S103, encoding each of the tensor instances based on a preset public identifier to obtain an encoding result of each of the tensor instances.

[0035] In applications, public identifiers are used to identify public strings in raw data. Raw data from different data sources is large in size and occupies large storage resources. In order to achieve low storage space storage in the form of tensors, public identifiers are used to encode public strings in tensor instances. The storage space required for the encoded tensor instance is lower than that of the tensor instance before encoding.

[0036] For example, Figure 2As shown in the figure, a tensor instance containing data from multiple weather stations. Among them, all temperature readings are converted to degrees Celsius, wind speed (wind_speed_kph) readings are converted to kilometers per hour, and other indicators are also standardized accordingly. Fields prefixed with "_" such as _timestamp and _sensor_id are internal elements, indicating that they are metadata or key identifiers. Fields without the prefix "_" such as temperature and humidity are optional elements, indicating that they are actual measurements for further analysis. Among them, the public identifier "1002" is used to represent the public string "temperature" or "temperature"; the public identifier "1004" is used to represent the public string "wind speed" or "wind_speed_kph". Therefore, at least the "temperature" and "wind speed" in the tensor instance can be encoded by the public identifier "1002" and the public identifier "1004", resulting in an encoding result with lower storage space required.

[0037] Step S104, storing the encoding result of each tensor instance into a designated storage space.

[0038] In the application, the encoding result of the tensor instance of each original data is stored in the specified storage space to achieve persistent storage of the original data in tensor form.

[0039] In this embodiment, by obtaining raw data from different data sources and generating standardized tensor instances based on the general tensor class of the specified object, and then encoding and storing these tensor instances, the consistency, efficiency and interoperability of data processing can be significantly improved. First, it ensures that all raw data describing the same specified object, regardless of its source, are converted into tensor instances that follow a unified format and semantic constraints, thereby eliminating the obstacles caused by data format differences. Secondly, each tensor instance is encoded using a preset identifier, which further enhances the consistency and parsability of data representation, making cross-platform and cross-system data exchange smoother. Finally, the encoded tensor instance is stored in the specified storage space, which can realize the persistent storage of raw data in the form of tensors, reduce data storage redundancy, and improve data storage efficiency. In one embodiment, Figure 3 As shown, the implementation process of step 102 includes the following steps S201 to S202: Step S201, creating and obtaining collation information of the general tensor class of the specified object, wherein the collation information is used to describe the format and semantics of each field in the general tensor class.

[0040] In the application, the fields in the general tensor class are associated with corresponding collation information, which is used to describe the clear semantics and format requirements of the field. The collation information is stored in JSON format to ensure that it is easy to parse and use. JSON is a lightweight data exchange format widely used between various programming languages ​​and platforms. Each general tensor class has its own specific collation information in JSON format, which ensures that the collation information between general tensor classes of different specified objects will not be confused, and can be customized according to the different needs of the class.

[0041] Step S202: Based on the proofreading information and the original data of each of the data sources, instantiate the general tensor class to obtain a tensor instance of the original data of each of the data sources.

[0042] In the application, based on the universal tensor class, the actual value of each field is selected from the original data of each data source, and the actual value of the field is standardized in combination with the proofreading information of the field to obtain the target value in a standard format, and finally the tensor instance of the original data of each data source relative to the universal tensor class is obtained.

[0043] In this embodiment, by generating raw data tensor instances of each data source based on a general tensor class of a specified object, the consistency and efficiency of data processing can be significantly improved. Specifically, first obtain the proofreading information describing the format and semantics of each field in the tensor class to ensure a clear and unified understanding of the data structure. Then, use these proofreading information in combination with the raw data of each data source to perform instantiation operations on the general tensor class to generate standardized tensor instances. This not only simplifies the integration process of multi-source heterogeneous data, ensures the consistency of format and semantics of data from different sources, but also enhances the flexibility and reliability of data processing. Through a unified data representation, cross-platform data exchange and analysis tasks can be supported more efficiently, reducing parsing errors caused by data format or semantic differences.

[0044] In one embodiment, Figure 4 As shown, the implementation process of step 103 includes the following steps S301 to S303: Step S301, obtaining a string table matching the tensor type of the general tensor class, wherein the string table is used to describe the correspondence between public strings and public identifiers.

[0045] In the application, in order to reduce the storage space occupied by the tensor instance, the computer device can compress each tensor instance of the general tensor class through the string table adapted to the general tensor class to reduce the storage space requirements of the tensor instance. The string table is used to describe the correspondence between strings and identifiers. The string table avoids repeatedly storing these strings in each tensor instance by mapping common string values ​​to unique identifiers (usually integer IDs). In this way, when storing tensor instances, the strings in the tensor instances can be replaced with corresponding identifiers, which can effectively reduce the demand for storage space. The working principle of the string table is as follows: when writing a tensor instance, the computer device checks whether the string in the tensor instance already exists in the string table. If it exists, only the identifier ID corresponding to the string is stored; if it does not exist, the new string is added to the string table and a new identifier ID is assigned. In this way, the storage space is significantly reduced, especially when there are many repeated strings. The query efficiency is improved because the corresponding string can be quickly found by ID.

[0046] Step S302 : Reading a target identifier of a common string associated with a tensor instance from the string table.

[0047] In the application, for each tensor instance of a specified object, a target identifier corresponding to each string in the specified tensor instance is read from a string table corresponding to a general tensor class of the specified object.

[0048] Step S303: Encode each of the tensor instances based on each of the target identifiers to obtain an encoding result of each of the tensor instances.

[0049] In the application, based on the target identifier corresponding to each string, the string in the tensor instance is replaced, and the target tensor instance is generated by encoding in a preset order.

[0050] In this embodiment, by encoding the tensor instance based on a preset public identifier, more efficient and consistent data processing can be achieved. Specifically, first, a string table matching the tensor type is obtained, which describes in detail the correspondence between the string and the identifier. Then, the target identifier of the string in each tensor instance is read from this string table. Finally, each tensor instance is encoded based on these target identifiers to obtain the encoded tensor instance. This method not only simplifies the data processing process, but also ensures the semantic consistency and accuracy of data from different sources. For example, when processing data involving multiple units (such as currency units USD, EUR or temperature units Celsius, Fahrenheit), it can ensure that the specific meaning of the field (such as counters, process IDs, user IDs, etc.) is correctly parsed and uniformly represented. This helps to improve the efficiency of data processing, reduce errors caused by inconsistent units or semantics, and thus enhance the reliability and maintainability of the system.

[0051] In one embodiment, the specific implementation process of step S303 is as follows: if the format type of the original data is a key-value pair type, then obtain the target identifier of each key or value in the original data; based on the target identifier of each key or each value, encode each tensor instance in a preset order to obtain the encoding result of each tensor instance.

[0052] For example, Figure 5 As shown in the figure, the raw data is stored in the Influx database in JSON format. The format type of the raw data is a key-value pair type. The key or value in the raw data usually corresponds to the field in the tensor instance. In the Influx model, the keys in the raw data are often repeated, resulting in intrinsic data redundancy, including the following information: {"temperature": 81, "humidity": 43}, the key information is "temperature" and "humidity", and the preset string table corresponding to the key information or value information is constructed as follows: "temperature" -> 1002, "humidity" -> 1003, then the target identifier of each key or value is read from the preset string table, and the tensor instance corresponding to the raw data is encoded in the preset order. The encoding result of the tensor instance is as follows: {1,1002,2,81,3,1003,4,43}, where 1 in "1,1002,2" is the sequence number of the preset order. The encoding result of the tensor instance is a column-oriented tensor. Compared to the original data, column-oriented tensor storage can achieve 50% compression and data editing.

[0053] In this embodiment, when the format type of the original data is a key-value pair type, by obtaining the target identifier of each key or value and encoding it based on a preset order, the consistency and efficiency of data processing can be significantly improved. Specifically, first extract the identifier of each field in the tensor instance to ensure that each field has a clear and unique representation. Then, encode the identifiers of these fields in a preset order to generate a standardized encoding result. This not only simplifies the data structure and facilitates subsequent processing and analysis, but also ensures that the comparison and operation between different tensor instances are highly consistent.

[0054] In one embodiment, if the format type of the original data is a variable-length string type, the string length of each variable string in the original data is obtained; based on the target identifier and string length of each variable string, each tensor instance is encoded to obtain the encoding result of each tensor instance.

[0055] In the application, the variable-length string type raw data can be genomic data, which is the binary encoding of gene scan data and the index of the scan encoding. Genomic data is usually in the form of FASTQ, BAM, SAM, BAI and other files. BAI files are constructed afterwards. It does not consider the data efficiency issues of FP32 and FP8, and there is considerable read amplification. For example, Figure 6 The format of the FASTQ file data is shown.

[0056] For example, Figure 7 As shown, taking the original data as genome data as an example, the original data is {TAGAATACGCCAGGT \n", "ACGTATGGC\n", "TAGAATACGCCAGGT \n"}, the public string is "TAGAATACGCCAGGT", and the public string is stored in the string table: the public identifier corresponding to "TAGAATACGCCAGGT" is 2001, then the tensor instance corresponding to the original data is encoded based on the public string in the string table, and the encoded tensor instance is {1,15,2001,2,9,ACGTATGGC ,3,15,2001}. {1, 15, 2001} is the first string, the string length is 15, and the identifier is 2001.

[0057] In this embodiment, by encoding the tensor instances based on each target identifier, especially for the case where the original data is a variable-length string type, the method significantly improves the efficiency and consistency of data processing. Specifically, the string length of the variable string in each tensor instance is first obtained, and then the tensor instance is encoded in combination with the target identifier and length information of these strings to obtain a standardized encoding result. This method not only ensures the effective representation of strings of different lengths, but also improves the compactness of data storage and transmission and reduces redundancy.

[0058] In one embodiment, Figure 8 As shown, before storing the encoding results of each tensor instance into the designated storage space, the computer device may also perform a tensor deduplication operation, and the specific implementation process includes the following steps S401 to S402: Step S401: for each of the tensor instances, determine the distance between the tensor instance and other tensor instances.

[0059] In the application, when storing tensor instances, the computer device can perform tensor deduplication based on a preset distance measurement method so that duplicate tensor instances can be stored only once. The preset distance measurement methods include similarity distance and difference distance. Among them, the similarity distance is used to measure the similarity between two tensor instances. The similarity between the tensor instances is determined by determining the number or proportion of common columns between the two tensor instances to be compared. If the two tensor instances have the same data in multiple columns, then the similarity distance between them will be smaller, indicating that they have a higher similarity in these columns. The similarity distance can be used to measure the commonalities between two data sets in specific features. The difference distance focuses on the unique columns (i.e., unique columns) in the two tensor instances to be compared. The uniqueness of each tensor can be measured by calculating its unique columns. A larger difference distance means that the two tensors are different in multiple columns, that is, they show different data features in these columns. To explain the similarity distance, first determine the common columns between the two tensor instances to be compared; normalize the data of these common columns from -1.0 to 1.0 to ensure that data of different scales can be compared using the same standard; apply the selected distance formula (such as Euclidean distance, Manhattan distance, etc.) to calculate the similarity distance.

[0060] For the difference distance, we explain the unique columns in each tensor instance (unique columns). We normalize the data of these unique columns from -1.0 to 1.0. Since unique columns may have a greater impact on the overall distance, we can adjust their weights according to the actual situation (weight adjustment).

[0061] In the application, the computer device determines the distance between the tensor instances according to a preset distance measurement method.

[0062] Step S402: Based on the distance, deduplicate tensors to obtain tensor instances to be stored.

[0063] In the application, the computer device compares the distance with the preset distance threshold. If the comparison result indicates that the distance reaches the distance threshold, the two tensor instances representing the distance association are equivalent. Therefore, only one tensor instance needs to be retained as the tensor instance to be stored.

[0064] In this embodiment, by introducing the concepts of similarity distance and difference distance, the relationship between column-oriented tensors can be more comprehensively understood and analyzed, which can not only capture common features but also identify unique attributes, thereby providing more detailed data insights.

[0065] In one embodiment, if the preset distance measurement method includes a similarity distance, such as Fig. 9 As shown, the implementation process of step S401 includes the following steps S501 to S503: Step S501: for each of the tensor instances, determine the common columns between the tensor instance and other tensor instances.

[0066] Among them, the similarity distance measures the similarity between the common columns (i.e., common columns) in the two tensor instances to be compared, which helps to identify common features between different data sets. The computer device determines the common columns between the two tensor instances to be compared.

[0067] Step S502 , normalizing each of the common columns respectively to obtain a normalized result of each of the common columns.

[0068] In applications, in order to perform effective distance calculation within the same range, the data in the common columns are normalized, such as normalizing the data in the common columns to data between [-1.0, 1.0].

[0069] In applications, normalization operations can ensure that data of different scales can be compared within the same range, preventing some columns from dominating distance calculations due to excessively large value ranges. Common methods include min-max normalization, which scales all values ​​to the range of [-1.0, 1.0].

[0070] Step S503 : determining the similarity distance between the targeted tensor instance and the other tensor instances based on a preset distance determination method and the normalization results of the common columns.

[0071] In the application, the selected distance formula (such as Euclidean distance, Manhattan distance, etc.) is applied in combination with the normalized results of the common columns to determine the similarity distance between two tensor instances.

[0072] For example, Fig.10 As shown in the figure, there are two tensor instances to be compared, Tensor A and Tensor B, which have common columns "temperature_c" and "humidity". The data of the common columns are normalized to obtain the corresponding normalized results. Finally, the Euclidean distance formula is used to calculate the similarity distance:

[0073] In this embodiment, by determining the similarity distance between each tensor instance and other tensor instances, the accuracy and efficiency of data analysis and machine learning tasks can be significantly improved. Specifically, first determine the common columns between each tensor instance and other tensor instances to ensure that the same features or attributes are compared; then normalize these common columns to eliminate the influence of different scales or units to ensure the fairness and accuracy of distance calculation. The normalized results can more truly reflect the relative differences between data. Finally, based on the preset distance determination method (such as Euclidean distance, cosine similarity, etc.), combined with the normalized common column results, the similarity distance between the tensor instance and other tensor instances is accurately calculated. This method not only improves the scientificity and rationality of the distance metric, but also enhances the model's adaptability to multi-source heterogeneous data, reduces the deviation caused by differences in data characteristics, and thus supports more complex data analysis tasks and application scenarios.

[0074] In one embodiment, if the preset distance metric includes a difference distance, such as Fig.11 As shown, the implementation process of step S502 includes the following steps S601 to S603: Step S601: for each of the tensor instances, determine the unique columns of the tensor instance and other tensor instances respectively.

[0075] In the application, for the tensor instance to be compared, the computer device determines the unique column of the tensor instance, where the unique column is a column that does not exist in another tensor instance, which helps to identify the unique characteristics of each tensor instance.

[0076] Step S602: normalize each unique column to obtain a normalized result of each unique column.

[0077] In the application, the data of these unique columns are normalized from -1.0 to 1.0; since unique columns may have a greater impact on the overall distance, their weights can be adjusted according to actual conditions (weight adjustment).

[0078] Step S603: Determine the difference distance between the targeted tensor instance and the other tensor instances based on the normalization results of the unique columns.

[0079] In applications, the difference distance can be calculated by simply adding the normalized results of these unique columns or applying other appropriate distance formulas. For example, the difference distance can be determined by calculating the number of unique columns, that is, the difference distance can be the number of unique columns in each tensor instance; the difference distance can also be determined by calculating the ratio of unique columns, that is, the difference distance can be the number of unique columns of each tensor instance divided by the total number of all columns. In practical applications, it may be necessary to select an appropriate calculation method based on the specific situation. For example, if the number of columns of tensor instances varies greatly, it may be more appropriate to use a ratio. If the number of columns is similar, it may be more intuitive to use the number. In addition, depending on the specific application scenario, other factors may need to be considered, such as the importance or weight of the column.

[0080] For example, Fig.12 As shown, there are two tensor instances Tensor A and Tensor B, each with some unique columns. The unique column of Tensor A is "wind_speed_kph", and the unique column of Tensor B is "precipitation_mm". The data of the unique columns are normalized from -1.0 to 1.0. The difference distance is determined to be 0.7 (0.3+0.4) by the number of unique columns.

[0081] In this embodiment, by determining the difference distance between each tensor instance and other tensor instances, the feature differentiation ability in data analysis and machine learning tasks can be significantly enhanced. The specific process includes: first, identifying the unique columns of each tensor instance, which reflect the unique attributes or characteristics of each tensor instance; then normalizing these unique columns to eliminate the influence of different scales or units to ensure the fairness and accuracy of distance calculation. Based on the normalized unique column results, the difference distance between the tensor instance and other tensor instances is accurately calculated. This method can not only effectively capture the unique differences between each instance, avoid the problem of common features covering important individual features, but also enhance the model's understanding depth of complex data sets and improve the effect of data mining. In addition, by focusing on the distance metric of unique columns, it helps to discover subtle differences in the data, support more accurate data classification, anomaly detection, and personalized recommendation applications, thereby improving the overall performance and reliability of the system.

[0082] It should be noted that for data in the form of persistent tensors, applications can add new tensor instances to the running system in real time without restarting or interrupting existing services. This flexibility is critical to adapting to changing data needs.

[0083] The present application also provides a data storage device for executing the steps in the above data storage method embodiment. Fig.13 As shown, the data storage device 1300 provided in the embodiment of the present application includes: The acquisition module 1310 is used to acquire original data from different data sources, where the original data from each data source is used to describe the same designated object or the same data usage purpose.

[0084] The generation module 1320 is used to define a general tensor class based on the specified object or the data usage purpose, and generate a tensor instance of the original data of each of the data sources based on the general tensor class.

[0085] The encoding module 1330 is used to encode each of the tensor instances based on a preset public identifier to obtain an encoding result of each of the tensor instances.

[0086] The storage module 1340 is used to store the encoding results of each tensor instance into a designated storage space.

[0087] In one embodiment, the generation module is also used to create and obtain the proofreading information of the general tensor class of the specified object, where the proofreading information is used to describe the format and semantics of each field in the general tensor class; based on the proofreading information and the original data of each of the data sources, the general tensor class is instantiated to obtain a tensor instance of the original data of each of the data sources.

[0088] In one embodiment, the encoding module is also used to obtain a string table matching the tensor type of the general tensor class, wherein the string table is used to describe the correspondence between common strings and common identifiers; read the target identifier of the common string associated with the tensor instance from the string table; and encode each of the tensor instances based on each of the target identifiers to obtain an encoded tensor instance.

[0089] In one embodiment, the encoding module is also used to obtain the target identifier of each key or value in the original data if the format type of the original data is a key-value pair type; based on the target identifier of each key or each value, encode each tensor instance in a preset order to obtain the encoding result of each tensor instance.

[0090] In one embodiment, the encoding module is also used to obtain the string length of each variable string in the tensor instance if the format type of the original data is a variable-length string type; based on the target identifier and string length of each variable string, encode each tensor instance to obtain the encoding result of each tensor instance.

[0091] In one embodiment, the storage module is further used to determine, for each of the tensor instances, the distance between the tensor instance and other tensor instances; based on the distance, perform tensor deduplication to obtain the tensor instance to be stored.

[0092] In one embodiment, the storage module is also used to determine, for each of the tensor instances, the common columns between the targeted tensor instance and other tensor instances; normalize each of the common columns respectively to obtain the normalized results of each of the common columns; and determine the similarity distance between the targeted tensor instance and the other tensor instances based on a preset distance determination method and the normalized results of each of the common columns.

[0093] In one embodiment, the storage module is also used to determine, for each of the tensor instances, the unique columns of the targeted tensor instance and other tensor instances respectively; normalize each of the unique columns respectively to obtain the normalized results of each of the unique columns; and determine the difference distance between the targeted tensor instance and the other tensor instances based on the normalized results of each of the unique columns.

[0094] In applications, each module in the data storage device may be a software program module, or may be implemented by different logic circuits integrated in a processor, or may be implemented by multiple distributed processors.

[0095] Fig.14 This is a schematic diagram of the structure of a computer device for implementing data storage provided in one embodiment of the present application. Fig.14 As shown, the computer device 1400 of this embodiment includes: at least one processor 1410 ( Fig.14 Only one is shown in the figure) a processor, a memory 1420, and a computer program 1430 stored in the memory 1420 and executable on the at least one processor 1410, wherein when the processor 1410 executes the computer program 1430, the steps in any of the above-mentioned robot control method embodiments are implemented.

[0096] The computer device 1400 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device 1400 may include, but is not limited to, a processor 1410 and a memory 1420. Those skilled in the art will appreciate that Fig.14 This is only an example of the computer device 1400 and does not constitute a limitation on the computer device 1400 . The computer device 1400 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0097] The processor 1410 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0098] In some embodiments, the memory 1420 may be an internal storage unit of the computer device 1400, such as a hard disk or memory of the computer device 1400. In other embodiments, the memory 1420 may also be an external storage device of the computer device 1400, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 1400. Further, the memory 1420 may also include both an internal storage unit of the computer device 1400 and an external storage device. The memory 1420 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 1420 may also be used to temporarily store data that has been output or is to be output.

[0099] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0100] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0101] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0102] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0104] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0105] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0106] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0108] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data storage method, characterized in that: include: Acquiring raw data from different data sources, where the raw data from each of the data sources is used to describe the same specified object or the same data usage purpose; Based on the specified object or the data usage purpose, a general tensor class is defined, and a tensor instance of the original data of each of the data sources is generated based on the general tensor class; Based on a preset public identifier, encoding each of the tensor instances to obtain encoding results of each of the tensor instances; The encoding result of each tensor instance is stored in a specified storage space.

2. The data storage method according to claim 1, characterized in that: The generating a tensor instance of the original data of each data source based on the general tensor class includes: Create and obtain collation information of a general tensor class of the specified object, where the collation information is used to describe the format and semantics of each field in the general tensor class; Based on the collation information and the original data of each of the data sources, the general tensor class is instantiated to obtain a tensor instance of the original data of each of the data sources.

3. The data storage method according to claim 1, characterized in that: The encoding of each tensor instance based on a preset public identifier to obtain an encoding result of each tensor instance includes: Obtain a string table matching a tensor type of a general tensor class, where the string table is used to describe a correspondence between a public string and a public identifier; Reading a target identifier of a public string associated with the tensor instance from the string table; Based on each of the target identifiers, each of the tensor instances is encoded to obtain an encoding result of each of the tensor instances.

4. The data storage method according to claim 3, characterized in that: The encoding of each tensor instance based on each target identifier to obtain an encoding result of each tensor instance includes: If the format type of the original data is a key-value pair type, obtaining a target identifier for each key or value in the original data; Based on the target identifier of each of the keys or each of the values, each of the tensor instances is encoded in a preset order to obtain an encoding result of each of the tensor instances.

5. The data storage method according to claim 3, characterized in that: The encoding of each tensor instance based on each target identifier to obtain an encoding result of each tensor instance includes: If the format type of the original data is a variable-length string type, then obtaining the string length of each variable string in the original data; Based on the target identifier and the string length of each variable string, each tensor instance is encoded to obtain an encoding result of each tensor instance.

6. The data storage method according to claim 1, characterized in that: Before storing the encoding results of each tensor instance into the designated storage space, the method further includes: For each of the tensor instances, determine the distance between the tensor instance and other tensor instances; Based on the distance, tensor deduplication is performed to obtain tensor instances to be stored.

7. The data storage method according to claim 6, characterized in that: The step of determining, for each of the tensor instances, a distance between the tensor instance and other tensor instances includes: For each of the tensor instances, determine the common columns between the tensor instance and other tensor instances; Normalizing each of the common columns respectively to obtain normalized results of each of the common columns; Based on a preset distance determination method and the normalization results of each of the common columns, a similarity distance between the targeted tensor instance and the other tensor instances is determined.

8. The data storage method according to claim 6, characterized in that: The step of determining, for each of the tensor instances, a distance between the tensor instance and other tensor instances includes: For each of the tensor instances, determining the unique columns of the tensor instance and other tensor instances respectively; Normalizing each of the unique columns respectively to obtain normalized results of each of the unique columns; Based on the normalized results of each of the unique columns, the difference distance between the targeted tensor instance and the other tensor instances is determined.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

10. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables the method according to any one of claims 1 to 8 to be performed.

Images