Method and system for referencing data

Abstract

A method for referencing data is provided. The method includes receiving a request including data to be stored. The data to be stored is broken down into a plurality of bytes by byte. Each byte of the data is represented by a key value. The broken down data is vectorized to obtain a first vector of the data to be stored. The first vector includes at least one value pair. Each value pair includes a first value and a second value. The first value represents one or more bytes of the broken down data having a unique key value. The second value represents an instance of the one or more bytes of the broken down data having the unique key value present. It is determined whether the vectorized data exists in one or more data centers. A reference to the data to be stored is returned according to a result of the determining step.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of data science, and in particular to a method for referencing data locally or remotely and a system capable of performing the method thereof. BACKGROUND

[0002] Data centers are commonly used to store and share applications and data. The design of a data center is based on computer networks and storage resources that can transmit shared applications and data. A data center can include servers, data storage systems, network devices such as routers and switches, and security systems such as firewalls and encryption systems.

[0003] As the demand for storing more information continues to increase and is difficult to control, the size of data centers is also expanding. For example, data and applications are stored in data centers. The servers of a data center are computer hardware and / or software programs that provide services to other programs or devices such as clients. Servers are usually classified by their purpose. Some examples of these server categories include web servers, application servers, proxy servers, virtual servers, file servers, database servers, and printer servers. For example, the purpose of a database server is to be responsible for hosting one or more databases. Client applications can perform database queries to retrieve data from or write data to databases hosted by the server. Another example is a file server, which is responsible for storing and managing data files (such as text, images, sound, pictures, video) so that computers on the same network can access these files when needed.

[0004] The design of a file server becomes complex due to competing demands for storage space, access speed, recoverability, security, and budget. The complexity is further exacerbated as the environment is constantly changing, with new hardware and technology quickly rendering old equipment obsolete. The storage space of a server is quickly filled with new data being generated continuously, and more servers are needed to store new or modified data, which requires more infrastructure, resulting in additional costs, space, and energy requirements.

[0005] In addition, the security of data stored and shared by a data center is always a focal point.

[0006] The disadvantage of the current method and / or system for storing / sharing data or files in a data center is that as more and more files or data are shared or stored on the Internet over time, the data center will expand indefinitely. This results in increasing the size and maintenance costs of the data center.

[0007] Another disadvantage of the current methods and / or systems for storing / sharing data or files in a data center is the risk of cyber attacks. If a data center is attacked by hackers intentionally or unintentionally, the risk of data leakage is high, which is not desirable for users. SUMMARY

[0008] Therefore, it is advantageous to implement a method and system to overcome or at least alleviate the above-mentioned disadvantages. In particular, it is desirable to recycle data that has been stored in a data center and to ensure the security of the data stored in the data center. In order to better solve one or more of the above problems, a method, a system and a non-transitory computer readable storage medium having stored thereon a computer program having the features defined in the method claim are provided. Preferred embodiments are defined in the dependent claims.

[0009] Therefore, according to a first aspect, there is provided a method for referencing data, the method being performed by a processor operatively connected to one or more data centers in a network. Each of the one or more data centers is configured to store files and respective vectors of the files. The method comprises receiving a request comprising data to be stored; and breaking the data to be stored into a plurality of bytes by byte. Each byte of the data is represented by a key value. The method further comprises vectorizing the broken data to obtain a first vector of the data to be stored. The first vector comprises at least one value pair. Each value pair comprises a first value and a second value. The first value represents one or more bytes of the broken data having a unique key value. The second value represents instances of the one or more bytes having the unique key value present in the broken data. The method further comprises determining whether the vectorized data exists in the one or more data centers, and returning a reference of the data to be stored according to a result of the determining step.

[0010] The advantage of the present invention is that this method can effectively recycle existing data, which can be data in a local storage device or data in a shared device on the Internet. In this way, the size of the data center is almost fixed even if the data to be shared or stored continues to increase. With a data center of almost fixed size, it means that even if the data to be shared or stored continues to increase, i.e. the information stored in the data center will continue to increase, but the physical size of the data center will not increase as in the traditional data center, because the data already stored in the data center can be recycled. In other words, compared with the data to be added to the data center, this method greatly reduces the increase speed of the physical size of the data center.

[0011] The term "file" represents any type of data file, such as text, image, sound, picture, video. The data file can comprise a sequence of bytes. Each sequence of bytes can comprise one or more bytes.

[0012] The term "key value" is used to represent a byte in data. All bytes in data can be represented by a corresponding key value. Data can be represented in different number systems. For example, decimal number system, binary number system, octal number system, hexadecimal number system. For example, if data is represented in hexadecimal number system, each byte in data is represented by a hexadecimal value. The hexadecimal value is the key value of the byte in data.

[0013] The term "vector" is used to represent data using only unique key values of the data. A vector includes at least one value pair. Each value pair includes a first value and a second value. The first value represents one or more bytes in data having a unique key value, and the second value represents an instance of the one or more bytes having the unique key value present in the data. A vector including at least one value pair can include all unique key values that can exist in the data as the first value of the at least one value pair. For example, data can be represented in hexadecimal values. In this case, each value pair in the vector can include a first value having a unique hexadecimal value present in the hexadecimal data, and a second value representing an instance of the one or more bytes having the unique hexadecimal value present in the hexadecimal data. A vector including at least one value pair can include all unique hexadecimal values that can exist in the hexadecimal data as the first value.

[0014] The phrase "one or more data centers" is used to represent data centers capable of storing files and vectors thereof.

[0015] According to some embodiments of the present application, determining whether the vectorized data exists in the one or more data centers and returning a reference to the data to be stored according to the result of the determining step can further include comparing the first vector to vectors in the one or more data centers. If the first vector exists in the one or more data centers, determining whether the data to be stored is duplicated with a first file already existing in the one or more data centers. The first file corresponds to a vector having the same value pairs as the first vector. If the data to be stored is duplicated with the first file already existing in the one or more data centers, returning a reference to the user. The reference is a reference ID of the first file already existing in the one or more data centers.

[0016] According to some embodiments of the application, determining whether the vectorized data exists in the one or more data centers can further comprise comparing the first vector to vectors in the one or more data centers. If the first vector does not exist in any of the one or more data centers, or if the data to be stored does not duplicate a first file already existing in the one or more data centers, selecting a vector in the one or more data centers having a highest similarity value compared to the first vector. Determining whether the vectorized data exists in the one or more data centers can further comprise determining one or more duplicate byte sequences in the data to be stored and in a second file corresponding to the selected vector in the one or more data centers, wherein a length of the duplicate byte sequences is equal to or less than a first threshold. Returning the reference to the data to be stored can further comprise returning a first reference, the first reference comprising one or more sub-references, each sub-reference corresponding to a respective one of the duplicate byte sequences, each sub-reference comprising an index and a range of the duplicate byte sequence in the second file.

[0017] According to some embodiments of the application, the method can further comprise injecting one or more byte sequences not found in the second file, wherein a length of the not found byte sequences is less than a second threshold.

[0018] According to some embodiments of the application, the method can further comprise vectorizing each byte sequence not found in the second file to obtain a respective second vector, wherein a length of each byte sequence not found in the second file is equal to or greater than a second threshold. For each byte sequence, the following steps are performed: (i) determining whether the second vector exists in the one or more data centers. (ii) If the second vector exists in the one or more data centers, determining whether the byte sequence duplicates a third file already existing in the one or more data centers, wherein the third file corresponds to a vector having the same value pair as the second vector. (iii) If the byte sequence duplicates the third file, returning a second reference. The second reference is a reference ID of the third file, or comprises an index and a range of the duplicated byte in the third file. (iv) If the second vector does not exist in the one or more data centers, or if the byte sequence does not duplicate the third file, adding the byte sequence to the one or more data centers and returning a third reference. The third reference is a reference ID of the byte sequence that does not duplicate the third file.

[0019] According to some embodiments of the application, the returned reference to the data to be stored can comprise at least one of the first reference, the injected byte sequences, the one or more second references, and the one or more third references.

[0020] According to some embodiments of the application, the method can further comprise reassembling the data to be stored according to the references.

[0021] According to some embodiments of the present application, the breaking down of the data to be stored into bytes can comprise representing each byte in hexadecimal form.

[0022] According to a second aspect, a system for referencing data comprises a processor and a memory having instructions stored therein. The instructions, when executed by the processor, cause the processor to perform the method described above.

[0023] According to a third aspect, a non-transitory computer-readable storage medium having instructions stored thereon. The instructions, when executed by a computer, cause the computer to perform the method of the first aspect.

[0024] The method and system have the advantage that when data is to be stored or shared, what is returned is a reference rather than the complete data to be stored or shared. Thus, what is stored at the user terminal is only the reference to the data rather than the complete data, and the complete data can be obtained through the reference to the data. In this way, the data storage space at the user terminal is reduced. Furthermore, since all the data stored in the data center can be recycled and combined into different data, when data is to be added to the data center, there is no need to save the complete data each time, and the size of the data center can be controlled to be almost fixed.

[0025] The method and system can also further benefit from the concept that all the data stored in the data center can be recycled and combined into different data, thereby significantly improving the security of the data stored therein. BRIEF DESCRIPTION OF DRAWINGS

[0026] These and other aspects will now be described in more detail, with reference to the embodiments illustrated in the drawings.

[0027] Figure 1 An exemplary flowchart of a method for referencing data is shown, the method being performed by a processor in operative connection with one or more data centers in a network.

[0028] Figure 1 (a) An exemplary flowchart of a method for referencing data is shown, the method being additionally / alternatively performed by a processor in operative connection with one or more data centers in a network.

[0029] Figure 2 A system for referencing data is schematically shown.

[0030] All figures are schematic, not necessarily to scale, and generally only show parts of a device as necessary for illustration of the embodiments, other parts can be omitted or implied. DETAILED DESCRIPTION

[0031] Various aspects of the application are described more fully below with reference to the drawings, in which current, preferred embodiments are shown. This disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and fully convey the scope of the application to those skilled in the art.

[0032] Reference will now be made to the figures in which the various aspects of the application are shown. The following description is presented to enable any person skilled in the art to make and use the application. Descriptions of specific devices and applications are provided only as examples. Any such specific examples of components and applications are not intended to limit the scope of the application which is defined solely by the appended claims. Figure 1, a flowchart of a method 100 for referencing data is shown, the method is performed by a processor operatively connected to one or more data centers in a network. Each of the one or more data centers is configured to store files and respective vectors of the files. The method 100 includes: step 102, receiving a request including data to be stored. For example, the request to store data can be received from a user through an application in a mobile terminal or software on a computer, etc. In some embodiments, the data to be stored can be any suitable data type, such as text, image, sound, picture, video, or a combination of any of the types. For example, the data to be stored is a text message: "Hello world". In addition, the method also includes step 104, decomposing the data to be stored into a plurality of bytes by byte. In some embodiments, different number systems can be used to further represent the decomposed data to be stored. For example, decimal number system, binary number system, octal number system, hexadecimal number system. Each byte of the decomposed data is represented by a corresponding key value. For example, if the decomposed data is represented by hexadecimal number system, each byte in the data is represented by a hexadecimal value. The hexadecimal value is the key value of the byte in the data. Taking the text message "Hello world" as an example, when it is decomposed, the bytes can be represented by hexadecimal values as [48, 65, 6C, 6C, 6F, 20, 77, 6F, 72, 6C, 64], where "H" in the text message "Hello world" is represented by the corresponding hexadecimal value "48" in the decomposed data [48, 65, 6C, 6C, 6F, 20, 77, 6F, 72, 6C, 64]. Similarly, "l" in the text message "Hello world" appears three times, each time it appears in the text message, it is represented by the corresponding hexadecimal value "6C" in the decomposed data [48, 65, 6C, 6C, 6F, 20, 77, 6F, 72, 6C, 64]. In step 106, the decomposed data is vectorized to obtain a first vector of the data to be stored. The first vector includes at least one value pair. Each value pair includes a first value and a second value. The first value represents one or more bytes having a unique key value in the decomposed data. The second value represents the instance of the one or more bytes having a unique key value in the decomposed data. Taking the text message "Hello world" as an example, when the decomposed data is vectorized, a first vector {48=1, 65=1, 6C=3, 6F=2,...} is obtained. For example, the first vector {48=1, 65=1, 6C=3, 6F=2,...} includes a first value pair "48=1", where "48" is the first value in the first value pair "48=1", and "1" is the second value.The first value of the pair "48=1" in the first value pair "48=1" indicates one or more bytes of the decomposed hexadecimal representation data [48, 65, 6C, 6C, 6F, 20, 77, 6F, 72, 6C, 64] having the unique hexadecimal value "48". The second value of the pair "48=1" in the first value pair "48=1" indicates a unique instance of one or more bytes (i.e., H) of the decomposed hexadecimal representation data [48, 65, 6C, 6C, 6F, 20, 77, 6F, 72, 6C, 64] having the unique hexadecimal value "48". Likewise, the second value of the pair "6C=3" in the third value pair "6C=3" of the first vector {48=1, 65=1, 6C=3, 6F=2,...} indicates three instances of one or more bytes (i.e., l) of the decomposed hexadecimal representation data [48, 65, 6C, 6C, 6F, 20, 77, 6F, 72, 6C, 64] having the unique hexadecimal value "6C". The method further includes a step 108 of determining whether the vectorized data exists in one or more data centers. The determination can include checking the vectorized data in the one or more data centers for duplication. Different subsequent steps can be performed depending on the result of the determination. In some embodiments, the step can be implemented by, for example, determining whether all of the value pairs of the first vector exist in a vector of a file stored in the one or more data centers, and if so, checking whether the data to be stored is duplicated in the one or more data centers, which will be explained in more detail below in connection with Figure 1 (a) are explained in more detail. Other embodiments, for example, determine that the data to be stored is not duplicated in the one or more data centers, which will be explained in more detail below in connection with Figure 1 (a) are explained in more detail. Finally, in step 110, a reference to the data to be stored is returned depending on the result of the determination step 108. In some embodiments, returning the reference can include returning a reference to a file determined in the one or more data centers. According to some embodiments, the file determined in the one or more data centers includes all of the value pairs in the first vector, and the determined file is duplicated with the data to be stored. The reference can include, for example, a reference ID. The reference can be returned to the same or a different user, software on a computer, etc. This process will be explained in more detail below.

[0033] Reference is made to Figure 1 (a), a flowchart of a method 100 for referencing data is shown, which is additionally / alternatively performed by a processor operatively connected to one or more data centers in a network. Figure 1 (a) further details different scenarios that can occur when referencing data. Steps 102 to 108 are explained in more detail in connection with Figure 1The execution. In step 108, the method determines whether the vectorized data exists in the one or more data centers. The determination can further comprise: step 109, comparing the first vector with the vectors in the one or more data centers; and step 111, if it is determined in step 109 that the first vector exists in the one or more data centers, determining whether the data to be stored is duplicated with the first file already existing in the one or more data centers. The first file corresponds to the vectors having the same value pairs as the first vector. Take the text message "Hello world" as an example, in the example case, it is determined whether all the value pairs in the first vector {48=1, 65=1, 6C=3, 6F=2,...} exist in the vectors of the files stored in the one or more data centers. If it is determined that all the value pairs in the first vector exist in the vectors of the files stored in the one or more data centers, step 111, determining duplication, can be performed in any suitable manner. According to some embodiments, the duplication is determined by checking the order of the bytes in the data to be stored. In other words, the data to be stored is compared with the first file byte by byte to determine whether the order of the bytes in the data to be stored is the same as the order of the bytes in the first file. According to other embodiments, the duplication is determined by using a hash algorithm. In one case, if the order is the same, the data corresponding to the first vector is duplicated with the first file already existing in the one or more data centers. If the data to be stored is duplicated with the first file already existing in the one or more data centers, a reference is returned to the user in step 110. The reference can be a reference ID of the first file already stored in the one or more data centers. According to the present application, only the reference is returned and stored at the user end without the user's awareness. The method can further comprise step 122, reassembling the data to be stored according to the reference. The reassembling can comprise recovering the data to be stored according to the reference to the first file already stored in the one or more data centers. According to some embodiments, the user can use the reference to obtain the first file without the user's awareness through an application in the mobile terminal or software in the computer.

[0034] Back to step 108, further different scenarios of referencing data are explained. Step 109, comparing the first vector with vectors in one or more data centers. If the first vector is not found in any of the one or more data centers upon comparison, or the data to be stored is determined not to be duplicated in the first file in step 111, step 112 is performed, selecting a vector in the one or more data centers with the highest similarity value compared with the first vector. According to some embodiments, the similarity value between the first vector and the selected vector in the one or more data centers can be calculated using cosine similarity. The following explanation will be based on the similarity value calculated using cosine similarity. After calculation, a scalar value between 0 and 1 will be returned, indicating the similarity value. If the returned similarity value is 1, it indicates that the file corresponding to the selected vector in the one or more data centers is likely to be duplicated with the first vector. Various other suitable methods can also be used when calculating the similarity value. Therefore, the present application is not limited to calculating the similarity value based on cosine similarity. In step 113, one or more duplicated byte sequences in the data to be stored and in a second file corresponding to the vector selected in step 112 in the one or more data centers are determined. The length of the duplicated byte sequence is equal to or less than a first threshold value. The threshold value can be any numerical value. The second file corresponding to the selected vector can be different from or the same as the first file in the one or more data centers. The second file is the file corresponding to the selected vector with the highest similarity value in the one or more data centers. In step 114, a first reference of the second file is returned. The first reference includes one or more sub-references. Each sub-reference corresponds to a duplicated byte sequence respectively. Each sub-reference includes an index and a range of the duplicated byte sequence in the second file. The index can be a file ID, for example, the file ID of the second file, which can be represented as FileID = i02. The range can be defined using the starting byte number in the file corresponding to the selected vector (for example, the second file), and the length of the number of bytes in the file corresponding to the selected vector (for example, the second file), which is duplicated with one of the one or more duplicated byte sequences in the second file. The syntax of each sub-reference can be as follows: FileID_StartByte_length. In addition, when the length of the byte sequence not found in the second file is less than a second threshold value, step 121 is performed, injecting the one or more byte sequences not found in the second file. The injection can be performed as the injection of the byte or byte sequence when the reference is returned in step 110. Generally, the second threshold value is less than the first threshold value. For example, if the data is represented in hexadecimal notation, the injection can be performed by injecting a hexadecimal value for each byte in the one or more byte sequences not found in the second file. However, when the length of each byte sequence not found in the second file is equal to or greater than the second threshold value, step 115 is performed, vectorizing each byte sequence not found in the second file to obtain a respective second vector.The following steps 116-120 can be performed for each byte sequence. In step 116, it is determined whether the second vector exists in one or more data centers. The determination in step 116 can further include comparing the second vector with vectors in the one or more data centers. If the second vector exists in the one or more data centers, step 117 is performed, which determines whether the byte sequence has a duplicate in a third file already existing in the one or more data centers. The third file corresponds to vectors having the same value pairs as the second vector. The comparison and duplicate checking step is performed similarly as described above. If the byte sequence has a duplicate in the third file, step 118 is performed, which returns a second reference. The second reference can be a reference ID of the third file, or include an index and a range of the duplicate byte in the third file. Specifically, when the byte sequence duplicates the entire third file, the second reference can be a reference ID of the third file. When the byte sequence duplicates a portion of the third file, the second reference can include an index and a range of the duplicate byte in the third file. However, if the second vector does not exist in the one or more data centers, or the byte sequence does not have a duplicate in the third file, step 119 is performed, which adds the byte sequence to the one or more data centers. Thus, in step 120, a third reference of the byte sequence added to the one or more data centers is returned. The third reference can be a reference ID of the byte sequence without a duplicate in the third file. The reference 110 of the data to be stored returned can include at least one of the first reference, the injected byte sequence, the one or more second references, and the one or more third references. The method 100 can further include step 122, which reassembles the data to be stored according to the references. The reassembling can include recovering the data to be stored according to the references already stored in the one or more data centers.

[0035] Referring to Figure 2 According to one embodiment, a system 200 for referencing data includes a processor 202 and a memory 204. The processor 202 in the system 200 is operatively connected with one or more data centers 206a, 206b, 206c, 206d in a network. One or more data centers (e.g., 206a, 206b) can be located in a remote network, such as a network connected through the Internet or a cloud network, etc. One or more data centers (e.g., 206c, 206d) can also be located in a local network. Each of the one or more data centers 206a, 206b, 206c, 206d is configured to store files and corresponding vectors of the files. The memory 204 has instructions stored thereon. The instructions, when executed by the processor 202, cause the processor 202 to perform the method 100 for referencing data as described in connection with the preceding figures. The memory 204 is shown as being connected to the processor 202 independently. However, those skilled in the art will understand that the memory can be built-in in the processor 202, or arranged to be configured outside the system 200.

[0036] The memory 204 can be a non-transitory computer readable storage medium having instructions stored thereon. When the instructions are executed by the computer, the computer is caused to perform the method 100 for referencing data described in connection with any one or combination of Figure 1 and 1(a)

[0037] Those skilled in the art will realize that the application is in no way limited to the preferred embodiments described above. On the contrary, a number of modifications and variations are possible within the scope of the appended claims. For example, the processors performing the method of referencing data are shown as a single processor. However, they can also perform certain parts of the method collectively as a group of processors. Thus, the embodiments presented in the disclosure are for example purposes only and should not be construed as limiting the scope.​

Claims

1. A method for referencing data, performed by a processor operatively connected to one or more data centers in a network, each of the one or more data centers configured to store files and respective vectors of files, the method comprising: receiving a request including data to be stored; breaking the data to be stored into a plurality of bytes by byte, each byte of data represented by a key value; vectorizing the data broken by byte to obtain vectorized data, the vectorized data including a first vector of the data to be stored, wherein the first vector includes at least one value pair, each value pair including a first value and a second value, wherein the first value represents one or more bytes having a unique key value among the data broken by byte, and the second value represents instances of the one or more bytes having the unique key value present among the data broken by byte; determining whether the first vector is stored in the one or more data centers; and returning a reference of the data to be stored according to a result of the determination.

2. The method of claim 1, wherein, Determining whether the first vector is stored in the one or more data centers and returning the reference of the data to be stored according to a result of the determination further comprises: comparing the first vector with existing vectorized data stored in the one or more data centers; if the first vector is stored in the one or more data centers, determining whether the data to be stored is duplicated with a first file already existing in the one or more data centers, wherein the first file corresponds to a vector having the same value pair as the first vector; if the data to be stored is duplicated with the first file already existing in the one or more data centers, returning a reference to a user, the reference being a reference identifier ID of the first file already existing in the one or more data centers.

3. The method of claim 1, wherein, Determining whether the first vector is stored in the one or more data centers and returning the reference of the data to be stored according to a result of the determination further comprises: comparing the first vector with existing vectorized data stored in the one or more data centers; if the first vector does not exist in any of the one or more data centers, or if the data to be stored is not duplicated with a first file already existing in the one or more data centers, selecting a vector in the one or more data centers having a highest similarity value with the first vector; determining one or more duplicated byte sequences in the data to be stored and in a second file corresponding to the selected vector in the one or more data centers, wherein a length of the duplicated byte sequence is equal to or less than a first threshold value; and returning a first reference, the first reference including one or more sub-references, each sub-reference corresponding to one of the duplicated byte sequences respectively, and each sub-reference including an index and a range of the duplicated byte sequence in the second file.

4. The method of claim 3, further comprising: injecting one or more byte sequences not found in the second file, wherein a length of the not found byte sequence is less than a second threshold value.

5. The method of claim 4, further comprising: vectorizing each byte sequence not found in the second file to obtain a respective second vector, wherein each byte sequence not found in the second file has a length equal to or greater than the second threshold; performing the following (i) to (iv) on each byte sequence: (i) determining whether the second vector is stored in the one or more data centers; (ii) if the second vector is stored in the one or more data centers, determining whether the byte sequence has a duplicate in a third file already in the one or more data centers, wherein the third file corresponds to a vector having the same value pair as the second vector; (iii) if the byte sequence has a duplicate in the third file, returning a second reference, the second reference being a reference ID of the third file or including an index and a range of the duplicate byte in the third file; (iv) if the second vector does not exist in the one or more data centers or if the byte sequence does not have a duplicate in the third file, adding the byte sequence to the one or more data centers and returning a third reference, the third reference being a reference identification ID of the byte sequence without a duplicate in the third file.

6. The method of claim 5, wherein, The returned references of the data to be stored include at least one of the first reference, the injected byte sequence, one or more of the second references, and one or more of the third references.

7. The method of claim 1, further comprising reassembling the data to be stored according to the references.

8. The method of claim 1, wherein, Decomposing the data to be stored by byte includes representing each byte in hexadecimal form.

9. A system for referencing data, comprising a processor and a memory, the processor being operatively connected with one or more data centers in a network, each of the one or more data centers being configured to store files and respective vectors of the files, the memory storing instructions that, when executed by the processor, cause the processor to perform a method, the method comprising: receiving a request including data to be stored; decomposing the data to be stored by byte into a plurality of bytes, each byte of data being represented by a key value; vectorizing the decomposed data to obtain vectorized data, the vectorized data including a first vector of the data to be stored, wherein the first vector includes at least one value pair, each value pair including a first value and a second value, wherein the first value represents one or more bytes having a unique key value in the decomposed data, and the second value represents an instance of the one or more bytes having a unique key value present in the decomposed data; determining whether the first vector is stored in the one or more data centers; and returning references of the data to be stored according to a result of the determination.

10. The system of claim 9, wherein, Determining whether the first vector is stored in the one or more data centers and returning references of the data to be stored according to a result of the determination further comprise: comparing the first vector with existing vectorized data stored in the one or more data centers; if the first vector is stored in the one or more data centers, determining whether the data to be stored is duplicated in a first file already existing in the one or more data centers, wherein the first file corresponds to a vector having the same pair of values as the first vector; if the data to be stored is duplicated in the first file already existing in the one or more data centers, returning a reference to the user, the reference being a reference identifier ID of the first file already existing in the one or more data centers.

11. The system of claim 9, wherein, determining whether the first vector is stored in the one or more data centers and returning the reference of the data to be stored according to the determined result further comprises: comparing the first vector with existing vectorized data stored in the one or more data centers; if the first vector does not exist in any of the one or more data centers, or if the data to be stored is not duplicated in a first file already existing in the one or more data centers, selecting a vector in the one or more data centers having a highest similarity value with the first vector; determining one or more duplicated byte sequences in the data to be stored and in a second file corresponding to the selected vector in the one or more data centers, wherein the duplicated byte sequences have a length equal to or less than a first threshold value; and returning a first reference, the first reference comprising one or more sub-references, each sub-reference corresponding to one of the duplicated byte sequences respectively, and each sub-reference comprising an index and a range of the duplicated byte sequence in the second file.

12. The system of claim 11, further comprising: injecting one or more byte sequences not found in the second file, wherein the not found byte sequences have a length less than a second threshold value.

13. The system of claim 12, further comprising: vectorizing each byte sequence not found in the second file to obtain a respective second vector, wherein each byte sequence not found in the second file has a length equal to or greater than the second threshold value; and performing the following (i) to (iv) on each byte sequence: (i) determining whether the second vector is stored in the one or more data centers; (ii) if the second vector is stored in the one or more data centers, determining whether the byte sequence is duplicated in a third file already existing in the one or more data centers, wherein the third file corresponds to a vector having the same pair of values as the second vector; (iii) if the byte sequence is duplicated in the third file, returning a second reference, the second reference being a reference ID of the third file or comprising an index and a range of the duplicated byte in the third file; (iv) if the second vector does not exist in the one or more data centers, or if the byte sequence does not repeat in the third file, then adding the byte sequence to the one or more data centers and returning a third reference, the third reference being a reference ID of the byte sequence that does not repeat in the third file.

14. The system of claim 13, wherein, The returned reference of the data to be stored includes at least one of the first reference, the injected byte sequence, one or more of the second references, and one or more of the third references.

15. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a processor, cause the processor to perform a method, the processor being in operative connection with one or more data centers in a network, each data center of the one or more data centers being configured to store files and respective vectors of files, the method comprising: receiving a request including data to be stored; breaking the data to be stored into bytes, each byte of data being represented by a key value; vectorizing the broken-down data to obtain vectorized data, the vectorized data including a first vector of the data to be stored, wherein the first vector includes at least one value pair, each value pair including a first value and a second value, wherein the first value represents one or more bytes having a unique key value in the broken-down data, and the second value represents an instance of the one or more bytes having a unique key value that appears in the broken-down data; determining whether the first vector is stored in the one or more data centers; and returning a reference of the data to be stored according to a result of the determination.

16. The non-transitory computer-readable storage medium of claim 15, wherein, Determining whether the first vector is stored in the one or more data centers and returning a reference of the data to be stored according to a result of the determination further comprises: comparing the first vector with existing vectorized data stored in the one or more data centers; if the first vector is stored in the one or more data centers, determining whether the data to be stored is duplicated with an existing first file in the one or more data centers, wherein the first file corresponds to a vector having the same value pair as the first vector; if the data to be stored is duplicated with the existing first file in the one or more data centers, returning a reference to a user, the reference being a reference identification ID of the existing first file in the one or more data centers.

17. The non-transitory computer-readable storage medium of claim 15, wherein, Determining whether the first vector is stored in the one or more data centers and returning a reference of the data to be stored according to a result of the determination further comprises: comparing the first vector with existing vectorized data stored in the one or more data centers; if the first vector does not exist in any of the one or more data centers, or if the data to be stored is not duplicated with an existing first file in the one or more data centers, selecting a vector in the one or more data centers that has a highest similarity value with the first vector; determining one or more repeated byte sequences in the to-be-stored data and in a second file corresponding to a selected vector in the one or more data centers, wherein a length of the repeated byte sequence is equal to or less than a first threshold; and returning a first reference, the first reference including one or more sub-references, each sub-reference corresponding to a repeated byte sequence in the repeated byte sequences, and each sub-reference including an index and a range of the repeated byte sequence in the second file.

18. The non-transitory computer-readable storage medium of claim 17, further comprising: injecting one or more byte sequences not found in the second file, wherein a length of the not-found byte sequence is less than a second threshold.

19. The non-transitory computer-readable storage medium of claim 18, further comprising: vectorizing each byte sequence not found in the second file to obtain a respective second vector, wherein a length of each byte sequence not found in the second file is equal to or greater than the second threshold; and performing the following (i) to (iv) on each byte sequence: (i) determining whether the second vector is stored in the one or more data centers; (ii) if the second vector is stored in the one or more data centers, determining whether the byte sequence is repeated in a third file already in the one or more data centers, wherein the third file corresponds to a vector having a same value pair as the second vector; (iii) if the byte sequence is repeated in the third file, returning a second reference, the second reference being a reference ID of the third file or including an index and a range of the repeated byte in the third file; (iv) if the second vector does not exist in the one or more data centers or if the byte sequence is not repeated in the third file, adding the byte sequence to the one or more data centers and returning a third reference, the third reference being a reference ID of the byte sequence not repeated in the third file.

20. The non-transitory computer-readable storage medium of claim 19, wherein, The returned references of the to-be-stored data include at least one of the first reference, the injected byte sequence, the one or more second references, and the one or more third references.

Images