Data storage and search method and device

Abstract

The embodiment of the application provides a data storage and searching method and device, the method comprises the following steps: obtaining coding data representing the association relationship between first data and second data; storing the first data into a first field of a data table and storing the coding data into a second field of the data table which is associated with the first field; it can be seen that, in the first aspect, the data is stored into the same table in the scheme, and multiple tables do not need to be established and maintained, which is convenient for searching; in the second aspect, the first data and the second data associated therewith are not stored into a table together in the scheme, but the first data and the coding data representing the association relationship are stored into a table together, so that the association relationship between the first data and the second data can be directly obtained, and the association relationship between the tables does not need to be established again, which is further convenient for searching.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data storage and searching method and device. BACKGROUND

[0002] When storing associated data, the association between data is usually reflected. For example, in the scenario of machine learning with image as training data, algorithm information and image information are associated data, and the calling of each algorithm to the image needs to be reflected when storing the two kinds of associated data.

[0003] At present, the associated data is usually stored in different tables, and then the association between the tables is established. For example, image information is stored in table 1, algorithm information is stored in table 2, and the association between table 1 and table 2 is established.

[0004] However, in this scheme, multiple tables need to be established and maintained to store associated data, which is not convenient for searching. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a data storage and searching method and device to facilitate searching.

[0006] To achieve the above purpose, the embodiments of the present application provide a data storage method, comprising:

[0007] obtaining first data to be stored and second data associated with the first data;

[0008] obtaining code data representing the association between the first data and the second data;

[0009] storing the first data to a first field of a data table, and storing the code data to a second field of the data table associated with the first field.

[0010] Optionally, the obtaining code data representing the association between the first data and the second data comprises:

[0011] obtaining initial binary data of a preset bit number; each bit in the initial binary data corresponds to each second type of data, and the second type is the type of the second data;

[0012] adjusting the bit corresponding to the second data in the initial binary data from an initial value to a preset value to obtain code data representing the association between the first data and the second data.

[0013] Optionally, the first data is training data of machine learning, the second type of data is algorithm information used by the machine learning, the second data is algorithm information calling the training data, and each bit value in the initial binary data corresponds to an algorithm used by the machine learning.

[0014] Optionally, the first data is of a first type, and the second data is of a second type.

[0015] Optionally, the method further comprises:

[0016] Optionally, the method further comprises:

[0017] Optionally, the method further comprises:

[0018] Optionally, the method further comprises:

[0019] Optionally, the method further comprises:

[0020] Optionally, the method further comprises:

[0021] Optionally, the method further comprises:

[0022] Optionally, the method further comprises:

[0023] Optionally, the method further comprises:

[0024] Optionally, the method further comprises:

[0025] Optionally, the method further comprises:

[0026] Optionally, the method further comprises:

[0027] To achieve the above object, the embodiment of the present application further provides a data searching method, comprising:

[0028] receiving a searching instruction, wherein the searching instruction comprises second data;

[0029] determining a data table corresponding to the searching instruction, wherein the data table comprises a plurality of table entries, and each table entry stores first data and encoded data, and the encoded data represents an association between the first data and the second data;

[0030] reading the encoded data in the table entry in the data table, and judging whether the read encoded data and the second data satisfy a matching rule, and if yes, determining the first data in the table entry as searched data.

[0031] Optionally, the second data is second-type data, and the encoded data represents an association between the first data and each second-type data.

[0032] Optionally, the encoded data is binary data with a preset bit number, and each bit in the encoded data corresponds to each second-type data.

[0033] The judging whether the read encoded data and the second data satisfy the matching rule comprises:

[0034] converting the second data into binary data with the same bit number as the encoded data as converted data;

[0035] performing logical AND operation on the read encoded data and the converted data to obtain an operation result;

[0036] judging whether the operation result is a preset value.

[0037] Optionally, the searching instruction is an instruction for searching training data for a target algorithm, the second data is target algorithm information, the first data is training data for machine learning, the second-type data is various algorithm information used for machine learning, and each bit value in the encoded data corresponds to one algorithm used for machine learning.

[0038] Optionally, the reading the encoded data in the table entry comprises reading subfragments of the encoded data stored in different fields of the table entry in parallel.

[0039] The performing logical AND operation on the read encoded data and the converted data to obtain an operation result comprises: determining target bits in the converted data to which the subfragments are mapped in parallel, and performing logical AND operation on each subfragment and the target bit to which the subfragment is mapped in parallel to obtain an operation result.

[0040] The judging whether the operation result is the preset value comprises judging whether all the obtained operation results are the preset value.

[0041] To achieve the above object, the embodiment of the present application further provides a data storage device, comprising:

[0042] The first obtaining module is configured to obtain first data to be stored and second data associated with the first data.

[0043] The second obtaining module is configured to obtain encoding data representing an association relationship between the first data and the second data.

[0044] The storage module is configured to store the first data to a first field of a data table and store the encoding data to a second field of the data table associated with the first field.

[0045] Optionally, the second obtaining module is specifically configured to:

[0046] obtain initial binary data with a preset bit number; each bit in the initial binary data corresponds to each second type of data, and the second type is the type of the second data.

[0047] adjust a bit in the initial binary data corresponding to the second data from an initial value to a preset value to obtain the encoding data representing the association relationship between the first data and the second data.

[0048] Optionally, the first data is training data of machine learning, the second type of data is various algorithm information used by machine learning, the second data is algorithm information calling the training data, and each bit value in the initial binary data corresponds to one algorithm used by machine learning.

[0049] Optionally, the first data is of a first type, and the second data is of a second type; the device further comprises:

[0050] The statistical module is configured to count various candidate association relationships between the first type of data and the second type of data.

[0051] The encoding module is configured to encode the various candidate association relationships according to a preset encoding mode to obtain encoding data corresponding to the various candidate association relationships.

[0052] The second obtaining module is specifically configured to determine an association relationship between the first data and the second data as a to-be-matched relationship and find, in the encoding data corresponding to the various candidate association relationships, encoding data corresponding to the to-be-matched relationship.

[0053] Optionally, the storage module is specifically used for:

[0054] splitting the encoded data to obtain multiple sub-segments;

[0055] storing the multiple sub-segments into different fields of the data table associated with the first field respectively.

[0056] Optionally, the device further comprises:

[0057] a first judging module, configured to judge whether the data table stores a table entry matching the first data; if yes, trigger the updating module, and if no, trigger the storage module;

[0058] the updating module, configured to perform logical or operation on the encoded data and the data of the second field in the table entry to obtain an operation result, and replace the data of the second field in the table entry with the operation result.

[0059] To achieve the above object, the embodiment of the present application further provides a data searching device, comprising:

[0060] a receiving module, configured to receive a searching instruction, wherein the searching instruction comprises second data;

[0061] a first determining module, configured to determine a data table corresponding to the searching instruction, wherein the data table comprises multiple table entries, and each table entry stores first data and encoded data corresponding thereto, and the encoded data represents an association relationship between the first data and the second data;

[0062] a reading module, configured to read the encoded data in each table entry in the data table;

[0063] a second judging module, configured to judge whether the read encoded data and the second data satisfy a matching rule; if yes, trigger a second determining module;

[0064] the second determining module, configured to determine the first data in the table entry as searched data.

[0065] Optionally, the second data is second-type data, and the encoded data represents an association relationship between the first data and each kind of second-type data.

[0066] Optionally, the encoded data is binary data with a preset bit number, and each bit in the encoded data corresponds to each kind of second-type data respectively.

[0067] The second judging module is specifically configured to: convert the second data into binary data with the same number of bits as the encoded data as conversion data; perform logical AND operation on the read encoded data and the conversion data to obtain an operation result; and judge whether the operation result is a preset value.

[0068] Optionally, the searching instruction is an instruction for searching training data for a target algorithm, the second data is target algorithm information, the first data is training data for machine learning, the second type of data is algorithm information used by machine learning, and each bit value in the encoded data corresponds to an algorithm used by machine learning.

[0069] Optionally, the reading module is specifically configured to: read each sub-fragment of the encoded data stored in different fields of the table item in parallel.

[0070] The second judging module is specifically configured to: convert the second data into binary data with the same number of bits as the encoded data as conversion data; determine target bits in the conversion data to which each sub-fragment is mapped in parallel, perform logical AND operation on each sub-fragment and the target bit to which the sub-fragment is mapped in parallel to obtain an operation result; and judge whether the obtained operation result is a preset value.

[0071] To achieve the above object, an embodiment of the present application further provides an electronic device, which comprises a processor and a memory.

[0072] The memory is used for storing a computer program.

[0073] The processor is used for executing the program stored on the memory to realize any one of the above data storage and searching methods.

[0074] According to the embodiment of the present application, the encoded data representing the association relationship between the first data and the second data is obtained; the first data is stored in a first field of a data table, and the encoded data is stored in a second field of the data table associated with the first field; thus, in the first aspect, the data is stored in the same table in the present solution, and multiple tables do not need to be established and maintained, which is convenient for searching; in the second aspect, the first data and the second data associated with the first data are not stored in a table together, but the first data and the encoded data representing the association relationship are stored in a table together, so that the association relationship between the first data and the second data can be directly obtained, and the association relationship between tables does not need to be established, which is further convenient for searching.

[0075] Of course, implementing any product or method of the present application does not necessarily need to achieve all the advantages described above at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description only represent some embodiments of the present application, and for those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0077] Figure 1 The first flowchart of the data storage method provided by the embodiment of the present application;

[0078] Figure 2 The first flowchart of the data storage method provided by the embodiment of the present application;

[0079] Figure 3 The flowchart of the data searching method provided by the embodiment of the present application;

[0080] Figure 4 The structural diagram of the data storage device provided by the embodiment of the present application;

[0081] Figure 5 The structural diagram of the data searching device provided by the embodiment of the present application;

[0082] Figure 6 The structural diagram of the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0083] The technical solutions in the embodiments of the present application will be described clearly and completely below with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the scope of protection of the present application.

[0084] In order to achieve the above-mentioned purpose, the embodiments of the present application provide a data storage and searching method and device, which can be applied to various electronic devices, and the specific application is not limited. The data storage method will be described in detail first.

[0085] Figure 1 The first flowchart of the data storage method provided by the embodiment of the present application includes:

[0086] S101: obtaining the first data to be stored and the second data associated with the first data.

[0087] For the sake of distinction, the two kinds of acquired associated data are respectively referred to as first data and second data. Taking the machine learning scenario as an example, the first data can be training data such as image, voice data, text data, etc., and the specific implementation is not limited; the second data can be algorithm information used in the machine learning process, such as the name, ID (Identity Document, identity number), serial number, etc. of the algorithm, which can uniquely identify the algorithm, and the specific implementation is not limited.

[0088] The second data associated with the first data can be understood as algorithm information calling the training data. In the machine learning scenario, it is usually necessary to record the calling of each algorithm to the training data. For example, the first data acquired in S101 can be image A, and the second data can be various algorithm information calling image A: algorithm 101 and algorithm 102.

[0089] S102: Acquire encoding data representing the association relationship between the first data and the second data.

[0090] The encoding data can be data in a preset format. In an embodiment, the encoding data is binary data with a preset number of bits; for the sake of convenience, the type of the second data is referred to as the second type; each bit in the binary data corresponds to each type of the second data.

[0091] If the types of the second data are known, and the association relationship between the first data and the second type of data includes association and non-association, in this scenario, the number of bits of the binary data (related to the types of the second data) can be preset, and 0 and 1 are used to represent the association relationship: 1 represents association, and 0 represents non-association. This embodiment can be applied in this scenario.

[0092] For example, assuming that there are three types of second data: X, Y, and Z, the encoding data can be three-bit binary data, the first bit (counting from right to left) in the binary data represents whether the first data is associated with X, the second bit represents whether the first data is associated with Y, and the third bit represents whether the first data is associated with Z. The association can be represented as 1, and the non-association can be represented as 0.

[0093] In this embodiment, S102 can include: acquiring initial binary data with a preset number of bits; each bit in the initial binary data corresponds to each type of the second data, and the second type is the type of the second data; and adjusting the bit corresponding to the second data in the initial binary data from an initial value to a preset value to obtain the encoding data representing the association relationship between the first data and the second data.

[0094] If the first data is associated with the second type of data, represented as 1, or not associated, represented as 0, the initial value of each bit in the initial binary data can be 0. Assuming that the second data associated with the first data obtained in S101 is X and Y, the bits corresponding to X and Y in the initial binary data are adjusted from 0 to 1 to obtain 011 as the encoding data. Taking another example, assuming that the second data associated with the first data obtained in S101 is X and Z, the bits corresponding to X and Z in the initial binary data are adjusted from 0 to 1 to obtain 101 as the encoding data.

[0095] By applying the present embodiment, the encoding data can represent the association between the first data and multiple second types of data; in this way, in the subsequent search process, the association between multiple types of data can be determined based on the same data table, further facilitating the search.

[0096] Still taking the machine learning scenario as an example, the first data is: the training data of machine learning, the second type of data is: various algorithm information used in machine learning, and the second data is: algorithm information calling the training data. Each bit value in the initial binary data corresponds to an algorithm used in machine learning.

[0097] Continuing the above example, assuming that the first data obtained in S101 is image A, and assuming that the second type of data includes various algorithms used in the machine learning process: algorithm 101, algorithm 102, and algorithm 103. Assuming that the encoding data is a three-bit binary data, the first bit (from right to left) in the binary data represents whether algorithm 101 has called image A, the second bit represents whether algorithm 102 has called image A, and the third bit represents whether algorithm 103 has called image A.

[0098] If the algorithm calls image A, it is represented as 1, otherwise it is represented as 0. The initial value of each bit in the initial binary data can be 0. Continuing the above example, assuming that the second data associated with the first data obtained in S101 is algorithm 101 and algorithm 102, the first bit and the second bit in the initial binary data are adjusted from 0 to 1 to obtain 011 as the encoding data.

[0099] As can be seen, the encoding data can represent the calling situation of various algorithms on the training data, so that in the subsequent search process, based on the same data table, the training data can be searched for various algorithms or combinations of various algorithms, improving the search efficiency.

[0100] In another implementation, for the convenience of description, the type of the first data is referred to as a first type, and the type of the second data is referred to as a second type; various candidate association relationships between the first type of data and the second type of data can be counted; and the various candidate association relationships are encoded according to a preset encoding manner to obtain encoding data corresponding to the various candidate association relationships. In this way, S102 can include: determining an association relationship between the first data and the second data as a to-be-matched relationship; and searching for, in the encoding data corresponding to the various candidate association relationships, encoding data corresponding to the to-be-matched relationship.

[0101] Still taking the machine learning scenario as an example, the first type of data is various training data, and the second type of data is various algorithm information, and various candidate association relationships between the training data and the algorithms are counted. For example, it is assumed that three algorithms 101, 102, and 103 are used in the machine learning process, and for each piece of training data, it can be called by the algorithm 101, can be called by the algorithm 102, can be called by the algorithm 103, can be called by the algorithm 101 and the algorithm 102, can be called by the algorithm 101 and the algorithm 103, can be called by the algorithm 102 and the algorithm 103, and can be called by the algorithm 101, the algorithm 102, and the algorithm 103. The various possible calling relationships listed here are various candidate association relationships between the training data and the algorithms.

[0102] The various candidate association relationships (calling relationships) are encoded according to a preset encoding manner. The encoding here can be understood as assigning an identifier to the various candidate association relationships to distinguish these association relationships. The specific identifier type is not limited, for example, a serial number. If the training data is called by the algorithm 101, the encoding data is 1; if the training data is called by the algorithm 102, the encoding data is 2; if the training data is called by the algorithm 103, the encoding data is 3; if the training data is called by the algorithm 101 and the algorithm 102, the encoding data is 4; if the training data is called by the algorithm 101 and the algorithm 103, the encoding data is 5; if the training data is called by the algorithm 102 and the algorithm 103, the encoding data is 6; and if the training data is called by the algorithm 101, the algorithm 102, and the algorithm 103, the encoding data is 7.

[0103] It is assumed that the second data associated with the first data obtained in S101 is the algorithm 101 and the algorithm 102, and the encoding data corresponding to the algorithm 101 and the algorithm 102 is found to be 4. Taking another example, it is assumed that the second data associated with the first data obtained in S101 is the algorithm 101 and the algorithm 103, and the encoding data corresponding to the algorithm 101 and the algorithm 102 is found to be 5. And so on, which will not be listed one by one.

[0104] S103: store the first data to a first field of the data table, and store the encoded data to a second field of the data table associated with the first field.

[0105] The first data obtained in S101 and the encoded data obtained in S102 can be stored in different fields of the same table entry. Still taking the machine learning scenario as an example, the first data can be training data, and the second data can be various algorithm information used in the machine learning process. In this case, as shown in Table 1, the ID, storage path, data type, and creation time of the first data and the encoded data can be stored in different fields of the same table entry.

[0106] Table 1

[0107]

[0108] Other data can also be stored in the data table, such as a label of the first data, and the like, without limitation. In Table 1, 011 indicates that the algorithm 101 and the algorithm 102 are called, and 111 indicates that the algorithm 101, the algorithm 102, and the algorithm 103 are called.

[0109] In an embodiment, S103 can include: splitting the encoded data to obtain multiple sub-segments; and respectively storing the multiple sub-segments to different fields of the data table associated with the first field.

[0110] In the above embodiment, the encoded data is binary data of a preset number of bits. If the number of bits of the binary data is large, the binary data can be split into multiple sub-segments. For example, assuming that the encoded data is binary data of 100 bits, the binary data can be split into 10 sub-segments, each of which is binary data of 10 bits, and the 10 sub-segments are respectively stored in different fields. In this way, when the encoded data is read subsequently, the 10 sub-segments can be processed in parallel, the processing speed is improved, and the search efficiency is further improved.

[0111] Application of the present application Figure 1 In the embodiment, the first data and the second data associated with the first data are stored in the same table, without the need to establish and maintain multiple tables, facilitating search and reducing the amount of data storage. In the second aspect, the first data and the encoded data representing the association relationship are stored in the same table, without the need to establish the association relationship between tables, further facilitating search. In the third aspect, in an embodiment, the encoded data is split into multiple sub-segments for storage. In this way, when the encoded data is read subsequently, the sub-segments can be processed in parallel, the processing speed is improved, and the search efficiency is further improved.

[0112] Figure 2 The second flowchart of the data storage method provided by the embodiment of the present application is shown in FIG. 2. Figure 2 The embodiment shown mainly introduces the updating manner of the data table in the case of binary data as the encoding data. Figure 2 The embodiment shown includes the following steps.

[0113] S201: Obtain first data to be stored and second data associated with the first data.

[0114] S202: Obtain initial binary data with a preset number of bits; each bit in the initial binary data corresponds to each type of second data, and the second type is the type of the second data.

[0115] For the convenience of description, the type of the second data is referred to as the second type. If the types of the second type of data are known, and the association between the first data and the second type of data includes association and non-association, in this scenario, the number of bits of the binary data (related to the types of the second type of data) can be preset, and 0 and 1 are used to represent the association: 1 represents association, and 0 represents non-association. The initial value of each bit of the initial binary data can be 0.

[0116] Still taking the machine learning scenario as an example, the first data is training data for machine learning, the second type of data is various algorithm information used by machine learning, the second data is algorithm information for calling the training data, and each bit value in the initial binary data corresponds to one algorithm used by machine learning.

[0117] S203: Adjust the bit in the initial binary data corresponding to the second data from the initial value to a preset value to obtain encoding data representing the association between the first data and the second data.

[0118] Suppose that the first data obtained in S201 is image 1, and suppose that the second type of data includes various algorithms used in the machine learning process: algorithm 101, algorithm 102, and algorithm 103. Suppose that the encoding data is three-bit binary data, the first bit (counted from right to left) in the binary data represents whether algorithm 101 calls image 1, the second bit represents whether algorithm 102 calls image 1, and the third bit represents whether algorithm 103 calls image 1.

[0119] If the algorithm calls image 1, it is represented as 1, otherwise as 0, and the initial value of each bit in the initial binary data is 0. Suppose that the second data associated with the first data obtained in S201 is algorithm 103, then adjust the third bit in the initial binary data from 0 to 1 to obtain 100 as the encoding data.

[0120] S204: judging whether a table entry matching the first data is stored in the data table; if yes, performing S205, if no, performing S206.

[0121] S204 is to judge whether the first data is stored in the data table, if yes, performing S205 to update the table entry in which the first data is stored, if no, performing S206 to establish a new table entry in the data table to store the first data and the encoding data thereof.

[0122] S205: performing logical OR operation between the encoding data and the data in the second field of the table entry to obtain an operation result; and replacing the data in the second field of the table entry with the operation result.

[0123] The data in the second field of the table entry is the encoding data stored in the table entry, and the encoding data stored in the table entry can be updated by applying the embodiment.

[0124] In the logical OR operation, if one or more of the operands are true (1), the operation result is true; only if all the operands are 0, the operation result is 0.

[0125] Suppose that the embodiment shown in Figure 2 Before the embodiment shown in Figure 2 is executed, the image 1 has been called by the algorithm 101 and the algorithm 102, and the calling of the image 1 by the algorithm 101 and the algorithm 102 forms the first table entry in the above table 1. In the subsequent machine learning process, the image 1 is called by the algorithm 103, and the embodiment shown in

[0126] S206: storing the first data to the first field of the data table and storing the encoding data to the second field of the data table associated with the first field.

[0127] By applying the embodiment shown in Figure 2 , the first aspect is to update the encoding data in the original table entry for the first data that has been stored, and to establish a new table entry for the first data that has not been stored; in this way, each first data only occupies one table entry, facilitating the searching. The second aspect is to perform only simple bit operation (logical OR operation) to update the table entry, and the updating efficiency is relatively high.

[0128] If the above-mentioned embodiment of "encoding various candidate association relationships according to a preset encoding mode" is adopted, subsequent update of the stored encoding data is needed, the association relationship between the first data and the second type of data can be re-determined, and new encoding data is generated based on the re-determined association relationship.

[0129] For example, image 1 has been called by algorithm 101 and algorithm 102, and continuing the above example, the encoding data is a serial number, and the encoding data corresponding to image 1 is 4. For the calling of image 1 by algorithm 101 and algorithm 102, a table entry is formed in the data table:

[0130] First data: image 1 Encoded data: 4

[0131] In the subsequent machine learning process, algorithm 103 also calls image 1, that is, algorithm 101, algorithm 102 and algorithm 103 call image 1, and the encoding data in the above table entry is modified from 4 to 7. In this way, each piece of first data only occupies one table entry, which is convenient for searching.

[0132] Figure 3 A flowchart of a data searching method provided by an embodiment of the present application is shown in the figure, which includes:

[0133] S301: receiving a search instruction, the search instruction including second data.

[0134] Still taking the machine learning scenario as an example, the search instruction can be an instruction for searching training data for a target algorithm, and the second data is target algorithm information. The calling of each algorithm on the training data can be recorded in advance, so that when an algorithm is used, the training data that has not been called by the algorithm (target algorithm) can be searched, so as to reduce the repeated calling of a piece of training data by an algorithm and improve the training efficiency.

[0135] S302: determining a data table corresponding to the search instruction, the data table including a plurality of table entries, and the table entries correspondingly store first data and encoding data, the encoding data representing the association relationship between the first data and the second data.

[0136] The data table corresponding to the search instruction, that is, the data table storing the association relationship between the second data and the first data, can be stored in advance by using the storage method provided by the embodiment of the present application. Figure 1 The data table in the storage method provided by the embodiment of the present application does not store the first data and the second data correspondingly, but stores the first data and the encoding data representing the association relationship correspondingly.

[0137] For convenience of description, the type of the second data is referred to as a second type; in an implementation, the encoding data indicates an association between the first data and the data of each second type.

[0138] Continuing the above example, the search instruction is an instruction for searching training data for a target algorithm, and in this case, the data of the second type is information of various algorithms used by machine learning, and the first data is training data for machine learning. In this implementation, the encoding data can indicate the calling of the training data by each algorithm, so that based on the same data table, training data can be searched for various algorithms or combinations of various algorithms, thereby improving the search efficiency.

[0139] Therefore, by using the present implementation, various algorithm information can be included in the search instruction, and training data can be searched for the various algorithm information at the same time.

[0140] S303: For a table entry in the data table, read the encoding data in the table entry; determine whether the read encoding data and the second data satisfy a matching rule, and if so, determine the first data in the table entry as the searched data.

[0141] In an implementation, each table entry in the data table can be traversed, and the encoding data in each table entry can be read; and it is determined whether the encoding data and the second data satisfy a matching rule.

[0142] Or in another implementation, the data table can also be filtered first, such as de-duplication, removal of invalid data, etc., and then for each filtered table entry, the encoding data in the table entry is read, and it is determined whether the read encoding data and the second data satisfy a matching rule.

[0143] In an implementation, determining whether the read encoding data and the second data satisfy a matching rule can include determining, based on the read encoding data, whether the first data in the table entry and the second data in the search instruction have an association. Depending on actual conditions, first data having an association with the second data or first data not having an association with the second data can be searched for. For example, in the above example, training data not called by the target algorithm is searched for, i.e., first data not having an association is searched for.

[0144] In an implementation, the encoding data is binary data of a preset number of bits, and each bit in the encoding data corresponds to data of each second type; in this implementation, the second data can be converted into binary data of the same number of bits as the encoding data as converted data; the read encoding data and the converted data are subjected to a logical AND operation to obtain an operation result; it is determined whether the operation result is a preset value; if so, the first data in the table entry is determined as the searched data.

[0145] Alternatively, the second data itself can also be binary data with the same number of bits as the encoded data, in which case the conversion step can be omitted and the second data can be directly logically ANDed with the encoded data.

[0146] Still taking the machine learning scenario as an example, each bit value in the encoded data corresponds to an algorithm used in machine learning. Assume that the second type of data includes various algorithms used in the machine learning process: algorithm 101, algorithm 102, and algorithm 103. Assume that the encoded data is three-bit binary data, where the first bit (counting from right to left) indicates whether algorithm 101 has called the training data, the second bit indicates whether algorithm 102 has called the training data, and the third bit indicates whether algorithm 103 has called the training data. If an algorithm has called the training data, it can be represented as 1, otherwise as 0.

[0147] In this case, the second data can be converted into three-bit binary data, for example, if the second data is algorithm 101, the converted binary data (converted data) can be represented as 001, if the second data is algorithm 102, the converted binary data can be represented as 010, if the second data is algorithm 103, the converted binary data can be represented as 100, if the second data is algorithm 101 and algorithm 102, the converted binary data can be represented as 011, and so on.

[0148] In this case, the above preset value can be 0. In logical AND operation, only if both operands are true (1), the operation result is true (1). If the converted data of the second data and the same bit value in the encoded data are both 1, the operation result of the bit is 1, which indicates that the training data has been called by the target algorithm. If the operation result is 0 at each bit, it indicates that the training data has not been called by the target algorithm, and the training data can be used as the found data.

[0149] Assume that the search instruction is to search for training data for algorithm 103, and the second data is algorithm 103, which is converted into three-bit binary data 100. Assume that the data table is as shown in Table 1 above, and 100 is logically ANDed with the encoded data 011 in the first table entry in Table 1, and the operation result is 000, which is the preset value, and image 1 in the first table entry is used as the found data. The second table entry is processed similarly, and image 2 in the second table entry is also used as the found data. 100 is logically ANDed with the encoded data 111 in the third table entry in Table 1, and the operation result is 100, which is not the preset value 0, and video 3 in the third table entry is not used as the found data.

[0150] It can be seen that, by using the embodiment, complex condition screening is not required, and only simple bit operation (logical AND operation) is required to screen the data meeting the condition, thereby improving the searching efficiency. Even if multiple algorithm information is searched for training data at the same time, simple bit operation is also performed, and the searching efficiency is high.

[0151] In the above embodiment of the data storage, the encoded data is binary data with a preset number of bits, the encoded data is divided into multiple sub-segments, and the multiple sub-segments are respectively stored in different fields. In this embodiment, for each table entry, the sub-segments of the encoded data stored in different fields of the table entry can be read in parallel, the target bits in the conversion data to which the sub-segments are mapped are determined in parallel, the sub-segments and the target bits to which the sub-segments are mapped are subjected to logical AND operation in parallel, and the operation results are obtained. It is determined whether the obtained operation results are all preset values. If the operation results are all preset values, the first data in the table entry is determined as the searched data.

[0152] Suppose that the encoded data is binary data M with 100 bits, which is divided into 10 sub-segments M1-M10, each of which is binary data with 10 bits, and the 10 sub-segments are respectively stored in different fields. Suppose that the second data in the search instruction is converted to obtain conversion data N, which is also binary data with 100 bits. When searching based on the search instruction, the 10 sub-segments M1-M10 are read in parallel, the target bits in N to which each sub-segment is mapped are determined, and the sub-segments and the target bits corresponding to the sub-segments are subjected to logical AND operation in parallel to obtain operation results. That is, the encoded data and the second data are subjected to logical AND operation bit by bit, or logical AND operation is performed between the same bits. Since the number of bits is large, the sub-segments are subjected to parallel logical AND operation, thereby improving the operation efficiency.

[0153] In the above embodiment of the data storage, various candidate association relationships are encoded according to a preset encoding mode. The encoded data stored by using this embodiment can be a serial number or other identifier. In this way, when searching for data, the encoded data in a table entry in a data table is read. It is determined whether the second type of data to which the encoded data points contains the second data in the search instruction according to the preset encoding mode. If not, the first data in the table entry is determined as the searched data.

[0154] For example, suppose that image 1 is called by algorithm 101 and algorithm 102, and continue the above example, the encoded data is a serial number, and the encoded data corresponding to image 1 is 4. For the calling of image 1 by algorithm 101 and algorithm 102, a table entry in the data table is formed as follows:

[0155] First data: image 1 Encoded data: 4

[0156] Suppose the received search instruction is: search training data for algorithm 103. According to the preset encoding mode, it is determined that the second type of data (algorithm 101 and algorithm 102) pointed to by the encoded data 4 in the table entry does not contain the second data (algorithm 103) in the search instruction, and the first data in the table entry is determined as the searched data.

[0157] Or in other cases, according to different search conditions, if it is determined that the second type of data pointed to by the encoded data contains the second data in the search instruction, the first data in the table entry is determined as the searched data. The specific search condition is not limited.

[0158] Application of the present application Figure 3 In the embodiment shown, the first aspect is to search for associated data in the same data table, which is more convenient than searching for data in multiple associated tables. The second aspect is to directly obtain the association between the first data and the second data through the encoded data stored in the data table, and the search can be directly based on the association, which is efficient. The third aspect is that in one implementation, in the machine learning scenario, when using an algorithm, the training data that has not been called by the algorithm can be searched for, so as to reduce the case that one algorithm repeatedly calls one training data, and improve the training efficiency. The fourth aspect is that in the machine learning scenario, the encoded data can represent the calling situation of each algorithm to the training data, so that based on the same data table, training data can be searched for various algorithms or combinations of various algorithms, and the search efficiency is improved. The fifth aspect is that in one implementation, only simple bit operations (logical AND operations) need to be performed to filter out the data meeting the conditions, and the search efficiency is improved. Moreover, even if training data is searched for multiple algorithm information at the same time, simple bit operations are also performed, and the search efficiency is high. The sixth aspect is that in one implementation, the encoded data is divided into multiple sub-segments for storage, so that when the encoded data is read, these segments are processed in parallel, the processing speed is improved, and the search efficiency is further improved.

[0159] Corresponding to the method embodiments described above, the embodiments of the present application also provide a data storage device, as shown in Figure 4 The data storage device comprises:

[0160] The first obtaining module 401 is configured to obtain first data to be stored and second data associated with the first data.

[0161] The second obtaining module 402 is configured to obtain encoded data representing the association between the first data and the second data.

[0162] The storage module 403 is configured to store the first data to a first field of a data table, and store the encoded data to a second field of the data table associated with the first field.

[0163] In an implementation, the second obtaining module 402 is specifically configured to:

[0164] obtain initial binary data of a preset bit number; each bit in the initial binary data corresponds to a second type of data respectively, the second type being a type of the second data;

[0165] adjust a bit in the initial binary data corresponding to the second data from an initial value to a preset value, to obtain encoding data representing an association relationship between the first data and the second data.

[0166] In an implementation, the first data is training data of machine learning, the second type of data is various algorithm information used by machine learning, the second data is algorithm information for calling the training data, and each bit in the initial binary data corresponds to an algorithm used by the machine learning respectively.

[0167] In an implementation, the first data is of a first type, and the second data is of a second type; the apparatus further includes a counting module and an encoding module (not shown in the figure), wherein,

[0168] the counting module is configured to count various candidate association relationships between the first type of data and the second type of data;

[0169] the encoding module is configured to encode the various candidate association relationships in a preset encoding manner, to obtain encoding data corresponding to the various candidate association relationships;

[0170] the second obtaining module 402 is specifically configured to: determine an association relationship between the first data and the second data as a to-be-matched relationship; and search, in the encoding data corresponding to the various candidate association relationships, for encoding data corresponding to the to-be-matched relationship.

[0171] In an implementation, the storage module 403 is specifically configured to:

[0172] divide the encoding data to obtain multiple sub-segments;

[0173] store the multiple sub-segments into different fields associated with the first field in the data table respectively.

[0174] In an implementation, the apparatus further includes a first judging module and an updating module (not shown in the figure), wherein,

[0175] the first judging module is configured to judge whether a table entry matched with the first data is stored in the data table; if yes, trigger the updating module, and if no, trigger the storage module 403;

[0176] The updating module is configured to perform a logical OR operation on the encoded data and data in the second field of the table entry to obtain an operation result, and replace the data in the second field of the table entry with the operation result.

[0177] The application Figure 4 In the first aspect, the data is stored in the same table, and thus, there is no need to establish and maintain multiple tables, data searching is facilitated, and the amount of data storage is reduced. In the second aspect, the first data and the encoded data representing the association relationship are stored in the same table, and thus, the association relationship between the first data and the second data can be directly obtained without establishing the association relationship between tables, and data searching is further facilitated. In the third aspect, in an implementation, the encoded data is divided into multiple sub-segments for storage, and thus, when the encoded data is read, the sub-segments can be processed in parallel, the processing speed is improved, and the searching efficiency is improved. In the fourth aspect, in an implementation, for the first data that has been stored, the encoded data in the original table entry is updated, and for the first data that has not been stored, a new table entry is established, and thus, each piece of first data occupies only one table entry, data searching is facilitated, and only a simple bit operation (logical OR operation) is required to update the table entry, and the updating efficiency is high.

[0178] Corresponding to the method embodiments, the application also provides a data searching device, as shown in Figure 5 The device comprises:

[0179] The receiving module 501 is configured to receive a searching instruction, and the searching instruction comprises second data.

[0180] The first determining module 502 is configured to determine a data table corresponding to the searching instruction, and the data table comprises multiple table entries, and the table entries correspondingly store first data and encoded data, and the encoded data represents an association relationship between the first data and the second data.

[0181] The reading module 503 is configured to read the encoded data in the table entries in the data table.

[0182] The second judging module 504 is configured to judge whether the read encoded data and the second data satisfy a matching rule, and if yes, trigger the second determining module.

[0183] The second determining module 505 is configured to determine the first data in the table entry as searched data.

[0184] In an implementation, the second data is of a second type, and the encoded data represents a correlation between the first data and each second type of data.

[0185] In an implementation, the encoded data is binary data of a preset bit number, and each bit of the encoded data corresponds to each second type of data.

[0186] The second determining module 504 is specifically configured to: convert the second data into binary data of the same bit number as the encoded data as converted data; perform a logical AND operation on the read encoded data and the converted data to obtain an operation result; and determine whether the operation result is a preset value.

[0187] In an implementation, the lookup instruction is an instruction for searching training data for a target algorithm, the second data is target algorithm information, the first data is training data for machine learning, the second type of data is various algorithm information used for machine learning, and each bit value in the encoded data corresponds to one algorithm used for machine learning.

[0188] In an implementation, the reading module 503 is specifically configured to: read the sub-segments of the encoded data stored in different fields of the table item in parallel.

[0189] The second determining module 504 is specifically configured to: convert the second data into binary data of the same bit number as the encoded data as converted data; determine target bits in the converted data to which the sub-segments are mapped in parallel, perform a logical AND operation on each sub-segment and the target bit to which the sub-segment is mapped in parallel to obtain an operation result; and determine whether the obtained operation result is a preset value.

[0190] The application can be applied to Figure 5In the embodiment shown, the first aspect is that the associated data is found in the same data table, which is more convenient than finding the data in multiple associated tables. The second aspect is that the association between the first data and the second data is directly obtained through the encoded data stored in the data table, and the association can be directly used for searching, which is efficient. The third aspect is that in one implementation, in a machine learning scenario, when an algorithm is used, the training data that has not been called by the algorithm can be found, so as to reduce the case that the same training data is repeatedly called by the algorithm, and improve the training efficiency. The fourth aspect is that in a machine learning scenario, the encoded data can represent the calling condition of each algorithm to the training data, so that based on the same data table, the training data can be found for various algorithms or combinations of various algorithms, and the searching efficiency is improved. The fifth aspect is that in one implementation, only simple bit operations (logical AND operations) are needed to filter out the data meeting the condition, and the searching efficiency is improved. Moreover, even if the training data is searched for multiple algorithm information at the same time, only simple bit operations are performed, and the searching efficiency is high. The sixth aspect is that in one implementation, the encoded data is divided into multiple sub-segments and stored separately, so that when the encoded data is read, the sub-segments are processed in parallel, the processing speed is improved, and the searching efficiency is improved.

[0191] The embodiment of the present application also provides an electronic device, such as Figure 6 As shown, the electronic device includes a processor 601 and a memory 602.

[0192] The memory 602 is used to store a computer program.

[0193] The processor 601 is used to execute the program stored in the memory 602, and realize the above-mentioned any one data storage and searching method.

[0194] The memory mentioned in the above-mentioned electronic device can include a random access memory (RAM) and a non-volatile memory (NVM), for example, at least one disk memory. Optionally, the memory can also be at least one storage device located away from the above-mentioned processor.

[0195] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0196] In yet another embodiment provided by the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement any one of the data storage and searching methods described above.

[0197] In yet another embodiment provided by the present application, a computer program product is provided, and the computer program product includes instructions. When the computer program product is executed on a computer, the computer is caused to perform any one of the data storage and searching methods described above.

[0198] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0199] It is to be noted that, in the present document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily implying any actual relationship or order between such entities or actions. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0200] Each of the embodiments in the present specification is described in a related manner, and the same or similar parts between the embodiments can be mutually referred to. Each of the embodiments focuses on the difference from other embodiments. In particular, for the device embodiment, the equipment embodiment, the computer-readable storage medium embodiment, and the computer program product embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0201] The preferred embodiments of the present application are described above with reference to the drawings, but the present application is not limited to the above examples. Any modification, equivalent replacement, improvement and the like within the spirit and principle of the present application are included in the scope of the present application.

Claims

1. A data storage method, characterized by, The method comprises: obtaining first data to be stored and second data associated with the first data; obtaining encoding data representing the association relationship between the first data and the second data; storing the first data to a first field of a data table and storing the encoding data to a second field associated with the first field in the data table; wherein the obtaining of the encoding data representing the association relationship between the first data and the second data comprises: obtaining initial binary data of a preset bit number; each bit in the initial binary data corresponds to a second type of data, and different bits correspond to different second types of data, and the second type is the type of the second data; adjusting the bit corresponding to the second data in the initial binary data from an initial value to a preset value to obtain the encoding data representing the association relationship between the first data and the second data; wherein the bit being the initial value indicates that the first data is not associated with the second type of data corresponding to the bit, and the bit being the preset value indicates that the first data is associated with the second type of data corresponding to the bit; Alternatively, the type of the first data is a first type, and the type of the second data is a second type; the method further comprises: counting various candidate association relationships between the first type of data and the second type of data; encoding the various candidate association relationships according to a preset encoding manner to obtain encoding data corresponding to the various candidate association relationships, wherein the encoding indicates that an identifier is assigned to each candidate association relationship to distinguish the various candidate association relationships; the obtaining of the encoding data representing the association relationship between the first data and the second data comprises: determining the association relationship between the first data and the second data as a to-be-matched relationship; finding the encoding data corresponding to the to-be-matched relationship in the encoding data corresponding to the various candidate association relationships.

2. The method of claim 1, wherein, The first data is training data for machine learning, the second type of data is various algorithm information used by machine learning, the second data is algorithm information for calling the training data, and each bit value in the initial binary data corresponds to an algorithm used by machine learning.

3. The method of claim 1, wherein, The storing of the encoding data to the second field associated with the first field in the data table comprises: segmenting the encoding data to obtain multiple sub-segments; storing the multiple sub-segments to different fields associated with the first field in the data table, respectively.

4. The method of claim 1, wherein, After the adjusting of the bit corresponding to the second data in the binary data from the initial value to the preset value to obtain the encoding data representing the association relationship between the first data and the second data, the method further comprises: determining whether a table entry matching the first data is stored in the data table; if yes, performing a logical OR operation on the encoding data and the data in the second field of the table entry to obtain an operation result; and replacing the data in the second field of the table entry with the operation result. If not, the storing of the first data into a first field of a data table and the storing of the encoded data into a second field of the data table associated with the first field are performed.

5. A data search method characterized by, The method comprises: receiving a search instruction, the search instruction comprising second data; determining a data table corresponding to the search instruction, the data table comprising a plurality of table entries, each table entry storing first data and encoded data, the encoded data representing an association between the first data and the second data; for each table entry in the data table, reading the encoded data in the table entry, determining whether the read encoded data and the second data satisfy a matching rule, and if so, determining the first data in the table entry as searched data; wherein the second data is of a second type, the encoded data represents an association between the first data and each second type of data, the encoded data is binary data of a preset number of bits, each bit of the encoded data corresponds to a second type of data, different bits correspond to different second types of data, and the determination of whether the read encoded data and the second data satisfy the matching rule, and if so, the determination of the first data in the table entry as searched data, comprises: converting the second data into binary data of the same number of bits as the encoded data as converted data; performing a logical AND operation on the read encoded data and the converted data to obtain an operation result; determining whether the operation result is a preset value; if the operation result is the preset value, indicating that the first data in the table entry is first data that has no association with the second data, the first data in the table entry is determined as searched data; or, the encoded data is a serial number, and the determination of whether the read encoded data and the second data satisfy the matching rule, and if so, the determination of the first data in the table entry as searched data, comprises: determining, according to a preset encoding mode, whether the second type of data pointed to by the read encoded data contains the second data in the search instruction; if not, the first data in the table entry is determined as searched data.

6. The method of claim 5, wherein, The search instruction is an instruction for searching training data for a target algorithm, the second data is target algorithm information, the first data is training data for machine learning, the second type of data is various algorithm information used for machine learning, and each bit value in the encoded data corresponds to one algorithm used for machine learning.

7. The method of claim 5, wherein, The reading of the encoded data in the table entry comprises parallel reading of sub-pieces of the encoded data stored in different fields of the table entry; the performing of the logical AND operation on the read encoded data and the converted data to obtain an operation result comprises parallel determination of target bits in the converted data to which each sub-piece is mapped and parallel performance of a logical AND operation on each sub-piece and its mapped target bit to obtain an operation result; the determination of whether the operation result is a preset value comprises determination of whether all the obtained operation results are preset values.

8. A data storage device, characterized by The method comprises: The first obtaining module is configured to obtain first data to be stored and second data associated with the first data; The second obtaining module is configured to obtain encoded data representing an association relationship between the first data and the second data; The storage module is configured to store the first data to a first field of a data table and store the encoded data to a second field of the data table associated with the first field; The second obtaining module is specifically configured to obtain initial binary data with a preset number of bits, each bit in the initial binary data corresponding to a second type of data, different bits corresponding to different second types of data, the second type being a type of the second data, and adjusting a bit in the initial binary data corresponding to the second data from an initial value to a preset value to obtain the encoded data representing the association relationship between the first data and the second data, the bit being the initial value indicating that the first data is not associated with the second type of data corresponding to the bit, and the bit being the preset value indicating that the first data is associated with the second type of data corresponding to the bit. Alternatively, the first data is of a first type, and the second data is of a second type; and the apparatus further includes: The statistical module is configured to count various candidate association relationships between the first type of data and the second type of data; The encoding module is configured to encode the various candidate association relationships according to a preset encoding manner to obtain encoded data corresponding to the various candidate association relationships, the encoding representing that an identifier is assigned to each candidate association relationship to distinguish the various candidate association relationships; The second obtaining module is specifically configured to determine an association relationship between the first data and the second data as a to-be-matched relationship, and find, in the encoded data corresponding to the various candidate association relationships, encoded data corresponding to the to-be-matched relationship.

9. A data search apparatus, characterized by comprising: The receiving module is configured to receive a search instruction, the search instruction including second data; The first determining module is configured to determine a data table corresponding to the search instruction, the data table including a plurality of table entries, each table entry corresponding to stored first data and encoded data, the encoded data representing an association relationship between the first data and the second data; The reading module is configured to read the encoded data in each table entry in the data table; The second judging module is configured to judge whether the read encoded data and the second data satisfy a matching rule, and if so, trigger the second determining module; The second determining module is configured to determine the first data in the table entry as searched data. ​ The second data is second type data, the encoding data represents an association between the first data and each second type data, the encoding data is binary data with a preset bit number, each bit of the encoding data corresponds to a second type data, and different bits correspond to different second type data; the second judging module is specifically configured to convert the second data into binary data with the same bit number as the encoding data as converted data, perform logical AND operation on the read encoding data and the converted data to obtain an operation result, and determine whether the operation result is a preset value; if the operation result is the preset value, it indicates that the first data in the table entry is first data that has no association with the second data, and the second determining module is triggered. Alternatively, the encoding data is a serial number, and the second judging module is specifically configured to determine, according to a preset encoding mode, whether the second type data pointed to by the read encoding data contains the second data in the search instruction; if not, the second determining module is triggered.

Images