Data processing method and system

Abstract

The application discloses a data processing method and system. The method comprises the following steps: receiving first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; and outputting a storage result corresponding to the first image data. The application solves the technical problem of the related art that the index construction in the vector retrieval method needs data training, which leads to the need to re-construct the index when a new feature vector is written, and the high cost of index construction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data retrieval, in particular to a data processing method and system. BACKGROUND

[0002] At present, the nearest neighbor search method is widely used in the fields of computer vision, text, image retrieval, etc. Nearest neighbor search (NNS) can also be called "closest point search", which can be an optimization problem of finding the nearest point in a scale space. The problem description is as follows: given a point set S and a target point q in a scale space M, find the point in S closest to q. In most cases, M is a multi-dimensional Euclidean space, and the distance is determined by the Euclidean distance or Manhattan distance.

[0003] The process of the traditional nearest neighbor search method can be divided into two parts: index construction and index retrieval. In the index construction process, data training needs to be performed first, and then the index can be constructed. After the index is constructed, when new data is written, the index often needs to be reconstructed, which limits the use scene of retrieval and increases the cost of repeated index construction.

[0004] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0005] The embodiments of the present application provide a data processing method and system to at least solve the technical problem in the related art that the data training is needed in the index construction of the vector retrieval method, resulting in the need to reconstruct the index when a new feature vector is written, and the high cost of index construction.

[0006] According to an aspect of an embodiment of the present application, a data processing method is provided, comprising: receiving a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier to the first storage area; and outputting a storage result corresponding to the first feature vector.

[0007] According to another aspect of an embodiment of the present application, a data processing method is provided, comprising: obtaining a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; and storing the first feature vector and the first index identifier to the first storage area.

[0008] According to another aspect of the embodiments of the present application, a data processing method is further provided, comprising: obtaining first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index; and storing the first feature vector and the first index into the first storage area.

[0009] According to another aspect of the embodiments of the present application, a data processing method is further provided, comprising: obtaining first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index; and storing the first feature vector and the first index into the first storage area.

[0010] According to another aspect of the embodiments of the present application, a data processing method is further provided, comprising: obtaining first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index; and storing the first feature vector and the first index into the first storage area.

[0011] According to another aspect of the embodiments of the present application, a data processing method is further provided, comprising: obtaining first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index; and storing the first feature vector and the first index into the first storage area.

[0012] According to another aspect of the embodiments of the present application, a computer readable storage medium is further provided, which comprises a stored program, wherein the program, when running, controls a device where the computer readable storage medium is located to perform the data processing method described above.

[0013] According to another aspect of the embodiments of the present application, a computer terminal is further provided, comprising: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the data processing method described above.

[0014] According to another aspect of the embodiments of the present application, a data processing system is also provided, comprising: a processor; and a memory connected with the processor, configured to provide the processor with instructions to process the following processing steps: receiving first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; and outputting a storage result corresponding to the first image data.

[0015] In the embodiments of the present application, after receiving the first feature vector, the first index identifier corresponding to the first feature vector can be generated, and the first storage area in the first storage block corresponding to the first index identifier can be determined, so that the first feature vector and the first index identifier can be stored into the first storage area, and the corresponding storage result can be output, achieving the purpose of streaming writing of the feature vector. It is easy to note that, since the index identifier of the feature vector can be generated immediately after receiving the feature vector each time, without the index training process, the streaming index in the vector retrieval scenario is achieved, the technical effect of the feature vector being searched as soon as it is added is achieved, and the index construction cost is reduced, thereby solving the technical problem in the related art that the index construction in the vector retrieval method needs data training, resulting in the need to reconstruct the index when a new feature vector is written, and the index construction cost is high. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application, and the illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0017] Figure 1 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method according to an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of a computer terminal as a receiving end according to an embodiment of the present application;

[0019] Figure 3 is a flowchart of a first data processing method according to an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of an optional user interface according to an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of an optional index structure design according to an embodiment of the present application;

[0022] Figure 6is a flow chart of a second data processing method according to an embodiment of the present application;

[0023] Figure 7 is a flow chart of a third data processing method according to an embodiment of the present application;

[0024] Figure 8 is a flow chart of a fourth data processing method according to an embodiment of the present application;

[0025] Figure 9 is a schematic diagram of another optional index structure design according to an embodiment of the present application;

[0026] Figure 10 is a schematic diagram of a first data processing apparatus according to an embodiment of the present application;

[0027] Figure 11 is a schematic diagram of a second data processing apparatus according to an embodiment of the present application;

[0028] Figure 12 is a schematic diagram of a third data processing apparatus according to an embodiment of the present application;

[0029] Figure 13 is a schematic diagram of a fourth data processing apparatus according to an embodiment of the present application;

[0030] Figure 14 is a flow chart of a fourth data processing method according to an embodiment of the present application;

[0031] Figure 15 is a flow chart of a fifth data processing method according to an embodiment of the present application; and

[0032] Figure 16 is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should belong to the protection scope of the present application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0035] First, some of the nouns or terms appearing in the description of the embodiments of the present application are applicable to the following explanations:

[0036] Feature: can refer to a single attribute extracted from raw data, which can be a number. After the raw data is converted into a feature vector, it can be further stored and retrieved. For example, after converting a picture into a feature vector, a nearest neighbor retrieval method can be used to find similar pictures in a large number of pictures.

[0037] HNSW: Hierarchcal Navigable Small World graphs, a hierarchical navigable small world graph algorithm, can be a kind of graph index retrieval method without training, the main idea is: according to the length of the edge is divided into different layers, search the graph of multiple layers, can set the maximum node number of each node in each layer to a constant, which is independent of the size of the graph, and then find the local nearest point from the top layer, and then search from this point in the next layer, similar to the skip list structure in one-dimensional data.

[0038] Block: data block, which can be a standard length of bytes or bits, is the smallest unit of data block management storage.

[0039] MMAP: can refer to a method of memory mapping file, which can map a file or other object to the address space of a process, realize the corresponding relationship between file disk address and a segment of virtual address in process virtual address space, so as to realize file sharing between different processes.

[0040] Embodiment 1

[0041] According to the embodiments of the present application, a data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0042] The method embodiments provided by the embodiments of the present application can be executed in a mobile terminal, a computer terminal or similar computing device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing the data processing method is shown. As shown in the figure, Figure 1 The computer terminal 10 (or mobile device 10) can include one or more processors 102 (the processor 102 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports in the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that, Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 can also include more or less components than those shown in the figure, or have a different configuration from that shown in the figure. Figure 1 For example, the computer terminal 10 can also include more or less components than those shown in the figure, or have a different configuration from that shown in the figure. Figure 1 For example, the computer terminal 10 can also include more or less components than those shown in the figure, or have a different configuration from that shown in the figure.

[0043] It should be noted that the one or more processors 102 and / or other data processing circuits described above can be referred to as "data processing circuits" herein. The data processing circuit can be embodied in whole or in part as software, hardware, firmware or any other combination. In addition, the data processing circuit can be a single independent processing module, or any one of the other elements combined into the computer terminal 10 (or mobile device) in whole or in part. The data processing circuit serves as a processor control (for example, selection of a variable resistance terminal path connected to an interface).

[0044] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0045] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0046] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0047] Figure 1 The hardware structure block diagram shown can serve not only as an exemplary block diagram of the aforementioned computer terminal 10 (or mobile device), but also as an exemplary block diagram of the aforementioned server. In one optional embodiment, Figure 2 The use of the above is illustrated in a block diagram. Figure 1 The computer terminal 10 (or mobile device) shown is one embodiment of the receiving end. For example... Figure 2 As shown, computer terminal 10 (or mobile device) can be connected to one or more clients 20 via a data network connection or electronically, such as a mobile terminal, computer terminal, etc. In an optional embodiment, the computer terminal 10 (or mobile device) can be a server, providing a network server for the connected one or more clients. The data network connection can be a local area network connection, a wide area network connection, an Internet connection, or other types of data network connection. The network service is a network-based user service, such as social networks, cloud resources, email, online payments, or other online applications.

[0048] In the above operating environment, the application provides a data processing method as shown in Figure 3 . Figure 3 is a flowchart of a first data processing method according to an embodiment of the application. As shown in Figure 3 , the method can include the following steps:

[0049] Step S302, receiving a first feature vector.

[0050] The first feature vector in the above step can be a feature vector that needs to be written into a storage device.

[0051] In an optional embodiment, the first feature vector can be uploaded by a user to a server through a user terminal (such as a smartphone, a tablet computer, a notebook computer, etc.), and the feature vector is stored into the storage device by the server. In another optional embodiment, the first feature vector can be selected by the user directly on the user terminal, and the feature vector is stored locally by the user terminal.

[0052] It should be noted that in actual application scenarios, the user often wants to write image, text, video and other data, and retrieve the written data, on the basis of which the user needs to manually process the data that needs to be written to obtain the above-mentioned first feature vector. In order to simplify the user's operation, the user can also directly upload the data to be stored to the server, and the server performs feature extraction, and then stores the first feature vector; or the user directly selects the data, and the user terminal performs feature extraction, and then stores the first feature vector.

[0053] Further, in order to facilitate the user to upload files to the server, a user interface as shown in Figure 4 may be provided, and the user can click the "upload file" button in the interface to select and upload the file storing the first feature vector to the server, or the user can drag the file storing the first feature vector into the dashed box to complete the uploading purpose.

[0054] It should also be noted that in order to ensure that the server or the user terminal accurately knows whether the received feature vector needs to be written into a vector or retrieved, the corresponding processing instruction can be received at the same time as the first feature vector is received, and based on the processing instruction, it can be determined whether the first feature vector is written into a vector or retrieved.

[0055] Step S304, generating a first index identifier corresponding to the first feature vector.

[0056] The first index identifier in the above step can refer to an index id corresponding to the first feature vector.

[0057] In an alternative embodiment, in order to achieve the purpose of stream writing and query, an index id can be assigned to the first feature vector when the first feature vector is received, without the index training process. Alternatively, the first index id can be generated according to the self-incrementing number generation algorithm, that is, an index id starting from 0 and sequentially increasing can be assigned to each newly added feature vector. For example, when the first feature vector is received, if the last generated index id is 1101, the first index id of the first feature vector can be determined as 1102.

[0058] In step S306, the first storage area in the first storage block corresponding to the first index id is determined.

[0059] The first storage block in the above steps can be an already allocated block or a newly allocated block. Each block contains a plurality of nodes node, and each node is a storage area of a feature vector and can be used to store all data of the feature vector. Moreover, the size of each block in the same index file is consistent.

[0060] Alternatively, when the first storage block is a newly added block, a block of storage space can be allocated from the storage system, and the size of the first storage block is determined by the preset feature vector size and the preset number, wherein the preset number can be the number of neighbors of the feature vector. Here, the neighbor can refer to the data of a plurality of feature vectors with the closest feature distance determined based on the nearest neighbor algorithm. The neighbor data can be pre-set according to actual needs, for example, it can be 4 or 5, but is not limited thereto. Through the above scheme, each block can accommodate a power of 2 number of nodes, so that the corresponding block can be immediately located according to the index id without additional indexing, reducing storage access and improving performance.

[0061] In an alternative embodiment, after the first index id is generated, the block id and the offset in the block can be calculated through a specific algorithm. Based on the calculated block id, the first storage block can be determined, and based on the offset, the first storage area in the first storage block can be determined.

[0062] It should be noted that, in order to facilitate subsequent real-time vector retrieval, the calculated block id and offset can be stored in the storage space corresponding to the first index id, for example, the storage area of the index id is 32 bits, and the high 24 bits can be set as the block id and the low 8 bits can be set as the offset. At the same time, since the storage area of each index id is determined, multi-thread parallel writing can be set to be lock-free, without thread conflicts.

[0063] Step S308, store the first feature vector and the first index identifier into the first storage area.

[0064] In an optional embodiment, the first feature vector and the first index identifier can be stored into the first storage area in the first storage block, each storage area can be divided into a feature vector sub-storage area and an index identifier sub-storage area, wherein the index identifier sub-storage area is located after the feature vector sub-storage area.

[0065] It should be noted that since the block contains a plurality of storage areas, the sizes of the feature vectors and the index identifiers stored in each storage area are different, in order to ensure that each storage area has the same size, optionally, a preset padding data can be stored in the first storage area. The pedding can be all 0 data, and the sub-storage area storing the pedding is located at the end of the first storage area.

[0066] Step S310, output the storage result corresponding to the first feature vector.

[0067] The storage result in the above steps can be a prompt information indicating that the first feature vector is stored successfully, so as to facilitate the user to know whether the vector writing process is successful this time.

[0068] In an optional embodiment, after the server stores the first feature vector into the first storage area, the server can return the storage result indicating that the storage is successful to the user terminal, and the user terminal displays or plays the storage result, for example, the storage result can be displayed in the display area of the user interface as shown in Figure 4 In an optional embodiment, the user terminal can also display the storage result indicating that the storage is successful to the user after storing the first feature vector into the first storage area.

[0069] The preferred embodiment of the present application will be described in detail below in combination with the index format design diagram as shown in Figure 5 The index structure design can be deployed in notebook computers, PC computer terminals, and mobile terminals such as mobile phones, tablet computers, and palm computers. As shown in Figure 5 When a new feature vector is added to the index, an index id starting from 0 and sequentially increasing can be allocated in the index file, and the corresponding block id and offset in the block can be calculated according to the id. For each block, the mapping relationship between the index file and the memory block can be constructed by the mmap algorithm.

[0070] If it is determined that the block is a newly added block, a block of storage space can be allocated from a storage system (for example, can be a memory), the size of the block can be calculated in advance according to the feature vector size and the number of neighbors, and the sizes of all blocks corresponding to the same index file are consistent.

[0071] Each block contains a plurality of nodes, each node is a storage area corresponding to a feature vector, wherein the feature vector, the index id and the neighbor table of the feature vector can be stored, wherein the neighbor table contains the number of neighbors of the feature vector and the index id of each neighbor, for example, a certain neighbor table contains 4 neighbors, and the four index ids are 10, 17, 3 and 6. In addition, padding can be added at the end of the node to ensure that each storage area can be aligned according to a certain boundary.

[0072] Based on the scheme provided in the above embodiments of the application, after receiving the first feature vector, the first index identifier corresponding to the first feature vector can be generated, and the first storage area in the first storage block corresponding to the first index identifier can be determined, so that the first feature vector and the first index identifier can be stored in the first storage area, and the corresponding storage result can be output, achieving the purpose of feature vector streaming writing. It is easy to note that since the index identifier of the feature vector can be generated immediately after receiving the feature vector each time, there is no need to perform index training process, thereby realizing streaming index in the vector retrieval scenario, achieving the technical effect of feature vector increment and search, reducing the index construction cost, and further solving the technical problem in the related art that the index construction in the vector retrieval method needs data training, resulting in the need to rebuild the index when writing a new feature vector, and the index construction cost is high.

[0073] In the above embodiments of the application, step S306, determining the first storage area in the first storage block corresponding to the first index identifier includes: determining the data block identifier and the offset of the first storage block based on the first index identifier; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset.

[0074] In an optional embodiment, for an index file, the size of each block and the size of each storage area can be determined according to the index format design method, so that the correspondence between the index id and the block id and the offset can be determined, so as to obtain a specific algorithm. After assigning a new index id (i.e., the first index identifier described above), the data block identifier block id and the offset of the index id can be calculated according to the predetermined specific algorithm, so that the first storage block can be determined based on the block id, and the first storage area in the first storage block can be determined based on the offset.

[0075] For example, assuming that each block contains three nodes, and the size of each node is 1 MB, the specific algorithm can be determined as follows: the block id is the integer part of the index id / 3, and the offset is the difference between the index id and the block id*3.

[0076] In the above embodiments of the present application, after the first feature vector and the first index identifier are stored in the first storage area in step S308, the method further includes at least one of the following: mapping the first storage block to a disk; and storing the first storage block to a remote storage device through a preset interface.

[0077] The preset interface in the above steps can be a read / write interface.

[0078] In order to achieve the purpose of streaming writing and querying, the following two vector retrieval schemes can be used: the first scheme can use a linear index, and uses a brute force matching method for retrieval when retrieving a vector. This scheme does not require an index training process, but if the data volume is large, the retrieval cost and response time will increase exponentially, and it cannot be applied to large data volume application scenarios. The second scheme can use a type of index that does not require training, for example, using the HNSW graph method, real-time retrieval can be achieved, but the index can only be stored in the memory, and the data cannot be persisted in real time; the size of the index is limited by the size of the memory; in addition, after the process or device is down, it takes a long time to load the index again.

[0079] To solve the above problems, in an optional embodiment, when data persistence is required, the data stored in all blocks can be mapped to a local disk by mmap or mapped to a remote disk file by a read / write interface. On this basis, after the process or device is down, the data stored in the local disk or the remote disk does not need to be loaded into the memory, so that the millisecond-level loading of the index can be realized. In addition, since the data stored in the block can be mapped to the local or remote disk, there is no need to reside in the memory, and therefore the size of the index file can not be limited by the memory and the disk size.

[0080] In the above embodiment of the present application, before the first feature vector and the first index identifier are stored in the first storage area in step S308, the method further comprises: determining a second feature vector in the plurality of stored feature vectors based on a hierarchical navigable small world graph algorithm, wherein the feature distance between the second feature vector and the first feature vector is less than a preset distance; determining a second index identifier of the second feature vector; generating a data table corresponding to the first feature vector based on the second index identifier; and storing the first feature vector, the first index identifier and the data table in the first storage area.

[0081] The preset distance in the above step can be a feature distance for determining the similarity of two feature vectors, which can be calculated by the Euclidean distance algorithm, but is not limited thereto, and other algorithms can also be used. When the feature distance between two feature vectors is less than the preset distance, it can be determined that the similarity of the two feature vectors is high, and they can be neighbors of each other. The data table can be a data table for storing the second feature vector, i.e., a neighbor table of the first feature vector, which can store the index id of each neighbor, i.e., the index id of each second feature vector.

[0082] In an optional embodiment, to realize the streaming index of the vector index scenario, the HNSW algorithm can be used for nearest neighbor retrieval index, so that after each first feature vector is added, the HNSW algorithm can be used to determine a plurality of feature vectors closest to the feature vector to obtain a second feature vector, and further generate a neighbor table corresponding to the first feature vector based on the index id of the second feature vector, and further store the first feature vector, the index id of the first feature vector and the neighbor table in the determined first storage area.

[0083] It should be noted that the writing process of the first feature vector can be determined based on the HNSW algorithm. Based on the principle of the HNSW algorithm, all feature vectors can be constructed into a multi-layer interconnected graph. After receiving the first feature vector, the layer in which the feature vector is located is first determined, and then the node closest to the feature vector is determined layer by layer from the top layer. The node closest to the feature vector in each layer can be used as the starting node of the next layer, and the node closest to the feature vector in the lower layer can be used as the neighbor of the feature vector. The feature vector can be stored in the layer in which it is located and each layer below the layer. The node corresponding to the feature vector in each layer can be connected to its neighbor node.

[0084] In the above embodiments of the present application, based on the second index identifier, the data table corresponding to the first feature vector is generated, including: determining the number of second feature vectors; based on the number and the second index identifier, generating the data table.

[0085] In an optional embodiment, the preset number of neighbors of each feature vector can be determined according to actual retrieval needs, and therefore the number of second feature vectors is the above-mentioned preset number. After determining the plurality of second feature vectors corresponding to the first feature vector, the preset number and the index id (i.e. the above-mentioned second index identifier) of each second feature vector can be stored in the neighbor table. For example, as shown in Figure 5 the preset number of neighbors can be 4, and the second index identifiers are 10, 17, 3 and 6, respectively.

[0086] It should be noted that by storing the preset number of neighbors in the neighbor table, the index id of each neighbor can be accurately determined based on the preset number.

[0087] In the above embodiments of the present application, after storing the first feature vector and the first index identifier in the first storage area in step S308, the method further includes: receiving a third feature vector; determining a target storage area in a target storage block based on the hierarchical navigable small world graph algorithm; obtaining a target feature vector stored in the target storage area to obtain a retrieval result, wherein the feature distance between the target feature vector and the third feature vector is less than a preset distance; and outputting the retrieval result.

[0088] The third feature vector in the above step can be a feature vector that needs to be subjected to vector retrieval. The target feature vector can be the feature vector closest to the third feature vector based on the HNSW algorithm. The number of target feature vectors is the preset number of neighbors, and the storage area in which the target feature vector is stored is the above-mentioned target storage area. The storage block in which the target storage area is located is the above-mentioned target storage block.

[0089] In an alternative embodiment, the third feature vector can be uploaded by the user to the server through a user terminal (e.g. a smart phone, a tablet, a notebook, etc.), and the vector retrieval is performed by the server. After the retrieval result is retrieved, it can be returned to the user terminal and displayed by the user terminal. In another alternative embodiment, the third feature vector can be selected directly by the user on the user terminal, and the vector retrieval is performed by the user terminal. After the retrieval result is retrieved, it can be displayed to the user for viewing.

[0090] It should be noted that, as with the processing mode of the first feature vector, in order to simplify the operation of the user, the user can also directly upload the data to be retrieved to the server, and the server performs feature extraction, and then performs vector retrieval on the third feature vector; or the user directly selects the data, and the user terminal performs feature extraction, and then performs vector retrieval on the third feature vector.

[0091] As shown in the user interface shown in Figure 4 , the user can select a file storing the third feature vector and upload it to the server by clicking the "Upload File" button in the interface, or the user can complete the uploading purpose by dragging the file storing the third feature vector into the dashed box. The retrieval result can also be displayed in the display area of the user interface.

[0092] Further, the retrieval process of the target feature vector can be determined based on the HNSW algorithm. Based on the principle of the HNSW algorithm, after receiving the third feature vector, the nearest node to the feature vector can be determined layer by layer from the topmost layer. The nearest node retrieved in each layer can be used as the starting node of the next layer, and the nearest node detected in the lower layer can be used as the final retrieval result. In the entire retrieval process, the nearest node determined in each layer is the corresponding index id. Further, based on the correspondence between the index id, the block id and the offset, the feature vector corresponding to the index id can be obtained.

[0093] It should be noted that, for the above-mentioned method embodiments, in order to simplify the description, they are all described as a series of action combinations, but those skilled in the art should know that the application is not limited by the order of the described actions, because according to the application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the application.

[0094] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software on a general hardware platform, and of course can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disc), and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device) to execute the methods described in the various embodiments of the present application.

[0095] Embodiment 2

[0096] According to the embodiments of the present application, a data processing method is also provided, which can also be applied to the operating environment as shown in Figure 1 and Figure 2 . Figure 6 is a flowchart of a second data processing method according to an embodiment of the present application. As shown in Figure 6 , the method can include the following steps:

[0097] Step S602, obtaining a first feature vector.

[0098] The first feature vector in the above step can be a feature vector that needs to be written into a storage device.

[0099] Step S604, generating a first index identifier corresponding to the first feature vector.

[0100] The first index identifier in the above step can refer to an index id corresponding to the first feature vector.

[0101] Step S606, determining a first storage area in a first storage block corresponding to the first index identifier.

[0102] The first storage block in the above step can be an already allocated block, or a newly allocated block. Each block contains a plurality of nodes node, and each node is a storage area of a feature vector, which can be used to store all data of the feature vector. Moreover, in the same index file, the size of each block is consistent.

[0103] Optionally, when the first storage block is a newly added block, a storage space can be allocated from the storage system, and the size of the first storage block is determined by a preset feature vector size and a preset number, wherein the preset number can be the number of neighbors of the feature vector. Here, the neighbors can refer to the data of a plurality of feature vectors with the closest feature distance determined based on the nearest neighbor algorithm. The neighbor data can be pre-set according to actual needs, for example, can be 4 or 5, but is not limited thereto.

[0104] In step S608, the first feature vector and the first index identifier are stored in the first storage area.

[0105] In the above embodiments of the present application, in step S606, determining the first storage area in the first storage block corresponding to the first index identifier includes: determining the data block identifier and the offset of the first storage block based on the first index identifier; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset.

[0106] In the above embodiments of the present application, after step S608, the first feature vector and the first index identifier are stored in the first storage area, the method further includes at least one of the following: mapping the first storage block to a disk; and storing the first storage block in a remote storage device through a preset interface.

[0107] The preset interface in the above steps can be a read / write interface.

[0108] In the above embodiments of the present application, before step S608, the first feature vector and the first index identifier are stored in the first storage area, the method further includes: determining a second feature vector in the plurality of stored feature vectors based on a hierarchical navigable small world graph algorithm, wherein the feature distance between the second feature vector and the first feature vector is less than a preset distance; determining a second index identifier of the second feature vector; generating a data table corresponding to the first feature vector based on the second index identifier; and storing the first feature vector, the first index identifier, and the data table in the first storage area.

[0109] The preset distance in the above steps can be a feature distance pre-set for determining the similarity of two feature vectors. The feature distance of two feature vectors can be calculated by the Euclidean distance algorithm, but is not limited thereto, and other algorithms can also be used. When the feature distance between two feature vectors is less than the preset distance, it can be determined that the similarity of the two feature vectors is high, and they can be neighbors of each other. The data table can be a data table for storing the second feature vector, i.e., a neighbor table of the first feature vector. The neighbor table can store the index id of each neighbor, i.e., the index id of each second feature vector.

[0110] In the above embodiments of the present application, based on the second index identifier, the data table corresponding to the first feature vector is generated, including: determining the number of second feature vectors; and generating the data table based on the number and the second index identifier.

[0111] In the above embodiments of the present application, after storing the first feature vector and the first index identifier into the first storage area in step S608, the method further includes: obtaining a third feature vector; determining a target storage area in a target storage block based on the hierarchical navigable small world graph algorithm; and obtaining a target feature vector stored in the target storage area to obtain a search result, wherein a feature distance between the target feature vector and the third feature vector is less than a preset distance.

[0112] The third feature vector in the above step can be a feature vector that needs to be searched. The target feature vector can be a feature vector closest to the third feature vector based on the HNSW algorithm, the number of target feature vectors is a preset number of neighbors, the storage area storing the target feature vector is the target storage area described above, and the storage block where the target storage area is located is the target storage block described above.

[0113] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the schemes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0114] Embodiment 3

[0115] According to the embodiments of the present application, a data processing method is also provided, which can also be applied to the running environment as shown in Figure 1 and Figure 2 . Figure 7 is a flowchart of a third data processing method according to the embodiments of the present application. As shown in Figure 7 , the method can include the following steps:

[0116] Step S702, obtaining first image data.

[0117] The first image data in the above step can be an image that needs to be written into a storage device.

[0118] Step S704, performing feature extraction on the first image data to obtain a first feature vector.

[0119] Step S706, generating a first index identifier corresponding to the first feature vector.

[0120] The first index identifier in the above step can be an index id corresponding to the first feature vector.

[0121] Step S708, determining a first storage area in a first storage block corresponding to the first index identifier.

[0122] The first storage block in the above step can be an already allocated block or a newly allocated block. Each block contains a plurality of nodes, each of which is a storage area of a feature vector and can be used to store all data of the feature vector. Moreover, the size of each block in the same index file is consistent.

[0123] Optionally, when the first storage block is a newly added block, a block of storage space can be allocated from the storage system, and the size of the first storage block is determined by a preset feature vector size and a preset number, wherein the preset number can be the number of neighbors of the feature vector. Here, the neighbor can refer to the data of a plurality of feature vectors with the closest feature distance determined based on the nearest neighbor algorithm. The neighbor data can be pre-set according to actual needs, for example, it can be 4 or 5, but is not limited thereto.

[0124] In step S710, the first feature vector and the first index identifier are stored in the first storage area.

[0125] In the above embodiments of the present application, in step S708, determining the first storage area in the first storage block corresponding to the first index identifier includes: determining the data block identifier and the offset of the first storage block based on the first index identifier; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset.

[0126] In the above embodiments of the present application, after step S710, the first feature vector and the first index identifier are stored in the first storage area, the method further includes at least one of the following: mapping the first storage block to a disk; and storing the first storage block in a remote storage device through a preset interface.

[0127] The preset interface in the above step can be a read / write interface.

[0128] In the above embodiments of the present application, before step S710, the first feature vector and the first index identifier are stored in the first storage area, the method further includes: determining a second feature vector in the plurality of stored feature vectors based on a hierarchical navigable small world graph algorithm, wherein the feature distance between the second feature vector and the first feature vector is less than a preset distance; determining a second index identifier of the second feature vector; generating a data table corresponding to the first feature vector based on the second index identifier; and storing the first feature vector, the first index identifier, and the data table in the first storage area.

[0129] The preset distance in the above step can be a preset feature distance for determining the similarity of two feature vectors. The feature distance of two feature vectors can be calculated by a Euclidean distance algorithm, but is not limited thereto, and other algorithms can also be used. When the feature distance of two feature vectors is less than the preset distance, it can be determined that the two feature vectors have a high similarity and can be neighbors of each other. The data table can be a data table for storing the second feature vectors, that is, a neighbor table of the first feature vectors. The neighbor table can store the index id of each neighbor, that is, the index id of each second feature vector.

[0130] In the above embodiments of the present application, based on the second index identifier, the data table corresponding to the first feature vector is generated, including: determining the number of second feature vectors; and generating the data table based on the number and the second index identifier.

[0131] In the above embodiments of the present application, after the first feature vector and the first index identifier are stored in the first storage area in step S710, the method further includes: obtaining second image data; performing feature extraction on the second image data to obtain a third feature vector; determining a target storage area in a target storage block based on the hierarchical navigable small world graph algorithm; obtaining a target feature vector stored in the target storage area, wherein the feature distance between the target feature vector and the third feature vector is less than a preset distance; and determining target image data corresponding to the target feature vector to obtain a retrieval result corresponding to the second image data.

[0132] The second image data in the above step can be an image that needs to be subjected to image retrieval, and the third feature vector can be a feature vector that needs to be subjected to vector retrieval. The target feature vector can be a feature vector closest to the third feature vector based on the HNSW algorithm, the number of target feature vectors is a preset number of neighbors, the storage area in which the target feature vector is stored is the target storage area described above, and the storage block in which the target storage area is located is the target storage block described above.

[0133] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the schemes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0134] Embodiment 4

[0135] According to the embodiments of the present application, a data processing method is also provided. The method can also be applied to the operating environment as shown in Figure 1 and Figure 2 . Figure 8 is a flowchart of a fourth data processing method according to an embodiment of the present application. As shown in Figure 8 , the method can include the following steps:

[0136] Step S802, receiving first image data.

[0137] The first image data in the above steps can be an image that needs to be written into a storage device.

[0138] Step S804, performing feature extraction on the first image data to obtain a first feature vector.

[0139] Step S806, generating a first index identifier corresponding to the first feature vector.

[0140] The first index identifier in the above steps can be an index id corresponding to the first feature vector.

[0141] Step S808, determining a first storage area in a first storage block corresponding to the first index identifier.

[0142] The first storage block in the above steps can be an already allocated block or a newly allocated block. Each block contains multiple nodes, and each node is a storage area for a feature vector and can be used to store all data of the feature vector. Moreover, the size of each block in the same index file is consistent.

[0143] Optionally, when the first storage block is a newly added block, a block of storage space can be allocated from a storage system, and the size of the first storage block is determined by a preset feature vector size and a preset number, wherein the preset number can be the number of neighbors of the feature vector. Here, the neighbors can refer to the data of a plurality of feature vectors with the closest feature distance determined based on a nearest neighbor algorithm. The neighbor data can be pre-set according to actual needs, for example, it can be 4 or 5, but is not limited thereto.

[0144] Step S810, storing the first feature vector and the first index identifier into the first storage area.

[0145] Step S812, outputting a storage result corresponding to the first image data.

[0146] In the above embodiments of the present application, step S808 of determining the first storage area in the first storage block corresponding to the first index identifier comprises: determining a data block identifier and an offset of the first storage block based on the first index identifier; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset.

[0147] In the above embodiments of the present application, after step S810 of storing the first feature vector and the first index identifier into the first storage area, the method further comprises at least one of the following: mapping the first storage block to a disk; and storing the first storage block into a remote storage device through a preset interface.

[0148] The preset interface in the above step can be a read / write interface.

[0149] In the above embodiments of the present application, before the first feature vector and the first index identifier are stored into the first storage area in step S810, the method further comprises: determining a second feature vector in the plurality of stored feature vectors based on the hierarchical navigable small world graph algorithm, wherein the feature distance between the second feature vector and the first feature vector is less than a preset distance; determining a second index identifier of the second feature vector; generating a data table corresponding to the first feature vector based on the second index identifier; and storing the first feature vector, the first index identifier and the data table into the first storage area.

[0150] The preset distance in the above step can be a preset feature distance used to determine the similarity of two feature vectors. The feature distance between two feature vectors can be calculated by the Euclidean distance algorithm, but is not limited thereto, and other algorithms can also be used. When the feature distance between two feature vectors is less than the preset distance, it can be determined that the similarity of the two feature vectors is high, and they can be neighbors of each other. The data table can be a data table used to store the second feature vector, i.e., a neighbor table of the first feature vector. The neighbor table can store the index id of each neighbor, i.e., the index id of each second feature vector.

[0151] In the above embodiments of the present application, generating the data table corresponding to the first feature vector based on the second index identifier comprises: determining the number of second feature vectors; and generating the data table based on the number and the second index identifier.

[0152] In the above embodiments of the present application, after the first feature vector and the first index identifier are stored into the first storage area in step S810, the method further comprises: receiving second image data; performing feature extraction on the second image data to obtain a third feature vector; determining a target storage area in a target storage block based on the hierarchical navigable small world graph algorithm; obtaining a target feature vector stored in the target storage area, wherein the feature distance between the target feature vector and the third feature vector is less than a preset distance; determining target image data corresponding to the target feature vector to obtain a retrieval result corresponding to the second image data; and outputting the retrieval result.

[0153] The second image data in the above step can be an image that needs to be subjected to image retrieval, and the third feature vector can be a feature vector that needs to be subjected to vector retrieval. The target feature vector can be a feature vector closest to the third feature vector determined based on the HNSW algorithm. The number of target feature vectors is a preset number of neighbors. The storage area in which the target feature vector is stored is the target storage area described above. The storage block in which the target storage area is located is the target storage block described above.

[0154] The preferred embodiment of the present application is described in detail below in combination with Figure 9 the index structure design diagram shown in the figure. The index structure design can be deployed in the middle. As Figure 9 shown, the front-end client 20 can upload image data that needs to be stored to the server 10, and the server 10 can perform feature extraction on the image data after receiving the image data, obtain a feature vector of the image feature, and store the feature vector. After storage is complete, the server 10 can return the corresponding storage result to the front-end client 20. Optionally, when each new feature vector is added to the index, an index id that is sequentially incremented from 0 can be assigned in the index file, and the corresponding block id and offset in the block can be calculated according to the id. For each block, the mapping relationship between the index file and the memory block can be constructed by the mmap algorithm.

[0155] If it is determined that the block is a newly added block, a block of storage space can be allocated from a storage system (for example, it can be memory), the size of the block can be calculated in advance according to the size of the feature vector and the number of neighbors, and the sizes of all blocks corresponding to the same index file are consistent.

[0156] Each block contains a plurality of nodes, and each node is a storage area corresponding to a feature vector, which can store a feature vector, an index id, and a neighbor table of the feature vector, wherein the neighbor table contains the number of neighbors of the feature vector and the index id of each neighbor, for example, a neighbor table contains 4 neighbors, and the four index ids are 10, 17, 3 and 6. In addition, padding can be added at the end of the node to ensure that each storage area can be aligned according to a certain boundary.

[0157] It should be noted that the preferred implementation scheme and the embodiment 1 provided in the above embodiments of the present application have the same application scenarios and implementation processes as the scheme provided in the embodiment 1, but are not limited to the scheme provided in the embodiment 1.

[0158] Embodiment 5

[0159] According to the embodiments of the present application, a data processing device for implementing the above-mentioned data processing method is also provided, as Figure 10 shown, the device 1000 includes a receiving module 1002, a first generating module 1004, a first determining module 1006, a storage module 1008, and an output module 1010.

[0160] The receiving module 1002 is configured to receive a first feature vector; the first generating module 1004 is configured to generate a first index identifier corresponding to the first feature vector; the first determining module 1006 is configured to determine a first storage area in a first storage block corresponding to the first index identifier; the storage module 1008 is configured to store the first feature vector and the first index identifier into the first storage area; and the output module 1010 is configured to output a storage result corresponding to the first feature vector.

[0161] It should be noted that the receiving module 1002, the first generating module 1004, the first determining module 1006, the storage module 1008 and the output module 1010 correspond to steps S302 to S310 in Embodiment 1, and the five modules have the same instances and application scenarios as the corresponding steps, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the above modules can run in the computer terminal 10 provided in Embodiment 1 as part of the device.

[0162] In the above embodiments of the present application, the first determining module includes a first determining unit, a second determining unit and a third determining unit.

[0163] The first determining unit is configured to determine a data block identifier and an offset of the first storage block based on the first index identifier; the second determining unit is configured to determine the first storage block based on the data block identifier; and the third determining unit is configured to determine the first storage area based on the offset.

[0164] In the above embodiments of the present application, the device further includes at least one of a first mapping module and a second mapping module.

[0165] The first mapping module is configured to map the first storage block to a disk; and the second mapping module is configured to store the first storage block into a remote storage device through a preset interface.

[0166] In the above embodiments of the present application, the device further includes a second determining module, a third determining module and a second generating module.

[0167] The second determining module is configured to determine a second feature vector in the plurality of stored feature vectors based on a hierarchical navigable small-world graph algorithm, wherein a feature distance between the second feature vector and the first feature vector is less than a preset distance; the third determining module is configured to determine a second index identifier of the second feature vector; the second generating module is configured to generate a data table corresponding to the first feature vector based on the second index identifier; and the storage module is further configured to store the first feature vector, the first index identifier and the data table into the first storage area.

[0168] In the above embodiments of the present application, the second generating module includes a fourth determining unit and a generating unit.

[0169] The fourth determining unit is configured to determine a number of the second feature vectors; and the generating unit is configured to generate a data table based on the number and the second index identifier.

[0170] In the above-mentioned embodiments of the present application, the device further comprises a fourth determining module and an obtaining module.

[0171] The receiving module is further configured to receive a third feature vector; the fourth determining module is configured to determine a target storage area in a target storage block based on a hierarchical navigable small-world graph algorithm; the obtaining module is configured to obtain a target feature vector stored in the target storage area to obtain a search result, where a feature distance between the target feature vector and the third feature vector is less than a preset distance; and the output module is further configured to output the search result.

[0172] It should be noted that the preferred embodiments involved in the above-mentioned embodiments of the present application have the same application scenarios and implementation processes as the scheme provided in Embodiment 1, but are not limited to the scheme provided in Embodiment 1.

[0173] Embodiment 6

[0174] According to the embodiments of the present application, a data processing device for implementing the above-mentioned data processing method is further provided, as shown in the figure, the device 1100 comprises a first obtaining module 1102, a first generating module 1104, a first determining module 1106 and a storage module 1108. Figure 11

[0175] The first obtaining module 1102 is configured to obtain a first feature vector; the first generating module 1104 is configured to generate a first index identifier corresponding to the first feature vector; the first determining module 1106 is configured to determine a first storage area in a first storage block corresponding to the first index identifier; and the storage module 1108 is configured to store the first feature vector and the first index identifier into the first storage area.

[0176] It should be noted that the above-mentioned first obtaining module 1102, first generating module 1104, first determining module 1106 and storage module 1108 correspond to steps S602 to S608 in Embodiment 2, and the four modules have the same instances and application scenarios as the corresponding steps, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the above-mentioned modules as part of the device can run in the computer terminal 10 provided in Embodiment 1.

[0177] In the above-mentioned embodiments of the present application, the first determining module comprises a first determining unit, a second determining unit and a third determining unit.

[0178] ​The first determining unit is configured to determine the data block identifier and the offset of the first storage block based on the first index identifier; the second determining unit is configured to determine the first storage block based on the data block identifier; and the third determining unit is configured to determine the first storage region based on the offset.

[0179] In the above embodiments of the present application, the apparatus further comprises at least one of the following: a first mapping module and a second mapping module.

[0180] The first mapping module is configured to map the first storage block to the disk; and the second mapping module is configured to store the first storage block to the remote storage device through the preset interface.

[0181] In the above embodiments of the present application, the apparatus further comprises a second determining module, a third determining module and a second generating module.

[0182] The second determining module is configured to determine a second feature vector in the plurality of stored feature vectors based on the hierarchical navigable small world graph algorithm, wherein a feature distance between the second feature vector and the first feature vector is less than a preset distance; the third determining module is configured to determine a second index identifier of the second feature vector; and the second generating module is configured to generate a data table corresponding to the first feature vector based on the second index identifier; and the storage module is further configured to store the first feature vector, the first index identifier and the data table to the first storage region.

[0183] In the above embodiments of the present application, the second generating module comprises a fourth determining unit and a generating unit.

[0184] The fourth determining unit is configured to determine a number of the second feature vectors; and the generating unit is configured to generate the data table based on the number and the second index identifier.

[0185] In the above embodiments of the present application, the apparatus further comprises a fourth determining module and a second obtaining module.

[0186] The first obtaining module is configured to receive a third feature vector; the fourth determining module is configured to determine a target storage region in a target storage block based on the hierarchical navigable small world graph algorithm; and the second obtaining module is configured to obtain a target feature vector stored in the target storage region to obtain a search result, wherein a feature distance between the target feature vector and the third feature vector is less than a preset distance.

[0187] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the scheme provided in Embodiment 1, but are not limited to the scheme provided in Embodiment 1.

[0188] Embodiment 7

[0189] According to the embodiments of the present application, a data processing apparatus for implementing the above data processing method is further provided, as shown in the figure, the apparatus 1200 comprises: a first acquisition module 1202, an extraction module 1204, a first generation module 1206, a first determination module 1208 and a storage module 1210. Figure 12

[0190] The first acquisition module 1202 is configured to acquire first image data; the extraction module 1204 is configured to perform feature extraction on the first image data to obtain a first feature vector; the first generation module 1206 is configured to generate a first index identifier corresponding to the first feature vector; the first determination module 1208 is configured to determine a first storage area in a first storage block corresponding to the first index identifier; and the storage module 1210 is configured to store the first feature vector and the first index identifier into the first storage area.

[0191] It should be noted that the above first acquisition module 1202, extraction module 1204, first generation module 1206, first determination module 1208 and storage module 1210 correspond to steps S702 to S710 in Embodiment 3, and the five modules have the same instances and application scenarios as the corresponding steps, but are not limited to the above disclosed contents of Embodiment 1. It should be noted that the above modules as part of the apparatus can run in the computer terminal 10 provided in Embodiment 1.

[0192] In the above embodiments of the present application, the first determination module comprises: a first determination unit, a second determination unit and a third determination unit.

[0193] The first determination unit is configured to determine a data block identifier and an offset of the first storage block based on the first index identifier; the second determination unit is configured to determine the first storage block based on the data block identifier; and the third determination unit is configured to determine the first storage area based on the offset.

[0194] In the above embodiments of the present application, the apparatus further comprises at least one of: a first mapping module and a second mapping module.

[0195] The first mapping module is configured to map the first storage block to a disk; and the second mapping module is configured to store the first storage block into a remote storage device through a preset interface.

[0196] In the above embodiments of the present application, the apparatus further comprises: a second determination module, a third determination module and a second generation module.

[0197] ​The second determining module is configured to determine a second feature vector in the plurality of stored feature vectors based on the hierarchical navigable small-world graph algorithm, where a feature distance between the second feature vector and the first feature vector is less than a preset distance.

[0198] In the above embodiments of the present application, the second generating module comprises a fourth determining unit and a generating unit.

[0199] The fourth determining unit is configured to determine a number of the second feature vectors, and the generating unit is configured to generate the data table based on the number and the second index identifier.

[0200] In the above embodiments of the present application, the device further comprises a fourth determining module, a second obtaining module and a fifth determining module.

[0201] The first obtaining module is further configured to obtain second image data, the extracting module is further configured to perform feature extraction on the second image data to obtain a third feature vector, the fourth determining module is configured to determine a target storage region in a target storage block based on the hierarchical navigable small-world graph algorithm, the second obtaining module is configured to obtain a target feature vector stored in the target storage region, where a feature distance between the target feature vector and the third feature vector is less than a preset distance, and the fifth determining module is configured to determine target image data corresponding to the target feature vector to obtain a search result corresponding to the second image data.

[0202] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the scheme provided in Embodiment 1, but are not limited to the scheme provided in Embodiment 1.

[0203] Embodiment 8

[0204] According to the embodiments of the present application, a data processing device for implementing the above data processing method is further provided, as shown in FIG. 13, the device 1300 comprises a receiving module 1302, an extracting module 1304, a first generating module 1306, a first determining module 1308, a storage module 1310 and an output module 1312. Figure 13

[0205] ​The receiving module 1302 is configured to receive first image data; the extraction module 1304 is configured to perform feature extraction on the first image data to obtain a first feature vector; the first generation module 1306 is configured to generate a first index identifier corresponding to the first feature vector; the first determination module 1308 is configured to determine a first storage area in a first storage block corresponding to the first index identifier; the storage module 1310 is configured to store the first feature vector and the first index identifier into the first storage area; and the output module 1312 is configured to output a storage result corresponding to the first image data.

[0206] It should be noted that the receiving module 1302, the extraction module 1304, the first generation module 1306, the first determination module 1308, the storage module 1310 and the output module 1312 correspond to steps S802 to S812 in Embodiment 4, and the six modules have the same instances and application scenarios as the corresponding steps, but are not limited to the above-mentioned disclosed contents of Embodiment 1. It should be noted that the above-mentioned modules as part of the device can run in the computer terminal 10 provided in Embodiment 1.

[0207] In the above-mentioned embodiments of the present application, the first determination module comprises a first determination unit, a second determination unit and a third determination unit.

[0208] The first determination unit is configured to determine a data block identifier and an offset of the first storage block based on the first index identifier; the second determination unit is configured to determine the first storage block based on the data block identifier; and the third determination unit is configured to determine the first storage area based on the offset.

[0209] In the above-mentioned embodiments of the present application, the device further comprises at least one of a first mapping module and a second mapping module.

[0210] The first mapping module is configured to map the first storage block to a disk; and the second mapping module is configured to store the first storage block into a remote storage device through a preset interface.

[0211] In the above-mentioned embodiments of the present application, the device further comprises a second determination module, a third determination module and a second generation module.

[0212] The second determination module is configured to determine a second feature vector in the plurality of stored feature vectors based on a hierarchical navigable small world graph algorithm, wherein a feature distance between the second feature vector and the first feature vector is less than a preset distance; the third determination module is configured to determine a second index identifier of the second feature vector; the second generation module is configured to generate a data table corresponding to the first feature vector based on the second index identifier; and the storage module is further configured to store the first feature vector, the first index identifier and the data table into the first storage area.

[0213] In the above embodiments of the present application, the second generation module comprises a fourth determination unit and a generation unit.

[0214] The fourth determination unit is configured to determine the number of second feature vectors, and the generation unit is configured to generate the data table based on the number and the second index identifier.

[0215] In the above embodiments of the present application, the device further comprises a fourth determination module, an acquisition module and a fifth determination module.

[0216] The receiving module is further configured to receive second image data, and the extraction module is further configured to perform feature extraction on the second image data to obtain a third feature vector. The fourth determination module is configured to determine a target storage area in the target storage block based on the hierarchical navigable small-world graph algorithm. The acquisition module is configured to acquire a target feature vector stored in the target storage area, wherein the feature distance between the target feature vector and the third feature vector is less than a preset distance. The fifth determination module is configured to determine target image data corresponding to the target feature vector to obtain a retrieval result corresponding to the second image data. The output module is further configured to output the retrieval result.

[0217] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the schemes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0218] Embodiment 9

[0219] According to the embodiments of the present application, a data processing method is also provided, which can also be applied to the operating environment as shown in Figure 1 and Figure 2 . Figure 14 is a flowchart of a fourth data processing method according to the embodiments of the present application. As shown in Figure 14 , the method can comprise the following steps:

[0220] In step S1402, a first feature vector is acquired by calling a first interface, wherein the first interface comprises a first parameter, and the parameter value of the first parameter is the first feature vector.

[0221] The first interface in the above step can be an interface for data interaction between a server and a client. The client can pass the first feature vector into an interface function as a parameter of the interface function, so as to achieve the purpose of uploading the first feature vector to the server.

[0222] In step S1404, a first index identifier corresponding to the first feature vector is generated.

[0223] In step S1406, a first storage area in a first storage block corresponding to the first index identifier is determined.

[0224] Step S1408, store the first feature vector and the first index identifier to the first storage area.

[0225] Step S1410, output the storage result corresponding to the first feature vector by calling a second interface, wherein the second interface comprises a second parameter, and a parameter value of the second parameter is the storage result.

[0226] The second interface in the above steps can be an interface for data interaction between a server and a client. The server can pass the storage result into an interface function as a parameter of the interface function, so as to realize the purpose of issuing the storage result to the client.

[0227] In the above embodiments of the present application, step S1406, determining the first storage area in the first storage block corresponding to the first index identifier comprises: determining the data block identifier and the offset of the first storage block based on the first index identifier; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset.

[0228] In the above embodiments of the present application, after step S1408, storing the first feature vector and the first index identifier to the first storage area, the method further comprises at least one of the following: mapping the first storage block to a disk; and storing the first storage block to a remote storage device through a preset interface.

[0229] In the above embodiments of the present application, before step S1408, storing the first feature vector and the first index identifier to the first storage area, the method further comprises: determining a second feature vector in the plurality of stored feature vectors based on the hierarchical navigable small world graph algorithm, wherein a feature distance between the second feature vector and the first feature vector is less than a preset distance; determining a second index identifier of the second feature vector; generating a data table corresponding to the first feature vector based on the second index identifier; and storing the first feature vector, the first index identifier and the data table to the first storage area.

[0230] In the above embodiments of the present application, generating the data table corresponding to the first feature vector based on the second index identifier comprises: determining a number of the second feature vectors; and generating the data table based on the number and the second index identifier.

[0231] In the above embodiments of the present application, after step S1408, storing the first feature vector and the first index identifier to the first storage area, the method further comprises: receiving a third feature vector; determining a target storage area in a target storage block based on the hierarchical navigable small world graph algorithm; obtaining a target feature vector stored in the target storage area to obtain a retrieval result, wherein a feature distance between the target feature vector and the third feature vector is less than a preset distance; and outputting the retrieval result.

[0232] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the schemes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0233] Embodiment 10

[0234] According to the embodiments of the present application, a data processing method is also provided, which can also be applied to the running environment as shown in Figure 1 and Figure 2 . Figure 15 is a flowchart of a fifth data processing method according to the embodiments of the present application. As shown in Figure 15 , the method can include the following steps:

[0235] Step S1502, acquiring first image data by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is the first image data.

[0236] Step S1504, performing feature extraction on the first image data to obtain a first feature vector.

[0237] Step S1506, generating a first index identifier corresponding to the first feature vector.

[0238] Step S1508, determining a first storage area in a first storage block corresponding to the first index identifier.

[0239] Step S1510, storing the first feature vector and the first index identifier to the first storage area.

[0240] Step S1512, outputting a storage result corresponding to the first image data by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the storage result.

[0241] In the above embodiments of the present application, step S1508 of determining the first storage area in the first storage block corresponding to the first index identifier includes: determining a data block identifier and an offset of the first storage block based on the first index identifier; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset.

[0242] In the above embodiments of the present application, after step S1510 of storing the first feature vector and the first index identifier to the first storage area, the method further includes at least one of the following: mapping the first storage block to a disk; and storing the first storage block to a remote storage device through a preset interface.

[0243] In the above embodiments of the present application, before storing the first feature vector and the first index identifier into the first storage area at step S1510, the method further comprises: determining a second feature vector in the plurality of stored feature vectors based on the hierarchical navigable small world graph algorithm, wherein a feature distance between the second feature vector and the first feature vector is less than a preset distance; determining a second index identifier of the second feature vector; generating a data table corresponding to the first feature vector based on the second index identifier; and storing the first feature vector, the first index identifier and the data table into the first storage area.

[0244] In the above embodiments of the present application, generating the data table corresponding to the first feature vector based on the second index identifier comprises: determining a number of the second feature vectors; and generating the data table based on the number and the second index identifier.

[0245] In the above embodiments of the present application, after storing the first feature vector and the first index identifier into the first storage area at step S1510, the method further comprises: receiving second image data; performing feature extraction on the second image data to obtain a third feature vector; determining a target storage area in a target storage block based on the hierarchical navigable small world graph algorithm; obtaining a target feature vector stored in the target storage area, wherein a feature distance between the target feature vector and the third feature vector is less than a preset distance; determining target image data corresponding to the target feature vector to obtain a retrieval result corresponding to the second image data; and outputting the retrieval result.

[0246] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as the schemes and application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0247] Embodiment 11

[0248] According to the embodiments of the present application, a data processing system is further provided, comprising:

[0249] a processor, and

[0250] a memory connected with the processor, configured to provide the processor with instructions for processing the following processing steps: receiving first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; and outputting a storage result corresponding to the first image data.

[0251] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as the schemes and application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0252] Embodiment 12

[0253] The embodiment of the present application can provide a computer terminal, which can be any one of computer terminal devices in a computer terminal group. Alternatively, in the embodiment, the computer terminal can be replaced by a mobile terminal or other terminal device.

[0254] Alternatively, in the embodiment, the computer terminal can be located in at least one of a plurality of network devices in a computer network.

[0255] In the embodiment, the computer terminal can execute program codes of the following steps in the data processing method: receiving first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; and outputting a storage result corresponding to the first image data.

[0256] Alternatively, Figure 16 is a structural block diagram of a computer terminal according to an embodiment of the present application. As shown in Figure 16 The computer terminal A can include one or more (only one is shown in the figure) processors 1602 and a memory 1604.

[0257] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the data processing method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implements the above-mentioned data processing method. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the terminal A through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0258] The processor can call information and application programs stored in the memory through the transmission device to execute the following steps: receiving a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; and outputting a storage result corresponding to the first feature vector.

[0259] Optionally, the processor can further execute program codes of the following steps: determining the data block identifier and the offset of the first storage block based on the first index identifier; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset.

[0260] Optionally, the processor can further execute program codes of at least one of the following steps: mapping the first storage block to the disk after storing the first feature vector and the first index identifier to the first storage area; and storing the first storage block to the remote storage device through the preset interface.

[0261] Optionally, the processor can further execute program codes of the following steps: determining a second feature vector in the plurality of stored feature vectors based on the hierarchical navigable small world graph algorithm before storing the first feature vector and the first index identifier to the first storage area, wherein the feature distance between the second feature vector and the first feature vector is less than the preset distance; determining a second index identifier of the second feature vector; generating a data table corresponding to the first feature vector based on the second index identifier; and storing the first feature vector, the first index identifier and the data table to the first storage area.

[0262] Optionally, the processor can further execute program codes of the following steps: determining the number of the second feature vectors; and generating the data table based on the number and the second index identifier.

[0263] Optionally, the processor can further execute program codes of the following steps: receiving a third feature vector after storing the first feature vector and the first index identifier to the first storage area; determining a target storage area in a target storage block based on the hierarchical navigable small world graph algorithm; obtaining a target feature vector stored in the target storage area to obtain a retrieval result, wherein the feature distance between the target feature vector and the third feature vector is less than the preset distance; and outputting the retrieval result.

[0264] Optionally, the processor can further execute program codes of the following steps: obtaining a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; and storing the first feature vector and the first index identifier to the first storage area.

[0265] Optionally, the processor can further execute program codes of the following steps: obtaining a third feature vector after storing the first feature vector and the first index identifier to the first storage area; determining a target storage area in a target storage block based on the hierarchical navigable small world graph algorithm; obtaining a target feature vector stored in the target storage area to obtain a retrieval result, wherein the feature distance between the target feature vector and the third feature vector is less than the preset distance.

[0266] The processor can call information and application programs stored in the memory through the transmission device to execute the following steps: obtaining first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; and storing the first feature vector and the first index identifier into the first storage area.

[0267] Optionally, the processor can further execute program codes of the following steps: after storing the first feature vector and the first index identifier into the first storage area, obtaining second image data; performing feature extraction on the second image data to obtain a third feature vector; determining a target storage area in a target storage block based on the hierarchical navigable small-world graph algorithm; obtaining a target feature vector stored in the target storage area, wherein a feature distance between the target feature vector and the third feature vector is less than a preset distance; determining target image data corresponding to the target feature vector to obtain a retrieval result corresponding to the second image data.

[0268] The processor can call information and application programs stored in the memory through the transmission device to execute the following steps: receiving first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; and outputting a storage result corresponding to the first image data.

[0269] Optionally, the processor can further execute program codes of the following steps: after storing the first feature vector and the first index identifier into the first storage area, receiving second image data; performing feature extraction on the second image data to obtain a third feature vector; determining a target storage area in a target storage block based on the hierarchical navigable small-world graph algorithm; obtaining a target feature vector stored in the target storage area, wherein a feature distance between the target feature vector and the third feature vector is less than a preset distance; determining target image data corresponding to the target feature vector to obtain a retrieval result corresponding to the second image data; and outputting the retrieval result.

[0270] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: obtaining first image data by calling a first interface, wherein the first interface comprises: a first parameter, and a parameter value of the first parameter is the first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; and outputting a storage result corresponding to the first image data by calling a second interface, wherein the second interface comprises: a second parameter, and a parameter value of the second parameter is the storage result.

[0271] The processor can call information and application programs stored in the memory through the transmission device to perform the following steps: obtaining a first feature vector by calling a first interface, wherein the first interface comprises: a first parameter, and a parameter value of the first parameter is the first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; and outputting a storage result corresponding to the first feature vector by calling a second interface, wherein the second interface comprises: a second parameter, and a parameter value of the second parameter is the storage result.

[0272] By adopting the embodiment of the application, a vector writing and retrieval scheme is provided. The index identifier of the feature vector can be generated immediately after receiving the feature vector each time, without performing an index training process, thereby realizing streaming index in the vector retrieval scene, achieving the technical effect that the feature vector is added and searched immediately, and reducing the index construction cost, thereby solving the technical problem in the related art that the index construction in the vector retrieval method needs data training, resulting in the need to reconstruct the index when a new feature vector is written, and the index construction cost is high.

[0273] Those skilled in the art can understand that, Figure 16 The structure shown is only schematic, and the computer terminal can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, and the like. Figure 16 It does not limit the structure of the electronic device. For example, the computer terminal A can further include more or fewer components (such as a network interface, a display device, etc.) than Figure 16 shown, or have a different configuration from Figure 16 shown.

[0274] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing the terminal device related hardware through a program, and the program can be stored in a computer readable storage medium, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0275] Embodiment 13

[0276] The embodiments of the present application also provide a storage medium. Optionally, in the embodiments, the storage medium can be used to save the program code executed by the data processing method provided by the embodiments.

[0277] Optionally, in the embodiments, the storage medium can be located in any one of the computer terminals in the computer terminal group in the computer network, or in any one of the mobile terminals in the mobile terminal group.

[0278] Optionally, in the embodiments, the storage medium is configured to store program code for performing the following steps: receiving a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; and outputting a storage result corresponding to the first feature vector.

[0279] Optionally, the storage medium is further configured to store program code for performing the following steps: determining a data block identifier and an offset of the first storage block based on the first index identifier; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset.

[0280] Optionally, the storage medium is further configured to store program code for performing at least one of the following steps: mapping the first storage block to a disk after storing the first feature vector and the first index identifier into the first storage area; and storing the first storage block into a remote storage device through a preset interface.

[0281] Optionally, the storage medium is further configured to store program code for performing the following steps: determining a second feature vector in the plurality of stored feature vectors based on the hierarchical navigable small world graph algorithm before storing the first feature vector and the first index identifier into the first storage area, wherein a feature distance between the second feature vector and the first feature vector is less than a preset distance; determining a second index identifier of the second feature vector; generating a data table corresponding to the first feature vector based on the second index identifier; and storing the first feature vector, the first index identifier and the data table into the first storage area.

[0282] Optionally, the storage medium is further configured to store program code for determining a number of the second feature vectors; and generating a data table based on the number and the second index identifier.

[0283] Optionally, the storage medium is further configured to store program code for receiving a third feature vector after storing the first feature vector and the first index identifier into the first storage area; determining a target storage area in a target storage block based on the hierarchical navigable small-world graph algorithm; obtaining a target feature vector stored in the target storage area to obtain a search result, wherein a feature distance between the target feature vector and the third feature vector is less than a preset distance; and outputting the search result.

[0284] Optionally, in the embodiment, the storage medium is configured to store program code for obtaining the first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; and storing the first feature vector and the first index identifier into the first storage area.

[0285] Optionally, the storage medium is further configured to store program code for obtaining a third feature vector after storing the first feature vector and the first index identifier into the first storage area; determining a target storage area in a target storage block based on the hierarchical navigable small-world graph algorithm; obtaining a target feature vector stored in the target storage area, wherein a feature distance between the target feature vector and the third feature vector is less than a preset distance; and determining target image data corresponding to the target feature vector to obtain a search result corresponding to the second image data.

[0286] Optionally, in the embodiment, the storage medium is configured to store program code for obtaining the first image data; performing feature extraction on the first image data to obtain the first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; and storing the first feature vector and the first index identifier into the first storage area.

[0287] Optionally, the storage medium is further configured to store program code for obtaining second image data after storing the first feature vector and the first index identifier into the first storage area; performing feature extraction on the second image data to obtain a third feature vector; determining a target storage area in a target storage block based on the hierarchical navigable small-world graph algorithm; obtaining a target feature vector stored in the target storage area, wherein a feature distance between the target feature vector and the third feature vector is less than a preset distance; and determining target image data corresponding to the target feature vector to obtain a search result corresponding to the second image data.

[0288] Optionally, in the embodiment, the storage medium is configured to store program code for performing the following steps: receiving the first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; and outputting a storage result corresponding to the first image data.

[0289] Optionally, the storage medium is further configured to store program code for performing the following steps: after storing the first feature vector and the first index identifier into the first storage area, receiving second image data; performing feature extraction on the second image data to obtain a third feature vector; determining a target storage area in a target storage block based on the hierarchical navigable small-world graph algorithm; obtaining a target feature vector stored in the target storage area, wherein a feature distance between the target feature vector and the third feature vector is less than a preset distance; determining target image data corresponding to the target feature vector to obtain a search result corresponding to the second image data; and outputting the search result.

[0290] Optionally, in the embodiment, the storage medium is configured to store program code for performing the following steps: obtaining the first image data by calling a first interface, wherein the first interface comprises: a first parameter, and a parameter value of the first parameter is the first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; and outputting a storage result corresponding to the first image data by calling a second interface, wherein the second interface comprises: a second parameter, and a parameter value of the second parameter is the storage result.

[0291] Optionally, in the embodiment, the storage medium is configured to store program code for performing the following steps: obtaining the first feature vector by calling a first interface, wherein the first interface comprises: a first parameter, and a parameter value of the first parameter is the first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; and outputting a storage result corresponding to the first feature vector by calling a second interface, wherein the second interface comprises: a second parameter, and a parameter value of the second parameter is the storage result.

[0292] The serial numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0293] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0294] In several embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other manners. For example, the described unit embodiments are merely schematic, and the division of units is merely logical function division, and there can be other division manners in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, units or modules, and can be in electrical, mechanical or other forms.

[0295] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0296] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0297] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0298] The above description is merely the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. A data processing method, comprising: receiving first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index; storing the first feature vector and the first index into the first storage area; outputting a storage result corresponding to the first image data; wherein after storing the first feature vector and the first index into the first storage area, the method further comprises at least one of the following: mapping the first storage block into a disk; storing the first storage block into a remote storage device through a preset interface; wherein determining the first storage area in the first storage block corresponding to the first index comprises: determining a data block identifier and an offset of the first storage block based on the first index; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset. 2.A data processing method, comprising: receiving a first feature vector; generating a first index corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index; storing the first feature vector and the first index into the first storage area; outputting a storage result corresponding to the first feature vector; wherein after storing the first feature vector and the first index into the first storage area, the method further comprises at least one of the following: mapping the first storage block into a disk; storing the first storage block into a remote storage device through a preset interface; wherein determining the first storage area in the first storage block corresponding to the first index comprises: determining a data block identifier and an offset of the first storage block based on the first index; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset. 3.A data processing method, comprising: obtaining a first feature vector; generating a first index corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index; storing the first feature vector and the first index into the first storage area; wherein after storing the first feature vector and the first index into the first storage area, the method further comprises at least one of the following: mapping the first storage block into a disk; storing the first storage block into a remote storage device through a preset interface; wherein determining the first storage area in the first storage block corresponding to the first index comprises: determining a data block identifier and an offset of the first storage block based on the first index; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset.

4. The method of claim 3, wherein, before storing the first feature vector and the first index into the first storage area, the method further comprises: determine a second feature vector in the plurality of stored feature vectors based on the hierarchical navigable small world graph algorithm, wherein the second feature vector has a feature distance to the first feature vector less than a preset distance; determine a second index identifier of the second feature vector; generate a data table corresponding to the first feature vector based on the second index identifier; store the first feature vector, the first index identifier and the data table into the first storage area.

5. The method of claim 4, wherein, generating the data table corresponding to the first feature vector based on the second index identifier includes: determining a number of the second feature vectors; generating the data table based on the number and the second index identifier.

6. The method of claim 3, wherein, After storing the first feature vector and the first index identifier into the first storage area, the method further includes: obtaining a third feature vector; determining a target storage area in a target storage block based on the hierarchical navigable small world graph algorithm; obtaining a target feature vector stored in the target storage area to obtain a retrieval result, wherein the target feature vector has a feature distance to the third feature vector less than a preset distance.

7. The method of claim 3, wherein, The first index identifier is generated according to a self-incrementing number generation algorithm.

8. The method of claim 3, wherein, The size of the first storage block is determined by a preset feature vector size and a preset number.

9. The method of claim 3, wherein, Preset padding data is stored into the first storage area.

10. A data processing method, comprising: obtaining first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; wherein after storing the first feature vector and the first index identifier into the first storage area, the method further includes at least one of the following: mapping the first storage block into a disk; storing the first storage block into a remote storage device through a preset interface; wherein determining the first storage area in the first storage block corresponding to the first index identifier includes: determining a data block identifier and an offset of the first storage block based on the first index identifier; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset.

11. A data processing method, comprising: obtaining first image data by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is the first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; outputting a storage result corresponding to the first image data by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the storage result. wherein, after storing the first feature vector and the first index identifier into the first storage area, the method further comprises at least one of: mapping the first storage block into a disk; storing the first storage block into a remote storage device through a preset interface; wherein, determining the first storage area in the first storage block corresponding to the first index identifier comprises: determining a data block identifier and an offset of the first storage block based on the first index identifier; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset.

12. A data processing method, comprising: obtaining a first feature vector by calling a first interface, wherein the first interface comprises: a first parameter, and a parameter value of the first parameter is the first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; storing the first feature vector and the first index identifier into the first storage area; outputting a storage result corresponding to the first feature vector by calling a second interface, wherein the second interface comprises: a second parameter, and a parameter value of the second parameter is the storage result; wherein, after storing the first feature vector and the first index identifier into the first storage area, the method further comprises at least one of: mapping the first storage block into a disk; storing the first storage block into a remote storage device through a preset interface; wherein, determining the first storage area in the first storage block corresponding to the first index identifier comprises: determining a data block identifier and an offset of the first storage block based on the first index identifier; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset.

13. A computer readable storage medium, the computer readable storage medium comprising a stored program, wherein, controlling a device in which the computer readable storage medium is located to perform the data processing method of any one of claims 1 to 12 when the program runs.

14. A computer terminal comprising: a memory and a processor, wherein the processor is configured to run a program stored in the memory, and the program performs the data processing method of any one of claims 1 to 12 when the program runs.

15. A data processing system, comprising: a processor; and a memory connected with the processor, configured to provide the processor with instructions to process the following processing steps: receiving first image data; performing feature extraction on the first image data to obtain a first feature vector; generating a first index identifier corresponding to the first feature vector; determining a first storage area in a first storage block corresponding to the first index identifier; and storing the first feature vector and the first index identifier into the first storage area. output the storage result corresponding to the first image data; wherein, after storing the first feature vector and the first index identifier into the first storage area, the processing steps further include at least one of the following: mapping the first storage block to a disk; storing the first storage block to a remote storage device through a preset interface; wherein, determining the first storage area in the first storage block corresponding to the first index identifier includes: determining a data block identifier and an offset of the first storage block based on the first index identifier; determining the first storage block based on the data block identifier; and determining the first storage area based on the offset.

Images