Data processing method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of computer processing, provides a data processing method and device, equipment, a storage medium and a program product, and can avoid data loss caused by equipment failure or misoperation. The method comprises the following steps: when set backup time arrives, obtaining a vector data set of a vector database and an unstructured data set of object storage equipment; backing up the vector data set to a standby vector database; if the available storage space of the object storage equipment does not meet the requirement, dividing the unstructured data set into a first unstructured data set and a second unstructured data set according to the generation time and the search times of the unstructured data; migrating the first unstructured data set from the object storage device to the target object storage device, and backing up the second unstructured data set to the standby object storage device.

Description

Technical Field

[0001] The present application relates to the field of computer processing technology, and in particular to a data processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] Object Storage Device (OSD) is a device based on object storage technology. Object storage devices can store unstructured data such as images, audio and text. Unstructured data such as video, audio and text can use vector data as a search index; the search index of unstructured data can be stored in a vector database.

[0003] Vector databases and object storage devices may sometimes fail or fail to operate properly, resulting in partial data loss. Summary of the invention

[0004] Based on this, it is necessary to provide a data processing method, apparatus, computer equipment, storage medium and computer program product to address the above technical issues.

[0005] The present application provides a data processing method, the method comprising:

[0006] When the set backup time arrives, the vector data set of the vector database and the unstructured data set of the object storage device are obtained; the vector data is used to search for unstructured data;

[0007] Backing up the vector data set to a backup vector database;

[0008] If the available storage space of the object storage device does not meet the requirement, the unstructured data set is divided into a first unstructured data set and a second unstructured data set according to the generation time and search times of the unstructured data; the unstructured data in the first unstructured data set has a low search time and the generation time is far from the backup time, and the unstructured data in the second unstructured data set has a high search time or the generation time is close to the backup time;

[0009] The first unstructured data set is migrated from the object storage device to a target object storage device, and the second unstructured data set is backed up to a backup object storage device.

[0010] In one embodiment, the method further comprises:

[0011] In response to a data search request from a user device, determining target vector data in the vector database and sending the target vector data to the object storage device;

[0012] If the object storage device finds the corresponding unstructured data locally using the target vector data as an index, the corresponding unstructured data is fed back to the user device;

[0013] If the object storage device fails to find the corresponding unstructured data locally using the target vector data as an index, sending the target vector data to the target object storage device;

[0014] When the target object storage device finds the corresponding unstructured data locally using the target vector data as an index, the corresponding unstructured data is fed back to the user device.

[0015] In one embodiment, when there are multiple target object storage devices, sending the target vector data to the target object storage device includes:

[0016] Obtaining a time range for generating unstructured data associated with each of the target object storage devices;

[0017] Obtaining the generation time of the target vector data;

[0018] Acquire the generation time range of the target vector data within which the generation time of the target vector data falls, and determine the target object storage device associated with the determined generation time range of the target vector data;

[0019] The target vector data is sent to a determined target object storage device.

[0020] In one embodiment, the unstructured data set is divided into a first unstructured data set and a second unstructured data set according to the generation time and the number of searches of the unstructured data, including:

[0021] Determine, in the unstructured data set, unstructured data whose time difference between the generation time and the backup time is less than a time difference threshold, to form a candidate unstructured data set;

[0022] Sorting the unstructured data of the candidate unstructured data set according to the number of searches for the unstructured data to obtain an unstructured data sequence;

[0023] Determining unstructured data with a low search frequency in the unstructured data sequence to form a first unstructured data set;

[0024] A second unstructured data set is obtained according to the unstructured data in the unstructured data set except the first unstructured data set.

[0025] In one embodiment, determining unstructured data with a low search count in the unstructured data sequence includes:

[0026] If the unstructured data is sorted from high to low according to the number of search times, the unstructured data sorted later in the unstructured data sequence is determined as the unstructured data with a low number of search times;

[0027] If the unstructured data is sorted from low to high according to the number of search times, the unstructured data sorted first in the unstructured data sequence is determined as the unstructured data with a low number of search times.

[0028] In one embodiment, the method further comprises:

[0029] Obtaining available storage space of the object storage device;

[0030] If the available storage space is lower than the storage space threshold, it is determined that the available storage space of the object storage device does not meet the requirement.

[0031] The present application provides a data processing device, the device comprising:

[0032] The data acquisition module is used to acquire the vector data set of the vector database and the unstructured data set of the object storage device when the set backup time arrives; the vector data is used to search for unstructured data;

[0033] A vector data backup module, used for backing up the vector data set to a backup vector database;

[0034] An unstructured data partitioning module is used to, if the available storage space of the object storage device does not meet the requirement, divide the unstructured data set into a first unstructured data set and a second unstructured data set according to the generation time and search times of the unstructured data; the unstructured data in the first unstructured data set has a low search time and the generation time is far from the backup time, and the unstructured data in the second unstructured data set has a high search time or the generation time is close to the backup time;

[0035] The unstructured data processing module is used to migrate the first unstructured data set from the object storage device to a target object storage device, and back up the second unstructured data set to a backup object storage device.

[0036] The present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the above method.

[0037] The present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to execute the above method.

[0038] The present application provides a computer program product, on which a computer program is stored, and the computer program is executed by a processor to perform the above method.

[0039] The above-mentioned data processing method, device, computer equipment, storage medium and computer program product obtain the vector data set of the vector database and the unstructured data set of the object storage device when the set backup time arrives; the vector data is used to search for unstructured data; the vector data set is backed up to the backup vector database; if the available storage space of the object storage device does not meet the requirements, the unstructured data set is divided into a first unstructured data set and a second unstructured data set according to the generation time and search number of the unstructured data; the unstructured data in the first unstructured data set has a low search number and the generation time is far from the backup time, and the unstructured data in the second unstructured data set has a high search number or the generation time is close to the backup time; the first unstructured data set is migrated from the object storage device to the target object storage device, and the second unstructured data set is backed up to the backup object storage device. In the solution provided in the present application, when the set backup time arrives, the vector database and the object storage device are backed up to avoid data loss due to equipment failure or operational error; and, considering that the amount of unstructured data stored in the object storage device is large, therefore, when backing up the object storage device, it is possible to first determine whether the available storage space of the object storage device meets the requirements. If it does not meet the requirements, the unstructured data with a high number of lookups and a generation time far from the backup time can be migrated to the target object storage device to increase the available storage space of the object storage device; as for other unstructured data of the object storage device, it can be backed up to a backup object storage device. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 An application environment diagram of a data processing method in an embodiment;

[0042] Figure 2 is a flow chart of a data processing method in one embodiment;

[0043] Figure 3 A schematic diagram of active / standby switching in one embodiment;

[0044] Figure 4 is a flow chart of a data processing method in another embodiment;

[0045] Figure 5 is a structural block diagram of a data processing device in one embodiment;

[0046] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] The data processing method provided in this application involves Figure 1 The vector database, spare vector database, object storage device, spare object storage device and target object storage device shown. The steps included in the method can be executed by a computer device. The method includes Figure 2 The steps shown are:

[0049] Step S201, when the set backup time arrives, obtain the vector data set of the vector database and the unstructured data set of the object storage device.

[0050] Vector data is used to search unstructured data.

[0051] The user can pre-set the time when backup is required, which can be called the backup time. When the backup time arrives, the backup task can be executed to obtain the vector data set of the vector database and the unstructured data set of the object storage device. The vector data can be used as an index to find the unstructured data.

[0052] Step S202, backing up the vector data set to a backup vector database.

[0053] After the vector dataset is obtained, the vector dataset may be copied and the copied vector dataset may be stored in a standby database.

[0054] In some scenarios, users can read data from or write data to the vector database, or read data from or write data to the backup vector database. Figure 3 , through the active-standby switching, the user's read instruction or write instruction can be sent to the vector database or the standby vector database through the gateway. When the user's write instruction is sent to the standby vector database through the gateway, at this time, new data is written to the standby vector database; the new data can be synchronously backed up to the vector database.

[0055] After obtaining the unstructured data set of the object storage device, it can be determined whether the available storage space of the object storage device meets the requirement.

[0056] Specifically, the method provided in the present application also includes: obtaining available storage space of the object storage device; if the available storage space is lower than a storage space threshold, determining that the available storage space of the object storage device does not meet the requirement.

[0057] If the available storage space is greater than or equal to the storage space threshold, it is determined that the available storage space of the object storage device meets the requirement. At this time, the unstructured data set can be directly copied and the copied unstructured data can be stored in the backup object storage device.

[0058] Step S203: if the available storage space of the object storage device does not meet the requirement, the unstructured data set is divided into a first unstructured data set and a second unstructured data set according to the generation time and the number of searches of the unstructured data.

[0059] If the available storage space of the object storage device does not meet the requirement, the generation time and search times of the unstructured data can be obtained; the search times of the unstructured data are the number of times the unstructured data is searched.

[0060] According to the generation time and search times of the unstructured data, the unstructured data set can be divided into a first unstructured data set and a second unstructured data set. The unstructured data in the first unstructured data set has a low search time and the generation time is far from the backup time, and the unstructured data in the second unstructured data set has a high search time or the generation time is close to the backup time.

[0061] Step S204: Migrate the first unstructured data set from the object storage device to the target object storage device, and back up the second unstructured data set to the backup object storage device.

[0062] The unstructured data in the first unstructured data set has a low number of search times and is generated far from the backup time. The possibility of this type of unstructured data being subsequently searched by users is low. Therefore, the first unstructured data set can be migrated from the object storage device to another object storage device, which can be called a target object storage device.

[0063] The unstructured data in the second unstructured data set has a high number of search times or is generated close to the backup time; one type of unstructured data has a high number of search times and is more likely to be searched by users in the future; another type of unstructured data is generated close to the backup time, so it is difficult to judge the possibility of user search; therefore, the unstructured data with a high number of search times and the unstructured data generated close to the backup time can be copied, and the copied data can be stored in the backup object storage device.

[0064] In the above data processing method, when the set backup time arrives, the vector data set of the vector database and the unstructured data set of the object storage device are obtained; the vector data is used to search for unstructured data; the vector data set is backed up to the backup vector database; if the available storage space of the object storage device does not meet the requirements, the unstructured data set is divided into a first unstructured data set and a second unstructured data set according to the generation time and search times of the unstructured data; the unstructured data in the first unstructured data set has a low search time and the generation time is far from the backup time, and the unstructured data in the second unstructured data set has a high search time or the generation time is close to the backup time; the first unstructured data set is migrated from the object storage device to the target object storage device, and the second unstructured data set is backed up to the backup object storage device. In the solution provided in the present application, when the set backup time arrives, the vector database and the object storage device are backed up to avoid data loss due to equipment failure or operational error; and, considering that the amount of unstructured data stored in the object storage device is large, therefore, when backing up the object storage device, it is possible to first determine whether the available storage space of the object storage device meets the requirements. If it does not meet the requirements, the unstructured data with a high number of lookups and a generation time far from the backup time can be migrated to the target object storage device to increase the available storage space of the object storage device; as for other unstructured data of the object storage device, it can be backed up to a backup object storage device.

[0065] In one embodiment, the method provided by the present application further includes:

[0066] In response to a data search request from a user device, target vector data is determined in a vector database and the target vector data is sent to an object storage device; if the object storage device finds the corresponding unstructured data locally using the target vector data as an index, the corresponding unstructured data is fed back to the user device; if the object storage device does not find the corresponding unstructured data locally using the target vector data as an index, the target vector data is sent to a target object storage device; when the target object storage device finds the corresponding unstructured data locally using the target vector data as an index, the corresponding unstructured data is fed back to the user device.

[0067] In this embodiment, after receiving a data search request sent by a user through a user device, the data required by the user can be determined according to the data search request, and then the corresponding vector data can be determined in the vector database, and the vector data is called the target vector data. After determining the target vector data, the target vector data can be sent to the object storage device. The object storage device can use the target vector data as an index to search locally for corresponding unstructured data.

[0068] If the object storage device finds the corresponding unstructured data locally, the corresponding unstructured data can be obtained from the object storage device and fed back to the user device.

[0069] If the object storage device does not find the corresponding unstructured data locally, it means that the unstructured data corresponding to the target vector data is migrated to the target object storage device; therefore, the target vector data can be sent to the target object storage device.

[0070] The target object storage device can use the target vector data as an index to search for the corresponding unstructured data locally; obtain the corresponding unstructured data from the target object storage device and feed it back to the user device.

[0071] In one embodiment, when there are multiple target object storage devices, sending the target vector data to the target object storage device includes:

[0072] Obtain the unstructured data generation time range associated with each target object storage device; obtain the generation time of the target vector data; obtain the unstructured data generation time range within which the generation time of the target vector data falls, and determine the target object storage device associated with the determined unstructured data generation time range; send the target vector data to the determined target object storage device.

[0073] During each migration, the unstructured data to be migrated in the object storage device is migrated to a target object storage device (referred to as target object storage device A). After multiple consecutive migrations, the available storage space of the target object storage device A becomes less and less.

[0074] During a migration, if the available storage space of the target object storage device A does not meet the requirements, the unstructured data to be migrated in the object storage device can be migrated to another target object storage device (called target object storage device B). After multiple consecutive migrations, the available storage space of the target object storage device B becomes less and less.

[0075] During a migration, if the available storage space of the target object storage device B does not meet the requirements, the unstructured data to be migrated in the object storage device can be migrated to another target object storage device (referred to as the target object storage device C).

[0076] In the above manner, the unstructured data to be migrated in the object storage device is migrated to different target object storage devices in sequence.

[0077] For each target object storage device, the generation time range of the unstructured data associated with the target object storage device can be formed according to the earliest generation time and the latest generation time of the unstructured data stored in the target object storage device. Since the unstructured data stored in each target object storage device is stored sequentially in the above manner, the generation time range of the unstructured data associated with each target object storage device is different.

[0078] The generation time of the unstructured data is consistent with the generation time of the vector data. Therefore, after obtaining the target vector data, it is possible to determine in which target storage device the unstructured data corresponding to the target vector data is stored based on the generation time of the target vector data. Specifically, it is possible to determine the unstructured data generation time range within which the generation time of the target vector data falls, and the target object storage device associated with the unstructured data generation time range stores the unstructured data corresponding to the target vector data. The target vector data can be sent to the target object storage device.

[0079] In one embodiment, the unstructured data set is divided into a first unstructured data set and a second unstructured data set according to the generation time and the number of searches of the unstructured data, including:

[0080] Determine in an unstructured data set the unstructured data whose time difference between generation time and backup time is lower than a time difference threshold to form a candidate unstructured data set; sort the unstructured data of the candidate unstructured data set according to the number of searches of the unstructured data to obtain an unstructured data sequence; determine in the unstructured data sequence the unstructured data with a low number of searches to form a first unstructured data set; and obtain a second unstructured data set based on the unstructured data in the unstructured data set except the first unstructured data set.

[0081] In this embodiment, after obtaining the generation time of the unstructured data and the unstructured data set of the object storage device, the time difference between the generation time and the backup time of each unstructured data in the unstructured data set can be determined; based on the unstructured data whose time difference is lower than the time difference threshold, a candidate unstructured data set is formed.

[0082] According to the number of times the unstructured data is searched, the unstructured data in the candidate unstructured data set can be sorted to obtain an unstructured data sequence. Then, unstructured data with a low number of searches is determined in the unstructured data sequence to form a first unstructured data set; and a second unstructured data set is obtained based on the unstructured data in the unstructured data set except the first unstructured data set.

[0083] In one embodiment, determining unstructured data with a low search count in an unstructured data sequence includes:

[0084] If the unstructured data is sorted from high to low according to the number of search times, the unstructured data sorted later in the unstructured data sequence is determined to be the unstructured data with a low number of search times; if the unstructured data is sorted from low to high according to the number of search times, the unstructured data sorted earlier in the unstructured data sequence is determined to be the unstructured data with a low number of search times.

[0085] If the unstructured data in the candidate unstructured data set are sorted in descending order of the number of search times, the unstructured data ranked last N in the unstructured data sequence can be determined as unstructured data with a low number of search times, where N is an integer greater than or equal to 1.

[0086] If the unstructured data in the candidate unstructured data set are sorted in order of the number of search times from low to high, the first N unstructured data in the unstructured data sequence can be determined as unstructured data with a low number of search times.

[0087] In order to better understand the above method, an application example of the data processing method of the present application is described in detail below.

[0088] Reference Figure 4 , when the set backup time arrives, it can be determined to perform the backup; otherwise, it will not be performed. Then, the status of the vector database and the object storage device can be checked to determine whether the status is normal. If the status is abnormal, it indicates that the vector database and the object storage device are not suitable for backup.

[0089] If the status is normal, the characterization vector database and the object storage device are suitable for backup. At this time, the vector database can be backed up, and the vector data set of the vector database can be backed up to the backup vector database. Determine whether the available storage space of the object storage device meets the requirements. If the requirements are met, no migration is performed, the object storage device is backed up, and the unstructured data set of the object storage device is backed up to the backup object storage device. If the available storage space of the object storage device does not meet the requirements, the unstructured data set can be divided to obtain a first unstructured data set and a second unstructured data set; the unstructured data in the first unstructured data set has a low number of search times and the generation time is far from the backup time, and the unstructured data in the second unstructured data set has a high number of search times or the generation time is close to the backup time. Migrate the first unstructured data set from the object storage device to the target object storage device, and back up the second unstructured data set to the backup object storage device.

[0090] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0091] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0092] In one embodiment, Figure 5 As shown, a data processing device is provided, comprising:

[0093] The data acquisition module 501 is used to acquire the vector data set of the vector database and the unstructured data set of the object storage device when the set backup time arrives; the vector data is used to search for unstructured data;

[0094] A vector data backup module 502, used to back up the vector data set to a backup vector database;

[0095] The unstructured data partitioning module 503 is used to partition the unstructured data set into a first unstructured data set and a second unstructured data set according to the generation time and search times of the unstructured data if the available storage space of the object storage device does not meet the requirements; the unstructured data in the first unstructured data set has a low search time and the generation time is far from the backup time, and the unstructured data in the second unstructured data set has a high search time or the generation time is close to the backup time;

[0096] The unstructured data processing module 504 is configured to migrate the first unstructured data set from the object storage device to a target object storage device, and back up the second unstructured data set to a backup object storage device.

[0097] In one embodiment, the device further comprises a data feedback processing module, configured to:

[0098] In response to a data search request from a user device, target vector data is determined in the vector database and the target vector data is sent to the object storage device; if the object storage device finds the corresponding unstructured data locally using the target vector data as an index, the corresponding unstructured data is fed back to the user device; if the object storage device does not find the corresponding unstructured data locally using the target vector data as an index, the target vector data is sent to the target object storage device; when the target object storage device finds the corresponding unstructured data locally using the target vector data as an index, the corresponding unstructured data is fed back to the user device.

[0099] In one embodiment, when there are multiple target object storage devices, the data feedback processing module is further configured to:

[0100] Obtain the unstructured data generation time range associated with each of the target object storage devices; obtain the generation time of the target vector data; obtain the unstructured data generation time range within which the generation time of the target vector data falls, and determine the target object storage device associated with the determined unstructured data generation time range; send the target vector data to the determined target object storage device.

[0101] In one embodiment, the unstructured data partitioning module 503 is further used to:

[0102] Determine in the unstructured data set the unstructured data whose time difference between the generation time and the backup time is lower than a time difference threshold to form a candidate unstructured data set; sort the unstructured data of the candidate unstructured data set according to the number of searches of the unstructured data to obtain an unstructured data sequence; determine in the unstructured data sequence the unstructured data with a low number of searches to form a first unstructured data set; and obtain a second unstructured data set based on the unstructured data in the unstructured data set except the first unstructured data set.

[0103] In one embodiment, the unstructured data partitioning module 503 is further used to:

[0104] If the unstructured data is sorted from high to low according to the number of search times, the unstructured data sorted later in the unstructured data sequence is determined to be the unstructured data with a low number of search times; if the unstructured data is sorted from low to high according to the number of search times, the unstructured data sorted earlier in the unstructured data sequence is determined to be the unstructured data with a low number of search times.

[0105] In one embodiment, the apparatus further includes an available storage space determination module, which is used to: obtain the available storage space of the object storage device; if the available storage space is lower than a storage space threshold, determine that the available storage space of the object storage device does not meet the requirement.

[0106] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0107] In an exemplary embodiment, a computer device is provided, the internal structure of which can be shown as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the above method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0108] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0109] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned various method embodiments when executing the computer program.

[0110] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0111] In one embodiment, a computer program product is provided, on which a computer program is stored, and the computer program is used by a processor to execute the steps in the above-mentioned various method embodiments.

[0112] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0113] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0114] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0115] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: When the set backup time arrives, the vector data set of the vector database and the unstructured data set of the object storage device are obtained; the vector data is used to search for unstructured data; Backing up the vector data set to a backup vector database; If the available storage space of the object storage device does not meet the requirement, the unstructured data set is divided into a first unstructured data set and a second unstructured data set according to the generation time and search times of the unstructured data; the unstructured data in the first unstructured data set has a low search time and the generation time is far from the backup time, and the unstructured data in the second unstructured data set has a high search time or the generation time is close to the backup time; The first unstructured data set is migrated from the object storage device to a target object storage device, and the second unstructured data set is backed up to a backup object storage device.

2. The method according to claim 1, characterized in that The method further comprises: In response to a data search request from a user device, determining target vector data in the vector database and sending the target vector data to the object storage device; If the object storage device finds the corresponding unstructured data locally using the target vector data as an index, the corresponding unstructured data is fed back to the user device; If the object storage device fails to find the corresponding unstructured data locally using the target vector data as an index, sending the target vector data to the target object storage device; When the target object storage device finds the corresponding unstructured data locally using the target vector data as an index, the corresponding unstructured data is fed back to the user device.

3. The method according to claim 2, characterized in that In the case that there are multiple target object storage devices, sending the target vector data to the target object storage device includes: Obtaining a time range for generating unstructured data associated with each of the target object storage devices; Obtaining the generation time of the target vector data; Acquire the generation time range of the target vector data within which the generation time of the target vector data falls, and determine the target object storage device associated with the determined generation time range of the target vector data; The target vector data is sent to a determined target object storage device.

4. The method according to claim 1, characterized in that: According to the generation time and the number of searches of the unstructured data, the unstructured data set is divided into a first unstructured data set and a second unstructured data set, including: Determine, in the unstructured data set, unstructured data whose time difference between the generation time and the backup time is less than a time difference threshold, to form a candidate unstructured data set; Sorting the unstructured data of the candidate unstructured data set according to the number of searches for the unstructured data to obtain an unstructured data sequence; Determining unstructured data with a low search frequency in the unstructured data sequence to form a first unstructured data set; A second unstructured data set is obtained according to the unstructured data in the unstructured data set except the first unstructured data set.

5. The method according to claim 4, characterized in that Determining unstructured data with a low search frequency in the unstructured data sequence includes: If the unstructured data is sorted from high to low according to the number of search times, the unstructured data sorted later in the unstructured data sequence is determined as the unstructured data with a low number of search times; If the unstructured data is sorted from low to high according to the number of search times, the unstructured data sorted first in the unstructured data sequence is determined as the unstructured data with a low number of search times.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Obtaining available storage space of the object storage device; If the available storage space is lower than the storage space threshold, it is determined that the available storage space of the object storage device does not meet the requirement.

7. A data processing device, characterized in that: The device comprises: The data acquisition module is used to acquire the vector data set of the vector database and the unstructured data set of the object storage device when the set backup time arrives; the vector data is used to search for unstructured data; A vector data backup module, used for backing up the vector data set to a backup vector database; An unstructured data partitioning module is used to, if the available storage space of the object storage device does not meet the requirement, divide the unstructured data set into a first unstructured data set and a second unstructured data set according to the generation time and search times of the unstructured data; the unstructured data in the first unstructured data set has a low search time and the generation time is far from the backup time, and the unstructured data in the second unstructured data set has a high search time or the generation time is close to the backup time; The unstructured data processing module is used to migrate the first unstructured data set from the object storage device to a target object storage device, and back up the second unstructured data set to a backup object storage device.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images