Data processing method, device, electronic device and computer readable medium

Abstract

The embodiments of the present disclosure disclose a data processing method, device, electronic device and computer-readable medium. A specific implementation of the data processing method includes: using multiple data elimination threads to detect whether the memory usage of the memory exceeds a predetermined threshold, wherein the memory is divided into multiple memory partitions to store data according to preset rules; in response to exceeding the predetermined threshold, obtaining a target memory partition with the largest memory usage among the multiple memory partitions; using multiple data elimination threads to perform data elimination processing on the first data linked list in the target memory partition until the memory usage of the memory does not exceed the predetermined threshold. This implementation can improve the efficiency of data elimination processing in the memory and improve the stability of system operation.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a data processing method, device, electronic device, and computer-readable medium. Background Art

[0002] At present, the data elimination linked list corresponding to the memory data processing method can be uniquely determined. Among them, the article data in the memory can be protected by means of read-write locks. Here, while serving the upstream query calculation request, it is also necessary to perform the elimination operation of the system's internal data. Among them, the upstream query calculation and the elimination of internal data are multi-threaded concurrent operations. Therefore, when the upstream query calculation service load pressure is high, the activity of internal data elimination will greatly affect the processing delay of the upstream request and the system central processing unit (CPU) load, resulting in periodic oscillations in the upstream calculation delay and CPU load and low stability of system operation. Summary of the invention

[0003] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0004] Some embodiments of the present disclosure propose data processing methods, devices, equipment, and computer-readable media to solve the technical problems mentioned in the above background technology section.

[0005] In a first aspect, some embodiments of the present disclosure provide a data processing method, which includes: using multiple data elimination threads to detect whether a memory usage rate of a memory exceeds a predetermined threshold, wherein the above-mentioned memory is divided into multiple memory partitions to store data according to preset rules; in response to exceeding the above-mentioned predetermined threshold, obtaining a target memory partition with the largest memory usage rate among the multiple above-mentioned memory partitions; using the multiple above-mentioned data elimination threads to perform data elimination processing on a first data linked list in the above-mentioned target memory partition until the memory usage rate of the above-mentioned memory does not exceed the above-mentioned predetermined threshold.

[0006] In a second aspect, some embodiments of the present disclosure provide a data processing device, comprising: a detection unit, configured to use multiple data elimination threads to detect whether a memory usage rate of a memory exceeds a predetermined threshold, wherein the above-mentioned memory is divided into multiple memory partitions to store data according to preset rules; an acquisition unit, configured to, in response to exceeding the above-mentioned predetermined threshold, acquire a target memory partition with the largest memory usage rate among the multiple above-mentioned memory partitions; a processing unit, configured to use the multiple above-mentioned data elimination threads to perform data elimination processing on the first data linked list in the above-mentioned target memory partition until the memory usage rate of the above-mentioned memory does not exceed the above-mentioned predetermined threshold.

[0007] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any method as described in the first aspect.

[0008] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, any method in the first aspect is implemented.

[0009] Multiple embodiments of the above-mentioned various embodiments of the present disclosure have the following beneficial effects: First, multiple data elimination threads are used to detect whether the memory usage rate of the memory exceeds a predetermined threshold value to determine the load level of the current memory. In this way, it can be determined whether to perform data elimination processing based on the load level. Among them, the above-mentioned memory is divided into multiple memory partitions to store data according to preset rules. Then, in response to exceeding the above-mentioned predetermined threshold value, the target memory partition with the largest memory usage rate among the multiple above-mentioned memory partitions is obtained. Finally, the first data linked list in the above-mentioned target memory partition is subjected to data elimination processing by using multiple above-mentioned data elimination threads until the memory usage rate of the above-mentioned memory does not exceed the above-mentioned predetermined threshold value to reduce the load of the current memory. Here, the above-mentioned data processing method can improve the efficiency of data processing in the memory by partitioning the memory and processing the corresponding memory partition data elimination, reduce the occurrence of the problem of increased pressure cycle of the central processing unit, and improve the stability of system operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0011] Figure 1-2 is a schematic diagram of an application scenario diagram of the data processing method of some embodiments of the present disclosure;

[0012] Figure 3 is a flow chart of some embodiments of the data processing method according to the present disclosure;

[0013] Figure 4 is a schematic diagram of some application scenarios of determining a target memory partition according to some embodiments of the present disclosure;

[0014] Figure 5 is a schematic diagram of some application scenarios of data elimination processing according to some embodiments of the present disclosure;

[0015] Figure 6 is a schematic diagram of some application scenarios of a time series article calculation cache system according to some embodiments of the present disclosure;

[0016] Figure 7 are flow charts of other embodiments of the data processing method according to the present disclosure;

[0017] Figure 8 is a schematic diagram of the structure of some embodiments of the data processing device according to the present disclosure;

[0018] Fig. 9 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0021] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0025] Figure 1-Figure 2 It is a schematic diagram of an application scenario diagram of the data processing method of some embodiments of the present disclosure.

[0026] like Figure 1-Figure 2 As shown, the electronic device 101 can use multiple data elimination threads to detect whether the memory usage 103 of the memory 102 exceeds a predetermined threshold. Among them, the above-mentioned memory 102 is divided into multiple memory partitions to store data according to preset rules. In this application scenario, the above-mentioned memory 102 may include: memory partition 1021, memory partition 1022, memory partition 1023, and memory partition 1024. The above-mentioned memory usage 103 may be: 80%. The above-mentioned predetermined threshold may be 0.6. Then, in response to exceeding the above-mentioned predetermined threshold, the target memory partition 1024 with the largest memory usage among the multiple above-mentioned memory partitions is obtained. In this application scenario, the memory usage corresponding to the memory partition 1021 may be: 20%. The memory usage corresponding to the memory partition 1022 may be: 10%. The memory usage corresponding to the memory partition 1023 may be: 20%. The memory usage corresponding to the memory partition 1024 may be: 30%. Since the memory usage rate 103 is greater than the predetermined threshold, the target memory partition 1023 with the largest memory usage rate is determined from the multiple memory partitions, that is, the memory partition with a memory usage rate of 30%. Finally, the first data linked list 104 in the target memory partition 1024 is subjected to data elimination processing by using the multiple data elimination threads until the memory usage rate 103 of the memory 102 does not exceed the predetermined threshold. In this application scenario, the first data linked list 104 includes: data 1041, data 1042, data 1043, data 1044, and data 1045.

[0027] It should be noted that the electronic device 101 can be hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device. When the electronic device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0028] It should be understood that Figure 1The number of electronic devices in the embodiment is only for illustration. Any number of electronic devices may be provided according to implementation requirements.

[0029] Continue to refer Figure 3 , shows a process 300 of some embodiments of the data processing method according to the present disclosure. The data processing method comprises the following steps:

[0030] Step 301: Use multiple data elimination threads to detect whether the memory usage rate of the memory exceeds a predetermined threshold.

[0031] In some embodiments, the execution subject of the data processing method (for example Figure 1 The electronic device 101 shown can use multiple data elimination threads to detect whether the memory usage of the memory exceeds a predetermined threshold. Among them, the above-mentioned memory is divided into multiple memory partitions to store data according to preset rules. The above-mentioned memory usage rate can be a memory water level, which refers to the CPU occupancy in the current instance task. The above-mentioned data elimination thread can be used to perform data elimination operations on the linked list data in the memory. The above-mentioned data elimination thread can be multithreading, and can also refer to the technology of implementing concurrent execution of multiple threads from software or hardware. Computers with multithreading capabilities can execute more than one thread at the same time due to hardware support, thereby improving overall processing performance. Systems with this capability include symmetric multiprocessors, multi-core processors, and chip-level multiprocessing or simultaneous multithreading processors. In a program, these independently running program fragments can be called "threads".

[0032] As an example, the above memory can be divided into 16 memory partitions.

[0033] In some optional implementations of some embodiments, the execution entity may utilize the plurality of data elimination threads to periodically detect whether the memory usage of the memory exceeds the predetermined threshold value according to a first predetermined time period.

[0034] Step 302: in response to exceeding the predetermined threshold, obtaining a target memory partition with the highest memory usage rate among the plurality of memory partitions.

[0035] In some embodiments, in response to exceeding the predetermined threshold, the execution entity may obtain a target memory partition having the highest memory usage rate among the plurality of memory partitions.

[0036] As an example, in response to exceeding the predetermined threshold, the execution subject may use a related interface to obtain the memory usage of each of the multiple memory partitions, and then determine the memory partition with the highest memory usage among the multiple memory partitions as the target memory partition.

[0037] As an example, Figure 4 As shown, the memory is divided into a first memory partition 401, a second memory partition 402, a third memory partition 403, a fourth memory partition 404 and a fifth memory partition 405. Among them, the data elimination linked list in the first memory partition 401 stores 45% of the above data volume. The data elimination linked list in the second memory partition 402 stores 80% of the above data volume. The data elimination linked list in the third memory partition 403 stores 40% of the above data volume. The data elimination linked list in the fourth memory partition 404 stores 15% of the above data volume. The data elimination linked list in the fifth memory partition 405 stores 87% of the above data volume. Since the data elimination linked list in the fifth memory partition 405 stores the largest amount of data volume, the fifth memory partition 405 is determined as the target memory partition.

[0038] Step 303: Use the plurality of data elimination threads to perform data elimination processing on the first data linked list in the target memory partition until the memory usage rate of the memory does not exceed the predetermined threshold.

[0039] In some embodiments, the execution subject may utilize the plurality of data elimination threads to perform data elimination processing on the first data linked list in the target memory partition until the memory usage rate of the memory does not exceed the predetermined threshold. The first data linked list may be a least recently used (LRU) data linked list (LRU List). The LRU List may be used to maintain a cache area in the memory, and the data is processed according to the LRU algorithm.

[0040] In some optional implementations of some embodiments, performing data elimination processing on the first data linked list in the target memory partition by using the plurality of data elimination threads until the memory usage rate of the memory does not exceed the predetermined threshold may include the following steps:

[0041] The first step is to process the data in the first data linked list through the following processing steps:

[0042] The first sub-step is to determine whether each of the plurality of data elimination threads can obtain the lock information of the target position data in the first data linked list in the target memory partition. As an example, the lock information may be the read-write lock information of the target position data in the target memory partition.

[0043] The second sub-step is, in response to determining that the data elimination thread acquires the lock information of the target location data, using the data elimination thread to perform data elimination processing on the target location data.

[0044] The third sub-step is, in response to determining that the data elimination thread has not obtained the lock information of the target position data, determining whether the data elimination thread can obtain the lock information of the next position data of the target position in the first data linked list in the target memory partition.

[0045] Here, in response to determining that the data elimination thread has not obtained the lock information of the target position data, it indicates that other threads are processing the data. For the processing of the current target position data, there is a thread processing conflict problem, so it is necessary to determine whether the data elimination thread can obtain the lock information of the next position data of the target position in the first data linked list in the target memory partition to process the next position data of the target position. This implementation method can greatly improve the data elimination processing speed.

[0046] As an example, Figure 5 As shown, in response to detecting that the memory usage rate of the above memory exceeds the above predetermined threshold, the target data elimination thread 501 in the multiple data elimination threads determines whether it can obtain the lock information 5036 of the target position data 5035 in the first data linked list 503 in the above target memory partition 502. Then, in response to determining that the above target data elimination thread 501 has not obtained the lock information 5036 of the above target position data, the lock information 5037 of the next position data 5034 of the target position in the first data linked list 503 in the above target memory partition 502 is obtained. Among them, the above first data linked list 503 includes: data 5031, data 5032, data 5033, data 5034 and data 5035.

[0047] The fourth sub-step is, in response to determining that the data elimination thread obtains the lock information of the next position data of the target position, using the data elimination thread to perform data elimination processing on the next position data of the target position.

[0048] In the second step, in response to determining that the data elimination thread cannot obtain the lock information of the next position data of the target position, the next position data of the target position is determined as the target position data and the processing steps are executed.

[0049] In some optional implementations of some embodiments, the data of each memory partition in the above memory is stored through the following steps:

[0050] The first step is to convert the identification information of each data in the data set into a corresponding hash value to obtain a hash value set. The first identification information may be the Uid (User Identification) of each data.

[0051] In the second step, based on the above hash value set, each data in the above data set is stored in a corresponding memory partition in the above multiple memory partitions. As an example, the above execution subject can evenly store each data in a corresponding memory partition in the above multiple memory partitions according to the size of the first digit in all characters in the hash value.

[0052] Optionally, based on the hash value set, storing each data in the data set into a corresponding memory partition in the multiple memory partitions may include the following steps:

[0053] The first step is to determine the corresponding memory partition based on the target character in the hash value corresponding to the above data. As an example, the memory partition corresponding to the data can be determined based on the hexadecimal size of the first three characters of all the characters in the hash value.

[0054] The second step is to store the above data into the corresponding memory partition.

[0055] Optionally, the target character in the hash value corresponding to the data is a character at a target continuous position in the hash value.

[0056] Multiple embodiments of the above-mentioned various embodiments of the present disclosure have the following beneficial effects: First, multiple data elimination threads are used to detect whether the memory usage rate of the memory exceeds a predetermined threshold value to determine the load level of the current memory. In this way, it can be determined whether to perform data elimination processing based on the load level. Among them, the above-mentioned memory is divided into multiple memory partitions to store data according to preset rules. Then, in response to exceeding the above-mentioned predetermined threshold value, the target memory partition with the largest memory usage rate among the multiple above-mentioned memory partitions is obtained. Finally, the first data linked list in the above-mentioned target memory partition is subjected to data elimination processing by using multiple above-mentioned data elimination threads until the memory usage rate of the above-mentioned memory does not exceed the above-mentioned predetermined threshold value to reduce the load of the current memory. Here, the above-mentioned data processing method can improve the efficiency of data processing in the memory by partitioning the memory and processing the corresponding memory partition data elimination, reduce the occurrence of the problem of increased pressure cycle of the central processing unit, and improve the stability of system operation.

[0057] Figure 6 It is a schematic diagram of some application scenarios of the time series article calculation cache system according to some embodiments of the present disclosure.

[0058] Real-time calculation of upstream data and organization and persistence of memory data by background threads can be performed on the target system. The above target system can be a time series article calculation cache system (IPS, Instance Profile Service). The overall architecture is divided into three layers, including: user layer, IPS system layer, and data persistence layer. This architecture is based on the Remote Procedure Call (RPC) framework and service discovery mechanism, and uses the underlying consistent storage to persist data, so as to realize the recording and query of user feature vectors at the cache level.

[0059] like Figure 6 As shown, the overall architecture includes: user layer 601, IPS system layer 602, and data persistence layer 603. Among them, the user layer 601 can perform: article acquisition, model training related to article data, and data prediction related to article data. The user layer 601 and the IPS system layer 602 implement data interaction by calling the (RPC, Remote Procedure Call) framework. Among them, the hash consistency strategy is adopted in the (RPC, Remote Procedure Call) framework. The IPS system layer 602 can implement storage and update of data with the data persistence layer 603. In addition, the framework uses consul 606 to implement service registration and service discovery between the user layer 601 and the IPS system layer 602.

[0060] Continue to refer Figure 7 , shows a process 700 of another embodiment of the data processing method according to the present disclosure. The data processing method comprises the following steps:

[0061] Step 701: Use multiple data elimination threads to detect whether the memory usage rate of the memory exceeds a predetermined threshold.

[0062] Step 702: in response to exceeding the predetermined threshold, obtaining a target memory partition with the highest memory usage rate among the plurality of memory partitions.

[0063] Step 703: Use the plurality of data elimination threads to perform data elimination processing on the first data linked list in the target memory partition until the memory usage rate of the memory does not exceed the predetermined threshold.

[0064] In some embodiments, the specific implementation of steps 701-703 and the technical effects brought about can be referred to Figure 3 The corresponding steps 301-303 in the embodiments are not described in detail here.

[0065] Step 704, determining whether there is data to be eliminated in the first data linked list.

[0066] In some embodiments, an execution entity (e.g. Figure 1 The electronic device 101) shown can determine whether there is data to be eliminated in the first data linked list by querying the first data linked list.

[0067] In some optional implementations of some embodiments, the execution entity may periodically determine whether the to-be-eliminated data exists in the first data linked list according to a second predetermined time period.

[0068] Here, by periodically determining whether the data to be eliminated exists in the first data linked list according to the second predetermined time period, the data to be eliminated that may exist in the first data linked list can be efficiently processed in a timely manner, thereby improving data processing efficiency.

[0069] Step 705: In response to determining that the data exists, search whether there is data corresponding to the data to be eliminated in the downstream storage of the memory.

[0070] In some embodiments, in response to determining that there is, the execution subject may search whether there is data corresponding to the data to be eliminated in the memory downstream storage. The data corresponding to the data to be eliminated may be data of the same type as the data to be eliminated. The data corresponding to the data to be eliminated may also be data that is very similar to the data to be eliminated.

[0071] Step 706: In response to the existence of the corresponding data, determine whether the corresponding data and the to-be-eliminated data are the same data.

[0072] In some embodiments, in response to the existence of the corresponding data, the execution entity may determine whether the corresponding data and the to-be-eliminated data are the same data.

[0073] As an example, in response to the existence of the corresponding data, the execution subject may determine whether the corresponding data and the to-be-eliminated data are the same data by analyzing the data type and data structure.

[0074] Step 707, in response to the corresponding data being different from the data to be eliminated or the data corresponding to the data to be eliminated not existing, the data to be eliminated is stored in a second data linked list in the corresponding memory partition, and multiple data persistence threads are used to persist the data to be eliminated in the second data linked list.

[0075] In some embodiments, in response to the corresponding data being different from the to-be-eliminated data or the absence of data corresponding to the to-be-eliminated data, the execution subject may store the to-be-eliminated data into a second data linked list in the corresponding memory partition, and use multiple data persistence threads to perform data persistence on the to-be-eliminated data in the second data linked list. Each of the multiple memory partitions corresponds to multiple data persistence threads.

[0076] from Figure 7 It can be seen that Figure 3 Compared with the description of some corresponding embodiments, Figure 7 The data processing method flow 700 in some corresponding embodiments further highlights the specific steps of persisting the data to be eliminated in the first data linked list. Therefore, the schemes described in these embodiments can perform persistent processing on the data to be eliminated in the memory in a timely manner, avoid excessive memory load, and thus improve the operation of the system.

[0077] Continue to refer Figure 8 As an implementation of the above-mentioned methods in the above-mentioned figures, the present disclosure provides some embodiments of a data processing device, and these device embodiments are Figure 3 Corresponding to the above method embodiments, the device can be specifically applied to various electronic devices.

[0078] like Figure 8 As shown, the data processing device 800 of some embodiments includes: a detection unit 801, an acquisition unit 802 and a processing unit 803. The detection unit 801 is configured to use multiple data elimination threads to detect whether the memory usage of the memory exceeds a predetermined threshold, wherein the above-mentioned memory is divided into multiple memory partitions to store data according to preset rules. The acquisition unit 801 is configured to, in response to exceeding the above-mentioned predetermined threshold, acquire a target memory partition with the largest memory usage among the multiple above-mentioned memory partitions. The processing unit 803 is configured to use the multiple above-mentioned data elimination threads to perform data elimination processing on the first data linked list in the above-mentioned target memory partition until the memory usage of the above-mentioned memory does not exceed the above-mentioned predetermined threshold.

[0079] In some optional implementations of some embodiments, the data processing device 800 further includes: a determination unit, a search unit, a judgment unit, and a storage persistence unit (not shown in the figure). The determination unit is configured to: determine whether there is data to be eliminated in the above-mentioned first data linked list. The search unit is configured to: in response to determining that there is, search whether there is data corresponding to the above-mentioned data to be eliminated in the downstream storage of the memory. The judgment unit is configured to: in response to the existence of the above-mentioned corresponding data, judge whether the above-mentioned corresponding data and the above-mentioned data to be eliminated are the same data. The storage persistence unit is configured to: in response to the above-mentioned corresponding data and the above-mentioned data to be eliminated being different data or the data corresponding to the above-mentioned data to be eliminated not existing, store the above-mentioned data to be eliminated in the second data linked list in the corresponding memory partition, and use multiple data persistence threads to persist the data to be eliminated in the above-mentioned second data linked list, wherein each of the multiple memory partitions corresponds to multiple data persistence threads.

[0080] In some optional implementations of some embodiments, the processing unit 803 in the data processing device 800 can be further configured to: process the data in the above-mentioned first data linked list through the following processing steps: determine whether each data elimination thread among the multiple data elimination threads can obtain the lock information of the target position data in the first data linked list in the above-mentioned target memory partition; in response to determining that the above-mentioned data elimination thread obtains the lock information of the above-mentioned target position data, use the above-mentioned data elimination thread to perform data elimination processing on the above-mentioned target position data; in response to determining that the above-mentioned data elimination thread does not obtain the lock information of the above-mentioned target position data, determine whether the above-mentioned data elimination thread can obtain the lock information of the next position data of the target position in the first data linked list in the above-mentioned target memory partition; in response to determining that the above-mentioned data elimination thread obtains the lock information of the next position data of the above-mentioned target position, use the above-mentioned data elimination thread to perform data elimination processing on the next position data of the above-mentioned target position; in response to determining that the above-mentioned data elimination thread cannot obtain the lock information of the next position data of the above-mentioned target position, determine the next position data of the above-mentioned target position as the above-mentioned target position data and execute the above-mentioned processing steps.

[0081] In some optional implementations of some embodiments, the detection unit 801 in the data processing device 800 may be further configured to: periodically detect whether the memory usage of the above-mentioned memory exceeds the above-mentioned predetermined threshold value using the plurality of the above-mentioned data elimination threads according to a first predetermined time period.

[0082] In some optional implementations of some embodiments, the determination unit in the data processing device 800 is further configured to: periodically determine whether the above-mentioned data to be eliminated exists in the above-mentioned first data linked list according to a second predetermined time length.

[0083] It is understood that the units described in the device 800 are similar to those described in the reference Figure 3 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 800 and the units included therein, and will not be described in detail here.

[0084] Reference below Fig. 9 , which shows a structural schematic diagram of an electronic device 900 suitable for implementing some embodiments of the present disclosure. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0085] like Fig. 9 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0086] Typically, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Fig. 9 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0087] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 909, or installed from the storage device 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0088] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0089] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0090] The computer-readable medium may be included in the device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: uses multiple data elimination threads to detect whether the memory usage of the memory exceeds a predetermined threshold, wherein the memory is divided into multiple memory partitions to store data according to preset rules; in response to exceeding the predetermined threshold, obtains the target memory partition with the largest memory usage among the multiple memory partitions; uses multiple data elimination threads to perform data elimination processing on the first data linked list in the target memory partition until the memory usage of the memory does not exceed the predetermined threshold.

[0091] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0092] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0093] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a detection unit, an acquisition unit, and a processing unit. The names of these units do not constitute a limitation on the units themselves in some cases. For example, the detection unit may also be described as "a unit that uses multiple data elimination threads to detect whether the memory usage of the memory exceeds a predetermined threshold."

[0094] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0095] According to one or more embodiments of the present disclosure, a data processing method is provided, comprising: using multiple data elimination threads to detect whether a memory usage rate of a memory exceeds a predetermined threshold, wherein the memory is divided into multiple memory partitions to store data according to preset rules; in response to exceeding the predetermined threshold, obtaining a target memory partition with the largest memory usage rate among the multiple memory partitions; and using the multiple data elimination threads to perform data elimination processing on a first data linked list in the target memory partition until the memory usage rate of the memory does not exceed the predetermined threshold.

[0096] According to one or more embodiments of the present disclosure, the method further includes: determining whether there is data to be eliminated in the first data linked list; in response to determining that there is data to be eliminated, searching whether there is data corresponding to the data to be eliminated in the downstream storage of the memory; in response to the existence of the corresponding data, determining whether the corresponding data is the same as the data to be eliminated; in response to the corresponding data being different from the data to be eliminated or the data corresponding to the data to be eliminated not existing, storing the data to be eliminated in the second data linked list in the corresponding memory partition, and performing data persistence on the data to be eliminated in the second data linked list using multiple data persistence threads, wherein each of the multiple memory partitions corresponds to multiple data persistence threads.

[0097] According to one or more embodiments of the present disclosure, the method of performing data elimination processing on the first data linked list in the target memory partition by using the plurality of data elimination threads until the memory usage rate of the memory does not exceed the predetermined threshold comprises: processing the data in the first data linked list by the following processing steps: determining whether each of the plurality of data elimination threads can obtain the lock information of the target position data in the first data linked list in the target memory partition; in response to determining that the data elimination thread obtains the lock information of the target position data, performing data elimination processing on the target position data by using the data elimination thread; in response to determining that the data elimination thread does not obtain the lock information of the target position data, determining whether the data elimination thread can obtain the lock information of the next position data of the target position in the first data linked list in the target memory partition; in response to determining that the data elimination thread obtains the lock information of the next position data of the target position, performing data elimination processing on the next position data of the target position by using the data elimination thread; in response to determining that the data elimination thread cannot obtain the lock information of the next position data of the target position, determining the next position data of the target position as the target position data and executing the above processing steps.

[0098] According to one or more embodiments of the present disclosure, the above-mentioned use of multiple data elimination threads to detect whether the memory usage rate of the memory exceeds the predetermined threshold includes: according to a first predetermined time length, using the multiple data elimination threads to periodically detect whether the memory usage rate of the above-mentioned memory exceeds the above-mentioned predetermined threshold.

[0099] According to one or more embodiments of the present disclosure, the determining of the data to be eliminated in the first data linked list includes: periodically determining whether the data to be eliminated exists in the first data linked list according to a second predetermined time period.

[0100] According to one or more embodiments of the present disclosure, the data of each memory partition in the above-mentioned memory is stored through the following steps: converting the identification information of each data in the data set into a corresponding hash value to obtain a hash value set; based on the above-mentioned hash value set, storing each data in the above-mentioned data set into a corresponding memory partition in the above-mentioned multiple memory partitions.

[0101] According to one or more embodiments of the present disclosure, based on the above-mentioned hash value set, each data in the above-mentioned data set is stored in a corresponding memory partition among the above-mentioned multiple memory partitions, including: determining the corresponding memory partition based on the target character in the hash value corresponding to the above-mentioned data; storing the above-mentioned data in the above-mentioned corresponding memory partition.

[0102] According to one or more embodiments of the present disclosure, there is provided a data processing device, comprising: a detection unit, configured to use multiple data elimination threads to detect whether a memory usage rate of a memory exceeds a predetermined threshold, wherein the memory is divided into multiple memory partitions to store data according to preset rules; an acquisition unit, configured to, in response to exceeding the predetermined threshold, acquire a target memory partition with the largest memory usage rate among the multiple memory partitions; and a processing unit, configured to use the multiple data elimination threads to perform data elimination processing on a first data linked list in the target memory partition until the memory usage rate of the memory does not exceed the predetermined threshold.

[0103] According to one or more embodiments of the present disclosure, the data processing device further includes: a determination unit, a search unit, a judgment unit and a storage and persistence unit (not shown in the figure). The determination unit is configured to: determine whether there is data to be eliminated in the above-mentioned first data linked list. The search unit is configured to: in response to determining that there is, search whether there is data corresponding to the above-mentioned data to be eliminated in the downstream storage of the memory. The judgment unit is configured to: in response to the existence of the above-mentioned corresponding data, judge whether the above-mentioned corresponding data and the above-mentioned data to be eliminated are the same data. The storage and persistence unit is configured to: in response to the above-mentioned corresponding data and the above-mentioned data to be eliminated being different data or the data corresponding to the above-mentioned data to be eliminated not existing, store the above-mentioned data to be eliminated in the second data linked list in the corresponding memory partition, and use multiple data persistence threads to persist the data to be eliminated in the above-mentioned second data linked list, wherein each of the multiple memory partitions corresponds to multiple data persistence threads.

[0104] According to one or more embodiments of the present disclosure, the processing unit in the data processing device can be further configured to: process the data in the above-mentioned first data linked list through the following processing steps: determine whether each data elimination thread among the multiple data elimination threads can obtain the lock information of the target position data in the first data linked list in the above-mentioned target memory partition; in response to determining that the above-mentioned data elimination thread obtains the lock information of the above-mentioned target position data, use the above-mentioned data elimination thread to perform data elimination processing on the above-mentioned target position data; in response to determining that the above-mentioned data elimination thread does not obtain the lock information of the above-mentioned target position data, determine whether the above-mentioned data elimination thread can obtain the lock information of the next position data of the target position in the first data linked list in the above-mentioned target memory partition; in response to determining that the above-mentioned data elimination thread obtains the lock information of the next position data of the above-mentioned target position, use the above-mentioned data elimination thread to perform data elimination processing on the next position data of the above-mentioned target position; in response to determining that the above-mentioned data elimination thread cannot obtain the lock information of the next position data of the above-mentioned target position, determine the next position data of the above-mentioned target position as the above-mentioned target position data and execute the above-mentioned processing steps.

[0105] According to one or more embodiments of the present disclosure, the detection unit in the data processing device may be further configured to: periodically detect whether the memory usage of the memory exceeds the predetermined threshold using the plurality of data elimination threads according to a first predetermined time period.

[0106] According to one or more embodiments of the present disclosure, the determination unit in the data processing device is further configured to: periodically determine whether the to-be-eliminated data exists in the first data linked list according to a second predetermined time period.

[0107] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A data processing method, comprising: Using a plurality of data elimination threads to detect whether a memory usage of a memory exceeds a predetermined threshold, wherein the memory is divided into a plurality of memory partitions to store data according to a preset rule; In response to exceeding the predetermined threshold, obtaining a target memory partition having the highest memory usage rate among the plurality of memory partitions; The data elimination process is performed on the first data linked list in the target memory partition by using the plurality of data elimination threads through the following processing steps until the memory usage rate of the memory does not exceed the predetermined threshold: Determine whether each of the plurality of data elimination threads can obtain lock information of target position data in the first data linked list in the target memory partition; In response to determining that the data elimination thread acquires the lock information of the target location data, using the data elimination thread to perform data elimination processing on the target location data; In response to determining that the data elimination thread has not obtained the lock information of the target location data, determining whether the data elimination thread can obtain the lock information of the next location data of the target location in the first data linked list in the target memory partition; In response to determining that the data elimination thread obtains the lock information of the next position data of the target position, using the data elimination thread to perform data elimination processing on the next position data of the target position; In response to determining that the data elimination thread cannot obtain the lock information of the next position data of the target position, the next position data of the target position is determined as the target position data and the processing step is performed.

2. The method according to claim 1, wherein: The method further comprises: Determine whether there is data to be eliminated in the first data linked list; In response to determining that the data exists, searching whether there is data corresponding to the data to be eliminated in the downstream storage of the memory; In response to the existence of the corresponding data, determining whether the corresponding data and the to-be-eliminated data are the same data; In response to the corresponding data being different from the data to be eliminated or the absence of data corresponding to the data to be eliminated, the data to be eliminated is stored in a second data linked list in the corresponding memory partition, and the data to be eliminated in the second data linked list is persisted using multiple data persistence threads, wherein each of the multiple memory partitions corresponds to multiple data persistence threads.

3. According to the method of claim 1, the step of using a plurality of data elimination threads to detect whether the memory usage rate of the memory exceeds a predetermined threshold comprises: According to the first predetermined time period, the plurality of data elimination threads are used to periodically detect whether the memory usage rate of the memory exceeds the predetermined threshold.

4. The method according to claim 2, wherein: The determining the data to be eliminated in the first data linked list includes: According to a second predetermined time period, it is periodically determined whether the to-be-eliminated data exists in the first data linked list.

5. The method according to claim 1, wherein: The data of each memory partition in the memory is stored by the following steps: Convert the identification information of each data in the data set into a corresponding hash value to obtain a hash value set; Based on the hash value set, each data in the data set is stored in a corresponding memory partition among the multiple memory partitions.

6. The method according to claim 5, wherein: The storing each data in the data set into a corresponding memory partition among the multiple memory partitions based on the hash value set includes: Determine a corresponding memory partition based on a target character in a hash value corresponding to the data; The data is stored in the corresponding memory partition.

7. A data processing device, comprising: a detection unit configured to detect whether a memory usage of a memory exceeds a predetermined threshold using a plurality of data elimination threads, wherein the memory is divided into a plurality of memory partitions to store data according to a preset rule; An acquiring unit is configured to acquire a target memory partition having the largest memory usage rate among the plurality of memory partitions in response to exceeding the predetermined threshold; The processing unit is configured to perform data elimination processing on the first data linked list in the target memory partition by using the plurality of data elimination threads through the following processing steps until the memory usage rate of the memory does not exceed the predetermined threshold: determining whether each of the plurality of data elimination threads can obtain the lock information of the target position data in the first data linked list in the target memory partition; in response to determining that the data elimination thread obtains the lock information of the target position data, using the data elimination thread to perform data elimination processing on the target position data; in response to determining that the data elimination thread does not obtain the lock information of the target position data, determining whether the data elimination thread can obtain the lock information of the next position data of the target position in the first data linked list in the target memory partition; in response to determining that the data elimination thread obtains the lock information of the next position data of the target position, using the data elimination thread to perform data elimination processing on the next position data of the target position; in response to determining that the data elimination thread cannot obtain the lock information of the next position data of the target position, determining the next position data of the target position as the target position data and executing the processing steps.

8. An electronic device comprising: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images