File receiving state monitoring method and related device

Abstract

The invention discloses a file receiving state monitoring method and a related device, and relates to the technical field of data processing, and the file receiving state monitoring method comprises the steps that after a first data file is received, file information of the first data file is determined, and the file information comprises a first file theme and a first file identifier; mapping the first file identifier by using hash functions of a first function number corresponding to the first file theme to obtain a first hash value of the first function number, the first function number being determined based on a first expected file total amount and a first bit number corresponding to the first file theme, the first bit number is a bit array size of a first bitmap corresponding to the first file theme, and the first bit number is determined based on a first expected misjudgment rate corresponding to the first file theme and a first expected total file amount; and setting a bit taking each first hash value as a bit index on the first bitmap as a first state value so as to update the first bitmap. The processing efficiency of state monitoring is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and related apparatus for monitoring the status of file reception. Background Technology

[0002] In modern information systems (such as bank record management systems), massive amounts of report files from different upstream systems and on different topics need to be received. To monitor whether any specific report files are missing, traditional solutions typically require storing all detailed data of received report files in a database, and then querying to confirm the existence of a specific file. Therefore, during the recording and retrieval process, the system needs to frequently interact with the database, which severely restricts processing efficiency. Summary of the Invention

[0003] In view of the above problems, this application provides a method and related apparatus for monitoring file reception status. The specific solution is as follows:

[0004] The first aspect of this application provides a method for monitoring file reception status, including:

[0005] Upon receiving the first data file, determine the file information of the first data file, which includes the first file subject and the first file identifier;

[0006] The first file identifier is mapped using a hash function corresponding to the first function number of the first file topic to obtain a first hash value of the first function number. The first function number is determined based on the first expected total number of files and the first bit number corresponding to the first file topic. The first bit number is the size of the bit array of the first bit diagram corresponding to the first file topic. The first bit number is determined based on the first expected false positive rate and the first expected total number of files corresponding to the first file topic.

[0007] The first bitmap is updated by setting the bits indexed by the first hash value to the first state value.

[0008] In one possible implementation, the file reception status monitoring method also includes:

[0009] In response to the arrival of the second data file reception status verification timing, the file information of the second data file is determined, the file information of the second data file includes the second file topic and the second file identifier;

[0010] The second file identifier is mapped using the hash function of the second function number corresponding to the second file topic to obtain the second hash value of the second function number. The second function number is determined based on the second expected total number of files and the second bit number corresponding to the second file topic. The second bit number is the size of the bit array of the second bitmap corresponding to the second file topic. The second bit number is determined based on the second expected false positive rate and the second expected total number of files corresponding to the second file topic.

[0011] On the second bitmap, it is determined whether each bit indexed by the second hash value is a first state value;

[0012] If not, then it is determined that the second data file is in an unreceived state;

[0013] If so, then the second data file is determined to be in a received state.

[0014] In one possible implementation, the file reception status monitoring method also includes:

[0015] Execute on any target file topic:

[0016] Based on the target file topic, determine the target expected total number of files and the target expected misclassification rate. The target file topic includes a first file topic and a second file topic.

[0017] The target number of bits is determined based on the total number of target expected files and the target expected false positive rate. The target number of bits is positively correlated with the total number of target expected files and negatively correlated with the target expected false positive rate.

[0018] The number of objective functions is determined based on the target number of bits and the target expected total number of files. The number of objective functions is inversely correlated with the target expected total number of files and positively correlated with the target number of bits.

[0019] Based on the number of objective functions, a corresponding sequence of objective functions is configured for the subject of the target file, wherein the sequence of objective functions includes the number of objective hash functions specified in the objective functions.

[0020] Based on the target number of bits, an initialized target bitmap is configured for the target file topic, and the bits at each bit index of the initialized target bitmap are second state values; the size of the bit array of the target bitmap is the target number of bits.

[0021] In one possible implementation, the total number of target expected files and the target expected false positive rate are determined based on the target file topic, including:

[0022] The total number of data files belonging to the subject of the target file is taken as the total number of the target expected files;

[0023] Based on the remaining memory ratio, the storage efficiency coefficient corresponding to the target file topic is determined, and the storage efficiency coefficient is inversely correlated with the remaining memory ratio.

[0024] Based on the importance of the target file topic, a data integrity coefficient corresponding to the target file topic is determined, and the importance of the target file topic is positively correlated with the data integrity coefficient;

[0025] When the data integrity coefficient is greater than the preset target threshold and greater than the storage efficiency coefficient, the target expected misjudgment rate is determined to be the first value within the preset numerical range;

[0026] When the storage efficiency coefficient is greater than the target threshold and also greater than the data integrity coefficient, the target expected false positive rate is determined to be the second value within the specified range.

[0027] When both the data integrity coefficient and the storage efficiency coefficient are not greater than the target threshold, or when both the data integrity coefficient and the storage efficiency coefficient are not less than the target threshold, the target expected misjudgment rate is determined to be the third value within the numerical range.

[0028] The first value is less than the third value, and the third value is less than the second value.

[0029] In one possible implementation, determining the target number of bits based on the target expected total number of files and the target expected false positive rate includes:

[0030] The error cost is obtained by multiplying the total number of target expected files by the natural logarithm of the target expected misclassification rate.

[0031] The target number of bits is obtained by dividing the absolute value of the error cost by the square of a preset first constant, where the first constant is the natural logarithm of 2.

[0032] In one possible implementation, the number of objective functions is determined based on the target number of bits and the target expected total number of files, including:

[0033] The average bit budget of the data file is obtained by dividing the target number of bits by the target expected total file size.

[0034] The number of objective functions is obtained by multiplying the average bit budget by the first constant.

[0035] A second aspect of this application provides a document receiving status monitoring device, comprising:

[0036] The file information acquisition unit is used to determine the file information of the first data file after receiving the first data file. The file information of the first data file includes a first file topic and a first file identifier.

[0037] The file mapping unit is used to map the first file identifier using a hash function of the first function number corresponding to the first file topic to obtain a first hash value of the first function number. The first function number is determined based on the first expected total number of files corresponding to the first file topic and the first bit number. The first bit number is the size of the bit array of the first bit map corresponding to the first file topic. The first bit number is determined based on the first expected false positive rate and the first expected total number of files corresponding to the first file topic.

[0038] The bitmap update unit is used to set the bits in the first bitmap with each bit indexed by the first hash value to a first state value in order to update the first bitmap.

[0039] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the file receiving status monitoring method described in the first aspect or any implementation thereof.

[0040] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0041] The memory is used to store computer programs;

[0042] The processor is used to execute the computer program so that the electronic device can implement the file receiving status monitoring method of the first aspect or any implementation thereof.

[0043] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to perform a file reception status monitoring method as described in the first aspect or any implementation thereof.

[0044] By employing the above technical solution, this application provides a file reception status monitoring method and related apparatus. Upon receiving a first data file, the method determines the file information of the first data file, which includes a first file topic and a first file identifier. Using a hash function corresponding to a first function quantity of the first file topic, the first file identifier is mapped to obtain a first hash value for the first function quantity. The first function quantity is determined based on the first expected total number of files corresponding to the first file topic and a first bit quantity. The first bit quantity is the size of the bit array of the first bit map corresponding to the first file topic and is determined based on the first expected false positive rate and the first expected total number of files corresponding to the first file topic. The bits in the first bit map, with each first hash value as their bit index, are set to first state values ​​to update the first bit map. In summary, this application records file reception status by constructing and updating bitmaps for different file topics, avoiding frequent interactions between the system and the database. It transforms the file status recording operation from time-consuming disk read / write operations to microsecond-level memory bit operations, thereby greatly improving the processing efficiency of massive file reception status monitoring and reducing storage space consumption. Attached Figure Description

[0045] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0046] Figure 1 A schematic diagram of a system architecture is provided for this application;

[0047] Figure 2 A flowchart illustrating a document receiving status monitoring method provided in this application;

[0048] Figure 3 A flowchart illustrating another document receiving status monitoring method provided in this application;

[0049] Figure 4 A flowchart illustrating another document receiving status monitoring method provided in this application;

[0050] Figure 5 A flowchart illustrating another document receiving status monitoring method provided in this application;

[0051] Figure 6 A schematic diagram of a document receiving status monitoring device provided in this application;

[0052] Figure 7 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0053] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0054] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0055] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same properties in the description of embodiments of this application. Furthermore, the terms "including" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0056] The file receiving status monitoring method provided in this application can be applied in the field of data processing, specifically in scenarios where the system needs to receive a large number of data files, to record and query the receiving status of the data files.

[0057] Taking an archive management system as an example, such a system receives reports daily from various upstream systems, themes, and institutions, with the cumulative number of reports reaching millions or even tens of millions per day. If the archive management system wants to monitor reports for a specific theme to see if any reports for a particular institution have been missed, the traditional approach would be to store detailed data of all received reports and then search for the existence of that data. This would consume significant database resources, and the frequent interaction between the system and the database during recording and retrieval would slow down the report processing speed.

[0058] In response, this application provides a file reception status monitoring method that improves the data processing efficiency of recording and querying by using bit operations based on a bitmap to record the reception status of data files.

[0059] See Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 can provide the methods provided in the embodiments of this application to one or more terminals.

[0060] The terminal 100 may be equipped with a file receiving status monitoring application. The application and webpage can provide an interface. The terminal 100 can receive relevant parameters input by the user on the application interface and send the parameters to the server 200. The server 200 can obtain the processing result based on the received parameters and return the processing result to the terminal 100.

[0061] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters on its own, without the need for the server to cooperate. This application embodiment is not limited to this.

[0062] The following description Figure 1 The product form of the mid-terminal 100;

[0063] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0064] Terminal 100 may include a radio frequency unit, memory, input unit, display unit, camera (optional), audio circuitry (optional), speaker (optional), microphone (optional), headphone jack (optional), processor, external interface, power supply, and other components. Those skilled in the art will understand that the above-mentioned components are merely examples and do not constitute a limitation on the terminal or multifunctional device; it may include more or fewer components, or a combination of certain components, or different components.

[0065] The input unit can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the portable multi-functional device. Specifically, the input unit may include a touchscreen (optional) and / or other input devices. Other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0066] Among them, the input device can receive input data, etc.

[0067] The display unit can be used to display information input by the user or information provided to the user, various menus of the terminal, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiments of this application, the display unit can be used to display an interface for monitoring file reception status, processing results, etc.

[0068] The memory can be used to store software code related to the file receiving status monitoring method, the processor can execute the steps of the file receiving status monitoring method, and can also schedule other units (such as the above-mentioned input unit and display unit) to achieve the corresponding functions.

[0069] This radio frequency unit (optional) can be used to receive and send signals during information transmission or calls.

[0070] In this embodiment of the application, the radio frequency unit can send data to the server 200 and receive the processing results sent by the server 200.

[0071] It should be understood that this radio frequency unit is optional and can be replaced with other communication interfaces, such as a network port.

[0072] Terminal 100 also includes a power source (such as a battery) for supplying power to the various components.

[0073] Terminal 100 also includes an external interface, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect terminal 100 to other devices for communication or to connect a charger to charge terminal 100.

[0074] Server 200 includes a bus, a processor, a communication interface, and memory. The processor, memory, and communication interface communicate with each other via the bus.

[0075] The memory can be used to store software code related to the file receiving status monitoring method, the processor can execute the steps of the chip's file receiving status monitoring method, and can also schedule other units to achieve the corresponding functions.

[0076] This application provides a file receiving status monitoring method, which aims to improve the monitoring efficiency of receiving massive amounts of data files and reduce storage space consumption. The file receiving status monitoring method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0077] Reference Figure 2 , Figure 2 This is a flowchart illustrating a file reception status monitoring method provided in an embodiment of this application, as follows: Figure 2As shown in the figure, the file receiving status monitoring method provided in this application embodiment may include steps S201 to S203, which are described in detail below.

[0078] S201. After receiving the first data file, determine the file information of the first data file.

[0079] In this embodiment, the file information of the first data file includes a first file subject and a first file identifier. The first file subject refers to the business category or business scenario to which the data file belongs, such as branch check-in status. The first file identifier refers to a code or keyword that can uniquely identify different specific files under the same file subject, such as the organization number that generated the report file.

[0080] In this embodiment, the first data file refers to the data file that the system actually receives and whose reception status needs to be recorded, such as a check-in report from a bank branch.

[0081] S202. Using the hash function corresponding to the number of first functions in the first file topic, map the first file identifier to obtain the first hash value of the number of first functions.

[0082] In this embodiment, the number of first functions is determined based on the total number of first expected files corresponding to the first file topic and the number of first bits. The number of first bits is the size of the bit array of the first bit map corresponding to the first file topic, and the number of first bits is determined based on the first expected false positive rate and the total number of first expected files corresponding to the first file topic.

[0083] In this embodiment, the system pre-establishes and manages independent monitoring resources, such as bitmaps and hash function sequences, for different file topics. A hash function is a function that can convert input into a fixed-length output value through a hash algorithm. Based on the current topic of the first data file, i.e., the first file topic, the system finds the optimal hash function sequence tailored to the first file topic. Then, using the first number of hash functions in the hash function sequence, parallel hash calculations are performed on the first file identifier to generate a corresponding number of hash values, i.e., the first hash values.

[0084] S203. On the first bit diagram, set the bits indexed by each first hash value to the first state value to update the first bit diagram.

[0085] In this embodiment, the first bit map refers to a data structure specifically allocated for the first file topic to record the receiving status of all files under that topic; it is essentially a binary bit array. A bit index refers to the position number or subscript of a specific binary bit in the bit map. A bit refers to a single binary digit in the bit map.

[0086] In this embodiment, the first status value refers to the status indicating that the file has been received, which is usually represented by 1 in a binary bitmap.

[0087] In this embodiment, all first hash values ​​are used as target positions (i.e., bit indices). The corresponding binary bits for these positions are found in the first bit map, and all corresponding binary bits are set to the first state value, i.e., set to 1. This bit operation is equivalent to performing an efficient state record of the arrival of this file in memory, thereby leaving a mark in the bit map that the file has been received.

[0088] As can be seen from the above technical solution, the file receiving status monitoring method provided in this application, after receiving a first data file, determines the file information of the first data file, which includes a first file topic and a first file identifier. Using a hash function corresponding to the first function quantity of the first file topic, the first file identifier is mapped to obtain a first hash value for the first function quantity. The first function quantity is determined based on the first expected total number of files and the first bit quantity corresponding to the first file topic. The first bit quantity is the size of the bit array of the first bit map corresponding to the first file topic, and is determined based on the first expected false positive rate and the first expected total number of files. The bits in the first bit map, with each first hash value as their bit index, are set as first status values ​​to update the first bit map. In summary, this application records the file receiving status by constructing and updating bitmaps for different file topics, avoiding frequent interactions between the system and the database. It transforms the file status recording operation from time-consuming disk read / write to microsecond-level memory bit operations, thereby greatly improving the processing efficiency of massive file receiving status monitoring and reducing redundant storage space consumption.

[0089] Reference Figure 3 , Figure 3 This is a flowchart illustrating another file reception status monitoring method provided in an embodiment of this application. Figure 3 This illustrates a specific implementation method for determining the reception status (received or not received) of a data file based on a bitmap, such as... Figure 3 As shown in the figure, the file receiving status monitoring method provided in this application embodiment may include steps S301 to S305, which are described in detail below.

[0090] S301. In response to the arrival of the second data file reception status verification timing, determine the file information of the second data file.

[0091] In this embodiment, the second data file refers to the specific target data file that needs to be verified for reception status. The file information of the second data file includes the second file subject and the second file identifier.

[0092] In this embodiment, the timing of receiving status verification refers to a specific time point or event that triggers the verification of whether a certain file has been received, such as when performing batch verification at the end of each business day.

[0093] S302. Using the hash function corresponding to the second function number of the second file topic, map the second file identifier to obtain the second hash value of the second function number.

[0094] In this embodiment, the number of second functions is determined based on the total number of second expected files corresponding to the second file topic and the number of second bits. The number of second bits is the size of the bit array of the second bitmap corresponding to the second file topic, and the number of second bits is determined based on the second expected false positive rate and the total number of second expected files corresponding to the second file topic.

[0095] In this embodiment, the second number of functions refers to the number of hash functions pre-calculated and configured for data such as the second file topic. The second hash value refers to the numerical result obtained by inputting the second file identifier into the hash function. The second expected total number of files refers to the total number of data files expected to be received for the second file topic. The second number of bits refers to the size of the bit array of the bitmap configured for the second file topic, that is, the total number of binary bits contained in the bitmap. The second expected false positive rate refers to the highest acceptable probability of incorrect judgment for the second file topic.

[0096] In this embodiment, the system calls a sequence of hash functions customized for the topic to which the file to be verified belongs, i.e., the second file topic, and uses this set of hash functions, which is equal to the number of second functions, to perform parallel calculations on the file's identifier and generate a corresponding number of second hash values.

[0097] S303. On the second bitmap, determine whether the bits indexed by each second hash value are all first state values.

[0098] In this embodiment, the second bitmap refers to a data structure specifically allocated for the second file topic to record the receiving status of all files under that topic; its essence is a binary bit array.

[0099] In this embodiment, all the calculated second hash values ​​are used as target positions. The corresponding binary bits of these positions are searched in the second bitmap, and it is checked whether they are all set to 1.

[0100] S304. If not, then determine that the second data file is in an unreceived state.

[0101] In this embodiment, if in the second bitmap, at least one bit that is not the first state value (i.e., not 1) is located by the second hash value, then it can be determined that the file has not been received.

[0102] S305. If so, then determine that the second data file is in the received state.

[0103] In this embodiment, if all bits located by the second hash value in the second bitmap are all of the first state value (i.e., all 1), then the probability that the second data file has been received is greater than a preset threshold. That is, although the first state value of all bits located by the second hash value may be set by mapping other files, the false positive rate is within expectations, and it can be determined that the second data file has been received.

[0104] As can be seen from the above technical solution, the file reception status monitoring method provided in this application, for the data file whose reception status needs to be verified, i.e., the second data file, achieves a fast and accurate determination of the reception status of the second data file by querying the bitmap. It is evident that by directly verifying the status value of the corresponding bit in the bitmap using the hash mapping result of the file subject and identifier, it is possible to determine whether the data file has been received without querying the original database, thereby greatly improving the efficiency of status verification and the system response speed.

[0105] Reference Figure 4 , Figure 4 This is a flowchart illustrating a file reception status monitoring method provided in an embodiment of this application. Figure 4 This illustrates the specific implementation of a bitmap and hash function for configuring an arbitrary file theme, such as... Figure 4 As shown in the figure, the file receiving status monitoring method provided in this application embodiment may include steps S401 to S405, which are described in detail below.

[0106] S401. Based on the target file topic, determine the total number of target expected files and the target expected misjudgment rate.

[0107] In this embodiment, the target file topic refers to any file topic that needs to be monitored for file reception status. Specifically, the target file topic includes a first file topic and a second file topic. The target expected total number of files refers to the total number of data files expected to be received for the target file topic. The target expected false positive rate refers to the highest probability of incorrect judgment that the system can accept for the target file topic, that is, the maximum probability that an unreceived file is incorrectly judged as received.

[0108] S402. Determine the target number of bits based on the target expected total number of files and the target expected misjudgment rate.

[0109] In this embodiment, the number of target bits is positively correlated with the total number of target expected files and negatively correlated with the target expected misjudgment rate.

[0110] In this embodiment, the number of objective functions refers to the number of optimal hash functions that need to be configured to process the subject of the target file. The number of objective functions is inversely related to the total number of target files, meaning that the more files there are, the fewer bit resources each file can occupy, and therefore the number of optimal hash functions also decreases accordingly. Conversely, it is positively related to the number of target bits, meaning that the larger the allocated bitmap space, the more hash functions can be supported.

[0111] In one optional embodiment, a specific method for determining the target number of bits based on the target expected total number of files and the target expected false positive rate includes:

[0112] Calculate the error cost by multiplying the target expected total number of files by the natural logarithm of the target expected false positive rate. Calculate the absolute value of the error cost by dividing it by the square of a preset first constant, which is the natural logarithm of 2, to obtain the target number of bits.

[0113] S403. Determine the number of objective functions based on the target number of bits and the target expected total number of files.

[0114] In this embodiment, the number of objective functions is inversely correlated with the total number of target expected files and positively correlated with the number of target bits.

[0115] Specifically, the number of objective functions refers to the optimal number of hash functions that need to be configured to process the subject of the target file. The number of objective functions is inversely related to the total number of target files, meaning that the more data files expected to be received, the fewer bit resources each data file can occupy, and therefore the fewer optimal hash functions are required. It is positively related to the target number of bits, meaning that the larger the allocated bitmap space, the more hash functions can be supported.

[0116] In one optional embodiment, a specific implementation method for determining the number of objective functions based on the target number of bits and the target expected total number of files includes:

[0117] Calculate the target number of bits and divide it by the total desired file size to obtain the average bit budget of the data file. Multiply the average bit budget by a first constant to obtain the number of objective functions.

[0118] S404. Based on the number of objective functions, determine the sequence of objective functions to be configured for the subject of the objective file.

[0119] In this embodiment, the objective function sequence refers to a set of hash functions containing the number of objective functions, which will be used to calculate the mapping of file identifiers.

[0120] Specifically, based on the calculated optimal number of objective functions, a corresponding number of hash functions are selected or generated to form a sequence of hash functions specifically for the subject of the target file.

[0121] S405. Based on the target bit count, configure an initialized target bitmap for the target file subject.

[0122] In this embodiment, the bits at each bit index of the initialized target bitmap are the second state values. That is, the initialized target bitmap refers to the initial state in which all binary bits are in the second state value. The second state value is usually represented by 0, which means that the file has not been received.

[0123] In this embodiment, the target bitmap refers to a data structure created for the target file subject to record the file reception status; essentially, it is a binary bit array. The size of the target bitmap's bit array is equal to the target number of bits.

[0124] In this embodiment, a bitmap of a specific size is allocated and initialized in memory based on the calculated target number of bits to prepare for the subsequent recording of the file reception status.

[0125] As can be seen from the above technical solutions, the file receiving status monitoring method provided in this application achieves an optimal balance between resource consumption and judgment accuracy by calculating and configuring the bitmap size and the number of hash functions based on the expected size of the file topic (target expected total number of files) and the fault tolerance requirement (target expected false judgment rate). This application provides customized and theoretically optimal initialization configurations for different file topics, ensuring the efficiency and reliability of the monitoring method from the bottom layer of the system.

[0126] Reference Figure 5 , Figure 5 This is a flowchart illustrating another file reception status monitoring method provided in an embodiment of this application. Figure 5 The following illustrates S401, which details the implementation process for determining the total number of target files and the target expected misclassification rate based on the target file topic. Figure 5 As shown in the figure, the file receiving status monitoring method provided in this application embodiment may include steps S501 to S506, which are described in detail below.

[0127] S501. The total number of data files belonging to the subject of the target file is taken as the total number of target expected files.

[0128] In this embodiment, based on business rules or historical data, the total number of all expected data files belonging to the target file topic is directly determined as the total number of target expected files.

[0129] S502. Based on the proportion of remaining memory, determine the storage efficiency coefficient corresponding to the target file topic.

[0130] In this embodiment, the storage efficiency coefficient is inversely correlated with the proportion of remaining memory.

[0131] In this embodiment, the remaining memory ratio refers to the percentage of currently available memory resources in the system relative to the total memory. The storage efficiency coefficient is a quantitative indicator reflecting the current resource scarcity of the system, used to prioritize memory saving in decision-making regarding false positive rates. The storage efficiency coefficient is inversely correlated with the remaining memory ratio; that is, the lower the remaining memory ratio, the more strained the memory resources, and the larger the value of the storage efficiency coefficient, indicating a higher requirement for storage efficiency. In this case, a more space-saving bitmap configuration and / or hash function configuration should be determined.

[0132] S503. Based on the importance of the target file topic, determine the data integrity coefficient corresponding to the target file topic.

[0133] In this embodiment, the importance of the target file topic is positively correlated with the data integrity coefficient.

[0134] In this embodiment, the importance of the target file topic refers to the criticality or value of the data file corresponding to that topic. The data integrity coefficient reflects a quantitative indicator of the business-level requirements for monitoring accuracy, used to prioritize data integrity in decision-making regarding false positive rates. The importance of the target file topic is positively correlated with the data integrity coefficient; that is, the more important the business topic, the larger the data integrity coefficient, indicating a higher requirement for the accuracy of monitoring results regarding data reception status. In this case, bitmap configurations and / or hash function configurations that can improve the accuracy of monitoring results should be determined.

[0135] S504. When the data integrity coefficient is greater than the preset target threshold and greater than the storage efficiency coefficient, the target expected misjudgment rate is determined to be the first value within the preset numerical range.

[0136] In this embodiment, the target threshold is pre-configured as a preset baseline value to determine the impact of the storage efficiency coefficient or data integrity coefficient. The first value refers to a smaller value within the selectable range of the false positive rate, representing a configuration strategy of high accuracy and low false positive rate.

[0137] In this embodiment, the data integrity coefficient is greater than the target threshold and greater than the storage efficiency coefficient, indicating that the integrity requirements of the data files under the target file topic are high and their priority is higher than the storage resource constraints. Therefore, the target expected misjudgment rate is set to a smaller value within the selectable range of misjudgment rates, that is, the first value, to ensure the highest monitoring accuracy.

[0138] S505. When the storage efficiency coefficient is greater than the target threshold and greater than the data integrity coefficient, the target expected misclassification rate is determined to be the second value within the numerical range.

[0139] In this embodiment, the second value refers to a larger value within the selectable range of the false positive rate, representing a configuration strategy with high space efficiency and high false positive rate.

[0140] In this embodiment, the storage efficiency coefficient is greater than the target threshold and greater than the data integrity coefficient, indicating that the system resource constraints are very tight and the priority of memory resource utilization efficiency is higher than the business accuracy requirement. Therefore, the target expected misjudgment rate is set to a larger value within the numerical range, that is, the second value, in order to prioritize the utilization efficiency of memory resources.

[0141] S506. When both the data integrity coefficient and the storage efficiency coefficient are not greater than the target threshold, or when both the data integrity coefficient and the storage efficiency coefficient are not less than the target threshold, the target expected misclassification rate is determined to be the third value within the numerical range.

[0142] In this embodiment, the first value is less than the third value, and the third value is less than the second value. That is, the third value is an intermediate value within the selectable range, representing a configuration strategy that strikes a balance between accuracy requirements and space efficiency requirements.

[0143] In this embodiment, when both the data integrity coefficient and the storage efficiency coefficient are not greater than the target threshold (i.e., when the influence of both coefficients is not significant), or when both the data integrity coefficient and the storage efficiency coefficient are not less than the target threshold (i.e., when the influence of both coefficients is significant and they compete with each other), the target expected misclassification rate is set to the third value. This ensures the rationality and gradient of the misclassification rate values ​​under different strategies.

[0144] For example, the preset target threshold is 5. The false positive rate ranges from [0, 5%], with the first value set to 0.1%, the third value to 1%, and the second value to 5%.

[0145] In the first scenario, the data integrity coefficient of the target file topic is 8, and the storage efficiency coefficient is 4. Since the data integrity coefficient is greater than the target threshold of 5 and also greater than the storage efficiency coefficient of 4, an expected false positive rate of 0.1% is selected for this topic to prioritize accuracy.

[0146] In the second scenario, the data integrity coefficient of the target file topic is 3, and the storage efficiency coefficient is 7. Since the storage efficiency coefficient is greater than the target threshold of 5 and also greater than the data integrity coefficient of 3, a 5% expected false positive rate is selected for this topic to prioritize resource conservation.

[0147] In the third case, the data integrity coefficient is 4 and the storage efficiency coefficient is 3, both of which are not greater than the target threshold of 5.

[0148] In the fourth case, the data integrity coefficient is 7 and the storage efficiency coefficient is 6, both of which are not less than the target threshold of 5.

[0149] In the third and fourth cases, a balanced strategy is adopted, selecting an expected misjudgment rate of 1% for the topic.

[0150] As can be seen from the above technical solutions, the file receiving status monitoring method provided in this application dynamically and adaptively determines the target misjudgment rate by comprehensively considering the real-time resource status of the system (remaining memory ratio) and the importance of the business topic. This achieves a balance between system resource constraints and accuracy requirements under business scenarios. As a result, the monitoring method can flexibly adjust the monitoring strategy according to the operating environment and business priorities, thereby optimizing the overall resource utilization efficiency while ensuring the integrity of core business data.

[0151] Next, taking a file management system as an example, a document receiving status monitoring method provided in this application embodiment will be described in detail.

[0152] As an archive management system, it needs to receive data files, i.e., report files, from different upstream systems, different themes, and different institutions every day. The cumulative number of report files increases to millions or tens of millions every day. The archive management system monitors report files of a specified theme to see if there are any reports missing from a specific institution. In the traditional method, the archive management system needs to store the detailed data of all received report files and then search for the existence of the data. This will consume a lot of database resources, and the system will frequently interact with the database during the recording and retrieval process, which will reduce the processing speed of report files.

[0153] To address this, the file reception status monitoring method provided in this application is based on bitmaps and Bloom filters to automatically monitor the file reception status of report files, thereby determining whether files are missing. Specifically, based on the report file topics currently supported by the document management system, a bitmap is created for each topic. Using a Bloom filter multi-bit hash mapping mechanism, a specific number of bits in the bitmap are dynamically allocated to record the arrival status of each file under each topic. During the report file reception process, for each specific report file parsed, the bit corresponding to the bit index on the corresponding bitmap is set to 1 through multi-bit hash mapping. At the end of each day, the system checks whether the corresponding bits are all 1 based on the report file's hash mapping, thereby determining whether the report file is missing, i.e., not received.

[0154] First, based on all the report file themes that the document management system already supports, a corresponding bitmap is created for each theme's report file. The size of the bitmap's bit array is determined by the number of document organization numbers supported by the document management system, i.e., the total number of report files expected to be received daily. For example, for a bank's operations management system, each branch generates a daily attendance report file. At the end of each day, the operations management system packages all branch report files and sends them to the document management system. The document management system registers 20,000 branches of the bank, so it is expected to receive a maximum of 20,000 branch attendance report files daily. In this case, a bitmap with the theme of branch attendance should be created. Initially, each bit is 0. After generation, the bitmap is stored in Redis, with each bit index in the initial bitmap corresponding to the second state value, i.e., 0.

[0155] Furthermore, a multi-bit hash mapping algorithm is determined. For example, to detect these 20,000-bit report files, the system designs three hash mapping algorithms: Hash1(), Hash2(), and Hash3(). When the system processes a specific organization's branch check-in report file, it takes the organization number, assuming it is A00001, and performs the above three hash mapping algorithms as follows:

[0156] Hash1(A00001)= 1, Hash2(A00001)= 2, Hash3(A00001)= 7.

[0157] Furthermore, using Redis's bit manipulation methods, the bits with indexes (i.e., bit indices) of 1, 2, and 7 in the bitmap above are set to 1 respectively.

[0158] It should be noted that if the report file is received from another organization on the same day, the same algorithm will be used to keep the corresponding position 1 until the end of the day, which is the time when the document management system stops receiving report files, for example, 24:00 every night.

[0159] Furthermore, after the document management system stops receiving report files, it verifies whether it has received a report file from a certain institution, such as A00009. It performs the same hash mapping on the file to determine if all corresponding bits in the bitmap are 1. If so, the report file is considered received, indicating it may have been received and processed. If not, the report file is considered unreceived, meaning it has definitely not been received and processed.

[0160] Furthermore, regarding how to choose the appropriate Bloom filter size and the number of hash functions, the following is a summary:

[0161] The size of the Bloom filter (i.e., the size of the bit array m) and the number of hash functions k need to be calculated based on the expected total number of files and the maximum acceptable false positive rate (i.e., the expected false positive rate).

[0162] Specifically, the size of the bit array m = -n×ln(p) / (ln2)^2. The number of hash functions k = m / n·ln2 ≈0.7×m / n. Where n is the expected number of elements to be inserted (i.e., the total number of files expected to be received per day), p is the expected false positive rate (usually set to 0.1%~1%), m is the size of the bit array, and k is the number of hash functions.

[0163] It should be noted that the false positive rate is related to the probability of empty positions in the bit array. After inserting n elements, the probability that a certain bit is still 0 is (1-1 / m)^(k·n). A false positive will only occur when all k hashes are 1. Therefore, the false positive rate p ≈ (1-e^(-k·n / m))^k, which simplifies to the expression for m. Furthermore, by taking the derivative of the false positive rate p with respect to k, and setting the derivative to 0, we obtain the optimal k.

[0164] For example, if the expected number of report files received daily is n=1 million, and the expected false positive rate is p=0.01%, then:

[0165] m = -1000000 × ln(0.0001) / (ln2)² ≈ -1000000 × (-9.2103) / (0.4804) ≈ 19162065 bits (approximately 2.4MB).

[0166] k = 19162065 / 1000000 × 0.6931(ln2≈0.6931)≈ 13.29 ≈ 14.

[0167] In other words, for the report file, a 19,162,065-bit bitmap needs to be created to determine 14 hash mappings.

[0168] Furthermore, to verify whether the false positive rate meets the standard, that is, substituting m=19162065, k=14, n=1 million, we calculate p ≈ (1-e^{-k×n / m})^k = (1-e^{-14×1000000 / 19162065})^14 ≈ (1-e^{-0.7306})^14 ≈ 0.0000998 ≈ 0.01%, which meets the expectations.

[0169] Furthermore, adjust p according to the business scenario, and thus adjust m and k, as follows:

[0170] In scenarios sensitive to misjudgment (such as financial transactions), since the data files are of high importance in this business scenario, we set p=0.01%~0.1% and increase m and k, sacrificing space for accuracy. In space-sensitive scenarios (such as log statistics), we set p=1%~5% and reduce m and k, trading an acceptable misjudgment rate for space storage efficiency.

[0171] For example, sign-in report files are less important than financial transaction data files and have lower sensitivity to misjudgments. Therefore, a small amount of accuracy can be sacrificed in exchange for space storage efficiency.

[0172] In summary, the file receiving status monitoring method provided in the embodiments of this application can achieve at least the following technical effects:

[0173] First, it reduces the consumption of storage resources and saves database resources.

[0174] Bitmap storage is essentially a binary array, with each element occupying 1 bit (1 / 8 byte). Taking the storage of 1 million file-received data entries as an example, if stored as a database, this would add 1 million rows of data. Assuming each record contains at least the fields shown in Table 1, each record would occupy approximately 10 + 3 + 20 + 6 + 8 = 47 bytes. Therefore, the total number of bytes occupied by 1 million records would be approximately 47 × 1 million = 47,000,000 bytes, or about 45 MB.

[0175] Table 1

[0176] Field meaning Field type length Primary Key ID varchar 10 bytes File arrival time date 3 bytes Report file name varchar The average length is 20 bytes. Organization Number varchar The average length is 6 bytes. Report File Theme varchar The average length is 8 bytes.

[0177] If a bitmap combined with a Bloom filter is used, assuming the expected false positive rate is no more than 0.1% (p = 0.001), then m = -n×ln(p) / (ln2)^2, which requires approximately 14,379,164 bits of space. Converted to bytes (1 byte = 8 bits), this is approximately 1,797,000 bytes, or about 1.8 MB of space.

[0178] As can be seen, this saves dozens of times more space compared to the previous method.

[0179] Second, improve data processing efficiency.

[0180] According to the traditional database storage method, each file record requires an interaction with the database. Assuming that the database uses the batch insertion method and submits every 3,000 records, the insertion and submission time for every 3,000 records is about 1 second. Then, the time required to insert 1 million records can be estimated as follows: 1 million records / 3,000 records per time × 1 second per time, the total time is about 333 seconds, or 5.6 minutes.

[0181] The bit operations used in this application are very fast, typically taking microseconds per operation. Therefore, for 14,379,164 bit operations, assuming each operation takes one microsecond, the total time would be approximately 14,379,164 / 1,000,000 ≈ 14.4 seconds.

[0182] As can be seen, this saves dozens of times the time compared to other methods.

[0183] Third, improve the accuracy of monitoring results.

[0184] Specifically, Bloom filters can support a false positive rate of 0.1% to 1% with a balance between bit space and mapping number, and have high accuracy.

[0185] The above describes a file receiving status monitoring method provided by the embodiments of this application. The following describes the apparatus for performing the above file receiving status monitoring method.

[0186] Please see Figure 6 , Figure 6 This is a schematic diagram of a file receiving status monitoring device provided in an embodiment of this application. Figure 6 As shown, the document receives a status monitoring device 600, which includes:

[0187] The file information acquisition unit 601 is used to determine the file information of the first data file after receiving the first data file. The file information of the first data file includes a first file topic and a first file identifier.

[0188] The file mapping unit 602 is used to map the first file identifier using a hash function of the first function number corresponding to the first file topic to obtain a first hash value of the first function number. The first function number is determined based on the first expected total number of files corresponding to the first file topic and the first bit number. The first bit number is the size of the bit array of the first bit map corresponding to the first file topic. The first bit number is determined based on the first expected false positive rate and the first expected total number of files corresponding to the first file topic.

[0189] Bitmap update unit 603 is used to set the bits in the first bitmap with each bit indexed by the first hash value to a first state value in order to update the first bitmap.

[0190] In one possible implementation, the file receiving status monitoring device further includes a status monitoring unit for:

[0191] In response to the arrival of the second data file reception status verification timing, the file information of the second data file is determined, the file information of the second data file includes the second file topic and the second file identifier;

[0192] The second file identifier is mapped using the hash function of the second function number corresponding to the second file topic to obtain the second hash value of the second function number. The second function number is determined based on the second expected total number of files and the second bit number corresponding to the second file topic. The second bit number is the size of the bit array of the second bitmap corresponding to the second file topic. The second bit number is determined based on the second expected false positive rate and the second expected total number of files corresponding to the second file topic.

[0193] On the second bitmap, it is determined whether each bit indexed by the second hash value is a first state value;

[0194] If not, then it is determined that the second data file is in an unreceived state;

[0195] If so, then the second data file is determined to be in a received state.

[0196] In one possible implementation, the file receiving status monitoring device further includes a bitmap configuration unit for performing the following for any target file topic:

[0197] Based on the target file topic, determine the target expected total number of files and the target expected misclassification rate. The target file topic includes a first file topic and a second file topic.

[0198] The target number of bits is determined based on the total number of target expected files and the target expected false positive rate. The target number of bits is positively correlated with the total number of target expected files and negatively correlated with the target expected false positive rate.

[0199] The number of objective functions is determined based on the target number of bits and the target expected total number of files. The number of objective functions is inversely correlated with the target expected total number of files and positively correlated with the target number of bits.

[0200] Based on the number of objective functions, a corresponding sequence of objective functions is configured for the subject of the target file, wherein the sequence of objective functions includes the number of objective hash functions specified in the objective functions.

[0201] Based on the target number of bits, an initialized target bitmap is configured for the target file topic, and the bits at each bit index of the initialized target bitmap are second state values; the size of the bit array of the target bitmap is the target number of bits.

[0202] In one possible implementation, the bitmap configuration unit, when determining the target expected total number of files and the target expected misclassification rate based on the target file topic, specifically performs the following:

[0203] The total number of data files belonging to the subject of the target file is taken as the total number of the target expected files;

[0204] Based on the remaining memory ratio, the storage efficiency coefficient corresponding to the target file topic is determined, and the storage efficiency coefficient is inversely correlated with the remaining memory ratio.

[0205] Based on the importance of the target file topic, a data integrity coefficient corresponding to the target file topic is determined, and the importance of the target file topic is positively correlated with the data integrity coefficient;

[0206] When the data integrity coefficient is greater than the preset target threshold and greater than the storage efficiency coefficient, the target expected misjudgment rate is determined to be the first value within the preset numerical range;

[0207] When the storage efficiency coefficient is greater than the target threshold and also greater than the data integrity coefficient, the target expected false positive rate is determined to be the second value within the specified range.

[0208] When both the data integrity coefficient and the storage efficiency coefficient are not greater than the target threshold, or when both the data integrity coefficient and the storage efficiency coefficient are not less than the target threshold, the target expected misjudgment rate is determined to be the third value within the numerical range.

[0209] The first value is less than the third value, and the third value is less than the second value.

[0210] In one possible implementation, the bitmap configuration unit, when determining the target number of bits based on the target expected file total and the target expected misclassification rate, specifically performs the following:

[0211] The error cost is obtained by multiplying the total number of target expected files by the natural logarithm of the target expected misclassification rate.

[0212] The target number of bits is obtained by dividing the absolute value of the error cost by the square of a preset first constant, where the first constant is the natural logarithm of 2.

[0213] In one possible implementation, the bitmap configuration unit, when determining the number of target functions based on the target number of bits and the total number of the target desired file, is specifically used for:

[0214] The average bit budget of the data file is obtained by dividing the target number of bits by the target expected total file size.

[0215] The number of objective functions is obtained by multiplying the average bit budget by the first constant.

[0216] This application also provides an electronic device. Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, with reference to... Figure 7 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0217] like Figure 7 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. When the electronic device is powered on, the RAM 703 also stores various programs and data required for the operation of the electronic device. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0218] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, memory cards, hard drives, etc.; and communication devices 709. Communication device 709 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0219] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the file receiving status monitoring methods provided in this application.

[0220] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the file receiving status monitoring methods provided in this application.

[0221] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0222] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0223] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0224] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for monitoring file reception status, characterized in that, include: Upon receiving the first data file, determine the file information of the first data file, which includes the first file subject and the first file identifier; The first file identifier is mapped using a hash function corresponding to the first function number of the first file topic to obtain a first hash value of the first function number. The first function number is determined based on the first expected total number of files and the first bit number corresponding to the first file topic. The first bit number is the size of the bit array of the first bit diagram corresponding to the first file topic. The first bit number is determined based on the first expected false positive rate and the first expected total number of files corresponding to the first file topic. The first bitmap is updated by setting the bits indexed by the first hash value to the first state value.

2. The file receiving status monitoring method according to claim 1, characterized in that, The file receiving status monitoring method further includes: In response to the arrival of the second data file reception status verification timing, the file information of the second data file is determined, the file information of the second data file includes the second file topic and the second file identifier; The second file identifier is mapped using the hash function of the second function number corresponding to the second file topic to obtain the second hash value of the second function number. The second function number is determined based on the second expected total number of files and the second bit number corresponding to the second file topic. The second bit number is the size of the bit array of the second bitmap corresponding to the second file topic. The second bit number is determined based on the second expected false positive rate and the second expected total number of files corresponding to the second file topic. On the second bitmap, it is determined whether each bit indexed by the second hash value is a first state value; If not, then it is determined that the second data file is in an unreceived state; If so, then the second data file is determined to be in a received state.

3. The file receiving status monitoring method according to claim 2, characterized in that, The file receiving status monitoring method further includes: Execute on any target file topic: Based on the target file topic, determine the target expected total number of files and the target expected misclassification rate. The target file topic includes a first file topic and a second file topic. The target number of bits is determined based on the total number of target expected files and the target expected false positive rate. The target number of bits is positively correlated with the total number of target expected files and negatively correlated with the target expected false positive rate. The number of objective functions is determined based on the target number of bits and the target expected total number of files. The number of objective functions is inversely correlated with the target expected total number of files and positively correlated with the target number of bits. Based on the number of objective functions, a corresponding sequence of objective functions is configured for the subject of the target file, wherein the sequence of objective functions includes the number of objective hash functions specified in the objective functions. Based on the target number of bits, an initialized target bitmap is configured for the target file topic, and the bits at each bit index of the initialized target bitmap are second state values; the size of the bit array of the target bitmap is the target number of bits.

4. The file receiving status monitoring method according to claim 3, characterized in that, Based on the target file topic, determine the total number of target expected files and the target expected false positive rate, including: The total number of data files belonging to the subject of the target file is taken as the total number of the target expected files; Based on the remaining memory ratio, the storage efficiency coefficient corresponding to the target file topic is determined, and the storage efficiency coefficient is inversely correlated with the remaining memory ratio. Based on the importance of the target file topic, a data integrity coefficient corresponding to the target file topic is determined, and the importance of the target file topic is positively correlated with the data integrity coefficient; When the data integrity coefficient is greater than the preset target threshold and greater than the storage efficiency coefficient, the target expected misjudgment rate is determined to be the first value within the preset numerical range; When the storage efficiency coefficient is greater than the target threshold and also greater than the data integrity coefficient, the target expected false positive rate is determined to be the second value within the specified range. When both the data integrity coefficient and the storage efficiency coefficient are not greater than the target threshold, or when both the data integrity coefficient and the storage efficiency coefficient are not less than the target threshold, the target expected misjudgment rate is determined to be the third value within the numerical range. The first value is less than the third value, and the third value is less than the second value.

5. The file receiving status monitoring method according to claim 3, characterized in that, Determining the target number of bits based on the target expected total number of files and the target expected false positive rate includes: The error cost is obtained by multiplying the total number of target expected files by the natural logarithm of the target expected misclassification rate. The target number of bits is obtained by dividing the absolute value of the error cost by the square of a preset first constant, where the first constant is the natural logarithm of 2.

6. The file receiving status monitoring method according to claim 5, characterized in that, The number of objective functions is determined based on the target number of bits and the total number of target desired files, including: The average bit budget of the data file is obtained by dividing the target number of bits by the target expected total file size. The number of objective functions is obtained by multiplying the average bit budget by the first constant.

7. A file receiving status monitoring device, characterized in that, include: The file information acquisition unit is used to determine the file information of the first data file after receiving the first data file. The file information of the first data file includes a first file topic and a first file identifier. The file mapping unit is used to map the first file identifier using a hash function of the first function number corresponding to the first file topic to obtain a first hash value of the first function number. The first function number is determined based on the first expected total number of files corresponding to the first file topic and the first bit number. The first bit number is the size of the bit array of the first bit map corresponding to the first file topic. The first bit number is determined based on the first expected false positive rate and the first expected total number of files corresponding to the first file topic. The bitmap update unit is used to set the bits in the first bitmap with each bit indexed by the first hash value to a first state value in order to update the first bitmap.

8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the file reception status monitoring method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the file reception status monitoring method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the file reception status monitoring method as described in any one of claims 1 to 6.

Images