File data writing method, device and computer readable storage medium

Abstract

The present invention relates to a big data technology, and discloses a file data writing method, including: storing multiple acquired files and generating a file task list; calling the files in the file task list to collect them into a temporary file set, and classifying the files in the temporary file set to obtain a classification task table; selecting files with the same classification in the classification task table and putting them into a preset task queue, and when there is a vacancy in the task queue, selecting files with the same classification and putting them into the task queue, until all the files with the same classification in the classification task list are put into the task queue, selecting files of other classifications and continuing to put them into the task queue, and writing the files in the task queue into a preset electronic device. The present invention also proposes a file data writing device, an electronic device, and a computer-readable storage medium. The present invention can solve the problem of low efficiency and high cost in writing tens of millions of file data.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a file data writing method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the rise of big data, people generate a huge amount of data in their daily network activities, such as browsing the web, uploading data, downloading data, etc. These generated data will form tens of millions of file data, so the demand for processing tens of millions of file data is increasing.

[0003] However, since tens of millions of files contain a huge amount of data, too much computing memory is occupied when writing them, and the writing efficiency is low. Therefore, there is an increasing demand for how to write tens of millions of files at a low cost and high efficiency. Summary of the invention

[0004] The present invention provides a file data writing method, device, electronic device and computer-readable storage medium, the main purpose of which is to solve the problem of low efficiency and high cost in writing tens of millions of file data.

[0005] To achieve the above object, the present invention provides a method for writing file data, comprising:

[0006] Store the acquired files and generate a file task list;

[0007] Calling the files in the file task list to collect them into a temporary file set, and classifying the files in the temporary file set according to keywords;

[0008] Selecting files from the temporary file set according to the classification and putting them into a preset task queue;

[0009] The files in the task queue are written into a preset electronic device.

[0010] Optionally, classifying the files in the temporary file set according to keywords includes:

[0011] Calculating the probability of each information word contained in the temporary file set by using a first probability formula;

[0012] According to the probability of the information words, the sentence probability of the information words being arranged into sentences according to different permutations and combinations is calculated using a second probability formula;

[0013] According to the sentence probabilities, k permutations and combinations are selected to obtain a sentence probability set;

[0014] Calculating the similarity between the sentences in the sentence probability set and the keywords in the preset keyword set to obtain a similarity result;

[0015] Each file in the file set is classified according to the similarity result.

[0016] Optionally, the calculating the probability of the information word contained in each file in the temporary file set by using the first probability formula includes:

[0017] The probability P(C) of the occurrence of the nth information word is calculated using the following first probability calculation formula:

[0018] P(C)=P(W n |W1,…W n-1 )

[0019] Among them, W n is the nth information word, P(W n |W1,…W n-1 ) is a conditional probability calculation, which indicates the probability of the nth information word appearing when the n-1th information word appears.

[0020] Optionally, the calculating the sentence probability of the information words being arranged into sentences according to different permutations and combinations by using the second probability formula includes:

[0021] The information words in the file S are calculated by using the following second probability calculation formula: m The sentence probability P(S) of a sequence arrangement and combination into a sentence is:

[0022] P(S)=P(W1)P(W2|W1)…P(W m |W m-n+1 ,…W m-1 )

[0023] Among them, W m represents the mth information word in the file, P(W m |W m-n+1 ,…W m-1 ) is the conditional probability, indicating the Wth m-n+1 When the information word appears, W m Probability of occurrence.

[0024] Optionally, calculating the similarity between the sentences in the sentence probability set and the keywords in the preset keyword set to obtain a similarity result includes:

[0025] The similarity Sim between the sentences in the sentence probability set and the keywords in the preset keyword set is calculated using the following similarity calculation formula:

[0026]

[0027] Among them, g i The preset keyword set G = {g1, g2...g t} any keyword; q a is any sentence in the sentence probability set; count is a similarity calculation; argmax is a maximum calculation; and t is the number of keywords in the keyword set. Optionally, selecting a file from the temporary file set and putting it into a preset task queue according to the classification includes:

[0028] When there is a vacancy in the task queue, determining whether there is a file in the temporary file set that has the same classification as the file in the task queue;

[0029] If there are files in the temporary file set that have the same classification as the files in the task queue, then arbitrarily select one of the files in the same classification; and if there are no files in the temporary file set that have the same classification as the files in the task queue, then select a file in the next classification;

[0030] Calculate the access value Z of the selected file i :

[0031] Z i =P i +d i +T i ,

[0032] Among them, P i The queue priority for the selected file; d i The absolute deadline for the selected document; i a write cycle for the selected file;

[0033] If the admission value Z i is less than or equal to an admission threshold X, then the selected file is refused to enter the task queue;

[0034] If the admission value Z i If the value is greater than the admission threshold X, the selected file is allowed to enter the task queue.

[0035] Optionally, the method further includes performing abnormality monitoring on a process of writing files in the task queue to a preset electronic device, obtaining monitoring results, and outputting the monitoring results.

[0036] In order to solve the above problem, the present invention further provides a file data writing device, the device comprising:

[0037] The file acquisition module is used to store the acquired files and generate a file task list;

[0038] A keyword classification module, used for calling the files in the file task list to collect them into a temporary file set, and classifying the files in the temporary file set according to keywords;

[0039] A file writing module, used for selecting files from the temporary file set according to the classification and putting them into a preset task queue, and writing the files in the task queue into a preset electronic device;

[0040] The abnormality monitoring module is used to perform abnormality monitoring in the process of writing the files in the task queue to the preset electronic device, obtain the monitoring result, and output the monitoring result.

[0041] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0042] a memory storing at least one instruction; and

[0043] The processor executes the instructions stored in the memory to implement any one of the file data writing methods described above.

[0044] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement any one of the file data writing methods described above.

[0045] The embodiment of the present invention obtains multiple files, stores the files, and obtains a file task list. The number of the obtained files may be huge and difficult to process in a short time, so they are first stored for further processing later; the files in the file task list are called, collected into a temporary file set, and the files in the temporary file set are classified according to keywords to obtain a classification task table. According to the keywords, the files are classified according to the contents contained in the files, so that the files can be read and written more quickly later; files with the same classification in the classification task table are selected and put into a preset task queue. When there is a vacancy in the task queue, files with the same classification as in the task queue are preferentially selected and put into the task queue until all the files with the same classification as in the classification task list are put into the task queue, and files with different classifications in the task queue are randomly selected and continue to be put into the task queue, and the files in the task queue are written into a preset electronic device. Different types of files have different writing configurations, and configuration changes need to be made when writing different types of files. The present invention puts files with similar contents together for file writing, thereby realizing fast file writing with low computing memory occupancy. Therefore, the file data writing method, device and computer-readable storage medium proposed in the present invention can achieve low-cost and high-efficiency writing of tens of millions of file data. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a flow chart of a method for writing file data provided by an embodiment of the present invention;

[0047] Figure 2 Module diagram of a file data writing method provided by an embodiment of the present invention

[0048] Figure 3 A schematic diagram of the internal structure of an electronic device for a file data writing method provided by an embodiment of the present invention;

[0049] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0051] The present invention provides a method for writing file data. Figure 1 FIG. 1 is a flow chart of a method for writing file data provided by an embodiment of the present invention. The method can be executed by a device, and the device can be implemented by software and / or hardware.

[0052] In this embodiment, the file data writing method includes:

[0053] S1. Store the acquired files and generate a file task list.

[0054] In detail, storing the original file includes:

[0055] NAS (Network Attached Storage) is used to store files uploaded by users through the client.

[0056] The NAS is a device connected to the network and has a data storage function, so it is also called "network storage". It is a dedicated data storage server that is data-centric, completely separates storage devices from servers, and centrally manages data, thereby freeing up bandwidth, improving performance, and reducing costs. The embodiment of the present invention uses NAS to store the uploaded files, which can effectively improve storage efficiency.

[0057] When the storage is completed, the file task list is obtained.

[0058] S2. Call the files in the file task list to collect them into a temporary file set, and classify the files in the temporary file set according to keywords.

[0059] Preferably, the number of files acquired by the embodiment of the present invention is very large, and the acquired files need to be classified so as to achieve faster writing of the files later.

[0060] In detail, the S2 includes:

[0061] Calling multiple files in the file task list and collecting the multiple files into a temporary file set;

[0062] Calculating the probability of each information word contained in the temporary file set by using a first probability formula;

[0063] According to the probability of the information words, the sentence probability of the information words being arranged into sentences according to different permutations and combinations is calculated using a second probability formula;

[0064] According to the sentence probabilities, k permutations and combinations are selected to obtain a sentence probability set;

[0065] Calculating the similarity between the sentences in the sentence probability set and the keywords in the preset keyword set to obtain a similarity result;

[0066] Each file in the file set is classified according to the similarity result.

[0067] The information words refer to words containing characteristic information in the file, such as a file containing a network connection is defined as a web page file, and a file containing a computer code is defined as a source code file, etc.

[0068] In detail, the embodiment of the present invention can use a pessimistic locking method to call multiple files in the file task list.

[0069] The pessimistic locking mode means that it is assumed that each time the file is called, there will be other processes modifying the file, so the file will be locked each time it is called, so that other processes cannot modify the file.

[0070] After calling multiple files in the file task list, the multiple files are collected to obtain the temporary file set.

[0071] Furthermore, the method of calculating the probability of the information word contained in each file in the temporary file set by using the first probability formula includes:

[0072] The probability P(C) of the information word appearing at the nth information word is calculated using the following first probability calculation formula:

[0073] P(C)=P(W n |W1,…W n-1 )

[0074] Among them, W n is the nth information word, P(W n |W1,…W n-1 ) is a conditional probability calculation, which indicates the probability of the nth information word appearing when the n-1th information word appears.

[0075] Furthermore, the calculating, based on the probability of the information words, using a second probability formula to calculate the sentence probability of the information words being arranged into sentences according to different permutations and combinations includes:

[0076] For any file S={W1, W2…W m}, calculate the information words in the file S according to the information word probability P(C) according to W1, W2…W m The sentence probability P(S) of a sequence arrangement and combination into a sentence is:

[0077] P(S)=P(W1)P(W2|W1)…P(W m |W m-n+1 ,…W m-1 )

[0078] Among them, W m represents the mth information word in the file S, P(W m |W m-n+1 ,…W m-1 ) is the conditional probability, indicating the Wth m-n+1 When the information word appears, Wm Probability of occurrence.

[0079] Furthermore, the sentence probability P(S) of all possible information words is calculated, and according to the sentence probability P(S), k combinations of information words with the largest sentence probability are selected to obtain a sentence probability set, which is represented by Q = {q1, q2…q h}.

[0080] Furthermore, the calculating the similarity between the sentences in the sentence probability set and the keywords in the preset keyword set to obtain the similarity result includes:

[0081] The similarity calculation formula is used to calculate the similarity between the sentences in the sentence probability set and the preset keyword set G = {g1, g2…g t The similarity Sim of the most similar keyword in}:

[0082]

[0083] Among them, g i is any keyword in the keyword set; a is any sentence in the sentence probability set; count is a similarity operation; argmax is a maximum value operation; and t is the number of keywords in the keyword set.

[0084] After all the sentences in the sentence probability set have completed the above calculation, each file in the file set is classified according to the calculated similarity result, and the files with the same classification are collected together to obtain the classification task table.

[0085] S3. Selecting files from the temporary file set according to the classification and putting them into a preset task queue, and writing the files in the task queue into a preset electronic device.

[0086] In an embodiment of the present invention, since the amount of data contained in the files in the task queue is very large, if the files are to be written together, a large amount of computing memory will be occupied, and the writing will take a long time and be costly. Therefore, it is necessary to give priority to selecting files with the same classification in the classification task table and putting them into the task queue to achieve priority processing of file writing of the same type, reduce the computing memory usage, and reduce the cost of file writing.

[0087] The task queue is a queue with a preset length threshold. When the number of files in the task queue is less than the length threshold, files that meet the preset access principle are randomly selected from the classification task table and put into the task queue until the number of files in the task queue is equal to the length threshold;

[0088] When the number of files in the task queue is greater than or equal to the length threshold, the random selection of files that meet the admission principle from the classified task table and put into the task queue is suspended.

[0089] Furthermore, selecting files from the temporary file set according to the classification and putting them into a preset task queue includes:

[0090] When there is a vacancy in the task queue, determining whether there is a file in the temporary file set that has the same classification as the file in the task queue;

[0091] If there are files in the temporary file set that have the same classification as the files in the task queue, then arbitrarily select one of the files in the same classification; and if there are no files in the temporary file set that have the same classification as the files in the task queue, then select a file in the next classification;

[0092] Calculate the access value Z of the selected file i :

[0093] Z i =P i +d i +T i ,

[0094] Among them, P i The queue priority for the selected file; d i The absolute deadline for the selected document; i a write cycle for the selected file;

[0095] If the admission value Z i is less than or equal to an admission threshold X, then the selected file is refused to enter the task queue;

[0096] If the admission value Z i If the value is greater than the admission threshold X, the selected file is allowed to enter the task queue.

[0097] S4. Perform abnormality monitoring during the process of writing the files in the task queue into a preset electronic device, obtain monitoring results, and output the monitoring results.

[0098] In detail, the S4 includes:

[0099] Collecting write logs during the file writing process and storing the write logs;

[0100] Inputting the stored write log and the preset abnormal keyword set into a preset analysis algorithm to obtain analysis results, and aggregating the analysis results to obtain the monitoring results;

[0101] The monitoring result is outputted.

[0102] Furthermore, the writing log in the process of collecting file writing includes:

[0103] The existing agent technology is used to realize automatic collection of the writing logs generated during the process of writing the files in the task queue into the preset electronic device.

[0104] The agent technology is a computer system packaged in a certain environment that can achieve the designed purpose and can act flexibly and autonomously in the environment.

[0105] Furthermore, since the collected write logs may have a huge amount of data, they need to be stored first and then further analyzed.

[0106] In detail, the stored write log and the preset abnormal keyword set are input into a preset analysis algorithm to obtain analysis results, including:

[0107] The stored write log is analyzed and calculated using the following preset analysis algorithm to obtain the analysis result f:

[0108]

[0109]

[0110] Among them, Y i represents the ith abnormal keyword in the abnormal keyword set; n is the number of abnormal keywords in the abnormal keyword set, and W is the error factor; Write a log for the above.

[0111] Furthermore, when all stored write logs are calculated by the analysis algorithm, the analysis result is obtained, that is, the monitoring result, and the monitoring result is output.

[0112] The embodiment of the present invention obtains multiple files, stores the files, and obtains a file task list. The number of the obtained files may be huge and difficult to process in a short time, so they are first stored for further processing later; the files in the file task list are called, collected into a temporary file set, and the files in the temporary file set are classified according to keywords to obtain a classification task table. According to the keywords, the files are classified according to the contents contained in the files, so that the files can be read and written more quickly later; files with the same classification in the classification task table are selected and put into a preset task queue. When there is a vacancy in the task queue, files with the same classification as in the task queue are preferentially selected and put into the task queue until all the files with the same classification as in the classification task list are put into the task queue, and files with different classifications in the task queue are randomly selected and continue to be put into the task queue, and the files in the task queue are written into a preset electronic device. Different types of files have different writing configurations, and configuration changes need to be made when writing different types of files. The present invention puts files with similar contents together for file writing, thereby realizing fast file writing with low computing memory occupancy. Therefore, the file data writing method, device and computer-readable storage medium proposed in the present invention can achieve low-cost and high-efficiency writing of tens of millions of file data.

[0113] like Figure 2 The figure shows a functional module diagram of the file data writing device of the present invention.

[0114] The file data writing device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the file data writing device can include a file acquisition module 101, a keyword classification module 102, a file writing module 103 and an abnormality monitoring module 104. The module described in the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0115] In this embodiment, the functions of each module / unit are as follows:

[0116] The file acquisition module 101 is used to store the acquired multiple files and generate a file task list.

[0117] In detail, the file acquisition module 101 uses NAS (Network Attached Storage) technology to store the files uploaded by the user through the client.

[0118] The NAS is a device connected to the network and has a data storage function, so it is also called "network storage". It is a dedicated data storage server that is data-centric, completely separates storage devices from servers, and centrally manages data, thereby freeing up bandwidth, improving performance, and reducing costs. The embodiment of the present invention uses NAS to store the uploaded files, which can effectively improve storage efficiency.

[0119] When the storage is completed, the file task list is obtained.

[0120] The keyword classification module 102 is used to call the files in the file task list to collect them into a temporary file set, and classify the files in the temporary file set according to keywords.

[0121] Preferably, the number of files acquired by the embodiment of the present invention is very large, and the acquired files need to be classified so as to achieve faster writing of the files later.

[0122] In detail, the keyword classification module 102 calls multiple files in the file task list and aggregates the multiple files into a temporary file set;

[0123] Calculating the probability of each information word contained in the temporary file set by using a first probability formula;

[0124] According to the probability of the information words, the sentence probability of the information words being arranged into sentences according to different permutations and combinations is calculated using a second probability formula;

[0125] According to the sentence probabilities, k permutations and combinations are selected to obtain a sentence probability set;

[0126] Calculating the similarity between the sentences in the sentence probability set and the keywords in the preset keyword set to obtain a similarity result;

[0127] Each file in the file set is classified according to the similarity result.

[0128] The information words refer to words containing characteristic information in the document.

[0129] In detail, the embodiment of the present invention can use a pessimistic locking method to call multiple files in the file task list.

[0130] The pessimistic locking mode means that it is assumed that each time the file is called, there will be other processes modifying the file, so the file will be locked each time it is called, so that other processes cannot modify the file.

[0131] After the keyword classification module 102 calls multiple files in the file task list, the multiple files are collected to obtain the temporary file set.

[0132] Furthermore, the method of calculating the probability of the information word contained in each file in the temporary file set by using the first probability formula includes:

[0133] The probability P(C) of the information word appearing at the nth information word is calculated using the following first probability calculation formula:

[0134] P(C)=P(W n |W1,…W n-1 )

[0135] Among them, W n is the nth information word, P(W n |W1,…W n-1 ) is a conditional probability calculation, which indicates the probability of the nth information word appearing when the n-1th information word appears.

[0136] Furthermore, the calculating, based on the probability of the information words, using a second probability formula to calculate the sentence probability of the information words being arranged into sentences according to different permutations and combinations includes:

[0137] For any file S={W1, W2…W m}, calculate the information words in the file S according to the information word probability P(C) according to W1, W2…W m The sentence probability P(S) of a sequence arrangement and combination into a sentence is:

[0138] P(S)=P(W1)P(W2|W1)…P(W m |W m-n+1 ,…W m-1 )

[0139] Among them, W m represents the mth information word in the file S, P(W m |W m-n+1 ,…W m-1 ) is the conditional probability, indicating the Wth m-n+1 When the information word appears, W m Probability of occurrence.

[0140] Furthermore, the keyword classification module 102 calculates the sentence probability P(S) of all possible information words arranged into sentences, and selects k combinations of information words with the largest sentence probability according to the sentence probability P(S) to obtain a sentence probability set, which is represented by Q={q1,q2…q h}.

[0141] Furthermore, the calculating the similarity between the sentences in the sentence probability set and the keywords in the preset keyword set to obtain the similarity result includes:

[0142] The similarity calculation formula is used to calculate the similarity between the sentences in the sentence probability set and the preset keyword set G = {g1, g2…g t The similarity Sim of the most similar keyword in}:

[0143]

[0144] Among them, g i is any keyword in the keyword set; a is any sentence in the sentence probability set; count is a similarity operation; argmax is a maximum value operation; and t is the number of keywords in the keyword set.

[0145] When all the sentences in the sentence probability set have completed the above calculation, the keyword classification module 102 classifies each file in the file set according to the calculated similarity result, and collects the files with the same classification together to obtain the classification task table.

[0146] The file writing module 103 is used to select files from the temporary file set according to the classification and put them into a preset task queue, and write the files in the task queue into a preset electronic device.

[0147] In an embodiment of the present invention, since the amount of data contained in the files in the task queue is very large, if the files are to be written together, a large amount of computing memory will be occupied, and the writing will take a long time and be costly. Therefore, it is necessary to give priority to selecting files with the same classification in the classification task table and putting them into the task queue to achieve priority processing of file writing of the same type, reduce the computing memory usage, and reduce the cost of file writing.

[0148] The task queue is a queue with a preset length threshold. When the number of files in the task queue is less than the length threshold, files that meet the preset access principle are randomly selected from the classification task table and put into the task queue until the number of files in the task queue is equal to the length threshold;

[0149] When the number of files in the task queue is greater than or equal to the length threshold, the random selection of files that meet the admission principle from the classified task table and put into the task queue is suspended.

[0150] Furthermore, selecting files from the temporary file set according to the classification and putting them into a preset task queue includes:

[0151] When there is a vacancy in the task queue, determining whether there is a file in the temporary file set that has the same classification as the file in the task queue;

[0152] If there are files in the temporary file set that have the same classification as the files in the task queue, then arbitrarily select one of the files in the same classification; and if there are no files in the temporary file set that have the same classification as the files in the task queue, then select a file in the next classification;

[0153] Calculate the access value Z of the selected file i :

[0154] Z i =P i +d i +T i ,

[0155] Among them, P i The queue priority for the selected file; d i The absolute deadline for the selected document; i a write cycle for the selected file;

[0156] If the admission value Z i is less than or equal to an admission threshold X, then the selected file is refused to enter the task queue;

[0157] If the admission value Z i If the value is greater than the admission threshold X, the selected file is allowed to enter the task queue.

[0158] The abnormality monitoring module 104 is used to perform abnormality monitoring during the process of writing the files in the task queue to the preset electronic device, obtain monitoring results, and output the monitoring results.

[0159] In detail, performing abnormal monitoring in the process of writing the file in the task queue to the preset electronic device to obtain the monitoring result, and outputting the monitoring result includes:

[0160] Collecting write logs during the file writing process and storing the write logs;

[0161] Inputting the stored write log and the preset abnormal keyword set into a preset analysis algorithm to obtain analysis results, and aggregating the analysis results to obtain the monitoring results;

[0162] The monitoring result is outputted.

[0163] Furthermore, the writing log in the process of collecting file writing includes:

[0164] The existing agent technology is used to realize automatic collection of the writing logs generated during the process of writing the files in the task queue into the preset electronic device.

[0165] The agent technology is a computer system packaged in a certain environment that can achieve the designed purpose and can act flexibly and autonomously in the environment.

[0166] Furthermore, since the collected write logs may have a huge amount of data, they need to be stored first and then further analyzed.

[0167] In detail, the stored write log and the preset abnormal keyword set are input into a preset analysis algorithm to obtain analysis results, including:

[0168] The stored write log is analyzed and calculated using the following preset analysis algorithm to obtain the analysis result f:

[0169]

[0170]

[0171] Among them, Y i represents the ith abnormal keyword in the abnormal keyword set; n is the number of abnormal keywords in the abnormal keyword set, and W is the error factor; Write a log for the above.

[0172] Furthermore, when all stored write logs are calculated by the analysis algorithm, the analysis result is obtained, that is, the monitoring result, and the monitoring result is output.

[0173] like Figure 3 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing a method for writing file data according to the present invention.

[0174] The electronic device 1 may include a processor 10 , a memory 11 , and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10 , such as a file data writing program 12 .

[0175] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. The memory 11 may be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 may not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the file data writing program 12, etc., but also be used to temporarily store data that has been output or is to be output.

[0176] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing a file data writing program, etc.), and calls data stored in the memory 11, so as to execute various functions of the electronic device 1 and process data.

[0177] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0178] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0179] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0180] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0181] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0182] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0183] The file data writing program 12 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:

[0184] Store the acquired files and generate a file task list;

[0185] Calling the files in the file task list to collect them into a temporary file set, and classifying the files in the temporary file set according to keywords;

[0186] Selecting files from the temporary file set according to the classification and putting them into a preset task queue;

[0187] The file in the task queue is written to the preset electronic device. Specifically, the specific implementation method of the processor 10 for the above instruction can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0188] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0189] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0190] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0192] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0193] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0194] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for writing file data, characterized in that: The method comprises: Store the acquired files and generate a file task list; Calling the files in the file task list to collect them into a temporary file set, and classifying the files in the temporary file set according to keywords; Selecting files from the temporary file set according to the classification and putting them into a preset task queue; Writing the files in the task queue to a preset electronic device; The classifying of the files in the temporary file set according to keywords includes: using a first probability formula to calculate the probability of the information words contained in each file in the temporary file set; using a second probability formula to calculate the sentence probability of the information words arranged into sentences according to different permutations and combinations according to the probability of the information words; selecting k permutations and combinations according to the sentence probability to obtain a sentence probability set; calculating the similarity between the sentences in the sentence probability set and the keywords in the preset keyword set to obtain a similarity result; and classifying each file in the file set according to the similarity result; The selecting files from the temporary file set according to the classification and putting them into a preset task queue comprises: When there is a vacancy in the task queue, determining whether there is a file in the temporary file set that has the same classification as the file in the task queue; If there are files in the temporary file set that have the same classification as the files in the task queue, then arbitrarily select one of the files in the same classification; and if there are no files in the temporary file set that have the same classification as the files in the task queue, then select a file in the next classification; Calculate the access value Z of the selected file i : Z i =P i +d i +T i , Among them, P i The queue priority for the selected file; d i The absolute deadline for the selected document; i a write cycle for the selected file; If the admission value Z i is less than or equal to an admission threshold X, then the selected file is refused to enter the task queue; If the admission value Z i If the value is greater than the admission threshold X, the selected file is allowed to enter the task queue.

2. The file data writing method according to claim 1, characterized in that: The using the first probability formula to calculate the probability of the information word contained in each file in the temporary file set includes: The probability P(C) of the occurrence of the nth information word is calculated using the following first probability calculation formula: P(C)=P(W n |W1,…W n-1 ) Among them, W n is the nth information word, P(W n |W1,…W n-1 ) is the conditional probability calculation, which indicates the probability of the nth information word appearing when the n-1th information word appears.

3. The file data writing method according to claim 1, characterized in that: The method of calculating the sentence probability of the information words being arranged into sentences according to different permutations and combinations by using the second probability formula includes: The information words in the file S are calculated by using the following second probability calculation formula: m The sentence probability P(S) of a sequence arrangement and combination into a sentence is: P(S)=P(W1)P(W2|W1)…P(W m |W m-n+1 ,…W m-1 ) Among them, W m represents the mth information word in the file, P(W m |W m-n+1 ,…W m-1 ) is the conditional probability, indicating the Wth m-n+1 When the information word appears, W m Probability of occurrence.

4. The file data writing method according to claim 1, characterized in that: The calculating the similarity between the sentences in the sentence probability set and the keywords in the preset keyword set to obtain a similarity result includes: The similarity Sim between the sentences in the sentence probability set and the keywords in the preset keyword set is calculated using the following similarity calculation formula: Among them, g i The preset keyword set G = {g1, g2...g t } any keyword; q a is any sentence in the sentence probability set; count is a similarity operation; argmax is a maximum operation; and t is the number of keywords in the keyword set.

5. The file data writing method according to any one of claims 1 to 4, characterized in that: The method further comprises: The process of writing the files in the task queue into the preset electronic device is monitored for abnormalities, a monitoring result is obtained, and the monitoring result is output.

6. A file data writing device, used to implement the file data writing method according to any one of claims 1 to 5, characterized in that: The device comprises: The file acquisition module is used to store the acquired multiple files and generate a file task list; A keyword classification module, used for calling the files in the file task list to collect them into a temporary file set, and classifying the files in the temporary file set according to keywords to obtain a classification task table; A file writing module, used for selecting files with the same classification in the classification task table and putting them into a preset task queue, and writing the files in the task queue into a preset electronic device; The abnormality monitoring module is used to perform abnormality monitoring in the process of writing the files in the task queue to the preset electronic device, obtain the monitoring result, and output the monitoring result.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the file data writing method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the file data writing method according to any one of claims 1 to 5 is implemented.

Images