Method and device for processing small file data concurrent read and write based on relational database

Abstract

The application provides a small file data concurrent read-write method and device based on a relational database, relates to the field of data file storage, and can be applied to the financial field and other fields.The method comprises the following steps: obtaining business data and image file data according to received deposit message data analysis; generating a batch entity through the business data, positioning a plurality of image small files in the image file data to a corresponding large file or a database BLOB field partition for storage and recording deposit information of the image small files; generating corresponding image attribute information according to the deposit information and image description information of the image small files, and generating a corresponding image attribute entity according to the image attribute information; and after storing the image attribute entity and the batch entity in association, generating a write result through the batch entity.

Description

Technical Field

[0001] This application relates to the field of data file storage and can be applied to the financial field and other fields, particularly to a method and apparatus for concurrent reading and writing of small file data based on a relational database. Background Technology

[0002] In the banking system, tens of thousands of transactions occur daily. The paper receipts generated from these transactions are typically scanned into image files (hereinafter referred to as scanned copies) and stored in the background storage service. Due to the massive daily transaction volume, each transaction generates several scanned copies, resulting in a huge demand for file storage over time. To save storage space, each scanned copy is compressed into small files ranging from 20KB to 2MB. In terms of the number of files, these small files number in the billions or even tens of billions, and in terms of space occupied, they occupy thousands of terabytes of space. The entirety of such files in terms of size and quantity is referred to as massive small files.

[0003] The sheer volume of small files requires banks to have robust storage management capabilities. Furthermore, these files are continuously accessed by hundreds or even thousands of tellers simultaneously, creating high concurrency and severely impacting system performance. Each teller's access to these files follows certain principles, typically involving accessing several small files as a set, performing operations such as adding, deleting, modifying, and querying elements within the set. For example, a customer's information can form a set: when adding a new customer, the teller scans the customer's identification documents, account opening contract, and other materials, creating small files, and submits them to the storage service in the same set; subsequently, when the customer makes a transaction, the teller retrieves the corresponding file set from the storage service to verify the customer's identity; when the customer's information changes, the teller scans the changed portions to update the corresponding file set; and after the customer closes their account, the teller can delete the corresponding file set. For example, scanned copies of vouchers generated from a loan transaction can form a file set. When a customer applies for a loan, the teller scans the customer's information and adds a new file set. When the loan is approved, the approving teller stamps the materials and needs to scan the stamped materials again, then update the file set with the new scanned copies. After the loan application is approved and the loan is disbursed, the risk monitoring teller can query the file set to verify the loan status and monitor for any risks.

[0004] The requirements for handling massive amounts of small files in banking systems are roughly as described above: first, they need enormous storage space management capabilities; second, they need to provide massive data throughput capabilities; and third, they need to provide the ability to access small files in sets, allowing for adding, modifying, querying, and deleting files within those sets. Currently, banking systems use a common method to meet these requirements, directly storing massive amounts of small files on the operating system. However, typical operating systems have limited support for massive amounts of small files, and as the number of files increases, the following three drawbacks gradually become apparent:

[0005] First, the number of sub-files in a single file directory should not be too large, as this will affect access performance. For example, if you need to retrieve the set of files you need from a directory containing 20,000 sub-files, the response time will be relatively long.

[0006] Secondly, there is a limit to the number of files that an operating system can open. If an operating system opens too many files, it will experience slow response, denial of service, or even system crash.

[0007] Third, when concurrently reading and writing files in the operating system, the performance of file reading and writing is extremely low. The total bandwidth of all processes reading or writing can only reach a maximum of about 1MB / s, even if the disk bandwidth is 100MB / s. If a small file is 20KB in size, and 10 processes simultaneously write different files to a single operating system, each process can only write a maximum of 5 files per second.

[0008] As socio-economic activities increase, banks' business volume grows larger, and the demands on the efficiency of bank transaction processing become increasingly higher. If the methods mentioned above are still used to process massive amounts of small files, it will cause slow file access, which in turn will slow down transactions and seriously affect system efficiency. Summary of the Invention

[0009] The purpose of this application is to provide a method and apparatus for concurrent reading and writing of small file data based on relational databases, so as to meet the needs of the above-mentioned banking system for processing massive small files, and overcome the shortcomings of existing operating systems in supporting concurrent reading and writing of massive small file data.

[0010] To achieve the above objectives, this application provides a method for concurrent read / write of small file data based on a relational database. The method includes: parsing received storage message data to obtain business data and image file data; generating a batch entity from the business data; using the batch entity to locate multiple small image files in the image file data to their corresponding large files or database BLOB (Binary Large Object) field partitions for storage and recording the storage information of the small image files; generating corresponding image attribute information based on the storage information and the image description information of the small image files; generating a corresponding image attribute entity based on the image attribute information; associating and storing the image attribute entity with the batch entity; and generating a write result using the batch entity.

[0011] In the above method for concurrent reading and writing of small file data based on relational database, optionally, the process of parsing the received storage message data to obtain business data and image file data includes: obtaining business data by encoding conversion based on the storage message data, and decompressing and restoring the image small file in the storage message data to obtain a binary data stream; and obtaining image file data based on the binary data stream.

[0012] In the above method for concurrent read and write of small file data based on relational database, optionally, the process of locating multiple small image files in the image file data to the corresponding sequence number of the large file partition and storing and recording the storage information of the small image files through the batch entity includes: associating multiple small image files of the same set in the image file data through the batch entity; obtaining the location information of the corresponding large file according to the batch entity; storing the multiple associated small image files into the large file in the form of binary data streams through the location information and recording the corresponding storage information.

[0013] In the above method for concurrent reading and writing of small file data based on relational databases, optionally, the storage information includes storage path, offset and content length.

[0014] In the above method for concurrent reading and writing of small file data based on relational database, optionally, the method further includes: constructing one or more large files with corresponding name information on each file storage system according to the number of threads based on a preset correspondence rule between batch entities and large file name information; wherein each large file is written to by one thread.

[0015] In the above-described method for concurrent reading and writing of small file data based on relational database, optionally, the method further includes: after any batch of small image files is stored in the large file, detecting the data storage volume in the current large file; when the data storage volume is higher than a preset threshold, closing the write channel of the current large file and reconstructing a large file to complete the subsequent batch of small image file writing operations.

[0016] In the above method for concurrent read / write of small file data based on relational database, optionally, the method further includes: parsing the received query message data to obtain a batch entity, obtaining the corresponding business data based on the batch entity; locating the image attribute entity of the corresponding batch based on the batch entity, obtaining the corresponding image attribute information through the image attribute entity; obtaining the small image file from the corresponding large file or database BLOB field partition based on the image attribute information; and generating a query result based on the business data, the image attribute information, and the small image file.

[0017] This application also provides a device for concurrent read / write of small file data based on a relational database. The device includes a data receiving unit, a processing module, and a data sending unit. The data receiving unit is used to parse business data and image file data based on the received storage message data. The processing module is used to generate a batch entity through the business data, and to locate multiple small image files in the image file data into corresponding large files or database BLOB field partitions for storage and recording the storage information of the small image files through the batch entity. Furthermore, it generates corresponding image attribute information based on the storage information and the image description information of the small image files, and generates corresponding image attribute entities based on the image attribute information. After associating and storing the image attribute entities with the batch entities, it generates a write result through the batch entities. The data sending unit is used to feed back the write result to the sender of the storage message data.

[0018] In the above-described concurrent read / write device for processing small file data based on a relational database, optionally, the data receiving unit is further configured to parse the received query message data to obtain a batch entity; the processing module is further configured to obtain the corresponding business data based on the batch entity; locate the image attribute entity of the corresponding batch based on the batch entity, and obtain the corresponding image attribute information through the image attribute entity; obtain the image small file from the corresponding large file or database BLOB field partition based on the image attribute information; generate a query result based on the business data, the image attribute information, and the image small file; the data sending unit is further configured to feed back the query result to the query message data sender.

[0019] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0020] This application also provides a computer-readable storage medium storing a computer program that performs the above-described methods.

[0021] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0022] The beneficial technical effects of this application are as follows: it solves the shortcomings of using the operating system to store massive amounts of small files and meets the needs of the banking system for processing massive amounts of small files. At the same time, it improves the efficiency of concurrent access to massive amounts of small files by a large number of users. When using a regular file system to store massive amounts of small files, the maximum disk read / write bandwidth is 1MB / s, and the maximum number of files read / written is 50 / s. After applying this invention, the maximum disk read / write bandwidth is increased to 20MB / s, and the maximum number of files read / written reaches 400 / s. This performance improvement ensures that concurrent read / write of massive amounts of small files will no longer be a bottleneck for the banking system within a foreseeable timeframe. Furthermore, after using data files to store massive amounts of small files, the situation of the operating system running out of handles no longer occurs, eliminating the phenomenon of operating system denial of service. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. In the drawings:

[0024] Figure 1 This is a flowchart illustrating a method for concurrent reading and writing of small file data based on a relational database, provided in an embodiment of this application.

[0025] Figure 2 This is a schematic diagram of a small file storage process provided in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram illustrating the relationship between batches, image attributes, and image content provided in an embodiment of this application.

[0027] Figure 4 This is a schematic diagram of container logic provided in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram illustrating the principle of large file construction provided in an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the process for adding a large file according to an embodiment of this application;

[0030] Figure 7 This is a schematic diagram of a query process provided in an embodiment of this application;

[0031] Figure 8 This is a schematic diagram of the structure of a device for concurrent reading and writing of small file data based on a relational database, provided in an embodiment of this application.

[0032] Figure 9 This is a schematic diagram illustrating the application principle of a device for concurrent reading and writing of small file data based on a relational database, provided in an embodiment of this application.

[0033] Figure 10 This is a schematic diagram of a new batch process provided in an embodiment of this application;

[0034] Figure 11 This is a schematic diagram of a batch query process provided in an embodiment of this application;

[0035] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0036] The following will describe in detail the implementation methods of this application with reference to the accompanying drawings and embodiments, so as to fully understand how this application uses technical means to solve technical problems and achieve technical effects, and to implement it accordingly. It should be noted that, as long as there is no conflict, the various embodiments and features in each embodiment of this application can be combined with each other, and the resulting technical solutions are all within the protection scope of this application.

[0037] Furthermore, the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0038] Research has shown that transmitting a certain number of small files in a distributed manner takes significantly longer than transmitting them in a package. This phenomenon can be explained by the principles of file read / write operations: the operating system needs time to open a file (primarily for retrieving the file and preparing the file buffer), time to read and write file data, and time to close the file (for closing the file buffer). Distributed and packaged transmissions consume almost the same amount of time for reading and writing file data because the total file size is the same. However, distributed transmission requires opening and closing files far more times than packaged transmission. In distributed transmission, the number of times a file is opened and closed corresponds to the number of small files, while in packaged transmission, only one file needs to be opened and closed. Therefore, packaged transmission is more efficient than distributed transmission.

[0039] Inspired by the high efficiency of packet transmission, please refer to Figure 1 As shown, this application provides a method for concurrent read and write of small file data based on a relational database, the method comprising:

[0040] S101 parses the received stored message data to obtain service data and image file data;

[0041] S102 generates a batch entity through the business data, and uses the batch entity to locate multiple small image files in the image file data to the corresponding large file or database BLOB field partition for storage and records the storage information of the small image files.

[0042] S103 generates corresponding image attribute information based on the stored information and the image description information of the image file, and generates corresponding image attribute entities based on the image attribute information;

[0043] S104 After associating and storing the image attribute entity with the batch entity, a write result is generated through the batch entity.

[0044] Therefore, by associating and concatenating several small files belonging to the same set, the contents of a large number of small files are stored in a small number of large files that are opened for extended periods. This means storing different small file contents in different segments of a large file, reducing the number of times files are opened and closed, and ultimately improving the processing efficiency of massive numbers of small files. In practice, using this method to store the contents of small files requires using a relational database to record the offset and length of the small files within the large file. Furthermore, due to file size limitations and file system size limitations, multiple file systems must be mounted in the operating system, with each file system storing multiple large files, to accommodate the massive number of small files (thousands of terabytes). Specific methods will be described in detail in subsequent embodiments.

[0045] In the above embodiments, obtaining business data and image file data by parsing the received stored message data may include: obtaining business data through code conversion based on the stored message data, and decompressing and restoring the small image files in the stored message data to obtain a binary data stream; and obtaining image file data based on the binary data stream. In practice, this process mainly involves converting the message package submitted by the user client into a code format, restoring it to business data, and restoring the small image files in the message package to a binary data stream. Specific code format conversion and restoration operations can be implemented using techniques in this field, and will not be detailed further in this application.

[0046] Please refer to Figure 2As shown, in one embodiment of this application, the process of locating multiple small image files in the image file data to the corresponding sequence number of the large file partition and recording the storage information of the small image files through the batch entity includes:

[0047] S201 associates multiple small image files of the same set in the image file data through the batch entity;

[0048] S202 obtains the location information of the corresponding large file based on the batch entity positioning, and stores the multiple associated small image files into the large file in the form of binary data stream using the location information and records the corresponding storage information.

[0049] The storage information includes the storage path, offset, and content length.

[0050] Specifically, in practice, a collection of files can be defined as a batch, and the relationship between a batch and its individual image files is one-to-many. The batch entity model has attributes such as batch ID, creation date, and business information. Each batch has a unique batch ID, assigned using a serial number; the creation date records the batch's creation date in the format YYYYMMDD, where YYYY represents a 4-digit year, MM represents a 2-digit month (padded with leading zeros if necessary), and DD represents a 2-digit day (padded with leading zeros if necessary), for a total of 8 digits; the business information records the information needed by the batch during business processing. Each individual image file needs to record business information related to that image and also save its file content. Attributes such as business information, save path (the path of the larger file containing the smaller file), offset (the offset of the smaller file within the larger file), and content length can be stored in the database, while the image file content can be stored in the larger file. Therefore, two models need to be designed: an image attribute entity and an image content entity, used to record image attributes and file content respectively.

[0051] The image attribute entity model has attributes such as image ID, batch ID, business information, save path, offset, and content length. Each image file has a unique image ID, assigned using a serial number. Image attribute entities within the same batch have the same batch ID; the batch ID value is the batch ID of the corresponding batch entity. Business information records information needed by the image file during business processing; this information is different for each image file and also different from the business information of the corresponding batch entity. The image content entity model has image content attributes; image content is binary data. The relationship between batch entities, image attribute entities, and image content entities can be found in [reference needed]. Figure 3As shown, the relationship between batch entities and image attribute entities is one-to-many, while the relationship between image attribute entities and image content entities is one-to-one. One batch entity corresponds to multiple image attribute entities, and one image attribute entity corresponds to one image content entity. Using this data model, batches and images can be associated using batch IDs.

[0052] In one embodiment of this application, the method further includes: constructing one or more large files with corresponding name information on each file storage system according to the number of threads based on a preset correspondence rule between batch entities and large file name information; wherein each large file is written with data by one thread.

[0053] For details, please refer to Figure 4 As shown, in practice, different entities can be stored using different containers, such as batch containers, image attribute containers, and image content containers. Batch containers store batch entities and are implemented using database tables; image attribute containers store image attribute entities and are also implemented using database tables; image content containers store image content entities and are implemented using different large files stored on different file systems. Please refer to [further details]. Figure 5 As shown, the batch container and image attribute container are implemented using database tables; the image content container is implemented using multiple large files distributed across multiple file systems. The distribution of these large files can be specified using different algorithms, such as by the hash value of the batch number or by the batch's creation date. For example, in... Figure 5 The program specifies the names and storage locations of multiple large files by date. Each large file can be named according to the rule "large file YYYYMMDD_n", where YYYYMMDD is the batch generation date, YYYY represents a 4-digit year, MM represents a 2-digit month (padded with leading zeros if necessary), DD represents a 2-digit day (padded with leading zeros if necessary), and n is used to distinguish multiple large files generated on the same day. Each "large file YYYYMMDD_n" stores the image content of the batch generated within that date. For example, "large file 20120309_2" is used to store the image content from 0:00 on March 9, 2012 to 24:00 on March 9, 2012. This file is the second large file generated on that day.

[0054] Please refer to Figure 6 As shown, in one embodiment of this application, the method further includes:

[0055] S601 After any batch of small image files is stored in the large file, the data storage volume in the current large file is detected;

[0056] S602 When the amount of data stored is higher than a preset threshold, the write channel of the current large file is closed and a new large file is reconstructed to complete the subsequent batch of image small file write operations.

[0057] Specifically, in practice, when a large file is about to exceed its size limit, it needs to be closed, and a new large file needs to be opened for writing. Therefore, there are multiple large files per day. A large file can only be written to by one thread at a time; otherwise, the image content will be corrupted. To improve concurrency efficiency, when there is high user concurrency, multiple large files need to be opened and written to in parallel, resulting in multiple large files per day. Similarly, the implementation of using database BLOB fields to store image entity content is similar. Most relational databases currently support LOB type fields. LOB is short for Large Object. LOB is further divided into BLOB and CLOB, which are Binary LOB and Character LOB respectively. BLOB can store binary data of unlimited length. Image file content is also binary data, therefore BLOB can store image content entities. The implementation idea of ​​this solution is similar to that of using large files, so it will not be detailed here. Using BLOB fields in a relational database to store image content entities has the following advantages over using large files: 1. It can leverage the database's concurrent processing capabilities to increase user concurrency; 2. It can leverage the database's file management capabilities to reduce user intervention in file management; 3. It can leverage the database's BLOB field management capabilities to simplify the workload involved in location tracking.

[0058] Please refer to Figure 7 As shown, in one embodiment of this application, the method further includes:

[0059] S701 parses the received query message data to obtain the batch entity, and obtains the corresponding business data based on the batch entity;

[0060] S702 locates the image attribute entity corresponding to the batch based on the batch entity, and obtains the corresponding image attribute information by querying the image attribute entity;

[0061] S703 obtains the small image file from the corresponding large file or database BLOB field partition based on the image attribute information;

[0062] S704 generates query results based on the business data, the image attribute information, and the image file.

[0063] In this embodiment, based on the aforementioned storage operation, users can perform subsequent quick queries. The specific query principles and logic can be referred to the aforementioned embodiments, and will not be described in detail here.

[0064] Please refer to Figure 8As shown, this application also provides a device for concurrent read / write of small file data based on a relational database. The device includes a data receiving unit, a processing module, and a data sending unit. The data receiving unit is used to parse business data and image file data based on received storage message data. The processing module is used to generate a batch entity through the business data, and to locate multiple small image files in the image file data into corresponding large files or database BLOB field partitions for storage and recording the storage information of the small image files through the batch entity. Furthermore, it generates corresponding image attribute information based on the storage information and the image description information of the small image files, and generates a corresponding image attribute entity based on the image attribute information. After associating and storing the image attribute entity with the batch entity, it generates a write result through the batch entity. The data sending unit is used to feed back the write result to the sender of the storage message data.

[0065] Furthermore, the data receiving unit is also used to parse the received query message data to obtain batch entities; the processing module is also used to obtain the corresponding business data based on the batch entities; locate the image attribute entities of the corresponding batch based on the batch entities, and obtain the corresponding image attribute information through the image attribute entities; obtain the image small files from the corresponding large file or database BLOB field partition based on the image attribute information; generate query results based on the business data, the image attribute information, and the image small files; the data sending unit is also used to feed back the query results to the query message data sender.

[0066] For details, please refer to Figure 9 As shown, in practical operation, the concurrent read / write device for small file data processing based on a relational database consists of a data receiving unit 100, a data sending unit 200, a small file association unit 300, a small file location unit 400, and a small file read / write unit 500. The data receiving unit 100 receives business data and image files submitted by the user; the data sending unit 200 sends the processing results or the image files to be retrieved to the user; the small file association unit 300 associates image files within the same set; the small file location unit 400 locates the image file content within its larger file; and the small file read / write unit 500 handles CRUD requests for the small file set. Specifically, the data receiving unit 100 receives compressed message packages submitted by the user client, performs encoding conversion, restores the data to its original format, and converts the small image files within the compressed message package back into binary data streams. The data sending unit 200 forms the processing results or the image files to be retrieved into a message and then sends it to the user. The small file association unit 300, the small file positioning unit 400, and the small file read / write unit 500 need to work together to associate, locate, and access image files in the same set.

[0067] Based on the aforementioned structure of a device for concurrent read / write of small file data using a relational database, a new batch process can be added as follows: Figure 10 As shown:

[0068] The data receiving unit 100 receives the service data and image file content sent by the user;

[0069] The small file association unit 300 generates a batch ID and generates an image ID for each image in the batch, forming a batch entity and an image attribute entity. These two entities are saved to the database to form an association between the batch and the image.

[0070] The small file association unit 300 splices together the batch of small files. The small file positioning unit 400 locates the path of the large file or the database BLOB field partition according to the batch's generation date and the large file sequence number of the day. Then, the small file read / write unit 500 stores the spliced ​​image content entity into the corresponding large file or database BLOB field. The small file association unit 300 updates the image attribute entity's save path, offset, content length, and other attributes.

[0071] The data sending unit 200 sends the newly added batch ID to the user.

[0072] For batch query procedures, please refer to [link / reference]. Figure 11 As shown:

[0073] Data receiving unit 100 receives the batch ID that the user wants to query;

[0074] The small file association unit 300 obtains the corresponding batch entity and image attribute entity from the database based on the batch ID;

[0075] The small file positioning unit 400 locates the position of the partition of the large file or database BLOB field according to the save path in the image attribute entity, and the small file reading and writing unit 500 obtains the image content entity from the large file or database BLOB field according to the offset, content length and other attributes in the image attribute entity.

[0076] The data sending unit 200 forms a message from the batch entity, image attribute entity, and image content entity and sends it to the user.

[0077] The beneficial technical effects of this application are as follows: it solves the shortcomings of using the operating system to store massive amounts of small files and meets the needs of the banking system for processing massive amounts of small files. At the same time, it improves the efficiency of concurrent access to massive amounts of small files by a large number of users. When using a regular file system to store massive amounts of small files, the maximum disk read / write bandwidth is 1MB / s, and the maximum number of files read / written is 50 / s. After applying this invention, the maximum disk read / write bandwidth is increased to 20MB / s, and the maximum number of files read / written reaches 400 / s. This performance improvement ensures that concurrent read / write of massive amounts of small files will no longer be a bottleneck for the banking system within a foreseeable timeframe. Furthermore, after using data files to store massive amounts of small files, the situation of the operating system running out of handles no longer occurs, eliminating the phenomenon of operating system denial of service.

[0078] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0079] This application also provides a computer-readable storage medium storing a computer program that performs the above-described methods.

[0080] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0081] like Figure 12 As shown, the electronic device 600 may also include: a communication module 110, an input unit 120, an audio processing unit 130, a display 160, and a power supply 170. It is worth noting that the electronic device 600 does not necessarily need to include these components. Figure 12 All components shown; in addition, the electronic device 600 may also include Figure 12 For components not shown, please refer to existing technology.

[0082] like Figure 12 As shown, the central processing unit 100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device. The central processing unit 100 receives inputs and controls the operation of various components of the electronic device 600.

[0083] The memory 140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 100 may execute the program stored in the memory 140 to perform information storage or processing, etc.

[0084] Input unit 120 provides input to central processing unit 100. Input unit 120 may be, for example, a keypad or touch input device. Power supply 170 provides power to electronic device 600. Display 160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0085] The memory 140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 140 can also be some other type of device. The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application / function storage unit 142 for storing application programs and function programs or processes for executing the operation of the electronic device 600 via the central processing unit 100.

[0086] The memory 140 may also include a data storage unit 143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 144 of the memory 140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0087] The communication module 110 is a transmitter / receiver 110 that transmits and receives signals via antenna 111. The communication module (transmitter / receiver) 110 is coupled to the central processing unit 100 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0088] Based on different communication technologies, multiple communication modules 110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and receive audio input from the microphone 132, thereby enabling typical telecommunications functions. The audio processor 130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 130 is coupled to a central processing unit 100, enabling on-device recording via the microphone 132 and on-device playback of stored audio via the speaker 131.

[0089] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for concurrent read and write of small file data based on a relational database, characterized in that, The method includes: The business data and image file data are obtained by parsing the received stored message data; A batch entity is generated from the business data. The batch entity is used to locate multiple small image files in the image file data to their corresponding large files or database BLOB field partitions for storage and to record the storage information of the small image files. The batch entity has a batch ID and a generation date. The batch ID is assigned using a serial number. Based on the stored information and the image description information of the image file, generate corresponding image attribute information, and generate corresponding image attribute entities based on the image attribute information; After associating and storing the image attribute entity with the batch entity, the write result is generated through the batch entity; The method further includes: constructing one or more large files with corresponding name information on each file storage system according to the number of threads based on a preset correspondence rule between batch entities and large file name information; wherein, each large file is written with data by one thread; wherein, the large files are named according to the batch generation date, and each large file stores the image content of the batch generated within that date; Once any batch of small image files has been stored in the large file, the amount of data stored in the current large file is detected. When the amount of data stored exceeds a preset threshold, the write channel of the current large file is closed and a new large file is reconstructed to complete the subsequent batch of image small file write operations. The process of locating multiple small image files in the image file data to their corresponding large file partitions and storing them using the batch entity, and recording the storage information of the small image files, includes: associating multiple small image files of the same set in the image file data using the batch entity; obtaining the location information of the corresponding large file based on the batch entity location; storing the associated multiple small image files into the large file in the form of a binary data stream using the location information and recording the corresponding storage information.

2. The method for concurrent read / write of small file data based on relational databases according to claim 1, characterized in that, The business data and image file data obtained by parsing the received stored message data include: Based on the stored message data, the business data is obtained through code conversion, and the image small files in the stored message data are decompressed and restored to obtain a binary data stream; Image file data is obtained from the binary data stream.

3. The method for concurrent read / write of small file data based on relational databases according to claim 1, characterized in that, The storage information includes the storage path, offset, and content length.

4. The method for concurrent read / write of small file data based on relational databases according to claim 1, characterized in that, The method further includes: The batch entity is obtained by parsing the received query message data, and the corresponding business data is obtained based on the batch entity. Based on the batch entity, locate the corresponding batch of image attribute entities, and obtain the corresponding image attribute information by querying the image attribute entities; Based on the image attribute information, obtain the image small file from the corresponding large file or database BLOB field partition; Query results are generated based on the business data, the image attribute information, and the image file.

5. A device for concurrent read / write of small file data based on a relational database, characterized in that, The device includes a data receiving unit, a processing module, and a data sending unit; The data receiving unit is used to parse the received stored message data to obtain service data and image file data; The processing module is used to generate batch entities from the business data, locate multiple small image files in the image file data to their corresponding large files or database BLOB field partitions using the batch entities, and record the storage information of the small image files; and generate corresponding image attribute information based on the storage information and the image description information of the small image files, and generate corresponding image attribute entities based on the image attribute information; after associating and storing the image attribute entities with the batch entities, a write result is generated using the batch entities; wherein, the batch entity has a batch ID and a generation date, and the batch ID is assigned using a serial number; The data sending unit is used to send the writing result back to the storage message data sender; The device also includes a large file construction unit and a data writing unit; The large file construction unit is used to construct one or more large files with corresponding names on each file storage system according to the preset correspondence rules between batch entities and large file name information, based on the number of threads; wherein, each large file is written with data by one thread; wherein, the large files are named according to the batch generation date, and each large file stores the image content of the batch generated within that date; The data writing unit is used to detect the data storage volume in the current large file after any batch of small image files is stored in the large file; when the data storage volume is higher than a preset threshold, the writing channel of the current large file is closed and a new large file is reconstructed to complete the writing operation of subsequent batches of small image files. Specifically, the processing module is used to associate multiple small image files of the same set in the image file data through the batch entity; obtain the location information of the corresponding large file according to the batch entity location; and store the multiple associated small image files into the large file in the form of binary data streams through the location information and record the corresponding storage information.

6. The device for concurrent read / write of small file data based on relational database according to claim 5, characterized in that, The data receiving unit is also used to parse the received query message data to obtain batch entities; The processing module is also used to obtain the corresponding business data based on the batch entity; locate the image attribute entity of the corresponding batch based on the batch entity, and obtain the corresponding image attribute information by querying the image attribute entity; and obtain the image small file from the corresponding large file or database BLOB field partition based on the image attribute information. Query results are generated based on the business data, the image attribute information, and the image file. The data sending unit is also used to send the query results back to the query message data sender.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that enables a computer to execute the method of any one of claims 1 to 4.

9. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 4.

Images