MongoDB-based distributed picture caching method and system

Abstract

The invention provides a MongoDB-based distributed picture caching method and system, and the method comprises the steps: writing picture data uploaded by a picture uploading request into a local SSD in a server node, generating an object ID and picture metadata, and storing the object ID and the picture metadata into a MongoDB database; migrating the picture data in the local SSD into the SATA hard disk in batches, and updating the migration state; performing data cleaning on the local SSD based on the state of the picture data in the local SSD, and synchronously updating the picture metadata in the MongoDB database; and analyzing the picture object ID corresponding to the picture reading request, querying the memory cache according to the picture object ID, and if the picture object ID does not exist in the memory cache, reading the corresponding file from the local SSD or SATA hard disk according to the migration state queried in the MongoDB database. According to the method, the three-layer caching technology is utilized, and the concurrent reading performance of the pictures is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly relates to a method and system for distributed caching of pictures based on MongoDB. Background Art

[0002] With the development of technologies such as AI, big data, and cloud computing, the use of urban-level intelligent video clouds has gradually matured. The intelligent video cloud needs to connect a large number of devices from different manufacturers, such as cameras, NVRs, and various edge computing gateways.

[0003] These devices generate a large number of captured pictures every day, such as face, vehicle, and human body captured pictures. Moreover, the storage periods of different types of pictures may also be different in different business scenarios. For example, face pictures are stored for 30 days, and vehicle pictures are stored for 7 days. After receiving these pictures, the video cloud platform not only needs to perform local storage but also needs to perform a large amount of processing and analysis work based on these pictures. For example, these pictures are re-structured again to extract face features, or these pictures are shared with other platforms through the 1400 protocol. In this process, since it is faced with the reception, storage, and analysis of a large amount of pictures at the urban scale level, pictures need to be frequently read during data sharing. How to withstand so many concurrent accesses and reads, and since the sizes of the received pictures are also different, there are large pictures and small pictures, and some small pictures may only be stored for 7 days. How to optimize the concurrent read performance for small pictures has become an urgent problem to be solved currently. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to provide a method and system for distributed caching of pictures based on MongoDB to at least solve the deficiencies in the above technologies.

[0005] The present invention proposes a method for distributed caching of pictures based on MongoDB, including: Initializing a picture object table and its corresponding picture index, and allocating the received picture upload requests to preset server nodes; Writing the picture data uploaded by the picture upload request into the local SSD in the server node, and adding business attributes to the id field of the unique identifier of the picture data to generate a corresponding object ID and picture metadata and storing them in the MongoDB database; Based on the storage period and the size of the picture data, migrating the picture data in the local SSD to the SATA hard disk in batches, and updating the migration status; Performing data cleaning on the local SSD based on the status of the picture data in the local SSD, and synchronously updating the picture metadata in the MongoDB database; When a picture reading request is received, parse out the picture object ID corresponding to the picture reading request, and query the memory cache according to the picture object ID. If the picture object ID does not exist in the memory cache, query the corresponding migration status from the MongoDB database, and read the corresponding file from the local SSD or the SATA hard disk according to the migration status.

[0006] Further, the picture object table includes an object ID, a device ID, a node ID, a timestamp, a storage period, a picture size, a picture type, a picture location, a migration status, an expiration time, and a redundancy rule. The picture index includes a first composite index and a second composite index.

[0007] Further, the steps of writing the picture data uploaded by the picture upload request into the local SSD in the server node, and adding business attributes to the id field of the unique identifier of the picture data to generate a corresponding object ID and picture metadata and storing them in the MongoDB database include: In the server node, store the Http picture form data of the picture data of the picture upload request in the local SSD. Generate a corresponding object ID based on the current time, the current device ID, the current machine node information, and a random number of the picture data, and store the picture metadata of the picture data in the MongoDB database.

[0008] Further, the steps of migrating the picture data in the local SSD to the SATA hard disk in batches based on the storage period and the picture data size, and updating the migration status include: Read the minimum migration file parameter and the delayed migration parameter, and generate the maximum migration object ID according to the delayed migration parameter. Scan the MongoDB database, compare the picture data in the local SSD with the minimum migration file parameter. If the picture data in the local SSD is smaller than the minimum migration file parameter, do not migrate the picture data in the local SSD. If the picture data in the local SSD is larger than the minimum migration file parameter, obtain the picture metadata of the picture data in the local SSD, obtain a corresponding list of pictures to be migrated from the picture data in the local SSD based on a preset migration rule and the picture metadata, and migrate the corresponding picture data to the SATA hard disk according to the list of pictures to be migrated, and update the corresponding migration status.

[0009] Further, the steps of performing data cleaning on the local SSD based on the status of the picture data in the local SSD and synchronously updating the picture metadata in the MongoDB database include: Read the maximum capacity limit and the maximum cleaning capacity limit of the local SSD, and generate a maximum cleaning object ID according to the current time; If the current capacity of the local SSD is greater than the maximum capacity limit, obtain a list of pictures to be cleaned from the local SSD based on a preset scanning rule, and perform data cleaning on the local SSD based on the list of pictures to be cleaned; After the data cleaning of the local SSD is completed, accumulate and count the current cleaning capacity, and determine whether the current cleaning capacity is greater than the maximum cleaning capacity limit. If so, exit the data cleaning.

[0010] The present invention also proposes a picture distributed cache system based on MongoDB, including: An initialization module, configured to initialize a picture object table and its corresponding picture index, and allocate the received picture upload request to a preset server node; A data storage module, configured to write the picture data uploaded by the picture upload request into the local SSD in the server node, and add a service attribute to the id field of the unique identifier of the picture data to generate a corresponding object ID and picture metadata, and store them in the MongoDB database; A data migration module, configured to migrate the picture data in the local SSD to the SATA hard disk in batches based on the storage period and the picture data size, and update the migration status; A data cleaning module, configured to perform data cleaning on the local SSD based on the status of the picture data in the local SSD, and synchronously update the picture metadata in the MongoDB database; A file reading module, configured to, when receiving a picture reading request, parse out the picture object ID corresponding to the picture reading request, query the memory cache according to the picture object ID. If the picture object ID does not exist in the memory cache, query the corresponding migration status from the MongoDB database, and read the corresponding file from the local SSD or the SATA hard disk according to the migration status.

[0011] Further, the picture object table includes an object ID, a device ID, a node ID, a timestamp, a storage period, a picture size, a picture type, a picture location, a migration status, an expiration time, and a redundancy rule, and the picture index includes a first composite index and a second composite index.

[0012] Further, the data storage module includes: A data upload unit, configured to store the picture data uploaded by the picture upload request in the server node, and store the Http picture form data of the picture data in the local SSD; A data storage unit, configured to generate a corresponding object ID based on the current time, the current device ID, the current machine node information, and a random number of the picture data, and store the picture metadata of the picture data in a MongoDB database.

[0013] Further, the data migration module includes: A parameter reading unit, configured to read a minimum migration file parameter and a delay migration parameter, and generate a maximum migration object ID according to the delay migration parameter; A parameter comparison unit, configured to scan the MongoDB database, compare the picture data in the local SSD with the minimum migration file parameter, and if the picture data in the local SSD is less than the minimum migration file parameter, do not migrate the picture data in the local SSD; A data migration unit, configured to, if the picture data in the local SSD is greater than the minimum migration file parameter, obtain the picture metadata of the picture data in the local SSD, obtain a corresponding list of pictures to be migrated from the picture data in the local SSD based on a preset migration rule and the picture metadata, and migrate the corresponding picture data to a SATA hard disk according to the list of pictures to be migrated, and update the corresponding migration status.

[0014] Further, the data cleaning module includes: A data reading unit, configured to read a maximum capacity limit of the local SSD and a cleaning maximum capacity limit, and generate a maximum cleaning object ID according to the current time; A data cleaning unit, configured to, if the current capacity of the local SSD is greater than the maximum capacity limit, obtain a list of pictures to be cleaned from the local SSD based on a preset scanning rule, and perform data cleaning on the local SSD based on the list of pictures to be cleaned; A capacity judgment unit, configured to, after the data cleaning of the local SSD is completed, accumulate and count the current cleaning capacity, and judge whether the current cleaning capacity is greater than the cleaning maximum capacity limit, and if so, exit the data cleaning.

[0015] The present invention also provides a storage medium, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned picture distributed caching method based on MongoDB is implemented.

[0016] The present invention also provides a computer, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above-mentioned MongoDB-based picture distributed caching method is implemented.

[0017] In the MongoDB-based picture distributed caching method and system of the present invention, Nginx is used for distributed balancing to achieve dynamic expansion of access services to cope with high concurrency. Local SSD is used as a temporary cache, and the high IOPS of SSD is utilized to improve concurrent receiving and reading performance. Memory caching is used to compare the recent hot data in the cache to realize the situation of real-time picture analysis. The pictures of real-time analysis are directly hit in the memory, reducing the number of times of requesting to read picture data from SSD or SATA hard disk; the high-performance read and write characteristics and sharding technology of MongoDB are utilized to more conveniently perform dynamic expansion of the scale of picture metadata; the _id field of MongoDB is redesigned, and the combination of _id and business is realized. _id is the default primary key of MongoDB. Searching based on the default primary key in combination with the business scenario speeds up the search efficiency; the three-layer caching technology, namely memory, SSD, and SATA hard disk, is utilized to improve the concurrent reading performance of pictures. Hot data is placed in the memory, sub-hot data is placed in the SSD, and cold data is placed in the SATA hard disk. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of the MongoDB-based picture distributed caching method in the first embodiment of the present invention; Figure 2 It is an architecture relationship diagram between services in the first embodiment of the present invention; Figure 3 It is a partial identification schematic diagram in the MongoDB table in the first embodiment of the present invention; Figure 4 It is a flowchart of routing a picture query request to a machine in the first embodiment of the present invention; Figure 5 It is a structural block diagram of the MongoDB-based picture distributed caching system in the second embodiment of the present invention; Figure 6 It is a structural block diagram of the computer in the third embodiment of the present invention.

[0019] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. SPECIFIC EMBODIMENTS

[0020] For the convenience of understanding the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, these embodiments are provided to make the disclosure of the present invention more thorough and comprehensive.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs. The terms used in the description of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0022] Embodiment 1 Please refer to Figure 1 , which shows the picture distributed caching method based on MongoDB in the first embodiment of the present invention. The method specifically includes steps S101 to S105: S101, initialize the picture object table and its corresponding picture index, and allocate the received picture upload request to a preset server node; The distributed processing method provided in this embodiment is implemented through Nginx load balancing, picture receiving service, picture migration service, and picture deletion service. The architectural relationship between the services is as Figure 2 shown; Among them, Nginx load balancing is the unified picture access entry. Nginx transfers the picture to a certain machine according to the machine load request. Among them, the machine can be horizontally expanded to face the concurrent access pressure. On a certain machine, an SSD is used as a high-speed cache. The advantage of the SSD is high IOPS. Specifically, in this embodiment, a three-layer architecture of memory-SSD-SATA hard disk is adopted for data storage. Hot data is stored in the memory, sub-hot data is stored in the SSD, and cold data is stored in the SATA hard disk. A large-capacity SSD is deployed on each machine for temporary caching of picture data. When a certain machine node receives picture data, it will be gradually migrated to the SATA hard disk. At the same time, the machine node will also check the capacity of the current SSD. If the capacity limit is exceeded, the picture data on the SSD will be deleted.

[0023] Further, the picture receiving service adopted in this embodiment is used to process the reception of pictures. After receiving a picture, the data is written into the local SSD, and at the same time, the picture metadata is written into the MongoDB database; the picture migration service is used to gradually migrate the picture data of this machine node to the SATA hard disk and determine which data needs to be migrated and which does not; the picture deletion service is used to check the size of the local temporary cache of this machine node. If it exceeds a certain capacity, the picture data on the local SSD needs to be deleted according to certain rules to release enough space.

[0024] Specifically, please refer to Figures 3 to 4 , each document in the MongoDB table has a unique identifier, and this ID occupies 12 bytes. The ID is redefined and business attributes are added. For example: through the format of the object ID, it is possible to know the device type, production time, etc. corresponding to the object ID, and the role of the node ID is to be able to route the picture query request to the corresponding device when doing routing in Nginx.

[0025] Among them, in this embodiment, the object migration status is divided into three types, namely un-migrated, migrated, and disk-written. When the picture is first uploaded to the SSD, it is in the un-migrated state. When the picture is migrated from the SSD to the SATA hard disk, it becomes the migrated state. When the picture is deleted from the SSD, it becomes the disk-written state; Delayed migration: It means that the picture waits for a period of time after falling on the SSD and then is migrated to the SATA hard disk. The main purpose of this method is to give priority to reading pictures from the SSD.

[0026] Picture location: In this embodiment, the picture has 2 locations. When the picture is on the SSD, it is a local path, and when the picture is on the SATA hard disk, it is an internal address format.

[0027] Further, the picture object table includes object ID, device ID, node ID, timestamp, storage period, picture size, picture type, picture location, migration status, expiration time, and redundancy rules. The picture index includes a first composite index and a second composite index.

[0028] Among them, the first composite index: migration status, storage period, _id, is mainly used for the picture migration service to query; The second composite index: migration status, _id, is mainly used for the picture deletion service to query.

[0029] S102, write the picture data uploaded by the picture upload request into the local SSD in the server node, and add business attributes to the id field of the unique identifier of the picture data to generate the corresponding object ID and picture metadata and store them in the MongoDB database; Further, step S102 specifically includes steps S1021 to S1022: S1021, in the server node, store the Http image form data of the image data uploaded by the image upload request into the local SSD; S1022, generate a corresponding object ID based on the current time, current device ID, current machine node information, and random number of the image data, and store the image metadata of the image data into the MongoDB database.

[0030] In specific implementation, receive the Http image form data of the image data, save it on the local SSD, generate a corresponding object ID based on the current time, current device ID, current machine node information, and random number of the image data, and insert the image metadata of the image data into the MongoDB database. Among them, the metadata is the data corresponding to the above image object table, that is, the object ID, device ID, node ID, timestamp, storage period, image size, image type, image location, migration status, expiration time, redundancy rule.

[0031] S103, based on the storage period and the image data size, migrate the image data in the local SSD to the SATA hard disk in batches and update the migration status; Further, step S103 specifically includes steps S1031 to S1033: S1031, read the minimum migration file parameter and the delayed migration parameter, and generate the maximum migration object ID according to the delayed migration parameter; S1032, scan the MongoDB database, compare the image data in the local SSD with the minimum migration file parameter. If the image data in the local SSD is less than the minimum migration file parameter, do not migrate the image data in the local SSD; S1033, if the image data in the local SSD is greater than the minimum migration file parameter, obtain the image metadata of the image data in the local SSD, obtain the corresponding list of images to be migrated from the image data in the local SSD based on the preset migration rule and the image metadata, and migrate the corresponding image data to the SATA hard disk according to the list of images to be migrated and update the corresponding migration status.

[0032] In specific implementation, start multiple migration service threads for each storage period. For example, if the image has two storage periods of 7 days and 30 days, start two migration service threads to process the image migration services of different storage periods respectively; Specifically, read the minimum migration file parameter and the delayed migration parameter of the configuration, generate the maximum migration object ID (MaxOID) according to the delayed migration parameter, scan the MongoDB database, and compare the image data in the local SSD with the minimum migration file parameter. If the image data in the local SSD is less than the minimum migration file parameter, it means that the current image is too small and it is meaningless to migrate it to the SATA hard disk. Therefore, it can be deleted after storing in the local SSD for more than the storage period.

[0033] Furthermore, batch scan the MongoDB database, scanning 2000 pieces of data each time. If the image data in the local SSD is greater than the minimum migration file parameter and meets the following conditions: the migration status is not migrated, the storage period is equal to the storage period of this thread, the object node ID is equal to the node of this machine, and the object ID is less than the maximum migration object ID, the obtained search results are sorted by _id, and the object ID of the last record is assigned as CurrentOID, then the list of images to be migrated can be obtained. Parse the image location from the list and perform the following operations: read the image, migrate the image data to the SATA hard disk, and update the record to the migrated state; After obtaining CurrentOID, continue to traverse at the position of the last record. If the image data in the local SSD is greater than the minimum migration file parameter and meets the following conditions: the migration status is not migrated, the storage period is equal to the storage period of this thread, the object node ID is equal to the node of this machine, the object ID is less than the maximum migration object ID and greater than CurrentOID, the obtained search results are sorted by _id. And the object ID of the last record is assigned as CurrentOID, then the list of images to be migrated can be obtained. Parse the image location from the list and perform the following operations: read the image, migrate the image data to the SATA hard disk, and update the record to the migrated state.

[0034] Repeat scanning the MongoDB database until all migrations are completed.

[0035] S104. Perform data cleaning on the local SSD based on the status of the image data in the local SSD, and synchronously update the image metadata in the MongoDB database; Furthermore, the step S104 specifically includes steps S1041 to S1043: S1041. Read the maximum capacity limit of the local SSD and the maximum cleaning capacity limit, and generate the maximum cleaning object ID according to the current time; S1042. If the current capacity of the local SSD is greater than the maximum capacity limit, obtain the list of images to be cleaned from the local SSD based on the preset scanning rule, and perform data cleaning on the local SSD based on the list of images to be cleaned; S1043. After the local SSD finishes data cleaning, accumulate and count the current cleaning capacity, and determine whether the current cleaning capacity is greater than the maximum cleaning capacity limit. If so, exit the data cleaning.

[0036] In specific implementation, the picture deletion service will perform three parts of cleaning work, including: 1. Clean the capacity of the SSD. It is necessary to clean the picture data that has been migrated to the SATA hard disk on the SSD to release enough capacity to meet the writing of new pictures. Specifically, read the maximum capacity limit (MaxCapacity) and the maximum cleaning capacity limit (MaxCleanCapacity) of the local SSD, and generate the maximum cleaning object ID according to the current system time T1. Determine whether the current capacity is greater than the maximum capacity limit. If the current capacity is greater than the maximum capacity limit, batch scan the MongoDB database, scanning 2000 pieces of data each time. The scanned data needs to meet the following conditions: the migration status is migrated, the object node ID is equal to the local machine node, and the object ID is less than the maximum cleaning object ID. The obtained search results are sorted by _id, and the object ID of the last record is assigned to CurrentOID to obtain the list of pictures to be cleaned. Parse the picture location from the list and perform the following operations: delete the local file, modify the picture location to the SATA hard disk location, update the record to the disk-written state, and accumulate and count the current cleaning capacity as CurrentCleanCapacity. After obtaining CurrentOID, continue to traverse at the position of the last record, batch scan the MongoDB database, scanning 2000 pieces of data each time. The scanned data needs to meet the following conditions: the migration status is migrated, the object node ID is equal to the local machine node, the object ID is less than the maximum cleaning object ID and greater than CurrentOID. The obtained search results are sorted by _id, and the object ID of the last record is assigned to CurrentOID to obtain the list of pictures to be cleaned. Parse the picture location from the list and perform the following operations: delete the local file, modify the picture location to the SATA hard disk location, update the record to the disk-written state, and accumulate and count the current cleaning capacity as CurrentCleanCapacity. After deleting the file, check whether the current CurrentCleanCapacity is greater than the maximum cleaning capacity limit. If it is greater, exit the cleaning. If it is less, repeat scanning the database until the current CurrentCleanCapacity is greater than the maximum cleaning capacity limit.

[0037] 2. Clean the picture data that exceeds the storage period. Specifically, for each storage period, multiple cleaning service threads are started. For example, if there are two storage periods for pictures, 7 days and 30 days, then two cleaning service threads are started to handle the picture cleaning services for different storage periods respectively; Read the current system time T1, calculate the expiration time T2 of the picture data according to the current system time T1, where T2 = T1 - storage period. Generate the maximum cleaning object ID based on the calculated expiration time T2. Batch scan the MongoDB database, scanning 2000 pieces of data each time. The scanned data needs to meet the following conditions: the migration status is disk-written, the object node ID is equal to the local machine node, the storage period is equal to the storage period of this thread, and the object ID is less than the maximum cleaning object ID. The obtained search results are sorted by _id, and the object ID of the last record is assigned to CurrentOID, and the database records scanned are deleted in batches; After obtaining CurrentOID, continue to traverse at the position of the last record, batch scan the database, scanning 2000 pieces of data each time. The scanned data needs to meet the following conditions: the migration status is disk-written, the object node ID is equal to the local machine node, the storage period is equal to the storage period of this thread, the object ID is less than the maximum cleaning object ID and greater than CurrentOid. The search results are sorted by _id, and the object ID of the last record is assigned to CurrentOid, and the database records scanned are deleted in batches; Continuously scan the database until all expired picture data records are cleared.

[0038] 3. Clean the picture data that has exceeded the storage period but has not been migrated; Specifically, for each storage period, multiple cleaning service threads are started. Suppose there are two storage periods for pictures, 7 days and 30 days, then two cleaning service threads are started to handle the picture cleaning services for different storage periods respectively; Read the current system time as T1, calculate the expiration time T2 of the picture data according to the current system time T1, where T2 = T1 - storage period. Generate the maximum cleaning object ID based on the calculated expiration time T2.

[0039] 4. Batch scan the database, scanning 2000 pieces of data each time. The scanned data needs to meet the following conditions: the migration status is not migrated, the object node ID is equal to the local machine node, the storage period is equal to the storage period of this thread, and the object ID is less than the maximum cleaning object ID. The search results are sorted by _id, and the object ID of the last record is assigned to CurrentOID, and the list of pictures to be cleaned can be obtained. Parse the picture locations from the list and perform the following operations: delete the local files and delete the database records in batches; After obtaining the CurrentOID, continue traversing at the position of the last record, and batch scan the MongoDB database. Each time, scan 2000 pieces of data. The scanned data needs to meet the following conditions: the migration status is un-migrated, the object node ID is equal to the node of this machine, the storage period is equal to the storage period of this thread, the object ID is less than the maximum cleaning object ID and greater than the CurrentOiID. The search results are sorted by _id, and the object ID of the last record is assigned to CurrentOID. Then, the list of pictures to be cleaned can be obtained. Parse the picture positions from the list and perform the following operations: delete local files and batch delete database records; Repeat scanning the database until all expired un-migrated picture data is cleaned up.

[0040] S105. When receiving a picture reading request, parse the picture object ID corresponding to the picture reading request, and query the memory cache according to the picture object ID. If the picture object ID does not exist in the memory cache, query the corresponding migration status from the MongoDB database, and read the corresponding file from the local SSD or the SATA hard disk according to the migration status.

[0041] In specific implementation, when Nginx receives a picture reading request, parse the picture object ID corresponding to the picture reading request, obtain the corresponding node ID information from it, route the request to the actual node machine according to the node ID information, query the memory cache according to the picture object ID. If the picture object ID exists in the memory cache, return the picture data from the memory. If the picture object ID does not exist in the memory cache, query from the MongoDB database. If the _id field in the database record is equal to the object ID, return the database record. Query the migration status from the database record. If the status is un-migrated, read the file from the local SSD. If the status is migrated or disk-written, read the picture data from the SATA hard disk.

[0042] In summary, in the above embodiments of the present invention, the MongoDB-based distributed image caching method uses Nginx for distributed balancing to achieve dynamic expansion of the access service to handle high concurrency. It uses local SSD as a temporary cache to improve the concurrent reception and reading performance with the high IOPS of the SSD, and uses memory caching to compare the recent hot data in the cache to implement real-time image analysis. The real-time analyzed images directly hit the memory, reducing the number of times to read image data from the SSD or SATA hard disk. It utilizes the high-performance read and write characteristics and sharding technology of MongoDB to more conveniently perform dynamic expansion of the scale of image metadata. It uses the redesigned _id field of MongoDB and realizes the combination of _id and the business. _id is the default primary key of MongoDB, and searching based on the default primary key in combination with the business scenario speeds up the search efficiency. It uses a three-layer caching technology, namely memory, SSD, and SATA hard disk, to improve the concurrent reading performance of images. The hot data is placed in the memory, the semi-hot data is placed in the SSD, and the cold data is placed in the SATA hard disk.

[0043] Embodiment 2 On the other hand, the present invention also proposes a MongoDB-based distributed image caching system. Please refer to Figure 5 , which shows the MongoDB-based distributed image caching system in the second embodiment of the present invention. The system includes: An initialization module 11, configured to initialize the image object table and its corresponding image index, and allocate the received image upload request to a preset server node; Further, the image object table includes an object ID, a device ID, a node ID, a timestamp, a storage period, an image size, an image type, an image location, a migration status, an expiration time, and a redundancy rule. The image index includes a first composite index and a second composite index.

[0044] A data storage module 12, configured to write the image data uploaded by the image upload request into the local SSD in the server node, and add a business attribute to the id field of the unique identifier of the image data to generate a corresponding object ID and store the image metadata in the MongoDB database; Further, the data storage module 12 includes: A data upload unit, configured to store the Http image form data of the image data uploaded by the image upload request in the local SSD in the server node; A data storage unit, configured to generate a corresponding object ID based on the current time, the current device ID, the current machine node information, and a random number of the image data, and store the image metadata of the image data in the MongoDB database.

[0045] A data migration module 13, configured to batch-migrate the picture data in the local SSD to the SATA hard disk based on the storage period and the picture data size, and update the migration status; Further, the data migration module 13 includes: A parameter reading unit, configured to read the minimum migration file parameter and the delayed migration parameter, and generate a maximum migration object ID according to the delayed migration parameter; A parameter comparison unit, configured to scan the MongoDB database, compare the picture data in the local SSD with the minimum migration file parameter, and if the picture data in the local SSD is smaller than the minimum migration file parameter, do not migrate the picture data in the local SSD; A data migration unit, configured to, if the picture data in the local SSD is larger than the minimum migration file parameter, obtain the picture metadata of the picture data in the local SSD, obtain a corresponding list of pictures to be migrated from the picture data in the local SSD based on a preset migration rule and the picture metadata, and migrate the corresponding picture data to the SATA hard disk according to the list of pictures to be migrated, and update the corresponding migration status.

[0046] A data cleaning module 14, configured to perform data cleaning on the local SSD based on the status of the picture data in the local SSD, and synchronously update the picture metadata in the MongoDB database; Further, the data cleaning module 14 includes: A data reading unit, configured to read the maximum capacity limit of the local SSD and the maximum cleaning capacity limit, and generate a maximum cleaning object ID according to the current time; A data cleaning unit, configured to, if the current capacity of the local SSD is larger than the maximum capacity limit, obtain a list of pictures to be cleaned from the local SSD based on a preset scanning rule, and perform data cleaning on the local SSD based on the list of pictures to be cleaned; A capacity judgment unit, configured to, after the data cleaning of the local SSD is completed, cumulatively count the current cleaning capacity, and judge whether the current cleaning capacity is larger than the maximum cleaning capacity limit, and if so, exit the data cleaning.

[0047] A file reading module 15, configured to, when receiving a picture reading request, parse out the picture object ID corresponding to the picture reading request, query the memory cache according to the picture object ID, and if the picture object ID does not exist in the memory cache, query the corresponding migration status from the MongoDB database, and read the corresponding file from the local SSD or the SATA hard disk according to the migration status.

[0048] The functions or operation steps realized when the above-mentioned modules and units are executed are generally the same as those in the foregoing method embodiments, and will not be described in detail herein.

[0049] The picture distributed caching system based on MongoDB provided by the embodiments of the present invention has the same implementation principle and technical effects as those in the foregoing method embodiments. For a brief description, for the parts not mentioned in the system embodiments, reference may be made to the corresponding content in the foregoing method embodiments.

[0050] Embodiment III The present invention also proposes a computer. Please refer to Figure 6 , which shows the computer in the third embodiment of the present invention, including a memory 10, a processor 20, and a computer program 30 stored on the memory 10 and executable on the processor 20. When the processor 20 executes the computer program 30, the above-mentioned picture distributed caching method based on MongoDB is implemented.

[0051] Among them, the memory 10 includes at least one type of storage medium, and the storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 10 can be an internal storage unit of the computer in some embodiments, such as the hard disk of the computer. The memory 10 can also be an external storage device in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 10 can also include both an internal storage unit and an external storage device of the computer. The memory 10 can be used not only to store application software installed on the computer and various types of data, but also to temporarily store data that has been output or will be output.

[0052] Among them, the processor 20 can be an Electronic Control Unit (ECU, also known as a vehicle computer for short), a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments, and is used to run the program code stored in the memory 10 or process data, such as executing an access restriction program, etc.

[0053] It should be noted that Figure 6 The structure shown does not constitute a limitation on the computer. In other embodiments, the computer may include fewer or more components than shown in the figure, or combine some components, or have different component arrangements.

[0054] An embodiment of the present invention also provides a storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the above-mentioned distributed picture caching method based on MongoDB.

[0055] Those skilled in the art can understand that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0056] More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection part (electronic device) having one or more wirings, a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then stored in a computer memory.

[0057] It should be understood that each part of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following well-known technologies in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0058] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0059] The embodiments described above merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A distributed image caching method based on MongoDB, characterized in that, Including: Initializing the picture object table and its corresponding picture index, and allocating the received picture upload request to a preset server node; Writing the picture data uploaded by the picture upload request into the local SSD in the server node, and adding business attributes to the id field of the unique identifier of the picture data to generate a corresponding object ID and picture metadata, and storing them in the MongoDB database; Based on the storage period and the size of the picture data, migrating the picture data in the local SSD to the SATA hard disk in batches, and updating the migration status; Performing data cleaning on the local SSD based on the status of the picture data in the local SSD, and synchronously updating the picture metadata in the MongoDB database; When receiving a picture reading request, parsing out the picture object ID corresponding to the picture reading request, and querying the memory cache according to the picture object ID. If the picture object ID does not exist in the memory cache, querying the corresponding migration status from the MongoDB database, and reading the corresponding file from the local SSD or the SATA hard disk according to the migration status.

2. The method for distributed caching of pictures based on MongoDB according to claim 1, characterized in that The picture object table includes an object ID, a device ID, a node ID, a timestamp, a storage period, a picture size, a picture type, a picture location, a migration status, an expiration time, and a redundancy rule. The picture index includes a first composite index and a second composite index.

3. The method for distributed caching of pictures based on MongoDB according to claim 2, wherein The step of writing the picture data uploaded by the picture upload request into the local SSD in the server node, and adding business attributes to the id field of the unique identifier of the picture data to generate a corresponding object ID and picture metadata, and storing them in the MongoDB database includes: In the server node, storing the Http picture form data of the picture data of the picture upload request in the local SSD; Generating a corresponding object ID based on the current time, the current device ID, the current machine node information, and a random number of the picture data, and storing the picture metadata of the picture data in the MongoDB database.

4. The method for distributed caching of pictures based on MongoDB according to claim 2, wherein The step of migrating the picture data in the local SSD to the SATA hard disk in batches based on the storage period and the size of the picture data, and updating the migration status includes: Reading the minimum migration file parameter and the delayed migration parameter, and generating the maximum migration object ID according to the delayed migration parameter; Scanning the MongoDB database, comparing the picture data in the local SSD with the minimum migration file parameter. If the picture data in the local SSD is smaller than the minimum migration file parameter, the picture data in the local SSD will not be migrated; If the picture data in the local SSD is greater than the minimum migration file parameter, obtain the picture metadata of the picture data in the local SSD, obtain the corresponding list of pictures to be migrated from the picture data in the local SSD based on a preset migration rule and the picture metadata, and migrate the corresponding picture data to the SATA hard disk according to the list of pictures to be migrated, and update the corresponding migration status.

5. The method for distributed caching of pictures based on MongoDB according to claim 2, wherein The steps of performing data cleaning on the local SSD based on the status of the picture data in the local SSD and synchronously updating the picture metadata in the MongoDB database include: Read the maximum capacity limit of the local SSD and the maximum cleaning capacity limit, and generate the maximum cleaning object ID according to the current time; If the current capacity of the local SSD is greater than the maximum capacity limit, obtain the list of pictures to be cleaned from the local SSD based on a preset scanning rule, and perform data cleaning on the local SSD based on the list of pictures to be cleaned; After the local SSD completes data cleaning, accumulate and count the current cleaning capacity, and determine whether the current cleaning capacity is greater than the maximum cleaning capacity limit. If so, exit the data cleaning.

6. A picture distributed caching system based on MongoDB, characterized in that, Include: An initialization module, used to initialize the picture object table and its corresponding picture index, and allocate the received picture upload request to a preset server node; A data storage module, used to write the picture data uploaded by the picture upload request into the local SSD in the server node, and add a service attribute to the id field of the unique identifier of the picture data to generate a corresponding object ID and picture metadata and store them in the MongoDB database; A data migration module, used to migrate the picture data in the local SSD to the SATA hard disk in batches based on the storage period and the size of the picture data, and update the migration status; A data cleaning module, used to perform data cleaning on the local SSD based on the status of the picture data in the local SSD, and synchronously update the picture metadata in the MongoDB database; A file reading module, used to, when receiving a picture reading request, parse out the picture object ID corresponding to the picture reading request, query the memory cache according to the picture object ID. If the picture object ID does not exist in the memory cache, query the corresponding migration status from the MongoDB database, and read the corresponding file from the local SSD or the SATA hard disk according to the migration status.

7. The picture distributed cache system based on MongoDB according to claim 6, wherein The picture object table includes an object ID, a device ID, a node ID, a timestamp, a storage period, a picture size, a picture type, a picture location, a migration status, an expiration time, and a redundancy rule. The picture index includes a first composite index and a second composite index.

8. The picture distributed cache system based on MongoDB according to claim 7, wherein, The data storage module includes: A data upload unit, used to store the Http picture form data of the picture data uploaded by the picture upload request in the local SSD in the server node; A data storage unit, which is used to generate a corresponding object ID based on the current time, the current device ID, the current machine node information, and a random number of the picture data, and store the picture metadata of the picture data into a MongoDB database.

9. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the MongoDB-based picture distributed caching method according to any one of claims 1 to 5.

10. A computer, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the MongoDB-based picture distributed caching method according to any one of claims 1 to 5.

Images