Data storage and retrieval method and system

Abstract

The invention relates to a data storage and retrieval method and system, and belongs to the technical field of data processing.The method comprises the steps that the data structure of video data, the storage mode of the video data and the retrieval mode of the video data are obtained; under the condition that a storage request of the video data is received, determining a fragmentation rule based on the equipment identifier of the main document, and automatically distributing the video data to different fragmentation servers under the fragmentation rule; and under the condition that a retrieval request of the video data is received, performing retrieval from different fragment servers based on a retrieval condition. According to the data storage and retrieval method, an efficient, reliable and easy-to-expand data storage and retrieval scheme is realized through automatic fragmentation, flexible data retrieval and query, an optimized data storage structure and improvement of system integrity.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a data storage and retrieval method and system. Background Art

[0002] With the development of the Internet, the Internet of Things (IoT), social media, and sensor networks, the speed and scale of data generation far exceed the processing capabilities of traditional databases. Big data encompasses not only structured data but also semi-structured and unstructured data, such as text, images, and video. This places higher demands on storage and retrieval. The growing demand for real-time data processing and analysis requires databases to rapidly ingest, store, and retrieve data. However, traditional relational databases (such as MySQL and Oracle) face numerous challenges when processing big data. Relational database management systems (RDBMSs) rely on fixed table structures and are difficult to adapt to storing unstructured and semi-structured data. Traditional databases are prone to performance bottlenecks and scalability issues when faced with large data volumes and high concurrent requests. With the continuous connection of devices, the concurrency and volume of video data are also increasing. Therefore, how to better implement data storage and retrieval has become a pressing issue. Summary of the Invention

[0003] In view of the above shortcomings of the prior art, the purpose of the present invention is to provide a data storage and retrieval method and system, which realizes an efficient, reliable and easily scalable data storage and retrieval solution.

[0004] In a first aspect of the present invention, a data storage and retrieval method is proposed, comprising: S1, obtaining a data structure of video data, a storage method of the video data, and a retrieval method of the video data; S2, upon receiving a storage request for the video data, determining a sharding rule based on a device identification of a master document, and automatically distributing the video data to different shard servers under the sharding rule; S3, upon receiving a retrieval request for the video data, retrieving from the different shard servers based on retrieval conditions.

[0005] Furthermore, the data structure of the video data includes the structure of a main document, the structure of the main document includes the device identification, the creation date of the video data and a nested array document, wherein the attribute information of the nested array document includes the name, storage address, size and recording time of the video data, and the nested array document includes multiple video data, each of which corresponds to a unique identifier.

[0006] Furthermore, indexes are created on both the device identifier and the creation date of the video data. Each piece of video data corresponds to a unique identifier, and an index is created on the identifier.

[0007] Furthermore, the storage method of the video data includes a data management function, and the data management function includes a write function, a modification function and a delete function; wherein, in the case of the write function, the newly written video data is added to the nested array document in the current video document using the $push operator; in the case of the modification function, the target data in the nested array document is modified based on the identifier of the video data and the $set operator; in the case of the delete function, the target data is removed from the nested array document based on the identifier of the video data and the $pull operator.

[0008] Furthermore, a retrieval method for the video data is determined, including: using the $match operator, filtering out a main document containing the target video according to the device identifier and time interval; using the $unwind operator to filter out a nested array document from the main document, and splitting the nested array document into multiple independent documents, wherein each video data in the nested array document corresponds to an independent document; using the $match operator based on the target condition to filter out useless data in the nested array document; using the $group operator to perform a group query on the main document identifier, and combining the $push operator to reorganize the split independent documents into a first array document.

[0009] Furthermore, two MongoDB instances are deployed on the shard server and configured with the same replica set name. Shardsvr is used to declare them as shard instances, and replica set information is configured and shards are initialized.

[0010] Furthermore, it also includes: deploying multiple MongoDB instances on the configuration server, configuring them with the same replica set name, using configsvr to declare them as configuration service instances, configuring replica set information and initializing shards; deploying multiple MongoDB instances on the configuration routing server, using configdb to declare the configuration service cluster address, entering the routing service instance, using addShard to configure all shard clusters, and using enableSharding to specify the database that needs to be sharded, and using shardCollection to configure the shard key.

[0011] The second aspect of the present invention proposes a data storage and retrieval system, including: an acquisition module for acquiring the data structure of video data, the storage method of the video data, and the retrieval method of the video data; a storage module for determining a sharding rule based on the device identification of the main document when receiving a storage request for the video data, and automatically distributing the video data to different shard servers under the sharding rule; a retrieval module for searching from the different shard servers based on retrieval conditions when receiving a retrieval request for the video data.

[0012] In a third aspect of the present invention, an electronic device is proposed, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the methods described in the first aspect of the present invention.

[0013] A fourth aspect of the present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute any one of the methods according to the first aspect of the present invention.

[0014] The beneficial effects of the present invention are as follows:

[0015] The data storage and retrieval method and system described in this invention obtain the data structure, storage method, and retrieval method of video data. Upon receiving a request to store video data, the system determines a sharding rule based on the device identifier of the master document and automatically distributes the video data to different shard servers based on the sharding rule. Upon receiving a retrieval request for video data, the system retrieves data from the different shard servers based on the retrieval criteria. This data storage and retrieval method achieves an efficient, reliable, and easily scalable data storage and retrieval solution through automated sharding, flexible data retrieval and querying, an optimized data storage structure, and improved system integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are only for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. Throughout the drawings, the same reference numerals represent the same components. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings.

[0017] Figure 1 is a flow chart of a data storage and retrieval method according to one embodiment of the present invention;

[0018] Figure 2is a schematic diagram of a data storage and retrieval system according to an embodiment of the present invention;

[0019] Figure 3 It is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work should fall within the scope of protection of the present invention.

[0021] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0022] In the description of the present invention, it should be noted that, unless otherwise expressly specified and limited, the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second" and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be a communication between the internal parts of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with certain aspects of the present invention, as detailed in the appended claims.

[0024] The present invention provides a data storage and retrieval method, system, and related devices. Specifically, the data storage and retrieval method, system, and related devices according to embodiments of the present invention are described below with reference to the accompanying drawings.

[0025] Figure 1 This is a flow chart of a data storage and retrieval method according to one embodiment of the present invention. It should be noted that the data storage and retrieval method according to the embodiment of the present invention can be applied to the data storage and retrieval system according to the embodiment of the present invention. The data storage and retrieval system can be configured on an electronic device or on a server. This embodiment of the present application is not limited to this.

[0026] like Figure 1 As shown, the data storage and retrieval method comprises:

[0027] S110, obtaining the data structure of the video data, the storage method of the video data, and the retrieval method of the video data.

[0028] In an embodiment of the present invention, the data structure of the video data includes the structure of a main document, the structure of the main document includes a device identifier, a creation date of the video data, and a nested array document, wherein the attribute information of the nested array document includes the name, storage address, size, and recording time of the video data, and the nested array document includes multiple video data, each of which corresponds to a unique identifier.

[0029] Among them, indexes are created on both the device identification and the creation date of the video data. Each piece of video data corresponds to a unique identification, and an index is created on the identification.

[0030] Specifically, the data structure of the recorded data is designed, using the device ID and the creation date of the recorded data as the data foundation. The device ID, the set of recorded device videos for that day, and the creation date of the recorded data are nested in an array document as the primary document. The nested array document stores the actual recorded data, recording the name, storage address, size, time, and other information of the recorded data. A composite index is added to the device ID and the creation date of the recorded data to quickly locate the recorded data stored on a particular device on a particular day. An identifier is established for each recorded data item in the nested array document, and a unique identifier for each recorded data item is generated using MongoDB's objectId generation rules. An index is then established on the identifier of each recorded data item, facilitating quick queries for specific recordings. This means that, based on the same key data, the key data is used as the primary document, while each detailed data item is written as an embedded document within the corresponding primary document data. This allows for vertical scalability of the document data, significantly increasing data storage capacity.

[0031] In an embodiment of the present invention, the storage method of video data includes a data management function, and the data management function includes a write function, a modify function and a delete function; wherein, in the case of the write function, the newly written video data is added to the nested array document in the current video document using the $push operator; in the case of the modify function, the target data in the nested array document is modified based on the identifier of the video data and the $set operator; in the case of the delete function, the target data is removed from the nested array document based on the identifier of the video data and the $pull operator.

[0032] In an embodiment of the present invention, a method for retrieving video data is determined, including: using the $match operator to filter out a main document containing a target video according to a device identifier and a time interval; using the $unwind operator to filter out a nested array document from the main document, and splitting the nested array document into multiple independent documents, wherein each video data in the nested array document corresponds to an independent document; using the $match operator to filter out useless data in the nested array document based on a target condition; using the $group operator to perform a group query on the main document identifier, and combining the $push operator to reorganize the split independent documents into a first array document.

[0033] That is to say, use the aggregation pipeline query, use the device ID and time interval combined with the $match operator to assemble the first search condition, use the $unwind operator to disassemble the nested array document, use the $match operator to assemble the actual video query as the second search condition to filter out useless video data, and finally use the $group grouping operator to group the main document ID and combine the $push operator to reassemble the disassembled array document into a new array document to achieve the video data retrieval function.

[0034] S120 , when a storage request for video data is received, a sharding rule is determined based on the device identification of the main document, and the video data is automatically distributed to different sharding servers according to the sharding rule.

[0035] The device identifier of the main document is used as a fragmentation rule.

[0036] S130: When a retrieval request for video data is received, searching is performed from different shard servers based on the retrieval conditions.

[0037] The search conditions include but are not limited to device identification, time interval, etc.

[0038] In an embodiment of the present invention, two MongoDB instances are deployed on a shard server and configured with the same replica set name. Shardsvr is used to declare them as shard instances, and replica set information is configured and shards are initialized.

[0039] In an embodiment of the present invention, multiple MongoDB instances are deployed on a configuration server, configured with the same replica set name, declared as a configuration service instance using configsvr, configured with replica set information, and initialized with shards. Multiple MongoDB instances are deployed on a configuration routing server, and the configuration service cluster address is declared using configdb. Accessing the routing service instance, all shard clusters are configured using addShard, and databases to be sharded are specified using enableSharding. Shard keys are configured using shardCollection. Thus, data is automatically distributed across shards based on data-specific keys, enabling horizontal scalability of data storage and further increasing data storage capacity. Replica sets are configured on each shard to ensure high data availability and fault tolerance. Pipeline queries enable complex data processing and analysis. Creating indexes and nested array document indexes combined with pipeline queries can significantly improve query performance.

[0040] In a specific embodiment of the present invention, assume that there is a monitoring system that includes 1,000 monitoring devices, and each device generates multiple recording data every day. In the case of data storage, new recording data is written to the corresponding MongoDB master document every day based on the device ID and date. If the recording data of a certain device needs to be updated or deleted, the system will perform the corresponding operation based on the unique ID of the recording data. In the case of data retrieval, when it is necessary to retrieve the recording data of a certain device in a specific time interval, the system will quickly locate the relevant shard server based on the device ID and time interval. Using aggregate pipeline queries, the system can efficiently filter out the target recording data and return it to the requester.

[0041] According to the data storage and retrieval method of an embodiment of the present invention, the data structure, storage method, and retrieval method of video data are obtained. Upon receiving a request to store video data, sharding rules are determined based on the device identifier of the master document. Under these sharding rules, the video data is automatically distributed to different shard servers. Upon receiving a request to retrieve video data, retrieval is performed from the different shard servers based on the retrieval criteria. This data storage and retrieval method achieves an efficient, reliable, and easily scalable data storage and retrieval solution through automated sharding, flexible data retrieval and querying, an optimized data storage structure, and improved system integrity.

[0042] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0043] According to one aspect of an embodiment of the present invention, a data storage and retrieval system is also provided. Figure 2 is a schematic diagram of a data storage and retrieval system according to an embodiment of the present invention; Figure 2 Shown, including:

[0044] An acquisition module 210 is configured to acquire a data structure of video data, a storage method of the video data, and a retrieval method of the video data;

[0045] The storage module 220 is configured to, upon receiving a storage request for the video data, determine a sharding rule based on the device identifier of the master document, and automatically distribute the video data to different sharding servers according to the sharding rule;

[0046] The retrieval module 230 is configured to, upon receiving a retrieval request for the video data, perform retrieval from the different shard servers based on retrieval conditions.

[0047] According to an embodiment of the present invention, the data storage and retrieval system obtains the data structure, storage method, and retrieval method of video data. Upon receiving a request to store video data, it determines a sharding rule based on the device identifier of the master document and automatically distributes the video data to different shard servers based on the sharding rule. Upon receiving a retrieval request for video data, it retrieves data from different shard servers based on the retrieval criteria. This data storage and retrieval system achieves an efficient, reliable, and easily scalable data storage and retrieval solution through automated sharding, flexible data retrieval and querying, an optimized data storage structure, and improved system integrity.

[0048] Optionally, the data structure of the video data includes the structure of a main document, the structure of the main document includes the device identifier, the creation date of the video data and a nested array document, wherein the attribute information of the nested array document includes the name, storage address, size and recording time of the video data, and the nested array document includes multiple video data, each of which corresponds to a unique identifier.

[0049] Optionally, indexes are created on both the device identifier and the creation date of the video data, each piece of video data corresponds to a unique identifier, and an index is created on the identifier.

[0050] Optionally, the storage method of the video data includes a data management function, and the data management function includes a write function, a modify function and a delete function; wherein, in the case of the write function, the newly written video data is added to the nested array document in the current video document using the $push operator; in the case of the modify function, the target data in the nested array document is modified based on the identifier of the video data and the $set operator; in the case of the delete function, the target data is removed from the nested array document based on the identifier of the video data and the $pull operator.

[0051] Optionally, determining a retrieval method for the video data includes: using the $match operator to filter out a main document containing a target video based on a device identifier and a time interval; using the $unwind operator to filter out a nested array document from the main document, and splitting the nested array document into multiple independent documents, wherein each video data in the nested array document corresponds to an independent document; using the $match operator to filter out useless data in the nested array document based on a target condition; using the $group operator to perform a group query on the main document identifier, and combining the $push operator to reorganize the split independent documents into a first array document.

[0052] Optionally, deploy multiple MongoDB instances on the configuration server, configure them with the same replica set name, declare them as configuration service instances using configsvr, configure the replica set information, and initialize shards. Deploy multiple MongoDB instances on the configuration routing server, declare the configuration service cluster address using configdb, enter the routing service instance, configure all shard clusters using addShard, specify the databases that need sharding using enableSharding, and configure the shard key using shardCollection.

[0053] According to one aspect of an embodiment of the present invention, an electronic device is provided.

[0054] Figure 3 FIG. 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Figure 3 As shown, the electronic device may include one or more ( Figure 3Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD)) and a memory 104 for storing data. In an exemplary embodiment, the electronic device may further include a transmission device 106 and an input / output device 108 for communication functions. It will be understood by those skilled in the art that Figure 3 The structure shown is only for illustration and does not limit the structure of the above terminal device. Figure 3 More or fewer components than shown, or with Figure 3 Equivalent functions or comparisons shown Figure 3 Shown are different configurations with more functionality.

[0055] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer programs corresponding to the data storage and retrieval methods in the embodiments of the present invention. The processor 102 executes the computer programs stored in the memory 104 to perform various functional applications and data processing, thereby implementing the above-mentioned methods. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories may be connected to the terminal device via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0056] The transmission device 106 is used to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by a communications provider of a switching device. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0057] The present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute a data storage and retrieval method.

[0058] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation plans of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, and is not a limitation on the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

[0059] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0060] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and are not to be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0061] Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents; and such modifications or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be covered by the scope of protection of the present invention.

Claims

1. A data storage and retrieval method, characterized in that: include: S1, obtaining a data structure of video data, a storage method of the video data, and a retrieval method of the video data; S2, when receiving a storage request for the video data, determining a sharding rule based on the device identification of the main document, and automatically distributing the video data to different sharding servers under the sharding rule; S3: When receiving a search request for the video data, searching is performed from the different slice servers based on search conditions.

2. The data storage and retrieval method according to claim 1, characterized in that: The data structure of the video data includes the structure of a main document, the structure of the main document includes the device identification, the creation date of the video data and a nested array document, wherein the attribute information of the nested array document includes the name, storage address, size and recording time of the video data, and the nested array document includes multiple video data, each of which corresponds to a unique identification.

3. The data storage and retrieval method according to claim 2, characterized in that: An index is created on both the device identification and the creation date of the video data. Each piece of video data corresponds to a unique identification, and an index is created on the identification.

4. The data storage and retrieval method according to claim 1, characterized in that: The storage method of the video data includes a data management function, and the data management function includes a write function, a modify function and a delete function; wherein, In the case of the write function, the newly written video data is added to the nested array document in the current video document using the $push operator; In the case of the modification function, the target data in the nested array document is modified based on the identification of the video data and the $set operator; In the case of the delete function, the target data is removed from the nested array document based on the identification of the video data and the $pull operator.

5. The data storage and retrieval method according to claim 1, characterized in that: Determining a retrieval method for the video data includes: When the $match operator is used, the main document containing the target video is filtered out based on the device ID and time interval; Using the $unwind operator to filter out the nested array document from the main document, and split the nested array document into multiple independent documents, wherein each piece of video data in the nested array document corresponds to an independent document; Using the $match operator to filter useless data in the nested array document based on the target condition; The $group operator is used to perform group query on the main document identifier, and the split independent documents are reassembled into the first array document in combination with the $push operator.

6. The data storage and retrieval method according to claim 1, characterized in that: Deploy two MongoDB instances on the shard server and configure them with the same replication set name. Use shardsvr to declare them as shard instances, configure replication set information, and initialize shards.

7. The data storage and retrieval method according to claim 1, characterized in that: Also includes: Deploy multiple MongoDB instances on the configuration server and configure them with the same replication set name. Use configsvr to declare them as configuration service instances, configure replication set information, and initialize shards. Deploy multiple MongoDB instances on the configuration routing server, use configdb to declare the configuration service cluster address, enter the routing service instance, use addShard to configure all shard clusters, use enableSharding to specify the database that needs to be sharded, and use shardCollection to configure the shard key.

8. A data storage and retrieval system, characterized in that: include: An acquisition module, used to acquire a data structure of video data, a storage method of the video data, and a retrieval method of the video data; A storage module, configured to, upon receiving a storage request for the video data, determine a sharding rule based on a device identifier of a master document, and automatically distribute the video data to different sharding servers under the sharding rule; The retrieval module is used to retrieve from the different shard servers based on the retrieval conditions when receiving a retrieval request for the video data.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images