Method and system for managing vehicle-mounted internet of things card data

Abstract

The invention discloses a management method and system for vehicle-mounted internet of things card data, and the method comprises the steps: verifying a received data processing instruction, and accessing a database through the data processing instruction after the verification succeeds; wherein the data processing instruction comprises a data updating instruction and a data query instruction; based on the unique data identifier of the data updating instruction and a preset failure node list, distributing a plurality of storage nodes for to-be-updated data, and then performing fragmentation storage on the to-be-updated data in the storage nodes in a three-stage submission mode; and based on the data unique identifier of the data query instruction, determining a query node corresponding to to-be-queried data, and performing fragment query on the query node. Multi-node storage and query are performed on the vehicle-mounted Internet of Things card data, so that the data processing consistency and the data security are improved.

Description

Technical Field

[0001] This application belongs to the technical field of data storage and retrieval, and particularly relates to a method and system for managing vehicle-mounted Internet of Things card data. Background Art

[0002] With the rapid development of vehicle-mounted Internet of Things technology, the Internet of Vehicles has gradually become an important means to enhance vehicle functions and user experience. In the current Internet of Vehicles service scenario, the vehicle-mounted Internet of Things card platform should have the ability to manage and control a large amount of Internet of Things card data in real time, so as to provide storage and query services for key information such as the identification information of the Internet of Things card (ICCID), vehicle identification (VIN code), real-name authentication of vehicle owners, card status, package information, and remaining traffic for vehicle owners, vehicle enterprise technicians, and card operator technicians. However, the database system design of the current vehicle-mounted Internet of Things card platform still needs to be improved.

[0003] The current vehicle-mounted Internet of Things card platform still has technical bottlenecks in the storage and retrieval of a large amount of data, and it is difficult to ensure real-time response and fault troubleshooting when dealing with a large number of concurrent accesses and complex data queries. Most existing improvement solutions adopt a distributed database architecture to improve the response speed. However, the distributed database has great limitations in supporting strongly consistent transactions and is difficult to maintain real-time consistency in the dynamic data changes of a large number of vehicles. Therefore, the system reliability is low. At the same time, the real-time or timed status monitoring and fault detection methods adopted lack the management of task failure records, resulting in poor continuity of data services. Because the Internet of Vehicles scenario involves a large amount of privacy data of vehicles and vehicle owners, if a reliable data management method cannot be provided, it will lead to privacy leakage problems and affect data security. Summary of the Invention

[0004] This application proposes a method and system for managing vehicle-mounted Internet of Things card data, which improves the continuity of data processing and data security by storing and querying vehicle-mounted Internet of Things card data in multiple nodes.

[0005] The first aspect of this application provides a method for managing vehicle-mounted Internet of Things card data, and the method includes:

[0006] Verify the received data and processing instructions, and access the database through the data processing instructions after successful verification; wherein, the data processing instructions include a data update instruction and a data query instruction;

[0007] Based on the data unique identifier of the data update instruction and the preset list of failed nodes, allocate several storage nodes for the data to be updated, and then store the data to be updated in a sharded manner in the storage nodes through a three-phase commit method;

[0008] Based on the data unique identifier of the data query instruction, determine the query node corresponding to the data to be queried, and perform sharded query on the query node.

[0009] The above solution first verifies the user who sends the instruction, detects whether the user's permission allows corresponding operations, and ensures data security. For the operation instruction issued by the user, if it is to update the database (insert, delete, modify), the data unique identifier in the data update instruction is extracted, and based on the preset list of failed nodes, multiple currently valid nodes are allocated for the data to be updated to complete data update and backup, improving the continuity of data processing. And by means of three-phase commit, the data update request is sent to the storage node multiple times, reducing the impact on the data processing process when a failure occurs, and ensuring higher system stability. If it is a data query, based on the data unique identifier of the data query instruction, quickly locate the storage location corresponding to the data, and respond to the user's request more quickly.

[0010] In a possible implementation method of the first aspect, based on the data unique identifier of the data update instruction and the preset list of failed nodes, allocate several storage nodes for the data to be updated, and then store the data to be updated in a sharded manner in the storage nodes through the three-phase commit method, specifically:

[0011] Compare the data unique identifier with the hash values corresponding to each node in the preset database to determine the initial primary storage node;

[0012] According to the node status of the initial primary storage node, allocate several storage nodes for the data to be updated through the list of failed nodes; where the storage nodes include a primary storage node and several secondary storage nodes;

[0013] Send a data submission request to the storage node through the three-phase commit method, and when the storage node is in the committed state, submit the data to be updated to the storage node.

[0014] The above solution first determines an initial primary storage node through the data unique identifier for initial writing and storing data. Then, multiple secondary storage nodes are allocated for data backup, mainly synchronizing data copies from the primary storage node, improving the disaster tolerance of data storage. And when the primary storage node fails, data update can still be completed based on the secondary storage node, ensuring the continuity of data services. Then, send a three-phase data submission request to the multiple storage nodes allocated for the data to be updated, so that data update can still be independently completed when the system fails, ensuring higher system stability.

[0015] In a possible implementation method of the first aspect, compare the unique data identifier with the hash values corresponding to each node in a preset database to determine the initial primary storage node. Specifically:

[0016] Convert the unique data identifier into a flag hash value;

[0017] Based on the positions of the nodes in the preset database, compare the flag hash value with the hash values corresponding to the nodes in a clockwise direction, and use the first node whose found hash value is greater than or equal to the flag hash value as the initial primary storage node.

[0018] In a possible implementation method of the first aspect, according to the node status of the initial primary storage node, allocate a number of storage nodes for the data to be updated through the list of failed nodes. Specifically:

[0019] Based on the position of the initial primary storage node, allocate a number of secondary redundant nodes for the data to be updated;

[0020] Detect the node status of the initial primary storage node and the secondary redundant nodes according to the list of failed nodes, use the initial primary storage node in the valid state as the primary storage node, and use the secondary redundant nodes in the valid state as the secondary storage nodes;

[0021] If both the initial primary storage node and the secondary redundant nodes are not in the valid state, terminate the update and return an update failure message;

[0022] If the initial primary storage node is not in the valid state, randomly select a node from the secondary redundant nodes in the valid state as the primary storage node, and use the remaining secondary redundant nodes in the valid state as the secondary storage nodes.

[0023] The above solution first determines whether the allocated storage nodes are in the valid state through the list of failed nodes, thereby completing the screening of the storage nodes and obtaining valid nodes that can normally complete data updates. If all the allocated nodes are invalid, an update failure message is returned, indicating that the data update operation cannot be performed currently. If the initial primary storage node is invalid, an effective node is selected from the secondary redundant nodes to become the primary storage node, ensuring that the data update operation can continue, improving the continuity of data processing and the stability of the system.

[0024] In a possible implementation method of the first aspect, submit a data submission request to the storage node through the three-phase commit method. When the storage node is in the committed state, submit the data to be updated to the storage node. Specifically:

[0025] Send a first submission request to each of the storage nodes. If all the storage nodes return request approval information, then send a second submission request to each of the storage nodes again;

[0026] According to the second submission request, set the data to be updated to the pre-submission state and make the storage nodes return request approval information, and then send a third submission request to each of the storage nodes again;

[0027] According to the third submission request, the storage nodes write the data to be updated into the database, set the data to be updated to the committed state and return storage success information, and then return update success information.

[0028] The above solution first sends a first submission request to detect whether the current state of the storage nodes can complete the data update operation; after a positive response to the first submission request, send a second submission request to set the data to be updated to the pre-submission state, avoiding transaction conflicts caused by the storage nodes responding to other requests during data update, and further ensuring the continuity of data processing; finally, send a third submission request to the storage nodes to officially write the data to be updated into the storage nodes and complete the data update.

[0029] In a possible implementation method of the first aspect, based on the data unique identifier of the data query instruction, determine the query node corresponding to the data to be queried, and perform a sharded query on the query node, specifically:

[0030] Locate the query node of the data to be queried according to the data unique identifier of the data query instruction; wherein, the query node includes a main query node and several secondary redundant nodes;

[0031] Send a query request to the query nodes in the valid state and return the query result;

[0032] When there is no query node in the valid state, terminate the query and return query failure information.

[0033] In a possible implementation method of the first aspect, sending a query request to the query nodes in the valid state further includes:

[0034] If the main query node is not in the valid state, but there are secondary redundant nodes in the valid state, randomly select a secondary redundant node in the valid state as the new main query node and send a query request to it.

[0035] The above solution selects to read data from the secondary redundant nodes when the main query node is invalid, ensuring that the data query operation can still be completed in case of a failure.

[0036] In a possible implementation method of the first aspect, the received data processing instruction is verified, and after successful verification, the database is accessed through the data processing instruction. Specifically:

[0037] Match the user type and operation type corresponding to the data processing instruction to verify whether the data processing instruction is executable;

[0038] If the verification fails, an authority error message is returned, and the reason for the failure to execute the data processing instruction is recorded in the transaction log.

[0039] In a possible implementation method of the first aspect, it further includes:

[0040] Detect the node status of each node in the database at a preset frequency. When it is detected that the node status changes, update the list of failed nodes according to the detection result, and record the detection result in the transaction log;

[0041] Record the query failure information and the update failure information in the transaction log.

[0042] The above solution regularly detects the node status of each node and updates the list of failed nodes, timely feedbacks the fault problems, speeds up the fault recovery rate, and ensures the stability of the system. Moreover, the efficiently updated list of failed nodes can provide accurate data support for data processing.

[0043] The second aspect of the present application provides a management system for in-vehicle IoT card data. The system includes: an authority verification module, a data update module, and a data query module;

[0044] Among them, the authority verification module is used to verify the received data processing instruction, and after successful verification, access the database through the data processing instruction. Among them, the data processing instruction includes a data update instruction and a data query instruction;

[0045] The data update module is used to allocate a number of storage nodes for the data to be updated based on the data unique identifier of the data update instruction and the preset list of failed nodes, and then store the data to be updated in a sharded manner in the storage nodes through a three-phase commit method;

[0046] The data query module is used to determine the query node corresponding to the data to be queried based on the data unique identifier of the data query instruction, and perform a sharded query on the query node. Description of the Drawings

[0047] To more clearly illustrate the technical solutions of this application, the following will briefly introduce the drawings required for implementation. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0048] Figure 1 is a specific flowchart of a method for managing in-vehicle IoT card data provided by an embodiment of this application;

[0049] Figure 2 is a structural diagram of a system for managing in-vehicle IoT card data provided by an embodiment of this application. Specific Embodiments

[0050] The following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the drawings in the embodiments of this application. Obviously, the described embodiments are only some embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of this application.

[0051] It should be understood that the step numbers used in the text are only for convenient description and are not used to limit the execution order of the steps.

[0052] First Embodiment

[0053] To handle massive data storage and retrieval, the database system of the in-vehicle IoT card platform needs to improve the data management method to enhance the request response speed and data storage security. However, limited by the database architecture, existing distributed databases have significant limitations in supporting strongly consistent transactions and are difficult to maintain real-time consistency in the dynamic data changes of a large number of vehicles. Moreover, when a node in the database fails, it will affect other nodes, resulting in the inability to continue operations and respond to user requests. Therefore, how to quickly and accurately respond to user needs while improving data security during user operations of updating and querying the database is the main research direction of the embodiments of this application.

[0054] As Figure 1 shown, Figure 1 This is a specific flowchart of a method for managing in-vehicle IoT card data provided by an embodiment of this application. The method for managing in-vehicle IoT card data in this embodiment includes steps S1 to S3, which are described in detail as follows:

[0055] Step S1: Verify the received data processing instruction. After successful verification, access the database through the data processing instruction.

[0056] In the embodiment of the present application, according to their actual needs, users send corresponding data update instructions or data query instructions to the database system of the vehicle-mounted IoT card platform. These data processing instructions include a data unique identifier and the data to be operated on. Among them, the data unique identifier is the identity card number of the IoT card (abbreviated as ICCID), which can be used to uniquely identify a vehicle-mounted IoT card.

[0057] Identify the corresponding user type and operation type through the data processing instructions, and determine whether the user who issues the instructions has the corresponding operation authority. If the authority is insufficient, the fault management mechanism is used to feedback the authority error information to inform the user of the reason for the operation failure, and the reason for the execution failure of the data processing instructions is recorded in the transaction log. If the user has the authority corresponding to the data processing instructions, the interface gateway is automatically called to access the database platform to perform the corresponding operation on the data to be operated on.

[0058] By introducing a user authority verification mechanism, the usage authorities of different users can be defined, and the authorities can be inherited through the inheritance relationship between roles, so that there is a hierarchical relationship between users, which is more convenient for effectively monitoring the improper access and abuse of authorities by users, and improving the data security of the vehicle-mounted IoT card platform.

[0059] Step S2: Based on the data unique identifier of the data update instruction and the preset list of failed nodes, allocate several storage nodes for the data to be updated, and then store the data to be updated in a sharded manner in the storage nodes through a three-phase commit method.

[0060] In the embodiment of the present application, if the user issues a data update instruction, the data unique identifier and the data to be updated are extracted from the data update instruction, and the data to be updated is sharded according to the data unique identifier through a consistent hashing algorithm, and the data to be updated is divided into several data blocks for separate update operations.

[0061] For each data block, an initial primary storage node is allocated through the data unique identifier. Specifically: the data unique identifier is converted into a 128-bit hash value using the MD5 hashing algorithm, and then this hash value is converted into an integer to obtain a flag hash value. Then, the nodes are searched in the clockwise direction through the given sharding mapping table. When the first node with a hash value greater than or equal to the flag hash value is found, this node is used as the initial primary storage node.

[0062] Then, based on this initial primary storage node, several secondary redundant nodes are selected from the database for data block allocation. Since the initially found initial primary storage node may not be valid in the current state, it is necessary to screen and reallocate these storage nodes with the help of a preset list of failed nodes, and select the nodes in the valid state to successfully complete the data update. Among them, the list of failed nodes is regularly updated according to the detection data, and the nodes in the invalid state are recorded in the list to ensure that the status of each storage node can be accurately recorded.

[0063] Specifically, the screening and reallocation of storage nodes are mainly divided into the following three cases:

[0064] (1) If the initial primary storage node and all secondary redundant nodes are in the list of failed nodes, that is, these nodes are not valid, then return the storage node failure information and record it in the transaction log, and then return the update failure information to the user according to the updated transaction log to explain the operation failure result and reason, and terminate the data update process;

[0065] (2) If the initial primary storage node is in the valid state, and there are secondary redundant nodes not in the valid state, then use this initial primary storage node as the primary storage node, remove the invalid secondary redundant nodes, and use the secondary redundant nodes in the valid state as secondary storage nodes;

[0066] (3) If the initial primary storage node is invalid, randomly select a node from the secondary redundant nodes in the valid state as the primary storage node, and the remaining valid secondary redundant nodes as secondary storage nodes.

[0067] After completing the screening and reallocation of storage nodes, there is one primary storage node and multiple secondary storage nodes for the data to be updated to be used for data update and backup. Among them, the primary storage node is used for initial writing and storing vehicle data, that is, all data to be updated are first written into the primary storage node. And the secondary storage nodes do not directly participate in new data storage, but synchronize data copies from the primary storage node for repeated data backup to improve the disaster tolerance of data storage. When the primary storage node fails or becomes unavailable, the data update can be completed through the remaining secondary storage nodes to ensure the continuity of data services, so that even if some nodes in the database system fail, the user requests can be quickly responded to. In addition, on the premise of ensuring data consistency, the secondary storage nodes can also be used as a reliable data source. For example, when it is found that the content stored in the primary storage node is inconsistent with that in the secondary storage node, the data that is consistent with most nodes can be used as reliable data.

[0068] After determining the storage nodes, submit data submission requests to the storage nodes multiple times through the three-phase commit method, and ensure that the data update can be successfully completed even in the case of node unavailability through the pre-commit mechanism.

[0069] The three - stage submission method is specifically divided into three stages:

[0070] The first stage is used to determine whether the storage nodes are available. Specifically:

[0071] (1) Send a first submission request to all storage nodes. In the embodiments of this application, it is to send a "Can Commit" request to inquire whether each storage node can perform the storage of the data to be updated this time;

[0072] (2) Make a judgment on whether the storage nodes can be used according to the system status of each storage node, such as whether the resources are available and whether there are conflicts with the transactions currently being processed by the nodes. If the storage node is available, it returns a request approval message. In the embodiments of this application, it returns "Can Commit - Yes"; if the node is unavailable, it returns a request failure message. In the embodiments of this application, it returns "Can Commit - No";

[0073] (3) When all storage nodes return request approval messages, the second stage can be entered; if there is a storage node that returns a request failure message or times out without response, terminate the data update process and restore the node status of all storage nodes. At the same time, record the reason for the failure of this operation in the transaction log and return an update failure message to the user to explain the failure result and reason of the operation.

[0074] The second stage is used to complete the preparatory work for data update. Specifically:

[0075] (1) When all storage nodes return request approval messages, send a second submission request to each storage node. In the embodiments of this application, it is to send a "Pre Commit" request to prompt the storage nodes to prepare to enter the data storage state;

[0076] (2) After receiving the second submission request, all storage nodes lock the relevant data and switch the node status and relevant data to the pre - commit state, and record the data change operation in the transaction log. After completing these operations, the storage stage returns a request approval message. In the embodiments of this application, it is to return a "Pre Commit - Acknowledged" request;

[0077] (3) If all storage nodes return request approval messages, the third stage can be entered; if there is any storage node that returns a request failure message or times out without response, terminate the data update process and restore the node status of all storage nodes. At the same time, record the reason for the failure of this operation in the transaction log and return an update failure message to the user to explain the failure result and reason of the operation.

[0078] In the embodiments of the present application, a rollback operation is performed on these storage nodes to restore the node states of all storage nodes. After performing the rollback operation, all operations related to the data processing instructions that have been completed will be revoked, including the changes made in the second phase; the data of the storage nodes will also be restored to the state before the data processing instructions started, and the relevant data will be marked as uncommitted, and at the same time recorded in the transaction log for future recovery. After the rollback operation is completed, the storage node will no longer accept the second commit request until the transaction restarts.

[0079] The third phase is used to implement the submission of the data to be updated, specifically:

[0080] (1) When all storage nodes return request consent information, a third commit request is sent to all storage nodes, which is to send a "Do Commit" request in the embodiments of the present application;

[0081] (2) After receiving the third commit request, the storage node writes the data previously marked as pre-committed state into the database, and marks these data as committed state; after the submission is successful, the storage node returns a storage success message and returns an update success message to the user;

[0082] (3) If any storage node fails in the third phase, it will be retried according to the transaction log to ensure the consistency of the submission of the data to be updated.

[0083] As an improvement of the above solution, if a certain storage node fails in any phase, data recovery will be performed according to the transaction log, and the data submission and node rollback operations will be retried. In addition, a timeout time will be set in each phase. After a timeout occurs, a failure message will be fed back to the transaction log and the user will be informed of the timeout information according to the transaction log, terminating the transaction process, ensuring the consistency of the states of all nodes, being able to reduce transaction suspension when a failure occurs, reducing waiting and retries, and thus avoiding unnecessary resource occupation.

[0084] When the data to be updated is stored, the first-level cache (node memory cache) and relevant routing information are updated. If there is a second-level cache for the data to be updated, the second-level cache is synchronously updated. Among them, the second-level cache is the global cache, which is used to store and manage the hot data of the entire system and centrally store the high-frequency query content; the first-level cache is used to store the hot data of a single node and give priority to responding to query requests.

[0085] Step S3, based on the data unique identifier of the data query instruction, determine the query node corresponding to the data to be queried, and perform a sharded query on the query node.

[0086] In the embodiment of the present application, first, a data unique identifier is extracted from the data query instruction, and the secondary cache index is queried through the data unique identifier. If relevant information is found in the secondary cache index, the query result is directly returned to the user to complete the data query operation; if no relevant information is found in the secondary cache index, the query node is located through the shard mapping table using the data unique identifier.

[0087] Specifically, a primary query node and several secondary redundant nodes are allocated based on the list of failed nodes. If both the primary query node and the secondary redundant nodes fail, the query transaction is terminated and a query failure message is returned, and the query failure message is recorded in the transaction log; if the primary query node fails, a node randomly selected from the secondary redundant nodes in the valid state is used as the new primary query node and a query request is sent to it; if the primary query node is in the valid state, a query request is sent to this primary query node.

[0088] When the primary query node receives the query request, it will initiate a query to the primary cache index and return the query data to complete the data query request.

[0089] As an improvement to the above solution, in the embodiment of the present application, the node status of each node in the database is also detected at a preset frequency. When it is detected that the node status changes, the list of failed nodes is updated according to the detection result, and the detection result is recorded in the transaction log. When the node status is restored, data synchronization is automatically completed through the transaction log to update the list of failed nodes.

[0090] Furthermore, regular analysis of the transaction log is also carried out to mark and restrict the operations of some users with abnormal operations, so as to promptly detect improper access and abuse of permissions and prevent data leakage, improving the system security. Among them, the abnormal operations defined in the embodiment of the present application include excessive operation permission conflicts, abnormal access addresses, abnormal access times, and excessive access times in a short period, etc.

[0091] Implementing the embodiment of the present application has the following beneficial effects:

[0092] In the embodiments of the present application, the user who sends the instruction is first verified to detect whether the user has the permission to perform the corresponding operation, ensuring the security of the data. For the operation instruction issued by the user, if it is to update the database (add, delete, modify), the unique data identifier in the data update instruction is extracted, and based on the preset list of failed nodes, multiple currently valid nodes are allocated to the data to be updated to complete the data update and backup, improving the continuity of data processing. And the data update request is sent to the storage node multiple times through the three-phase commit method, reducing the impact on the data processing process when a failure occurs and ensuring higher system stability. If it is a data query, according to the unique data identifier of the data query instruction, the storage location corresponding to the data is quickly located to respond to the user's request more quickly.

[0093] Second Embodiment

[0094] Further, in order to execute the management system for in-vehicle Internet of Things card data corresponding to the above method embodiments to achieve the corresponding functions and technical effects, Figure 2 A structural diagram of a management system for in-vehicle Internet of Things card data is provided. For the sake of illustration, only the parts related to this embodiment are shown. The management system for in-vehicle Internet of Things card data provided by the embodiments of the present application includes:

[0095] A permission verification module 201, configured to verify the received data processing instruction, and access the database through the data processing instruction after successful verification; wherein, the data processing instruction includes a data update instruction and a data query instruction.

[0096] In the embodiments of the present application, the user type and operation type corresponding to the data processing instruction are matched to verify whether the data processing instruction is executable;

[0097] If the verification fails, a permission error message is returned, and the reason for the failure to execute the data processing instruction is recorded in the transaction log.

[0098] A data update module 202, configured to allocate a number of storage nodes for the data to be updated based on the unique data identifier of the data update instruction and the preset list of failed nodes, and then store the data to be updated in a sharded manner in the storage nodes through the three-phase commit method.

[0099] In the embodiments of the present application, the unique data identifier is compared with the hash values corresponding to each node in the preset database to determine the initial primary storage node;

[0100] According to the node status of the initial primary storage node, a number of storage nodes are allocated for the data to be updated through the list of failed nodes; wherein, the storage node includes a primary storage node and a number of secondary storage nodes;

[0101] Submit a data submission request to the storage node through a three-phase commit method. When the storage node is in the committed state, submit the data to be updated to the storage node.

[0102] The data query module 203 is configured to determine a query node corresponding to the data to be queried based on the data unique identifier of the data query instruction, and perform a sharded query on the query node.

[0103] In an embodiment of the present application, send a first commit request to each of the storage nodes. If all the storage nodes return request approval information, then send a second commit request to each of the storage nodes again;

[0104] According to the second commit request, set the data to be updated to the pre-committed state and make the storage node return request approval information, and then send a third commit request to each of the storage nodes again;

[0105] According to the third commit request, the storage node writes the data to be updated into the database, sets the data to be updated to the committed state and returns a storage success message, and then returns an update success message.

[0106] In some embodiments, the permission verification module 201 is specifically:

[0107] The user sends a corresponding data update instruction or data query instruction to the database system of the vehicle-mounted IoT card platform according to his own actual needs. These data processing instructions include a data unique identifier and the data to be operated. Among them, the data unique identifier is the identity card number of the IoT card (abbreviated as ICCID), which can be used to uniquely identify a vehicle-mounted IoT card.

[0108] Identify the corresponding user type and operation type through the data processing instruction, and determine whether the user who issues the instruction has the corresponding operation permission. If the permission is insufficient, feedback a permission error message through the fault management mechanism to inform the user of the reason for the operation failure, and record the reason for the execution failure of the data processing instruction in the transaction log. If the user has the permission corresponding to the data processing instruction, automatically call the interface gateway to access the database platform to perform the corresponding operation on the data to be operated.

[0109] By introducing a user permission verification mechanism, the usage permissions of different users can be defined, and the permissions can be inherited through the inheritance relationship between roles, so that there is a hierarchical relationship between users, which is more convenient to effectively monitor the improper access and permission abuse behaviors of users and improve the data security of the vehicle-mounted IoT card platform.

[0110] In some embodiments, the data update module 202 is specifically:

[0111] If the instruction sent by the user is a data update instruction, extract the data unique identifier and the data to be updated from the data update instruction, and fragment the data to be updated according to the data unique identifier through the consistent hashing algorithm, and divide the data to be updated into several data blocks for separate update operations.

[0112] For each data block, assign an initial primary storage node through the data unique identifier. Specifically: use the MD5 hashing algorithm to convert the data unique identifier into a hash value with a length of 128 bits, and then convert this hash value into an integer to obtain a flag hash value. Then, look up the node in the clockwise direction through the given sharding mapping table. When the first node with a hash value greater than or equal to the flag hash value is found, use this node as the initial primary storage node.

[0113] Then, based on this initial primary storage node, select several secondary redundant nodes from the database for the data block. Since the initially found initial primary storage node may not be valid in the current state, it is also necessary to screen and reassign these storage nodes with the help of a preset list of failed nodes, and select nodes in the valid state to successfully complete the data update. Among them, the list of failed nodes will be updated regularly according to the detection data, and the nodes in the invalid state will be recorded in the list to ensure that the status of each storage node can be accurately recorded.

[0114] Specifically, the screening and reallocation of storage nodes are mainly divided into the following three cases:

[0115] (4) If the initial primary storage node and all secondary redundant nodes are in the list of failed nodes, that is, these nodes are not valid, return the storage node failure information and record it in the transaction log, and then return the update failure information to the user according to the updated transaction log to explain the operation failure result and reason, and terminate the data update process;

[0116] (5) If the initial primary storage node is in the valid state and there are secondary redundant nodes that are not in the valid state, use this initial primary storage node as the primary storage node, remove the invalid secondary redundant nodes, and use the secondary redundant nodes in the valid state as secondary storage nodes;

[0117] (6) If the initial primary storage node is invalid, randomly select a node from the secondary redundant nodes in the valid state as the primary storage node, and use the remaining valid secondary redundant nodes as secondary storage nodes.

[0118] After the screening and reallocation of storage nodes are completed, the data to be updated has a primary storage node and multiple secondary storage nodes available for data update and backup. Among them, the primary storage node is used for initially writing and storing in-vehicle data, that is, all the data to be updated is first written to the primary storage node. The secondary storage nodes do not directly participate in new data storage, but synchronize data copies from the primary storage node for repeated data backup to enhance the disaster tolerance of data storage. When the primary storage node fails or becomes unavailable, data update can be completed through the remaining secondary storage nodes to ensure the continuity of data services, so that even if some nodes in the database system fail, user requests can be quickly responded to. In addition, on the premise of ensuring data consistency, the secondary storage nodes can also be used as reliable data sources. For example, when it is found that the content stored in the primary storage node is inconsistent with that in the secondary storage nodes, the data that is consistent among most nodes can be used as reliable data.

[0119] After determining the storage nodes, the data submission requests are submitted to the storage nodes multiple times through the three-phase commit method, and the pre-commit mechanism is used to ensure that data update can be successfully completed even if there are nodes that cannot be used.

[0120] The three-phase commit method is specifically divided into three phases:

[0121] The first phase is used to determine whether the storage nodes are available. Specifically:

[0122] (4) Send the first commit request to all storage nodes. In the embodiment of the present application, it is to send a "Can Commit" request to inquire whether each storage node can execute the storage of the data to be updated this time;

[0123] (5) Make a judgment on whether the storage nodes can be used according to the system status of each storage node, such as whether the resources are available and whether there is a conflict with the transaction currently processed by the node; if the storage node is available, return the request approval information. In the embodiment of the present application, return "Can Commit-Yes"; if the node is unavailable, return the request failure information. In the embodiment of the present application, return "Can Commit-No";

[0124] (6) When all storage nodes return the request approval information, the second phase can be entered; if there is a storage node that returns the request failure information or times out without response, terminate the data update process and restore the node status of all storage nodes, and at the same time record the reason for the failure of this operation in the transaction log and return an update failure information to the user to explain the failure result and reason of the operation.

[0125] The second phase is used to complete the preparatory work for data update. Specifically:

[0126] (4) After all storage nodes have returned request consent messages, send a second commit request to each storage node. In the embodiment of the present application, it is to send a "Pre Commit" request to prompt the storage node to prepare to enter the data storage state;

[0127] (5) After receiving the second commit request, all storage nodes lock the relevant data and switch the node state and relevant data to the pre-commit state, and record the data change operation in the transaction log. After completing these operations, the storage phase returns a request consent message. In the embodiment of the present application, it is to return a "Pre Commit-Acknowledged" request;

[0128] (6) If all storage nodes return request consent messages, the third phase can be entered; if any storage node returns a request failure message or times out without response, terminate the data update process and restore the node state of all storage nodes. At the same time, record the reason for the failure of this operation in the transaction log and return an update failure message to the user to explain the failure result and reason.

[0129] In the embodiment of the present application, the node state of all storage nodes is restored by performing a rollback operation on these storage nodes. After performing the rollback operation, all operations related to the data processing instruction that have been completed will be revoked, including the changes made in the second phase; the data of the storage node will also be restored to the state before the data processing instruction started, and the relevant data will be marked as uncommitted state, and recorded in the transaction log for future recovery. After completing the rollback operation, the storage node will no longer accept the second commit request until the transaction restarts.

[0130] The third phase is used to implement the submission of the data to be updated, specifically:

[0131] (4) When all storage nodes return request consent messages, send a third commit request to all storage nodes. In the embodiment of the present application, it is to send a "Do Commit" request;

[0132] (5) After receiving the third commit request, the storage node writes the data previously marked as the pre-commit state into the database and marks these data as the committed state; after the submission is successful, the storage node returns a storage success message and returns an update success message to the user;

[0133] (6) If any storage node fails in the third phase, it will be retried according to the transaction log to ensure the consistency of the submission of the data to be updated.

[0134] As an improvement to the above solution, if a storage node fails at any stage, data recovery will be performed according to the transaction log, and the data submission and node rollback operations will be retried. In addition, a timeout will be set for each stage. After a timeout occurs, a failure message will be fed back to the transaction log and the user will be informed of the timeout information according to the transaction log, terminating the transaction process to ensure that the states of all nodes are consistent, reducing transaction suspension in case of failures, reducing waiting and retries, and thus avoiding unnecessary resource occupation.

[0135] After the data to be updated is stored, the first-level cache (node memory cache) and related routing information are updated. If there is a second-level cache for the data to be updated, the second-level cache is synchronously updated. Among them, the second-level cache is the global cache, which is used to store and manage the hot data of the entire system and centrally store the frequently queried content; the first-level cache is used to store the hot data of a single node and give priority to responding to query requests.

[0136] In some embodiments, the data query module 203 is specifically:

[0137] First, extract the data unique identifier from the data query instruction and query the second-level cache index through the data unique identifier. If relevant information is found in the second-level cache index, the query result is directly returned to the user to complete the data query operation; if no relevant information is found in the second-level cache index, the query node is located through the shard mapping table using the data unique identifier.

[0138] Specifically, a primary query node and several secondary redundant nodes are allocated based on the list of failed nodes. If both the primary query node and the secondary redundant nodes fail, the query transaction is terminated and a query failure message is returned, and the query failure message is recorded in the transaction log; if the primary query node fails, a node is randomly selected from the secondary redundant nodes in the valid state as the new primary query node and a query request is sent to it; if the primary query node is in the valid state, a query request is sent to this primary query node.

[0139] When the primary query node receives the query request, it will initiate a query to the first-level cache index and return the query data to complete the data query request.

[0140] As an improvement to the above solution, the embodiments of the present application also detect the node status of each node in the database at a preset frequency. When it is detected that the node status changes, the list of failed nodes is updated according to the detection result, and the detection result is recorded in the transaction log. When the node status is restored, data synchronization is automatically completed through the transaction log to update the list of failed nodes.

[0141] Furthermore, the transaction log will be regularly analyzed, and users with abnormal operations will be marked and restricted. This can promptly detect improper access and abuse of permissions and prevent data leakage, thereby enhancing the system security. Among them, the abnormal operations defined in the embodiments of this application include excessive operation permission conflicts, abnormal access addresses, abnormal access times, and excessive access times in a short period, etc.

[0142] Implementing the embodiments of this application has the following beneficial effects:

[0143] In the embodiments of this application, the user who sends the instruction is first verified to detect whether the user's permission allows the corresponding operation, ensuring the security of the data. For the operation instruction sent by the user, if it is to update (add, delete, modify) the database, the unique data identifier in the data update instruction is extracted, and multiple currently valid nodes are allocated to the data to be updated based on the preset list of failed nodes to complete the data update and backup, improving the continuity of data processing. And the data update request is sent to the storage node multiple times through the three-phase commit method, reducing the impact on the data processing process when a failure occurs and ensuring higher system stability. If it is a data query, the storage location corresponding to the data can be quickly located according to the unique data identifier in the data query instruction, and the user's request can be responded to more quickly.

[0144] The specific embodiments described above further elaborate on the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above are only specific embodiments of this application and are not used to limit the protection scope of this application. In particular, for those skilled in the art, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included in the protection scope of this application.

Claims

1. A method for managing vehicle-mounted Internet of Things card data, characterized in that: include: Verifying the received data processing instructions, and accessing the database through the data processing instructions after successful verification; wherein the data processing instructions include data update instructions and data query instructions; Based on the data unique identifier of the data update instruction and the preset invalid node list, a number of storage nodes are allocated for the data to be updated, and then the data to be updated is stored in the storage nodes in fragments through a three-phase commit method; Based on the data unique identifier of the data query instruction, the query node corresponding to the data to be queried is determined, and a shard query is performed on the query node.

2. The method for managing vehicle-mounted Internet of Things card data according to claim 1, characterized in that: The data unique identifier based on the data update instruction and the preset invalid node list are used to allocate a number of storage nodes for the data to be updated, and then the data to be updated is stored in the storage nodes in fragments through a three-phase commit method, specifically: Compare the data unique identifier with the hash value corresponding to each node in the preset database to determine the initial primary storage node; According to the node status of the initial primary storage node, a number of storage nodes are allocated to the data to be updated through the failed node list; wherein the storage nodes include a primary storage node and a number of secondary storage nodes; A data submission request is submitted to the storage node in a three-phase submission manner, and when the storage node is in a submitted state, the data to be updated is submitted to the storage node.

3. The method for managing vehicle-mounted Internet of Things card data according to claim 2, characterized in that: The data unique identifier is compared with the hash value corresponding to each node in the preset database to determine the initial primary storage node, specifically: Convert the unique identifier of the data into a signature hash value; Based on the position of each node in a preset database, the mark hash value is compared with the hash value corresponding to the node in a clockwise direction, and the first node found whose hash value is greater than or equal to the mark hash value is used as the initial main storage node.

4. The method for managing vehicle-mounted Internet of Things card data according to claim 2, characterized in that: The method of allocating a number of storage nodes for the data to be updated through the failed node list according to the node status of the initial primary storage node is specifically as follows: Based on the position of the initial primary storage node, allocating a number of secondary redundant nodes for the data to be updated; Detecting the node status of the initial primary storage node and the secondary redundant node according to the failed node list, taking the initial primary storage node in a valid state as the primary storage node, and taking the secondary redundant node in a valid state as the secondary storage node; If both the initial primary storage node and the secondary redundant node are not in a valid state, terminating the update and returning update failure information; If the initial primary storage node is not in a valid state, a node is randomly selected from the secondary redundant nodes in a valid state as the primary storage node, and the remaining secondary redundant nodes in a valid state are used as the secondary storage nodes.

5. The method for managing vehicle-mounted Internet of Things card data according to claim 4, characterized in that: The step of submitting a data submission request to the storage node in a three-phase submission manner and submitting the data to be updated to the storage node when the storage node is in a submitted state is as follows: Sending a first submission request to each of the storage nodes, and if the storage nodes all return request approval information, sending a second submission request to each of the storage nodes again; According to the second submission request, the data to be updated is set to a pre-submission state and the storage node is caused to return request approval information, and then a third submission request is sent to each storage node again; According to the third submission request, the storage node writes the data to be updated into the database, sets the data to be updated to a submitted state and returns storage success information, and then returns update success information.

6. The method for managing vehicle-mounted Internet of Things card data according to claim 1, characterized in that: The data unique identifier based on the data query instruction determines the query node corresponding to the data to be queried, and performs a shard query on the query node, specifically: Locate the query node of the data to be queried according to the data unique identifier of the data query instruction; wherein the query node includes a primary query node and a plurality of secondary redundant nodes; Sending a query request to the query node in a valid state and returning the query result; When there is no query node in a valid state, the query is terminated and query failure information is returned.

7. The method for managing vehicle-mounted Internet of Things card data according to claim 6, characterized in that: The sending of a query request to the query node in a valid state further includes: If the primary query node is not in a valid state, but there is a secondary redundant node in a valid state, the secondary redundant node in a valid state is randomly selected as a new primary query node and a query request is sent to it.

8. The method for managing vehicle-mounted Internet of Things card data according to claim 1, characterized in that: The received data processing instruction is verified, and after successful verification, the database is accessed through the data processing instruction, specifically: Matching the user type and operation type corresponding to the data processing instruction to verify whether the data processing instruction is executable; If the verification fails, permission error information is returned, and the reason for the failure to execute the data processing instruction is recorded in the transaction log.

9. The method for managing vehicle-mounted Internet of Things card data according to any one of claims 1 to 8, characterized in that: Also includes: Detecting the node status of each node in the database at a preset frequency, and when detecting that the node status changes, updating the failed node list according to the detection result, and recording the detection result in the transaction log; The query failure information and the update failure information are recorded in a transaction log.

10. A management system for vehicle-mounted Internet of Things card data, comprising: Permission verification module, data update module and data query module; The permission verification module is used to verify the received data processing instructions, and access the database through the data processing instructions after successful verification; wherein the data processing instructions include data update instructions and data query instructions; The data update module is used to allocate a number of storage nodes for the data to be updated based on the data unique identifier of the data update instruction and the preset invalid node list, and then store the data to be updated in the storage nodes in fragments through a three-phase commit method; The data query module is used to determine the query node corresponding to the data to be queried based on the data unique identifier of the data query instruction, and perform a shard query on the query node.

Images