An autonomous driving data storage system, method, device and storage medium

Abstract

The application discloses an automatic driving data storage system, method, equipment and storage medium, the system comprises an object storage system, a container computing platform and a gateway service, the container computing platform comprises a container node; the object storage system is used for storing automatic driving data; the container computing platform is used for downloading the automatic driving data from the object storage system for processing, and storing the data file obtained by processing into the container node; the gateway service is used for receiving an access request of a client accessing a target file, determining routing information according to the access request, if the routing information points to the container node, forwarding the access request to the corresponding container node for operation, and returning the operation result to the client, if the routing information points to the object storage system, forwarding the access request to the object storage system for operation, and returning the operation result to the client, which improves the technical problem of resource consumption of the container platform and the object storage system caused by the automatic driving data in the caching process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, and particularly relates to an automatic driving data storage system, method, device and storage medium. BACKGROUND

[0002] The automatic driving field needs to process a large amount of automatic driving data such as road test data and map data every day. A container computing platform needs to download a large amount of automatic driving data from an object storage system to a local container for simulation, training and other processing. The processed result data is first stored locally and then uploaded to the object storage system. A client accesses the object storage system to operate the required data. If the processed result data is uploaded to the object storage system, a large amount of resources of the container platform and the object storage system need to be consumed, including local IO operation, local network transmission bandwidth, object storage system IO operation, object storage system network transmission bandwidth and object storage system storage space, which causes resource waste.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The present application provides an automatic driving data storage system, method, device and storage medium, which is used to improve the technical problem of resource consumption of a container platform and an object storage system caused by automatic driving data in a caching process.

[0005] Therefore, the present application provides an automatic driving data storage system in the first aspect, which comprises an object storage system, a container computing platform and a gateway service, wherein the container computing platform comprises a container node.

[0006] The object storage system is configured to store automatic driving data.

[0007] The container computing platform is configured to download the automatic driving data from the object storage system for processing and store the data file obtained by processing into the container node.

[0008] The gateway service is configured to receive an access request of a client for accessing a target file, determine routing information according to the access request, forward the access request to the corresponding container node for operation if the routing information is directed to the container node and return an operation result to the client, and forward the access request to the object storage system for operation if the routing information is directed to the object storage system and return an operation result to the client.

[0009] Optionally, the container computing platform further comprises a metadata service.

[0010] The container node is configured to listen to a change state of a local data file, and upload first metadata information of a new data file to the metadata service, the first metadata information comprising an identifier of the container node where the data file is located, a bucket name, a file name, a file size, and a file modification time.

[0011] The metadata service is configured to store the first metadata information of the data file in the container node, and initialize second metadata information of the data file, the second metadata information comprising a creation time, a reading time, a deletion time, a reading frequency, and a persistence type.

[0012] Optionally, the container node is further configured to:

[0013] report self state information to the metadata service, and store, by the metadata service, state information of the container node, the state information comprising an identifier of the container node, a used space size, and a remaining space size.

[0014] Optionally, the container computing platform is further configured to output the data file into a data directory of the container node according to a format of the bucket / file path.

[0015] The container node is specifically configured to listen to a change state of a data file in a local data directory, and upload first metadata information of a new data file to the metadata service.

[0016] Optionally, the gateway service is specifically configured to perform the following steps:

[0017] S1, receiving an access request of a client for accessing a target file;

[0018] S2, obtaining a target bucket and a target file path of the target file according to path information in the access request;

[0019] S3, querying whether the target bucket of the target file hits a bucket in the metadata service, if not, performing step S5, and if yes, querying whether the metadata service exists a target file corresponding to the target file path and whether a persistence type of the target file is a container computing platform;

[0020] S4, if the metadata service is queried to exist the target file corresponding to the target file path and the persistence type of the target file is the container computing platform, obtaining routing information of a container node associated with the target file, forwarding the access request to the corresponding container node according to the routing information for operation, and returning an operation result to the client, if the metadata service is queried to not exist the target file corresponding to the target file path or the persistence type of the target file is not the container computing platform, performing step S5;

[0021] S5, determining that the routing information is directed to the object storage system, forwarding the access request to the object storage system according to the routing information, and returning an operation result to the client.

[0022] Optionally, the gateway service further comprises the following steps:

[0023] According to the header information in the access request, a control signal is obtained, if the control signal comprises skipping metadata service verification, step S5 is executed, and if the control signal does not comprise skipping metadata service verification, step S3 is executed.

[0024] Optionally, the container node provides an object storage service interface, and the object storage service interface is used to read or delete a target file in the container node.

[0025] The container node is further configured to upload a reading result of the target file to the metadata service, and update a reading time and a reading frequency of the target file by the metadata service, or upload a deletion result of the target file to the metadata service, and update a deletion time of the target file by the metadata service.

[0026] Optionally, the automatic driving data storage system further comprises a data service, and the data service is configured to:

[0027] Scan metadata information of each data file in the metadata service, wherein the metadata information comprises the first metadata information and the second metadata information.

[0028] According to the reading frequency in the metadata information, copy a data file with an access frequency higher than a first preset threshold from a container node to an object storage system, delete the data file from the container node, and update a persistent type of the data file in the metadata service to the object storage system.

[0029] Optionally, the data service is further configured to:

[0030] Scan state information of the container node in the metadata service.

[0031] If it is detected that a used space size of the container node exceeds a second preset threshold, clean up data files in the container node until the used space size of the container node is lower than a third preset threshold.

[0032] Optionally, the data service is further configured to clean up the data files in the container node according to a preset life cycle.

[0033] Optionally, the data service is further configured to detect whether a deletion time of the data file is greater than a preset time according to the metadata information, and if yes, delete the file object entry in the metadata information corresponding to the data file.

[0034] The second aspect of the present application provides an automatic driving data storage method, applied to any one of the automatic driving data storage systems of the first aspect, and the method comprises:

[0035] storing the automatic driving data through the object storage system;

[0036] downloading the automatic driving data from the object storage system for processing through the container computing platform, and storing the data file obtained through processing into the container node;

[0037] receiving an access request of a client accessing a target file through the gateway service, determining routing information according to the access request, forwarding the access request to the corresponding container node for operation if the routing information points to the container node, and returning an operation result to the client, and forwarding the access request to the object storage system for operation if the routing information points to the object storage system, and returning an operation result to the client.

[0038] The third aspect of the present application provides an automatic driving data storage device, comprising a processor and a memory;

[0039] The memory is configured to store program code and transmit the program code to the processor;

[0040] The processor is configured to execute the automatic driving data storage method according to the instructions in the program code.

[0041] The fourth aspect of the present application provides a computer readable storage medium, configured to store program code, and the program code is executed by the processor to realize the automatic driving data storage method of the second aspect.

[0042] From the above technical solutions, the present application has the following advantages:

[0043] The application provides an automatic driving data storage system, comprising an object storage system, a container computing platform and a gateway service, the container computing platform comprising a container node; the object storage system is used for storing automatic driving data; the container computing platform is used for downloading the automatic driving data from the object storage system for processing and storing the data file obtained by processing into the container node; the gateway service is used for receiving an access request of a client accessing a target file, determining routing information according to the access request, forwarding the access request to the corresponding container node for operation if the routing information points to the container node, and returning the operation result to the client, and forwarding the access request to the object storage system for operation if the routing information points to the object storage system, and returning the operation result to the client.

[0044] In the application, the container computing platform downloads the required automatic driving data from the object storage system for processing, and stores the data file obtained by processing in the container node locally, without uploading it to the object storage system, and uses the container computing platform as an extension of the object storage system, while providing a unified access intersection, so that the client can access the container computing platform and the object storage system through the gateway service to operate the target file, thereby reducing the resource consumption of the container computing platform and the object storage system, and improving the technical problem of resource consumption of the container platform and the object storage system caused by the automatic driving data in the caching process. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0046] Figure 1 A structural schematic diagram of an automatic driving data storage system provided by the embodiment of the application;

[0047] Figure 2 Another structural schematic diagram of an automatic driving data storage system provided by the embodiment of the application;

[0048] Figure 3 A flowchart of an automatic driving data storage method applied to an automatic driving data storage system provided by the embodiment of the application. DETAILED DESCRIPTION

[0049] The application provides an automatic driving data storage system, method, device and storage medium, which are used for improving the technical problem of resource consumption of the container platform and the object storage system caused by the automatic driving data in the caching process.

[0050] In order for those skilled in the art to better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0051] For ease of understanding, please refer to Figure 1 The embodiment of the present application provides an automatic driving data storage system, comprising: an object storage system, a container computing platform and a gateway service, the container computing platform comprising a container node;

[0052] The object storage system is used for storing automatic driving data.

[0053] The container computing platform is used for downloading the automatic driving data from the object storage system for processing, and storing the data file obtained by processing into the container node.

[0054] The gateway service is used for receiving an access request of a client accessing a target file, determining routing information according to the access request, forwarding the access request to the corresponding container node for operation if the routing information points to the container node, and returning the operation result to the client, forwarding the access request to the object storage system for operation if the routing information points to the object storage system, and returning the operation result to the client.

[0055] The automatic driving system uploads the collected automatic driving data to the object storage system, and the object storage system stores the received automatic driving data. The object storage system in the embodiment of the present application can be a distributed object storage system. When the automatic driving data is needed for simulation, training, etc., the container computing platform downloads the required automatic driving data from the object storage system for simulation or training, etc. In the embodiment of the present application, the container computing platform outputs the data file obtained by processing (i.e. the result data file obtained by processing) to the local container node after processing the automatic driving data. The number of container nodes in the container computing platform is one or more.

[0056] When the client needs to access the target file, the client sends an access request to the gateway service, and the gateway service determines the routing information of the data link according to the access request. After receiving the access request sent by the client for accessing the target file, the gateway service queries the container computing platform to determine whether the target file exists. If the target file exists, the gateway service routes to the container node corresponding to the container computing platform, at this time, the gateway service forwards the access request to the corresponding container node to perform a corresponding operation (a read operation or a delete operation), and returns an operation result to the client. If the target file does not exist, the gateway service routes to the object storage system, at this time, the gateway service forwards the access request to the object storage system to perform a corresponding operation, and returns an operation result to the client. It should be noted that when the gateway service routes to the object storage system, all operations are the same as operations on the original object storage system, and the object storage system is not modified in the embodiment of the application.

[0057] In the embodiment of the application, the container computing platform downloads the required automatic driving data from the object storage system for processing, and stores the data file obtained by processing in the container node locally, without uploading the data file to the object storage system. In this way, the container computing platform is used as an extension of the object storage system, and a unified access portal is provided. The client can access the container computing platform and the object storage system through the gateway service to operate the target file, thereby reducing the resource consumption of the container computing platform and the object storage system, and thus improving the technical problem of resource consumption of the container computing platform and the object storage system caused by the caching of automatic driving data.

[0058] The above is one embodiment of the automatic driving data storage system provided by the application. The following is another embodiment of the automatic driving data storage system provided by the application.

[0059] For reference Figure 2 The automatic driving data storage system provided by the embodiment of the application includes an object storage system, a container computing platform, and a gateway service. The container computing platform includes a container node.

[0060] The object storage system is configured to store automatic driving data.

[0061] The container computing platform is configured to download the automatic driving data from the object storage system for processing, and store the data file obtained by processing in the container node.

[0062] The gateway service is configured to receive an access request of a client for accessing a target file, determine routing information according to the access request, forward the access request to a corresponding container node for operation if the routing information points to the container node, and return an operation result to the client, and forward the access request to the object storage system for operation if the routing information points to the object storage system, and return an operation result to the client.

[0063] Further, the container computing platform in the embodiment of the present application further comprises a metadata service;

[0064] The container node is configured to listen to the change state of the local data file, and upload first metadata information of the new data file to the metadata service, wherein the first metadata information comprises an identifier (such as a container node UUID) of the container node where the data file is located, a bucket name, a file name, a file size, and a file modification time.

[0065] The metadata service is configured to store the first metadata information of the data file in the container node, and initialize second metadata information of the data file, wherein the second metadata information comprises a creation time, a read time, a deletion time, a read frequency, and a persistence type.

[0066] The autonomous driving system uploads the collected autonomous driving data to an object storage system, and the object storage system stores the received autonomous driving data. The object storage system in the embodiment of the present application can be a distributed object storage system. When the autonomous driving data needs to be simulated or trained, the container computing platform downloads the required autonomous driving data from the object storage system for simulation or training. In the embodiment of the present application, the container computing platform outputs the data file obtained by processing to the local container node after processing the autonomous driving data. The number of container nodes in the container computing platform is one or more.

[0067] Further, the container computing platform can also mount the data directory of the host machine in the container according to the business nature, and output the obtained data file to the data directory of the container node of the host machine in the format of storage bucket / file path for some low-frequency or short-life data files. The container node is specifically configured to listen to the change state of the data file in the local data directory, and upload first metadata information of the new data file to the metadata service when a new data file is found in the data directory. The metadata service is configured to store the first metadata information of the data file in the container node, and initialize second metadata information of the data file, wherein the second metadata information comprises a creation time, a read time, a deletion time, a read frequency, and a persistence type. The persistence type in the embodiment of the present application includes two types of container computing platform and object storage system, and the default value is the container computing platform. When initializing the persistence type, the default value is initialized.

[0068] Further, the container node in the embodiment of the present application is further configured to report its own state information to the metadata service, and the metadata service saves the state information of the container node, wherein the state information comprises an identifier of the container node, a used space size, and a remaining space size. The container node can report its own state information to the metadata service at a regular time, and the metadata service saves the received state information of the container node, and updates the state information of the container node to the latest received state information.

[0069] When the client needs to access the target file, an access request is sent to the gateway service, and the gateway service judges the routing information of the data link according to the access request. The gateway service is specifically used to perform the following steps:

[0070] S1, receiving an access request of a client accessing a target file;

[0071] S2, obtaining a target storage bucket and a target file path of the target file according to path information in the access request;

[0072] S3, querying whether the target storage bucket of the target file hits a storage bucket in a metadata service, if not, performing step S5; if yes, querying whether the target file corresponding to the target file path exists in the metadata service and whether the persistent type of the target file is a container computing platform;

[0073] S4, if it is queried that the target file corresponding to the target file path exists in the metadata service and the persistent type of the target file is the container computing platform, obtaining routing information pointing to a container node associated with the target file, forwarding the access request to the corresponding container node for operation according to the routing information, and returning an operation result to the client; if it is queried that the target file corresponding to the target file path does not exist in the metadata service or the persistent type of the target file is not the container computing platform, performing step S5;

[0074] S5, determining that the routing information points to an object storage system, forwarding the access request to the object storage system for operation according to the routing information, and returning an operation result to the client.

[0075] Further, the gateway service is also used to perform the following steps:

[0076] According to the header information in the access request, a control signal is obtained, if the control signal includes skipping metadata service verification, step S5 is performed, if the control signal does not include skipping metadata service verification, step S3 is performed.

[0077] The gateway service can specifically judge the routing information of the data link according to the path information and the header information in the access request. The target storage bucket and the target file path of the target file can be obtained from the path information, for example, the path information (such as URL path) in a certain access request is / video / 2022-01-01 / data.mp4, the target storage bucket of the target file is video and the target file path is 2022-01-01 / data.mp4. The gateway service can also obtain a custom control signal from the header information (Headers information), the control signal includes Authorization authentication information, skipping metadata service verification and other options.

[0078] The access request of the gateway service defaults to the client requests the metadata service, and the header information in the access request contains an exception of skipping the metadata service verification. If the gateway service obtains the control signal including skipping the metadata service verification according to the header information in the access request, it is determined that the routing information is directed to the object storage system, and the access request is directly routed to the object storage system. At this time, the access request is forwarded to the object storage system for operation, and the operation result is returned to the client. If the control signal does not include skipping the metadata service verification, the gateway service queries whether the target storage bucket of the target file hits the storage bucket in the metadata service. If not, it is determined that the routing information is directed to the object storage system, and the access request is forwarded to the object storage system according to the routing information for operation, and the operation result is returned to the client. If it is hit, the gateway service further queries whether the target file corresponding to the target file path exists in the metadata service and whether the persistent type of the target file is the container computing platform. If it is queried that the target file corresponding to the target file path exists in the metadata service and the persistent type of the target file is the container computing platform, the routing information directed to the container node (i.e., the container node storing the target file, which can be obtained through metadata information) associated with the target file is obtained. At this time, the gateway service node is routed to the corresponding container node according to the routing information, the access request is forwarded to the corresponding container node for operation, and the operation result is returned to the client. If it is queried that the target file corresponding to the target file path does not exist in the metadata service or the persistent type of the target file is not the container computing platform, the gateway service determines that the routing information is directed to the object storage system at this time, and the access request is forwarded to the object storage system according to the routing information for operation, and the operation result is returned to the client.

[0079] Further, the container node in the embodiment of the application provides an object storage service interface, which is used to read or delete the target file in the container node. The object storage service interface can only operate the data file of the current container node and belongs to a single container node. The object storage service interface includes an object reading interface and an object deleting interface. The object reading interface is used to read the target file and return the read data content to the gateway service, which is returned to the client by the gateway service. At this time, the container node is further used to upload the read result of the target file to the metadata service, and the metadata service updates the read time and the read times of the target file. The metadata service updates the read time of the target file as the current time, and the read times of the target file are increased by one to obtain the updated read times. The object deleting interface is used to delete the target file and return the deletion result to the gateway service, which is returned to the client by the gateway service. At this time, the container node is further used to upload the deletion result of the target file to the metadata service, and the metadata service updates the deletion time of the target file, specifically updates the deletion time of the target file as the current time.

[0080] Further, the automatic driving data storage system in the embodiments of the present application further comprises a data service, which can perform life cycle management, data cleaning and data file transfer from the container node to the persistent object storage system according to the metadata information in the metadata service.

[0081] Specifically, the data service is configured to:

[0082] scan the metadata information of each data file in the metadata service, wherein the metadata information comprises first metadata information and second metadata information;

[0083] copy the data file with the access frequency higher than the first preset threshold from the container node to the object storage system according to the read times in the metadata information, delete the data file from the container node, and update the persistent type of the data file in the metadata service to the object storage system.

[0084] The data service is a separate service, which can scan the metadata information of each data file in the metadata service at a regular time, copy the data file with the access frequency higher than the first preset threshold from the corresponding container node to the object storage system according to the read times of each data file in the metadata information, call the object deletion interface of the container node to delete the data file from the container node, and update the persistent type of the data file in the metadata service to the object storage system, thereby realizing the data persistence. The data service obtains the data file with high access frequency by obtaining the metadata information in the metadata service, and stores the data file in the object storage system, so that important data can be retained for a longer period.

[0085] Further, the data service is further configured to:

[0086] scan the state information of the container node in the metadata service;

[0087] if it is detected that the used space size of the container node exceeds the second preset threshold, clean the data file in the container node until the used space size of the container node is lower than the third preset threshold.

[0088] Further, the data service is further configured to clean the data file in the container node according to the preset life cycle.

[0089] In the embodiments of the present application, the data service can implement data cleaning on the container node in two modes. In one mode, the data service can periodically scan the state information of the container node in the metadata service. If it is detected that the used space size of the container node exceeds a second preset threshold, the data service cleans the data files in the container node until the used space size of the container node is less than a third preset threshold. The data service can call the object deletion interface of the container node to clean the data files in the container node. Specifically, the data service can clean the data file with the longest creation time, the data file that has not been accessed for the longest time, or the data file with the least access frequency.

[0090] In another mode, the data service cleans the data files in the container node according to a preset life cycle. The data service cleans the data file stored in the container node for more than the corresponding life cycle by calling the object deletion interface of the container node. It should be noted that the data service can set different life cycles for the data files in the container node according to the attributes (such as type) of the data files.

[0091] Further, the data service is also configured to detect whether the deletion time of the data file is greater than a preset time according to the metadata information. If yes, the data service deletes the file object entry in the metadata information corresponding to the data file. The data service can also clean the metadata information in the metadata service. After the data file is deleted by calling the object deletion interface, the data file will continue to be saved in the metadata information in the metadata service. The data service detects whether the deletion time of the data file is greater than a preset time according to the metadata information. After it is detected that the deletion time of the data file is greater than a preset time, the data service completely deletes the file object entry in the metadata information, so as to prevent the metadata information from being too large and occupying too many resources.

[0092] In the embodiments of the present application, the data service can manage the life cycle of the data file in the container node, clean the data file in the container node, and transfer the data file in the container node to the persistent object storage system according to the metadata information in the metadata service. The data service manages the life cycle of the data file in the container node. The important data with high access frequency can be transferred to the object storage system to obtain a longer retention period, and the temporary data can be transferred to the persistent object storage system. The data with low frequency or short life cycle can fully utilize the resources of the container node, and the container computing platform can be used as an extension of the object storage system. The computing platform can output the result data to the local of the computing node according to the specific business scenario, and there is no need to upload all the data in the container computing platform, which reduces the resource consumption of the container computing platform and the object storage system. At the same time, the data service provides a unified access portal for users, and the users can access the data in the object storage system and the container node at the same time without special processing.

[0093] The above is another embodiment of an automatic driving data storage system provided by the present application, and the following is an embodiment of an automatic driving data storage method provided by the present application.

[0094] Referring to Figure 3 The automatic driving data storage method provided by the present application is applied to the automatic driving data storage system in the foregoing embodiment, and the method comprises the following steps.

[0095] Step 301: Store the automatic driving data through the object storage system.

[0096] The automatic driving system uploads the collected automatic driving data to the object storage system, and the object storage system stores the received automatic driving data.

[0097] Step 302: Download the automatic driving data from the object storage system through the container computing platform for processing, and store the data files obtained by processing into the container nodes.

[0098] When the automatic driving data is needed for simulation and training, the container computing platform downloads the required automatic driving data from the object storage system for simulation or training and the like. In the present embodiment, after processing the automatic driving data, the container computing platform outputs the data files obtained by processing (i.e. the result data files obtained by processing) to the local container nodes. The number of container nodes in the container computing platform is one or more.

[0099] Step 303: Receive an access request for accessing a target file from a client through the gateway service, determine routing information according to the access request, if the routing information points to a container node, forward the access request to the corresponding container node for operation, and return the operation result to the client, if the routing information points to the object storage system, forward the access request to the object storage system for operation, and return the operation result to the client.

[0100] When the client needs to access the target file, an access request is sent to the gateway service, and the gateway service judges the routing information of the data link according to the access request. After receiving the access request sent by the client for accessing the target file, the gateway service queries the container computing platform to determine whether the target file exists, and if so, routes to the container node corresponding to the container computing platform, at this time, the access request is forwarded to the corresponding container node for corresponding operation (read operation or delete operation), and the operation result is returned to the client, and if not, routes to the object storage system, at this time, the access request is forwarded to the object storage system for corresponding operation, and the operation result is returned to the client. It should be noted that when the gateway service routes to the object storage system, all operations are the same as the operation on the original object storage system, and the object storage system is not modified in the embodiment of the application.

[0101] In the embodiment of the application, the container computing platform downloads the required autonomous driving data from the object storage system for processing, and stores the data file obtained by processing in the local container node, without uploading it to the object storage system, thereby realizing the extension of the container computing platform as the object storage system, providing a unified access intersection, and enabling the client to access the container computing platform and the object storage system through the gateway service to operate the target file, thereby reducing the resource consumption of the container computing platform and the object storage system, and improving the technical problem of resource consumption of the container platform and the object storage system caused by the caching process of autonomous driving data.

[0102] In one embodiment, the autonomous driving data storage method in the embodiment of the application further comprises:

[0103] The container node listens to the change state of the local data file, and uploads the first metadata information of the new data file to the metadata service, the first metadata information including the identifier of the container node where the data file is located, the storage bucket name, the file name, the file size and the file modification time;

[0104] The metadata service stores the first metadata information of the data file in the container node, and initializes the second metadata information of the data file, the second metadata information including the creation time, the read time, the deletion time, the read frequency and the persistence type.

[0105] In the embodiments of the present application, the container computing platform can mount the data directory of the host machine in the container according to the business nature, and output the processed data file in the format of the storage bucket / file path to the data directory of the container node of the host machine. The container node listens to the change state of the data file in the local data directory, and when it is detected that there is a new data file in the data directory, the first metadata information of the new data file is uploaded to the metadata service. The metadata service stores the first metadata information of the data file in the container node, and initializes the second metadata information of the data file, which includes the creation time, the reading time, the deletion time, the reading frequency and the persistence type. The persistence type in the embodiments of the present application includes two types, i.e., the container computing platform and the object storage system, and the default value is the container computing platform. When initializing the persistence type, the default value is initialized.

[0106] In an embodiment, the automatic driving data storage method in the embodiments of the present application further includes:

[0107] The container node reports the state information of the container node to the metadata service, and the metadata service saves the state information of the container node, which includes the identification of the container node, the used space size and the remaining space size.

[0108] The container node can report the state information of the container node to the metadata service at a regular time, and the metadata service saves the received state information of the container node, and updates the state information of the container node to the latest received state information.

[0109] In an embodiment, the step 303 in the embodiments of the present application specifically includes:

[0110] S3031, receiving an access request of a client accessing a target file;

[0111] S3032, obtaining the target storage bucket and the target file path of the target file according to the path information in the access request;

[0112] S3033, querying whether the target storage bucket of the target file hits the storage bucket in the metadata service, if not, performing step S3035, and if yes, querying whether the target file corresponding to the target file path exists in the metadata service and whether the persistence type of the target file is the container computing platform;

[0113] S3034, if the metadata service is found to exist the target file corresponding to the target file path and the persistence type of the target file is a container computing platform, the routing information pointing to the container node associated with the target file is obtained, the access request is forwarded to the corresponding container node for operation according to the routing information, and the operation result is returned to the client; if the metadata service is found to not exist the target file corresponding to the target file path or the persistence type of the target file is not a container computing platform, step S3035 is performed;

[0114] S3035, the routing information points to the object storage system, the access request is forwarded to the object storage system for operation according to the routing information, and the operation result is returned to the client.

[0115] In an embodiment, step 303 further comprises:

[0116] S3036, the control signal is obtained according to the header information in the access request, if the control signal includes skipping metadata service verification, step S3035 is performed, and if the control signal does not include skipping metadata service verification, step S3033 is performed.

[0117] Wherein, step S3036 and step S3032 can be executed synchronously. The gateway service can determine the routing information of the data link according to the path information and the header information in the access request, and can obtain the target storage bucket and the target file path of the target file from the path information, for example, the path information (such as URL path) in a certain access request is / video / 2022-01-01 / data.mp4, and the target storage bucket of the target file is video and the target file path is 2022-01-01 / data.mp4. The gateway service can also obtain the custom control signal from the header information (Headers information), and the control signal includes Authorization authentication information, skipping metadata service verification and other options.

[0118] The access request of the gateway service to the default client requests the metadata service, and the header information in the access request contains an exception of skipping the metadata service verification. If the gateway service obtains a control signal including skipping the metadata service verification according to the header information in the access request, the routing information is determined to point to the object storage system, and the access request is directly routed to the object storage system. At this time, the access request is forwarded to the object storage system for operation, and the operation result is returned to the client. If the control signal does not include skipping the metadata service verification, the gateway service queries whether the target storage bucket of the target file hits the storage bucket in the metadata service. If not, the routing information is determined to point to the object storage system, and the access request is forwarded to the object storage system according to the routing information for operation, and the operation result is returned to the client. If it is hit, the gateway service further queries whether the target file corresponding to the target file path exists in the metadata service and whether the persistent type of the target file is the container computing platform. If it is queried that the target file corresponding to the target file path exists in the metadata service and the persistent type of the target file is the container computing platform, the routing information pointing to the container node (that is, the container node storing the target file, which can be obtained through metadata information) associated with the target file is obtained. At this time, the gateway service node routes to the corresponding container node according to the routing information, forwards the access request to the corresponding container node for operation, and returns the operation result to the client. If it is queried that the target file corresponding to the target file path does not exist in the metadata service or the persistent type of the target file is not the container computing platform, the gateway service determines that the routing information points to the object storage system, and forwards the access request to the object storage system according to the routing information for operation, and returns the operation result to the client.

[0119] In an embodiment, the automatic driving data storage method in the embodiment of the application further includes:

[0120] The container node uploads the read result of the target file to the metadata service, and the metadata service updates the read time and read times of the target file, or the container node uploads the deletion result of the target file to the metadata service, and the metadata service updates the deletion time of the target file.

[0121] The container node in the embodiment of the present application provides an object storage service interface, which is used to read or delete a target file in the container node. The object storage service interface can only operate data files of the current container node and belongs to a single container node. The object storage service interface includes an object reading interface and an object deletion interface. The object reading interface is used to read the target file and return the read data content to the gateway service, which is returned to the client by the gateway service. At this time, the container node can upload the reading result of the target file to the metadata service, and the metadata service updates the reading time and reading times of the target file. The metadata service updates the reading time of the target file to the current time, and the reading times of the target file are increased by one to obtain the updated reading times. The object deletion interface is used to delete the target file and return the deletion result to the gateway service, which is returned to the client by the gateway service. At this time, the container node can also upload the deletion result of the target file to the metadata service, and the metadata service updates the deletion time of the target file, specifically updates the deletion time of the target file to the current time.

[0122] In an embodiment, the automatic driving data storage method in the embodiment of the present application further includes:

[0123] The data service scans the metadata information of each data file in the metadata service, and the metadata information includes first metadata information and second metadata information.

[0124] According to the reading times in the metadata information, the data files with access frequencies higher than a first preset threshold are copied from the container node to the object storage system, and the data files are deleted from the container node, and the persistent type of the data files in the metadata service is updated to the object storage system.

[0125] The data service is a separate service, which can scan the metadata information of each data file in the metadata service through the data service, and then can copy the data files with access frequencies higher than a first preset threshold from the corresponding container node to the object storage system according to the reading times of each data file in the metadata information, call the object deletion interface of the container node to delete the data files from the container node, and update the persistent type of the data files in the metadata service to the object storage system, so as to realize the persistence of data. The data service obtains the data files with high access frequencies by obtaining the metadata information in the metadata service, and stores the data files in the object storage system, so that important data has a longer retention period.

[0126] In an embodiment, the automatic driving data storage method in the embodiment of the present application further includes:

[0127] The data service scans state information of the container node in the metadata service, and if it is detected that the used space size of the container node exceeds a second preset threshold, the data service cleans up data files in the container node until the used space size of the container node is lower than a third preset threshold.

[0128] In an embodiment, the automatic driving data storage method further includes:

[0129] The data service cleans up the data files in the container node according to a preset life cycle.

[0130] In an embodiment, the data service has two modes to clean up the data in the container node. In one mode, the data service can periodically scan state information of the container node in the metadata service, and if it is detected that the used space size of the container node exceeds a second preset threshold, the data service cleans up data files in the container node until the used space size of the container node is lower than a third preset threshold. The data service can call an object deletion interface of the container node to clean up the data files in the container node, and specifically, the data files with the longest creation time, the longest time of not being accessed, or the least access frequency can be cleaned up.

[0131] In another mode, the data service can clean up the data files in the container node according to a preset life cycle by calling an object deletion interface of the container node to clean up the data files stored for more than the corresponding life cycle. It should be noted that the data service can set different life cycles for the data files in the container node according to the attributes (such as type) of the data files.

[0132] In an embodiment, the automatic driving data storage method further includes:

[0133] The data service detects whether the deletion time of the data file is greater than a preset time according to the metadata information, and if yes, deletes the file object entry in the metadata information corresponding to the data file.

[0134] In an embodiment, the data service can clean up the metadata information in the metadata service. After the data file is deleted by calling the object deletion interface, the data file is still saved in the metadata information in the metadata service. The data service detects whether the deletion time of the data file is greater than a preset time according to the metadata information, and if yes, deletes the file object entry in the metadata information corresponding to the data file, so as to prevent the metadata information from being too large and occupying too many resources.

[0135] In the embodiments of the present application, the data service can perform life cycle management, data cleaning and data file conversion in the container node to the persistent object storage system according to the metadata information in the metadata service. The data service performs life cycle management on the data file in the container node. For important data with high access frequency, the data can be converted to the object storage system to obtain a longer retention period, and temporary data can be converted to the persistent object storage system. For low-frequency or short-life cycle data, the resources of the container node can be fully utilized, and the container computing platform can be extended as an object storage system. The computing platform can output the result data to the local computing node according to the specific business scenario, and does not need to upload all data in the container computing platform, thereby reducing the resource consumption of the container computing platform and the object storage system. Meanwhile, a unified access portal is provided for users to access the data in the object storage system and the container node without special processing, and no intrusion operation is performed on the original object storage system.

[0136] The embodiments of the present application also provide an automatic driving data storage device, which comprises a processor and a memory.

[0137] The memory is used for storing program code and transmitting the program code to the processor.

[0138] The processor is used for executing the automatic driving data storage method in the foregoing method embodiments according to the instructions in the program code.

[0139] The embodiments of the present application also provide a computer readable storage medium, which is used for storing program code, and the program code is executed by the processor to implement the automatic driving data storage method in the foregoing method embodiments.

[0140] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0141] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0142] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed mutual units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0143] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0144] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0145] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, includes a plurality of instructions for executing all or part of the steps of the method described in various embodiments of the present application by a computer device (which can be a personal computer, a server, or a network device, etc.). The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), a random access memory (English full name: Random Access Memory, English abbreviation: RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0146] The above-described and the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An autonomous driving data storage system, characterized by, The application relates to a system and method for processing autonomous driving data. The system comprises an object storage system, a container computing platform and a gateway service, wherein the container computing platform comprises container nodes. The object storage system is configured to store autonomous driving data. The container computing platform is configured to download autonomous driving data from the object storage system for processing and store the processed data files in the container nodes. The gateway service is configured to receive an access request of a client for accessing a target file, determine routing information according to the access request, forward the access request to a corresponding container node for operation if the routing information points to the container node and return an operation result to the client, and forward the access request to the object storage system for operation if the routing information points to the object storage system and return an operation result to the client. The container computing platform further comprises a metadata service. The container nodes are configured to listen to the change state of local data files, upload first metadata information of new data files to the metadata service, and the first metadata information comprises the identifier of the container node where the data file is located, the storage bucket name, the file name, the file size and the file modification time. The metadata service is configured to store the first metadata information of the data files in the container nodes and initialize second metadata information of the data files, and the second metadata information comprises the creation time, the reading time, the deletion time, the reading frequency and the persistence type.

2. The autonomous driving data storage system of claim 1, wherein, The container nodes are further configured to: report self state information to the metadata service, and save the state information of the container nodes by the metadata service, wherein the state information comprises the identifier of the container node, the used space size and the remaining space size.

3. The autonomous driving data storage system of claim 1, wherein, The container computing platform is further configured to output the data files to the data directory of the container nodes in the format of the storage bucket / file path. The container nodes are specifically configured to listen to the change state of the data files in the local data directory and upload the first metadata information of the new data files to the metadata service.

4. The autonomous driving data storage system of claim 1, wherein, The gateway service is specifically configured to perform the following steps: S1, receiving an access request of a client for accessing a target file; S2, obtaining the target storage bucket and the target file path of the target file according to the path information in the access request; S3, querying whether the target storage bucket of the target file hits the storage bucket in the metadata service, and if not, performing step S5; if yes, querying whether the metadata service exists the target file corresponding to the target file path and whether the persistence type of the target file is the container computing platform; S4, if the metadata service is queried to exist the target file corresponding to the target file path and the persistence type of the target file is the container computing platform, obtaining routing information pointing to the container node associated with the target file, forwarding the access request to the corresponding container node according to the routing information for operation, and returning an operation result to the client, if the metadata service is queried to not exist the target file corresponding to the target file path or the persistence type of the target file is not the container computing platform, performing step S5. S5, determining that the routing information is directed to the object storage system, forwarding the access request to the object storage system according to the routing information, and returning an operation result to the client.

5. The autonomous driving data storage system of claim 4, wherein, The gateway service is further configured to perform the following steps: According to the header information in the access request, a control signal is obtained, if the control signal includes skipping metadata service verification, step S5 is performed, if the control signal does not include skipping metadata service verification, step S3 is performed.

6. The autonomous driving data storage system of claim 1, wherein, The container node provides an object storage service interface, which is used to read or delete a target file in the container node. The container node is further configured to upload a read result of the target file to the metadata service, update a read time and a read frequency of the target file by the metadata service, or upload a deletion result of the target file to the metadata service, and update a deletion time of the target file by the metadata service.

7. The autonomous driving data storage system of claim 2, wherein, The autonomous driving data storage system further comprises a data service, and the data service is configured to: Scan metadata information of each data file in the metadata service, wherein the metadata information comprises the first metadata information and the second metadata information; According to the read frequency in the metadata information, copy a data file with an access frequency higher than a first preset threshold from a container node to an object storage system, delete the data file from the container node, and update a persistent type of the data file in the metadata service to the object storage system.

8. The autonomous driving data storage system of claim 7, wherein, The data service is further configured to: Scan state information of the container node in the metadata service; If it is detected that a used space size of the container node exceeds a second preset threshold, clean up data files in the container node until the used space size of the container node is lower than a third preset threshold.

9. The autonomous driving data storage system of claim 7, wherein, The data service is further configured to clean up data files in the container node according to a preset life cycle.

10. The autonomous driving data storage system of claim 7, wherein, The data service is further configured to detect whether a deletion time of a data file is greater than a preset time according to the metadata information, and if so, delete a file object entry in metadata information corresponding to the data file.

11. An automatic driving data storage method characterized by comprising: The method is applied to the autonomous driving data storage system of any one of claims 1-10, and the method comprises: Storing autonomous driving data through an object storage system; Downloading autonomous driving data from the object storage system through a container computing platform for processing, and storing data files obtained by processing into a container node; Receiving an access request of a client for accessing a target file through a gateway service, determining routing information according to the access request, forwarding the access request to a corresponding container node for operation if the routing information is directed to the container node, and returning an operation result to the client, forwarding the access request to the object storage system for operation if the routing information is directed to the object storage system, and returning an operation result to the client; The container node listens to the change state of the local data file, and uploads first metadata information of the new data file to the metadata service, the first metadata information including the identification of the container node where the data file is located, the bucket name, the file name, the file size and the file modification time; The metadata service stores the first metadata information of the data file in the container node, and initializes second metadata information of the data file, the second metadata information including the creation time, the reading time, the deletion time, the reading frequency and the persistence type.

12. An autonomous driving data storage device, comprising: The device comprises a processor and a memory; The memory is configured to store program code and transmit the program code to the processor; The processor is configured to execute the automatic driving data storage method according to the instructions in the program code.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store program code, and the program code is executed by the processor to implement the automatic driving data storage method.

Images