A data storage method and apparatus

Abstract

The embodiment of the application discloses a data storage method and device, the method comprises the following steps: obtaining vehicle data to be stored; obtaining generation algorithm text data required when a preset generation algorithm is used to generate the vehicle data; and storing the generation algorithm text data into a preset artificial intelligence (AI) data storage model. Through the embodiment scheme, the storage space occupied by the vehicle data is greatly reduced, and the storage space is saved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to intelligent driving data storage technology, and in particular to a data storage method and device. BACKGROUND

[0002] With the continuous progress of automobile intelligent driving technology, and the continuous introduction and improvement of relevant policies in the field of intelligent driving by the state, the degree of intelligence of automobiles is getting higher and higher, and the amount of data generated is also getting larger and larger. Ten intelligent driving road sampling cars generate about 300 TB of unstructured (picture, audio, video, object) data per day; in terms of mass-produced cars, about 80 TB of intelligent driving data is generated per day (about 180 TB before screening, generated by 1.7 million vehicles). Currently, static storage modes such as block storage, file storage, and object storage are mainly used, and about 380 TB of disk capacity is needed per day to meet data storage requirements, and about 1.1 PB of storage resources are needed per month. With the gradual increase of intelligent driving scenarios, the rapid expansion of road sampling scale and intelligent driving data range, the storage demand is getting larger and larger, showing a geometric upward trend. According to the estimation of 10 new road sampling cars per year and 1.5 million car sales, nearly 3 PB of storage is needed per month by 2025, and 40 million capital costs are needed for storage resources every year, and the cost of data storage resources is too high.

[0003] Currently, the industry can only reduce data capacity through compression technology to save storage resources. However, the space for data compression is very limited, and the compression rate of different data types and file types is different, especially in terms of audio and video data file compression, the compression rate is very limited. SUMMARY

[0004] Embodiments of the present application provide a data storage method and device, which can greatly reduce the storage space occupied by vehicle data and save storage space.

[0005] The data storage method provided by the embodiments of the present application can include:

[0006] Obtaining vehicle data to be stored;

[0007] Obtaining generation algorithm text data required for generating the vehicle data using a preset generation algorithm;

[0008] Storing the generation algorithm text data into a preset artificial intelligence (AI) data storage model.

[0009] In the exemplary embodiments of the present application, before storing the generation algorithm text data into the preset AI data storage model, the method can further include:

[0010] Compressing the generation algorithm text data.

[0011] In the example embodiment of the present application, the method can further include:

[0012] Pre-acquiring generation algorithm text data of a plurality of vehicle data as training data;

[0013] Training a pre-created neural network model for storing data using the training data to obtain an AI data storage model.

[0014] In the example embodiment of the present application, after storing the generation algorithm text data into a preset artificial intelligence AI data storage model, the method can further include:

[0015] When the vehicle data needs to be queried, the vehicle data to be queried is generated by a preset data generator according to a query request, the generation algorithm, and the corresponding generation algorithm text data stored in the AI data storage model.

[0016] In the example embodiment of the present application, the vehicle data to be queried generated by the preset data generator according to the query request, the generation algorithm, and the corresponding generation algorithm text data stored in the AI data storage model can include:

[0017] The data generator sends a query instruction to the AI data storage model according to the query request, and acquires the generation algorithm text data queried by the AI data storage model according to the query instruction;

[0018] The data generator decompresses the queried generation algorithm text data, and executes the generation algorithm according to the decompressed generation algorithm text data to generate the vehicle data to be queried, and returns the generated vehicle data to an external data service querying the vehicle data.

[0019] In the example embodiment of the present application, the generation algorithm can include a deep convolutional generative adversarial network Dcgan algorithm and / or a mask autoencoder MAE algorithm.

[0020] In the example embodiment of the present application, after the vehicle data to be queried is generated by the preset data generator according to the query request, the generation algorithm, and the corresponding generation algorithm text data stored in the AI data storage model, the method can further include:

[0021] The generated vehicle data is discriminated and corrected by a preset discriminator.

[0022] In the example embodiment of the present application, the discriminator uses a sample data model in the Dcgan algorithm system.

[0023] In the example embodiments of the present application, the obtaining vehicle data to be stored can include:

[0024] According to a predetermined data reporting frequency, the vehicle data collected by the vehicle end is transmitted to a preset cloud data buffer through data express or network transmission.

[0025] The embodiments of the present application also provide a data storage device, which can include a processor and a computer readable storage medium, and the computer readable storage medium stores instructions, when the instructions are executed by the processor, the data storage method is realized.

[0026] Compared with the related art, the embodiments of the present application can include: obtaining vehicle data to be stored; obtaining generation algorithm text data required for generating the vehicle data by using a preset generation algorithm; and storing the generation algorithm text data into a preset artificial intelligence AI data storage model. Through the embodiment scheme, the occupation of storage space by vehicle data is greatly reduced, and the storage space is saved.

[0027] Other features and advantages of the present application will be described in the following description, and some will become apparent from the description, or will be understood by those skilled in the art. Other advantages of the present application can be achieved and obtained by the schemes described in the specification and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0028] The accompanying drawings are used to provide an understanding of the technical solutions of the present application, and constitute a part of the specification, and are used to explain the technical solutions of the present application together with the embodiments of the present application, and do not constitute a limitation on the technical solutions of the present application.

[0029] Figure 1 The data storage method flowchart of the embodiments of the present application is shown in the figure;

[0030] Figure 2 The data storage method structure schematic diagram of the embodiments of the present application is shown in the figure;

[0031] Figure 3 The data storage method structure schematic diagram of the embodiments of the present application is shown in the figure;

[0032] Figure 4 The data storage device block diagram of the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0033] The present application describes a plurality of embodiments, but the description is exemplary rather than limiting, and it will be apparent to those of ordinary skill in the art that there can be many embodiments and implementations that are within the scope of the embodiments described in the present application. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are possible. Unless specifically intended otherwise, any feature or element of any embodiment can be used with any other feature or element of any other embodiment, or in any other embodiment, whether or not that feature or element is specifically disclosed in combination with the other feature or element in any embodiment. Unless specifically intended otherwise, any feature or element of any embodiment can be replaced by any other feature or element of any other embodiment, or in any other embodiment, whether or not that feature or element is specifically disclosed in combination with the other feature or element in any embodiment.

[0034] The present application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features and elements disclosed herein can also be combined with any conventional feature or element to form a unique application of the presently claimed application that is not specifically disclosed. Any feature or element of any embodiment can also be combined with features or elements from other applications to form another unique application of the presently claimed application that is not specifically disclosed. Thus, it should be understood that any feature shown and / or discussed in the present application can be implemented alone or in any suitable combination. Accordingly, the embodiments are not to be restricted, except as by the appended claims and their equivalents. Additionally, various modifications and changes can be made within the scope of the claims.

[0035] Furthermore, in describing representative embodiments, the specification can have presented the method and / or process as a particular sequence of steps. However, to the extent that the method or process depends on more than one step, the method or process should not be limited to the particular sequence of steps described. Other sequences of steps can be possible, and are within the scope of the embodiments. Thus, man skilled in the art will appreciate that the specific order of the steps in the descriptions of the specification is not an inherent part of the embodiments. Additionally, the claims should not be limited to the steps in accordance with the order written herein, but can be carried out in other sequences other than the one recited in the claims.

[0036] The embodiments of the present application provide a data storage method, as shown in Figure 1 , Figure 2 The method can include steps S101-S103:

[0037] S101, obtaining vehicle data to be stored;

[0038] S102, obtaining generation algorithm text data required when generating the vehicle data by using a preset generation algorithm;

[0039] S103, store the generated algorithm text data into a preset artificial intelligence AI data storage model.

[0040] In the example embodiments of the present application, due to the rapid increase of intelligent driving research and development and mass production scene scale, intelligent driving data storage resources increase geometrically, and storage cost investment is increasingly high. From a technical point of view, how to improve the use value of intelligent driving storage resources, reduce the demand for intelligent driving storage resources, while meeting the intelligent driving data storage needs, is an urgent issue that needs to be researched and solved.

[0041] In the example embodiments of the present application, the amount of data occupying storage space is reduced from the perspective of artificial intelligence algorithm, and the storage principle is completely different from the traditional storage method.

[0042] In the example embodiments of the present application, the generation algorithm text data of the vehicle data to be stored is obtained by artificial intelligence algorithm, and the generation algorithm text data is stored in the artificial intelligence storage model, so that the generation algorithm text data is stored instead of the traditional storage of original vehicle data. Since the storage space occupied by the generation algorithm text data is much smaller than the storage space occupied by the original vehicle data, the purpose of reducing the storage space occupation is achieved.

[0043] In the example embodiments of the present application, the original vehicle data to be stored can include but is not limited to video data, picture data, voice data, radar data and various forms of data.

[0044] In the example embodiments of the present application, for example, when the vehicle data to be stored is a video data, directly storing the original video data will occupy a large space, at this time, a preset generation algorithm can be used to generate the video data, and the generation algorithm text data generated in the process of generating the video data is obtained as a storage object, and the original video data is not directly stored. The storage space occupied by the generation algorithm text data corresponding to the video data is much smaller than the storage space occupied by the video data itself, therefore, the storage space occupation is greatly reduced by the scheme.

[0045] In the example embodiments of the present application, the storage space saving rate is not dependent on or for a specific file type, whether it is ordinary text data, or video data, voice data, corresponding generation algorithm text data can be generated.

[0046] In the example embodiments of the present application, the generation algorithm text data can refer to all or part of parameters, variables, and other data required in the generation of vehicle data (e.g., a video or a picture) according to a preset generation algorithm; each vehicle data (e.g., a video or a picture) can correspond to a set of generation algorithm text data; and the generation algorithm text data can include one or a set of data expressions.

[0047] In the example embodiments of the present application, by changing the storage data form (transforming the original vehicle data into the generation algorithm text data corresponding to the vehicle data), the saved is not the original vehicle data, but the generation algorithm text data capable of regenerating the vehicle data, thereby reducing the data occupation of the storage space and improving the use value of the storage space.

[0048] In the example embodiments of the present application, for example, a 10MB picture data has a generation algorithm of about 1.2MB, and the data compression can greatly reduce the storage resource occupation. For another example, assuming that 380TB of intelligent driving data is newly added every day, if the traditional direct storage of the original vehicle data storage mode is used for saving, 380TB of storage space resources are required to complete the saving of the data. If the storage method of the embodiments of the present application is used, 380TB of data remains unchanged, but only 46TB (space saving ratio ≈ 0.87) of storage space is required to meet the data saving requirement; and the storage resources used for storing the data of one day can now store the data of nine days by using the storage method of the present application.

[0049] In the example embodiments of the present application, the generation algorithm can include but is not limited to a deep convolutional generative adversarial network (Dcgan) algorithm and / or a mask autoencoder (MAE) algorithm.

[0050] In the example embodiments of the present application, the obtaining of the vehicle data to be stored can include:

[0051] According to a predetermined data reporting frequency, the vehicle data collected by the vehicle end is transmitted to a preset cloud data buffer through data express or network transmission.

[0052] In the example embodiments of the present application, the vehicle end collects and temporarily stores the intelligent driving related vehicle data through a camera, a sensor, a radar, and the like, and stores the data in a safe medium (e.g., a hard disk). According to a predetermined data reporting frequency, the vehicle data is transmitted to a cloud data buffer through data express (manual transportation of a storage medium) or a secure private network (network transmission).

[0053] In the example embodiments of the present application, the preset storage converter can obtain the generation algorithm text data corresponding to the vehicle data according to the preset generation algorithm and the vehicle data in the cloud data buffer, and store the generation algorithm text data into an AI (artificial intelligence) data storage model, complete the saving of the generation algorithm text data, which is equivalent to completing the saving of the corresponding original vehicle data, and the original data in the cloud data buffer can be cleared at this time. In the example embodiments of the present application, the saved is the generation algorithm text data for regenerating the original vehicle data, not the original vehicle data.

[0054] In the example embodiments of the present application, before storing the generation algorithm text data into the preset AI data storage model, the method can further include:

[0055] Compressing the generation algorithm text data.

[0056] In the example embodiments of the present application, the compression of the generation algorithm text data can further reduce the space occupied by the generation algorithm text data.

[0057] In the example embodiments of the present application, the lossy compression or lossless compression can be selected according to the integrity requirement of the intelligent driving data.

[0058] In the example embodiments of the present application, the compression ratio depends on the selected compression algorithm. If the lossy compression is used, the compression rate is high, but the data will be distorted; if the lossless compression is used, the compression rate is low, but the data restoration degree is high. The corresponding compression algorithm can be selected according to the integrity requirement of different data.

[0059] In the example embodiments of the present application, the method can further include:

[0060] Pre-acquiring the generation algorithm text data of a plurality of vehicle data as training data;

[0061] Training the pre-created neural network model for storing data by using the training data to obtain an AI data storage model.

[0062] In the example embodiments of the present application, before implementing the example embodiments of the present application, the AI data storage model can be first obtained by model training.

[0063] In the example embodiments of the present application, a neural network model can be created in advance for storing data, and based on the neural network model, a large amount of training data is collected (for example, video data, picture data, radar data, sensor data, etc. obtained at the vehicle end are collected, and these video data, picture data, radar data, sensor data, etc. are generated through a preset generation algorithm, and the generation algorithm text data corresponding to these data is obtained respectively, and these generation algorithm text data is used as training data), the training data collected is used to train the neural network model, so that the neural network model only saves the generation algorithm text data corresponding to the vehicle data, and sets an appropriate identifier (for example, time, number, address, etc.) for each saved generation algorithm text data to facilitate subsequent data query; after the neural network model is trained multiple times, the AI data storage model described above can be obtained, which is used to identify and store the generation algorithm text data corresponding to the vehicle data.

[0064] In the example embodiments of the present application, in the process of real-time data storage, after obtaining the generation algorithm text data corresponding to the vehicle data to be stored, the generation algorithm text data can be input into the trained AI data storage model, and the AI data storage model is used to identify and store the generation algorithm text data, if the current data is not identified as generation algorithm text data, or is not the generation algorithm text data corresponding to the vehicle data, an error can be reported, or it can be directly ignored and not stored.

[0065] In the example embodiments of the present application, the storage occupation of the generation algorithm text data is much smaller than that of the original vehicle data, which can greatly reduce the storage resource occupation and greatly improve the use value of the storage resources. In addition, the algorithm data in the AI data storage model can be managed through a data governance model, for example, but not limited to, data cleaning, data desensitization, data daily management, etc.

[0066] In the example embodiments of the present application, as shown in Figure 3 After the generation algorithm text data is stored in the preset artificial intelligence AI data storage model, the method can further include:

[0067] When the vehicle data needs to be queried, the vehicle data to be queried is generated by a preset data generator according to the query request, the generation algorithm, and the corresponding generation algorithm text data stored in the AI data storage model.

[0068] In the example embodiments of the present application, the vehicle data to be queried generated by the preset data generator according to the query request, the generation algorithm, and the corresponding generation algorithm text data stored in the AI data storage model can include:

[0069] sending a query instruction to the AI data storage model according to the query request, and obtaining generated algorithm text data queried by the AI data storage model according to the query instruction;

[0070] decompressing the generated algorithm text data queried by the data generator, and executing the generated algorithm according to the decompressed generated algorithm text data to generate the vehicle data to be queried, and returning the generated vehicle data to the external data service querying the vehicle data.

[0071] In the example embodiments of the present application, according to the query instruction, the Dcgan algorithm or the MAE algorithm (or other similar algorithms) can be executed to generate data equivalent to the original vehicle data, and the generated data can be verified according to the algorithm parameters.

[0072] In the example embodiments of the present application, after the vehicle data to be queried is generated by the preset data generator according to the query request, the generated algorithm, and the corresponding generated algorithm text data stored in the AI data storage model, the method can further include:

[0073] discriminating and correcting the generated vehicle data by a preset discriminator.

[0074] In the example embodiments of the present application, the generated data can be discriminated and corrected again by the discriminator to further improve the consistency with the original vehicle data.

[0075] In the example embodiments of the present application, the discriminator can use a sample data model in the Dcgan algorithm system to discriminate the consistency of the generated data with the original vehicle data.

[0076] In the example embodiments of the present application, at least the following advantages are included:

[0077] 1. The storage space saving rate is very high, whether it is 10MB of data or 100MB of data, the size of the generated algorithm text data is about 1MB, and the storage space saving rate is much higher than the traditional storage method, especially in the intelligent driving field with large amount of non-structured data.

[0078] 2. The storage space saving rate is not dependent on or for specific file types, whether it is ordinary text data, video data, picture data, voice data, radar data, etc., corresponding generated algorithm text data can be generated.

[0079] The present application also provides a data storage device 1, as shown in Figure 4As shown, the apparatus can include a processor 11 and a computer readable storage medium 12 having instructions stored therein that, when executed by the processor 11, implement the data storage method.

[0080] In the exemplary embodiments of the present application, any of the above-mentioned data storage methods are applicable to the apparatus embodiments, and thus are not repeated here.

[0081] Those of ordinary skill in the art will understand that all or some of the steps in the above-disclosed methods and the functional modules / units in the systems and apparatuses can be implemented as software, firmware, hardware, or appropriate combinations thereof. In a hardware implementation, the division of the functional modules / units between the above description can not necessarily correspond to physical divisions of the components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components working together. Some or all of the components can be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on computer readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer. Further, it should be understood by those of ordinary skill in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

Claims

1. A data storage method, characterized by, The method comprises: acquiring vehicle data to be stored; acquiring generation algorithm text data required for generating the vehicle data by using a preset generation algorithm; compressing the generation algorithm text data; storing the generation algorithm text data into a preset artificial intelligence (AI) data storage model; when the vehicle data needs to be queried, generating the vehicle data to be queried by a preset data generator according to a query request, the generation algorithm, and corresponding generation algorithm text data stored in the AI data storage model; judging and correcting the generated vehicle data by a preset discriminator; wherein the vehicle data comprises one or more of the following: video data, picture data, voice data, and radar data; the generation algorithm comprises a deep convolutional generative adversarial network (Dcgan) algorithm and / or a mask autoencoder (MAE) algorithm; the compression of the generation algorithm text data comprises lossless compression or lossy compression of the generation algorithm text data.

2. The data storage method of claim 1, wherein, The method further comprises: pre-acquiring generation algorithm text data of various vehicle data as training data; training a pre-created neural network model for storing data by using the training data to obtain an AI data storage model.

3. The data storage method of claim 1, wherein, The generation of the vehicle data to be queried by the preset data generator according to the query request, the generation algorithm, and the corresponding generation algorithm text data stored in the AI data storage model comprises: sending a query instruction to the AI data storage model by the data generator according to the query request, and acquiring generation algorithm text data queried by the AI data storage model according to the query instruction; decompressing the acquired generation algorithm text data by the data generator, executing the generation algorithm according to the decompressed generation algorithm text data to generate the vehicle data to be queried, and returning the generated vehicle data to an external data service querying the vehicle data.

4. The data storage method of claim 1, wherein, The discriminator uses a sample data model in the Dcgan algorithm system.

5. The data storage method according to claim 1 or 2, wherein, The acquisition of the vehicle data to be stored comprises: transmitting vehicle data collected by a vehicle end to a preset cloud data buffer through data express or network transmission according to a predetermined data reporting frequency.

6. A data storage device, characterized by A device comprises a processor and a computer readable storage medium, wherein the computer readable storage medium stores instructions which, when executed by the processor, implement the data storage method of any one of claims 1-5.

Images