Data storage optimization method and system based on data compression

Abstract

The invention relates to the technical field of data storage optimization, in particular to a data storage optimization method and system based on data compression, and the method comprises the steps: obtaining data collected by a robot, carrying out the preprocessing operation of the collected data, carrying out the first byte stream reading operation of image data, and carrying out the second byte stream reading operation of the image data; performing format judgment on the read first byte stream data, selecting a corresponding first decoding function for the determined format, and converting the first byte stream data into an image matrix by adopting the first decoding function; selecting an image compression level, and compressing the image matrix according to the image compression level to obtain compressed image data; compressing the depth data to obtain compressed depth data; and storing the compressed image data, the compressed depth data, the robot action data and the robot task instruction data in an HDF5 file. The method has the advantages of reducing the storage cost and standardizing the data format.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage optimization, and in particular to a data storage optimization method and system based on data compression. Background Art

[0002] In the field of data storage and management, especially in scenarios involving large amounts of images and multi-type data, there are many problems:

[0003] 1. High storage costs: Uncompressed raw data files, such as images stored in nparray, take up a lot of space. For example, one file was 339.9MB. This not only increases storage device costs but also negatively impacts data transmission and processing efficiency.

[0004] 2. Inconsistent data formats: The lack of standardized data storage formats makes it difficult to share and interact with data across different systems and applications. Data from different sources may have different structures and encoding methods, increasing the difficulty of data integration and analysis.

[0005] 3. Challenges in Using HDF5: While HDF5 is a powerful data storage format, it presents several challenges in practical applications. The learning and usage barriers are high, and for beginners and non-computer science professionals, mastering its complex API and data organization requires significant time and effort. Compatibility issues exist between different versions of the HDF5 library, and portability across different operating systems and computer architectures is also limited, hindering file portability and data exchange and sharing. Summary of the Invention

[0006] In order to solve the deficiencies of the prior art, the present invention provides a data storage optimization method and system based on data compression;

[0007] In one aspect, a data storage optimization method based on data compression is provided, comprising:

[0008] Acquire the data collected by the robot, perform preprocessing operations on the collected data, and divide the collected data into image data, depth data, robot motion data, and robot task instruction data;

[0009] Performing a first byte stream reading operation on the image data, determining the format of the read first byte stream data, selecting a first decoding function corresponding to the determined format, and converting the first byte stream data into an image matrix using the first decoding function; selecting an image compression level, compressing the image matrix according to the image compression level, and obtaining compressed image data; performing a second byte stream reading operation on the depth data, selecting a second decoding function corresponding to the format of the read second byte stream, and decoding the second byte stream data into a depth data matrix using the second decoding function; selecting a lossless compression algorithm, and compressing the depth data matrix according to the selected lossless compression algorithm, and obtaining compressed depth data;

[0010] The compressed image data, compressed depth data, robot motion data, and robot task instruction data are stored in HDF5 files.

[0011] On the other hand, a data storage optimization system based on data compression is provided, including:

[0012] an acquisition module configured to: acquire data collected by the robot, perform preprocessing operations on the collected data, and divide the collected data into image data, depth data, robot motion data, and robot task instruction data;

[0013] a compression module configured to: perform a first byte stream reading operation on the image data, determine the format of the read first byte stream data, select a first decoding function corresponding to the determined format, and convert the first byte stream data into an image matrix using the first decoding function; select an image compression level, and compress the image matrix according to the image compression level to obtain compressed image data;

[0014] performing a second byte stream reading operation on the depth data, selecting a second decoding function corresponding to the format of the read second byte stream, and decoding the second byte stream data into a depth data matrix using the second decoding function; selecting a lossless compression algorithm, and compressing the depth data matrix according to the selected lossless compression algorithm to obtain compressed depth data;

[0015] The storage module is configured to store the compressed image data, the compressed depth data, the robot motion data and the robot task instruction data in an HDF5 file.

[0016] The above technical solution has the following advantages or beneficial effects:

[0017] Reduce storage costs: Use appropriate compression algorithms to reduce the space required for data storage while ensuring the integrity of data information or an acceptable degree of loss.

[0018] Standardize data formats: Develop a unified data standard format to ensure consistency in data storage, transmission, and use, facilitating data management, sharing, and analysis.

[0019] Optimize HDF5 applications: To address issues encountered in HDF5 usage, provide more efficient, convenient, and stable HDF5 data processing solutions, lower the usage threshold, and improve compatibility, performance, data security, and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0021] Figure 1 This is a flow chart of the method of embodiment 1. DETAILED DESCRIPTION

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0023] Explanation of terms:

[0024] 1.nparray: numpy array, an efficient data structure for processing numerical data in Python.

[0025] 2.HDF5: A file format for storing and managing large-scale scientific data, supporting efficient data reading and writing and complex data organization.

[0026] 3. RGB camera: A camera used to collect red, green, and blue color information and obtain color image data.

[0027] 4. Depth data: Data representing the distance between an object and the camera, often used to obtain three-dimensional information of a scene.

[0028] Example 1

[0029] This embodiment provides a data storage optimization method based on data compression;

[0030] like Figure 1 As shown, the data storage optimization method based on data compression includes:

[0031] S101: Acquire data collected by the robot, perform preprocessing operations on the collected data, and divide the collected data into image data, depth data, robot motion data, and robot task instruction data;

[0032] S102: performing a first byte stream reading operation on the image data, determining the format of the read first byte stream data, selecting a first decoding function corresponding to the determined format, and converting the first byte stream data into an image matrix using the first decoding function; selecting an image compression level, and compressing the image matrix according to the image compression level to obtain compressed image data;

[0033] performing a second byte stream reading operation on the depth data, selecting a second decoding function corresponding to the format of the read second byte stream, and decoding the second byte stream data into a depth data matrix using the second decoding function; selecting a lossless compression algorithm, and compressing the depth data matrix according to the selected lossless compression algorithm to obtain compressed depth data;

[0034] S103: Storing the compressed image data, compressed depth data, robot motion data, and robot task instruction data into an HDF5 file.

[0035] Furthermore, S101: obtaining data collected by the robot and performing preprocessing operations on the collected data, including:

[0036] Delete the data of the file that occupies zero memory size;

[0037] Delete data whose file format is unrecognizable;

[0038] Based on the file suffix, determine whether the data classification is correct.

[0039] It should be understood that the system continuously monitors various data acquisition devices, such as RGB cameras and depth sensors, as well as designated data storage directories. These devices and directories serve as the primary sources of data. Once new data is generated or transmitted to the storage directory, the system immediately initiates the data processing process.

[0040] Initial data screening: When new data is detected, the system first obtains basic information about the data file, including file name, file size, creation time, etc. Based on pre-set rules, the data is initially screened to exclude data that clearly does not meet the requirements, such as data with a file size of 0 or an unrecognizable file format.

[0041] Classification feature extraction: After preliminary screening, the system extracts data features for classification. The system analyzes the file storage path, looking for specific identifiers, such as "chest_color," which may indicate RGB image data, or "chest_depth," which may indicate depth data. The system also checks file format suffixes, such as .jpg and .png, to help determine image data. The system also reads the file header, which may contain key information such as the data type and encoding method, to further determine the data type.

[0042] Furthermore, the S101: dividing the collected data into image data, depth data, robot motion data and robot task instruction data includes:

[0043] If the collected data file format meets the RGB image characteristics, the collected data is determined to be RGB image data; the RGB image characteristics include: the file format suffix is the image format suffix, or the file header information is image data;

[0044] If the collected data format meets the depth data characteristics, the collected data is determined to be depth data; the depth data characteristics include: the data format suffix is the depth data file format, or the file header information indicates that it is depth data; the depth data file format is RGBD; the file header information indicates that it is depth data, for example: the first 13 bytes in the PNG format contain a magic number and an IHDR data block;

[0045] If the collected data contains a robot motion data identifier, the collected data is determined to be robot motion data; the robot motion data identifier includes: a data structure in the file content that contains information consistent with the posture of the robot arm and the gripper;

[0046] If the collected data is a task content described in natural language and complies with the format specifications of task instruction data, the collected data is determined to be robot task instruction data; the format specifications of task instruction data include: action and object descriptions.

[0047] Furthermore, the step S102 of performing a first byte stream reading operation on the image data includes:

[0048] The image data is opened in binary mode, and the file content is read byte by byte to obtain the first byte stream data.

[0049] It should be understood that for files determined to be RGB image data, the system opens the file in binary mode and reads the file content byte by byte to obtain byte stream data. This is the basis for subsequent processing to ensure that the complete image data information is obtained.

[0050] Furthermore, the step S102 of determining the format of the read first byte stream data, selecting a first decoding function corresponding to the determined format, and converting the first byte stream data into an image matrix using the first decoding function includes:

[0051] Determine the format of the first byte stream data;

[0052] If the format of the first byte stream data is png format, call cv2.imdecode function to convert the byte stream into an image matrix.

[0053] It should be understood that cv.imdecode is an OpenCV function that decodes image data in memory into a numpy.ndarray. The buf parameter is a byte array containing the image data, and flags determines the color and depth mode of the decoded image, such as color or grayscale. Success returns an image array; failure returns None. The image matrix is a three-dimensional array containing the image's height, width, and color channel information.

[0054] It should be understood that the characteristics of the byte stream data are analyzed to determine its original encoding format, such as common formats such as PNG and JPEG. Based on the determined format, an appropriate decoding library is selected. For example, using OpenCV's cv2 library, if the PNG format is used, the cv2.imdecode function is called to convert the byte stream into an image matrix for easy processing. The image matrix is typically a three-dimensional array containing the image's height, width, and color channel information, providing structured data for subsequent compression operations.

[0055] Furthermore, the step S102: selecting an image compression level includes:

[0056] If the size of the compressed image in memory is required to be no larger than the first set threshold, and the compression time is required to be longer than the second set threshold, then the compression level is selected as 7 or 8;

[0057] If the required compression duration is less than the third set threshold, the compression level 3 or 4 is selected.

[0058] It should be understood that the image matrix is compressed using the lossless PNG format. Before compression, the system selects an appropriate compression level based on the requirements of the actual application scenario. The compression level generally ranges from 1 to 9, with higher values resulting in a higher compression ratio but also longer compression time. For example, in image storage scenarios where storage volume requirements are extremely high and compression time requirements are not stringent, a higher compression level, such as 7 or 8, can be selected; in real-time image processing scenarios where processing speed requirements are high, a lower compression level, such as 3 or 4, can be selected.

[0059] Furthermore, the S102: compressing the image matrix according to the image compression level to obtain compressed image data;

[0060] After the compression level is selected, the image matrix is compressed according to the compression function to obtain compressed image data.

[0061] It should be understood that after selecting a compression level, the system compresses the image matrix using relevant functions (such as cv2.imencode) and converts it into a PNG format byte stream. During the compression process, the function optimizes the image data according to the selected compression level, removing redundant information and thus reducing the file size.

[0062] Furthermore, performing a second byte stream reading operation on the depth data includes: opening the depth data in a binary mode, and performing a byte stream reading operation on the depth data.

[0063] Furthermore, selecting a corresponding second decoding function for the format of the read second byte stream includes:

[0064] If the second byte stream data format is PNG format, use the Pillow library to read the PNG image to obtain the depth data, and then use the numpy library to convert it into a depth data matrix.

[0065] Furthermore, the adopting of the second decoding function to decode the second byte stream data into a depth data matrix includes:

[0066] If the second byte stream data format is PNG, in the decode_png_depth_data function, BytesIO(byte_stream) converts the byte stream into a file object for the Pillow library to read, np.frombuffer(depth_info, dtype = np.uint16) converts the byte data in the custom data block into a numpy array, and reshape((height, width)) converts the one-dimensional array into a two-dimensional depth data matrix according to the actual image height and width.

[0067] The decode_png_depth_data function processes the PNG-formatted depth data byte stream and converts it into a two-dimensional depth data matrix suitable for subsequent analysis. First, BytesIO(byte_stream) converts the byte stream into a file object, making it easier for the Pillow library to read the data. Next, np.frombuffer(depth_info, dtype = np.uint16) extracts the bytes from the data block as unsigned 16-bit integers and converts them into a one-dimensional NumPy array. Finally, reshape((height, width)) reshapes the one-dimensional array into a two-dimensional depth data matrix based on the actual image height and width, representing depth information at different locations in the scene.

[0068] BytesIO (byte_stream) is a class provided by the Python io module that creates a file-like object in memory. byte_stream represents the incoming byte stream data, which in this case represents a sequence of PNG-formatted depth data bytes. Using BytesIO (byte_stream), you can wrap the byte stream into a file object, allowing the Pillow library to read the data as if it were a regular file. This facilitates subsequent parsing and processing of the PNG-formatted depth data, avoids the complexity of directly manipulating the byte stream, and improves the flexibility and efficiency of data processing.

[0069] The NumPy function np.frombuffer(depth_info, dtype = np.uint16) converts the byte sequence depth_info storing depth information into a one-dimensional NumPy array of unsigned 16-bit integers. It parses the data in np.uint16 format, combining two bytes into an integer. When processing PNG depth data, this function converts extracted custom data blocks into a numeric array that can be further processed, meeting the requirements of depth measurement data processing.

[0070] reshape((height, width)) is a NumPy array method used to change the shape of an array. Height and width represent the height and width of the converted two-dimensional array, respectively. When processing depth data, after the np.frombuffer function converts the byte data into a one-dimensional NumPy array, it is necessary to use reshape((height, width)) to rearrange the one-dimensional array into a two-dimensional depth data matrix based on the actual height and width of the image. This way, each element corresponds to the depth value of a pixel in the image, facilitating subsequent operations such as depth analysis and visualization.

[0071] It should be understood that for depth data files, the system also opens the file in binary mode to read the byte stream data. Depending on the encoding format of the depth data (which may be a specific lossless or lossy format, which needs to be pre-configured or determined through file header information), the corresponding decoding tool (such as the decoding function for the specific depth data format in the cv2 library) is used to decode the byte stream into a depth data matrix. The depth data matrix is typically a two-dimensional array, and the element values represent the depth information at different positions. Accurate decoding is a key step in preserving the accuracy of the depth data.

[0072] Furthermore, selecting a lossless compression algorithm includes:

[0073] For depth data with a continuous similarity value data amount greater than a fourth set threshold, the selected lossless compression algorithm is the LZW algorithm;

[0074] For deep data whose data distribution does not have continuous similar value clustering features and whose data features are relatively random and uniform, the DEFLATE algorithm is selected as the lossless compression algorithm.

[0075] It should be understood that the LZW (Lempel-Ziv-Welch) algorithm is a dictionary-based data compression algorithm. During the compression process, it constructs a dictionary that stores the correspondence between strings and codes. Initially, the dictionary contains all individual characters and their corresponding codes. When scanning data, if a string already in the dictionary is encountered, the scan continues backward; if a string not in the dictionary is encountered, the code of the previous string is output and the new string is added to the dictionary. For example, for the string "abababc", the initial dictionary contains {'a':0,'b':1}. When "ab" is scanned, it is not in the dictionary, so the code 0 of "a" is output and "ab" is added to the dictionary (assuming the code is 2); then "ab" is scanned and the code 2 is output; then "abc" is scanned again, which is not in the dictionary, so the code 2 of "ab" is output and "abc" is added to the dictionary (assuming the code is 3), and so on, ultimately achieving data compression.

[0076] The DEFLATE algorithm is a lossless data compression algorithm that combines the LZ77 algorithm and Huffman coding. First, the LZ77 algorithm performs a match search on the data, replacing it with pointers to the previously encountered data and the remaining unmatched characters. Huffman coding then encodes these pointers and characters, assigning shorter codes to frequently occurring symbols and longer codes to less frequently occurring symbols, thereby achieving data compression. For example, the LZ77 algorithm might convert the data "aaaaabbbccd" into a representation similar to (5,0,'a'),(3,0,'b'),(2,0,'c'),'d', which is then further compressed using Huffman coding.

[0077] It should be understood that the depth data matrix is compressed using the tif byte stream lossless compression method. Among the many lossless compression algorithms, such as LZW and DEFLATE, the appropriate algorithm is selected based on the data characteristics and performance requirements.

[0078] Furthermore, compressing the depth data matrix according to the selected lossless compression algorithm to obtain compressed depth data includes:

[0079] Use the cv2.imencode function to convert the depth data matrix into tif byte stream data.

[0080] During compression, the function processes the depth data matrix according to the selected algorithm to compress the data.

[0081] When the LZW algorithm is selected, the data compression encoding method is the dictionary construction and encoding process of the LZW algorithm, which converts the data into the encoded output in the dictionary;

[0082] When the DEFLATE algorithm is selected, the data compression encoding method is to first use the LZ77 algorithm for data matching and pointer replacement, and then use Huffman coding for symbol encoding.

[0083] These encoding methods are designed to reduce the space required for data storage while ensuring that depth information is not lost. They ensure that the depth information is not changed during the compression process, maintaining the integrity and accuracy of the data.

[0084] It should be understood that the cv2.imencode function is used to encode image data into a memory buffer (byte stream) in a specified format. The function prototype is cv2.imencode(ext,img[,params]), where ext is the specified file extension (such as .tif), indicating the image format to be encoded; img is the image data to be encoded, or the depth data matrix when processing depth data; params is an optional parameter used to specify specific encoding parameters, such as compression quality. When processing depth data, cv2.imencode('.tif',depth_matrix) encodes the depth data matrix according to the selected lossless compression algorithm (such as LZW or DEFLATE), converting it into byte stream data that conforms to the TIFF format specification for storage or transmission.

[0085] It should be understood that TIFF (Tagged Image File Format) byte stream data is binary data encoded according to the TIFF file format specification. The TIFF format supports a variety of data types, image compression algorithms, and image layouts. For depth data, taking the tif byte stream data generated by the cv2.imencode function as an example, it contains the image metadata (such as image size, data type, etc.) and the compressed and encoded depth data content. These byte stream data can be stored directly as a TIFF file, or they can exist in the form of a byte stream in network transmission or memory, and can be restored to a depth data matrix through corresponding decoding operations when needed.

[0086] Furthermore, the step S103 of storing the compressed image data, compressed depth data, robot motion data, and robot task instruction data into an HDF5 file includes:

[0087] (3-1): Create an HDF5 file, create a dataset in the HDF5 file, and assign different types of data to corresponding datasets;

[0088] (3-2): Evaluate the data to be stored and select the corresponding compression algorithm and compression level based on the evaluation results;

[0089] (3-3): Store the data into the corresponding dataset according to the selected compression algorithm and compression level;

[0090] (3-4): Assign a unique identifier to each data object and store different types of data in different groups.

[0091] It should be understood that to create an HDF5 file, the relevant libraries must be introduced: Before data storage operations, the system ensures that Python's h5py library and numpy library have been introduced in the runtime environment. The h5py library is used to operate HDF5 files, and the numpy library is used to process numerical data, providing the necessary functional support for subsequent data storage and processing. To create an HDF5 file object: Use the File function of the h5py library to create an HDF5 file object, specifying the file name and open mode. For example, "with h5py.File('example.hdf5','w')as f:" creates a file object "f" named "example.hdf5" in write mode, providing a container for data storage.

[0092] Furthermore, the (3-1) allocates different types of data to corresponding data sets, including:

[0093] Write the compressed RGB image data into the "observations / RGB" dataset;

[0094] Write depth data to the "observations / Depth" dataset;

[0095] For robot action data, according to its data format: the robot arm pose data is in the form of [x (m), y (m), z (m), roll (angle), pitch (angle), yaw (angle)], the data type is 32-float, and shape = (n, 6) is written to the "action / robot_arm / arm" dataset;

[0096] Write the task instruction data (including Chinese and English expressions) into the "instruction" dataset and its sub-datasets "instruction_Chinese" and "instruction_English".

[0097] It should be understood that according to the planned dataset structure, the corresponding dataset is created in the HDF5 file using the create_dataset function of the h5py library. For example, "f.create_dataset('observations / RGB', data = rgb_data)" writes the compressed RGB image data "rgb_data" to the "observations / RGB" dataset, ensuring that the data is stored in a standard format.

[0098] Furthermore, the step (3-2): evaluating the data to be stored includes:

[0099] Evaluate the data type (numeric, image, or other types), data volume, data accuracy requirements, and subsequent data usage scenarios (such as whether frequent reading is required and whether high real-time requirements are required).

[0100] It should be understood that the HDF5 library provides a variety of data compression algorithms, such as gzip and szip. Before selecting an algorithm, evaluate the characteristics of the data to be stored, including factors such as the data type (numeric, image, or other type), the data size, the data accuracy requirements, and the subsequent use scenarios of the data.

[0101] Furthermore, (3-2): selecting a corresponding compression algorithm and compression level according to the evaluation result includes:

[0102] For scientific experimental data with accuracy requirements higher than the set threshold, choose the lossless compression algorithm gzip and set a high compression level (such as level 9) to ensure data integrity;

[0103] For data with high storage space requirements and a set threshold, and for scenarios with accuracy requirements lower than the set threshold, select the lossy but high compression ratio algorithm szip and set the compression level to 5.

[0104] After selecting the compression algorithm and level, apply these settings to the corresponding dataset when storing data. For example, specifying the compression algorithm and level through parameters when creating a dataset, "f.create_dataset('observations / RGB',data=rgb_data,compression='gzip',compression_opts=9)", ensures that the data is optimally compressed when stored.

[0105] Furthermore, the step (3-4): assigning a unique identifier to each data object includes:

[0106] To facilitate data management and retrieval, the system assigns a unique identifier (GUID) to each data object. For common data objects, GUIDs are generated using Python's uuid library. For example, using the statement "import uuid; guid = uuid.uuid4()" generates a unique GUID. This GUID serves as the identifier for the data object, facilitating subsequent rapid data location and management.

[0107] Furthermore, the (3-4) also includes: special processing of channel data: for channel data (a special case of robot motion data), due to its large amount of data, in order to save space, a GUID is not assigned to each channel data, but a GUID is assigned and saved to its parent object data table.

[0108] When referencing a channel, use the table object's GUID plus the channel's index in the table to locate it. For example, if the table "table1" has a GUID of "123e4567-e89b-12d3-a456-426614174000" and the channel "channel1" has an index of 0 in the table, then use "123e4567-e89b-12d3-a456-426614174000_0" to reference the channel.

[0109] Furthermore, the (3-4) stores different types of data into different groups.

[0110] Use the grouping and attribute functions of HDF5 files to classify and describe data. Create different groups to store different types of data. "Group 1" stores the data of Experiment 1, and "Group 2" stores the data of Experiment 2, which facilitates data organization and management.

[0111] Add attributes to the data object, such as "f['observations / RGB'].attrs['description'] = 'RGB images captured by camera1'", to add description attributes to the "observations / RGB" dataset to improve the readability and maintainability of the data.

[0112] Furthermore, the method further comprises:

[0113] S104: Create a README file for explanation.

[0114] Furthermore, the S104: creating an explanation README file includes:

[0115] (4-1): Determine the location of the README file;

[0116] (4-2): Fill in the file address information in the README file;

[0117] (4-3): Record environment information and device information in the README file;

[0118] (4-4): Describe the task content in the README file;

[0119] (4-5): Describe the data details in the README file;

[0120] (4-6): Evaluate data quality and score in the README file;

[0121] (4-7): Perform data verification operations in the README file.

[0122] Furthermore, (4-1): determining the location of the README file includes creating a README file (in JSON format) for each subtask and each group. The storage location of the README file is determined based on the data storage structure. For example, a README.json file is created in the "group_0" subfolder of the "robot_task_data" folder to ensure that the README file is closely associated with the corresponding data file.

[0123] Furthermore, (4-2): fill in the file address information in the README file, including:

[0124] Fill in the HDF5 file address in the README file and record the data storage location according to a unified format specification (such as "{Robot_Dataset} / {Task} / {subtask} / {group} / {subtask-group-episode.hdf5}") to facilitate users to quickly find the corresponding HDF5 file.

[0125] Furthermore, (4-3): recording the environment information and device information in the README file includes:

[0126] Indicate the collection environment, such as "real" for data collected in a real environment, and "sim" and the specific sim environment, such as "sim_unity" for data collected in the Unity simulation environment; record the type of robotic arm or humanoid robot, such as "JAKA", so that users can understand the data collection background and device information.

[0127] Furthermore, (4-4): describing the task content in the README file includes:

[0128] Describe the task content in detail, such as "Move the blue rectangle a little to the right of the pink sphere", so that other users can quickly understand the task scenario corresponding to the data and facilitate data analysis and application.

[0129] Furthermore, (4-5) above: describe the data details in the README file, including:

[0130] Detailed description of each data type, including the specific format and meaning of actions, observations, instructions, and flags. For example, "robot_arm data in action represents the position of the robot arm, with x, y, and z as coordinates and roll, pitch, and yaw as attitude angles." This provides detailed guidance for data usage, making it easier for users to understand and use the data.

[0131] Furthermore, (4-6): Evaluate and score the data quality in the README file, including:

[0132] Data quality is assessed and scored (out of 100), such as "85." Scoring is based on multiple factors, including accuracy, completeness, and consistency, to help users quickly assess the reliability and usability of data and decide whether to use it for subsequent analysis and research.

[0133] Furthermore, the step (4-7): performing data verification in the README file includes:

[0134] Data integrity verification: check whether there are missing values in the data set. Compare the number of statistical records with the expected number. If the numbers are inconsistent, there may be missing values. Check whether the size of the data file is within the expected range. If the file size is abnormal, the data may be damaged.

[0135] Data accuracy verification: For depth data, you can select some objects with known distances, compare the actual depth data with the true distance, and calculate the error range. If the error exceeds the acceptable range, there is a problem with the data accuracy.

[0136] For robot motion data, check whether the pose information is within a reasonable physical range, such as whether the coordinate values of the robot arm exceed its workspace. Through these verification operations, ensure the quality and reliability of the data, and record the verification results in the README file.

[0137] Save storage resources: After compression, data storage space is significantly reduced. For example, the compressed episode 0 file is 26MB, an order of magnitude smaller than the uncompressed version, effectively reducing storage costs.

[0138] Improve data management and sharing efficiency: A unified data standard format makes data interaction between different systems and applications more convenient, reduces format conversion and compatibility issues during data processing, and facilitates data integration, analysis, and sharing.

[0139] Optimizing the HDF5 Application Experience: This solution addresses issues encountered with HDF5 usage by providing a simpler, easier-to-use approach, lowering the learning curve and user experience. By selecting appropriate compression algorithms and optimizing data storage structures, HDF5 improves data processing performance, reduces compression and decompression time, and enhances performance in multi-threaded and distributed computing environments. Furthermore, data security and reliability are enhanced through data validation and improved management mechanisms.

[0140] You can try using other image compression algorithms instead of PNG and tif compression, such as WebP, PNG-LZ77, and other algorithms. WebP may have an advantage in compression ratio, but its compatibility with some older systems may be poor. In some older embedded devices or specific operating systems, WebP format decompression may not be directly supported, requiring additional conversion or adaptation work. PNG-LZ77 may provide better compression effects in specific scenarios, but it may have certain issues with compression speed or implementation complexity. When processing large amounts of data, its compression speed may be slow, affecting the efficiency of data storage. When choosing an alternative algorithm, it is necessary to comprehensively compare its compression ratio, compression speed, data loss, and compatibility with different systems, and make a reasonable choice based on the needs of the actual application scenario.

[0141] Explore the use of other data formats as alternatives to HDF5, such as NetCDF and Parquet. NetCDF is widely used in fields such as meteorology and oceanography and has good cross-platform capabilities, but it may not be as flexible as HDF5 in processing complex data structures and supporting multiple data types. For example, for some data sets containing multiple data types (such as images, text, and numerical data), the storage and management of NetCDF may be relatively complex. Parquet performs well in big data processing frameworks and is suitable for the storage and analysis of large-scale data, but it may lack data reading and writing flexibility and efficiency in processing small data volumes. When processing small amounts of data, Parquet may incur high storage overhead, and its read and write operations may be relatively complex. However, these formats differ from HDF5 in terms of supported functions, data reading and writing efficiency, and data structure organization, and require detailed evaluation and selection based on specific application scenarios, such as data types, processing requirements, and usage environment.

[0142] Example 2

[0143] This embodiment provides a data storage optimization system based on data compression, including:

[0144] an acquisition module configured to: acquire data collected by the robot, perform preprocessing operations on the collected data, and divide the collected data into image data, depth data, robot motion data, and robot task instruction data;

[0145] a compression module configured to: perform a first byte stream reading operation on the image data, determine the format of the read first byte stream data, select a first decoding function corresponding to the determined format, and convert the first byte stream data into an image matrix using the first decoding function; select an image compression level, and compress the image matrix according to the image compression level to obtain compressed image data;

[0146] performing a second byte stream reading operation on the depth data, selecting a second decoding function corresponding to the format of the read second byte stream, and decoding the second byte stream data into a depth data matrix using the second decoding function; selecting a lossless compression algorithm, and compressing the depth data matrix according to the selected lossless compression algorithm to obtain compressed depth data;

[0147] The storage module is configured to store the compressed image data, the compressed depth data, the robot motion data and the robot task instruction data in an HDF5 file.

[0148] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A data storage optimization method based on data compression, characterized in that: include: Acquire the data collected by the robot, perform preprocessing operations on the collected data, and divide the collected data into image data, depth data, robot motion data, and robot task instruction data; Performing a first byte stream reading operation on the image data, determining the format of the read first byte stream data, selecting a first decoding function corresponding to the determined format, and converting the first byte stream data into an image matrix using the first decoding function; selecting an image compression level, compressing the image matrix according to the image compression level, and obtaining compressed image data; performing a second byte stream reading operation on the depth data, selecting a second decoding function corresponding to the format of the read second byte stream, and decoding the second byte stream data into a depth data matrix using the second decoding function; Select a lossless compression algorithm, and compress the depth data matrix according to the selected lossless compression algorithm to obtain compressed depth data; The compressed image data, compressed depth data, robot motion data, and robot task instruction data are stored in HDF5 files.

2. The data storage optimization method based on data compression according to claim 1, characterized in that: Obtain the data collected by the robot and perform preprocessing operations on the collected data, including: Delete data whose memory size is zero; delete data whose file format cannot be recognized; determine whether the data classification is correct based on the file suffix.

3. The data storage optimization method based on data compression according to claim 1, characterized in that: The collected data is divided into image data, depth data, robot motion data and robot task instruction data, including: If the collected data file format meets the RGB image characteristics, the collected data is determined to be RGB image data; the RGB image characteristics include: the file format suffix is the image format suffix, or the file header information is image data; If the format of the collected data meets the depth data characteristics, the collected data is determined to be depth data; the depth data characteristics include: the data format suffix is a depth data file format, or the file header information indicates that it is depth data; If the collected data contains a robot motion data identifier, the collected data is determined to be robot motion data; the robot motion data identifier includes: a data structure in the file content that contains information consistent with the posture of the robot arm and the gripper; If the collected data is a task content described in natural language and complies with the format specifications of task instruction data, the collected data is determined to be robot task instruction data; the format specifications of task instruction data include: action and object descriptions.

4. The data storage optimization method based on data compression according to claim 1, characterized in that: Performing a first byte stream reading operation on the image data includes: The image data is opened in binary mode, and the file content is read byte by byte to obtain the first byte stream data.

5. The data storage optimization method based on data compression according to claim 1, characterized in that: Performing format determination on the read first byte stream data, selecting a first decoding function corresponding to the determined format, and converting the first byte stream data into an image matrix using the first decoding function, including: Determine the format of the first byte stream data; If the format of the first byte stream data is png format, call cv2.imdecode function to convert the byte stream into an image matrix.

6. The data storage optimization method based on data compression according to claim 1, characterized in that: Select the image compression level, including: If the size of the compressed image in memory is required to be no larger than the first set threshold, and the compression time is required to be longer than the second set threshold, then the compression level is selected as 7 or 8; If the required compression duration is less than the third set threshold, the compression level is selected as level 3 or 4; Compress the image matrix according to the image compression level to obtain compressed image data; After the compression level is selected, the image matrix is compressed according to the compression function to obtain compressed image data.

7. The data storage optimization method based on data compression according to claim 1, characterized in that: The performing a second byte stream reading operation on the depth data includes: opening the depth data in a binary mode, and performing a byte stream reading operation on the depth data; The selecting a corresponding second decoding function for the format of the read second byte stream includes: If the second byte stream data format is PNG format, use the Pillow library to read the PNG image to obtain the depth data, and then use the numpy library to convert it into a depth data matrix.

8. The data storage optimization method based on data compression according to claim 1, characterized in that: The selecting of the lossless compression algorithm comprises: For depth data with a continuous similarity value data amount greater than a fourth set threshold, the selected lossless compression algorithm is the LZW algorithm; For deep data with relatively random and uniform data features and no continuous similar value clustering, the DEFLATE algorithm is selected as the lossless compression algorithm; The compressing the depth data matrix according to the selected lossless compression algorithm to obtain compressed depth data includes: Use the cv2.imencode function to convert the depth data matrix into tif byte stream data; During compression, the function processes the depth data matrix according to the selected algorithm to compress the data; When the LZW algorithm is selected, the data compression encoding method is the dictionary construction and encoding process of the LZW algorithm, which converts the data into the encoded output in the dictionary; When the DEFLATE algorithm is selected, the data compression encoding method is to first use the LZ77 algorithm for data matching and pointer replacement, and then use Huffman coding for symbol encoding.

9. The data storage optimization method based on data compression according to claim 1, characterized in that: The compressed image data, compressed depth data, robot motion data, and robot task instruction data are stored in HDF5 files, including: (3-1): Create an HDF5 file, create a dataset in the HDF5 file, and assign different types of data to the corresponding datasets; (3-2): Evaluate the data to be stored and select the corresponding compression algorithm and compression level based on the evaluation results; (3-3): Store the data in the corresponding dataset based on the selected compression algorithm and compression level; (3-4): Assign a unique identifier to each data object and store different types of data in different groups; The method also includes: creating an explanatory README file; the creating of the explanatory README file includes: (4-1): determining the location of the README file; (4-2): filling in the file address information in the README file; (4-3): recording the environment information and the equipment information in the README file; (4-4): describing the task content in the README file; (4-5): describing the data details in the README file; (4-6): evaluating the data quality and scoring in the README file; (4-7): performing data verification operations in the README file.

10. A data storage optimization system based on data compression, characterized in that: include: an acquisition module configured to: acquire data collected by the robot, perform preprocessing operations on the collected data, and divide the collected data into image data, depth data, robot motion data, and robot task instruction data; a compression module configured to: perform a first byte stream reading operation on the image data, determine the format of the read first byte stream data, select a first decoding function corresponding to the determined format, and convert the first byte stream data into an image matrix using the first decoding function; select an image compression level, and compress the image matrix according to the image compression level to obtain compressed image data; Performing a second byte stream reading operation on the depth data, selecting a second decoding function corresponding to the format of the read second byte stream, and decoding the second byte stream data into a depth data matrix using the second decoding function; Select a lossless compression algorithm, and compress the depth data matrix according to the selected lossless compression algorithm to obtain compressed depth data; The storage module is configured to store the compressed image data, the compressed depth data, the robot motion data and the robot task instruction data in an HDF5 file.

Images