Data processing method and system based on Java service, terminal and storage medium

Abstract

The invention discloses a data processing method and system based on Java service, a terminal and a storage medium, and belongs to the technical field of data processing and storage. The method comprises the steps of receiving target to-be-uploaded data, and uploading a target to-be-uploaded file at the same time based on multiple threads in a Java service; extracting initial metadata of the target to-be-uploaded data, performing standardization processing on the initial metadata to obtain target metadata with uniform data format and code, and constructing a target index based on the target metadata; and classifying the target to-be-uploaded data based on the target index, and storing the target to-be-uploaded data in a corresponding data storage bucket. According to the invention, the efficiency, accuracy and reliability of file uploading processing are effectively improved, and the utilization of system resources is optimized.

Description

Technical Field

[0001] This invention relates to the field of data processing and storage technology, and in particular to a data processing method, system, terminal and storage medium based on Java services. Background Technology

[0002] Driven by the wave of digital transformation, enterprise applications are experiencing explosive growth in their demand for processing unstructured data. Traditional file storage solutions are gradually revealing systemic flaws when dealing with such scenarios: file systems based on single-machine architectures are limited by disk I / O throughput and storage capacity, resulting in single-file upload latency reaching the second level when handling high-concurrency upload tasks (such as tens of millions of users simultaneously uploading high-definition videos). More importantly, traditional solutions lack automated metadata management mechanisms, and file classification relies on manual configuration of path rules, leading to storage space utilization of less than 40%. Furthermore, data migration or expansion requires downtime for maintenance, making it difficult to meet business continuity requirements.

[0003] With the evolution of cloud computing technology, distributed object storage systems (such as Minio and AWS S3) have improved storage performance by more than 10 times through data sharding, erasure coding, and horizontal scaling architecture, with a single cluster supporting EB-level data capacity. However, how to achieve efficient integration with Minio in Java services and optimize performance and handle metadata during the upload process has become an urgent problem to be solved.

[0004] Therefore, there is an urgent need for a data management technology that combines Java services and efficient integrated storage to meet the requirements of efficient data processing and storage.

[0005] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0006] The main objective of this invention is to provide a data processing method, system, terminal, and storage medium based on Java services, aiming to solve the problem of the lack of effective management of file metadata processing and storage in the prior art.

[0007] To achieve the above objectives, a first aspect of the present invention provides a data processing method based on Java services, wherein the data processing method based on Java services includes: Receive the target data to be uploaded, and simultaneously upload the target file to be uploaded based on multi-threading in a Java service; Extract the initial metadata of the target data to be uploaded, standardize the initial metadata to obtain target metadata with unified data format and encoding, and build a target index based on the target metadata; The target data to be uploaded is categorized based on the target index and stored in the corresponding data storage bucket.

[0008] In one implementation, the simultaneous uploading of the target file to be uploaded based on multiple threads in a Java service includes: Obtain the target hardware configuration of the local server, and based on the target hardware configuration, configure a target number of threads to execute the upload of the target file to be uploaded.

[0009] In one implementation, extracting the initial metadata of the target data to be uploaded includes: The Java file operation class is used to obtain the file name, file size, creation time and modification time of the target data to be uploaded, and the file type of the target data to be uploaded is determined based on the file extension or file header information, so as to obtain the initial metadata.

[0010] In one implementation, the initial metadata is standardized, including: Convert the date and time information from different sources in the initial metadata into the target time format; Map category fields to predefined enumeration values; Perform unit normalization on numerical attributes; The character encoding of the initial metadata is detected using the Java standard library and re-encoded into UTF-8 format.

[0011] In one implementation, constructing a target index based on the target metadata includes: A directed graph model is used to construct the entity relationship network of the target metadata; The target index type is obtained based on the query scenario. The target index type is a combination of one or more local index types, including hash index, B+ tree index, inverted index, hierarchical index, and bitmap index. The entity relationship network is structured based on the target index type to obtain the target index.

[0012] In one implementation, classifying the target data to be uploaded based on the target index and storing it in the corresponding data storage bucket includes: Multiple target data storage buckets are constructed based on the target index type, and a mapping relationship is established between the target index and the multiple target data storage buckets. The target data storage buckets are divided into different levels of data storage buckets based on the index level of the target index. The target data to be uploaded is categorized based on the minimum level of the target index and stored in the corresponding target data storage bucket.

[0013] In one implementation, after storing the data in the corresponding data storage bucket, the method further includes: A metadata comparison task is triggered at each target time interval. After obtaining the actual metadata of the file from each of the data storage buckets, it is compared with the expected metadata recorded locally. If the actual metadata is found to be inconsistent with the expected metadata, a data repair operation is performed or an alarm notification is triggered.

[0014] In one implementation, before storing the data in the corresponding data storage bucket, the method further includes: When the Java service starts, the Minio client is initialized through the Minio Java SDK, the network address, access key and secret key parameters of the Minio server are configured, and a secure connection is established with the target Minio object storage system. The target Minio object is used as a storage container to store the target data to be uploaded.

[0015] In one implementation, after the receiving target has uploaded the data, it further includes: It provides real-time feedback on the upload progress, metadata extraction results, and storage status information of the target data to be uploaded.

[0016] A second aspect of the present invention provides a data processing system based on Java services, comprising: The file upload module is used to receive target data to be uploaded and upload the target file to be uploaded simultaneously using multi-threading in a Java service. The metadata processing module is used to extract the initial metadata of the target data to be uploaded, standardize the initial metadata to obtain target metadata with unified data format and encoding, and build a target index based on the target metadata; The storage module is used to classify the target data to be uploaded based on the target index and store it into the corresponding data storage bucket.

[0017] A third aspect of the present invention provides a terminal, wherein the terminal includes: a memory, a processor, and a Java service-based data processing program stored in the memory and executable on the processor, wherein the Java service-based data processing program, when executed by the processor, implements the steps of the Java service-based data processing method as described above.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer storage medium stores one or more programs that can be executed by one or more processors to implement the steps of the Java service-based data processing method described in any of the preceding claims.

[0019] Beneficial Effects: Compared with existing technologies, this invention provides a data processing method, system, terminal, and storage medium based on Java services. In the data processing method based on Java services provided by this invention, when managing IoT data, target data to be uploaded is received, and the target file is simultaneously uploaded using multiple threads in a Java service. Then, the initial metadata of the target data to be uploaded is extracted, standardized, and target metadata with unified data format and encoding is obtained. A target index is then constructed based on the target metadata. Finally, the target data to be uploaded is classified based on the target index and stored in corresponding data storage buckets. This invention provides users with a data processing method based on Java services, solving the problem of ineffective management of file metadata processing and storage in existing technologies. When a user uploads a file, multi-threaded concurrent file transfer is used, while simultaneously completing the standardized parsing and index construction of metadata to ensure efficient flow. Finally, a distributed storage system automatically sorts the files to corresponding storage buckets according to a preset strategy. This effectively improves the efficiency, accuracy, and reliability of file upload processing and optimizes system resource utilization. Attached Figure Description

[0020] Figure 1 A flowchart illustrating an embodiment of the Java service-based data processing method provided by the present invention; Figure 2 Metadata processing flowchart of the Java service-based data processing method provided by the present invention; Figure 3 A storage flowchart for the data processing method based on Java services provided by this invention; Figure 4 A schematic diagram illustrating the structural principle of an embodiment of the Java service-based data processing system provided by the present invention; Figure 5 A schematic diagram of the operating environment of an embodiment of the terminal provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0022] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0023] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0024] The present invention provides a data processing method based on Java services, which can be applied to terminals with computing capabilities. The terminal can execute the data processing method based on Java services provided by the present invention to manage and control edge sub-devices.

[0025] Example 1 This embodiment provides a data processing method based on Java services, belonging to the field of data processing and storage technology. It focuses on addressing the performance bottlenecks often encountered when uploading large-scale, diverse files, such as slow upload speeds and high server resource consumption. Furthermore, the lack of effective management of file metadata processing and storage makes it difficult to meet the demands for efficient data processing and storage.

[0026] In this embodiment, a Java-Minio integrated storage system for high-concurrency file processing scenarios is proposed. By optimizing the transmission protocol, reconstructing the metadata processing flow, and implementing an intelligent storage routing mechanism, the system systematically addresses the technical bottlenecks of traditional solutions in terms of performance, accuracy, and resource utilization.

[0027] The Java service-based data processing method provided in this embodiment can be applied to various Internet of Things (IoT) scenarios, such as smart homes, intelligent transportation, industrial IoT, and IoT healthcare.

[0028] Specifically, such as Figure 1 As shown, the data processing method based on Java services provided in this embodiment includes the following steps: S100: Receive the target data to be uploaded, and simultaneously upload the target file to be uploaded based on multi-threading in the Java service.

[0029] The simultaneous uploading of the target file to be uploaded using multiple threads in a Java service includes: Obtain the target hardware configuration of the local server, and based on the target hardware configuration, configure a target number of threads to execute the upload of the target file to be uploaded.

[0030] Specifically, when a customer needs to store the target data to be uploaded, the file will be uploaded based on the file upload module.

[0031] This embodiment supports a variety of common file formats. Specifically, it covers several major categories, including documents, images, audio, and video. By supporting these common file formats, users can easily upload various types of files to the system without worrying about format incompatibility issues, greatly improving the system's usability and user experience.

[0032] Specifically, upon receiving the target data to be uploaded, which could be various types of files such as documents, images, audio, or video, the system will initiate a multi-threaded mechanism within the Java service environment to simultaneously upload the target files.

[0033] Specifically, when multiple threads in a Java service simultaneously upload the target file, the target hardware configuration of the local server is first obtained. Specifically, various hardware parameters of the local server are thoroughly detected and collected, including but not limited to the number of CPU cores, memory capacity, and disk I / O performance. A comprehensive understanding of this hardware configuration information allows for an accurate assessment of the server's processing power and resource status.

[0034] After obtaining the target hardware configuration, the system intelligently configures multiple threads to execute the upload task of the target file based on the target hardware configuration information. For example, if the server has a large number of CPU cores and sufficient memory, the number of threads can be appropriately increased to improve upload efficiency; conversely, if the server's hardware resources are relatively limited, the number of threads will be reduced accordingly to avoid excessive resource consumption and system performance degradation. This method of dynamically adjusting the number of threads based on hardware configuration can fully utilize server resources and ensure the efficiency and stability of the file upload process. For example, for a server equipped with a 4-core CPU and 8GB of memory, after comprehensively considering various factors, 4-8 threads can be allocated to execute file upload tasks. This allocation strategy aims to fully utilize the server's computing and memory resources, avoiding excessive resource contention and high system load caused by too many threads, while also preventing the server's performance advantages from being fully utilized due to too few threads. By reasonably setting the number of threads, each thread can efficiently complete its part of the file upload work within its lifecycle, thereby significantly improving the overall file upload efficiency and providing users with a faster and smoother service experience.

[0035] In addition, this embodiment also includes a resume upload function, which is mainly achieved by accurately recording the file upload offset. Specifically, before officially starting the file upload operation, the system first checks whether there are any records related to the currently uploaded portion of the file. This checking mechanism is based on querying historical data records in the local storage area or a specific database table. If a corresponding record of the uploaded portion is detected, the system will accurately locate the position where the last file upload was interrupted based on the offset information in these records. Subsequently, the subsequent file content upload process will continue from that position, instead of re-uploading the entire file from the beginning. This method greatly reduces the amount of duplicate data transmission caused by network failures, temporary server failures, or other unexpected interruptions, effectively saving network bandwidth resources and user waiting time, improving the overall reliability and success rate of file uploads, and ensuring that even in the event of unforeseen circumstances, the file upload task can continue steadily until it is finally completed.

[0036] For example, if a user is uploading a large video file and the network connection drops halfway through for some reason, the resume upload feature allows the user to immediately continue uploading once the network is restored, without having to start over. They simply need to wait for the remaining data to complete. This significantly improves upload efficiency, especially when handling large files or in unstable network environments. Simultaneously, this feature enhances the system's reliability and fault tolerance, giving users greater peace of mind. Furthermore, in this embodiment, a user-friendly file upload interface is provided on the client side, allowing users to select local files for upload.

[0037] S200. Extract the initial metadata of the target data to be uploaded, standardize the initial metadata to obtain target metadata with unified data format and encoding, and build a target index based on the target metadata.

[0038] Specifically, refer to Figure 2 The step of extracting the initial metadata of the target data to be uploaded includes: The Java file operation class is used to obtain the file name, file size, creation time and modification time of the target data to be uploaded, and the file type of the target data to be uploaded is determined based on the file extension or file header information, so as to obtain the initial metadata.

[0039] The initial metadata is standardized, including: S210. Convert the date and time information from different sources in the initial metadata into the target time format; S220. Map the category field to a predefined enumeration value; S230. Perform unit normalization on numerical attributes; S240. Use the Java standard library to detect the character encoding of the initial metadata and re-encode it into UTF-8 format.

[0040] Specifically, in processing the target data to be uploaded, the first step is to extract the metadata of the target data. This step provides basic information for subsequent data processing and analysis. Specifically, in this embodiment, Java file operation classes are used to obtain key attributes of the target data to be uploaded, such as filename, file size, creation time, and modification time. Simultaneously, information related to the file content is also obtained, such as keywords in the document, image resolution, and audio duration, to gain a preliminary understanding of the file's basic information.

[0041] In addition to basic attributes, this embodiment also includes determining the file type based on the file extension or header information. This is because different file types may have different structures and uses, so accurate file type identification is crucial to ensuring the correctness of data processing. In this way, rich initial metadata can be extracted from the target data to be uploaded, laying the foundation for subsequent processing.

[0042] Then, to ensure data consistency and comparability, this embodiment also includes standardization processing of the initial metadata. Specifically, this includes converting date and time information from different sources into a unified target time format. This avoids data confusion caused by inconsistent date and time formats. Simultaneously, categorical fields are mapped and converted into predefined enumerated values. This allows previously scattered and diverse categorical information to be integrated into a limited number of categories, improving data readability and manageability. Furthermore, for numerical attributes, unit normalization is performed to ensure all values ​​are represented using the same units. This not only helps eliminate the impact of unit differences but also makes data analysis more accurate and reliable.

[0043] The final step is to process the character encoding. Since data from different sources may use different character encoding methods, this can lead to garbled text or incorrect display. Therefore, in this embodiment, the Java standard library is used to detect the character encoding of the initial metadata and re-encode it into UTF-8 format. UTF-8, as a widely used general-purpose character encoding standard, has good compatibility and stability, and can effectively solve text display problems in cross-platform and cross-language environments. Through the above series of standardization measures, the target metadata with unified data format and encoding is finally obtained, providing strong support for the further processing and application of the target data to be uploaded.

[0044] After obtaining the target metadata, the following steps are also included: Target index is constructed based on the target metadata.

[0045] Refer to this Figure 2 The construction of the target index based on the target metadata includes: S250. Construct the entity relationship network of the target metadata using a directed graph model; S260. Obtain the target index type according to the query scenario. The target index type is a combination of one or more local index types, including hash index, B+ tree index, inverted index, hierarchical index and bitmap index. S270. The entity relationship network is structured based on the target index type to obtain the target index.

[0046] Specifically, when constructing the target index based on the target metadata, a directed graph model is first used to build the entity relationship network of the target metadata. In this process, it is necessary to carefully analyze each element in the target metadata and its interrelationships. For example, in a dataset containing personal information, "name," "age," and "occupation" can all be considered entities, and the relationships between them are represented by specific edges, such as the connection "someone engages in a certain occupation." In this way, the originally scattered data points are connected into an organic whole, providing a clear structural foundation for subsequent processing.

[0047] Next, the target index type is obtained based on the query scenario. The target index type is a combination of one or more local index types. Specifically, in this embodiment, the local index types cover various common forms such as hash indexes, B+ tree indexes, inverted indexes, hierarchical indexes, and bitmap indexes. Different query scenarios have different requirements for the efficiency and methods of data retrieval, therefore, it is necessary to select appropriate combinations of index types. For example, when facing full-text retrieval of large-scale text data, it may be necessary to simultaneously use inverted indexes to quickly locate documents where keywords appear, and combine them with hierarchical indexes to optimize the loading performance of long lists, ensuring that users can smoothly browse search results.

[0048] Finally, the previously constructed entity relationship network is structured based on one or more selected target index types to obtain the final target index. This ensures that the generated target index can meet efficient access requirements for specific queries while maintaining good scalability and flexibility to adapt to constantly changing data environments and business logic.

[0049] Furthermore, in this embodiment, different indexing strategies are adopted for different application scenarios: for equality query scenarios, hash indexes are mainly used, which can complete accurate matching and searching in constant time by utilizing their unique key-value mapping characteristics; for range query scenarios, B+ tree indexes are preferred, as this tree structure can effectively support ordered traversal within the range and improve the speed of range scanning; as for full-text search scenarios, inverted indexes are usually selected, supplemented by hierarchical indexes to further improve the presentation efficiency of large result sets; in addition, when encountering low cardinality columns (i.e., the column has few value types), bitmap indexes become the ideal choice in this case due to their compact storage format and fast Boolean operation capabilities.

[0050] S300. Based on the target index, classify the target data to be uploaded and store it in the corresponding data storage bucket.

[0051] Before storing the data into the corresponding data storage bucket, the process also includes: When the Java service starts, the Minio client is initialized through the Minio Java SDK, the network address, access key and secret key parameters of the Minio server are configured, and a secure connection is established with the target Minio object storage system. The target Minio object is used as a storage container to store the target data to be uploaded.

[0052] Specifically, in Java services, interaction with the Minio object storage system requires the use of the Minio Java SDK. Minio is a high-performance, distributed object storage system suitable for storing massive amounts of unstructured data, such as images, videos, and log files. Its Java SDK provides developers with convenient access to Minio within a Java environment.

[0053] When a Java service starts, initializing the Minio client is a crucial first step. This is like building a bridge, enabling the Java service to communicate with the Minio server. During initialization, a series of parameters need to be configured accurately, the most important of which is the Minio server address. This specifies the target location, informing the client of the Minio server's network location and ensuring that requests are sent accurately to the correct server.

[0054] Access keys and secret keys are crucial credentials for authentication. The access key is similar to a username, identifying the user; the secret key is like a password, allowing only users with the correct keys to legitimately access resources in the Minio object storage system. Correct configuration of these parameters not only ensures a stable connection between the service and Minio but also guarantees the security and privacy of data transmission, preventing unauthorized access and data leakage. In this way, Java services can smoothly leverage Minio's powerful object storage capabilities to provide users with efficient and reliable data storage and management services.

[0055] Reference Figure 3 The step of classifying the target data to be uploaded based on the target index and storing it in the corresponding data storage bucket includes: S310. Construct multiple target data storage buckets based on the target index type, and construct a mapping relationship between the target index and the multiple target data storage buckets. The target data storage buckets are divided into different levels of data storage buckets based on the index level of the target index. S320. Classify the target data to be uploaded based on the minimum level of the target index and store it in the corresponding target data storage bucket.

[0056] After storing the data in the corresponding data storage bucket, the process also includes: S330. Trigger a metadata comparison task at each target time interval. After obtaining the actual metadata of the file from each of the data storage buckets, compare it with the expected metadata recorded locally. If the actual metadata is found to be inconsistent with the expected metadata, perform a data repair operation or trigger an alarm notification.

[0057] Specifically, after constructing the target index, the target data to be uploaded is categorized based on the target index and stored in the corresponding data storage buckets. First, in this embodiment, multiple target data storage buckets are constructed based on the target index type, and a mapping relationship is established between the target index and the multiple target data storage buckets. The target data storage buckets are divided into different levels based on the index level of the target index. During this process, it is necessary to determine how to divide the different levels and the number of storage buckets corresponding to each level based on the characteristics and requirements of the target index. For example, factors such as data access frequency and importance can be used to determine which data should be placed in higher-level storage buckets and which can be placed in lower-level storage buckets. Simultaneously, it is crucial to ensure an accurate mapping relationship is established so that data can be quickly and correctly routed to the appropriate storage location subsequently.

[0058] Specifically, when using the Minio Java SDK, you first need to import the corresponding dependency libraries, and then create and configure a MinioClient instance during the Java service startup phase. During this process, you need to provide the accurate server endpoint URL and valid credentials (Access Key and Secret Key) to ensure a successful connection to the target Minio service instance. Next, when processing a specific file upload request, the system will automatically determine which category the file should belong to based on preset rules, and select the appropriate target storage space (i.e., bucket) accordingly.

[0059] Then, the target data to be uploaded is categorized based on the minimum level of the target index and stored in the corresponding target data storage bucket. Specifically, based on the minimum level of the target index, it can be ensured that each piece of data to be uploaded is assigned to the correct storage bucket according to its characteristics. For example, if the index is based on a timestamp, it may be necessary to group by date range; if it is a content-related index, methods such as keyword matching need to be considered to achieve effective differentiation.

[0060] After data is stored in the corresponding data storage bucket, to ensure data integrity and consistency in the data storage system, the system will automatically trigger a metadata comparison task at preset time intervals. In this embodiment, the target time can be 12 hours; in many embodiments, the target time can be less or more depending on accuracy requirements. This mechanism is one of the important means to ensure data security and accuracy. After each target time interval, the system will start a background process responsible for traversing all data storage buckets and extracting the actual metadata information for each file. This metadata may include, but is not limited to, key attributes such as filename, creation date, last modification date, and size.

[0061] After obtaining the actual metadata, the next step is to compare it in detail with the expected metadata pre-recorded in the local database or other persistent storage media. This process requires highly precise verification of every field to ensure complete consistency. If any inconsistencies are found during this process—whether due to accidental damage caused by hardware failure, improper alteration caused by human error, or even unauthorized access or tampering resulting from potential security threats—the system will immediately take appropriate measures to address the situation.

[0062] Specifically, to ensure the accuracy of the status of all resources within the platform, this embodiment uses periodic tasks to verify whether the attributes of objects located in the cloud match the information stored in the local database. If any abnormal changes are detected, appropriate corrective measures are immediately taken or the administrator is notified for intervention.

[0063] Specifically, the remedial actions will differ depending on the type of problem detected. For issues that can be easily resolved by automated scripts, such as minor timestamp discrepancies or small-scale numerical differences, the system can attempt to fix these issues online directly without human intervention. However, in more complex situations, such as large-scale data loss or obvious security vulnerabilities, an alert should be promptly issued to relevant technical personnel, reminding them to investigate and address the problem as soon as possible. Furthermore, in some cases, it may be necessary to adjust subsequent operational procedures based on the specific circumstances to prevent similar incidents from recurring. This comprehensive monitoring and response mechanism effectively improves the stability and reliability of the entire platform, enhances overall security, and provides users with a superior service experience.

[0064] After receiving the data to be uploaded, the receiving target also includes: It provides real-time feedback on the upload progress, metadata extraction results, and storage status information of the target data to be uploaded.

[0065] In this embodiment, the system adopts a layered architecture design, consisting of a client, a service layer, and a storage layer forming a complete processing chain. The client provides a user-friendly file upload interface, supporting users to select local files for upload. It also displays the file upload progress, metadata extraction results, and upload status information.

[0066] The service layer provides Java services. It includes a file upload module, a metadata processing module, and an interface for interacting with the Minio storage module. It is responsible for receiving file upload requests from clients, calling the file upload module to upload the file, calling the metadata processing module to process the metadata, and then uploading the processed file and metadata to the Minio storage module.

[0067] The storage layer, also known as Minio object storage, is primarily responsible for storing uploaded files and their metadata, providing reliable data storage and access services.

[0068] In summary, this embodiment provides a data processing method based on Java services. During data upload and storage, it receives target data to be uploaded and simultaneously uploads the target file using multiple threads within the Java service. Then, it extracts the initial metadata of the target data, standardizes it to obtain target metadata with unified data format and encoding, and constructs a target index based on this metadata. Finally, it classifies the target data to be uploaded based on the target index and stores it in corresponding data storage buckets. This embodiment provides users with a data processing method based on Java services, solving the problem of ineffective management of file metadata processing and storage in existing technologies. When a user uploads a file, multi-threaded concurrent file transfer is used, while simultaneously completing the standardized parsing and index construction of metadata to ensure efficient flow. Finally, a distributed storage system automatically sorts the data to the corresponding storage bucket according to a preset strategy. This effectively improves the efficiency, accuracy, and reliability of file upload processing and optimizes system resource utilization.

[0069] It should be understood that although the steps in the flowcharts shown in the accompanying drawings are displayed sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of the steps in this invention, and these steps can be executed in other orders. Moreover, at least a portion of the steps in this invention may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0070] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program using signal-related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM). ROM Programmable ROM ( PROM ), electrically programmable ROM ( EPROM Electrically erasable programmable ROM ( EEPROM ) or flash memory. Volatile memory may include random access memory (RAM) RAM Alternatively, an external cache memory. This is for illustrative purposes only and not as a limitation. RAM It can be obtained in various forms, such as static RAM ( SRAM ),dynamic RAM ( DRAM ),synchronous DRAM ( SDRAM ), double data rate SDRAM ( DDR SDRAM ), Enhanced SDRAM ( ESDRAM ), Synchronization Link ( Synchlink ), DRAM ( SLDRAM ), memory bus ( Rambus )direct RAM ( RDRAM ), Direct Memory Bus Dynamics RAM ( DRDRAM ), and memory bus dynamics RAM ( RDRAM )wait.

[0071] Example 2 Based on the above embodiments, the present invention also provides a data processing system based on Java services, the functional module diagram of which is shown below. Figure 4 As shown, this Java service-based data processing system includes: The file upload module is used to receive target data to be uploaded and upload the target file to be uploaded simultaneously using multi-threading in Java services, as described in Example 1. The metadata processing module is used to extract the initial metadata of the target data to be uploaded, standardize the initial metadata to obtain target metadata with unified data format and encoding, and build a target index based on the target metadata, as described in Embodiment 1. The storage module is used to classify the target data to be uploaded based on the target index and store it in the corresponding data storage bucket, as described in Embodiment 1.

[0072] Example 3 like Figure 5 As shown, based on the above-mentioned data processing method based on Java services, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0073] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as the terminal's hard drive or memory. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard drive or smart memory card equipped on the terminal. SmartMediaCard , SMC ), Secure Digital ( SecureDigital , SD ) card, flash memory card ( FlashCard Furthermore, the memory 20 may include both internal storage units and external storage devices of the terminal. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a Java service-based data processing program 40, which can be executed by the processor 10 to implement the Java service-based data processing method of this application.

[0074] In some embodiments, the processor 10 may be a central processing unit (CPU). Central Processing Unit , CPU (a microprocessor or other data processing chip) is used to run program code stored in the memory 20 or process data, such as executing the data processing method based on Java services.

[0075] The display 30 may be, in some embodiments, led Monitors, LCD monitors, touch LCD monitors and OLED ( OrganicLight - EmittingDiodeThe display 30 includes components such as organic light-emitting diodes (OLEDs) and touchscreens. It is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0076] In one embodiment, when the processor 10 executes the Java service-based data processing program 40 in the memory 20, the following steps are performed: Receive the target data to be uploaded, and simultaneously upload the target file to be uploaded based on multi-threading in a Java service; Extract the initial metadata of the target data to be uploaded, standardize the initial metadata to obtain target metadata with unified data format and encoding, and build a target index based on the target metadata; The target data to be uploaded is categorized based on the target index and stored in the corresponding data storage bucket.

[0077] The simultaneous uploading of the target file based on multiple threads in a Java service includes: Obtain the target hardware configuration of the local server, and based on the target hardware configuration, configure a target number of threads to execute the upload of the target file to be uploaded.

[0078] The step of extracting the initial metadata of the target data to be uploaded includes: The Java file operation class is used to obtain the file name, file size, creation time and modification time of the target data to be uploaded, and the file type of the target data to be uploaded is determined based on the file extension or file header information, so as to obtain the initial metadata.

[0079] The standardization process for the initial metadata includes: Convert the date and time information from different sources in the initial metadata into the target time format; Map category fields to predefined enumeration values; Perform unit normalization on numerical attributes; The character encoding of the initial metadata is detected using the Java standard library and re-encoded into UTF-8 format.

[0080] The construction of the target index based on the target metadata includes: A directed graph model is used to construct the entity relationship network of the target metadata; The target index type is obtained based on the query scenario. The target index type is a combination of one or more local index types, including hash index, B+ tree index, inverted index, hierarchical index, and bitmap index. The entity relationship network is structured based on the target index type to obtain the target index.

[0081] The step of classifying the target data to be uploaded based on the target index and storing it in the corresponding data storage bucket includes: Multiple target data storage buckets are constructed based on the target index type, and a mapping relationship is established between the target index and the multiple target data storage buckets. The target data storage buckets are divided into different levels of data storage buckets based on the index level of the target index. The target data to be uploaded is categorized based on the minimum level of the target index and stored in the corresponding target data storage bucket.

[0082] The process of storing the data in the corresponding data storage bucket also includes: A metadata comparison task is triggered at each target time interval. After obtaining the actual metadata of the file from each of the data storage buckets, it is compared with the expected metadata recorded locally. If the actual metadata is found to be inconsistent with the expected metadata, a data repair operation is performed or an alarm notification is triggered.

[0083] Before storing the data in the corresponding data storage bucket, the process also includes: When the Java service starts, the Minio client is initialized through the Minio Java SDK, the network address, access key and secret key parameters of the Minio server are configured, and a secure connection is established with the target Minio object storage system. The target Minio object is used as a storage container to store the target data to be uploaded.

[0084] The receiving target, after uploading data, also includes: It provides real-time feedback on the upload progress, metadata extraction results, and storage status information of the target data to be uploaded.

[0085] Example 4 The present invention also provides a computer-readable storage medium having stored thereon one or more programs that can be executed by one or more processors to implement the steps of the data processing method based on Java services described in the above embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method based on Java services, characterized in that, The Java service-based data processing method includes: Receive the target data to be uploaded, and simultaneously upload the target file to be uploaded based on multi-threading in a Java service; Extract the initial metadata of the target data to be uploaded, standardize the initial metadata to obtain target metadata with unified data format and encoding, and build a target index based on the target metadata; The target data to be uploaded is categorized based on the target index and stored in the corresponding data storage bucket.

2. The data processing method based on Java services according to claim 1, characterized in that, The simultaneous uploading of the target file to be uploaded using multiple threads in a Java service includes: Obtain the target hardware configuration of the local server, and based on the target hardware configuration, configure a target number of threads to execute the upload of the target file to be uploaded.

3. The data processing method based on Java services according to claim 1, characterized in that, The extraction of initial metadata of the target data to be uploaded includes: The Java file operation class is used to obtain the file name, file size, creation time and modification time of the target data to be uploaded, and the file type of the target data to be uploaded is determined based on the file extension or file header information, so as to obtain the initial metadata.

4. The data processing method based on Java services according to claim 1, characterized in that, The initial metadata is standardized, including: Convert the date and time information from different sources in the initial metadata into the target time format; Map category fields to predefined enumeration values; Perform unit normalization on numerical attributes; The character encoding of the initial metadata is detected using the Java standard library and re-encoded into UTF-8 format.

5. The data processing method based on Java services according to claim 1, characterized in that, Constructing a target index based on the target metadata includes: A directed graph model is used to construct the entity relationship network of the target metadata; The target index type is obtained based on the query scenario. The target index type is a combination of one or more local index types, including hash index, B+ tree index, inverted index, hierarchical index, and bitmap index. The entity relationship network is structured based on the target index type to obtain the target index.

6. The data processing method based on Java services according to claim 5, characterized in that, The step of classifying the target data to be uploaded based on the target index and storing it in the corresponding data storage bucket includes: Multiple target data storage buckets are constructed based on the target index type, and a mapping relationship is established between the target index and the multiple target data storage buckets. The target data storage buckets are divided into different levels of data storage buckets based on the index level of the target index. The target data to be uploaded is categorized based on the minimum level of the target index and stored in the corresponding target data storage bucket.

7. The data processing method based on Java services according to claim 6, characterized in that, After storing the data in the corresponding data storage bucket, the process also includes: A metadata comparison task is triggered at each target time interval. After obtaining the actual metadata of the file from each of the data storage buckets, it is compared with the expected metadata recorded locally. If the actual metadata is found to be inconsistent with the expected metadata, a data repair operation is performed or an alarm notification is triggered.

8. The data processing method based on Java services according to claim 6, characterized in that, Before storing the data into the corresponding data storage bucket, the process also includes: When the Java service starts, the Minio client is initialized through the Minio Java SDK, the network address, access key and secret key parameters of the Minio server are configured, and a secure connection is established with the target Minio object storage system. The target Minio object is used as a storage container to store the target data to be uploaded.

9. The data processing method based on Java services according to claim 1, characterized in that, After receiving the data to be uploaded, the receiving target also includes: It provides real-time feedback on the upload progress, metadata extraction results, and storage status information of the target data to be uploaded.

10. A data processing system based on Java services, characterized in that, The system includes: The file upload module is used to receive target data to be uploaded and upload the target file to be uploaded simultaneously using multi-threading in a Java service. The metadata processing module is used to extract the initial metadata of the target data to be uploaded, standardize the initial metadata to obtain target metadata with unified data format and encoding, and build a target index based on the target metadata; The storage module is used to classify the target data to be uploaded based on the target index and store it into the corresponding data storage bucket.

11. A smart terminal, characterized in that, The smart terminal includes a memory, a processor, and a Java service-based data processing program stored in the memory and executable on the processor. When the Java service-based data processing program is executed by the processor, it implements the steps of the Java service-based data processing method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a Java service-based data processing program, which, when executed by a processor, implements the steps of the Java service-based data processing method as described in any one of claims 1-9.

Images