Multi-format industrial file intelligent analysis and vectorization storage method and device

Abstract

This invention belongs to the field of industrial document data storage technology, and discloses a method and apparatus for intelligent parsing and vectorized storage of multi-format industrial documents. The method includes: pushing the file metadata of the acquired raw industrial documents to a message queue; consuming the file metadata in the message queue, creating and executing a parsing task according to a preset strategy; receiving the parsing task, dynamically loading a parsing plugin matching the file type based on the file type in the file metadata, performing file parsing and generating a standard intermediate file in a unified format, writing the structured data from the standard intermediate file into a business database, extracting file content features, converting them into feature vectors, and writing them into a vector database. This approach achieves unified parsing and dual-mode (structured and vectorized) storage of multi-format industrial documents, supports dynamic expansion of parsing capabilities through a plug-in architecture, and supports in-depth analysis and intelligent retrieval of industrial data based on a vector database, thereby improving the efficiency and depth of industrial data analysis.

Description

Technical Field

[0001] This invention relates to the field of industrial document data storage technology, and in particular to a method and apparatus for intelligent parsing and vectorization storage of multi-format industrial documents. Background Technology

[0002] In modern industrial sectors such as semiconductor manufacturing and high-end equipment manufacturing, massive amounts of industrial data are generated during the production process, including semiconductor testing standard data format files, wafer defect detection format files, industrial image files, equipment operation logs, and process parameter records. These files carry critical information such as product quality, process status, and equipment health, forming the foundation for intelligent industrial analysis.

[0003] However, existing industrial document processing technologies suffer from the following pain points: First, file formats are heterogeneous and incompatible. File formats generated by different equipment manufacturers and different testing processes vary significantly. Traditional processing methods require developing dedicated parsing programs for each format, leading to severe duplication of development and poor system scalability. Second, parsing plugins are difficult to extend. Adding new formats requires modifying the core code and recompiling and deploying, making dynamic expansion impossible. Third, end-to-end control capabilities are lacking. Industrial document processing involves multiple stages, including acquisition, scheduling, parsing, and storage. Existing solutions lack unified task scheduling, strategy management, process monitoring, and data source management mechanisms. Fourth, data storage models are limited. Traditional industrial data platforms primarily use relational databases to store structured data, but the numerous unstructured features contained in industrial documents cannot be effectively expressed and retrieved. Simple structured storage is insufficient to meet the needs of in-depth analysis such as content-based similarity retrieval and root cause analysis.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a method and apparatus for intelligent parsing and vectorized storage of multi-format industrial documents, aiming to solve the technical problems of incompatible heterogeneous formats, difficulty in expanding parsing plugins, lack of full-process control, and single storage mode in existing industrial document processing technologies.

[0006] To achieve the above objectives, this invention provides a method for intelligent parsing and vectorized storage of multi-format industrial documents, which includes the following steps: By connecting to the industrial raw files through the file acquisition access layer, the file metadata of the industrial raw files is obtained, and the file metadata is pushed to the message queue; The global scheduling and control layer consumes the file metadata in the message queue, creates a parsing task according to a preset strategy, and allocates system resources to the parsing task to start task execution. The multi-format intelligent parsing layer receives the parsing task, dynamically loads parsing plugins matching the file type according to the file type in the file metadata through a plug-in architecture, and uses the parsing plugins to perform file parsing and generate a standard intermediate file in a unified format. The data distribution and vectorization storage layer reads the standard intermediate file, extracts the structured data from the standard intermediate file and writes it into the business database, and simultaneously extracts the file content features, converts them into feature vectors and writes them into the vector database.

[0007] In one embodiment, the file acquisition access layer supports FTP, SFTP, and HDFS file protocols through a file server management module, and maintains machine identification, data generation cycle, and file path mapping relationship through an industrial machine management module. The file metadata includes at least file format, data volume, business priority, and source machine identification. The message queue is a Kafka message queue to decouple the file acquisition and parsing stages and to smooth out peaks and valleys.

[0008] In one embodiment, the preset strategy is a multi-dimensional scheduling strategy based on file type, data volume, and business priority. The global scheduling and control layer dynamically manages the connection pools of various data sources through the data source management module, supports multi-data source routing, and adopts a multi-threaded task scheduling framework for concurrency control and system resource allocation. The parsing task includes multiple lifecycle states such as creation, allocation, execution, completion, and failure. The global scheduling and control layer manages the full lifecycle state transitions of the parsing task.

[0009] In one embodiment, the plug-in architecture is implemented based on the Java Service Provider Interface (SPI) mechanism, and the parsing plug-in is an independent loader component. Each loader component implements a unified parsing interface, supporting dynamic loading and hot-plugging. When there is a need to parse new file formats, add the corresponding Loader component and configure the SPI file for dynamic expansion; When no built-in Loader component matches the file type, a scripted Loader component is invoked. This scripted Loader component supports executing user-defined parsing logic using Groovy or JavaScript.

[0010] In one embodiment, the Loader component includes at least: a semiconductor test standard data format (STDF) Loader component, a wafer defect detection format (KLARF) Loader component, and a label image file format (TIFF) Loader component. The STDF Loader component is used to parse semiconductor test standard data format files and extract test items, test results, and yield information. The KLARF Loader component is used to parse wafer defect detection format files and extract defect coordinates, defect classification, and defect size information. The TIFF Loader component is used to parse industrial image files, extract image metadata and pixel matrices, and serialize the file into a standard intermediate file format (SIFF) using JavaScript object representation (JSON) or the Avro data serialization system. The data structure includes file header information, structured data fields extracted from each Loader component, feature vector fields, and extended attribute fields.

[0011] In one embodiment, the step of reading the standard intermediate file by the data distribution and vectorization storage layer, extracting the structured data from the standard intermediate file and writing it into the business database, and simultaneously extracting file content features and converting them into feature vectors to write into the vector database includes: The structured fields are extracted from the SIFF file by the structured parsing unit and mapped to the relational database table structure; Data is written through a multi-path distribution unit, based on the configured target database type, via the corresponding Java database connection JDBC driver or native software development kit (SDK). The feature extraction unit converts the file content features into vector representations. For numerical test data, the features are directly concatenated after normalization. For text log data, text features are extracted using the TF-IDF or Word2Vec algorithm to form feature vectors. The transformed feature vectors are written to the vector database in batches using the vector writing unit, and an index is built based on the open-source vector database. The multi-path distribution unit writes structured data into the business database in parallel according to a data distribution strategy configured based on file type or business line.

[0012] In one embodiment, the method further includes: The execution results written to the global scheduling and control layer database update the lifecycle status of the parsing task, and the monitoring and management module collects and records the task execution time, resource consumption, and success rate indicators in real time. When the indicators are abnormal, an automatic alarm is triggered. The system receives an industrial data analysis request, converts the query parameters contained in the request into a query vector, performs a similarity search on the query vector using the vector database, and returns feature vectors and associated structured data of several historical industrial documents similar to the query vector.

[0013] Furthermore, to achieve the above objectives, this invention also proposes a multi-format industrial document intelligent parsing and vectorization storage device, which is applied to the multi-format industrial document intelligent parsing and vectorization storage method described above. The device includes: The file acquisition and access module is used to connect to industrial raw files to obtain the file metadata of the industrial raw files and push the file metadata to the message queue; The global scheduling and control module is used to consume file metadata in the message queue, create parsing tasks according to preset strategies, and allocate system resources to the parsing tasks to start task execution. The multi-format intelligent parsing module is used to receive the parsing task, dynamically load parsing plugins that match the file type according to the file type in the file metadata through a plug-in architecture, and use the parsing plugins to perform file parsing and generate a standard intermediate file in a unified format. The distribution and storage module is used to read the standard intermediate file, extract the structured data from the standard intermediate file and write it into the business database, and at the same time extract the file content features, convert them into feature vectors and write them into the vector database.

[0014] Furthermore, to achieve the above objectives, the present invention also proposes a multi-format industrial document intelligent parsing and vectorization storage device, the multi-format industrial document intelligent parsing and vectorization storage device comprising: a memory, a processor, and a multi-format industrial document intelligent parsing and vectorization storage program stored on the memory and executable on the processor, the multi-format industrial document intelligent parsing and vectorization storage program being configured to implement the steps of the multi-format industrial document intelligent parsing and vectorization storage method described above.

[0015] Furthermore, to achieve the above objectives, the present invention also proposes a storage medium storing a multi-format industrial document intelligent parsing and vectorization storage program, wherein when the multi-format industrial document intelligent parsing and vectorization storage program is executed by a processor, it implements the steps of the multi-format industrial document intelligent parsing and vectorization storage method described above.

[0016] This invention pushes the file metadata of acquired raw industrial documents to a message queue; consumes the file metadata from the message queue, creates and executes parsing tasks according to a preset strategy; receives the parsing tasks, dynamically loads parsing plugins matching the file type based on the file type in the file metadata, performs file parsing, generates a standard intermediate file in a unified format, writes the structured data from the standard intermediate file to the business database, extracts file content features, converts them into feature vectors, and writes them to a vector database. This approach achieves unified parsing and dual-mode (structured and vectorized) storage of multi-format industrial documents, supports dynamic expansion of parsing capabilities through a plug-in architecture, and supports in-depth analysis and intelligent retrieval of industrial data based on a vector database, thus improving the efficiency and depth of industrial data analysis. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the first embodiment of the intelligent parsing and vectorized storage method for multi-format industrial documents of the present invention; Figure 2 This is a structural block diagram of the first embodiment of the intelligent parsing and vectorization storage device for multi-format industrial documents of the present invention.

[0018] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0020] This invention provides a method for intelligent parsing and vectorized storage of multi-format industrial documents, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the intelligent parsing and vectorized storage method for multi-format industrial documents according to the present invention.

[0021] In this embodiment, the intelligent parsing and vectorized storage method for multi-format industrial documents includes the following steps: Step S10: Connect to the industrial raw files through the file acquisition access layer to obtain the file metadata of the industrial raw files, and push the file metadata to the message queue.

[0022] In this embodiment, the executing entity is a multi-format industrial document intelligent parsing and vectorization storage device. This multi-format industrial document intelligent parsing and vectorization storage device has functions such as data processing, data communication, and program execution. The multi-format industrial document intelligent parsing and vectorization storage device can be a computer terminal device or other network device, or other devices with similar functions. This embodiment does not limit this.

[0023] It should be noted that existing industrial document processing technologies suffer from the following pain points: First, file formats are heterogeneous and incompatible. File formats generated by different equipment manufacturers and different testing processes vary significantly. Traditional processing methods require developing dedicated parsing programs for each format, leading to severe duplication of development and poor system scalability. Second, parsing plugins are difficult to extend. Adding new formats requires modifying the core code and recompiling and deploying, making dynamic expansion impossible. Third, end-to-end control capabilities are lacking. Industrial document processing involves multiple stages, including acquisition, scheduling, parsing, and storage. Existing solutions lack unified task scheduling, strategy management, process monitoring, and data source management mechanisms. Fourth, data storage models are limited. Traditional industrial data platforms primarily use relational databases to store structured data, but the numerous unstructured features contained in industrial documents cannot be effectively expressed and retrieved. Simple structured storage is insufficient to meet the needs of in-depth analysis such as content-based similarity retrieval and root cause analysis.

[0024] To address the aforementioned technical issues, this embodiment pushes the file metadata of the acquired raw industrial files to a message queue; consumes the file metadata from the message queue, creates and executes parsing tasks according to a preset strategy; receives the parsing tasks, dynamically loads parsing plugins matching the file type based on the file type in the file metadata, performs file parsing, generates a standard intermediate file in a unified format, writes the structured data from the standard intermediate file to the business database, extracts file content features, converts them into feature vectors, and writes them to a vector database. This approach achieves unified parsing and dual-mode (structured and vectorized) storage of multi-format industrial files, supports dynamic expansion of parsing capabilities through a plug-in architecture, and supports in-depth analysis and intelligent retrieval of industrial data based on a vector database, improving the efficiency and depth of industrial data analysis. Specifically, it can be implemented as follows.

[0025] In this embodiment, the intelligent parsing and vectorized storage system for multi-format industrial documents is developed using the Java full-stack language and includes four core layers: The file acquisition and access layer includes a file server management module and an industrial machine management module. It interfaces with raw industrial files and uses a Kafka message queue to decouple file acquisition and parsing, pushing file metadata and acquisition status to subsequent layers. The file server management module supports multiple file protocols such as FTP, SFTP, and HDFS, while the industrial machine management module maintains machine identifiers, data generation cycles, and file path mappings. This layer is implemented based on a Spring Boot microservice architecture, integrating a message queue through a Kafka Java client to ensure decoupling and peak / valley smoothing between acquisition and parsing processes.

[0026] The global scheduling and control layer, comprising a policy management module, a process management module, a monitoring management module, and a data source management module, is used to achieve full lifecycle management of parsing tasks. The policy management module supports multi-dimensional scheduling policy configuration based on file type, data volume, and business priority; the process management module maintains the lifecycle state of parsing tasks, including state transitions such as creation, allocation, execution, completion, and failure; the monitoring management module integrates metric collection and alarm mechanisms, collecting metrics such as task execution time, resource consumption, and success rate in real time; and the data source management module dynamically manages various data source connection pools and supports multi-data source routing. This layer employs a multi-threaded task scheduling framework to implement concurrent control and resource allocation for tasks.

[0027] The multi-format intelligent parsing layer, based on a Java plug-in architecture, includes a collection of extensible industrial file parsing Loader components. This layer utilizes the Java SPI (Service Provider Interface) mechanism to dynamically load and hot-plug Loader components. Each industrial file format corresponds to an independent Loader implementation class. Through a unified parsing interface definition, adding new formats only requires adding the corresponding Loader component without modifying the core framework code. The Loader components cover at least the following industrial file formats: STDF Loader, used to parse semiconductor test standard data format files, extracting test items, test results, and yield information; KLARF Loader, used to parse wafer defect detection file formats, extracting defect coordinates, defect classification, and defect size information; TIFF Loader, used to parse industrial image files, extracting image metadata and pixel matrices; and a scriptable Loader, allowing users to customize parsing logic using scripting languages ​​such as Groovy and JavaScript for rapid format expansion.

[0028] Each loader outputs the parsing results as a standard intermediate file (SIFF, or Standard Intermediate File Format). This format uses JSON or Avro serialization and includes file header information, structured data fields, feature vector fields, and extended attribute fields, providing a unified data view for subsequent distribution.

[0029] The data distribution and vectorized storage layer, including the loader-resolver core distribution module, is used for structured data extraction and industrial feature vectorization transformation of SIFF standard intermediate files. This module includes: a structured parsing unit, used to extract structured fields from SIFF files and map them to relational database table structures; a feature extraction unit, used to convert file content into vector representations using pre-built industrial feature extraction algorithms. Feature extraction can be performed in various ways: for numerical test data, it is directly concatenated into feature vectors after normalization; for text-based log data, text features are extracted using TF-IDF or Word2Vec algorithms; a multi-path distribution unit, used to write data according to the configured target database type using the corresponding JDBC driver or native SDK; and a vector writing unit, based on the Milvus Java SDK, to implement batch writing of vector data, index building, and similarity retrieval. This layer supports distributed writing to business databases such as Oracle, openGauss, and StarRocks, and also supports writing vectorized data to the Milvus vector database, achieving dual-mode storage of structured data and unstructured feature vectors. Through dynamic data source adapters, the system supports parallel writing to multiple business databases, and the data distribution strategy is configured for routing based on file type or business line.

[0030] In its implementation, the file acquisition and access layer is based on a Spring Boot microservice architecture, comprising a file server management module and an industrial machine management module. The file server management module supports multiple file protocols including FTP, SFTP, and HDFS. It defines the file server address, port, authentication information, and file path matching rules through configuration, and discovers newly arriving industrial raw files through periodic scanning or event-triggered methods. The industrial machine management module maintains machine identifiers, data generation cycles, and file path mappings, associating acquired files with production machines and generating file metadata. This file metadata includes at least the file format, data volume, business priority, and source machine identifier, providing basic information for subsequent scheduling and parsing.

[0031] In one embodiment, the file acquisition and access layer is implemented based on a Spring Boot microservice architecture, including a file server management module and an industrial machine management module. The file server management module supports multiple file protocols, including FTP, SFTP, and HDFS. It defines the file server address, port, authentication information, and file path matching rules through configuration, and discovers newly arriving industrial raw files through periodic scanning or event-triggered methods. The industrial machine management module maintains machine identifiers, data generation cycles, and file path mapping relationships, associating acquired files with production machines and generating file metadata. This file metadata includes at least the file format, data volume, business priority, and source machine identifier, providing basic information for subsequent scheduling and parsing.

[0032] The file acquisition and access layer integrates a message queue through a Kafka Java client, pushing file metadata to the Kafka message queue. The purpose of using the Kafka message queue is to decouple the file acquisition and parsing processes and to smooth out peak loads. When a large number of files arrive simultaneously, the message queue acts as a buffer, preventing the parsing layer from being overwhelmed by a sudden high load, thus ensuring the stability and scalability of the system.

[0033] Step S20: The global scheduling and control layer consumes the file metadata in the message queue, creates a parsing task according to a preset strategy, and allocates system resources to the parsing task to start task execution.

[0034] In the specific implementation, the global scheduling and control layer, as the core scheduling hub of the system, includes a policy management module, a process management module, a monitoring management module, and a data source management module, and is responsible for implementing full lifecycle control of the parsing task.

[0035] It should be noted that the global scheduling and control layer consumes file metadata from the Kafka message queue through consumer groups. The policy management module supports multi-dimensional scheduling policy configuration, and the preset policy is a comprehensive scheduling policy based on file type, data volume, and business priority. For example, for STDF files with small data volume and high business priority, resources are allocated for processing first; for TIFF image files with large data volume, more parallel processing resources are allocated to prevent processing latency. The data source management module dynamically manages the connection pools of various data sources, supports multi-data source routing, and writes data to the corresponding target database according to different file types and business lines. The global scheduling and control layer uses a multi-threaded task scheduling framework for concurrency control and system resource allocation to ensure efficient execution of parsing tasks.

[0036] It should be noted that the process management module maintains the lifecycle state of the parsing task, which includes multiple lifecycle states such as creation, allocation, execution, completion, and failure. When new file metadata is consumed, the process management module creates a parsing task instance, and the state is "created"; after the scheduling framework allocates processing threads and computing resources to the task according to the policy configuration, the state changes to "allocated"; when the multi-format intelligent parsing layer starts parsing, the state changes to "execution"; after successful parsing, the state changes to "completed"; and when parsing encounters an error, the state changes to "failed". The global scheduling and control layer manages the full lifecycle state transitions of the parsing task, and the task status is trackable in real time.

[0037] Step S30: The multi-format intelligent parsing layer receives the parsing task, dynamically loads parsing plugins matching the file type according to the file type in the file metadata through a plug-in architecture, and uses the parsing plugins to perform file parsing and generate a standard intermediate file in a unified format.

[0038] In its implementation, the multi-format intelligent parsing layer adopts a plug-in architecture based on the Java Service Provider Interface (SPI) mechanism. Each parsing plugin is an independent Loader component, and each Loader component implements a unified parsing interface. The interface definition includes methods for file format determination and parsing execution. The Java SPI mechanism supports dynamic discovery and loading of interface implementation classes at runtime, enabling dynamic expansion and hot-plugging of parsing capabilities without modifying the core framework code.

[0039] In its implementation, when the multi-format intelligent parsing layer receives a parsing task, it first searches for a matching Loader in the registered Loader component set based on the file type field in the file metadata. If no built-in Loader component matches the file type, a scripted Loader component is invoked. This scripted Loader component supports executing user-defined parsing logic using Groovy or JavaScript. The scripted Loader provides maximum flexibility to the system, allowing users to quickly adapt to new file formats without writing Java code, reducing the adaptation cycle from days to hours. When there is a need to parse new file formats, developers only need to write the corresponding Loader implementation class, configure the SPI file in the META-INF / services directory, and package the Loader component into the specified directory to achieve dynamic expansion.

[0040] In one embodiment, the Loader component includes at least: a semiconductor test standard data format (STDF) Loader component, used to parse semiconductor test standard data format files and extract test items, test results, and yield information; a wafer defect detection format (KLARF) Loader component, used to parse wafer defect detection format files and extract defect coordinates, defect classification, and defect size information; and a label image file format (TIFF) Loader component, used to parse industrial image files and extract image metadata and pixel matrix.

[0041] After each Loader component parses the file, it outputs the parsing results as a standardized intermediate file in SIFF format. This standardized intermediate file is a SIFF file serialized using JavaScript object notation (JSON) or the Avro data serialization system. Its data structure includes file header information, structured data fields extracted from each Loader component, feature vector fields, and extended attribute fields. By converting heterogeneous file formats into a unified standardized intermediate format, the incompatibility problem of industrial file formats is completely solved, providing a unified data view for subsequent data distribution and vectorized storage.

[0042] Step S40: The data distribution and vectorization storage layer reads the standard intermediate file, extracts the structured data from the standard intermediate file and writes it into the business database, and at the same time extracts the file content features, converts them into feature vectors and writes them into the vector database.

[0043] In its implementation, the data distribution and vectorized storage layer includes a loader-resolver core distribution module. This module is responsible for extracting structured data and vectorizing industrial features from the SIFF standard intermediate file. Specifically, this includes the following processing steps: A structured parsing unit extracts structured fields from the SIFF file and maps them to relational database table structures. Based on configured mapping rules, the structured parsing unit establishes a one-to-one correspondence between the structured fields in the SIFF file and the columns of the target table in the business database, generating corresponding SQL insert or update statements.

[0044] Furthermore, through a multi-path distribution unit, data is written to the target database using the corresponding Java database connection JDBC driver or native software development kit (SDK) based on the configured target database type. The business databases include at least Oracle, openGauss, and StarRocks databases, and the system supports parallel writing to multiple business databases through a dynamic data source adapter. The multi-path distribution unit writes structured data to multiple business databases in parallel according to a data distribution strategy configured based on file type or business line. For example, test data from STDF files is written to an Oracle database for production report analysis, defect data from KLARF files is written to an openGauss database for quality traceability, and metadata from TIFF files is written to a StarRocks database for statistical analysis.

[0045] Furthermore, the feature extraction unit converts the file content features into vector representations. Different feature extraction methods are used for different types of industrial data: for numerical test data, the feature vectors are directly concatenated after normalization; for text-based log data, text features are extracted using the TF-IDF (Term Frequency-Inverse Document Frequency) or Word2Vec algorithm to construct the feature vectors. The feature extraction unit transforms the unstructured features (such as defect patterns, test waveform features, image texture features, etc.) contained in the original industrial files into fixed-dimensional feature vectors.

[0046] Furthermore, through the vector writing unit, the transformed feature vectors are written to the vector database in batches based on the open-source vector database Milvus Java SDK, and an index is built. The Milvus vector database supports efficient vector similarity retrieval, providing a data foundation for subsequent intelligent applications such as content-based similarity retrieval, defect root cause analysis, and process parameter recommendation. Through the above dual-mode storage architecture of structured and vectorized data, the system not only meets the needs of traditional statistical analysis for structured data, but also supports deep intelligent analysis applications based on vector similarity.

[0047] In one embodiment, the method further includes: a global scheduling and control layer receiving the execution results written to the database, updating the lifecycle status of the parsing task, and a monitoring and management module collecting and recording task execution time, resource consumption, and success rate indicators in real time, triggering automatic alarms when the indicators are abnormal. For example, when the parsing success rate of a certain Loader component is lower than a threshold, the monitoring and management module generates alarm information to notify maintenance personnel to intervene and handle the issue, ensuring the timeliness and reliability of industrial data processing.

[0048] In one embodiment, the method further includes a vector retrieval step: receiving an industrial data analysis request, converting the query parameters contained in the industrial data analysis request into a query vector, performing a similarity search on the query vector through the vector database, and returning feature vectors and associated structured data of several historical industrial documents similar to the query vector. For example, in a semiconductor yield analysis scenario, if an engineer discovers a defect in a batch of chips, the defect characteristics of that batch can be converted into a query vector. The vector database can then be used to quickly retrieve the historical batches in the historical database that are most similar to the defect pattern, and return the structured data (such as test parameters, process conditions, etc.) of these batches. This enables defect root cause analysis and abnormal batch tracing, significantly improving analysis efficiency compared to traditional manual comparison methods.

[0049] In this embodiment, the file metadata of the acquired raw industrial files is pushed to a message queue; the file metadata in the message queue is consumed, and a parsing task is created and executed according to a preset strategy; the parsing task is received, and a parsing plugin matching the file type is dynamically loaded according to the file type in the file metadata; file parsing is performed, and a standard intermediate file with a unified format is generated; the structured data from the standard intermediate file is written to the business database, and file content features are extracted, converted into feature vectors, and written to a vector database. This method achieves unified parsing and dual-mode (structured and vectorized) storage of multi-format industrial files, supports dynamic expansion of parsing capabilities through a plug-in architecture, and supports in-depth analysis and intelligent retrieval of industrial data based on a vector database, improving the efficiency and depth of industrial data analysis.

[0050] Furthermore, this embodiment of the invention also proposes a storage medium storing a multi-format industrial document intelligent parsing and vectorization storage program. When the multi-format industrial document intelligent parsing and vectorization storage program is executed by a processor, it implements the steps of the multi-format industrial document intelligent parsing and vectorization storage method described above.

[0051] Reference Figure 2 , Figure 2 This is a structural block diagram of the first embodiment of the intelligent parsing and vectorization storage device for multi-format industrial documents of the present invention.

[0052] like Figure 2 As shown, the intelligent parsing and vectorization storage device for multi-format industrial documents proposed in this embodiment of the invention includes: The file acquisition and access module 10 is used to connect to industrial raw files to obtain the file metadata of the industrial raw files and push the file metadata to the message queue; The global scheduling and control module 20 is used to consume file metadata in the message queue, create parsing tasks according to a preset strategy, and allocate system resources to the parsing tasks to start task execution. The multi-format intelligent parsing module 30 is used to receive the parsing task, dynamically load parsing plugins that match the file type according to the file type in the file metadata through a plug-in architecture, and use the parsing plugins to perform file parsing and generate a standard intermediate file in a unified format. The distribution and storage module 40 is used to read the standard intermediate file, extract the structured data from the standard intermediate file and write it into the business database, and at the same time extract the file content features, convert them into feature vectors and write them into the vector database.

[0053] In this embodiment, the file metadata of the acquired raw industrial files is pushed to a message queue; the file metadata in the message queue is consumed, and a parsing task is created and executed according to a preset strategy; the parsing task is received, and a parsing plugin matching the file type is dynamically loaded according to the file type in the file metadata; file parsing is performed, and a standard intermediate file with a unified format is generated; the structured data from the standard intermediate file is written to the business database, and file content features are extracted, converted into feature vectors, and written to a vector database. This method achieves unified parsing and dual-mode (structured and vectorized) storage of multi-format industrial files, supports dynamic expansion of parsing capabilities through a plug-in architecture, and supports in-depth analysis and intelligent retrieval of industrial data based on a vector database, improving the efficiency and depth of industrial data analysis.

[0054] This application embodiment also provides a multi-format industrial document intelligent parsing and vectorization storage device, including a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other through the communication bus. The memory is used to store the multi-format industrial document intelligent parsing and vectorization storage program. When the processor executes the program stored in the memory, it implements the above-mentioned multi-format industrial document intelligent parsing and vectorization storage method.

[0055] The communication bus mentioned in the aforementioned multi-format industrial document intelligent parsing and vectorization storage device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0056] The communication interface is used for communication between the aforementioned multi-format industrial file intelligent parsing and vectorization storage device and other devices.

[0057] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0058] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0059] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0060] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0061] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0062] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0063] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0064] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0065] In addition, for technical details not described in detail in this embodiment, please refer to the method for intelligent parsing and vectorized storage of multi-format industrial documents provided in any embodiment of the present invention, which will not be repeated here.

[0066] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0067] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0068] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0069] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

[0070] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

Claims

1. A method for intelligent parsing and vectorized storage of multi-format industrial documents, characterized in that, The intelligent parsing and vectorized storage method for multi-format industrial documents includes: By connecting to the industrial raw files through the file acquisition access layer, the file metadata of the industrial raw files is obtained, and the file metadata is pushed to the message queue; The global scheduling and control layer consumes the file metadata in the message queue, creates a parsing task according to a preset strategy, and allocates system resources to the parsing task to start task execution. The multi-format intelligent parsing layer receives the parsing task, dynamically loads parsing plugins matching the file type according to the file type in the file metadata through a plug-in architecture, and uses the parsing plugins to perform file parsing and generate a standard intermediate file in a unified format. The data distribution and vectorization storage layer reads the standard intermediate file, extracts the structured data from the standard intermediate file and writes it into the business database, and simultaneously extracts the file content features, converts them into feature vectors and writes them into the vector database.

2. The intelligent parsing and vectorized storage method for multi-format industrial documents as described in claim 1, characterized in that, The file acquisition and access layer supports FTP, SFTP, and HDFS file system access through the file server management module, and maintains the machine identification, data generation cycle, and file path mapping relationship through the industrial machine management module. The file metadata includes at least the file format, data volume, business priority, and source machine identification. The message queue is a Kafka message queue to decouple the file acquisition and parsing links and to smooth out peaks and valleys.

3. The intelligent parsing and vectorized storage method for multi-format industrial documents as described in claim 1, characterized in that, The preset strategy is a multi-dimensional scheduling strategy based on file type, data volume, and business priority. The global scheduling and control layer dynamically manages the connection pools of various data sources through the data source management module, supports multi-data source routing, and adopts a multi-threaded task scheduling framework for concurrency control and system resource allocation. The parsing task includes multiple lifecycle states such as creation, allocation, execution, completion, and failure. The global scheduling and control layer manages the full lifecycle state transitions of the parsing task.

4. The intelligent parsing and vectorized storage method for multi-format industrial documents as described in claim 1, characterized in that, The plug-in architecture is implemented based on the Java Service Provider Interface (SPI) mechanism. The parsing plug-in is an independent loader component. Each loader component implements a unified parsing interface and supports dynamic loading and hot-plugging. When there is a need to parse new file formats, add the corresponding Loader component and configure the SPI file for dynamic expansion; When no built-in Loader component matches the file type, the scripted Loader component is invoked. The scripted Loader component supports executing user-defined parsing logic through a scripting language.

5. The intelligent parsing and vectorized storage method for multi-format industrial documents as described in claim 4, characterized in that, The Loader component includes at least: a semiconductor test standard data format STDF Loader component, a wafer defect detection format KLARF Loader component, and a label image file format TIFF Loader component; wherein, the STDF Loader component is used to parse semiconductor test standard data format files and extract test items, test results, and yield information; the KLARF Loader component is used to parse wafer defect detection format files and extract defect coordinates, defect classification, and defect size information; the TIFF Loader component is used to parse industrial image files, extract image metadata and pixel matrices, and use the standard intermediate file format SIFF file serialized using JavaScript object representation JSON or the data serialization system Avro. The data structure includes file header information, structured data fields extracted from each Loader component, feature vector fields, and extended attribute fields.

6. The intelligent parsing and vectorized storage method for multi-format industrial documents as described in claim 5, characterized in that, The process of reading the standard intermediate file by the data distribution and vectorization storage layer, extracting the structured data from the standard intermediate file and writing it into the business database, and simultaneously extracting file content features and converting them into feature vectors to write into the vector database includes: The structured fields are extracted from the SIFF file by the structured parsing unit and mapped to the relational database table structure; Data is written through a multi-path distribution unit, based on the configured target database type, via the corresponding Java database connection JDBC driver or native software development kit (SDK). The feature extraction unit converts the file content features into vector representations. For numerical test data, the features are directly concatenated after normalization. For text log data, text features are extracted using the TF-IDF or Word2Vec algorithm to form feature vectors. The transformed feature vectors are written to the vector database in batches using the vector writing unit, and an index is built based on the open-source vector database. The multi-path distribution unit writes structured data into the business database in parallel according to a data distribution strategy configured based on file type or business line.

7. The intelligent parsing and vectorized storage method for multi-format industrial documents as described in claim 1, characterized in that, The method further includes: The execution results written to the global scheduling and control layer database update the lifecycle status of the parsing task, and the monitoring and management module collects and records the task execution time, resource consumption, and success rate indicators in real time. When the indicators are abnormal, an automatic alarm is triggered. The system receives an industrial data analysis request, converts the query parameters contained in the request into a query vector, performs a similarity search on the query vector using the vector database, and returns feature vectors and associated structured data of several historical industrial documents similar to the query vector.

8. A multi-format industrial document intelligent parsing and vectorization storage device, characterized in that, The intelligent parsing and vectorization storage device for multi-format industrial documents is applied to the intelligent parsing and vectorization storage method for multi-format industrial documents as described in any one of claims 1 to 7, and the device comprises: The file acquisition and access module is used to connect to industrial raw files to obtain the file metadata of the industrial raw files and push the file metadata to the message queue; The global scheduling and control module is used to consume file metadata in the message queue, create parsing tasks according to preset strategies, and allocate system resources to the parsing tasks to start task execution. The multi-format intelligent parsing module is used to receive the parsing task, dynamically load parsing plugins that match the file type according to the file type in the file metadata through a plug-in architecture, and use the parsing plugins to perform file parsing and generate a standard intermediate file in a unified format. The distribution and storage module is used to read the standard intermediate file, extract the structured data from the standard intermediate file and write it into the business database, and at the same time extract the file content features, convert them into feature vectors and write them into the vector database.

9. A multi-format industrial document intelligent parsing and vectorization storage device, characterized in that, The multi-format industrial document intelligent parsing and vectorization storage device includes: a memory, a processor, and a multi-format industrial document intelligent parsing and vectorization storage program stored on the memory and executable on the processor, wherein the multi-format industrial document intelligent parsing and vectorization storage program is configured to implement the steps of the multi-format industrial document intelligent parsing and vectorization storage method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a multi-format industrial document intelligent parsing and vectorization storage program, which, when executed by a processor, implements the steps of the multi-format industrial document intelligent parsing and vectorization storage method as described in any one of claims 1 to 7.

Images