A processing method and device for multimodal data acquisition in an intelligent manufacturing system

Abstract

The present application provides a method and device for processing multimodal data acquisition in an intelligent manufacturing system. The method provided in the present application includes: constructing a data acquisition point catalog library based on the target production task of the intelligent manufacturing system; determining a collection plan based on the target production task and the data acquisition point catalog library; collecting multimodal data from the collection points in the data acquisition point catalog library based on the collection plan, and preprocessing the multimodal data according to the data parsing algorithm corresponding to each collection point, the data format and data type of each collection point are different, and the data format of the preprocessed multimodal data is the same; inputting the preprocessed multimodal data into the data fusion model, matching the fusion algorithm, fusing and storing the multimodal data. The method and device for processing multimodal data acquisition in an intelligent manufacturing system provided in the present application are used to complete data acquisition accurately and efficiently.

Description

Technical Field

[0001] The present application relates to the field of intelligent manufacturing technology, and in particular to a method and device for processing multimodal data acquisition in an intelligent manufacturing system. Background Art

[0002] In today's rapidly developing information age, data collection, processing, and analysis have become core tasks in various fields, particularly in intelligent manufacturing. Acquiring multimodal data is a crucial component of intelligent manufacturing. Multimodal data consists of multiple modalities, including text, images, audio, video, sensor data, and device data. Furthermore, different sensor models output data in different formats, and the resulting multimodal data provides a rich and comprehensive source of information for the system. Therefore, how to efficiently and accurately collect, process, and fuse multimodal data has become a key issue that needs to be addressed in the intelligent manufacturing field.

[0003] Existing data processing methods, when dealing with large-scale, diverse data, rely solely on manual labeling to identify distinct data. This often results in slow processing speeds and low efficiency, making it difficult to meet the real-time data processing needs of intelligent manufacturing systems. Furthermore, because intelligent manufacturing systems have different requirements for different tasks, it is difficult for technical personnel to obtain accurate and comprehensive data, impacting the accuracy of subsequent analysis and processing, and failing to meet practical needs. Summary of the Invention

[0004] In view of this, the present application provides a method and device for processing multimodal data acquisition in an intelligent manufacturing system, so as to complete data acquisition accurately and efficiently.

[0005] Specifically, this application is implemented through the following technical solutions:

[0006] In a first aspect, the present application provides a method for processing multimodal data acquisition in an intelligent manufacturing system, the method comprising:

[0007] Building a data collection point catalog based on the target production task of the intelligent manufacturing system; the data collection point catalog is a collection point associated with the target production task determined from various monitoring points within the control range of the intelligent manufacturing system, including a plurality of collection point configuration information, each of which includes the data type, location information and communication information of the collection point;

[0008] Determining a collection plan based on the target production task and the data collection point catalog library, wherein the collection plan at least includes a collection sequence for different collection points, a collection template and a data parsing algorithm corresponding to each collection point, the collection templates and data parsing algorithms being different for collection points of different data types, and the collection cycle and collection sequence being determined based on the timeliness information of the target production task;

[0009] Collecting multimodal data from the collection points in the data collection point catalog based on the collection scheme, and preprocessing the multimodal data according to the data analysis algorithm corresponding to each collection point, wherein the data format and data type of each collection point are different, and the data format of the multimodal data after preprocessing is the same;

[0010] The pre-processed multimodal data is input into a data fusion model, a fusion algorithm is matched, and the multimodal data is fused and stored.

[0011] The second aspect of the present application provides a processing device for multimodal data acquisition of an intelligent manufacturing system, the device comprising a construction module, a determination module, a collection module and a fusion module; wherein,

[0012] The construction module is used to construct a data collection point catalog library based on the target production task of the intelligent manufacturing system; the data collection point catalog library is a collection point associated with the target production task determined from various monitoring points within the control range of the intelligent manufacturing system, including a plurality of collection point configuration information, each of which is a data type, location information, and communication information of the collection point;

[0013] The determination module is configured to determine a collection plan based on the target production task and the data collection point catalog library, wherein the collection plan includes at least a collection sequence for different collection points, a collection template and a data parsing algorithm corresponding to each collection point, the collection templates and data parsing algorithms being different for collection points of different data types, and the collection cycle and collection sequence being determined based on the timeliness information of the target production task;

[0014] The acquisition module is configured to acquire multimodal data from the acquisition points in the data acquisition point catalog based on the acquisition plan, and preprocess the multimodal data according to a data analysis algorithm corresponding to each acquisition point, wherein the data formats and data types of the respective acquisition points are different, and the data formats of the preprocessed multimodal data are the same;

[0015] The fusion module is used to input the pre-processed multimodal data into a data fusion model, match the fusion algorithm, and fuse and store the multimodal data.

[0016] The third aspect of the present application provides a processing device for multimodal data acquisition of an intelligent manufacturing system, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods provided in the first aspect of the present application are implemented.

[0017] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods provided in the first aspect of the present application.

[0018] The processing method and device for multimodal data acquisition in an intelligent manufacturing system provided by the present application switches the traditional comprehensive management mode of a single collection point to a standardized management mode in a database with a unified format. By constructing a data collection point catalog library, all collection points associated with the target production task are recorded in detail, including key configuration information such as data type, location information, and communication information, so that all the points to be collected in the intelligent manufacturing system can be efficiently managed in a unified and standardized format in the database, which is conducive to conveniently obtaining the data format and communication protocol of each collection point in the post-collection control process, generating targeted collection templates, and making the data collection process more orderly and efficient through the data collection point catalog library, reducing the collection of redundant data and improving the pertinence and accuracy of data collection. Furthermore, the collection scheme takes into account the diversity of data types, customizes different collection templates and data parsing algorithms for each data type, ensures the accuracy and consistency of data during the collection process, and reasonably sets the collection cycle and collection sequence to meet the requirements of the intelligent manufacturing system for data collection timeliness. Furthermore, through the data fusion model, the corresponding fusion algorithm is matched according to the characteristics of the data and the needs of the business scenario, and the data of different modalities are fused into a unified data set, forming a more comprehensive and accurate data representation. In this way, by integrating the data of each collection point with the production tasks corresponding to the intelligent manufacturing system, the data acquisition of the production tasks is guaranteed efficiently and accurately, providing a guarantee for the completion of the production tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flowchart of a first embodiment of a method for processing multimodal data acquisition in an intelligent manufacturing system provided in this application;

[0020] Figure 2 This is a schematic diagram of a data collection point catalogue building process shown in an exemplary embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of a collection plan determination process shown in an exemplary embodiment of the present application;

[0022] Figure 4 This is a schematic diagram of a multimodal data preprocessing process according to an exemplary embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of a multimodal data fusion process shown in an exemplary embodiment of the present application;

[0024] Figure 6A hardware structure diagram of the processing device for multimodal data acquisition of the intelligent manufacturing system in this application;

[0025] Figure 7 This is a structural diagram of embodiment 1 of the processing device for multimodal data acquisition in an intelligent manufacturing system provided in this application. DETAILED DESCRIPTION

[0026] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0027] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0028] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0029] The present application provides a method and device for processing multimodal data acquisition in an intelligent manufacturing system, which are used to complete data acquisition accurately and efficiently.

[0030] Specific embodiments are given below to introduce the technical solutions of the present application in detail.

[0031] Figure 1 This is a flowchart of the first embodiment of the processing method for multimodal data acquisition of the intelligent manufacturing system provided by this application. Please refer to Figure 1 The method provided in this embodiment may include:

[0032] S101. Construct a data collection point catalog based on the target production task of the intelligent manufacturing system. The data collection point catalog is a collection point associated with the target production task determined from various monitoring points within the control range of the intelligent manufacturing system, including a plurality of collection point configuration information, each of which includes a data type, location information, and communication information of the collection point.

[0033] Specifically, the intelligent manufacturing system is a production system that integrates advanced manufacturing technology and information technology. Through the intelligent manufacturing system, intelligent monitoring, optimization and management of the production process can be achieved.

[0034] Furthermore, the target production task is a specific task or goal set in the intelligent manufacturing system to complete the production of a specific product or service. In specific implementation, the specific content of the target production task is set according to actual needs and is not limited in this embodiment.

[0035] Furthermore, the data collection point catalog is a database or directory built to complete the target production task. It should be noted that the data collection point catalog records the detailed information of all data collection points associated with the task.

[0036] Furthermore, the monitoring points within the control range of the intelligent manufacturing system are various devices, sensors, systems, etc. used to monitor the production process in real time. Various types of data can be collected through the collection points.

[0037] Furthermore, the configuration information of each collection point includes the data type of the data collected by the collection point, the location information of the collection point, the communication information of the collection point, etc. The collection point configuration information can guide the implementation of data collection work.

[0038] In specific implementation, the data type in the collection point configuration information refers to the type of data collected, which may include temperature, pressure, image, sound, etc.; the location information in the collection point configuration information is used to determine the installation location and layout of the collection point; the communication information in the collection point configuration information refers to the communication method and protocol between the collection point and the intelligent manufacturing system, and the communication information is used to ensure that the data can be accurately transmitted to the data center.

[0039] The following is a specific example to introduce the process of building a data collection point catalog library in detail:

[0040] (1) Determine target devices included in the intelligent manufacturing system based on registration data of the intelligent manufacturing system.

[0041] Specifically, the target devices are all devices included in the intelligent manufacturing system. The specific types of target devices are set according to actual needs and are not limited in this embodiment. For example, in one embodiment, the target devices may include devices, sensors, and systems.

[0042] In specific implementation, the intelligent manufacturing system records target device information such as equipment, sensors, and systems, including device name, model, location, interface type, data format, etc., which is used for intelligent manufacturing system management and data collection preparation.

[0043] (2) Filtering the target equipment associated with the target production task to obtain the collection point configuration information.

[0044] During specific implementation, target devices required to achieve the target production task are determined based on the target production task, and information corresponding to these target devices is obtained.

[0045] It should be noted that detailed information such as the device type, location, interface, data format, etc. of the target device associated with the target production task can be obtained, and the target device can be determined as a collection point, and the information corresponding to the obtained target device can be determined as collection point configuration information.

[0046] (3) The configuration information of the collection points that have passed the test is saved to obtain the data collection point catalog library.

[0047] Specifically, the collection points may be tested based on a traditional collection point testing method, or may be tested based on a neural network collection point testing method, which is not limited in this embodiment.

[0048] Furthermore, the collection points are tested, and those that cannot normally obtain data are eliminated. The collection point configuration information corresponding to the remaining collection points is saved in the data collection point catalog library. In specific implementation, based on the collection points that have passed the test and the actual location of the target equipment corresponding to the production environment, the data collection gateway is installed as needed, and the installation-related information is registered in the corresponding data collection point catalog library. The information contained in the database includes the actual installation location of the collection point, the required collection items, the interface communication protocol, the reporting server information, the data communication protocol used for reporting, and the data format.

[0049] As an optional embodiment, the screening of the target equipment associated with the target production task includes: determining the equipment to be used and the production equipment associated with the equipment to be used according to the target production task, determining the production equipment that needs to be monitored according to the idle status of the production equipment, and determining the equipment to be used and the production equipment that needs to be monitored as the target equipment.

[0050] The processing method for multimodal data acquisition in an intelligent manufacturing system provided in this embodiment determines the required target equipment by the target production task, so that data acquisition is more focused on the collection points that are useful for the production task, thereby improving the relevance and effectiveness of data acquisition. Furthermore, the collection points are tested to ensure that they can operate normally and accurately report data. This process ensures that the collection points can accurately provide data and ensures the smooth completion of the target production task. In this way, by verifying and configuring the target equipment corresponding to the target production task, an accurate and reliable data collection point catalog can be constructed, ensuring the smooth completion of the target production task.

[0051] Figure 2 This is a schematic diagram of the data collection point cataloging library construction process shown in an exemplary embodiment of this application. Figure 2 , select the data source type based on the device, sensor, and system, and identify it as a collection point. Fill in their corresponding location, type, parameters, interface, data format, and data communication method. Then determine whether each collection point requires a collection gateway, install the gateway for the required collection points, and finally perform a docking test. Correct any that fail the test until all collection points are working properly. Based on this, a data collection point catalog is generated.

[0052] S102. Determine a collection plan based on the target production task and the data collection point catalog library, wherein the collection plan at least includes a collection sequence for different collection points, a collection template and a data parsing algorithm corresponding to each collection point, and the collection templates and data parsing algorithms for collection points of different data types are different. The collection cycle and collection sequence are determined based on the timeliness information of the target production task.

[0053] Specifically, a collection plan is a data collection plan and strategy developed for a target production task. It should be noted that the specific content of a collection plan is determined based on actual needs and is not limited in this embodiment. In practice, a collection plan may include elements such as data collection points, the corresponding collection timing, data types, collection cycles, and analysis algorithms.

[0054] Furthermore, the collection timing is a time indicator for collecting data at a collection point, and data is collected according to the collection timing.

[0055] Furthermore, the collection template is used to guide the specific operations during the data collection process to ensure the consistency and accuracy of the data. The data parsing algorithm is used to convert the collected raw data into a format that can be processed and analyzed. Different data types require different collection templates and parsing algorithms.

[0056] Furthermore, the collection cycle refers to the frequency or time interval of data collection. This can be determined based on the timeliness of the target production task and the need for data updates. The collection sequence refers to the order and timing of data collection. This ensures an orderly data collection process and avoids data loss or confusion.

[0057] Furthermore, a data parsing algorithm is an algorithm or program required to convert raw data into a format that is easy to process or analyze. It should be noted that due to the diverse data formats collected at various collection points, different data parsing algorithms are required to adapt to different data types and formats. The data parsing algorithm's corresponding components or plug-ins convert the content into human-readable plain text information. These components or plug-ins are developed by the data format provider or a third party based on the interface documentation.

[0058] In the specific implementation, in this step, the data type corresponding to the branch data point determines the appropriate collection template, and the collection cycle and timing are determined according to the target production task to complete the construction of the collection plan.

[0059] A specific embodiment is given below to introduce the process of determining the acquisition plan in detail:

[0060] (1) Determine the collection timing based on the target production task and the idle time of each collection point.

[0061] Specifically, the target production task includes a data demand time point, and the collection timing corresponding to each collection point is determined based on the demand time point and the idle time available for collection at the collection point.

[0062] (2) Obtaining a collection template corresponding to each collection point based on the data collection point catalog library.

[0063] In specific implementation, a collection template is generated based on the interface standards and other data contained in the factory information of the target device, combined with the data requirements of the intelligent manufacturing system.

[0064] (3) Extracting the feature vector corresponding to the data in the acquisition point, and obtaining the extraction algorithm and the data analysis algorithm corresponding to the feature vector.

[0065] Specifically, the feature vectors of the data in the collection points may be obtained based on a traditional feature extraction method, or based on a neural network feature extraction method, which is not limited in this embodiment.

[0066] In specific implementations, for each acquisition point, a corresponding feature vector is obtained. Matching is performed based on the feature vectors to determine the extraction algorithm and data parsing algorithm corresponding to the feature vector. The resulting extraction algorithm and data parsing algorithm are then used as the extraction algorithm and data parsing algorithm for this acquisition point. It can be understood that each acquisition point corresponds to a feature extraction algorithm, which converts the raw data into a representative feature vector for easier analysis and processing.

[0067] (3) Determine the acquisition plan based on the acquisition template, the extraction algorithm and the data analysis algorithm.

[0068] During implementation, a detailed collection plan is developed by integrating the collection template, extraction algorithm, and data parsing algorithm. It should be noted that the collection plan should clearly define the data collection time, frequency, data format requirements, feature extraction, and data parsing steps for each collection point.

[0069] The method for processing multimodal data acquisition in intelligent manufacturing systems provided in this embodiment ensures standardized and efficient data acquisition through the use of acquisition templates. By selecting appropriate extraction and data analysis algorithms based on the feature vectors of the acquisition points, it is possible to specifically process individual acquisition points with different data types. Thus, by combining the acquisition template, extraction algorithm, and data analysis algorithm to generate an acquisition plan for each acquisition point, the reliability and accuracy of data acquisition can be guaranteed.

[0070] Figure 3 This is a schematic diagram of the acquisition plan determination process shown in an exemplary embodiment of this application. Figure 3 First, select the collection points in the data collection point catalog. These collection points can include devices, sensors, and systems. Configure the interface standards provided by the manufacturer for these collection points, as well as the data requirements of the intelligent manufacturing system. Based on this, generate a collection template. Then, determine whether each collection point has periodicity, and configure the collection period for periodic collection points. For collection points that do not have periodicity, select the corresponding data parsing algorithm and test it. Collection points that fail the test will return to the step of configuring the interface standards provided by the manufacturer. This process continues until all collection points have been tested successfully, resulting in the final collection plan.

[0071] S103: Collect multimodal data from the collection points in the data collection point catalog based on the collection plan, and preprocess the multimodal data according to the data analysis algorithm corresponding to each collection point. The data formats and data types of the collection points are different, and the data formats of the preprocessed multimodal data are the same.

[0072] Specifically, multimodal data refers to a collection of information consisting of two or more data in different forms or sources. Multimodal data has broad application value in intelligent manufacturing systems. It should be noted that the specific data types included in multimodal data are determined based on actual needs and are not limited in this embodiment. In specific implementations, multimodal data can include data types such as text, images, audio, video, and sensor data.

[0073] Furthermore, by pre-processing the multimodal data, the quality and usability of the data can be improved. For example, in one embodiment, the multimodal data is pre-processed through steps such as data cleaning, format conversion, and annotation.

[0074] In specific implementation, the data indicator basis for removing noise can be formed based on the manufacturer's parameter standards corresponding to the collection point, and then the data items required for the target production task can be obtained and bound to the collected multimodal data to determine the valid items of the collected data.

[0075] A specific embodiment is given below to introduce the process of multimodal data collection in detail:

[0076] (1) Acquire each acquisition point according to the acquisition sequence and acquisition period, wherein real-time multimodal data is acquired according to the real-time acquisition scheme, and non-real-time multimodal data is acquired according to the non-real-time acquisition scheme, wherein the non-real-time acquisition scheme at least includes an acquisition trigger condition.

[0077] Specifically, based on the timeliness information obtained from the target production task, the collection period of each collection point is determined, and multimodal data is collected according to the collection period. In specific implementation, multimodal data can be collected every other sampling period.

[0078] It should be noted that, with time as the primary index, the multimodal data collected regularly according to the collection cycle will be stored in the corresponding target server memory in real time.

[0079] Furthermore, the specific data type and content of the triggering data are set according to actual needs and are not limited in this embodiment.

[0080] (2) The real-time multimodal data is stored in the target server memory, and the non-real-time multimodal data is stored in the target hard disk.

[0081] Specifically, the real-time multimodal data in the target server memory can be stored in a time series database according to business needs. When triggering data is detected, the multimodal data of each collection point is collected and stored in the target hard disk.

[0082] It should be noted that when triggering data overlaps with a certain collection cycle, simultaneous real-time and non-real-time collection is required. In this case, the collection priority is determined according to the sequence of planned collection timings. For example, in one embodiment, non-real-time tasks are collected first, followed by real-time tasks. For another example, in another embodiment, real-time tasks are collected first, followed by non-real-time tasks.

[0083] Furthermore, in order to ensure the security of multimodal data and prevent tampering of multimodal data, the corresponding multimodal data is encrypted during the multimodal data transmission process. After the server receives the encrypted data, it decrypts it accordingly and restores it to multimodal data.

[0084] Furthermore, multimodal data can adopt a distributed storage architecture. In specific implementation, distributed storage systems such as Hado0p and HDFS can be used to solve large-scale data storage problems and improve data access speed and scalability.

[0085] The processing method for multimodal data acquisition in an intelligent manufacturing system provided by this embodiment ensures the timeliness of data by performing regular acquisition according to a set acquisition cycle, enabling the system to obtain the latest production status information in a timely manner. Furthermore, the non-real-time acquisition method with trigger data as the main index enables the system to respond quickly when a specific event or condition triggers, further improving the accuracy and pertinence of data acquisition. Furthermore, the multimodal data collected in real time is stored in the target server memory, and the fast access characteristics of the memory are utilized to improve the processing speed of real-time data, enabling the business system to respond quickly and make decisions. Furthermore, the data collected in non-real time is stored in the target hard disk. The large capacity and persistence characteristics of the hard disk ensure the long-term preservation and accessibility of historical data at any time, providing a solid foundation for subsequent data analysis and decision support. At the same time, the order and time arrangement of real-time and non-real-time acquisition are rationally planned through the acquisition time sequence, avoiding resource conflicts and data confusion during the data acquisition process, and improving the overall performance and stability of the system. In this way, by rationally acquiring multimodal data, the system can obtain more comprehensive and rich production information, providing more dimensional input for data analysis, and improving the operating efficiency and reliability of the intelligent manufacturing system.

[0086] A specific embodiment is given below to introduce the preprocessing process of multi-source data in detail:

[0087] (1) Generate a data indicator basis based on the parameter standards carried by the intelligent manufacturing system, and determine valid data based on the data indicator basis; the data indicator basis includes data items required for the target production task.

[0088] Specifically, parameter standards are a set of pre-defined parameters and rules within intelligent manufacturing systems that guide data collection and processing. In practice, these standards can include data format, sampling frequency, communication protocol, data processing accuracy, and more. These standards are crucial for determining data collection and processing methods.

[0089] Furthermore, the data related to the target production task in the intelligent manufacturing system is determined by the data indicator basis. In specific implementation, the data included in the data indicator basis can be determined as data collection points useful for the target production task, and the data collected at these collection points can be determined as valid data.

[0090] (2) Processing the valid data of the corresponding collection points based on the data analysis algorithm to generate the multimodal data in a unified format.

[0091] Specifically, the data parsing algorithm converts the collected valid data into data with a unified format and easy to process, to ensure that the processed data meets the further processing requirements of the intelligent manufacturing system.

[0092] Furthermore, the data corresponding to a collection point can be processed using a data analysis algorithm to produce a corresponding feature. In this way, different collection points can be processed in a targeted manner, ensuring the accuracy of preprocessing.

[0093] The processing method for multimodal data collection in an intelligent manufacturing system provided in this embodiment generates a data indicator basis based on the parameter standards carried by the intelligent manufacturing system and determines valid data accordingly, ensuring that the collected data has direct relevance and practical value to the target production task, thereby improving the validity and relevance of the data. Furthermore, the application of a data parsing algorithm unifies the format of valid data, eliminating format differences between data from different sources and providing a unified data format for subsequent data analysis, thereby improving the efficiency of data processing. In this way, by improving the overall quality of the data, a more comprehensive and accurate information foundation is provided for the intelligent manufacturing system.

[0094] In one possible implementation, multimodal data of each acquisition point is acquired according to the acquisition sequence, and the acquisition process of each acquisition point is specified by the acquisition template corresponding to the acquisition point; valid data content in the acquired multimodal data is filled into a data storage template, where different data types correspond to different data storage templates, while the same data storage templates correspond to the same data type, and the values in the data storage template are different; the acquisition template is parsed based on a data parsing algorithm to extract feature information of the values in the acquisition template, where the data parsing algorithm corresponds to the acquisition template.

[0095] The method provided in this embodiment ensures standardized and regularized data collection by establishing a corresponding collection template for each collection point. The collection process at each collection point is strictly carried out in accordance with the template, avoiding arbitrariness and uncertainty in data collection and improving data accuracy and reliability. By filling the data storage template with valid data content, this method not only achieves structured data management but also facilitates rapid data retrieval and access. This orderly data storage method helps improve the efficiency of data storage and management and reduce data maintenance costs. Thus, by standardizing the collection process, improving data collection efficiency, flexibly addressing the diversity of data types, and ensuring efficient and accurate data analysis.

[0096] Figure 4 This is a schematic diagram of a multimodal data preprocessing process shown in an exemplary embodiment of the present application. Figure 4 , complete the reporting of multimodal data according to the standards provided by the manufacturer, then obtain the data required for the target production task and bind it with the collected data. After format conversion, filter out non-compliant information, and define the attributes of the filtered data by label assignment. After the feature value extraction model extracts the feature representation, generate a suitable data preprocessing model to complete the preprocessing process of multimodal data.

[0097] S104: Input the pre-processed multimodal data into a data fusion model, match the fusion algorithm, and fuse and store the multimodal data.

[0098] Specifically, the data fusion model integrates information from multiple data sources or modalities through appropriate algorithms and technical means to obtain a more comprehensive and accurate data representation.

[0099] Furthermore, a fusion algorithm is a specific algorithm or method used in the data fusion process to merge data from multiple data sources or modalities into a unified data set. It should be noted that the specific type of fusion algorithm is determined based on actual needs and is not limited in this embodiment. For example, fusion algorithms may include data merging algorithms, data aggregation algorithms, data splitting algorithms, etc.

[0100] In specific implementation, an appropriate fusion algorithm is selected to fuse multimodal data based on the characteristics of the multimodal data and the business requirements of the target production task.

[0101] It should be noted that after the multimodal data is preprocessed, the label information in the multimodal data can be assigned a value. In specific implementation, the label information of the multimodal data can be a kind of metadata, which is attached to the multimodal data through metadata to provide additional information about the multimodal data to help understand and process the multimodal data. It should be noted that the specific content of the label information is determined according to actual needs, and is not limited to this in this embodiment. In specific implementation, the label information may include the multimodal data type, multimodal data source, multimodal data quality, multimodal data owner, multimodal data creation time, and multimodal data relationship, where the assignment information mainly refers to the actual data filling for these tags, and the filling source is the automatic filling in of the prior binding and data acquisition.

[0102] Furthermore, after labeling the multimodal data, a corresponding feature extraction algorithm is selected and bound based on the data type and the processing objectives of the target task. In specific implementations, the multimodal data corresponding to a single acquisition point is mapped to one or more feature extraction algorithms. These algorithms are then used to convert the multimodal data into representative feature vectors for subsequent processing and analysis. It should be noted that the feature extraction process can be performed based on a pre-built feature value extraction model.

[0103] A specific embodiment is given below to illustrate the multimodal data fusion process in detail.

[0104] (1) Based on the target production task, multiple target collection points that need to be fused are determined.

[0105] Specifically, according to the specific requirements of the target production task, multiple target collection points where data collection is required are first determined.

[0106] In the specific implementation, according to the specific requirements of the target production task, the target collection points are determined to include sensors, surveillance cameras, audio collection equipment, etc. on the production line.

[0107] (2) Determine a fusion algorithm based on the business scenario corresponding to the target production task and the characteristic requirements of the target production task.

[0108] Specifically, a business scenario refers to the actual production environment and workflow in which the target production task occurs. Different business scenarios may have different requirements for data collection and processing, so it is necessary to determine the appropriate data collection and processing strategy based on the specific scenario.

[0109] Furthermore, according to the needs of the target production task, the characteristics of the fused data are determined and identified as characteristic requirements.

[0110] In specific implementation, factors such as the data modality type, data volume, real-time requirements, and subsequent analysis needs are considered in combination with feature requirements and business scenarios, and relevant fusion algorithms are bound. Through the corresponding fusion algorithms, the data can be unified into a data set.

[0111] (3) fusing the multimodal data of the multiple target acquisition points based on the fusion algorithm to generate the target data set.

[0112] In specific implementation, the data of each collection point is read based on the fusion algorithm, the fusion algorithm is applied to integrate the data, and the fused data is saved to obtain the target data set.

[0113] Furthermore, through the fusion of multimodal data, data belonging to the same time can be divided together. At the same time, through the fusion of multimodal data, data representing the same quality information can be fused to facilitate subsequent analysis.

[0114] It should be noted that data from multiple different collection points can be fused.

[0115] Furthermore, the fused data can represent information from different dimensions of the intelligent manufacturing system. In practice, the fused data can be used to obtain process information, process information, equipment information, and other information corresponding to the target production task.

[0116] The method provided by the present invention matches the corresponding fusion algorithm according to the target production task, that is, the fusion algorithm corresponds to the target production task and is determined according to the task requirement matching, rather than being processed from the data perspective. The associated multimodal data related to the target production task is determined, and the associated multimodal data is fused using the fusion algorithm. Among them, a target production task may have multiple data usage requirements, and each requirement requires data fusion from multiple points to obtain information, that is, there may be multiple requirements, and each requirement corresponds to a different fusion algorithm because the target information and information source it wants to obtain are different. The associated data is selected according to the requirement, and then the associated multimodal data is fused according to the fusion algorithm according to the requirements of serving the target production task.

[0117] This embodiment provides a method for processing multimodal data collection in intelligent manufacturing systems. By fusing multimodal data from collection points, a more comprehensive and accurate data representation can be obtained. Furthermore, based on the business scenario and specific requirements, the fusion algorithm most suitable for the current target production task can be selected, ensuring the reliability and accuracy of data fusion.

[0118] The following is a specific example to explain the process of saving multimodal data in detail:

[0119] (1) Determine the target collection point as an upper branch of a tree structure according to the association relationship of the multiple target collection points.

[0120] Specifically, in a tree structure, an upper branch generally refers to a portion closer to the root node. In a specific implementation, the target collection point is determined as an upper branch. In other words, the target collection point is used as the top level or high-level classification of data organization.

[0121] Furthermore, the target production task corresponds to several target collection points, and the tree structure may also have several corresponding upper branches.

[0122] (2) The multimodal data is sequentially filled into the upper branches according to the attribute information to obtain corresponding lower branches; the tree structure is provided with cataloging information based on the attributes.

[0123] Specifically, lower-level branches are the parts of the tree structure that are located below the upper-level branches. They represent further subdivision or specific content of the data. In this context, lower-level branches will be assigned to each upper-level branch based on the source of the multimodal data.

[0124] Furthermore, cataloging information is the process of classifying, identifying, and indexing multimodal data. In a tree structure, cataloging information is used to describe the specific content and structure of each node (including upper and lower branches), facilitating the retrieval and management of multimodal data.

[0125] It should be noted that multimodal data can be located and retrieved through cataloging information.

[0126] In specific implementation, a tree structure and cataloging information are constructed for all target collection points and corresponding multimodal data until all multimodal data are processed.

[0127] The processing method for multimodal data collection in an intelligent manufacturing system provided in this embodiment provides an intuitive, hierarchical data organization method through a tree structure, allowing multimodal data to be arranged in an orderly manner according to specific logic and rules, helping to reduce data confusion and redundancy and improve data manageability and accessibility. Furthermore, by identifying and indexing each node of the tree structure through catalog information, the efficiency of data retrieval can be greatly improved. Users can quickly locate specific collection points or data sets as needed without having to traverse the entire target data set. Furthermore, the tree structure makes data management more flexible and convenient. The levels and nodes of the tree structure can be adjusted according to actual needs, collection points and target data sets can be added or deleted, and catalog information can be updated to adapt to the ever-changing production environment and business needs. At the same time, constructing multimodal data into a tree structure helps to better utilize the inherent connections and patterns of the data during the data fusion and analysis stages. By analyzing the relationships and differences between different nodes in the tree structure, potential connections and trends between data can be discovered, providing more comprehensive and in-depth information for production optimization and decision support.

[0128] A specific embodiment is given below to illustrate the display process of multimodal data in detail:

[0129] (1) Determine data query requirements based on the real-time service module.

[0130] Specifically, a communication protocol is the rules and agreements that must be followed by both parties during data transmission. It's important to note that in intelligent manufacturing systems, communication protocols are used to ensure smooth data exchange between different devices and systems. These protocols define the data format, transmission method, and error detection and correction mechanisms.

[0131] Furthermore, the real-time service module is the part of the intelligent manufacturing system responsible for real-time data processing and display. The real-time service module receives real-time data from various collection points according to the communication protocol of the intelligent manufacturing system, processes and analyzes these data, and finally displays them to the user in real time through a graphical interface or other means.

[0132] During implementation, the real-time service module's configuration and parameters are determined based on the intelligent manufacturing system's communication protocol. The module then establishes connections with various data collection points over the network, receiving the multimodal data they transmit. This real-time data is then processed and analyzed, and presented to the user using a pre-defined display method.

[0133] During specific implementation, when data query is required, relevant data query requirements are generated.

[0134] (2) Determine a data retrieval scope and a data retrieval condition based on the data query requirement and the tree structure, and locate and display the multimodal data based on the data retrieval scope and the data retrieval condition.

[0135] Specifically, the data retrieval range refers to the boundary of the multimodal data set to be searched, specified by the user, when performing multimodal data retrieval. In a specific implementation, in a tree structure, the data retrieval range may correspond to one or more branches in the tree structure.

[0136] Furthermore, data retrieval conditions refer to specific rules or conditions set by users when performing multimodal data retrieval for filtering multimodal data. In specific implementations, data retrieval conditions can be defined based on attributes, values, or relationships between multimodal data.

[0137] In specific implementations, the scope of data retrieval is determined based on the constructed tree structure. This scope can be the entire tree, a subtree, or multiple specific nodes within the tree structure. Then, based on the data retrieval criteria set by the user, the tree structure is searched and filtered, and branches that meet the criteria are located. The resulting multimodal data is then presented to the user using the pre-defined display method. This process allows users to quickly find and view the data they are interested in, improving the efficiency and convenience of data access.

[0138] It should be noted that the data displayed in non-real time is multimodal data collected in non-real time and stored in the target hard disk.

[0139] The processing method for multimodal data acquisition of an intelligent manufacturing system provided in this embodiment determines a real-time service module based on the communication protocol of the intelligent manufacturing system and displays multimodal data in real time, so that enterprises can instantly obtain key data in the production process. This real-time data visualization not only improves the transparency of the production process, but also enables managers to quickly respond to changes in production, adjust production strategies in a timely manner, and ensure the smooth operation of the production process. Furthermore, the data retrieval range and data retrieval conditions are determined based on the tree structure, making the data retrieval process more intuitive and efficient. Users can quickly locate specific data sets as needed, avoiding the trouble of blindly searching in massive data. In this way, non-real-time saved data can be quickly queried, improving the efficiency and accuracy of data utilization.

[0140] Figure 5 This is a schematic diagram of a multimodal data fusion process shown in an exemplary embodiment of the present application. Figure 5 ,First, the collection points are selected as the fusion objects, and the data of the ,collection points are acquired, and the fusion objects are cataloged using a tree structure. ,Multiple methods are used to bind the corresponding fusion algorithms for ,fusion objects. Finally, a data fusion model is generated according to the ,selected fusion algorithm to fuse the data of the collection points.

[0141] The processing method for multimodal data acquisition in intelligent manufacturing systems provided in this embodiment can process diverse data from different devices, sensors, and systems, supports multiple data formats and communication protocols, and significantly improves the flexibility and coverage of data acquisition. By constructing a data acquisition source catalog library, the acquisition scope is clearly defined and managed, making the data acquisition process more orderly and efficient. Data acquisition gateways are installed based on the acquisition source attributes, enabling rapid connection and system access between physical devices and sensors, improving the convenience of data acquisition. Furthermore, the collected data is cleaned, formatted, and annotated, effectively improving its quality and usability, providing a solid foundation for subsequent data analysis and decision-making. Furthermore, a tree structure is used to catalog the fused data, facilitating rapid retrieval of required information and improving the efficiency and accuracy of data retrieval. The use of distributed storage systems such as Hadoop and HDFS solves the problem of large-scale data storage and improves data access speed and scalability. Data is encrypted during data transmission to ensure data security and prevent data tampering. After receiving the data on the server, it is decrypted to ensure data integrity and security. Furthermore, the tree structure allows for convenient access and use of collected, processed, and integrated multimodal data, supporting various application scenarios such as production monitoring and optimized decision-making, thereby improving the scientific nature and accuracy of decision-making.

[0142] Corresponding to the aforementioned embodiment of a processing method for multimodal data acquisition of an intelligent manufacturing system, the present application also provides an embodiment of a processing device for multimodal data acquisition of an intelligent manufacturing system.

[0143] The embodiment of the processing device for multimodal data acquisition of an intelligent manufacturing system in the present application can be applied to the processing equipment for multimodal data acquisition of an intelligent manufacturing system. The embodiment of the device can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the processing equipment for multimodal data acquisition of an intelligent manufacturing system in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 6 As shown, this is a hardware structure diagram of the processing device for multimodal data acquisition of the intelligent manufacturing system of this application, except Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, the processing device for multimodal data acquisition of the intelligent manufacturing system in which the device in the embodiment is located may also include other hardware according to the actual function of the processing device for multimodal data acquisition of the intelligent manufacturing system, which will not be described in detail.

[0144] Figure 7 This is a structural diagram of the first embodiment of the processing device for multimodal data acquisition of intelligent manufacturing system provided by this application. Figure 7 The device provided in this embodiment includes a construction module 710, a determination module 720, a collection module 730 and a fusion module 740; wherein,

[0145] The construction module 710 is used to construct a data collection point catalog based on the target production task of the intelligent manufacturing system; the data collection point catalog is a collection point associated with the target production task determined from various monitoring points within the control range of the intelligent manufacturing system, including a plurality of collection point configuration information, each of which includes data type, location information, and communication information of the collection point;

[0146] The determination module 720 is configured to determine a collection plan based on the target production task and the data collection point catalog library, wherein a collection template and a data parsing algorithm are determined based on the data type, with the collection template and data parsing algorithm being different for each data type, and a collection period and a collection sequence are determined based on the timeliness information of the target production task;

[0147] The acquisition module 730 is configured to acquire multimodal data from the data acquisition point catalog based on the acquisition plan, and preprocess the multimodal data according to a data parsing algorithm corresponding to each data. The multimodal data may have different data formats and data types, but the preprocessed multimodal data may have the same data format.

[0148] The fusion module 740 is configured to input the pre-processed multimodal data into a data fusion model, match the fusion algorithm, and fuse and store the multimodal data.

[0149] The device of this embodiment can be used to perform Figure 1 The steps, specific implementation principles and implementation processes of the method embodiment shown are similar and will not be repeated here.

[0150] Optionally, the fusion module 740 is specifically configured to determine a plurality of target collection points requiring data fusion based on the target production task;

[0151] The fusion module 740 is further specifically configured to determine a fusion algorithm based on the business scenario corresponding to the target production task and the characteristic requirements of the target production task;

[0152] The fusion module 740 is further specifically configured to fuse the multimodal data from the same target acquisition point based on the fusion algorithm to generate the target data set.

[0153] Optionally, the fusion module 740 is further configured to determine the target collection point as an upper branch of the tree structure;

[0154] The fusion module 740 is further specifically configured to sequentially fill the upper branches with the multimodal data according to attribute information to obtain corresponding lower branches; the tree structure is provided with cataloging information based on the attributes.

[0155] Optionally, the construction module 710 is specifically configured to determine a target device included in the intelligent manufacturing system based on registration data of the intelligent manufacturing system;

[0156] The construction module 710 is further specifically configured to filter the target device associated with the target production task to obtain the collection point configuration information;

[0157] The construction module 710 is further specifically configured to save the configuration information of the collection points that have passed the test to obtain the data collection point catalog library.

[0158] Optionally, the determining module 720 is specifically configured to obtain a collection template corresponding to each collection point based on the data collection point catalog library;

[0159] The determination module 720 is further specifically configured to extract a feature vector corresponding to the data at the collection point, and obtain an extraction algorithm corresponding to the feature vector and the data parsing algorithm;

[0160] The determination module 720 is further specifically configured to determine the acquisition plan based on the acquisition template, the extraction algorithm, and the data parsing algorithm.

[0161] Optionally, the fusion module 740 is further specifically configured to determine a real-time service module based on the communication protocol of the intelligent manufacturing system, and display the multimodal data in real time based on the real-time service module;

[0162] The fusion module 740 is further specifically configured to determine a data retrieval range and a data retrieval condition based on the tree structure, and locate and display the multimodal data based on the data retrieval range and the data retrieval condition.

[0163] Optionally, the acquisition module 730 is specifically configured to regularly acquire the multimodal data according to the acquisition period and store the multimodal data in a target server memory;

[0164] The acquisition module 730 is further specifically configured to acquire the multimodal data in a non-real-time manner using trigger data as a primary index, and store the multimodal data in a target hard disk; wherein the order of the real-time acquisition and the non-real-time acquisition is planned based on the acquisition timing.

[0165] Optionally, the acquisition module 730 is further specifically configured to generate a data indicator basis based on the parameter standards carried by the intelligent manufacturing system, and determine valid data based on the data indicator basis; the data indicator basis includes data items required for the target production task;

[0166] The acquisition module 730 is further specifically configured to process the valid data of the corresponding acquisition point based on the data analysis algorithm to generate the multimodal data in a unified format.

[0167] Please continue to refer to Figure 6 The present application also provides a processing device for multimodal data acquisition of an intelligent manufacturing system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of any one of the methods provided in the first aspect of the present application are implemented.

[0168] The present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any one of the methods provided in the present application when the program is executed by a processor.

[0169] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0170] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0171] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for processing multimodal data acquisition in an intelligent manufacturing system, characterized in that: The method comprises: Building a data collection point catalog based on the target production task of the intelligent manufacturing system; the data collection point catalog is a collection point associated with the target production task determined from various monitoring points within the control range of the intelligent manufacturing system, including a plurality of collection point configuration information, each of which includes the data type, location information and communication information of the collection point; Determining a collection plan based on the target production task and the data collection point catalog library, wherein the collection plan at least includes a collection sequence for different collection points, a collection template and a data parsing algorithm corresponding to each collection point, the collection templates and data parsing algorithms being different for collection points of different data types, and the collection cycle and collection sequence being determined based on the timeliness information of the target production task; Collecting multimodal data from the collection points in the data collection point catalog based on the collection scheme, and preprocessing the multimodal data according to the data analysis algorithm corresponding to each collection point, wherein the data format and data type of each collection point are different, and the data format of the multimodal data after preprocessing is the same; The pre-processed multimodal data is input into a data fusion model, a fusion algorithm is matched, and the multimodal data is fused and stored.

2. The method according to claim 1, characterized in that The step of inputting the pre-processed multimodal data into a data fusion model, matching a fusion algorithm, fusing and storing the multimodal data includes: Determining multiple target collection points requiring data fusion based on the target production task; Determine a fusion algorithm based on the business scenario corresponding to the target production task and the characteristic requirements of the target production task; The multimodal data of the multiple target acquisition points are fused based on the fusion algorithm to generate a target data set.

3. The method according to claim 2, characterized in that The step of fusing the multimodal data of the plurality of target acquisition points based on the fusion algorithm to generate a target data set includes: Determining the target collection points as upper branches of a tree structure according to the association relationship of the multiple target collection points; The multimodal data is sequentially filled into the upper branches according to the attribute information to obtain corresponding lower branches; the tree structure is provided with cataloging information based on the attributes.

4. The method according to claim 1, wherein The data collection point cataloging library is constructed based on the target production tasks of the intelligent manufacturing system, including: Determining target devices included in the intelligent manufacturing system based on registration data of the intelligent manufacturing system; Filtering the target device associated with the target production task to obtain the collection point configuration information; The configuration information of the collection points that pass the test is saved to obtain the data collection point catalog library.

5. The method according to claim 1, wherein The determining of the collection plan based on the target production task and the data collection point catalog library includes: Determine the collection timing based on the target production task and the idle time of each collection point; Acquire a collection template corresponding to each collection point based on the data collection point catalog library; Extracting a feature vector corresponding to the data at the acquisition point, and obtaining an extraction algorithm and a data analysis algorithm corresponding to the feature vector; The acquisition plan is determined based on the acquisition template, the extraction algorithm and the data parsing algorithm.

6. The method according to claim 3, characterized in that After fusing and storing the multimodal data, the method further includes: Determine data query requirements based on real-time service modules; A data retrieval scope and a data retrieval condition are determined based on the data query requirement and the tree structure, and the multimodal data are located and displayed based on the data retrieval scope and the data retrieval condition.

7. The method according to claim 1, characterized in that The acquisition scheme includes at least a real-time acquisition scheme and a non-real-time acquisition scheme, and the acquisition of multimodal data from the data acquisition point catalog library based on the acquisition scheme includes: Collecting each collection point according to the collection timing and collection period, wherein real-time multimodal data is collected according to the real-time collection scheme, and non-real-time multimodal data is collected according to the non-real-time collection scheme, wherein the non-real-time collection scheme at least includes a collection trigger condition; The real-time multimodal data is stored in the target server memory, and the non-real-time multimodal data is stored in the target hard disk.

8. The method according to claim 1, characterized in that The collecting multimodal data from the data collection point catalog library based on the collection plan includes: Generate a data indicator basis based on the parameter standards carried by the intelligent manufacturing system, and determine valid data based on the data indicator basis; the data indicator basis includes data items required for the target production task; The valid data of the corresponding collection points are processed based on the data analysis algorithm to generate the multimodal data in a unified format.

9. The method according to claim 1, characterized in that The collecting of multimodal data from the collection points in the data collection point catalog based on the collection scheme, and preprocessing of the multimodal data according to the data analysis algorithm corresponding to each collection point, includes: Collecting multimodal data at each collection point according to the collection time sequence, wherein the collection process of each collection point is specified by the collection template corresponding to the collection point; Fill the valid data content in the collected multimodal data into the data storage template. Different data types correspond to different data storage templates. The same data type corresponds to the same data storage template, but the values in the data storage template are different. The acquisition template is parsed based on a data parsing algorithm to extract feature information of the numerical values in the acquisition template, and the data parsing algorithm corresponds to the acquisition template.

10. A processing device for multimodal data acquisition, characterized in that: The device includes a construction module, a determination module, a collection module and a fusion module; wherein, The construction module is used to construct a data collection point catalog library based on the target production task of the intelligent manufacturing system; the data collection point catalog library is a collection point associated with the target production task determined from various monitoring points within the control range of the intelligent manufacturing system, including a plurality of collection point configuration information, each of which is a data type, location information, and communication information of the collection point; The determination module is configured to determine a collection plan based on the target production task and the data collection point catalog library, wherein the collection plan includes at least a collection sequence for different collection points, a collection template and a data parsing algorithm corresponding to each collection point, the collection templates and data parsing algorithms being different for collection points of different data types, and the collection cycle and collection sequence being determined based on the timeliness information of the target production task; The acquisition module is configured to acquire multimodal data from the acquisition points in the data acquisition point catalog based on the acquisition plan, and preprocess the multimodal data according to a data analysis algorithm corresponding to each acquisition point, wherein the data formats and data types of the respective acquisition points are different, and the data formats of the preprocessed multimodal data are the same; The fusion module is used to input the pre-processed multimodal data into a data fusion model, match the fusion algorithm, and fuse and store the multimodal data.

Images