Processing method and device for multi-modal data acquisition of intelligent manufacturing system

Abstract

The invention provides a processing method and device for multi-modal data acquisition of an intelligent manufacturing system. The method provided by the invention comprises the following steps: constructing a data acquisition point catalogue library based on a target production task of an intelligent manufacturing system; determining an acquisition scheme based on the target production task and the data acquisition point catalogue library; multi-modal data are collected from collection points of the data collection point catalogue library based on the collection scheme, the multi-modal data are preprocessed according to data analysis algorithms corresponding to the collection points, the data formats and the data types of the collection points are different, and the data formats of the preprocessed multi-modal data are the same; and inputting the preprocessed multi-modal data into the data fusion model, matching a fusion algorithm, and fusing and storing the multi-modal data. The processing method and device for multi-modal data acquisition of the intelligent manufacturing system provided by the invention are used for accurately and efficiently completing data acquisition.

Description

Technical Field

[0001] The present application relates to the field of intelligent manufacturing technology, and in particular to a method and device for processing multimodal data acquisition in an intelligent manufacturing system. Background Art

[0002] In today's rapidly developing information age, data collection, processing and analysis have become one of the core tasks in various fields, especially in the field of intelligent manufacturing. Acquiring multimodal data in the field of intelligent manufacturing is an important part of intelligent manufacturing. Multimodal data consists of multiple modalities such as text, images, audio, video, sensor data and device data. At the same time, different types of sensor devices output data in different formats. The combined multimodal data also provides a rich and comprehensive source of information for the system. Therefore, how to efficiently and accurately collect, process and fuse multimodal data has become a key issue that needs to be solved in the field of intelligent manufacturing.

[0003] When dealing with large-scale and diverse data, existing data processing methods can only identify different data through manual labeling, which often faces the problems of slow processing speed and low efficiency, and it is difficult to meet the needs of intelligent manufacturing systems for real-time data processing. At the same time, since intelligent manufacturing systems have different requirements for different tasks, it is difficult for technicians to obtain accurate and comprehensive data, which affects the accuracy of subsequent analysis and processing and cannot meet the actual needs. Summary of the invention

[0004] In view of this, the present application provides a method and device for processing multimodal data acquisition in an intelligent manufacturing system, so as to complete data acquisition accurately and efficiently.

[0005] Specifically, the present application is implemented through the following technical solutions:

[0006] The first aspect of the present application provides a method for processing multimodal data acquisition of an intelligent manufacturing system, the method comprising:

[0007] Building a data collection point catalog library based on the target production task of the intelligent manufacturing system; the data collection point catalog library is a collection point associated with the target production task determined from various monitoring points within the control range of the intelligent manufacturing system, including multiple collection point configuration information, each of which is a data type, location information and communication information of the collection point;

[0008] Determine a collection plan based on the target production task and the data collection point catalog library, wherein the collection plan at least includes the collection timing of different collection points, the collection template and data parsing algorithm corresponding to each collection point, the collection template and data parsing algorithm of collection points of different data types are different, and the collection cycle and collection timing are determined based on the timeliness information of the target production task;

[0009] Based on the collection scheme, multimodal data is collected from the collection points of the data collection point catalog library, and the multimodal data is preprocessed according to the data analysis algorithm corresponding to each collection point, the data format and data type of each collection point are different, and the data format of the preprocessed multimodal data is the same;

[0010] The pre-processed multimodal data is input into a data fusion model, a fusion algorithm is matched, and the multimodal data is fused and stored.

[0011] The second aspect of the present application provides a processing device for multimodal data acquisition of an intelligent manufacturing system, the device comprising a construction module, a determination module, a collection module and a fusion module; wherein:

[0012] The construction module is used to construct a data collection point catalog library based on the target production task of the intelligent manufacturing system; the data collection point catalog library is a collection point associated with the target production task determined from various monitoring points within the control range of the intelligent manufacturing system, including a plurality of collection point configuration information, each of which is a data type, location information and communication information of the collection point;

[0013] The determination module is used to determine a collection plan based on the target production task and the data collection point catalog library, wherein the collection plan at least includes the collection timing of different collection points, the collection template and data parsing algorithm corresponding to each collection point, the collection template and data parsing algorithm of collection points of different data types are different, and the collection cycle and collection timing are determined based on the timeliness information of the target production task;

[0014] The acquisition module is used to acquire multimodal data from the acquisition points of the data acquisition point catalog library based on the acquisition scheme, and preprocess the multimodal data according to the data analysis algorithm corresponding to each acquisition point, the data format and data type of each acquisition point are different, and the data format of the preprocessed multimodal data is the same;

[0015] The fusion module is used to input the pre-processed multimodal data into a data fusion model, match the fusion algorithm, and fuse and store the multimodal data.

[0016] The third aspect of the present application provides a processing device for multimodal data acquisition of an intelligent manufacturing system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods provided in the first aspect of the present application are implemented.

[0017] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods provided in the first aspect of the present application.

[0018] The processing method and device for multimodal data collection of intelligent manufacturing system provided by the present application switches the traditional comprehensive management mode of a single collection point to a standardized management mode in a database of a unified format. By constructing a data collection point catalog library, all collection points associated with the target production task are recorded in detail, including key configuration information such as data type, location information and communication information, so that all the points to be collected in the intelligent manufacturing system can be efficiently managed in a unified and standardized format in the database, which is conducive to conveniently obtaining the data format and communication protocol of each collection point in the post-collection control process, generating targeted collection templates, and making the data collection process more orderly and efficient through the data collection point catalog library, reducing the collection of redundant data, and improving the pertinence and accuracy of data collection. Further, the collection scheme takes into account the diversity of data types, customizes different collection templates and data analysis algorithms for each data type, ensures the accuracy and consistency of data during the collection process, and reasonably sets the collection cycle and collection sequence to meet the requirements of the intelligent manufacturing system for the timeliness of data collection. Further, through the data fusion model, the corresponding fusion algorithm is matched according to the characteristics of the data and the needs of the business scenario, and the data of different modes are fused into a unified data set, forming a more comprehensive and accurate data representation. In this way, by integrating the data from each collection point with the production tasks corresponding to the intelligent manufacturing system, the data acquisition of the production tasks is guaranteed efficiently and accurately, providing a guarantee for the completion of the production tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A flowchart of a first embodiment of a method for processing multimodal data acquisition in an intelligent manufacturing system provided in the present application;

[0020] Figure 2 A schematic diagram of a data collection point catalog library construction process shown in an exemplary embodiment of the present application;

[0021] Figure 3 A schematic diagram of a collection scheme determination process shown in an exemplary embodiment of the present application;

[0022] Figure 4 A schematic diagram of a multimodal data preprocessing process shown in an exemplary embodiment of the present application;

[0023] Figure 5 A schematic diagram of a multimodal data fusion process shown in an exemplary embodiment of the present application;

[0024] Figure 6A hardware structure diagram of a processing device for multimodal data collection of an intelligent manufacturing system where the processing device for multimodal data collection of an intelligent manufacturing system of this application is located;

[0025] Figure 7 This is a structural diagram of a first embodiment of a processing device for multimodal data acquisition in an intelligent manufacturing system provided in the present application. DETAILED DESCRIPTION

[0026] Here, exemplary embodiments are described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application.

[0027] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0029] The present application provides a method and device for processing multimodal data acquisition in an intelligent manufacturing system, so as to complete data acquisition accurately and efficiently.

[0030] Specific embodiments are given below to introduce the technical solution of the present application in detail.

[0031] Figure 1 This is a flowchart of the first embodiment of the processing method for multimodal data collection of the intelligent manufacturing system provided by this application. Please refer to Figure 1 , the method provided in this embodiment may include:

[0032] S101. Construct a data collection point catalog library based on the target production task of the intelligent manufacturing system; the data collection point catalog library is a collection point associated with the target production task determined from various monitoring points within the control range of the intelligent manufacturing system, including multiple collection point configuration information, each of which is a data type, location information and communication information of the collection point.

[0033] Specifically, the intelligent manufacturing system is a production system that integrates advanced manufacturing technology and information technology. Through the intelligent manufacturing system, intelligent monitoring, optimization and management of the production process can be achieved.

[0034] Furthermore, the target production task is a specific task or goal set in the intelligent manufacturing system to complete the production of a specific product or service. In specific implementation, the specific content of the target production task is set according to actual needs, and this embodiment is not limited to this.

[0035] Furthermore, the data collection point catalog library is a database or directory constructed to complete the target production task. It should be noted that the data collection point catalog library records the detailed information of all data collection points associated with the task.

[0036] Furthermore, the monitoring points within the control range of the intelligent manufacturing system are various devices, sensors, systems, etc. used to monitor the production process in real time. Various types of data can be collected through the collection points.

[0037] Furthermore, the configuration information of each collection point includes the data type of the data collected by the collection point, the location information of the collection point, the communication information of the collection point, etc. The collection point configuration information can guide the implementation of data collection work.

[0038] In specific implementation, the data type in the collection point configuration information refers to the type of data collected, and the data type may include temperature, pressure, image, sound, etc.; the location information in the collection point configuration information is used to determine the installation location and layout of the collection point; the communication information in the collection point configuration information refers to the communication method and protocol between the collection point and the intelligent manufacturing system, and the communication information is used to ensure that the data can be accurately transmitted to the data center.

[0039] A specific embodiment is given below to introduce the construction process of the data collection point catalog library in detail:

[0040] (1) Determine a target device included in the intelligent manufacturing system based on registration data of the intelligent manufacturing system.

[0041] Specifically, the target devices are all devices included in the intelligent manufacturing system. The specific types of the target devices are set according to actual needs and are not limited in this embodiment. For example, in one embodiment, the target devices may include devices, sensors, and systems.

[0042] In specific implementation, the intelligent manufacturing system records target equipment information such as equipment, sensors, and systems, including equipment name, model, location, interface type, data format, etc., which is used for intelligent manufacturing system management and preparation for data collection.

[0043] (2) Filter the target device associated with the target production task to obtain the collection point configuration information.

[0044] In specific implementation, the target devices required to achieve the target production tasks are determined based on the target production tasks, and the corresponding information of these target devices is obtained.

[0045] It should be noted that detailed information such as device type, location, interface, data format, etc. of the target device associated with the target production task can be obtained, and the target device can be determined as a collection point, and the information corresponding to the obtained target device can be determined as collection point configuration information.

[0046] (3) The configuration information of the collection points that have passed the test is saved to obtain the data collection point catalog library.

[0047] Specifically, the collection points may be tested based on a traditional collection point testing method, or may be tested based on a neural network collection point testing method, which is not limited in this embodiment.

[0048] Furthermore, the collection points are tested, the collection points that cannot normally obtain data are eliminated, and the collection point configuration information corresponding to the remaining collection points is saved in the data collection point catalog library. In specific implementation, the data collection gateway is installed as needed in combination with the collection points that have passed the test and the actual location of the target device corresponding to the production environment, and the installation-related information is registered in the corresponding data collection point catalog library, which contains information including the actual installation location of the collection point, the required collection item information, the interface communication protocol, the reporting server information, the data communication protocol used for reporting, and the data format.

[0049] As an optional embodiment, the screening of the target equipment associated with the target production task includes: determining the equipment to be used and the production equipment associated with the equipment to be used according to the target production task, determining the production equipment that needs to be monitored according to the idle state of the production equipment, and determining the equipment to be used and the production equipment that needs to be monitored as the target equipment.

[0050] The processing method for multimodal data collection of the intelligent manufacturing system provided in this embodiment determines the required target equipment through the target production task, so that data collection is more focused on the collection points that are useful for the production task, thereby improving the relevance and effectiveness of data collection. Furthermore, the collection points are tested to ensure that they can work normally and report data accurately. This process ensures that the collection points can accurately provide data and ensures the smooth realization of the target production task. In this way, the target equipment corresponding to the target production task is verified and configured, and an accurate and reliable data collection point catalog library can be constructed to ensure the smooth completion of the target production task.

[0051] Figure 2 This is a schematic diagram of the data collection point catalog library construction process shown in an exemplary embodiment of the present application. Figure 2 , select the data source type according to the device, sensor, and system, and determine it as the collection point, fill in their corresponding location, type, parameters, interface, data format, and data communication method, etc. Then determine whether each collection point needs a collection gateway, install the gateway for the required collection point, and finally perform a docking test, correct those that fail the test, until all collection points that can work normally are obtained, based on which a data collection point catalog library is generated.

[0052] S102. Determine a collection plan based on the target production task and the data collection point catalog library, wherein the collection plan at least includes a collection sequence of different collection points, a collection template and a data parsing algorithm corresponding to each collection point, and the collection templates and data parsing algorithms for collection points of different data types are different. The collection cycle and collection sequence are determined based on the timeliness information of the target production task.

[0053] Specifically, the collection plan is a plan and strategy for data collection formulated for the target production task. It should be noted that the specific content of the collection plan is determined according to actual needs, and is not limited in this embodiment. In specific implementation, the collection plan may include data collection points, collection timing corresponding to the collection points, data types, collection cycles, analysis algorithms and other elements.

[0054] Furthermore, the collection timing is a time indicator for collecting data at a collection point, and data is collected according to the collection timing.

[0055] Furthermore, the collection template is used to guide the specific operations in the data collection process to ensure the consistency and accuracy of the data. The data parsing algorithm is used to convert the collected raw data into a format that can be processed and analyzed. Different data types require different collection templates and parsing algorithms.

[0056] Furthermore, the collection cycle refers to the frequency or time interval of data collection, which can be determined based on the timeliness of the target production task and the need for data update. The collection sequence refers to the order and time arrangement of data collection. The collection sequence can ensure the orderly progress of the data collection process and avoid data loss or confusion.

[0057] Furthermore, a data parsing algorithm is an algorithm or program required to convert raw data into a format that is easy to process or analyze. It should be noted that due to the diversity of data formats collected at various collection points, different data parsing algorithms are required to adapt to different data types and formats, and the content is converted into plain text information that can be recognized by humans through the components or plug-ins corresponding to the data parsing algorithm. The components or plug-ins are generated by the data format provider or third party based on the interface document development.

[0058] In the specific implementation, in this step, the data type corresponding to the branch data point determines the appropriate collection template, and the collection cycle and timing are determined according to the target production task to complete the construction of the collection plan.

[0059] A specific embodiment is given below to introduce the process of determining the acquisition plan in detail:

[0060] (1) Determine the collection timing based on the target production task and the idle time of each collection point.

[0061] Specifically, the target production task includes a demand time point for data, and the collection timing corresponding to each collection point is determined according to the demand time point and the idle time available for collection at the collection point.

[0062] (2) Obtaining a collection template corresponding to each collection point based on the data collection point catalog library.

[0063] In specific implementation, a collection template is generated based on the interface standards and other data contained in the factory information of the target device, combined with the data requirements of the intelligent manufacturing system.

[0064] (3) Extracting feature vectors corresponding to the data at the acquisition points, and obtaining extraction algorithms and data analysis algorithms corresponding to the feature vectors.

[0065] Specifically, the feature vector of the data in the collection point may be obtained based on a traditional feature extraction method, or may be obtained based on a neural network feature extraction method, which is not limited in this embodiment.

[0066] In specific implementation, for each acquisition point, the corresponding feature vector is obtained. Matching is performed based on the feature vector to determine the extraction algorithm and data parsing algorithm corresponding to the feature vector, and the obtained extraction algorithm and data parsing algorithm are determined as the extraction algorithm and data parsing algorithm of this acquisition point. It can be understood that one acquisition point corresponds to one feature extraction algorithm, and the feature extraction algorithm is used to convert the original data into a representative feature vector for easy analysis and processing.

[0067] (3) Determine the acquisition plan based on the acquisition template, the extraction algorithm and the data analysis algorithm.

[0068] In the specific implementation, a detailed collection plan is formulated by integrating the collection template, extraction algorithm and data analysis algorithm. It should be noted that the collection plan should clearly specify the specific steps such as data collection time, frequency, data format requirements, feature extraction and data analysis for each collection point.

[0069] The processing method for multimodal data collection of the intelligent manufacturing system provided in this embodiment can ensure the standardization and efficiency of data collection through the collection template. By selecting the appropriate extraction algorithm and data analysis algorithm through the feature vector of the collection point, each collection point with different data types can be processed in a targeted manner. In this way, by combining the collection template, the extraction algorithm and the data analysis algorithm, a collection plan is generated corresponding to each collection point, which can ensure the reliability and accuracy of data collection.

[0070] Figure 3 This is a schematic diagram of the collection scheme determination process shown in an exemplary embodiment of the present application. Figure 3 First, select the collection points in the data collection point catalog library. These collection points can include equipment, sensors, and systems. Configure the interface standards provided by the manufacturer for these collection points, and configure the data requirements of the intelligent manufacturing system. Generate a collection template based on this, and then determine whether each collection point is periodic, and configure the collection cycle for periodic collection points. Select the corresponding data analysis algorithm for the collection points that are not periodic and test them. Return the collection points that fail the test to the step of configuring the interface standards provided by the manufacturer until all collection points are tested successfully and the final collection plan is obtained.

[0071] S103: Collect multimodal data from the collection points of the data collection point catalog library based on the collection scheme, and pre-process the multimodal data according to the data analysis algorithm corresponding to each collection point, wherein the data format and data type of each collection point are different, and the data format of the pre-processed multimodal data is the same.

[0072] Specifically, multimodal data refers to a collection of information consisting of data in two or more different forms or sources. Multimodal data has a wide range of application value in intelligent manufacturing systems. It should be noted that the specific data types contained in multimodal data are determined according to actual needs, and this is not limited in this embodiment. In specific implementation, multimodal data can include data types such as text, images, audio, video, and sensor data.

[0073] Furthermore, by preprocessing the multimodal data, the quality and usability of the data can be improved. For example, in one embodiment, the multimodal data is preprocessed by steps such as data cleaning, format conversion, and annotation.

[0074] In specific implementation, a data indicator basis for removing noise can be formed based on the manufacturer's parameter standards corresponding to the collection points, and then the data items required for the target production task can be obtained and bound to the collected multimodal data to determine the valid items of the collected data.

[0075] A specific embodiment is given below to introduce the process of multimodal data collection in detail:

[0076] (1) Collecting each collection point according to the collection timing and collection period, wherein the real-time multimodal data is collected according to the real-time collection scheme, and the non-real-time multimodal data is collected according to the non-real-time collection scheme, and the non-real-time collection scheme at least includes a collection trigger condition.

[0077] Specifically, based on the timeliness information obtained from the target production task, the collection period of each collection point is determined, and the multimodal data is collected according to the collection period. In specific implementation, the multimodal data can be collected once every sampling period.

[0078] It should be noted that the multimodal data collected regularly according to the collection cycle is indexed based on time, and the multimodal data collected in real time is stored in the memory of the corresponding target server.

[0079] Furthermore, the specific data type and specific content of the triggering data are set according to actual needs, and are not limited in this embodiment.

[0080] (2) The real-time multimodal data is stored in the target server memory, and the non-real-time multimodal data is stored in the target hard disk.

[0081] Specifically, the real-time multimodal data in the target server memory can be stored in the time series database according to business needs. When triggering data is detected, the multimodal data of each collection point is collected and stored in the target hard disk.

[0082] It should be noted that when the trigger data overlaps with a certain collection cycle, real-time collection and non-real-time collection need to be performed simultaneously. At this time, the priority in the collection process is determined according to the order of the collection sequence planning. For example, in one embodiment, non-real-time tasks are collected first according to the collection sequence, and then real-time collection tasks are collected. For another example, in another embodiment, real-time tasks are collected first according to the collection sequence, and then non-real-time collection tasks are collected.

[0083] Furthermore, in order to ensure the security of multimodal data and prevent tampering of multimodal data, the corresponding multimodal data is encrypted during the multimodal data transmission process. After the server receives the encrypted data, it performs corresponding decryption processing on it and restores it to multimodal data.

[0084] Furthermore, multimodal data can adopt a distributed storage architecture. In specific implementation, distributed storage systems such as Hado0p and HDFS can be used to solve large-scale data storage problems and improve data access speed and scalability.

[0085] The processing method for multimodal data collection of the intelligent manufacturing system provided in this embodiment performs regular collection through the set collection cycle, ensures the timeliness of the data, and enables the system to obtain the latest production status information in a timely manner. Further, the non-real-time collection method with trigger data as the main index enables the system to respond quickly when a specific event or condition is triggered, further improving the accuracy and pertinence of data collection. Further, the multimodal data collected in real time is stored in the target server memory, and the fast access characteristics of the memory are used to improve the processing speed of real-time data, so that the business system can respond quickly and make decisions. Further, the non-real-time collected data is stored in the target hard disk, and the large capacity and persistence characteristics of the hard disk ensure the long-term preservation and access to historical data at any time, providing a solid foundation for subsequent data analysis and decision support. At the same time, the order and time arrangement of real-time collection and non-real-time collection are reasonably planned through the collection timing, avoiding resource conflicts and data confusion in the data collection process, and improving the overall performance and stability of the system. In this way, by reasonably collecting multimodal data, the system can obtain more comprehensive and rich production information, provide more dimensional input for data analysis, and improve the operation efficiency and reliability of the intelligent manufacturing system.

[0086] A specific embodiment is given below to introduce the preprocessing process of multi-source data in detail:

[0087] (1) Generate a data indicator basis based on the parameter standard carried by the intelligent manufacturing system, and determine valid data based on the data indicator basis; the data indicator basis includes data items required for the target production task.

[0088] Specifically, parameter standards are a series of parameters and rules pre-set in the intelligent manufacturing system to guide data collection and processing. In specific implementation, parameter standards can include data format, sampling frequency, communication protocol, data processing accuracy, etc. Parameter standards are an important basis for determining data collection and processing methods.

[0089] Furthermore, the data related to the target production task in the intelligent manufacturing system is determined by the data indicator basis. In specific implementation, the data included in the data indicator basis can be determined as data collection points useful for the target production task, and the data collected at these collection points can be determined as valid data.

[0090] (2) Processing the valid data of the corresponding collection point based on the data analysis algorithm to generate the multimodal data in a unified format.

[0091] Specifically, the data analysis algorithm converts the collected valid data into data with a unified format and easy to process, so as to ensure that the processed data meets the further processing requirements of the intelligent manufacturing system.

[0092] Furthermore, the data corresponding to a collection point can be processed using a data analysis algorithm to produce a corresponding feature. In this way, different collection points can be processed in a targeted manner, ensuring the accuracy of preprocessing.

[0093] The processing method for multimodal data collection of the intelligent manufacturing system provided in this embodiment generates a data indicator basis through the parameter standards carried by the intelligent manufacturing system, and determines the valid data accordingly, ensuring that the collected data has direct relevance and practical value to the target production task, and improving the validity and relevance of the data. Furthermore, the data parsing algorithm is applied to unify the format of the valid data, eliminating the format differences between data from different sources, and providing a unified data format for subsequent data analysis, thereby improving the efficiency of data processing. In this way, by improving the overall quality of the data, a more comprehensive and accurate information foundation is provided for the intelligent manufacturing system.

[0094] In a possible implementation, multimodal data of each collection point is collected according to the collection sequence, and the collection process of each collection point is specified by the collection template corresponding to the collection point; the valid data content in the collected multimodal data is filled into the data storage template, the data storage templates corresponding to different data types are different, the data storage templates corresponding to the same data type are the same, and the values ​​in the data storage template are different; the collection template is parsed based on a data parsing algorithm, and feature information of the values ​​in the collection template is extracted, and the data parsing algorithm corresponds to the collection template.

[0095] The method provided in this embodiment ensures the standardization and normalization of data collection by formulating a corresponding collection template for each collection point. The collection process of each collection point is strictly carried out in accordance with the template regulations, avoiding the arbitrariness and uncertainty of data collection and improving the accuracy and reliability of the data. By filling the data storage template with valid data content, the method not only realizes the structured management of data, but also facilitates the rapid retrieval and call of data. This orderly data storage method helps to improve the efficiency of data storage and management and reduce data maintenance costs. In this way, by standardizing the collection process, improving data collection efficiency, and flexibly responding to the diversity of data types, the efficiency and accuracy of data analysis can be ensured.

[0096] Figure 4 This is a schematic diagram of a multimodal data preprocessing process shown in an exemplary embodiment of the present application. Figure 4 , complete the reporting of multimodal data according to the standards provided by the manufacturer, then obtain the data required for the target production task and bind it to the collected data. After format conversion, filter out non-compliant information, define the attributes of the filtered data by label assignment, and generate a suitable data preprocessing model after extracting feature representation through the feature value extraction model to complete the preprocessing process of multimodal data.

[0097] S104: input the preprocessed multimodal data into a data fusion model, match the fusion algorithm, and fuse and store the multimodal data.

[0098] Specifically, the data fusion model integrates information from multiple data sources or modalities through appropriate algorithms and technical means to obtain a more comprehensive and accurate data representation.

[0099] Furthermore, the fusion algorithm is a specific algorithm or method used in the data fusion process, which is used to merge data from multiple data sources or modalities into a unified data set. It should be noted that the specific type of fusion algorithm is determined according to actual needs, and is not limited in this embodiment. For example, the fusion algorithm may include a data merging algorithm, a data aggregation algorithm, a data splitting algorithm, etc.

[0100] In specific implementation, a suitable fusion algorithm is selected to fuse the multimodal data according to the characteristics of the multimodal data and the business requirements of the target production task.

[0101] It should be noted that after the multimodal data is preprocessed, the label information in the multimodal data can be assigned. In specific implementation, the label information of the multimodal data can be a kind of metadata, which is attached to the multimodal data through the metadata to provide additional information about the multimodal data to help understand and process the multimodal data. It should be noted that the specific content of the label information is determined according to actual needs, and this is not limited in this embodiment. In specific implementation, the label information may include the multimodal data type, multimodal data source, multimodal data quality, multimodal data owner, multimodal data creation time and multimodal data relationship, where the assignment information mainly refers to the actual data filling for these tags, and the filling source is the automatic filling in the prior binding and data acquisition.

[0102] Furthermore, after the label information amplitude of the multimodal data is calculated, the corresponding feature extraction algorithm is selected and bound according to the data type of the multimodal data and the processing target corresponding to the target birthday task. In specific implementation, the multimodal data corresponding to a collection point corresponds to one or more feature extraction algorithms, and the multimodal data is converted into a representative feature vector based on the feature extraction algorithm for subsequent processing and analysis. It should be noted that the feature extraction process can be completed based on a pre-built feature value extraction model.

[0103] A specific embodiment is given below to illustrate the fusion process of multimodal data in detail.

[0104] (1) Determine multiple target collection points that require data fusion based on the target production task.

[0105] Specifically, according to the specific requirements of the target production task, a plurality of target collection points where data collection is required are first determined.

[0106] In the specific implementation, according to the specific requirements of the target production task, the target collection points are determined, including sensors, surveillance cameras, audio collection equipment, etc. on the production line.

[0107] (2) Determine a fusion algorithm based on the business scenario corresponding to the target production task and the characteristic requirements of the target production task.

[0108] Specifically, the business scenario refers to the actual production environment and workflow in which the target production task is located. Different business scenarios may have different requirements for data collection and processing, so it is necessary to determine the appropriate data collection and processing strategy based on the specific scenario.

[0109] Furthermore, according to the needs of the target production task, the characteristics of the fused data are determined and identified as characteristic requirements.

[0110] In the specific implementation, factors such as the data modality type, data volume, real-time requirements, and subsequent analysis requirements are considered in combination with feature requirements and business scenarios, and related fusion algorithms are bound. The data can be unified into a data set through the corresponding fusion algorithm.

[0111] (3) Based on the fusion algorithm, the multimodal data of the multiple target acquisition points are fused to generate the target data set.

[0112] In specific implementation, the data of each collection point is read based on the fusion algorithm, the fusion algorithm is applied to integrate the data, the fused data is saved, and the target data set is obtained.

[0113] Furthermore, through the fusion of multimodal data, data belonging to the same time can be divided together. At the same time, through the fusion of multimodal data, data representing the same quality information can be fused to facilitate subsequent analysis.

[0114] It should be noted that data from multiple different collection points can be fused.

[0115] Furthermore, the fused data can represent information of different dimensions of the intelligent manufacturing system. In specific implementation, the process information, process information, equipment information, etc. corresponding to the target production task can be obtained through the fused data.

[0116] The method provided by the present invention matches the corresponding fusion algorithm according to the target production task, that is, the fusion algorithm corresponds to the target production task, and is determined according to the task requirement matching rather than being processed from the data perspective, to determine the associated multimodal data related to the target production task, and to fuse the associated multimodal data using the fusion algorithm, wherein a target production task may have multiple data usage requirements, and each requirement requires data fusion from multiple points to obtain information, that is, there may be multiple requirements, and each requirement corresponds to a different fusion algorithm because the target information and information source to be obtained are different. The associated data is selected according to the requirement, and then the associated multimodal data is fused according to the fusion algorithm according to the requirements of serving the target production task.

[0117] The processing method for multimodal data collection of the intelligent manufacturing system provided in this embodiment can obtain a more comprehensive and accurate data representation by fusing multimodal data from the collection points. Furthermore, the fusion algorithm that best suits the current target production task can be selected based on the business scenario and characteristic requirements, thereby ensuring the reliability and accuracy of data fusion.

[0118] The following is a specific example to explain the multimodal data storage process in detail:

[0119] (1) Determining the target collection points as upper branches of a tree structure according to the association relationship between the multiple target collection points.

[0120] Specifically, in a tree structure, an upper branch usually refers to a portion closer to a root node. In a specific implementation, the target collection point is determined as an upper branch, in other words, the target collection point is used as a top level or high-level classification of data organization.

[0121] Furthermore, the target production task corresponds to several target collection points, and the tree structure may also have several corresponding upper branches.

[0122] (2) The multimodal data is sequentially filled into the upper branches according to the attribute information to obtain corresponding lower branches; the tree structure is provided with cataloging information based on the attributes.

[0123] Specifically, the lower branches are the parts of the tree structure located below the upper branches, which represent further subdivision or specific content of the data. In this context, the lower branches will be allocated to each upper branch according to the source of the multimodal data.

[0124] Furthermore, cataloging information is the process of classifying, identifying and indexing multimodal data. In the tree structure, cataloging information is used to describe the specific content and structure of each node (including upper and lower branches) to facilitate the retrieval and management of multimodal data.

[0125] It should be noted that multimodal data can be located and retrieved through cataloging information.

[0126] In specific implementation, a tree structure and cataloging information are constructed for all target collection points and corresponding multimodal data until all multimodal data are processed.

[0127] The processing method for multimodal data collection of the intelligent manufacturing system provided in this embodiment provides an intuitive and hierarchical data organization method through a tree structure, so that multimodal data can be arranged in order according to specific logic and rules, which helps to reduce the confusion and redundancy of data and improve the manageability and accessibility of data. Further, by identifying and indexing each node of the tree structure through catalog information, the efficiency of data retrieval can be greatly improved, and users can quickly locate specific collection points or data sets as needed without traversing the entire target data set. Further, the tree structure makes data management more flexible and convenient, and the levels and nodes of the tree structure can be adjusted according to actual needs, collection points and target data sets can be added or deleted, and catalog information can be updated to adapt to the ever-changing production environment and business needs. At the same time, forming a tree structure of multimodal data helps to better utilize the inherent connections and laws of data in the data fusion and analysis stage. By analyzing the relationship and difference between different nodes in the tree structure, the potential associations and trends between the data can be found, providing more comprehensive and in-depth information for production optimization and decision support.

[0128] A specific embodiment is given below to introduce the display process of multimodal data in detail:

[0129] (1) Determine data query requirements based on the real-time service module.

[0130] Specifically, the communication protocol is the rules and agreements that both parties must follow during the data transmission process. It should be noted that in the intelligent manufacturing system, the communication protocol is used to ensure smooth data exchange between different devices and systems. These protocols define the data format, transmission method, error detection and correction mechanism, etc.

[0131] Furthermore, the real-time service module is the part of the intelligent manufacturing system that is responsible for real-time data processing and display. The real-time service module receives real-time data from various collection points according to the communication protocol of the intelligent manufacturing system, processes and analyzes the data, and finally displays it to the user in real time through a graphical interface or other means.

[0132] In the specific implementation, the configuration and parameters of the real-time service module are determined according to the communication protocol of the intelligent manufacturing system. Then, the real-time service module establishes a connection with each data collection point through the network and receives the multimodal data sent by them. Then, these real-time data are processed and analyzed, and the processed multimodal data are displayed to the user through the set display method.

[0133] In specific implementation, when data query is needed, relevant data query requirements are generated.

[0134] (2) Determine a data retrieval scope and a data retrieval condition based on the data query requirement and the tree structure, and locate and display the multimodal data based on the data retrieval scope and the data retrieval condition.

[0135] Specifically, the data search range refers to the boundary of the multimodal data set to be searched, which is specified by the user when performing multimodal data search. In specific implementation, in a tree structure, the data search range may correspond to one or more branches in the tree structure.

[0136] Furthermore, the data retrieval condition refers to a specific rule or condition set by a user for filtering multimodal data when performing multimodal data retrieval. In specific implementation, the data retrieval condition can be defined based on the attributes, values, or relationships between the multimodal data.

[0137] In specific implementation, the scope of data retrieval is determined according to the constructed tree structure. This scope can be the entire tree, a subtree, or multiple specific nodes in the tree structure. Then, according to the data retrieval conditions set by the user, the tree structure is searched and filtered, the branches that meet the conditions are located, and the obtained multimodal data is displayed to the user in a set display method. This process allows users to quickly find and view the data parts they are interested in, improving the efficiency and convenience of data access.

[0138] It should be noted that the data displayed in non-real time is multimodal data collected in non-real time and stored in the target hard disk.

[0139] The processing method for multimodal data acquisition of the intelligent manufacturing system provided in this embodiment determines the real-time service module based on the communication protocol of the intelligent manufacturing system, and displays the multimodal data in real time, so that the enterprise can instantly obtain the key data in the production process. This real-time data visualization not only improves the transparency of the production process, but also enables managers to respond quickly to changes in production, adjust production strategies in time, and ensure the smooth operation of the production process. Furthermore, the data retrieval range and data retrieval conditions are determined based on the tree structure, making the data retrieval process more intuitive and efficient. Users can quickly locate specific data sets as needed, avoiding the trouble of blindly searching in massive data. In this way, non-real-time saved data can be quickly queried, improving the efficiency and accuracy of data utilization.

[0140] Figure 5 This is a schematic diagram of a multimodal data fusion process shown in an exemplary embodiment of the present application. Figure 5 ,Firstly, the collection points are selected as the fusion objects, and the data of the collection points are obtained. The fusion objects are catalogued using a tree structure. Various methods are used to bind the corresponding fusion algorithms to the fusion objects. Finally, a data fusion model is generated according to the selected fusion algorithm to fuse the data of the collection points.

[0141] The processing method for multimodal data acquisition of the intelligent manufacturing system provided in this embodiment can process diversified data from different devices, sensors and systems, support multiple data formats and communication protocols, and significantly improve the flexibility and coverage of data acquisition. By constructing a data acquisition source catalog library, the clear definition and management of the acquisition scope are realized, making the data acquisition process more orderly and efficient. The data acquisition gateway is installed according to the acquisition source attributes, which realizes the rapid connection and access system of physical devices and sensors, and improves the convenience of data acquisition. Further, the collected data is cleaned, formatted and annotated, which effectively improves the quality and availability of the data and provides a solid foundation for subsequent data analysis and decision-making. Further, the tree structure is used to catalog the fused data, which is convenient for quickly extracting the required information and improving the efficiency and accuracy of data retrieval. Distributed storage systems such as Hadoop and HDFS are used to solve the problem of large-scale data storage and improve the speed and scalability of data access. In addition, the data is encrypted during the data transmission process to ensure data security and prevent data tampering. At the same time, the data is decrypted after the server receives the data, which ensures the integrity and security of the data. Furthermore, the tree structure can be used to conveniently retrieve and use multimodal data that has been collected, processed and integrated, supporting various application scenarios such as production monitoring and optimized decision-making, thereby improving the scientificity and accuracy of decision-making.

[0142] Corresponding to the aforementioned embodiment of a processing method for multimodal data acquisition of an intelligent manufacturing system, the present application also provides an embodiment of a processing device for multimodal data acquisition of an intelligent manufacturing system.

[0143] The embodiment of the processing device for multimodal data collection of an intelligent manufacturing system in the present application can be applied to the processing equipment for multimodal data collection of an intelligent manufacturing system. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the processing equipment for multimodal data collection of the intelligent manufacturing system in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 6 As shown, it is a hardware structure diagram of the processing device of multimodal data collection of the intelligent manufacturing system where the processing device of multimodal data collection of the intelligent manufacturing system of the present application is located, except Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, the processing device for multimodal data acquisition of the intelligent manufacturing system in which the device in the embodiment is located may also include other hardware according to the actual function of the processing device for multimodal data acquisition of the intelligent manufacturing system, which will not be described in detail.

[0144] Figure 7 This is a structural diagram of the first embodiment of the processing device for multimodal data acquisition of the intelligent manufacturing system provided by this application. Figure 7 The device provided in this embodiment includes a construction module 710, a determination module 720, a collection module 730 and a fusion module 740; wherein,

[0145] The construction module 710 is used to construct a data collection point catalog library based on the target production task of the intelligent manufacturing system; the data collection point catalog library is a collection point associated with the target production task determined from various monitoring points within the control range of the intelligent manufacturing system, including multiple collection point configuration information, each of which is a data type, location information and communication information of the collection point;

[0146] The determination module 720 is used to determine a collection plan based on the target production task and the data collection point catalog library, wherein a collection template and a data parsing algorithm are determined based on the data type, and the collection template and data parsing algorithm for each data type are different, and a collection cycle and a collection sequence are determined based on the timeliness information of the target production task;

[0147] The acquisition module 730 is used to acquire multimodal data from the data acquisition point catalog library based on the acquisition scheme, and pre-process the multimodal data according to the data analysis algorithm corresponding to each data, the data format and data type of the multimodal data are different, and the data format of the pre-processed multimodal data is the same;

[0148] The fusion module 740 is used to input the pre-processed multimodal data into a data fusion model, match the fusion algorithm, and fuse and store the multimodal data.

[0149] The device of this embodiment can be used to perform Figure 1 The steps, specific implementation principles and implementation processes of the method embodiment shown are similar and will not be repeated here.

[0150] Optionally, the fusion module 740 is specifically used to determine a plurality of target collection points that need to be data fused based on the target production task;

[0151] The fusion module 740 is further specifically used to determine a fusion algorithm in combination with the business scenario corresponding to the target production task and the characteristic requirements of the target production task;

[0152] The fusion module 740 is further specifically configured to fuse the multimodal data from the same target acquisition point based on the fusion algorithm to generate the target data set.

[0153] Optionally, the fusion module 740 is further specifically configured to determine the target collection point as an upper branch of the tree structure;

[0154] The fusion module 740 is further specifically used to fill the upper branches with the multimodal data in sequence according to the attribute information to obtain the corresponding lower branches; the tree structure is provided with cataloging information based on the attributes.

[0155] Optionally, the construction module 710 is specifically configured to determine a target device included in the intelligent manufacturing system based on the registration data of the intelligent manufacturing system;

[0156] The construction module 710 is further specifically used to filter the target device associated with the target production task to obtain the collection point configuration information;

[0157] The construction module 710 is further specifically used to save the configuration information of the collection points that have passed the test to obtain the data collection point catalog library.

[0158] Optionally, the determination module 720 is specifically configured to obtain a collection template corresponding to each collection point based on the data collection point catalog library;

[0159] The determination module 720 is further specifically used to extract the feature vector corresponding to the data in the collection point, and obtain the extraction algorithm corresponding to the feature vector and the data analysis algorithm;

[0160] The determination module 720 is further specifically configured to determine the acquisition plan based on the acquisition template, the extraction algorithm and the data parsing algorithm.

[0161] Optionally, the fusion module 740 is further specifically configured to determine a real-time service module based on the communication protocol of the intelligent manufacturing system, and to display the multimodal data in real time based on the real-time service module;

[0162] The fusion module 740 is further specifically configured to determine a data retrieval scope and a data retrieval condition based on the tree structure, and locate and display the multimodal data based on the data retrieval scope and the data retrieval condition.

[0163] Optionally, the acquisition module 730 is specifically used to regularly acquire the multimodal data according to the acquisition period, and store the multimodal data in the target server memory;

[0164] The acquisition module 730 is also specifically used to acquire the multimodal data in non-real time with trigger data as the main index, and store the multimodal data in a target hard disk; wherein the sequence of the real-time acquisition and the non-real-time acquisition is planned based on the acquisition timing.

[0165] Optionally, the acquisition module 730 is further specifically used to generate a data indicator basis based on the parameter standard carried by the intelligent manufacturing system, and determine valid data based on the data indicator basis; the data indicator basis includes data items required for the target production task;

[0166] The acquisition module 730 is further specifically configured to process the valid data of the corresponding acquisition point based on the data analysis algorithm to generate the multimodal data in a unified format.

[0167] Please continue to refer to Figure 6 The present application also provides a processing device for multimodal data acquisition of an intelligent manufacturing system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the methods provided in the first aspect of the present application are implemented.

[0168] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of any one of the methods provided in the present application are implemented.

[0169] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0170] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0171] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for processing multimodal data acquisition in an intelligent manufacturing system, characterized in that: The method comprises: Building a data collection point catalog library based on the target production task of the intelligent manufacturing system; the data collection point catalog library is a collection point associated with the target production task determined from various monitoring points within the control range of the intelligent manufacturing system, including multiple collection point configuration information, each of which is a data type, location information and communication information of the collection point; Determine a collection plan based on the target production task and the data collection point catalog library, wherein the collection plan at least includes the collection timing of different collection points, the collection template and data parsing algorithm corresponding to each collection point, the collection template and data parsing algorithm of collection points of different data types are different, and the collection cycle and collection timing are determined based on the timeliness information of the target production task; Based on the collection scheme, multimodal data is collected from the collection points of the data collection point catalog library, and the multimodal data is preprocessed according to the data analysis algorithm corresponding to each collection point, the data format and data type of each collection point are different, and the data format of the preprocessed multimodal data is the same; The pre-processed multimodal data is input into a data fusion model, a fusion algorithm is matched, and the multimodal data is fused and stored.

2. The method according to claim 1, characterized in that The step of inputting the pre-processed multimodal data into a data fusion model, matching a fusion algorithm, fusing and storing the multimodal data includes: Determine multiple target collection points that require data fusion based on the target production task; Determine a fusion algorithm based on the business scenario corresponding to the target production task and the characteristic requirements of the target production task; The multimodal data of the multiple target acquisition points are fused based on the fusion algorithm to generate the target data set.

3. The method according to claim 2, characterized in that The step of fusing the multimodal data of the plurality of target acquisition points based on the fusion algorithm to generate the target data set includes: Determining the target collection points as upper branches of the tree structure according to the association relationship of the multiple target collection points; The multimodal data is sequentially filled into the upper branches according to the attribute information to obtain the corresponding lower branches; the tree structure is provided with cataloging information based on the attributes.

4. The method according to claim 1, characterized in that The data collection point catalog library is constructed based on the target production task of the intelligent manufacturing system, including: Determine the target device included in the intelligent manufacturing system based on the registration data of the intelligent manufacturing system; Filtering the target device associated with the target production task to obtain the collection point configuration information; The configuration information of the collection points that have passed the test is saved to obtain the data collection point catalog library.

5. The method according to claim 1, characterized in that The determining of the collection scheme based on the target production task and the data collection point catalog library includes: Determine the collection timing based on the target production task and the idle time of each collection point; Acquire a collection template corresponding to each collection point based on the data collection point catalog library; Extracting a feature vector corresponding to the data in the acquisition point, and obtaining an extraction algorithm corresponding to the feature vector and the data analysis algorithm; The acquisition plan is determined based on the acquisition template, the extraction algorithm and the data parsing algorithm.

6. The method according to claim 3, characterized in that After fusing and storing the multimodal data, the method further includes: Determine data query requirements based on real-time service modules; A data retrieval scope and a data retrieval condition are determined based on the data query requirement and the tree structure, and the multimodal data is located and displayed based on the data retrieval scope and the data retrieval condition.

7. The method according to claim 1, characterized in that The acquisition scheme includes at least a real-time acquisition scheme and a non-real-time acquisition scheme, and the acquisition of multimodal data from the data acquisition point catalog library based on the acquisition scheme includes: Collecting each collection point according to the collection timing and collection period, wherein the real-time multimodal data is collected according to the real-time collection scheme, and the non-real-time multimodal data is collected according to the non-real-time collection scheme, and the non-real-time collection scheme at least includes a collection trigger condition; The real-time multimodal data is stored in the target server memory, and the non-real-time multimodal data is stored in the target hard disk.

8. The method according to claim 1, characterized in that The collecting multimodal data from the data collection point catalog library based on the collection scheme includes: Generate a data indicator basis based on the parameter standard carried by the intelligent manufacturing system, and determine valid data based on the data indicator basis; the data indicator basis includes data items required for the target production task; The valid data of the corresponding collection point is processed based on the data analysis algorithm to generate the multimodal data in a unified format.

9. The method according to claim 1, characterized in that: The method of collecting multimodal data from the collection points of the data collection point catalog library based on the collection scheme, and preprocessing the multimodal data according to the data analysis algorithm corresponding to each collection point, includes: Collecting multimodal data of each collection point according to the collection time sequence, wherein the collection process of each collection point is specified by the collection template corresponding to the collection point; Fill the valid data content in the collected multimodal data into the data storage template, where different data types correspond to different data storage templates, and the same data storage templates correspond to the same data type, but the values ​​in the data storage templates are different; The acquisition template is analyzed based on a data analysis algorithm to extract characteristic information of numerical values ​​in the acquisition template, and the data analysis algorithm corresponds to the acquisition template.

10. A processing device for multimodal data acquisition, characterized in that: The device includes a construction module, a determination module, a collection module and a fusion module; wherein, The construction module is used to construct a data collection point catalog library based on the target production task of the intelligent manufacturing system; the data collection point catalog library is a collection point associated with the target production task determined from various monitoring points within the control range of the intelligent manufacturing system, including a plurality of collection point configuration information, each of which is a data type, location information and communication information of the collection point; The determination module is used to determine a collection plan based on the target production task and the data collection point catalog library, wherein the collection plan at least includes the collection timing of different collection points, the collection template and data parsing algorithm corresponding to each collection point, the collection template and data parsing algorithm of collection points of different data types are different, and the collection cycle and collection timing are determined based on the timeliness information of the target production task; The acquisition module is used to acquire multimodal data from the acquisition points of the data acquisition point catalog library based on the acquisition scheme, and preprocess the multimodal data according to the data analysis algorithm corresponding to each acquisition point, the data format and data type of each acquisition point are different, and the data format of the preprocessed multimodal data is the same; The fusion module is used to input the pre-processed multimodal data into a data fusion model, match the fusion algorithm, and fuse and store the multimodal data.

Images