Data acquisition method and system, electronic equipment and storage medium

Abstract

The invention provides a data collection method and system, electronic equipment and a storage medium, and the data collection method comprises the steps: carrying out the real-time monitoring of the content of a log file, and obtaining a monitoring result; determining whether a preset data acquisition condition is met or not according to the monitoring result; if the preset data acquisition condition is met, extracting corresponding log data from the log file, and performing format conversion on the log data to obtain target data meeting a format requirement; and calling an interface of a manufacturing execution system MES, and sending the target data to the MES through the interface. According to the technical scheme, the unified processing method is provided for the problem that factory data sources are diversified, the problems that data formats and protocols are not unified and integration difficulty is large are solved, and the real-time performance of data is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data collection and processing, and in particular to a data collection method and system, an electronic device and a storage medium. BACKGROUND

[0002] MES (Manufacturing Execution System) is a "production command center" connecting upper enterprise resource planning and lower production equipment. The core functions of MES include production scheduling, data collection, quality traceability, equipment management and report analysis, and has become the core of factory production management. The effective operation of MES depends on the timely and accurate input of data from underlying devices, sensors and business systems.

[0003] However, the existing MES-oriented data collection has the following problems:

[0004] There are many sources of factory data, including relational databases (such as Oracle, MySQL, SQL Server, etc.), time series databases, NoSQL databases and various APIs (Application Programming Interface), and the data formats and protocols are not unified, which is difficult to integrate. Poor real-time performance, traditional data processing tools are mostly in batch processing mode, and the data delay is from several minutes to several hours, which cannot meet the real-time requirements of MES for production site data (such as device status, material consumption, working hours). Insufficient reliability, network jitter, system failure and other abnormal conditions will cause data transmission to be interrupted, and due to the lack of effective breakpoint resume mechanism, it is easy to cause data loss or duplication. Complex configuration and low flexibility, the existing solutions often require a lot of coding work, and the response to business changes is slow, when the data table structure or MES interface changes, the code needs to be modified and redeployed, with high maintenance cost. Lack of state monitoring, data transmission is like a "black box", and it is difficult for operation and maintenance personnel to real-time understand the data flow state and locate the fault point.

[0005] Therefore, the present application is proposed. SUMMARY

[0006] The present application provides a data collection method, system, electronic device and storage medium, which proposes a unified processing method for the problem of multiple sources of factory data, solves the problem of non-uniform data format and protocol, and ensures the real-time performance of data.

[0007] In a first aspect, the embodiments of the present application provide a data collection method, comprising: monitoring the content of a log file in real time to obtain a monitoring result;

[0008] determine whether the preset data collection condition is met according to the monitoring result;

[0009] If the preset data collection condition is met, corresponding log data is extracted from the log file, and the log data is format-converted to obtain target data meeting the format requirement;

[0010] An interface of a manufacturing execution system (MES) is called, and the target data is sent to the MES through the interface.

[0011] According to the technical scheme provided in the embodiments of the present application, optionally, the log data is format-converted to obtain target data meeting the format requirement, which comprises:

[0012] According to a data mapping rule, a preset field in the log data is converted into a specified field, and data is assembled according to a set template to obtain target data meeting the format requirement.

[0013] According to the technical scheme provided in the embodiments of the present application, optionally, the technical scheme further comprises:

[0014] If a notification message of successful sending returned by the interface is received, the matched monitoring index is updated.

[0015] According to the technical scheme provided in the embodiments of the present application, optionally, the technical scheme further comprises:

[0016] If a notification message of failed sending returned by the interface is received, or the interface of the manufacturing execution system (MES) fails to be called, the target data is re-sent according to a set strategy.

[0017] According to the technical scheme provided in the embodiments of the present application, optionally, before the interface of the manufacturing execution system (MES) is called and the target data is sent to the MES through the interface, the technical scheme further comprises:

[0018] The target data meeting the format requirement is locally saved or cached.

[0019] According to the technical scheme provided in the embodiments of the present application, optionally, the technical scheme further comprises:

[0020] The target data of failed sending is stored, and an identification mark is set.

[0021] In a second aspect, the embodiments of the present application further provide a data collection system, comprising: a configuration management module configured to configure data source connection information, and receive and store data mapping rules and preset data collection conditions.

[0022] The data collection and monitoring module is in communication connection with the configuration management module, is used for establishing connection with a corresponding database according to data source connection information configured in the configuration management module, and is used for monitoring the content of a log file in real time, obtaining a monitoring result, and determining whether the preset data collection condition is met according to the monitoring result.

[0023] The data conversion and processing module is used for converting the monitoring result according to the data mapping rule if it is determined that the preset data collection condition is met, and processing the converted data into target data meeting format requirements.

[0024] The data transmission module is used for calling an interface of a manufacturing execution system (MES), and sending the target data to the MES through the interface.

[0025] In a third aspect, an electronic device is also provided in the embodiments of the present application, and the electronic device comprises:

[0026] a processor and a memory;

[0027] The processor is used for executing the steps of the data collection method according to any embodiment by calling programs or instructions stored in the memory.

[0028] In a fourth aspect, a computer readable storage medium is also provided in the embodiments of the present application, and the computer readable storage medium stores programs or instructions, which make a computer execute the steps of the data collection method according to any embodiment.

[0029] In summary, the present application provides a data collection method, which realizes millisecond-level data change capture by using a database transaction log monitoring technology (which is embodied in monitoring the content of a log file in real time, obtaining a monitoring result, determining whether a preset data collection condition is met according to the monitoring result, and extracting corresponding log data from the log file if the preset data collection condition is met), and gets rid of the delay problem caused by batch processing. The log-based data collection method provided in the present application does not need to establish a trigger on a source database or modify a business table structure, and has little influence on an existing system. In view of the problem of diverse data sources in a factory, the present application provides a unified processing method, solves the problems of non-uniform data format and protocol and great integration difficulty, and guarantees the real-time performance of data. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a flowchart of a data collection method provided in the embodiments of the present application;

[0031] Figure 2 is a flowchart of another data collection method provided in the embodiments of the present application;

[0032] Figure 3 is a structural schematic diagram of a data acquisition system provided by an embodiment of the present application;

[0033] Figure 4 is an architectural schematic diagram of a data acquisition system provided by an embodiment of the present application;

[0034] Figure 5 is a timing flow of a data acquisition method provided by an embodiment of the present application;

[0035] Figure 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0036] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description.

[0037] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0038] Figure 1 is a flowchart of a data acquisition method provided by an embodiment of the present application. The data acquisition method is a method for real-time acquisition, conversion and transmission of data for a manufacturing execution system (MES), and is suitable for sampling data of downstream equipment monitored by the MES. Referring to Figure 1 , the data acquisition method specifically includes the following steps:

[0039] S110, real-time monitoring the content of the log file to obtain a monitoring result.

[0040] Specifically, the log source (for example, a database binlog) is connected in real time to obtain the incremental content of the log, and the obtained log content is converted into readable structured data, which is the monitoring result.

[0041] Among them, the log file includes a change log, for example, MySQL binlog and PostgreSQL WAL are log files specially designed for recording data changes, with clear structure and high identification efficiency. For such log files, key information can be extracted through a parsing tool to obtain the "table name", "primary key" and data before and after the change.

[0042] The log file also includes logs for general business logs, such as application logs, sensor logs, etc., which do not have preset change event types (such as addition, deletion, update, etc. change event types), and for such logs, key fields can be extracted therefrom by using regular expressions or log parsing libraries to obtain monitoring results.

[0043] It can be understood that before real-time monitoring of the content of the log file, the following steps are further included:

[0044] The data source connection information is configured, such as database address, port, or MES station information. For example, in the configuration management interface, the SQLserver database of the safety test station (i.e. a specific MES station) is selected as the data source, and the connection parameters are configured, the database table to be listened to is selected, such as FunctionTestSoftware1.

[0045] The data mapping rules are configured. Specifically, the user can define the mapping, conversion, and cleaning rules (such as data type conversion, null value processing, code mapping) between the source data fields and the target MES API fields by dragging or scripting. For example, the data mapping rule is: the "KL30_V" field of the source table is mapped to the "insulation resistance" field of the MES API; the "DCW" field is mapped to the "voltage resistance" field; and the trigger rule is defined, when the "Collect" field becomes '0', the data transmission is triggered.

[0046] The REST API endpoint address and authentication information of the MES system are configured to ensure that the REST API service of the MES system is connected legally and correctly.

[0047] Finally, a communication connection with the source database is established according to the configured information.

[0048] S120, according to the monitoring result, it is determined whether the preset data acquisition condition is met.

[0049] S130, if the preset data acquisition condition is met, the corresponding log data is extracted from the log file, and the log data is format-converted to obtain target data meeting the format requirements.

[0050] For example, the preset data acquisition condition is, for example, that the "Collect" field becomes '0', that is, when the "Collect" field becomes '0', it is determined that the preset data acquisition condition is met.

[0051] The log data is format-converted to obtain target data meeting the format requirements, including: converting preset fields in the log data into specified fields according to a data mapping rule, and assembling data according to a set template to obtain target data meeting the format requirements.

[0052] For example, the "KL30_V" field in the log data is mapped to the specified field "insulation resistance" of the MES API; and the "DCW" field in the log data is mapped to the specified field "voltage resistance" of the MES API.

[0053] S140, calling an interface of a manufacturing execution system (MES), and sending the target data to the MES through the interface.

[0054] By means of the database transaction log monitoring technology (which is embodied in real-time monitoring of the content of a log file to obtain a monitoring result, determining whether a preset data collection condition is met according to the monitoring result, and extracting corresponding log data from the log file if the preset data collection condition is met), millisecond-level data change capture is achieved, and the delay problem caused by batch processing is eliminated. The log-based data collection method proposed in the present application does not need to establish a trigger on a source database or modify a business table structure, and has little impact on an existing system. In view of the problem of diverse data sources in a factory, the present application proposes a unified processing method, solves the problems of non-uniform data format and protocol and great integration difficulty, and ensures the real-time performance of data.

[0055] In some embodiments, after the interface of the manufacturing execution system (MES) is called and the target data is sent to the MES through the interface, the following steps are further included:

[0056] If a notification message returned by the interface indicating successful sending is received, the matched monitoring indicators (such as the number of successful transmissions) are updated.

[0057] If a notification message returned by the interface indicating failed sending is received, or the interface of the manufacturing execution system (MES) fails, the target data is re-sent according to a set strategy. For example, the target data is re-sent again after 1s, 3s or 10s, until the sending is successful, so as to ensure the stability and reliability of data sending.

[0058] In some embodiments, in order to ensure the consistency of data in network failure scenarios or system failure scenarios and effectively prevent data loss, the following step is added: before the interface of the manufacturing execution system (MES) is called and the target data is sent to the MES through the interface, the target data meeting the format requirements is locally saved or cached. The purpose is to prevent data loss caused by sudden system failure (such as network disconnection or power failure).

[0059] The target data of sending failure is stored, and an identification mark is set, so as to retransmit the data according to a set strategy.

[0060] In some embodiments, referring to a flowchart of a data collection method as shown in Figure 2 The method specifically includes the following steps:

[0061] S210, loading configuration.

[0062] The configuration includes but is not limited to database address, API of MES, mapping rule.

[0063] S220, listening to database transaction log.

[0064] S230, determining whether the data is changed, if yes, continuing to execute step S240, if not, continuing to execute step S220.

[0065] S240, extracting changed data, including data before change and data after change.

[0066] S250, data conversion.

[0067] S260, temporarily storing the data to local cache, realizing breakpoint resume.

[0068] S270, calling API of MES to transmit data.

[0069] S280, determining whether the data is transmitted successfully, if yes, continuing to execute step S290, if not, executing step 291.

[0070] S290, recording successful state and cleaning cache.

[0071] S291, storing the data into retry queue and retransmitting according to strategy.

[0072] S292, determining whether the maximum retry number is reached, if yes, executing step S293, if not, returning to execute S270 and continuing to retransmit data.

[0073] S293, marking the data of this transmission as failed data and sending alarm notification, and meanwhile, transferring the data of this transmission into dead letter queue for manual processing.

[0074] Based on the same inventive concept, the present application also provides a data collection system corresponding to the method of any of the above embodiments, referring to Figure 3 The data collection system includes:

[0075] The configuration management module 310 is configured to configure data source connection information, receive and store data mapping rules and preset data collection conditions. Specifically, a graphical interface is provided to configure data source connection information (e.g., database IP, port, authentication method, MES station, etc.). A data mapping rule configuration function is provided, and a user can define mapping, conversion and cleaning rules (e.g., data type conversion, null value processing, code mapping rule) between source data fields and target MES API fields through drag and drop or script. Trigger strategy configuration is provided to support incremental capture based on database transaction logs (e.g., MySQL binlog, Oracle redolog) to achieve real-time data collection, rather than polling.

[0076] The data collection and monitoring module 320 is in communication connection with the configuration management module 310, configured to establish a connection with a corresponding database according to the data source connection information configured in the configuration management module 310, and to monitor the content of the log file in real time to obtain a monitoring result, and to determine whether the preset data collection condition is met according to the monitoring result. Specifically, the data collection and monitoring module 320 is connected with the configuration management module 310, and establishes a connection with a source database according to the information configured by the configuration management module 310. The core is that data change events (add, delete, modify) are captured in real time by analyzing the transaction logs of the database. The data collection and monitoring module 320 is disguised as a slave database of the database, and continuously monitors log streams such as binlog. Once there is a related data change, a standardized data change message is immediately captured and generated.

[0077] The data conversion and processing module 330 is configured to convert the monitoring result according to the data mapping rules if it is determined that the preset data collection condition is met, and process the converted data into target data that meets the format requirements. Specifically, the data change message from the data collection and monitoring module 320 is received. The data mapping and conversion rules predefined in the configuration management module 310 are loaded and executed. For example, the test result is converted from the word "OK" to "1"; multiple source table fields such as test date and test time are combined into test time, etc. The data verification function is provided to mark or transfer the data that does not meet the rules to the dead letter queue to ensure the data quality input to the MES.

[0078] The data transmission module 340 is configured to call an interface of a manufacturing execution system (MES), and transmit the target data to the MES through the interface. Specifically, the converted data is packaged into a data format recognizable by the MES system, a JSON message is generated, and the MES interface is called through a REST API protocol. A retry and confirmation mechanism is implemented: if the MES interface fails to be called, the system will automatically retry according to an exponential backoff strategy. Only after a successful response from the MES is received, it is confirmed that the data has been processed. The system has local cache and breakpoint resume capabilities: before data transmission, a copy is cached in the local disk or memory. When the system restarts or the network recovers, the transmission can continue from the last interrupted position, ensuring that no data is lost.

[0079] In some embodiments, the data acquisition system further comprises a monitoring and alarm module configured to display one or more of a data acquisition rate, a transmission delay, and a transmission success rate in real time, and send an alarm notification for data with a transmission delay and / or a transmission failure. A visual interface can also be provided to display the overall topology and status of the data flow.

[0080] For the convenience of description, the above system is described in various modules according to functions. Of course, the functions of the modules can be implemented in one or more software and / or hardware when implementing the present application.

[0081] The system of the above embodiments is used to implement the corresponding data acquisition method in any of the preceding embodiments, and has the beneficial effects of the corresponding method embodiments, which are not described here again.

[0082] On the basis of the above data acquisition system, a method for real-time acquisition, conversion and transmission of data for an MES mainly comprises the following steps: S1, configuring a data source, a data mapping rule and a triggering strategy (preferably based on transaction log monitoring) through a configuration management module. S2: the data acquisition and monitoring module is started, connected to the source database, and begins to monitor the transaction log thereof. S3: when a change event of a target data table appears in the transaction log, the event is immediately captured, and the changed data (including the data mapping before and after the change) is extracted. S4: the data conversion and processing module cleans, converts and packages the captured raw data according to the predefined rules. S5: the data transmission module attempts to send the processed data to the MES system. S6: if the sending is successful, the success status is recorded, and the next event is waited for; if the sending fails, the retry process is entered, and the data is temporarily stored in the local cache. S7: the monitoring and alarm module collects indicators in the entire process, and provides a monitoring view and an abnormal alarm.

[0083] Compared with the prior art, the data acquisition method provided by the present application has the following significant advantages:

[0084] High real-time, through the database transaction log monitoring technology, the realization of millisecond level data change capture, get rid of the delay problem caused by batch processing. High reliability, through the retry mechanism, local cache and breakpoint resume design, ensure the final consistency of data in the network or system failure scenario, effectively prevent data loss. High flexibility and low code, through the graphical configuration interface and configurable mapping rules, greatly reduce the development and use threshold, business personnel after training can complete the configuration of most data integration tasks, respond to changes quickly. Strong observability, built-in all-round monitoring and alarm system, make the data transmission process transparent, greatly improve the maintainability and operation efficiency of the system. Low invasion to the source end, based on the log collection method, no need to establish trigger on the source database or modify the business table structure, minimal impact on the existing system.

[0085] Further, referring to a kind of architecture schematic diagram of data acquisition system as shown in Figure 4 It includes configuration management module 410, data acquisition and monitoring module 420, data conversion and processing module 430, data transmission module 440, monitoring and alarm module 450, source database 460 and target system MES 470.

[0086] Based on Figure 4 System architecture, referring to the time sequence flow of data acquisition method as shown in Figure 5 It includes the following steps: data acquisition and monitoring module loads data source and monitoring configuration from configuration management module, data conversion and processing module loads data mapping and conversion rule from configuration management module, data transmission module loads target API and transmission configuration from configuration management module.

[0087] When source database occurs business data change event, corresponding transaction log is generated, data acquisition and monitoring module parses log, captures change event, and reports acquisition event count to monitoring and alarm module. After data acquisition and monitoring module parses original change data from log, it is sent to data conversion and processing module, data conversion and processing module executes cleaning, conversion and encapsulation processing to original change data, and reports conversion success or failure count to monitoring and alarm module according to processing result. If conversion succeeds, data conversion and processing module sends encapsulated data message to data transmission module, data transmission module writes data into local cache backup, and requests API transmission data to MES system, if transmission succeeds, when receiving the success response returned by MES system, mark the cached data as completed, and report transmission success count to monitoring and alarm module, if transmission fails, when receiving the failure response returned by MES system, report transmission failure event to monitoring and alarm module, monitoring and alarm module executes alarm operation when receiving transmission failure event, and data transmission module executes retransmission logic until transmission succeeds or reaches maximum retry number.

[0088] In a specific implementation, assume a discrete manufacturing plant needs to synchronize the "test data" from the safety testing station to the MES system in real time. During the configuration phase, in the configuration management interface, select the SQL Server database of the safety testing station as the data source and configure the connection parameters. Select the database table to be monitored, FunctionTestSoftware1. Configure data mapping rules: map the "KL30_V" field of the source table to the "insulation resistance" field of the MES API; map the "DCW" field to the "withstand voltage" field; and define a trigger rule that triggers transmission only when the "Collect" field becomes '0'. Configure the REST API endpoint address and authentication information of the MES system.

[0089] During the runtime phase (corresponding to steps S2-S6 above): The system starts up, and the data acquisition and monitoring module connects to the SQL Server database and begins parsing its redo log. When a work order record in the FunctionTestSoftware1 table has its Collect value updated to '0', this UPDATE operation is recorded in the redo log. The data acquisition and monitoring module immediately captures this change from the log and extracts the insulation resistance and withstand voltage data.

[0090] The data conversion and processing module checked the rules and confirmed the status was "0". Therefore, the data was assembled into {"LineCode":"PL_MC_2_0","LineName":"Electronic Control Assembly 2.0 Platform Assembly Production Line"}.

[0091] ","StationCode":"MC2_0_OP27","StationName":"MC2_0_ST270","PointCode":"","PointName":"","Value":"C10GL12PC1SBC119","FilePath":"","OperateType":30,"IsPass":true,"CheckDa taList":[{"CheckProject":"Insulation Resistance","CheckValue":"11.97","CheckResult":"","UpperLimitValue":"","LowerLimitValue":"","StandardValue":""},{"CheckProject":"Withstand voltage","CheckValue":"

[0092] 297.9","CheckResult":"","UpperLimitValue":"","LowerLimitValue":"","StandardValue":""}]} format. The data transmission module calls the MES's api / dataportal / invoke interface and sends the JSON data. The MES returns "1" (OK, data transmission success) and the data transmission module records success. The "successful transmission number" indicator on the monitoring board is incremented by 1.

[0093] Abnormal processing: if the MES server restarts at this time, the interface returns "2" (data transmission failure). The data transmission module places this request in the retry queue and attempts again after 1 second, 3 seconds, and 10 seconds until success.

[0094] Figure 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 6 The electronic device 500 includes one or more processors 501 and a memory 502.

[0095] The processor 501 can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device 500 to perform desired functions.

[0096] The memory 502 can include one or more computer program products, which can include various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM), cache memory, and / or the like. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer readable storage medium, and the processor 501 can run the program instructions to implement the data acquisition method of any embodiment of the present application described above and / or other desired functions. Various contents such as initial extrinsic parameters, threshold values, and the like can also be stored in the computer readable storage medium.

[0097] In one example, the electronic device 500 can also include an input device 503 and an output device 504, which are interconnected by a bus system and / or other forms of connection mechanism (not shown). The input device 503 can include, for example, a keyboard, a mouse, and the like. The output device 504 can output various information to the outside, including pre-warning prompt information, braking force, and the like. The output device 504 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0098] Of course, in order to simplify, Figure 6 Only some of the components in the electronic device 500 related to the present application are shown in the figure, and components such as buses, input / output interfaces, and the like are omitted. In addition, the electronic device 500 can also include any other appropriate components according to specific application cases.

[0099] In addition to the above method and device, an embodiment of the present application can also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the data acquisition method provided by any embodiment of the present application.

[0100] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of the present application, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as "C" language or similar programming languages. The program code can be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0101] In addition, an embodiment of the present application can also be a computer readable storage medium, which stores computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the data acquisition method provided by any embodiment of the present application.

[0102] The computer readable storage medium can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or apparatuses, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more conductive wires, portable disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any appropriate combination of the above.

[0103] It should be noted that the terms "comprises", "comprising", "includes", "including", "contains", "containing" or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, includes, contains or contains a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises, includes, contains or contains the identified element.

[0104] It should also be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", and the like, specify the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and simplify the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise expressly specified and limited, the terms "mount", "connect", "connect" and the like should be broadly understood, for example, can be fixedly connected, can be detachably connected, or integrally connected; can be mechanically connected, can be electrically connected; can be directly connected, can be indirectly connected through an intermediate medium, or can be connected between two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0105] The principles and implementation modes of the present application are described by applying specific examples, and the above example descriptions are only used to help understand the method and its core idea of the present application. The above description is only the preferred embodiment of the present application. It should be noted that due to the limited nature of the language expression, there are objectively infinite specific structures, and for those of ordinary skill in the art, without departing from the principles of the present application, a number of improvements, refinements or changes can be made, or the above technical features can be combined in an appropriate manner; these improvements, refinements, changes or combinations, or the application of the inventive concept and technical solution to other occasions without improvement, shall be regarded as the protection scope of the present application.

Claims

1. A data acquisition method, characterized in that, include: Real-time monitoring of log file contents yields monitoring results; Determine whether the preset data collection conditions are met based on the monitoring results; If the preset data collection conditions are met, the corresponding log data is extracted from the log file, and the log data is converted to a new format to obtain target data that meets the format requirements. The target data is sent to the MES by calling the interface of the Manufacturing Execution System (MES).

2. The data acquisition method according to claim 1, characterized in that, The log data is format-converted to obtain target data that meets the format requirements, including: According to the data mapping rules, the preset fields in the log data are converted into specified fields, and the data is assembled according to the set template to obtain target data that meets the format requirements.

3. The data acquisition method according to claim 1, characterized in that, Also includes: If a notification message indicating successful transmission is received from the interface, the corresponding monitoring metrics are updated.

4. The data acquisition method according to claim 1, characterized in that, Also includes: If a notification message indicating a transmission failure is received from the interface, or if the call to the Manufacturing Execution System (MES) interface fails, the target data will be retransmitted according to the set strategy.

5. The data acquisition method according to claim 1, characterized in that, Before calling the Manufacturing Execution System (MES) interface and sending the target data to the MES through the interface, the method further includes: The target data that meets the format requirements can be saved locally or cached.

6. The data acquisition method according to claim 1, characterized in that, Also includes: Store the target data that failed to be sent and set an identification tag.

7. A data acquisition system, characterized in that, include: The configuration management module is used to configure data source connection information, as well as receive and store data mapping rules and preset data collection conditions; The data acquisition and monitoring module is communicatively connected to the configuration management module. It is used to establish a connection with the corresponding database according to the data source connection information configured in the configuration management module, and to monitor the contents of the log file in real time to obtain the monitoring results. Based on the monitoring results, it is determined whether the preset data acquisition conditions are met. The data conversion and processing module is used to convert the monitoring results according to the data mapping rules if it is determined that the preset data acquisition conditions are met, and to process the converted data into target data that meets the format requirements; The data transmission module is used to call the interface of the Manufacturing Execution System (MES) and send the target data to the MES through the interface.

8. The data acquisition system according to claim 7, characterized in that, Also includes: The monitoring and alarm module is used to display one or more of the following in real time: data acquisition rate, transmission latency, and transmission success rate, and to send alarm notifications for data with transmission latency and / or transmission failure.

9. An electronic device, characterized in that, The electronic device includes: Processor and memory; The processor executes the steps of the data acquisition method as described in any one of claims 1 to 6 by calling the program or instructions stored in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that cause a computer to perform the steps of the data acquisition method as described in any one of claims 1 to 6.

Images