Method and system for data intercommunication between ERP and MES based on non-intrusive interface acquisition

Abstract

This invention belongs to the field of industrial data interaction technology. It provides a method and system for ERP and MES data interoperability based on non-intrusive interface acquisition. The method includes: acquiring dynamically reconstructed interface images of the ERP system; identifying anchor bounding boxes corresponding to key fields using a trained semantic anchor detection model; performing global OCR recognition on the images to obtain candidate text boxes with coordinates; using anchor points as reference nodes and text boxes as target nodes; calculating the distance and orientation features between nodes; constructing a spatial topology graph of interface elements; filtering target text matching the semantic anchor positions based on the topology; parsing key-value pair format process production data; and finally converting the data into an MES-compatible format to complete cross-system synchronization. This invention does not require software interface integration, can adapt to dynamically changing ERP systems, and enables automatic, code-free capture and cross-system transfer of production data.

Description

Technical Field

[0001] This invention belongs to the field of industrial data interaction technology, specifically a method and system for ERP and MES data interoperability based on non-intrusive interface acquisition. Background Technology

[0002] In the field of high-end customized equipment manufacturing, products are mostly developed on an order-based model, characterized by deep nesting of product BOM structures, frequent engineering changes, and highly customized process routes. To adapt to these frequent structural changes, the legacy ERP systems used in such scenarios lack fixed page rendering templates. They recursively reconstruct pages in real time based on the current product BOM level, number of parts, and process nodes, dynamically generating interface elements such as folded trees, multi-layered nested tables, and floating child windows. This results in dynamic and random changes to page control IDs, DOM structure, window layout, and field positions, with no stable XPath paths or fixed screen display layout. Furthermore, due to stringent cybersecurity regulations, secondary development, direct database connections, open API interfaces, and middleware deployments on ERP and MES systems are prohibited. Data exchange between the two systems can only be achieved through non-intrusive front-end interface data collection, rendering conventional interface integration and modification solutions completely unsuitable.

[0003] Existing non-intrusive ERP and MES data acquisition solutions are mainly divided into two categories: DOM tree parsing and crawling and fixed coordinate OCR crawling. Neither of them can adapt to the above-mentioned dynamic interface reconstruction scenarios and has significant technical defects.

[0004] Among them, the data collection scheme based on DOM tree parsing relies on fixed control IDs, stable DOM hierarchy and XPath path positioning data. However, in this scenario, the ERP page is reconstructed as a whole with the change of BOM, and there are no stable positioning characteristics, which directly leads to the failure of DOM parsing positioning and makes it impossible to complete the effective collection of core production data.

[0005] OCR acquisition solutions based on fixed screen coordinates rely on fixed screenshot areas to recognize data. They cannot adapt to the dynamic characteristics of tree node folding, table row and column expansion and contraction, and floating window offset in this scenario. They are prone to problems such as field misalignment, data omission, and text misrecognition. The acquisition accuracy is extremely low, and it only supports one-way data reading, which cannot achieve bidirectional data backfilling and interoperability between ERP and MES.

[0006] Furthermore, existing dynamic webpage RPA data collection technologies can only adapt to simple scenarios such as regular partial front-end refresh and random changes of single control IDs. They cannot cope with extreme working conditions such as recursive reconstruction of the entire page tree structure, nested multi-level windows, and dynamic scaling of table dimensions. In applications, problems such as script crashes, data matching failures, and data collection interruptions frequently occur, resulting in extremely poor stability.

[0007] In summary, existing non-intrusive data acquisition technologies cannot adapt to the dynamic reconstruction of the overall interface caused by frequent changes in the BOM in high-end equipment manufacturing scenarios. Based on this, this invention proposes a non-intrusive ERP and MES data interoperability method that adapts to the dynamic reconstruction interface to solve the shortcomings of existing technologies.

[0008] To this end, the present invention provides a method and system for ERP and MES data interoperability based on non-intrusive interface acquisition. Summary of the Invention

[0009] In order to overcome the shortcomings of the prior art, at least one technical problem raised in the background art is solved.

[0010] The technical solution adopted by this invention to solve its technical problem is:

[0011] One of the objectives of this invention is to provide a method for data interoperability between ERP and MES based on non-intrusive interface acquisition, comprising:

[0012] Step 1: Obtain the current interface image of the dynamically reconstructed ERP system;

[0013] Step 2: Construct a semantic anchor detection model. Input the current interface image into the semantic anchor detection model to identify and output the semantic anchor bounding boxes corresponding to preset key fields.

[0014] Step 3: Perform global optical character recognition on the current interface image to obtain a set of candidate text boxes containing coordinate information;

[0015] Step 4: Using the semantic anchor bounding box as the reference node and the candidate text box set as the target node, calculate the spatial distance and orientation features between each node to construct a spatial topology graph of the interface elements.

[0016] Step 5: Based on the spatial topology graph, extract the target text boxes that spatially match the bounding boxes of each semantic anchor point, and parse them to obtain the process production data in key-value pair form;

[0017] Step 6: Convert the process production data into a data format compatible with the MES system and synchronize it.

[0018] As a further improvement of the present invention, the specific process of obtaining the current interface image of the dynamically reconstructed ERP system is as follows:

[0019] Obtain the process identifier of the dynamically reconstructed ERP system running in the ERP operating system, lock the target window handle based on the process identifier, and obtain the window size information of the dynamically reconstructed ERP system and the current screen visible area size information through the target window handle.

[0020] If the window size information is less than or equal to the visible area size information, the underlying screenshot interface of the ERP operating system is directly called to capture the visible image corresponding to the target window handle as the current interface image.

[0021] If the window size is larger than the visible area size, a long screenshot stitching mechanism is triggered: the window of the dynamically reconstructed ERP system is controlled to scroll at a preset scrolling step size. After each scrolling operation, a local interface image is captured until the window edge is reached. The feature points of the overlapping area of ​​two adjacent local interface images are extracted. Based on the feature points of the overlapping area, all local interface images are stitched together to generate a complete panoramic image, which is then used as the current interface image.

[0022] As a further improvement of the present invention, the specific process of constructing the semantic anchor detection model is as follows:

[0023] Collect ERP interface images of different BOM levels generated by dynamic reconstruction, manually label the areas where preset key fields representing business attributes are located in the ERP interface images, generate a training set with real bounding box labels, select a general convolutional object detection network as the initial model backbone, use the training set to iteratively train the initial model, calculate the localization loss and classification loss between the predicted bounding box and the real bounding box, backpropagate to update the network weights until the loss function converges, and obtain the trained semantic anchor detection model, thus completing the construction of the semantic anchor detection model.

[0024] As a further improvement of the present invention, the specific process of identifying and outputting the semantic anchor bounding box corresponding to the preset key field is as follows:

[0025] After the current interface image is normalized in size, it is input into the trained semantic anchor detection model. The semantic anchor detection model extracts features through forward propagation and outputs multiple candidate anchor prediction boxes, as well as the classification confidence of the preset key fields corresponding to each candidate anchor prediction box.

[0026] As a further improvement of the present invention, the specific steps of identifying and outputting the semantic anchor bounding box corresponding to the preset key field also include:

[0027] The classification confidence score is compared with the confidence score threshold, and candidate anchor prediction boxes with a classification confidence score lower than the confidence score threshold are removed to obtain a set of high-confidence prediction boxes.

[0028] Calculate the intersection-union ratio (IUR) between any two high-confidence prediction boxes. If the IUR is greater than the IUR threshold, discard the high-confidence prediction boxes with lower classification confidence. Output the final retained prediction boxes as semantic anchor bounding boxes. The semantic anchor bounding box contains the two-dimensional coordinate information of the anchor point in the current interface image and the corresponding preset key field category.

[0029] As a further improvement of the present invention, the specific process of obtaining the candidate text box set containing coordinate information is as follows:

[0030] The acquired current interface image is sequentially processed by grayscale and binarization to obtain an enhanced binarized interface image. The binarized interface image is then input into a preset text detection network to extract connected component features from the binarized interface image, locate all regions containing text, and output multiple initial text bounding boxes and the two-dimensional coordinate information of each initial text bounding box.

[0031] Based on the two-dimensional coordinate information, the corresponding local text images are cropped from the binarized interface image, and the local text images are input one by one into the preset sequence recognition network. The output is the string content and character recognition confidence of each initial text bounding box.

[0032] The character recognition confidence level is compared with the preset character confidence level threshold one by one. Initial text bounding boxes with character recognition confidence levels lower than the preset character confidence level threshold are removed. The remaining initial text bounding boxes, their corresponding string content, and two-dimensional coordinate information are packaged to obtain a candidate text box set.

[0033] As a further improvement of the present invention, the specific process of constructing the spatial topology diagram of interface elements is as follows:

[0034] Obtain all semantic anchor bounding boxes as reference nodes in the graph structure, obtain all text boxes in the candidate text box set as target nodes in the graph structure, extract the center point coordinates for each reference node, extract the center point coordinates for each target node, and calculate the Euclidean distance between the center point coordinates of the reference node and the center point coordinates of the target node as the spatial distance.

[0035] A local rectangular coordinate system is established with the coordinates of the center point of the reference node as the origin. The polar angle of the center point of the target node in the local rectangular coordinate system is calculated and the polar angle is used as the orientation feature.

[0036] If the spatial distance is greater than the search radius threshold, no connection edge is established; if the spatial distance is less than or equal to the search radius threshold, a directed connection edge is established between the reference node and the target node, and the calculated spatial distance and orientation features are assigned as edge weight attributes to the directed connection edge, generating a spatial topology graph with the reference node as the core and the directed connection edge as the topology link.

[0037] As a further improvement of the present invention, the specific process of parsing to obtain the process production data in key-value pair form is as follows:

[0038] Obtain each reference node in the spatial topology graph, obtain all target nodes that have directed edges connected to the reference nodes, and extract the orientation features and spatial distances from the edge weight attributes corresponding to each directed edge.

[0039] The system presets horizontal and vertical matching sectors. The horizontal matching sector is defined as the area with a polar angle between -15 degrees and +15 degrees, and the vertical matching sector is defined as the area with a polar angle between 75 degrees and 105 degrees.

[0040] Determine whether the orientation features of each target node fall within the horizontal or vertical matching sector. Eliminate target nodes that do not fall within either sector and retain the remaining target nodes as valid candidate nodes. Among the valid candidate nodes, select the valid candidate node with the smallest spatial distance as the target text box that matches the baseline node.

[0041] Extract the preset key fields corresponding to the baseline node as data key names, and extract the string content corresponding to the target text box as data key values. Bind the data key names and data key values ​​to generate process production data in key-value pair form.

[0042] As a further improvement of the present invention, the specific process of converting the process production data into a data format compatible with the MES system and synchronizing it is as follows:

[0043] The process involves obtaining production data, extracting data key names and data key values, converting data key names into standard English field names corresponding to the MES system, removing redundant spaces and special characters, and converting string format data key values ​​into standardized key values ​​according to the data type requirements of standard English field names.

[0044] Standard English field names and standardized key values ​​are assembled to generate a serialized data packet in JSON format, which is then used as the data format for the MES system.

[0045] Call the MES system's program interface to send the serialized data packet to the MES system's data receiver, listen for the response status code returned by the MES system. If the reception is successful, end the current acquisition task; if the reception fails, push the serialized data packet into the retry queue and initiate resynchronization at preset time intervals until the reception is successful or the preset maximum retry threshold is reached.

[0046] The second objective of this invention is to provide an ERP and MES data interoperability system based on non-intrusive interface acquisition, comprising the following modules:

[0047] Image acquisition module: Acquires the current interface image of the dynamically reconstructed ERP system;

[0048] Anchor point target detection module: Constructs a semantic anchor point detection model, inputs the current interface image into the semantic anchor point detection model, identifies and outputs the semantic anchor point bounding boxes corresponding to preset key fields;

[0049] Global text recognition module: Performs global optical character recognition on the current interface image to obtain a set of candidate text boxes containing coordinate information;

[0050] Topology construction module: Using the semantic anchor bounding box as the reference node and the candidate text box set as the target node, calculate the spatial distance and orientation features between each node to construct a spatial topology graph of interface elements;

[0051] Key-value pair data parsing module: Based on the spatial topology graph, extract the target text boxes that spatially match the bounding boxes of each semantic anchor point, and parse them to obtain process production data in key-value pair form;

[0052] Data forwarding and synchronization module: Converts the process production data into a data format compatible with the MES system and synchronizes it.

[0053] The beneficial effects of this invention are as follows:

[0054] 1. This invention adopts a non-intrusive image acquisition solution of window screenshot + long image stitching. It does not require calling the internal interface of ERP software or modifying the original ERP program. It can adapt to scenarios with dynamic interface reconstruction and windows exceeding the visible area, with strong compatibility. It reduces the development and modification costs of industrial software integration. It relies on the trained semantic anchor point detection model to lock key fields, and then combines it with OCR to recognize all text. It uses anchor points and text boxes to construct a spatial topology relationship graph, and relies on directional sectors and distance to filter and match key values. It is not affected by minor adjustments to the interface layout or small changes in text position, and the data capture stability is higher.

[0055] 2. By filtering target text through polar angle partitioning matching rules, only horizontally or vertically adjacent text is selected as the corresponding value, effectively avoiding recognition interference caused by a large amount of irrelevant text in the interface, greatly improving the accuracy of key-value pair data parsing, automatically completing field standardization conversion, generating JSON data packets and supporting failure retry mechanism, and can stably synchronize process production data in ERP to MES system automatically, realize automatic data flow across industrial software, reduce manual input, and improve the efficiency of factory production data flow.

[0056] 3. The entire solution relies on machine vision and spatial topology logic to extract data. It is an external interface acquisition solution that does not intrude on the business database. The data acquisition process is secure and controllable, and it is suitable for ERP systems with multiple versions and dynamically adjustable interfaces. Attached Figure Description

[0057] The invention will now be further described with reference to the accompanying drawings.

[0058] Figure 1 This is a flowchart of the steps of the ERP and MES data interoperability method based on non-intrusive interface acquisition in this invention;

[0059] Figure 2 This is a system module diagram of the ERP and MES data interoperability system based on non-intrusive interface acquisition, as described in this invention. Detailed Implementation

[0060] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0061] Example 1

[0062] like Figure 1 As shown in the embodiment of the present invention, the method for ERP and MES data interoperability based on non-intrusive interface acquisition includes:

[0063] Step 1: Obtain the current interface image of the dynamically reconstructed ERP system;

[0064] In some embodiments, the specific process of obtaining the current interface image of the dynamically reconstructed ERP system in step one is as follows:

[0065] Obtain the process identifier of the dynamically reconstructed ERP system running in the ERP operating system, and lock the target window handle based on the process identifier;

[0066] The window size information of the dynamically reconstructed ERP system and the visible area size information of the current screen are obtained through the target window handle.

[0067] Determine whether the window size information is larger than the visible area size information;

[0068] If the window size information is less than or equal to the visible area size information, the screenshot interface of the ERP operating system is directly called to capture the visible image corresponding to the target window handle and use it as the current interface image.

[0069] If the window size information is larger than the visible area size information, it indicates that the interface is too long due to the BOM hierarchy expansion. At this time, a long screenshot stitching mechanism is triggered: the window of the dynamically reconstructed ERP system is controlled to scroll at a preset scroll step; after each scroll operation, a local interface image is captured until the window edge is scrolled; the feature points of the overlapping area of ​​two adjacent local interface images are extracted, and all local interface images are stitched together based on the feature points of the overlapping area to generate a complete panoramic image, which is used as the current interface image.

[0070] It should be noted that the preset scrolling step size is, for example, set to 80% of the height value in the visible area size information. The basis for this setting is that when performing long screenshot stitching, a certain image overlap area must be retained for the feature point matching algorithm to align. Retaining 20% ​​overlap area can ensure that the image stitching algorithm has enough overlapping pixels for accurate alignment, avoiding stitching misalignment or gaps caused by too many similar tables in the dynamically reconstructed interface, and can also maximize the effective capture area for each scroll, thereby achieving the best balance between acquisition accuracy and acquisition efficiency.

[0071] In other embodiments, before performing window scrolling and image capture, the top-level floating pop-up of the window is identified first and the pop-up layer is temporarily frozen to avoid the pop-up from obscuring the business table and causing incomplete screenshots; at the same time, the page rendering refresh action is locked to ensure that the page image remains stable during each scroll capture; and non-business areas such as the window title bar and menu bar are automatically cropped, retaining only the business visible area where the BOM tree table is located, reducing invalid image data;

[0072] Step 2: Construct a semantic anchor detection model. Input the current interface image into the semantic anchor detection model to identify and output the semantic anchor bounding boxes corresponding to preset key fields.

[0073] In some embodiments, the specific process of constructing the semantic anchor detection model in step two is as follows:

[0074] Constructing a semantic anchor detection model: Collect ERP interface images from different historical BOM levels generated by dynamic reconstruction. Manually label the regions in the images containing the preset key fields representing business attributes (the preset key fields include, but are not limited to, fixed header text such as "material code", "specification", "required quantity" and "process route") to generate a training set with real bounding box labels. Build an initial object detection model based on a deep convolutional neural network. In this embodiment, a general convolutional object detection network is selected as the backbone of the initial model, without modifying the basic network structure. Iteratively train the initial object detection model using the training set. Calculate the localization loss and classification loss between the predicted bounding box and the real bounding box, and backpropagate to update the network weights until the loss function converges, thus obtaining the trained semantic anchor detection model.

[0075] The semantic anchor detection model is used to output multiple candidate anchor prediction boxes, and the classification confidence of each candidate anchor prediction box corresponding to the preset key field;

[0076] In some embodiments, the specific process of inputting the current interface image into the semantic anchor detection model, identifying and outputting the semantic anchor bounding box corresponding to the preset key field in step two is as follows:

[0077] After normalizing the size of the current interface image obtained in step one, it is input into the trained semantic anchor detection model.

[0078] The semantic anchor detection model extracts features through forward propagation and outputs multiple candidate anchor prediction boxes, as well as the classification confidence of each candidate anchor prediction box corresponding to the preset key field.

[0079] The classification confidence level is compared with a preset confidence threshold, and candidate anchor prediction boxes with classification confidence levels lower than the preset confidence threshold are removed to obtain a set of high-confidence prediction boxes.

[0080] A non-maximum suppression algorithm is applied to the set of high-confidence prediction boxes to calculate the intersection-union ratio (IUR) between any two high-confidence prediction boxes. If the IUR is greater than a preset IUR threshold, the high-confidence prediction box with the lower classification confidence is removed, and the remaining prediction boxes are output as the semantic anchor bounding boxes. The semantic anchor bounding boxes include the two-dimensional coordinate information of the anchor point in the current interface image and the corresponding preset key field category.

[0081] It should be noted that the preset confidence threshold is set to 0.85, for example. The basis for this setting is that although the ERP interface will be dynamically reconstructed, the visual characteristics (font, font size, background color) of the header text (such as "material code") used as anchor points are relatively fixed. Setting a high threshold of 0.85 can effectively filter out similar interference text that occasionally appears in the interface data rows (for example, a part name happens to contain the word "code"), ensuring the absolute accuracy of anchor point positioning.

[0082] It should be further explained that the preset intersection-union ratio (IUU) threshold is set to, for example, 0.45. The reason for this setting is that in a dynamically reconstructed compact ERP table, adjacent different headers (such as "unit price" and "total price") may be very close in physical space. If the threshold is set too low (such as 0.1), the non-maximum suppression algorithm will mistakenly delete adjacent independent anchor points as redundant boxes, resulting in anchor point loss. If the threshold is set too high (such as 0.8), it will not be able to effectively eliminate multiple overlapping predicted boxes generated by model jitter at the same anchor point. Setting it to 0.45 can perfectly balance the needs of deduplication and preventing missed detections.

[0083] Step 3: Perform global optical character recognition on the current interface image to obtain a set of candidate text boxes containing coordinate information;

[0084] In some embodiments, the specific process of performing global optical character recognition on the current interface image to obtain a set of candidate text boxes containing coordinate information in step three is as follows:

[0085] The current interface image obtained in step one is sequentially subjected to grayscale processing and adaptive binarization processing to eliminate image noise caused by uneven backlighting of industrial field displays and screen reflection, resulting in an enhanced binarized interface image.

[0086] The binarized interface image is input into a preset text detection network to extract connected component features from the image, locate all regions containing text, and output multiple initial text bounding boxes and the two-dimensional coordinate information of each initial text bounding box.

[0087] In this embodiment, a general text detection network based on the differentiable binarization (DBNet) architecture is selected as the initial detection backbone without modifying its underlying mathematical model. The network is pre-trained using an open-source text detection dataset containing industrial software interface features to obtain the preset text detection network.

[0088] Based on the two-dimensional coordinate information, the corresponding local text images are cropped from the binarized interface image, and the local text images are input one by one into the preset sequence recognition network for character sequence decoding, and the string content and character recognition confidence of each initial text bounding box are output.

[0089] In this embodiment, a general sequence recognition network based on convolutional recurrent neural network (CRNN) combined with connection temporal classification (CTC) algorithm is selected as the initial recognition backbone. Its basic network structure is not modified. Iterative training is performed using a synthetic text dataset containing Chinese, English and industrial special symbols to obtain the preset sequence recognition network.

[0090] The character recognition confidence level is compared with the preset character confidence threshold one by one;

[0091] The initial text bounding boxes with character recognition confidence scores lower than the preset character confidence score threshold are removed, and the remaining initial text bounding boxes, their corresponding string content, and two-dimensional coordinate information are packaged to construct the candidate text box set.

[0092] It should be noted that the preset character confidence threshold is set to 0.60, for example, based on the following: In the dynamically reconstructed ERP interface, there are a large number of table border lines, dashed lines of folded tree diagrams, and UI separators; conventional optical character recognition is very likely to misidentify these non-text UI structural elements as the letter "l", the number "1", or the dash "-", but the confidence of such misidentification is usually low (generally between 0.3 and 0.5); setting the character confidence threshold to 0.60 can accurately filter out pseudo-text boxes caused by table lines, while retaining as much of the real business data as possible (whose confidence is usually above 0.7) due to the slight blurring of fonts caused by the aging of the industrial control computer screen, thus providing a clean underlying data source for the subsequent construction of spatial topology relationships;

[0093] Step 4: Using the semantic anchor bounding box as the reference node and the candidate text box set as the target node, calculate the spatial distance and orientation features between each node to construct a spatial topology graph of the interface elements.

[0094] In some embodiments, the specific process of constructing a spatial topology graph of interface elements in step four, using the semantic anchor bounding box as the reference node and the candidate text box set as the target node, to calculate the spatial distance and orientation features between each node, is as follows:

[0095] Traverse all the semantic anchor bounding boxes output in step two and use them as the base node in the graph structure; at the same time, traverse all the text boxes in the candidate text box set obtained in step three and use them as the target node in the graph structure.

[0096] For each of the reference nodes, extract its center point coordinates; and for each of the target nodes, extract its center point coordinates.

[0097] Calculate the Euclidean distance between the coordinates of the center point of the reference node and the coordinates of the center point of the target node, and use it as the spatial distance;

[0098] A local rectangular coordinate system is established with the coordinates of the center point of the reference node as the origin. The polar angle of the center point of the target node in this local rectangular coordinate system is calculated, and the polar angle is used as the orientation feature.

[0099] The spatial distance is compared with a preset search radius threshold. If the spatial distance is greater than the search radius threshold, it is determined that the target node and the reference node are not related in terms of physical layout, and no connection edge is established.

[0100] If the spatial distance is less than or equal to the search radius threshold, a directed connection edge is established between the reference node and the target node, and the calculated spatial distance and the orientation feature are assigned as edge weight attributes to the directed connection edge.

[0101] After traversing and calculating all node pairs, a spatial topology graph is generated with the reference node as the core and the directed connecting edges as topological links.

[0102] It should be noted that the preset search radius threshold is set to, for example, 15% of the width of the current interface image. The basis for this setting is that in a dynamically reconstructed ERP interface, whether it is a horizontally arranged "key-value" pair (such as a table header on the left and data on the right) or a vertically arranged table (such as a table header on top and data at the bottom), the corresponding business data text boxes are usually closely surrounded by the table header anchor point. Setting it to 15% of the image width can effectively cover the maximum physical distance between adjacent cells, while avoiding forcibly including irrelevant data that is too far away into the topology calculation, thereby significantly reducing the computational complexity of subsequent graph neural networks or heuristic rules and preventing the topology graph from experiencing dimensionality explosion.

[0103] Step 5: Based on the spatial topology graph, extract the target text boxes that spatially match the bounding boxes of each semantic anchor point, and parse them to obtain the process production data in key-value pair form;

[0104] In some embodiments, the specific process of extracting target text boxes that spatially and logically match the bounding boxes of each semantic anchor point based on the spatial topology graph and parsing them to obtain key-value pair form of process production data in step five is as follows:

[0105] Traverse each of the reference nodes in the spatial topology graph and obtain all target nodes that have the directed connection edge with the reference node;

[0106] Read the edge weight attributes corresponding to each of the directed connection edges, and extract the orientation features and spatial distances therefrom;

[0107] Based on the layout rules of the screen coordinate system, horizontal matching sectors and vertical matching sectors are preset; wherein, the horizontal matching sector is defined as the area with a polar angle between -15 degrees and +15 degrees, indicating that the physical position is located directly to the right of the reference node; the vertical matching sector is defined as the area with a polar angle between 75 degrees and 105 degrees, indicating that the physical position is located directly below the reference node.

[0108] Determine whether the azimuth features of each target node fall within the horizontal matching sector or the vertical matching sector;

[0109] Remove target nodes that do not fall into any of the above sectors, and keep the remaining target nodes as valid candidate nodes;

[0110] Among the valid candidate nodes, their spatial distances are compared, and the valid candidate node with the smallest spatial distance is selected as the target text box that matches the baseline node in spatial logic.

[0111] Extract the preset key field corresponding to the baseline node as the data key name, and extract the string content corresponding to the target text box as the data key value;

[0112] Bind the data key name to the data key value to generate process production data in the form of key-value pairs;

[0113] It should be noted that the angle tolerance threshold (i.e., ±15 degrees) set in the horizontal and vertical matching sectors is based on the following: In the dynamically reconstructed ERP interface, the dynamic changes in the text alignment (such as left alignment, center alignment, right alignment) and padding within the table cells can cause the center point of the target text box to not be on an absolute horizontal or vertical line with the center point of the reference node. Setting an angle tolerance of 15 degrees can perfectly accommodate the physical coordinate offset caused by different UI rendering engines and alignment methods, ensuring the robustness of the matching and preventing the misidentification of diagonally opposite irrelevant text boxes as the target value.

[0114] Step 6: Convert the process production data into a data format compatible with the MES system and synchronize it.

[0115] In some embodiments, the specific process of converting the process production data into a data format compatible with the MES system and synchronizing it in step six is ​​as follows:

[0116] Obtain the process production data in the form of key-value pairs generated in step five, and extract the data key name and the data key value therein;

[0117] The system calls a preset field mapping dictionary to convert the data key name (e.g., the Chinese header "material code") into the standard English field name (e.g., "MaterialCode") corresponding to the underlying database of the MES system.

[0118] The data key values ​​are cleaned and converted in format to remove redundant spaces and special symbols that may remain during the visual recognition process. Based on the data type requirements of the standard English field names, the string format data key values ​​are converted into corresponding integer, floating-point, or date data to obtain standardized key values.

[0119] The standard English field names and the standardized key values ​​are assembled to generate a serialized data packet in JSON format, and the serialized data packet is used as a data format compatible with the MES system.

[0120] The standard application programming interface of the MES system is invoked to send the serialized data packet to the data receiving end of the MES system;

[0121] Listen to the response status code returned by the MES system. If the response status code indicates successful synchronization, the current collection task ends. If the response status code indicates synchronization failure or network timeout, the serialized data packet is pushed into the retry queue, and resynchronization is initiated at preset time intervals until synchronization is successful or the preset maximum retry threshold is reached.

[0122] It should be noted that the preset maximum retry threshold is set to, for example, 3 times. The basis for setting the maximum retry threshold is that in the industrial workshop, the network environment may be affected by electromagnetic interference caused by the start-up and shutdown of large equipment, resulting in instantaneous network jitter or packet loss. Setting it to 3 retries can effectively overcome the synchronization failure caused by occasional network fluctuations, ensure the absolute reliability of data communication between ERP and MES, and avoid the data acquisition end from getting stuck in an infinite loop of retries and exhausting the memory resources of the edge computing node when the MES system is actually down or under maintenance.

[0123] It should be further noted that the preset system field mapping dictionary refers to a set of mapping relations pre-configured in the system initialization and deployment stage to eliminate the semantic gap between heterogeneous industrial software; this dictionary essentially stores the strict corresponding relations between the natural language table headers displayed on the front-end interface of the ERP system and the underlying database fields of the MES system; specifically, the preset system field mapping dictionary includes a plurality of mapping entries, and each mapping entry includes at least a source field name, a target field name and a target data type; wherein the source field name corresponds to the data key name extracted by visual parsing (e.g., Chinese "Material Code" or "Demand Quantity"), the target field name corresponds to the standard English field name compatible with the MES system (e.g., "MaterialCode" or "RequireQty"), and the target data type is used to guide subsequent format cleaning and type conversion of the data key value (e.g., specified as String or Integer); by introducing the preset system field mapping dictionary, the front-end display text obtained through non-invasive visual collection can be accurately converted into standardized structured data recognizable by the underlying interface of industrial software, thereby completely breaking through the data barrier between the visual collection end and the MES receiving end;

[0124] Example 2

[0125] As Figure 2 shown, based on the specific implementation process of Embodiment 1, the present invention provides an ERP and MES data interconnection system based on non-invasive interface collection, which includes the following modules:

[0126] Image acquisition module: acquire the current interface image of the dynamically reconstructed ERP system;

[0127] Anchor point target detection module: construct a semantic anchor point detection model, input the current interface image into the semantic anchor point detection model, identify and output the semantic anchor bounding box corresponding to the preset key field;

[0128] Global optical character recognition module: perform global optical character recognition on the current interface image to obtain a candidate text box set containing coordinate information;

[0129] Topological relation construction module: take the semantic anchor bounding box as the reference node and the candidate text box set as the target node, calculate the spatial distance and orientation characteristics between each node, and construct a spatial topological graph of interface elements;

[0130] Key-value pair data parsing module: based on the spatial topological graph, extract target text boxes that are spatially logically matched with each semantic anchor bounding box, and parse to obtain process production data in the form of key-value pairs;

[0131] Data forwarding and synchronization module: Converts the process production data into a data format compatible with the MES system and synchronizes it.

[0132] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for data interoperability between ERP and MES based on non-intrusive interface acquisition, characterized by: include: Step 1: Obtain the current interface image of the dynamically reconstructed ERP system; Step 2: Construct a semantic anchor detection model. Input the current interface image into the semantic anchor detection model to identify and output the semantic anchor bounding boxes corresponding to preset key fields. Step 3: Perform global optical character recognition on the current interface image to obtain a set of candidate text boxes containing coordinate information; Step 4: Using the semantic anchor bounding box as the reference node and the candidate text box set as the target node, calculate the spatial distance and orientation features between each node to construct a spatial topology graph of the interface elements. Step 5: Based on the spatial topology graph, extract the target text boxes that spatially match the bounding boxes of each semantic anchor point, and parse them to obtain the process production data in key-value pair form; Step 6: Convert the process production data into a data format compatible with the MES system and synchronize it.

2. The method for ERP and MES data interoperability based on non-intrusive interface acquisition according to claim 1, characterized in that, The specific process for obtaining the current interface image of the dynamically reconstructed ERP system is as follows: Obtain the process identifier of the dynamically reconstructed ERP system running in the ERP operating system, lock the target window handle based on the process identifier, and obtain the window size information of the dynamically reconstructed ERP system and the current screen visible area size information through the target window handle. If the window size information is less than or equal to the visible area size information, the underlying screenshot interface of the ERP operating system is directly called to capture the visible image corresponding to the target window handle as the current interface image. If the window size is larger than the visible area size, a long screenshot stitching mechanism is triggered: the window of the dynamically reconstructed ERP system is controlled to scroll at a preset scrolling step size. After each scrolling operation, a local interface image is captured until the window edge is reached. The feature points of the overlapping area of ​​two adjacent local interface images are extracted. Based on the feature points of the overlapping area, all local interface images are stitched together to generate a complete panoramic image, which is then used as the current interface image.

3. The method for ERP and MES data interoperability based on non-intrusive interface acquisition according to claim 1, characterized in that, The specific process for constructing the semantic anchor detection model is as follows: Collect ERP interface images of different BOM levels generated by dynamic reconstruction, manually label the areas where preset key fields representing business attributes are located in the ERP interface images, generate a training set with real bounding box labels, select a general convolutional object detection network as the initial model backbone, use the training set to iteratively train the initial model, calculate the localization loss and classification loss between the predicted bounding box and the real bounding box, backpropagate to update the network weights until the loss function converges, and obtain the trained semantic anchor detection model, thus completing the construction of the semantic anchor detection model.

4. The method for ERP and MES data interoperability based on non-intrusive interface acquisition according to claim 1, characterized in that, The specific process of identifying and outputting the semantic anchor bounding box corresponding to the preset key fields is as follows: After the current interface image is normalized in size, it is input into the trained semantic anchor detection model. The semantic anchor detection model extracts features through forward propagation and outputs multiple candidate anchor prediction boxes, as well as the classification confidence of the preset key fields corresponding to each candidate anchor prediction box.

5. The method for ERP and MES data interoperability based on non-intrusive interface acquisition according to claim 4, characterized in that, The specific steps of identifying and outputting the semantic anchor bounding boxes corresponding to preset key fields also include: The classification confidence score is compared with the confidence score threshold, and candidate anchor prediction boxes with a classification confidence score lower than the confidence score threshold are removed to obtain a set of high-confidence prediction boxes. Calculate the intersection-union ratio (IUR) between any two high-confidence prediction boxes. If the IUR is greater than the IUR threshold, discard the high-confidence prediction boxes with lower classification confidence. Output the final retained prediction boxes as semantic anchor bounding boxes. The semantic anchor bounding box contains the two-dimensional coordinate information of the anchor point in the current interface image and the corresponding preset key field category.

6. The method for ERP and MES data interoperability based on non-intrusive interface acquisition according to claim 1, characterized in that, The specific process for obtaining the set of candidate text boxes containing coordinate information is as follows: The acquired current interface image is sequentially processed by grayscale and binarization to obtain an enhanced binarized interface image. The binarized interface image is then input into a preset text detection network to extract connected component features from the binarized interface image, locate all regions containing text, and output multiple initial text bounding boxes and the two-dimensional coordinate information of each initial text bounding box. Based on the two-dimensional coordinate information, the corresponding local text images are cropped from the binarized interface image, and the local text images are input one by one into the preset sequence recognition network. The output is the string content and character recognition confidence of each initial text bounding box. The character recognition confidence level is compared with the preset character confidence level threshold one by one. Initial text bounding boxes with character recognition confidence levels lower than the preset character confidence level threshold are removed. The remaining initial text bounding boxes, their corresponding string content, and two-dimensional coordinate information are packaged to obtain a candidate text box set.

7. The method for ERP and MES data interoperability based on non-intrusive interface acquisition according to claim 1, characterized in that, The specific process of constructing the spatial topology diagram of interface elements is as follows: Obtain all semantic anchor bounding boxes as reference nodes in the graph structure, obtain all text boxes in the candidate text box set as target nodes in the graph structure, extract the center point coordinates for each reference node, extract the center point coordinates for each target node, and calculate the Euclidean distance between the center point coordinates of the reference node and the center point coordinates of the target node as the spatial distance. A local rectangular coordinate system is established with the coordinates of the center point of the reference node as the origin. The polar angle of the center point of the target node in the local rectangular coordinate system is calculated and the polar angle is used as the orientation feature. If the spatial distance is greater than the search radius threshold, no connection edge is established; if the spatial distance is less than or equal to the search radius threshold, a directed connection edge is established between the reference node and the target node, and the calculated spatial distance and orientation features are assigned as edge weight attributes to the directed connection edge, generating a spatial topology graph with the reference node as the core and the directed connection edge as the topology link.

8. The method for ERP and MES data interoperability based on non-intrusive interface acquisition according to claim 1, characterized in that, The specific process of parsing to obtain key-value pair form of process production data is as follows: Obtain each reference node in the spatial topology graph, obtain all target nodes that have directed edges connected to the reference nodes, and extract the orientation features and spatial distances from the edge weight attributes corresponding to each directed edge. The system presets horizontal and vertical matching sectors. The horizontal matching sector is defined as the area with a polar angle between -15 degrees and +15 degrees, and the vertical matching sector is defined as the area with a polar angle between 75 degrees and 105 degrees. Determine whether the orientation features of each target node fall within the horizontal or vertical matching sector. Eliminate target nodes that do not fall within either sector and retain the remaining target nodes as valid candidate nodes. Among the valid candidate nodes, select the valid candidate node with the smallest spatial distance as the target text box that matches the baseline node. Extract the preset key fields corresponding to the baseline node as data key names, and extract the string content corresponding to the target text box as data key values. Bind the data key names and data key values ​​to generate process production data in key-value pair form.

9. The method for ERP and MES data interoperability based on non-intrusive interface acquisition according to claim 1, characterized in that, The specific process of converting the production data into a data format compatible with the MES system and synchronizing it is as follows: The process involves obtaining production data, extracting data key names and data key values, converting data key names into standard English field names corresponding to the MES system, removing redundant spaces and special characters, and converting string format data key values ​​into standardized key values ​​according to the data type requirements of standard English field names. Standard English field names and standardized key values ​​are assembled to generate a serialized data packet in JSON format, which is then used as the data format for the MES system. Call the MES system's program interface to send the serialized data packet to the MES system's data receiver, listen for the response status code returned by the MES system, and end the current data collection task if the reception is successful. If reception fails, the serialized data packet is pushed into the retry queue, and resynchronization is initiated at preset time intervals until reception is successful or the preset maximum retry threshold is reached.

10. An ERP and MES data interoperability system based on non-intrusive interface acquisition, characterized in that: include: Image acquisition module: Acquires the current interface image of the dynamically reconstructed ERP system; Anchor point target detection module: Constructs a semantic anchor point detection model, inputs the current interface image into the semantic anchor point detection model, identifies and outputs the semantic anchor point bounding boxes corresponding to preset key fields; Global text recognition module: Performs global optical character recognition on the current interface image to obtain a set of candidate text boxes containing coordinate information; Topology construction module: Using the semantic anchor bounding box as the reference node and the candidate text box set as the target node, calculate the spatial distance and orientation features between each node to construct a spatial topology graph of interface elements; Key-value pair data parsing module: Based on the spatial topology graph, extract the target text boxes that spatially match the bounding boxes of each semantic anchor point, and parse them to obtain process production data in key-value pair form; Data forwarding and synchronization module: Converts the process production data into a data format compatible with the MES system and synchronizes it.

Images