AI reasoning implementation method and AI model training method and device
By using AI inference methods and AI models to detect product defects in the detection equipment, the serious problem of missed detection in the existing technology is solved, higher detection accuracy and lower manual re-judgment costs are achieved, and product yield and equipment productivity are improved.
Patent Information
- Application Number
- CN202411747649.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-29
- Publication Date
- 2025-03-11
AI Technical Summary
Existing detection equipment uses computer vision detection algorithms to detect product defects, resulting in a common phenomenon of over-checking, affecting product yield and equipment productivity.
AI inference methods and AI models are used to detect product defects, and the AI model trained by the training platform is obtained through the inference platform, and the data to be inferred to the device side are inferred to obtain inference results, and the data to be inferred and the results to be inferred are shared with the training platform to iteratively update the AI model.
It improves the accuracy of product defect detection, reduces the phenomenon of over-examination, reduces manual re-judgement costs, and improves product yield and equipment productivity.
Smart Images
Figure CN119599134A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the fields of artificial intelligence and industrial Internet technology, and in particular to a method for implementing AI reasoning, a method for training an AI model, and a device. Background Art
[0002] The semiconductor industry has a large number of testing equipment in the production and manufacturing process of products. The detection performance of the testing equipment directly affects the product yield, equipment utilization rate, etc. Existing testing equipment mostly uses traditional computer vision (CV) detection algorithms to detect product defects, resulting in a relatively common phenomenon of over-inspection and missed inspection, which easily leads to the risk of stockpiling and thus affects production capacity. A large number of products require personnel to re-judge, which has high labor costs. At the same time, the experience of personnel is different and the mobility is high, making it difficult to control the quality of over-inspection and missed inspection, which affects the product yield. Summary of the invention
[0003] The embodiments of the present invention provide an implementation method of AI reasoning, a training method and a device for an AI model, which are used to solve the problem in the related art that over- and missed-detection phenomena are common when using computer vision detection algorithms to detect product defects.
[0004] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:
[0005] In a first aspect, an embodiment of the present invention provides a method for implementing AI reasoning, including:
[0006] The inference platform obtains the AI model trained by the training platform and the data to be inferred on the device side;
[0007] The reasoning platform uses the AI model to reason about the data to be reasoned, and obtains a reasoning result;
[0008] The inference platform returns the inference result to the device end;
[0009] The inference platform stores the data to be inferred and the inference results, and the data to be inferred and the inference results can be used for iterative updating of the AI model of the training platform.
[0010] Optionally, also include:
[0011] The inference platform obtains the AI model iteratively updated by the training platform;
[0012] The reasoning platform uses the iteratively updated AI model to update the local AI model.
[0013] Optionally, the reasoning platform stores the data to be reasoned and the reasoning result, including:
[0014] The reasoning platform stores the data to be reasoned in a shared storage device, and stores the reasoning result in a database. The data in the shared storage device can be shared by the reasoning platform and the training platform.
[0015] In a second aspect, an embodiment of the present invention provides a method for training an AI model, comprising:
[0016] The training platform obtains the training data set;
[0017] The training platform trains the AI model according to the training data set;
[0018] The training platform publishes the trained AI model to the inference platform;
[0019] The training platform obtains the data to be inferred and the inference results from the inference platform, and expands the training data set according to the data to be inferred and the inference results;
[0020] The training platform trains the AI model based on the expanded training data set to obtain an iteratively updated AI model;
[0021] The training platform publishes the iteratively updated AI model to the inference platform.
[0022] In a third aspect, an embodiment of the present invention provides a reasoning platform, including: a management module and at least one reasoning module;
[0023] The management module is used to obtain the AI model trained by the training platform and push the AI model to at least one of the reasoning modules;
[0024] The inference module is used to obtain the AI model pushed by the management module and the data to be inferred on the device side; use the AI model to infer the data to be inferred to obtain an inference result; return the inference result to the device side; store the data to be inferred, and the data to be inferred and the inference result can be used for iterative update of the AI model of the training platform.
[0025] Optionally, the management module is used to obtain the iteratively updated AI model of the training platform, and push the iteratively updated AI model to the reasoning module;
[0026] The reasoning module is used to update the local AI model using the iteratively updated AI model.
[0027] Optionally, the inference module is used to store the data to be inferred in a shared storage device, and report the inference result to the management module, and the data of the shared storage device can be shared by the inference platform and the training platform;
[0028] The management module is used to store the reasoning result reported by the reasoning module in a database.
[0029] In a fourth aspect, an embodiment of the present invention provides a training platform, including a training module, a model publishing module and an iterative update module, wherein:
[0030] The training module is used to obtain a training data set; and train an AI model according to the training data set;
[0031] The model publishing module is used to publish the trained AI model to the inference platform;
[0032] The iterative update module is used to obtain the data to be inferred and the inference results from the inference platform, and expand the training data set according to the data to be inferred and the inference results;
[0033] The training module is further used to train the AI model based on the expanded training data set to obtain an iteratively updated AI model;
[0034] The model publishing module is also used to send the iteratively updated AI model to the reasoning platform.
[0035] In a fifth aspect, an embodiment of the present invention provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the method for implementing AI reasoning as described in the first aspect above are implemented.
[0036] In the sixth aspect, an embodiment of the present invention provides a server, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the training method of the AI model as described in the second aspect above are implemented.
[0037] In a seventh aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect or the second aspect.
[0038] In an embodiment of the present invention, an artificial intelligence (AI) model is used to improve the detection accuracy of product defects. The reasoning platform obtains a uniformly trained AI model from the training platform, thereby realizing the sharing of the trained AI model and avoiding repeated training of the AI model. The AI model can be quickly implemented. At the same time, the reasoning platform shares its own data to be inferred and reasoning results with the training platform for iterative updating of the AI model, thereby expanding the training data and improving the AI model. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0040] Figure 1 Schematic diagram of the architecture of an AI reasoning system according to an embodiment of the present invention;
[0041] Figure 2 A schematic diagram of the functional structure of the reasoning platform according to an embodiment of the present invention;
[0042] Figure 3 A schematic diagram of the framework structure of the reasoning platform of an embodiment of the present invention;
[0043] Figure 4 A schematic diagram of the framework structure of a training platform according to an embodiment of the present invention;
[0044] Figure 5 A flowchart of a method for implementing AI reasoning according to an embodiment of the present invention;
[0045] Figure 6 Schematic diagram of a flow chart of a training method for an AI model according to an embodiment of the present invention;
[0046] Figure 7 A schematic diagram of the interaction process between the reasoning platform and the training platform of an embodiment of the present invention;
[0047] Figure 8 A schematic diagram of a self-loading process under a model according to an embodiment of the present invention;
[0048] Fig. 9 A schematic diagram of a flow chart of model calling by the inference module of an embodiment of the present invention;
[0049] Fig.10 A schematic diagram of an example interface design of an inference module and a device end according to an embodiment of the present invention;
[0050] Fig.11 A schematic diagram of the software architecture of the RPA system according to an embodiment of the present invention;
[0051] Fig.12 A schematic diagram of the hardware architecture of an RPA system according to an embodiment of the present invention;
[0052] Fig.13 A flowchart of implementing AI reasoning in a cloud-edge-end collaboration in an RPA case according to an embodiment of the present invention;
[0053] Fig.14A schematic diagram of realizing matrix remote layout through process control center technology according to an embodiment of the present invention;
[0054] Fig.15 A schematic diagram of the algorithm architecture of an RPA case in an embodiment of the present invention;
[0055] Fig.16 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention;
[0056] Fig.17 Schematic diagram of the structure of a server according to an embodiment of the present invention. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0058] Please refer to Figure 1 and Figure 2 , Figure 1 Schematic diagram of the architecture of the AI reasoning system according to an embodiment of the present invention. Figure 2 The functional architecture diagram of the AI reasoning system of an embodiment of the present invention is shown in FIG. 1 , which includes a training platform and a reasoning platform.
[0059] The training platform (also called the training cloud platform) is a training platform for AI models, which may include one or more servers for training AI models based on the acquired training data sets. The training platform is deployed in the cloud.
[0060] The inference platform is connected to the training platform, and is used to obtain the AI model trained by the training platform, and use the AI model to infer the data to be inferred (such as product inspection pictures, etc.) sent by the device side, to obtain the inference result (such as information on whether the product in the marked inspection picture is a defective product), and return the inference result to the device side, and store the data to be inferred and the inference result, which can be used for iterative update of the AI model of the training platform.
[0061] like Figure 2 As shown, the AI reasoning system in the present invention belongs to a cloud-edge-end intelligent ecosystem. The cloud refers to the training center, that is, a collection of multiple factories, the edge refers to the reasoning platform, and the end refers to the device end, such as automatic detection and maintenance equipment.
[0062] Please refer to Figure 1 , Figure 2 and Figure 3 The reasoning platform of the embodiment of the present invention includes: a management module and at least one reasoning module. Among them:
[0063] The management module (also referred to as Main Agent) is used to obtain the AI model trained by the training platform and push the AI model to at least one of the reasoning modules; that is, the main functions of the management module include: communication interaction (interaction with the training platform and interaction with the reasoning module) and model management.
[0064] The inference module (also called inference agent or agent) is used to obtain the AI model pushed by the management module and the data to be inferred on the device side; use the AI model to infer the data to be inferred to obtain the inference result; return the inference result to the device side; store the data to be inferred, and the data to be inferred and the inference result can be used for iterative update of the AI model of the training platform. That is, the main functions of the inference module include receiving inference tasks and providing inference services.
[0065] In an embodiment of the present invention, the reasoning module and the management module may have a shared folder in a shared storage device (such as a shared network disk), the management module may store the AI model in the shared folder, and the reasoning module may pull the AI model from the shared folder.
[0066] In an embodiment of the present invention, the AI model can be applied to the field of automatic equipment detection and maintenance. The data to be inferred can be a detection picture of the equipment, and the inference result can be a detection picture with labeled information.
[0067] In an embodiment of the present invention, the inference platform obtains a uniformly trained AI model from the training platform, thereby realizing the sharing of the trained AI model and avoiding repeated training of the AI model. The AI model can be quickly implemented. At the same time, the inference platform shares its own data to be inferred and inference results with the training platform for iterative updating of the AI model, expanding the training data and thus improving the AI model.
[0068] In the embodiment of the present invention, the management module can be deployed in the cloud or locally, for example, it can be deployed in any PC that is connected to the training platform and each reasoning module network.
[0069] The inference module can be deployed in the cloud or locally. Optionally, the inference module can be deployed on a PC with a graphics processing unit (GPU) to improve computing power. When the inference module is deployed locally, the inference module and the device end do not need to be transmitted over the Internet, so the data to be inferred sent by the device end can achieve low-latency inference and improve processing speed. Figure 1 When the inference module is deployed locally, it can be deployed across a personal computer (PC) with the device side to provide computing power for inference, or it can be deployed on the PC where the device side is located, thereby further reducing latency.
[0070] In an embodiment of the present invention, multiple reasoning modules can be used to connect to different business demand scenarios. Each reasoning module can correspond to a device end, and the device end can include multiple types of device ends, such as a robotic process automation (RPA) system, an edge inspection system (EGIS) or an automatic basic product detection (Auto Cell Test, ACT) system. Taking RPA as an example, one application is an automatic operation and repair system for product defects, wherein the automatic operation and repair includes: 1) monitoring the PC information of the repair, capturing PC screenshots, and generating repair tasks; 2) receiving operation instructions, controlling the device through a keyboard, video or mouse (Keyboard Video Mouse, KVM), and performing product repair. The repaired PC and the RPA system are connected using KVM, and the RPA system communicates with the reasoning module through the network. The reasoning module uses an AI model to reason about the received screenshots, generates screenshots with product defects marked, and returns them to the RPA system, thereby realizing automatic detection and repair of the entire process, reducing manual re-judgment, and improving product yield and equipment utilization rate.
[0071] In some embodiments, the management module is optionally used to obtain the iteratively updated AI model of the training platform and push the iteratively updated AI model to the reasoning module; the reasoning module is used to update the local AI model using the iteratively updated AI model. Thus, the reasoning module can obtain the latest AI model.
[0072] Optionally, the management module may send a model update instruction to the reasoning module when the AI model is updated. The reasoning module receives the model update instruction sent by the management module and may pull the latest AI model through a shared folder with the management module.
[0073] In some embodiments, optionally, the inference module is used to store the data to be inferred in a shared storage device, and report the inference result to the management module, and the data of the shared storage device can be shared by the inference platform and the training platform; the management module is used to store the inference result reported by the inference module in a database. The training platform can obtain the data to be inferred from the shared storage device, and obtain the inference result from the database for iterative update of the AI model.
[0074] Of course, it should be noted that the reasoning module can also store the data to be inferred and the reasoning result in a shared storage device at the same time. The training platform can obtain the data to be inferred and the reasoning result from the shared storage device.
[0075] The role of shared storage is to provide data sharing between the training platform and the inference platform. The location where it is deployed needs to be interconnected with the training platform and the inference platform network, such as network attached storage (NAS, used for network expansion).
[0076] In some embodiments, the functions of the management module also include: configuring the information of the reasoning module, and user interface (UI) interaction, which is to interact with the user by providing a user interface. For example, the user can choose to configure the reasoning module, configure the model version, download the model, etc. through the UI.
[0077] The reasoning module can be an application (APP) installed on the device, which can be encapsulated as an executable .exe file. It can be started by the management module through instructions, and then download the AI model. When receiving the reasoning task on the device side, it can reason the reasoning task through the AI model and return the reasoning result to the device side.
[0078] In an embodiment of the present invention, the reasoning module can obtain the processing process of reasoning data by calling the algorithm software on the device side (such as AOI device algorithm software or third-party AI calling software), thereby assisting the AI model in completing reasoning.
[0079] (I) The following is an explanation of several main functions of the management module.
[0080] 1. Communication implementation:
[0081] 1.1 Communication between management module and reasoning module:
[0082] The management module and the reasoning module can communicate using the Transmission Control Protocol (TCP), the TCP client (Client) is the reasoning module, and the TCP server (Server) is the management module.
[0083] The reasoning module may perform at least one of the following operations through the first communication interface (the interface from the reasoning module to the management module):
[0084] Register with the management module and unbind from the management module when exiting;
[0085] Report the status to the management module, such as reporting the status once every first preset time (such as 1 minute). If the management module does not receive the normal status within a second preset time (such as 3 minutes), the inference module status is set to abnormal;
[0086] Report the reasoning results to the management module (the reasoning module processes the reasoning once and reports the reasoning results to the management module);
[0087] Get configuration from the management module.
[0088] The management module may perform at least one of the following operations through the second communication interface (an interface from the management module to the reasoning module):
[0089] Send model update instructions to the inference module;
[0090] Send configuration updates to the inference module.
[0091] 1.2 Communication between management module and training platform:
[0092] In some embodiments, the management module can be used as an Http Server and the training platform can be used as an Http Client. Through the interface, functions such as obtaining data to be inferred (such as detection images), inference results, AI model delivery, and heartbeat detection are completed.
[0093] 1) When the training platform obtains the data to be inferred (such as pictures) and / or the inference results from the management module, the inference module can upload the pictures to the corresponding File Transfer Protocol (FTP) directory (i.e., shared storage), and upload the inference results and the picture path to the management module at the same time. The database of the management module records the inference results and the picture path. The management module interacts with the training platform. The training platform sends HTTP picture transfer instructions to the management module. The management module generates a set of file list addresses to be uploaded in FTP. The management module sends the set of file list addresses to be uploaded to the training platform. The training platform downloads the data to be inferred (such as pictures) and / or the inference results from the FTP of the management module.
[0094] 2) Model delivery: The management module obtains the AI model from the training platform and defines the request and response instructions according to the Http protocol. When the training platform delivers the AI model to the management module, the management module can design the model file binding method: when a new AI model is delivered, a user interface with a list of model files to be bound pops up in the user interface of the AI model version information management. In this user interface, select to bind a certain AI model file delivered to the manually created model version information (when the binding operation occurs, the management module background will upload the model file to the FTP path associated with the model version information). If there is no corresponding model version information in the management module, it needs to be created manually first, so as to reduce the coupling with the training platform and reduce the data dependence on the training platform.
[0095] 3) Heartbeat detection function: The training platform adds a heartbeat detection request. The training platform sends a heartbeat to the management module every preset time (such as 30 seconds). Through the heartbeat detection request, it can monitor whether the management module and the cloud platform are connected, which is convenient for users to troubleshoot problems.
[0096] 1.3 The management module can also reserve an interface for communicating and interacting with external systems.
[0097] 2. Configuration Management:
[0098] In the embodiment of the present invention, the management module can configure the basic information required for the reasoning module registration, configure the defect group, product group and equipment group, and combine and bind the defect group, equipment group, product group and the reasoning module.
[0099] 3. Model management:
[0100] The management module can manage the AI models obtained from the training platform, and can also support the management of AI models uploaded locally.
[0101] 1) Model reception: Receive AI models issued by the training platform and receive AI models uploaded locally.
[0102] 2) Model upload: Upload the AI model to the training platform to realize the sharing of AI model.
[0103] 3) Model push: Push the AI model to the corresponding inference module. You can choose to push multiple inference modules. If multiple inference modules are installed on different devices, you can push them by device ID.
[0104] 4) Version management: Version management of AI models. The version information of AI models can be bound to necessary business attributes (such as device ID, inference module number, product group, site group, etc.).
[0105] 5) Version screening: screening of AI model versions.
[0106] 6) Configuration push: Update the corresponding configuration file to the corresponding inference module.
[0107] 7) Model verification: Manually confirm the availability of the selected AI model. You can generate a simulated service call message to verify whether the AI model is normally available (you can refer to the AI call service interface to make a service call request).
[0108] 4. Data writing
[0109] The management module provides a user-visible system interface for model configuration and management. The interface can present the following information: tables used for device management, model management, configuration management, and inference task management.
[0110] 1) Data storage: Write the inference results of the inference module into the local database (for example, including image name, image path, machine, time, etc.).
[0111] 2) Entering annotation information into the database: writing the verified annotation information (such as defect information) into the database.
[0112] 3) Exception records: Records include exception information from the reasoning module and the management module to facilitate cause investigation.
[0113] 4) Other data writing: Other necessary local or external interaction information is written into the database for recording.
[0114] 5. Image management
[0115] Management of data required for AI model training (i.e., the image information in the shared storage mentioned above).
[0116] 1) Platform interaction: Receive tasks initiated by the training platform and copy the corresponding images and annotation information to the specified path (NAS or other disk address).
[0117] 2) Image management: The inferenced images corresponding to the inference module can be copied to the specified path for management and deleted regularly.
[0118] (ii) Several main functions of the reasoning module are described below.
[0119] The inference module can be an application configured on the device that needs inference. The management module starts the inference module through instructions. The inference module downloads the AI model to complete the startup and returns the successful startup result to the management module. When an inference task appears on the device side, the inference task is sent to the inference module. After the inference module infers the inference task based on the AI model, it returns the inference result to the device side and reports it to the management module.
[0120] The reasoning module mainly includes the following functions: the function of the reasoning gateway, the communication between the reasoning module and the management module, the communication between the reasoning module and the device, model management, task reasoning, and information management.
[0121] 1. Inference Gateway
[0122] The inference gateway performs the following functions: receiving instructions for inference tasks; packaging and sending inference results; configuring switching instructions; gateway simulator; and handling communication exceptions.
[0123] 2. Communication between the inference module and the device
[0124] By configuring the IP and using the TCP protocol to define the interface, the inference module and the device can send and receive data to each other.
[0125] 3. Model management
[0126] Model management includes at least one of the following:
[0127] 1) After the AI model is downloaded, update the local AI model. If the local AI model is not running, replace it directly. If the local AI model is running, determine the current inference status and replace the local AI model in the idle state. If it is in inference, replace the local AI model after the inference is completed.
[0128] It should be noted that in addition to direct replacement, storing multiple versions of AI models is not excluded.
[0129] 2) Model service, model startup, model initialization, and model parameter configuration.
[0130] 3) Service verification, providing an inference test interface for external calls.
[0131] 4)Configuration update.
[0132] 5) Supports manual intervention and can manually start and stop the inference service.
[0133] 4. Task reasoning:
[0134] The inference module can implement scheduling and inference on the GPU. Its main functions are:
[0135] 1) Algorithm interface software development kit (SDK) encapsulation and implementation, including standard interfaces such as algorithm initialization, reasoning, model replacement, and service termination.
[0136] 2) The algorithm code library is plug-in-based and supports languages such as C++ and C#. Demo testing is available.
[0137] 3) Algorithms including reasoning services need to support common algorithms such as segmentation, classification, and detection.
[0138] 4) The inference module algorithm access supports C# and C++ formats, and the SDK supports these two languages.
[0139] 5) GPU scheduling and algorithm thread management of the device where the inference module is located.
[0140] 6) GPU management of the device where the inference module is located.
[0141] 7) Management of information such as model loading and model calling.
[0142] 8) Reasoning tasks, reasoning result judgment processing, and analysis of data to be reasoned.
[0143] 9) Encapsulation and conversion of reasoning results.
[0144] 10) Management of reasoning task status.
[0145] 5. Information Management
[0146] The management of the log files of the reasoning tasks can generate log files according to the predetermined naming rules, save them to the local device or the directory specified by the management module, realize regular automatic cleaning of logs, and upload the task reasoning logs to the management module.
[0147] When an exception occurs in reasoning, a record of the exception is saved and the exception log is uploaded to the management module.
[0148] The GPU information and the running status of the inference module are reported to the management module, which facilitates the management module interface viewing and abnormal troubleshooting management.
[0149] Save the images used for inference, save the inferred images to the specified path (configuration file), manage the image lifecycle, and clean up files and images regularly according to rules.
[0150] Please refer to Figure 8 , Figure 8 Schematic diagram of the self-loading process under the model of the embodiment of the present invention, from Figure 8 It can be seen that the algorithm personnel upload the algorithm container and operator package to the training platform (also known as the training cloud platform), and the operator remotely controls the screening of images (data to be inferred) on the management module and manually uploads them to the training platform. After completing data annotation, model training and model testing on the training platform, the model is published to the management module. The management module receives the model release message from the training platform, downloads the model and stores it in the shared network disk, and then updates the model version information. The management module notifies the corresponding reasoning module to update the model according to the model configuration module. When the local AI model is idle, the reasoning module actively pulls the updated AI model and replaces it.
[0151] like Fig. 9 The figure is a schematic diagram of the process of model calling by the reasoning module of an embodiment of the present invention. The reasoning module can call different AI models to provide differentiated extension functions.
[0152] like Fig.10 The figure shows a sample schematic diagram of the interface design of the inference module and the device end of the embodiment of the present invention. The interface implementation of the inference module and the device end can be implemented through the TCP / IP protocol. When the inference module is installed locally, the shared memory address and size can be directly passed for model call; when the inference module is installed across PCs, the image path can be passed or the network byte stream can be directly passed.
[0153] Please refer to Figure 4 The embodiment of the present invention further provides a training platform, including a training module, a model publishing module and an iterative update module, wherein:
[0154] The training module is used to obtain a training data set and train an AI model according to the training data set; the training module trains the uploaded operator (such as VGG, FAST RCN, YOLO, etc.) through the training data set, fills in hyperparameters (learning rate, batch sample size, optimizer parameters, and adjustable parameters of some loss functions, etc.), and generates a model.
[0155] The model publishing module is used to publish the trained AI model to the inference platform;
[0156] The iterative update module is used to obtain the data to be inferred and the inference results from the inference platform, and expand the training data set according to the data to be inferred and the inference results;
[0157] The training module is further used to train the AI model based on the expanded training data set to obtain an iteratively updated AI model;
[0158] The model publishing module is also used to send the iteratively updated AI model to the reasoning platform.
[0159] In an embodiment of the present invention, the model publishing module can notify the inference platform of the release of the AI model through the model publishing information, and the inference platform downloads the AI model from the training platform.
[0160] In an embodiment of the present invention, the training platform uniformly trains the AI model and sends it to the inference platform, thereby realizing the sharing of the trained AI model and avoiding repeated training of the AI model. The AI model can be quickly implemented. At the same time, the training platform obtains the data to be inferred and the inference results of the inference platform for iterative updating of the AI model, expanding the training data and thus improving the AI model.
[0161] In some embodiments, in addition to supporting the above-mentioned model training, the training platform can also support image data access and management, image data annotation (manual annotation), operator development, model management, model encapsulation, model services, etc. The training platform can support a variety of algorithm frameworks to meet the needs of different AI application scenarios. When the training platform is used, data, data sets, operators, models, applications, images, etc. will be formed, which can be shared in the sharing center of the sharing platform.
[0162] In an embodiment of the present invention, there may be multiple AI models trained in the training platform, each of which is connected to different business demand scenarios. Each AI model is independent and can be applied to different factories and different types of equipment.
[0163] In some embodiments, the training platform puts the obtained training data into a data pool, and divides the training data in the data pool into a training data set and a test data set. Since the data in the data pool will be expanded, the training data set and the test data set can also be divided into versions, for example, different versions are marked with information such as V1 and V2. When there is a test data set, the training module uses the test data set to generate a test report through offline testing, and the test report includes accuracy, recall rate, confusion matrix, etc. If the test results meet the expected results, the model publishing module performs model application encapsulation, selects the dependent environment required by the model, and distributes the model to the inference platform.
[0164] Please refer to Figure 5 , an embodiment of the present invention further provides a method for implementing AI reasoning, including:
[0165] Step S11: The inference platform obtains the AI model trained by the training platform and the data to be inferred on the device side;
[0166] In some embodiments, step S11 may be performed by a management module in the reasoning platform.
[0167] In some embodiments, the data to be inferred may be a detection picture.
[0168] In some embodiments, step S12 may be performed by an inference module in an inference platform.
[0169] In the embodiment of the present invention, for different device ends, the inference platform can obtain the data to be inferred on the device end in different ways. For example, the data to be inferred can be obtained through the storage path of the data to be inferred provided by the device end, or the data to be inferred sent by the receiving device end can be obtained, or the data to be inferred can be obtained from the shared memory of the device end.
[0170] Step S12: the reasoning platform uses the AI model to reason about the data to be reasoned, and obtains a reasoning result;
[0171] In some embodiments, step S12 may be performed by an inference module in an inference platform.
[0172] Step S13: the inference platform returns the inference result to the device end;
[0173] In some embodiments, step S13 may be performed by an inference module in an inference platform.
[0174] Step S14: The inference platform stores the data to be inferred and the inference results, and the data to be inferred and the inference results can be used for iterative update of the AI model of the training platform.
[0175] In some embodiments, in step S14, storing the data to be inferred may be performed by the inference module, and storing the inference result may be performed by the management module. Of course, this is not limited to this, and storing the data to be inferred and the inference result may also be performed by both the inference module or the management module.
[0176] In an embodiment of the present invention, the inference platform obtains a uniformly trained AI model from the training platform, thereby realizing the sharing of the trained AI model and avoiding repeated training of the AI model. The AI model can be quickly implemented. At the same time, the inference platform shares its own data to be inferred and inference results with the training platform for iterative updating of the AI model, expanding the training data and thus improving the AI model.
[0177] In some embodiments, optionally, the method for implementing AI reasoning further includes:
[0178] The inference platform obtains the AI model iteratively updated by the training platform;
[0179] The reasoning platform uses the iteratively updated AI model to update the local AI model.
[0180] In some embodiments, optionally, the inference platform stores the data to be inferred and the inference results, including: the inference platform stores the data to be inferred in a shared storage device, and stores the inference results in a database, and the data in the shared storage device can be shared by the inference platform and the training platform.
[0181] Please refer to Figure 6 , an embodiment of the present invention further provides a method for training an AI model, comprising:
[0182] Step S21: The training platform obtains a training data set;
[0183] Step S22: the training platform trains the AI model according to the training data set;
[0184] Step S23: The training platform publishes the trained AI model to the inference platform;
[0185] Step S24: the training platform obtains the data to be inferred and the inference result from the inference platform, and expands the training data set according to the data to be inferred and the inference result;
[0186] Step S25: the training platform trains the AI model based on the expanded training data set to obtain an iteratively updated AI model;
[0187] Step S26: The training platform publishes the iteratively updated AI model to the inference platform.
[0188] Please refer to Figure 7 , Figure 7 The following is a schematic diagram of the interaction process between the inference platform and the training platform of an embodiment of the present invention, wherein the process includes:
[0189] Model inference process:
[0190] Step 1.1: The inference module (i.e., the inference agent in the figure) obtains the data to be inferred (such as the detection image) on the device side.
[0191] In the embodiment of the present invention, for different device ends, the inference module can obtain the data to be inferred on the device end in different ways. For example, the detection image can be obtained through the image path provided by the device end, or the byte stream of the detection image sent by the receiving device end can be obtained, or the detection image can be obtained from the shared memory of the device end.
[0192] The byte stream may be a byte stream of an EGIS or RPA system, and the shared memory may be a shared memory of an ACT system.
[0193] Step 1.2: The reasoning module uses the AI model to reason about the data to be reasoned, obtains the reasoning result, and returns it to the device.
[0194] Step 1.3: The inference module stores the obtained detection images to a shared storage device.
[0195] Step 1.4: The reasoning module reports the reasoning results to the management module (Main Agent).
[0196] Step 1.5: The management module stores the inference results in the database.
[0197] Model iteration process:
[0198] Step 2.1: The training platform extracts the inference results from the database of the management module.
[0199] In the embodiment of the present invention, the inference result is a detection image with annotations.
[0200] Step 2.2: The training platform extracts the data to be inferred (in this embodiment, the detection image) from the shared storage device.
[0201] The training platform can manually annotate the extracted test images for iterative training of the AI model.
[0202] Step 2.3: The training platform expands the training data set based on the obtained inference results and the data to be inferred, and uses the expanded training data set to train the AI model to obtain an iteratively updated AI model;
[0203] Step 2.4: The training platform sends the iteratively updated AI model to the management module;
[0204] Step 2.5: The management module pushes the iteratively updated AI model to the inference module;
[0205] Step 2.6: The inference module replaces the locally stored AI model with the iteratively updated AI model.
[0206] Step 2.7: The management module performs reasoning verification on the iteratively updated AI model;
[0207] Step 2.8: After the inference verification is successful, the management module pushes the AI model after inference verification to the inference module.
[0208] Step 2.5 and step 2.6, and step 2.7 and step 2.8, can be performed either or both.
[0209] The above solution of the embodiment of the present invention is described below by taking the device end as an RPA system as an example. In this implementation case, the automatic detection and maintenance equipment scenario of a factory is taken as the research object.
[0210] like Fig.11 The software architecture diagram of the RPA system of the embodiment of the present invention is shown. The RPA system includes the operation layer (Windows system), data persistence layer (MySQL database, local storage), data access layer (message queue, Entity Framework, event monitoring), business layer (RPA process includes collection, proxy operation, reasoning, alarm, task distribution and execution of Quartz RPA task scheduling, KVM device management, system business), application layer (WPF page framework, PRISM open source framework, MVVM architecture) and presentation layer (RPA client and re-judgment client). The RPA system and the repair PC are connected through KVM, the RPA system and the reasoning platform are connected through the network, and the reasoning platform and the training platform are connected through the network.
[0211] like Fig.12 The figure shows a schematic diagram of the hardware architecture of the RPA system of an embodiment of the present invention. ROS ROOM is the abbreviation of Remote Operating System, which means a remote operating system, commonly known as an intelligent manufacturing room, which can remotely connect to the equipment in the line to achieve the purpose of completing the corresponding operation. KVM RX, IP KVM, RPA PC, ADC PC, and proxy operation station are deployed inside ROS ROOM. The group's single company computer room server cluster deploys a training platform (application server and training server). Central servers, cloud management, and display interfaces (dashboards) are deployed at the group level.
[0212] like Fig.13 The figure shows a flow chart of cloud-edge-end collaboration to implement AI reasoning in an RPA case according to an embodiment of the present invention.
[0213] like Fig.14 The figure shows a schematic diagram of a matrix remote layout (many-to-many, 1-to-many) implemented by a process control center (PCC, providing more comprehensive remote monitoring and remote operation capabilities, mainly used for global monitoring and remote operation of equipment) technology in an embodiment of the present invention to improve the detection capacity. If the existing ROS network is stable (for example, the optical fiber is not attenuated, etc.), and the existing optical fiber KVM inventory is sufficient, then a KVM is added to the existing remote basis to realize the matrix layout. If the annual BX investment on ROS is high (network + KVM related maintenance costs are high), the device end replaces TX with IP KVM to achieve domestic replacement, the remote end replaces RX with PC, and the network replaces the network cable & optical fiber according to actual needs.
[0214] The RPA system uses RPA technology to achieve automatic repair and save manpower. If there is no Auto Repair and Review PC, add a client PC to install the RPA system, and the number of units is configured according to the number of Repair machines. If there is AutoRepair and Review PC, but there are defects, implement the RPA function for the Repair device and Review PC respectively, and the number of units is configured according to the number of Repair machines and Review PCs.
[0215] The RPA system performs reasoning through the reasoning module, reducing manual secondary review and improving the yield rate.
[0216] like Fig.15 The schematic diagram of the algorithm architecture of the RPA case in the embodiment of the present invention is shown, including:
[0217] Acquiring capabilities: Install the inference module, access the network, and obtain cloud interaction capabilities.
[0218] Training request: Manually trigger the import of training data (training platform front end).
[0219] Completion notification: Training is completed, and the inference platform receives training platform information and model links.
[0220] Update model: Download model file and replace the model.
[0221] In the embodiment of the present invention, RPA is equivalent to a small tool, which transmits node data to the reasoning module. After the reasoning module reasoning, the RPA process continues. If the shared memory method is adopted, RPA and the reasoning module are on the same machine, and reasoning a picture of several hundred KB takes about 100 milliseconds to implement the scenario, which is very fast.
[0222] Please refer to Fig.16 An embodiment of the present invention further provides an electronic device 10, comprising a processor 11, a memory 12, and a computer program stored in the memory 12 and executable on the processor 11. When the computer program is executed by the processor 11, the various processes of the above-mentioned AI reasoning implementation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0223] Please refer to Fig.17 The embodiment of the present invention further provides a server 20, including a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the computer program is executed by the processor 21, the various processes of the above-mentioned AI model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0224] The embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the above-mentioned AI reasoning implementation method or AI model training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0225] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 5 or Figure 6 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.
[0226] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0227] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0228] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.
Claims
1. A method for implementing AI reasoning, characterized in that: include: The inference platform obtains the AI model trained by the training platform and the data to be inferred on the device side; The reasoning platform uses the AI model to reason about the data to be reasoned, and obtains a reasoning result; The inference platform returns the inference result to the device end; The inference platform stores the data to be inferred and the inference results, and the data to be inferred and the inference results can be used for iterative updating of the AI model of the training platform.
2. The method according to claim 1, characterized in that: Also includes: The inference platform obtains the AI model iteratively updated by the training platform; The reasoning platform uses the iteratively updated AI model to update the local AI model.
3. The method according to claim 1, characterized in that The reasoning platform stores the data to be reasoned and the reasoning results, including: The reasoning platform stores the data to be reasoned in a shared storage device, and stores the reasoning result in a database. The data in the shared storage device can be shared by the reasoning platform and the training platform.
4. A method for training an AI model, characterized in that: include: The training platform obtains the training data set; The training platform trains the AI model according to the training data set; The training platform publishes the trained AI model to the inference platform; The training platform obtains the data to be inferred and the inference results from the inference platform, and expands the training data set according to the data to be inferred and the inference results; The training platform trains the AI model based on the expanded training data set to obtain an iteratively updated AI model; The training platform publishes the iteratively updated AI model to the inference platform.
5. An inference platform, characterized in that: include: a management module and at least one reasoning module; The management module is used to obtain the AI model trained by the training platform and push the AI model to at least one of the reasoning modules; The inference module is used to obtain the AI model pushed by the management module and the data to be inferred on the device side; use the AI model to infer the data to be inferred to obtain an inference result; return the inference result to the device side; store the data to be inferred, and the data to be inferred and the inference result can be used for iterative update of the AI model of the training platform.
6. The reasoning platform according to claim 5, characterized in that: The management module is used to obtain the iteratively updated AI model of the training platform and push the iteratively updated AI model to the reasoning module; The reasoning module is used to update the local AI model using the iteratively updated AI model.
7. The reasoning platform according to claim 5, characterized in that: The inference module is used to store the data to be inferred in a shared storage device, and report the inference result to the management module, and the data of the shared storage device can be shared by the inference platform and the training platform; The management module is used to store the reasoning result reported by the reasoning module in a database.
8. A training platform, characterized in that: It includes training module, model publishing module and iterative update module, among which: The training module is used to obtain a training data set; and train an AI model according to the training data set; The model publishing module is used to publish the trained AI model to the inference platform; The iterative update module is used to obtain the data to be inferred and the inference results from the inference platform, and expand the training data set according to the data to be inferred and the inference results; The training module is further used to train the AI model based on the expanded training data set to obtain an iteratively updated AI model; The model publishing module is also used to send the iteratively updated AI model to the reasoning platform.
9. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the steps of the method for implementing AI reasoning as described in any one of claims 1 to 3 are implemented.
10. A server, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the AI model training method as described in claim 4.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for implementing AI reasoning as described in any one of claims 1 to 3; or, when executed by a processor, implements the steps of the method for training an AI model as described in claim 4.
12. A computer program product, characterized in that It includes computer instructions, which, when executed by a processor, implement the steps of the method for implementing AI reasoning as described in any one of claims 1 to 3, or, when executed by a processor, implement the steps of the method for training an AI model as described in claim 4.