Information interaction method and device, computing equipment and storage medium

CN118923096APending Publication Date: 2024-11-08BOE TECHNOLOGY GROUP CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380007834.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-02-17
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The traditional image classification method relies on manual detection, which has low efficiency and insufficient accuracy. In addition, there are equipment, parameters, operations and environmental interference in the production and manufacturing process, which makes it difficult to identify defective products and affects the quality of the final product.

Method used

By designing information interaction methods and devices for image classification systems, real-time, accurate, and stable information interaction between the image management system and the image classification system is realized, and AI technology is used to automatically identify and classify images acquired by AOI equipment, reducing manual intervention. .

Benefits of technology

It improves the efficiency and accuracy of image detection, reduces the consumption of human resources, ensures the improvement of product quality, and realizes the timely identification and correction of defective products.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118923096A_ABST
    Figure CN118923096A_ABST
Patent Text Reader

Abstract

The invention provides an information interaction method for an image classification system, and the method comprises the steps: transmitting a connection establishment message to an image management system, the connection establishment message comprising information indicating the identity of the image classification system; in response to the received confirmation of the connection establishment message from the image management system, establishing an information interaction connection with the image management system; in response to receiving a connection test message from the image management system, an acknowledgement of the connection test message is sent to the image management system. The image classification system is configured to identify the image allocated by the image management system to determine a category of the allocated image, and the image management system is configured to receive the image from the image acquisition device and to allocate the received image.
Need to check novelty before this filing date? Find Prior Art

Description

Information interaction method and device, computing device, and storage medium Technical Field

[0001] The present disclosure relates to the field of communication technology, and in particular, to an information interaction method and device for an image classification system, an information interaction method and device for an image management system, an image processing system, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] During the manufacturing process of products, such as screens, problems with equipment, parameters, operation, and environmental interference can lead to products failing to meet certain process requirements or even exhibiting defects. To promptly detect these issues and improve final product quality, products can be inspected after each process step to promptly identify defective products and implement timely corrections and improvements. For example, automated optical inspection (AOI) equipment can capture images of products, which are then inspected, identified, and classified according to process requirements. Traditional classification methods rely primarily on manual inspection. Generally, images captured by AOI equipment are uploaded to an image management system, such as a test management system (TMS). This system presents the images to operators, who then manually classify each image, which is then reviewed by engineers. This process not only consumes significant manpower and time, but also requires the operator's expertise and sustained focus. Consequently, manual inspection methods often suffer from low efficiency and inaccuracy.

[0003] In recent years, with the rapid development of artificial intelligence (AI), particularly the breakthroughs in image processing methods such as deep learning, across various fields, the use of AI to solve practical problems in manufacturing has become a trend. Designing specialized image classification systems for product manufacturing processes, such as Automatic Defect Classification (ADC) systems, leveraging AI to automatically inspect product images captured by AOI equipment can help improve inspection efficiency and accuracy. However, achieving automated inspection with image classification systems requires real-time, accurate, and stable information exchange between the image classification system and existing image management systems.

[0004] Summary of the Invention

[0005] In view of this, the present disclosure provides an information interaction method and device for an image classification system, an information interaction method and device for an image management system, an image processing system, a computing device, a computer-readable storage medium and a computer program product, which can alleviate, reduce or even eliminate the above-mentioned problems.

[0006] According to one aspect of the present disclosure, there is provided an information interaction method for an image classification system, wherein the image classification system is configured to identify images assigned by an image management system to determine the category of the assigned images, and the image management system is configured to receive images from an image acquisition device and assign the received images, the method comprising: sending a connection establishment message to the image management system, the connection establishment message including information indicating the identity of the image classification system; in response to receiving a confirmation of the connection establishment message from the image management system, establishing an information interaction connection with the image management system; and in response to receiving a connection test message from the image management system, sending a confirmation of the connection test message to the image management system.

[0007] In some embodiments, the above method also includes: receiving a product online operation, which is used to launch a target model corresponding to the target product; sending a first switch synchronization message to the image management system, which includes product information related to the target product and a switch status indicating that the target model has been launched.

[0008] In some embodiments, the above method also includes: receiving an image allocation message from an image management system, the image allocation message is used to allocate a target image related to a target product to an image classification system, wherein the image allocation message is sent in response to a target model related to the target product being in an available state; based on the image allocation message, obtaining the target image; inputting the target image into the target model; based on the output of the target model, feeding back the image classification result to the image management system, the image classification result including at least one of the following items: image category, confidence corresponding to the image category, and information indicating whether to adopt the classification result.

[0009] In some embodiments, the image allocation message includes information related to the storage path of the target image of the target product, and, based on the image allocation message, obtaining the target image includes: obtaining the storage path of the target image based on the image allocation message; and reading the target image from the image storage according to the storage path.

[0010] In some embodiments, the image allocation message includes the file name of the image path file, and, based on the image allocation message, obtaining the storage path of the target image includes: obtaining the image path file according to the file name of the image path file and a first preset path, the image path file including the storage path of the target image; and obtaining the storage path of the target image by parsing the image path file.

[0011] In some embodiments, feeding back the image classification result to the image management system includes: writing the image classification result into a classification result file via a second preset path; and sending an image classification feedback message to the image management system, where the image classification feedback message includes the file name of the classification result file.

[0012] In some embodiments, the above method also includes: receiving a product offline operation, where the product online operation is used to launch a target model corresponding to the target product; sending a second switch synchronization message to the image management system, where the second switch synchronization message includes product information related to the target product and a switch status indicating that the target model has been offline.

[0013] In some embodiments, the above method also includes: in response to receiving a review result message from the image management system, the review result message includes information related to the manual review result of the target image, and storing the manual review result for evaluation and / or iterative training of the target model.

[0014] In some embodiments, each of the connection establishment message, connection test message, first switch synchronization message, second switch synchronization message, image allocation message, image classification feedback message, and re-inspection result message includes a corresponding message header, and the message header includes one or more of the following items: message node type, message node number, message name, response type, number of text items, and message number.

[0015] According to another aspect of the present disclosure, an information interaction method for an image management system is provided, wherein the image management system is configured to receive images from an image acquisition device and assign the received images, the method comprising: in response to receiving a connection establishment message from an image classification system, sending a confirmation of the connection establishment message to the image classification system to establish an information interaction connection with the image classification system, wherein the connection establishment message includes information indicating the identity of the image classification system, the image classification system is configured to identify the images assigned by the image management system to determine the category of the assigned images; sending a connection test message to the image classification system; and receiving a confirmation of the connection test message from the image classification system.

[0016] In some embodiments, the above method also includes: in response to receiving a first switch synchronization message from the image classification system, the first switch synchronization message includes product information related to the target product and a switch status indicating that the target model is online, updating the availability of the target model related to the target product to an available status.

[0017] In some embodiments, the above method also includes: in response to receiving a second switch synchronization message from the image classification system, the second switch synchronization message includes product information related to the target product and a switch status indicating that the target model has been offline, updating the availability of the target model related to the target product to an unavailable state.

[0018] In some embodiments, the above method also includes: in response to receiving a target image related to the target product from the image acquisition device, and the target model related to the target product is in an available state, assigning the target image to the image classification system; obtaining the image classification result fed back by the image classification system, the image classification result including at least one of the following items: image category, confidence corresponding to the image category, and information indicating whether to adopt the classification result.

[0019] In some embodiments, allocating a target image to an image classification system includes: writing a storage path of the target image to an image path file via a first preset path; and sending an image allocation message to the image classification system, the image allocation message including a file name of the image path file.

[0020] In some embodiments, obtaining the image classification results fed back by the image classification system includes: receiving an image classification feedback message from the image classification system, the image classification feedback message includes the file name of the classification result file, and the classification result file includes the image classification result; obtaining the classification result file according to the file name of the classification result file and a second preset path; and obtaining the image classification result by parsing the classification result file.

[0021] In some embodiments, the above method further includes: in response to a confidence level corresponding to the image category being greater than a preset confidence threshold, determining a classification result of the target image based on the image category.

[0022] In some embodiments, the above method also includes: presenting the target image to the user in response to the confidence corresponding to the image category being less than a preset confidence threshold; receiving a manual review result from the user; and determining a classification result of the target image based on the manual review result.

[0023] In some embodiments, the above method further includes: sending a re-inspection result message to the image classification system, where the re-inspection result message includes information related to the manual re-inspection result of the target image.

[0024] In some embodiments, the above method also includes: in response to receiving a target image related to a target product from an image acquisition device, and the target model related to the target product is in an unavailable state, presenting the target image to a user; receiving a manual classification result from the user; and determining a classification result of the target image based on the manual classification result.

[0025] In some embodiments, each of the connection establishment message, connection test message, first switch synchronization message, second switch synchronization message, image allocation message, image classification feedback message, and re-inspection result message includes a corresponding message header, and the message header includes one or more of the following items: message node type, message node number, message name, response type, number of text items, and message number.

[0026] According to another aspect of the present disclosure, an information interaction device for an image classification system is provided, wherein the image classification system is configured to identify images assigned by an image management system to determine the category of the assigned images, and the image management system is configured to receive images from an image acquisition device, and store and assign the received images. The method includes: a first sending module, configured to: send a connection establishment message to the image management system, the connection establishment message including information indicating the identity of the image classification system; a receiving module, configured to: establish an information interaction connection with the image management system in response to receiving a confirmation of the connection establishment message from the image management system; and a second sending module, configured to: send a confirmation of the connection test message to the image management system in response to receiving a connection test message from the image management system.

[0027] According to another aspect of the present disclosure, an information interaction device for an image management system is provided, wherein the image management system is configured to receive images from an image acquisition device, and store and distribute the received images, and the method includes: a first sending module, configured to: in response to receiving a connection establishment message from an image classification system, send a confirmation of the connection establishment message to the image classification system to establish an information interaction connection with the image classification system, wherein the connection establishment message includes information indicating the identity of the image classification system, and the image classification system is configured to identify the image distributed by the image management system to determine the category of the distributed image; a second sending module, configured to: send a connection test message to the image classification system; and a receiving module, configured to: receive a confirmation of the connection test message from the image classification system.

[0028] According to another aspect of the present disclosure, there is provided an image processing system, comprising an image management system and an image classification system. The image management system comprises: a first interface configured to receive a connection establishment message from the image classification system, send a confirmation of the connection establishment message and a connection test message to the image classification system, receive images from an image acquisition device, and assign the received images, wherein the connection establishment message includes information indicating the identity of the image classification system; an image memory configured to store images from the image acquisition device. The image classification system comprises: a second interface configured to send a connection establishment message to the image management system, receive a confirmation of the connection establishment message and a connection test message from the image management system, and send a confirmation of the connection test message to the image management system; and an image classification model configured to identify images assigned by the image management system to determine the category of the assigned images.

[0029] In some embodiments, the first interface of the image management system is further configured to: receive a first switch synchronization message from the image classification system, the first switch synchronization message including product information related to the target product and a switch status indicating that the target model has been launched, and update the availability of the target model related to the target product to an available state according to the first switch synchronization message. In addition, the image classification system further includes: a user interface configured to: receive a product launch operation, the product launch operation being used to launch the target model corresponding to the target product; a database configured to: update the switch status of the target model corresponding to the target product to "already launched" according to the product launch operation, wherein the second interface is further configured to: send the first switch synchronization message to the image management system in response to the switch status of the target model corresponding to the target product in the database being updated to "already launched".

[0030] In some embodiments, the image management system further comprises: a file storage configured to store an image path file, the image path file including the storage path of a target image of a target product; wherein the first interface is further configured to, in response to a target model associated with the target product being in an available state, send an image allocation message to the image classification system, the image allocation message including the file name of the image path file; and wherein the image classification system further comprises: a shared memory configured to store the image path file downloaded from the file storage of the image management system and the target image downloaded from the image storage of the image management system; wherein the second interface is further configured to receive the image allocation message, download the image path file from the file storage based on the file name of the image path file, obtain the storage path of the target image by parsing the image path file, and download the target image from the image storage based on the storage path.

[0031] In some embodiments, the image classification system further includes: a task queue configured to store image classification tasks created based on a target image; a task management device configured to send the image classification tasks in the task queue to an image classification model and receive image classification results from the image classification model; wherein the database is further configured to store image information related to the target image and the image classification results; the second interface is further configured to upload a classification result file to a file storage device and send an image classification feedback message to the image management system, wherein the classification result file is generated based on the image classification result, and the image classification feedback message includes the file name of the classification result file. Furthermore, wherein the first interface in the image management system is further configured to receive the image classification feedback message, obtain the image classification result based on the image classification feedback message, and send a confirmation of the image classification feedback message to the image classification system.

[0032] In some embodiments, the image management system further includes a manual determination device, and the first interface is further configured to: in response to a confidence level corresponding to an image category being less than a preset confidence threshold, present a target image to a user via the manual determination device, receive a manual review result from the user, and send a review result message to the image classification system, the review result message including information related to the manual review result of the target image. Furthermore, the second interface in the image classification system is further configured to: receive the review result message, store the manual review result in a database based on the review result message, and send a confirmation of the review result message to the image management system.

[0033] According to another aspect of the present disclosure, a computing device is provided, comprising: a memory configured to store computer-executable instructions; and a processor configured to perform the method provided according to various embodiments of the aforementioned aspects when the computer-executable instructions are executed by the processor.

[0034] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions. When the computer-executable instructions are executed, the methods provided in the various embodiments according to the aforementioned aspects are performed.

[0035] According to another aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the method provided according to various embodiments of the aforementioned aspects.

[0036] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0038] FIG1 schematically illustrates an example of an image processing scenario according to some embodiments of the present disclosure;

[0039] FIG2 schematically shows an example flow chart of an information interaction method for an image classification system according to some embodiments of the present disclosure;

[0040] FIG3 schematically illustrates an example timing diagram of a process for establishing an information interaction connection according to some embodiments of the present disclosure;

[0041] FIG4 schematically illustrates an example format of a message header for inter-system information interaction according to some embodiments of the present disclosure;

[0042] 5A and 5B schematically illustrate example formats of message bodies used in a process of establishing an information interaction connection according to some embodiments of the present disclosure;

[0043] FIG6 schematically illustrates an example timing diagram for a switch data synchronization process and an image distribution process according to some embodiments of the present disclosure;

[0044] FIG7 schematically illustrates an example format of a message body used in a switch data synchronization process according to some embodiments of the present disclosure;

[0045] FIG8 schematically illustrates an example timing diagram for an image classification process according to some embodiments of the present disclosure;

[0046] FIG9 schematically illustrates an example format of a message body used in an image classification process according to some embodiments of the present disclosure;

[0047] FIG10 schematically illustrates an example timing diagram for a manual review process according to some embodiments of the present disclosure;

[0048] FIG11 schematically shows an example flow chart of an information interaction method for an image management system according to some embodiments of the present disclosure;

[0049] FIG12A schematically illustrates an example block diagram of an image processing system according to some embodiments of the present disclosure;

[0050] FIG12B schematically illustrates an example timing diagram for an image processing scenario according to some embodiments of the present disclosure;

[0051] FIG13 schematically shows an example block diagram of an information interaction device for an image classification system according to some embodiments of the present disclosure;

[0052] FIG14 schematically shows an example block diagram of an information interaction device for an image management system according to some embodiments of the present disclosure;

[0053] FIG15 schematically illustrates an example block diagram of a computing device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0054] FIG1 schematically illustrates an example of an image processing scenario 100 according to some embodiments of the present disclosure. As shown in FIG1 , the scenario 100 may include an image acquisition device 110 , an image management system 120 , a manual determination module 130 , and an image classification system 140 .

[0055] The image acquisition device 110 can be used to acquire images to be classified. For example, it can acquire images of products to be inspected so as to detect possible defects or undesirable conditions of the products based on the acquired images. Exemplarily, the image acquisition device 110 can be arranged at one or more stations in the product assembly line. A station can refer to a process node in the product assembly line where one or a group of devices complete a certain production process, or a defect detection node used to perform defect detection, or other types of physical or virtual nodes. Exemplarily, for a product defect detection scenario, the image acquisition device 110 can be the automatic optical inspection (AOI) equipment mentioned above, which can generate an image description of product details through optical means, thereby providing a basis for judging the process status and undesirable conditions of the product. In addition, the image acquisition device 110 can also be other cameras, cameras, or other sensing devices with image acquisition capabilities. The image acquisition device 110 can send the acquired images to the image management system 120, for example, to the image storage device therein.

[0056] The image management system 120 can receive images from the image acquisition device 110 and store and manage these images. Exemplarily, the image management system 120 can assign each image to the manual judgment device 130 or the image classification system 140 according to the preset logic, so as to detect the image to determine the defect or defect type of the relevant product. According to the determined defect or defect type, the relevant product can be circulated and repaired, and the relevant production line equipment can be optionally debugged. Exemplarily, for the product defect detection scenario, the image management system 120 can be the test management system (TMS) mentioned above, which can be responsible for the image management and test process management of the product test. Exemplarily, the image management system can include, for example, a file processing module, an image processing module, an interface module for information interaction with the image acquisition device 110 and the image classification system 140, etc.

[0057] The manual judgment device 130 can present the images assigned by the image management system 120 to the user (such as an operator), obtain the user's judgment result, and then feed back the judgment result to the image management system 120. Exemplarily, the image management system 120 can assign some images for which the image classification system 140 cannot give a classification result to the manual judgment system 130 for manual judgment. Exemplarily, in the related art, the manual judgment device 130 can be integrated with the image management system 120. For example, in a traditional product defect detection system, all images are judged manually, so the manual judgment device 130 can exist as a part of the image management system 120, and all images received by the image management system 120 can be handed over to manual judgment through the manual judgment device 130. Exemplarily, for a product defect detection scenario, the image management system 120 can be a JPC (Judgement Personal Computer) device.

[0058] The image classification system 140 can receive the image assigned by the image management system 120 and call the corresponding AI (Artificial Intelligence) algorithm model to identify the image to determine the category of the image, such as the defect or defect type, and feed the determined category back to the image management system 120. Exemplarily, for the product defect detection scenario, the image classification system 140 can be the ADC system mentioned above. Exemplarily, the image classification system 140 can include an AI algorithm module, a file storage module, a database module, a task queue module, an interface module for information interaction with the image management system 120, etc. The AI ​​algorithm module can include, for example, an AI model cluster, wherein each AI model can correspond to one or several products, and can correspond to one or several sites. Each AI model can be pre-trained using sample data, and can receive image input and output data such as the category corresponding to the image and the corresponding confidence level. Optionally, the AI ​​model can be a feedforward neural network (FNN) model, a convolutional neural network (CNN) model, a target detection convolutional neural network (faster-RCNN) model, a recurrent neural network (RNN) model, or a generative adversarial network (GAN) model, but is not limited thereto. Other neural network models known to those skilled in the art may also be used.

[0059] As previously described, after image acquisition device 110 captures product images, they are uploaded to image management system 120, which then stores, distributes, and summarizes the classification results. Therefore, to automatically classify images using image classification system 140, it is necessary to achieve real-time, accurate, and stable information exchange and synchronization between the image classification system and the image management system. This involves, for example, the exchange of image data, the exchange of classification result data, and the synchronization of model information corresponding to different products and sites. This prevents information anomalies such as information confusion and loss caused by unstable information exchange connections, as well as issues such as missed image inspections caused by information anomalies, thereby preventing adverse effects or losses on the production line.

[0060] In this disclosure, "product" can refer to any product object in different process steps on a screen production line. Different products can be distinguished by information such as product type, product ID, and site number. Alternatively, depending on the specific application scenario, "product" can also refer to other types of products. In other words, the concepts of this disclosure can also be used for image classification of products other than screens to achieve defect detection or other inspection purposes.

[0061] FIG2 schematically shows an example flow chart of an information interaction method 200 for an image classification system according to some embodiments of the present disclosure. Exemplarily, the information interaction method 200 may be performed by the image classification system 140 shown in FIG1 . In the information interaction method 200, the image classification system may refer to a system configured to identify images assigned by an image management system to determine the category of the assigned images, and the image management system may refer to a system configured to receive images from an image acquisition device and assign the received images. Each of the image classification system and the image management system may include one or more devices, each of which may perform corresponding functions and be combined together to implement image classification and image management functions.

[0062] In step 210, a connection establishment message may be sent to the image management system, and the connection establishment message may include information indicating the identity of the image classification system. Exemplarily, the information indicating the identity of the image classification system may include information such as a node type and a node number, wherein the node type may indicate that the message sender is an image classification system, and the node number may indicate the number of the image classification system (for example, there may be one or more image classification systems). Alternatively, depending on the specific application scenario, other types of information such as the name of the image classification system, an IP address (Internet Protocol Address), etc. may be used as information indicating the identity of the image classification system, as long as it can distinguish one image classification system from other image classification systems and / or other devices. Exemplarily, in addition to the information indicating the identity of the image classification system, the connection establishment message may also include various information such as message type information indicating that the message is a connection establishment message, response type information indicating a response type, text quantity information indicating the text quantity, and a message number for identifying the message. For example, after the image classification system is started or restarted, it can automatically send the connection establishment message to the image management system according to a preset program. For example, the image classification system may pre-store the port address information of the image management system and send the connection establishment message to the pre-stored port address. Alternatively, depending on specific application requirements, an administrator can manually trigger the connection establishment process between the image classification system and the image management system, thereby triggering the sending of the connection establishment message.

[0063] In step 220, in response to receiving a confirmation of the connection establishment message from the image management system, an information interaction connection with the image management system may be established. Exemplarily, after receiving the connection establishment message from the image classification system, the image management system may send a confirmation message such as Ack, and store address information related to the image classification system, etc., so as to subsequently send messages to the image classification system through the established connection. The established connection can be identified by a pair of addresses, and after the connection is established, the image management system and the image classification system will continue to listen for messages from each other's addresses. Exemplarily, after receiving the confirmation of the connection establishment message from the image management system, the image classification system may consider that the connection establishment is complete, and start to listen for messages from the image management system until the system is shut down or restarted, or the connection is disconnected for other reasons.

[0064] In step 230, in response to receiving a connection test message from the image management system, a confirmation of the connection test message can be sent to the image management system. For example, after the image management system sends a confirmation of the connection establishment message to the image classification system, the two parties can simultaneously establish a connection between them. Subsequently, to ensure the accuracy and stability of the connection establishment, the image management system can send a connection test message to the image classification system. After receiving the connection test message, the image classification system can send a confirmation of the connection test message to the image management system. Thus, the image management system can confirm that the message it sent can be properly received by the image classification system, thereby avoiding message loss during subsequent data communication. For example, the connection test message can include one or more of the following: information indicating the identity of the image management system (such as the node type, node number, node name, IP address, and other types of information mentioned above), message type information indicating that the message is a connection test message, response type information indicating the response type, body quantity information indicating the number of body messages, and a message number used to identify the message. Furthermore, optionally, depending on specific application requirements, the connection test message can also include other data used to test the connection status.

[0065] Through the above method 200, with the help of connection establishment messages, connection test messages and corresponding confirmation messages, a stable and reliable information interaction connection can be established between the image classification system and the image management system, thereby helping to achieve subsequent stable, accurate and real-time information interaction between the two systems.

[0066] Figure 3 schematically illustrates an example timing diagram of a process 300 for establishing an information exchange connection according to some embodiments of the present disclosure. Schematically, Figure 3 illustrates the process of establishing an information exchange connection between an image classification system and an image management system, using the ADC system and the TMS system as examples. As shown in the figure, the connection between the two can be established using their respective communication interfaces, such as an ADC gateway in the ADC system and a TMS interface in the TMS system. Alternatively, other communication interfaces can be used depending on specific application requirements. As shown in the figure, the ADC gateway can first send a connection establishment message, such as a CE (Communication Establishment) message, to the TMS system (the address of the TMS interface) to submit a connection establishment request to the TMS. After receiving the CE message, the TMS system can send an ACK message back to the ADC system (the address of the ADC gateway) via the TMS interface to indicate confirmation of the CE message. The ADC and TMS then establish a connection, recording each other's addresses and beginning to monitor each other's messages. To ensure the accuracy and stability of the connection establishment, the TMS system can continue to send connection test messages, such as LT (Link Test) messages, and wait for feedback from the ADC. When the ADC returns an ACK message, it can be considered that the ADC can normally receive the message sent by the TMS, and thus it can be considered that the connection between the two is established correctly, and various subsequent data interaction tasks can be performed on the connection.

[0067] Schematically, FIG4 shows an example format of a general message header for inter-system information interaction according to some embodiments of the present disclosure. As shown in FIG4 , the general header may include six items, namely, node type, node ID, message name, response type, number of body items, and message ID. The node type may indicate the type of the message sender, such as the ADC, TMS, and JPC mentioned above. The node ID may indicate the number of the message sender within the type, such as ADC01, ADC02, and TMS01. The message name may indicate the name of the current message, such as the CE message and LT message mentioned above, which may inform the message receiver of the type of message the current message is, so that the receiver may process the received message according to a pre-agreed protocol. The response type may indicate the response type of the current message, for example, 0 may indicate that the current message is a request message, 1 may indicate that the current message is an acknowledgment message (Ack message), and 2 may indicate that the current message is a non-acknowledgment message (Nack message). The number of body items may indicate the number of items in the body of the current message. The message ID may indicate a unique number of the message, so as to uniquely identify the message. The message ID may be composed of a combination of the node ID and the sending time, where the sending time may be accurate to milliseconds. It is understood that the header shown in Figure 4 can be used in various messages between the ADC system and the TMS system, such as the connection establishment message and connection test message mentioned above, as well as the switch synchronization message, image allocation message, image classification feedback message, recheck result message, and various response messages mentioned below, to achieve standardized data exchange between the two that complies with preset protocol requirements. For example, for an Ack message, when one party receives a message from the other party, it can add the header of the received message as the body after its header to return an Ack message. This allows the other party to accurately determine which message the Ack message is acknowledging upon receiving it. It should be understood that the header format shown in Figure 4 is exemplary and can be modified or redesigned based on actual needs.

[0068] 5A and 5B also schematically illustrate example formats of message bodies used in a process of establishing an information interaction connection according to some embodiments of the present disclosure.

[0069] As shown in Figure 5A , a connection establishment message, such as a CE message, can be formatted with only a header and no body. Its header can adopt the format shown in Figure 4 . For a CE message, the request can be a default ACK message. As shown in Figure 5A , an example of a connection establishment message is shown: ADC^ADC01^CE^0^0^ADC01_20170425_1553539000, where ADC represents the node type, ADC01 represents the node ID, CE represents the message name, two zeros indicate that the message is a request message and that the number of message body items is 0, and ADC01_20170425_1553539000 represents the message ID. Further exemplarily, FIG5A also shows an Ack message for the above-mentioned example CE message, that is, the TMS system can send JPCSVR^JPCSVR01^CE^1^0^JPCSVR01_20170425_1553539000^[Received message header] to indicate confirmation of the above-mentioned example CE message. In the "[Received message header]" field, the header of the above-mentioned example CE message can be filled in.

[0070] As shown in FIG5B , a connection test message, such as an LT message, can also be formatted as a header-only message without a body. Its header can be in the format shown in FIG4 . For an LT message, the request can be a default Ack message. As shown in FIG5B , an example of a connection test message is provided: JPCSVR^JPCSVR01^LT^0^0^JPCSVR01_20170425_1553539000 (here, for example, the TMS system and the JPC device are considered as one, denoted as JPCSVR). JPCSVR is the node type, JPCSVR01 is the node ID, LT is the message name, two zeros indicate that the message is a request message and that the number of message body items is 0, respectively. JPCSVR01_20170425_1553539000 is the message ID. Further exemplarily, Figure 5B also shows an Ack message for the above-mentioned example LT message, that is, the ADC system can send ADC^ADC01^LT^l^0^ADC01_20170425_1553539000^[Received message header] to indicate confirmation of the above-mentioned example LT message, and the header of the above-mentioned example LT message can be filled in at [Received message header].

[0071] In some embodiments, to ensure the stability of information exchange and the accuracy of interactive information, information exchange between the image classification system and the image management system can be performed based on the TCP (Transmission Control Protocol). That is, all or at least some of the aforementioned connection establishment and connection test messages, as well as the switch synchronization messages, image allocation messages, image classification feedback messages, re-examination result messages, and various response messages mentioned below, can be transmitted based on the TCP protocol. In addition, these messages can also be optionally transmitted based on other existing or custom protocols.

[0072] It is understandable that when using AI models to implement automatic classification of product images, images obtained for different products and / or at different process nodes (such as different sites) often need to be processed by different AI models, wherein each AI model can be pre-trained using relevant sample data. This training process often requires a lot of manpower and time costs, and because the AI ​​model is data-driven, a trained AI model is generally only suitable for receiving and processing images of a certain type or several types of products. However, on the production line of a factory, there are often many different products, and product changes, process improvements, etc. often occur. Therefore, it is difficult to ensure that there are corresponding available models for all products on the production line. Images of products with available models can be handed over to the corresponding model for automatic detection, while images of products without available models need to be handed over to manual judgment equipment for manual detection. As mentioned above, this image allocation is implemented by the image management system. Therefore, in order to achieve a reasonable distribution of product images between the image classification system and the manual judgment equipment, and to improve the efficiency and accuracy of image classification, while avoiding redundant communication and improving the efficiency of information exchange, the product management system needs to obtain the availability of the models corresponding to each product in the image classification system in real time. Therefore, in some embodiments of the present disclosure, a dual switch control concept is proposed, that is, the control switch for image distribution is set in the image management system, and the online and offline switch for the product model is set in the image classification system, and the control switch in the image management system can be controlled by the image classification system.

[0073] Specifically, in some embodiments, the method 200 may further include the following steps: receiving a product launch operation, the product launch operation being used to launch a target model corresponding to a target product; and sending a first switch synchronization message to the image management system, the first switch synchronization message including product information related to the target product and a switch status indicating that the target model has been launched. Furthermore, in some embodiments, the method 200 may further include the following steps: receiving a product offline operation, the product online operation being used to launch a target model corresponding to the target product; and sending a second switch synchronization message to the image management system, the second switch synchronization message including product information related to the target product and a switch status indicating that the target model has been offline.

[0074] For example, an image classification system may provide a user interface, such as a product launch button and a product delisting button, so that users can launch and delist products by clicking the corresponding buttons. These buttons may be physical buttons, virtual buttons displayed on the user interface, or buttons triggered by voice or gestures, among other things. For example, during the product launch process, a user can initiate the product launch operation in the image classification system by clicking the product launch button, thereby launching a target model corresponding to a target product. Launching a target model can be understood as ensuring that the target model is in an available state and can receive and process images of the target product. When a user executes the product launch operation, the image classification system may send the aforementioned first switch synchronization message to the image management system in real time to inform the image management system of the target product and that the corresponding model for the target product is available. Upon receiving this first switch synchronization message, the image management system may store corresponding control switch data, which may indicate that the target model for the target product is available. Upon subsequent receipt of images corresponding to the target product, the received images may be assigned to the image classification system for automatic detection. Furthermore, for example, during the product delisting process, a user can initiate a product delisting operation in the image classification system by clicking a product delisting button, thereby delisting the target model corresponding to the target product. A target model delisting can be understood as a target model becoming unavailable and no longer able to receive and process images of the target product. When the user performs this product delisting operation, the image classification system can send the aforementioned second switch synchronization message to the image management system in real time to inform the image management system of the target product and that the corresponding model for the target product is unavailable. After receiving this second switch synchronization message, the image management system can store corresponding control switch data, which can indicate that the target model for the target product is unavailable. When subsequently receiving images corresponding to the target product, the received images can be directly assigned to a manual judgment device for manual inspection without further interaction with the image classification system. Through this real-time synchronization mechanism based on the dual switch concept, the image management system can monitor the model availability status of each product in real time and immediately perform control judgment upon receiving a product image, promptly assigning the product image to the image classification system or manual judgment device to avoid unnecessary information interaction. Furthermore, a synchronization button can optionally be provided in the image classification system so that the user can manually initiate the on / off data synchronization process when needed. For example, after the user clicks the synchronization button, the image classification system can send an on / off synchronization message to the image management system. The message can include information related to product information of one or more products and the availability of corresponding models for each product.Exemplarily, the first and second switch synchronization messages may include one or more of the following information: a product number, a product type, a site number, a site status list, a switch update time, and the like, and may optionally include more information. The product type may be a type artificially distinguished based on production specifications, for example. The division of different product types may be determined based on specific application requirements. The product number may be a specific product number under a certain product type. The site number may be the number of one or more sites on a product assembly line. The site status list may be a list of the current available statuses of product models at several sites on a product assembly line. The switch update time may be a switch synchronization time (e.g., the time the switch synchronization message was sent). Optionally, when a model is added or removed from a product at a site, the corresponding control switch at the image management system may be updated by including the site number (and optionally the current available status of the product model) in the corresponding switch synchronization message, or by including the current available status of the product model for all sites in a preset order in the corresponding switch synchronization message (i.e., via the site status list).

[0075] Figure 6 schematically illustrates an example timing diagram for a switch data synchronization process and image distribution process 600 according to some embodiments of the present disclosure. Schematically, Figure 6 uses an ADC system and a TMS system as examples to illustrate the switch data synchronization process and image distribution process between an image classification system and an image management system. As shown, the ADC system may include an ADC client and an ADC interface, while the TMS system may include a TMS interface, a TMS image server, and a TMS database. Alternatively, other ADC system or TMS system structures may be employed depending on specific application requirements.

[0076] As shown in Figure 6, during the on-off data synchronization process, the user can first perform an on-offline or off-line operation on a product model through the ADC client. After receiving the on-offline or off-line operation, the ADC system can perform an on-offline or off-line operation on the corresponding product model and send an on-offline synchronization message, such as an ADC_PS (ADC Product & Step Information) message, to the TMS system through the ADC interface to inform the TMS that the corresponding product model has been updated to an online or offline state. The TMS system can receive this message via the TMS interface and send an Ack message back to the ADC system to confirm the message. The TMS system can then parse the message according to a preset protocol and write the corresponding model management information into the TMS database. The written model management information can indicate that the corresponding product model in the ADC system is in an available state.

[0077] Schematically, FIG7 illustrates an example format of a message body for a switch data synchronization process according to some embodiments of the present disclosure. For switch synchronization messages (such as the first and second switch synchronization messages described above), such as the ADC_PS message, its header can adopt the format shown in FIG4 , and its body can adopt the format shown in FIG7 . As shown in FIG7 , the message body of the switch synchronization message ADC_PS may include four items: product ID, product type, site list, and update time. The product ID indicates the product number, the product type indicates the category to which the product belongs (which can be artificially divided based on information such as production specifications), the site list indicates the model online and offline status (or availability status) of each site on the product assembly line, and the update time indicates the switch synchronization time (e.g., the time the message was sent). The message type requested by the switch synchronization message is a default Ack. That is, upon receiving a default Ack, it can be considered that the sent switch synchronization message has been correctly received. For example, FIG7 also shows an example of the body of the switch synchronization message TestProduct01^B6TE^TStep01, TStep02, TStep03^20211201_1221276200, where TestProduct01 is the product ID, B6TE is the product type code, TStep01, TStep02, and TStep03 are the model online and offline status of the three sites respectively, and the three are arranged in order to form a site list. 20211201_1221276200 is the update time. Accordingly, after adding the header, the complete switch synchronization message can be ADC^ADC01^ADC_PS^0^3^ADC_20211201_1221276200^TestProduct01^B6TE^TStep01, TStep02, TStep03^20211201_1221276200, and its expected default Ack (i.e., the request example shown) may be ADCSRV_ADCSRV01^ADC_PS^1^0^20211201_1221276200^ADC^ADC01^ADC_PS^0^3^ADC_20211201_1221276200.

[0078] In some embodiments, the above method 200 may further include the following steps: receiving an image allocation message from an image management system, where the image allocation message can be used to allocate a target image related to a target product to an image classification system, wherein the image allocation message is sent in response to a target model related to the target product being in an available state; acquiring a target image based on the image allocation message; inputting the target image into a target model; and feeding back an image classification result to the image management system based on an output of the target model, where the image classification result may include at least one of the following: image category, confidence level corresponding to the image category, and information indicating whether to adopt the classification result.

[0079] Exemplarily, when the image management system receives a target image related to a target product from an image management device, it can determine whether there is an available target model corresponding to the target product in the image classification system by querying the control switch status corresponding to the target product. If the query result shows that the target model related to the target product is in an available state, that is, an online state, an image allocation message can be sent to the image classification system to allocate the target image to the image classification system. Exemplarily, the image allocation information may include information related to the target image itself, the name or storage location of the target image, and may include product information corresponding to the target image. After receiving the image allocation message, the image classification system may obtain the target image based on the message content and locate the required target model based on the corresponding product information. Subsequently, the obtained target image may be input into the target model and the model output may be obtained. The model output may include, for example, at least one of the confidence of each image category, the determined image category (generally the category with the highest confidence), and the confidence of the determined image category. The image classification system can feed back the image classification results to the image management system based on the obtained model output, for example, feeding back the image category determined based on the model output, the confidence level corresponding to the determined image category, and information on whether to adopt the classification result (for example, whether to adopt the classification result can be determined based on a comparison between the confidence level and a preset confidence threshold), etc.

[0080] Alternatively, exemplarily, if the query result shows that the target model related to the target product is in an unavailable state, that is, the model has been offline or does not exist, the target image can be directly assigned to the manual judgment device. Exemplarily, information related to the target image itself, the name or storage location of the target image, etc. can be transmitted to the manual judgment device, and the product information corresponding to the target image can be transmitted. After receiving this information, the manual judgment device can present the target image and optionally the corresponding product information to the user (such as an operator) through a human-computer interaction interface (such as an interactive interface presented by a client), and receive the user's input for the image category. Subsequently, the manual judgment device can feedback the classification result of the target image to the image management system based on the user input, and the classification result may include the image category and other optional information.

[0081] Therefore, with the help of the real-time status of the control switches of each product, the target images can be reasonably distributed between the image classification system (automatic classification) and the manual judgment equipment (manual classification). It can not only use the image classification system to improve the classification efficiency and accuracy, but also use manual classification as a supplement to avoid processing errors or redundant information interactions due to the lack of available models, thereby helping to further improve the overall system efficiency and system reliability.

[0082] Refer to Figure 6, which schematically illustrates an example sequence of the image allocation process. As shown, after the AOI device generates an image of a product and uploads the generated image file to the TMS image server, the TMS can store the image information, such as the image storage path, image name, and image upload time, in the TMS database. Furthermore, the TMS can perform validity queries based on the product information corresponding to the image file to query the control switch status of the product model corresponding to the product, that is, to check whether the corresponding product model is in an available state. After obtaining the query results, the TMS can allocate the image files through the TMS interface. If the query results indicate that the corresponding product model is available, the image files can be allocated to the ADC system, such as sending defect information and image location to it, where the defect information can be a preliminary judgment of the defect situation (such as information on the presence or absence of defects determined by the AOI equipment) or manually determined defect information, etc., and the defect detection results fed back can be received to achieve automatic classification; if the query results indicate that the corresponding product model is not available, the image files can be directly allocated to the JPC, or if the ADC system fails to feedback a usable classification result, the image files can be allocated to the JPC, such as sending defect information (such as image classification determined by the ADC system, etc.) and image location to it, and receiving the defect detection results fed back by it to implement manual classification by the operator.

[0083] In some embodiments, the image acquisition device can acquire and upload images one by one to the image management system. However, the efficiency of such image acquisition and uploading is relatively low. Therefore, multiple images are usually uploaded and processed as a group of images. For example, multiple images acquired for the same product (such as the same screen) can be regarded as a group of images, or multiple images acquired for a batch (LOT) of products can be regarded as a group of images. 1 LOT can include a preset number of products, such as 20 screens, and the preset number can be designed according to specific application requirements. In this case, the image management system can also allocate a group of images when allocating images, or alternatively, it can allocate multiple groups of images corresponding to the same product at the same time. In some embodiments of the present disclosure, in order to reduce the data volume of interactive messages and improve interaction efficiency, the allocation of target images can be achieved through a combination of message interaction and file download operations.

[0084] Specifically, in some embodiments, after the image management system receives the target image of the target product uploaded from the image acquisition device, the received image can be stored in an image memory. Exemplarily, the image memory can be a part of the image management system and can be implemented using various forms of storage devices. Subsequently, the image management system can include information related to the storage path of the target image of the target product in the image allocation message sent. Thus, after the image classification system receives the image allocation message, it can obtain the storage path of the target image based on the image allocation message. For example, the image allocation message can include the storage path of the target image or other information that can indicate the storage path of the target image. The image classification system can obtain the storage path of the target image by parsing the image allocation message. Subsequently, the image classification system can obtain the target image from the image memory based on the obtained storage path. Exemplarily, the image classification system can initiate a download request to the image memory based on the obtained storage path of the target image and receive the downloaded target image.

[0085] To further reduce message transmission volume and improve message interaction efficiency, in some embodiments, after receiving a target image and storing it in an image storage device, the image management system can write the target image's storage path to an image path file via a first preset path. The data content in the image path file can, for example, be in a preset format, such as XML or a similar file format, for ease of parsing and management. Subsequently, the image management system can include the file name of the image path file in the transmitted image allocation message. Thus, upon receiving the image allocation message, the image classification system can obtain the file name of the image path file based on the image allocation message and, based on the image path file name and the first preset path, retrieve the image path file that includes the target image's storage path. Exemplarily, the image classification system can determine the complete storage path of the image path file based on a combination of the first preset path and the file name of the image path file. Based on this complete storage path, the image classification system can initiate a download request to the corresponding file storage device and receive the downloaded file, thereby retrieving the image path file. Subsequently, the image classification system can parse the image path file to obtain the target image's storage path, thereby retrieving the target image from the image storage device. This approach facilitates the management and distribution of large quantities of image data.

[0086] In some embodiments, batch feedback of image classification results can be conveniently implemented based on similar ideas, while minimizing the amount of message interaction. Specifically, after the image classification system completes the automatic classification of the target image using the target model, the image classification results can be written into a classification result file via a second preset path. Then, an image classification feedback message can be sent to the image management system. The image classification feedback message can include the file name of the classification result file. The second preset path can be the same as or different from the first preset path, which can be selected based on specific needs. After receiving the image classification feedback message, the image management system can obtain the file name of the classification result file from the image classification feedback message according to a preset protocol, and can obtain the classification result file based on the file name of the classification result file and the second preset path. For example, it can obtain the complete storage path of the classification result file based on the combination of the file name of the classification result file and the second preset path, and request to download the classification result file from the relevant file storage based on the complete storage path, and receive the downloaded file. Subsequently, the image management system can parse the obtained file to obtain the image classification result of the target image.

[0087] Figure 8 schematically illustrates an example timing diagram for an image classification process 800 according to some embodiments of the present disclosure. Schematically, Figure 8 illustrates the image classification process of an image classification system and an image management system, using an ADC system and a TMS system as examples. As shown, the ADC system may include an ADC interface and an AI algorithm system, where the AI ​​algorithm system may include multiple AI models, and the TMS system may include a TMS interface, a TMS image server, and a TMS XML server. Alternatively, other ADC system or TMS system structures may be employed based on specific application requirements.

[0088] As shown in Figure 8, after the TMS system receives the image uploaded by the AOI device, it can generate an XML file and store it on an XML server. If it determines that the ADC system has an available model for the corresponding product, it can send an image allocation message, such as a GS (Glass Start) message, to the ADC system via the TMS interface. This message can include information such as the product ID, site, and XML file name. After receiving the GS message, the ADC system can send an Ack message to indicate its confirmation of the GS message. The ADC system can then download the corresponding XML file from the TMS XML server based on the XML file name and the associated preset path. By parsing the downloaded XML file, it can obtain the image storage path and then download the image file from the TMS image server based on the image storage path. Exemplarily, this download process can be implemented by the ADC initiating a download request to the relevant server and receiving the data. This allows the ADC to obtain product information, site information, and image information, thereby generating an ADC task and passing it to the AI ​​algorithm system to invoke the corresponding AI model to automatically detect and classify the image. Exemplarily, the ADC task can include information indicating the product or product model, image information, and so on. The AI ​​algorithm system can process the ADC task, call the corresponding model and GPU (Graphics Processing Unit) resources to process the image in the ADC task, and feedback the detection results based on the model output. The detection results may include, for example, the image category, confidence level, and other data output by the model. Subsequently, the ADC system can generate a corresponding XML file based on the detection results, upload the generated XML file to the TMS XML server according to the preset path, and send the file name and product information of the generated XML file to the TMS system through an image classification feedback message (such as a GE (Glass End) message). After receiving the GE message, the TMS system can feedback an Ack message to the ADC system to indicate confirmation of the GE message, and can obtain the corresponding file from the XML server according to the file name included in the GE message, and obtain the aforementioned detection results by parsing the file.

[0089] Schematically, Figure 9 illustrates an example message body format for an image classification process according to some embodiments of the present disclosure. For an image allocation message, such as the aforementioned GS message or the ADC GS message shown in Figure 9 , its header can adopt the format shown in Figure 4 , and its body can adopt the format shown in Figure 9 . As shown in Figure 9 , the message body of the ADC GS image allocation message can include six items: GLS sequence number, GLS ID, LOT ID, site ID, product ID, and XML file name. The GLS sequence number is an automatically generated serial number, which can be understood as the serial number of the ADC GS message; the GLS ID is the number of the glass (glass substrate or screen); the LOT ID is the batch number; the site ID is the site number; the product ID is the product number; and the XML file name is the file name of the path file containing the image storage path. The message type requested by this image allocation message is a default ACK. That is, upon receiving a default ACK, it can be considered that the sent image allocation message has been correctly received. 9 also shows an example of the body of the image distribution message 755174E013BX ^ 755174E013 ^ A350A0N ^ PRODUCT ^ xxx.xml, wherein 55174E013BX is the GLS ID, 755174E013 is the LOT ID, A350A0N is the site ID, PRODUCT is the product ID, and xxx.xml is the XML file name. Accordingly, after automatically generating the GLS sequence number and adding the header, the complete image allocation message can be JPCSVR^JPCSVR01^ADC_GS^0^6^JPCSVR01_20170508_1221276200^3^755174E013BX^755174E013^A350A0N^PRODUCT^xxx.xml, and its expected default Ack (i.e., the request example shown) can be ADC^ADC01^ADC_GS^1^0^ADC01_20170508_1221276200^JPCSVR^JPCSVR01^ADC_GS^0^6^JPCSVR01_20170508_1221276200.

[0090] In some embodiments, after receiving the image classification result, the image management system can determine whether to adopt the classification result given by the image classification system (i.e., the image category determined by the image classification system) according to a preset confidence threshold. If the confidence of the image category fed back by the image classification system is higher than the preset confidence threshold, the image category fed back by the image classification system can be adopted, and the classification result of the corresponding image can be determined based on the image category; if the confidence of the image category fed back by the image classification system is lower than the preset confidence threshold, the image category fed back by the image classification system can be not adopted, and the corresponding image can be handed over to a manual judgment device for manual re-inspection. Specifically, in response to the confidence of the image category of the target image fed back by the image classification system being lower than the preset confidence threshold, the image management system can present the target image to the user through the manual judgment device, for example, through the interactive interface on the manual judgment device, and receive the user's input for the image category, i.e., the manual re-inspection result, and then the manual re-inspection result can be fed back to the image management system, and the image management system can determine the final classification result of the target image based on the manual re-inspection result. Thus, manual judgment can be used as a supplementary mechanism to improve the efficiency of image detection while ensuring the accuracy of image detection, thereby helping to ensure the quality of the final product. In addition, illustratively, in some embodiments, the above-mentioned manual review process can also be performed on sampled images to verify the accuracy of the model's automatic classification results, so as to promptly discover problems such as a decrease or insufficiency in model accuracy, so as not to affect the quality of the final product.

[0091] Figure 10 schematically illustrates an example timing diagram for a manual review process 1000 according to some embodiments of the present disclosure. Schematically, Figure 10 illustrates the manual review process using an ADC system and a TMS system as examples. As shown, the ADC system may include an ADC interface, and the TMS system may include a TMS interface, a TMS image server, and a TMS database. Alternatively, other ADC system or TMS system structures may be employed depending on specific application requirements.

[0092] As shown in Figure 10, when manual review is required, the TMS system sends a start inspection command to the JPC device and provides the image location of the target image. The JPC device then presents the target image to a user, such as an operator, and obtains classification input. Based on the user input, the JPC device generates defect detection results and stores them in the TMS database.

[0093] In order to make better use of the manual review results and to improve the performance of related product models, the image management system can feed back the manual review results to the image classification system for subsequent data analysis and iterative model training. For example, the image management system can send a review result message to the image classification system, and the review result message can include information related to the manual review result of the target image, wherein the target image can be an image for which the image classification system does not give an acceptable image classification result. After receiving the review result message from the image management system, the image classification system can obtain the manual review result based on the review result message, and store the manual review result for evaluation and / or iterative training of the target model. Exemplarily, the information related to the manual review result of the target image can be the manual review result of the target image, thereby, the image classification system can directly read the manual review result from the review result message; or, similar to the image assignment message and image classification feedback message in the previous text, the information related to the manual review result of the target image can also be the name of the review result file (for example, in XML file format) containing the manual review result, thereby, the image classification system can read the name of the review result file from the review result message, and download the review result file from the corresponding file server according to the preset path, and then parse the review result file to obtain the manual review result. In this way, the image classification system can obtain the final classification results of all images assigned to the image classification system, including automatic classification results and manual review results. These data can be used for subsequent model performance evaluation and iterative training processes, providing convenience for further model evaluation and improving model performance.

[0094] Continuing with FIG10 , illustratively, the TMS system can retrieve the manual review results of the target image from the TMS database by searching the learning data, and transmit the manual review results to the ADC system via a review result message (e.g., a GL (Glass Learning) message). After receiving the review result message, the ADC system can send an Ack message to indicate confirmation of the review result message.

[0095] FIG11 schematically illustrates an example flow chart of an information interaction method 1100 for an image management system according to some embodiments of the present disclosure. Exemplarily, the information interaction method 1100 may be executed by the image management system 120 shown in FIG1 . In the information interaction method 1100, the image classification system may refer to a system configured to identify images assigned by the image management system to determine the category of the assigned images, and the image management system may refer to a system configured to receive images from an image acquisition device and assign the received images. Each of the image classification system and the image management system may include one or more devices, each of which may perform corresponding functions and be combined together to implement image classification and image management functions.

[0096] As shown in Figure 11, in step 1100, in response to receiving a connection establishment message from the image classification system, a confirmation of the connection establishment message can be sent to the image classification system to establish an information interaction connection with the image classification system, wherein the connection establishment message includes information indicating the identity of the image classification system; in step 1200, a connection test message can be sent to the image classification system; and in step 1300, a confirmation of the connection test message can be received from the image classification system.

[0097] The information interaction method 1100 may have the same or similar advantages as the aforementioned method 200, which will not be described in detail here.

[0098] In some embodiments, the above method 1100 may also include: in response to receiving a first switch synchronization message from the image classification system, the first switch synchronization message includes product information related to the target product and a switch status indicating that the target model is online, updating the availability of the target model related to the target product to an available status.

[0099] In some embodiments, the above method 1100 may further include: in response to receiving a second switch synchronization message from the image classification system, the second switch synchronization message includes product information related to the target product and a switch status indicating that the target model has been offline, updating the availability of the target model related to the target product to an unavailable state.

[0100] In some embodiments, the above method 1100 may also include: in response to receiving a target image related to a target product from an image acquisition device, and the target model related to the target product is in an available state, assigning the target image to an image classification system; obtaining an image classification result fed back by the image classification system, the image classification result including at least one of the following items: image category, confidence level corresponding to the image category, and information indicating whether to adopt the classification result.

[0101] In some embodiments, the step of assigning a target image to an image classification system may include: writing the storage path of the target image to an image path file via a first preset path; and sending an image assignment message to the image classification system, the image assignment message including the file name of the image path file.

[0102] In some embodiments, the step of obtaining the image classification results fed back by the image classification system may include: receiving an image classification feedback message from the image classification system, the image classification feedback message including the file name of the classification result file, the classification result file including the image classification result; obtaining the classification result file according to the file name of the classification result file and a second preset path; and obtaining the image classification result by parsing the classification result file.

[0103] In some embodiments, the method 1100 may further include: in response to a confidence level corresponding to the image category being greater than a preset confidence threshold, determining a classification result of the target image based on the image category.

[0104] In some embodiments, the above method 1100 may also include: presenting the target image to the user in response to the confidence corresponding to the image category being less than a preset confidence threshold; receiving a manual review result from the user; and determining a classification result of the target image based on the manual review result.

[0105] In some embodiments, the method 1100 may further include: sending a review result message to the image classification system, where the review result message includes information related to a manual review result of the target image.

[0106] In some embodiments, the above method 1100 may also include: in response to receiving a target image related to a target product from an image acquisition device, and the target model related to the target product is in an unavailable state, presenting the target image to a user; receiving a manual classification result from the user; and determining a classification result of the target image based on the manual classification result.

[0107] In some embodiments, each of the aforementioned connection establishment message, connection test message, first switch synchronization message, second switch synchronization message, image allocation message, image classification feedback message, and re-inspection result message includes a corresponding message header. The message header includes one or more of the following: message node type, message node number, message name, response type, number of body items, and message number. Furthermore, illustratively, all or at least some of the aforementioned messages may be sent based on the TCP protocol.

[0108] The specific details and advantages of the above-mentioned various embodiments have been described in detail in the description of the embodiment of method 200 and will not be repeated here.

[0109] 12A schematically illustrates an example block diagram of an image processing system 1200A according to some embodiments of the present disclosure. For example, the image processing system may be a management system for detecting defects based on product images, for example, a system for detecting defects in screens.

[0110] As shown in FIG12A , an image processing system 1200A may include an image management system 1210 and an image classification system 1220. Optionally, the image processing system 1200A may further include an image acquisition device 1230. The image management system 1210 may include a first interface 1211 configured to receive a connection establishment message from the image classification system 1220, send a confirmation of the connection establishment message and a connection test message to the image classification system 1220, receive images from the image acquisition device, and distribute the received images, wherein the connection establishment message includes information indicating the identity of the image classification system; and an image storage 1212 configured to store images from the image acquisition device 1230 (or an external image acquisition device). The image classification system 1220 may include: a second interface 1221, which may be configured to: send a connection establishment message to the image management system 1210, receive a confirmation of the connection establishment message and a connection test message from the image management system 1210, and send a confirmation of the connection test message to the image management system 1210; an image classification model 1222, which may be configured to: identify the image assigned by the image management system 1210 to determine the category of the assigned image. It should be understood that the image classification model 1222 may include one or more classification models. For example, each classification model can be used to classify a class of images, such as a defective classification task corresponding to a class of products. Regarding the format and other relevant details and technical effects of the connection establishment message, connection test message and corresponding confirmation message, please refer to the various embodiments described above and will not be repeated here.

[0111] In some embodiments, the first interface 1211 of the image management system 1210 may be further configured to receive a first switch synchronization message from the image classification system 1220, the first switch synchronization message including product information related to the target product and a switch status indicating that the target model has been launched, and update the availability of the target model related to the target product to an available state based on the first switch synchronization message. Furthermore, the image classification system 1220 may further include a user interface 1223 configured to receive a product launch operation for launching the target model corresponding to the target product; and a database 1224 configured to update the switch status of the target model corresponding to the target product to "launched" based on the product launch operation. In this embodiment, the second interface 1221 may be further configured to send the first switch synchronization message to the image management system 1210 in response to the switch status of the target model corresponding to the target product in the database 1224 being updated to "launched." For the format of the first switch synchronization message and other related details and technical effects, please refer to the various embodiments described above and will not be further elaborated here. In addition, the image classification system 1220 can also receive product offline operations through the user interface 1223 and synchronize them to the image management system 1210 through the second switch synchronization message accordingly. This has been described in detail in the previous embodiment and will not be repeated here.

[0112] In some embodiments, the image management system 1210 may further include a file storage 1213 configured to store an image path file, the image path file including the storage path of a target image of a target product. The first interface 1211 may further be configured to, in response to a target model associated with the target product being in an available state, send an image assignment message to the image classification system 1220, the image assignment message including the file name of the image path file. The image classification system 1220 may further include a shared memory 1225 configured to store the image path file downloaded from the file storage 1213 of the image management system 1210 and the target image downloaded from the image storage 1212 of the image management system 1210. The second interface 1221 may further be configured to receive the image assignment message, download the image path file from the file storage 1213 based on the file name of the image path file, parse the image path file to obtain the storage path of the target image, and download the target image from the image storage 1212 based on the storage path. For details regarding the format of the image assignment message, various details of the image assignment process, and technical effects, please refer to the various embodiments described above and will not be repeated here.

[0113] In some embodiments, the image classification system 1220 may further include: a task queue 1226, which may be configured to store image classification tasks created based on a target image; a task management device 1227, which may be configured to send image classification tasks in the task queue 1226 to the image classification model 1222 and receive image classification results from the image classification model 1222; wherein the database 1224 may further be configured to store image information related to the target image and the image classification results; the second interface 1221 may further be configured to upload a classification result file to the file storage 1213 and send an image classification feedback message to the image management system 1210, wherein the classification result file is generated based on the image classification result and the image classification feedback message includes the file name of the classification result file. Furthermore, the first interface 1211 in the image management system 1210 may further be configured to receive the image classification feedback message, obtain the image classification result based on the image classification feedback message, and send a confirmation of the image classification feedback message to the image classification system 1220. Regarding the image classification process, the feedback process of the image classification results, the format of the image classification feedback message and other relevant details and technical effects, please refer to the various embodiments described above and will not be repeated here.

[0114] For ease of understanding, the image assignment and classification result feedback process is briefly described here. For other relevant details, please refer to the various embodiments described above. For example, after receiving an image assignment message, the second interface 1221 of the image classification system 1220 may parse the message to obtain relevant product information and write it to the database 1224. It may also obtain the file name of the image path file and download the image path file from the file storage 1213 based on the file name. The image path file may be stored in the shared memory 1225. The second interface 1221 may parse the stored image path file and obtain the storage path of the target image. It then downloads the target image from the image storage 1212 based on the obtained storage path. The downloaded target image may be stored in the shared memory 1225, and the relevant image information may be written to the database 1224. At this point, the target image is considered downloaded, and the second interface 1221 may store the image classification task associated with the target image in the task queue 1226. It should be understood that the target image may include one or more images. Task queue 1226 can follow a first-in, first-out principle. Task management device 1227 can read image classification tasks from the task queue according to this principle and send the read tasks to image classification model 1222. Image classification model 1222 can call the corresponding classification model to classify the images involved in the image classification task and feedback the classification results to task management device 1227. Task management device 1227 can directly write the classification results to database 1224, or write them to another classification result queue, and then write them to database 1224. Subsequently, second interface 1221 can generate a classification result file based on the classification results in the database, upload it to file storage 1213, and send an image classification feedback message to image management system 1210 to inform the storage path of the classification result file. After receiving confirmation of the image classification feedback message from image management system 1210, image classification system 1220 can consider the image classification task as completed and record the completion status in database 1224.

[0115] In some embodiments, the image management system 1210 may further include a manual determination device 1214, and the first interface 1211 may further be configured to: in response to the confidence level corresponding to the image category being less than a preset confidence threshold, present the target image to the user through the manual determination device 1214, receive the manual review result from the user, and send a review result message to the image classification system 1220, the review result message including information related to the manual review result of the target image. The second interface 1221 in the image classification system 1220 may further be configured to: receive the review result message, store the manual review result in the database 1224 based on the review result message, and send a confirmation of the review result message to the image management system 1210. For details on the manual review process, the format of the review result message, and other related information and technical effects, please refer to the various embodiments described above, which will not be repeated here.

[0116] To further facilitate understanding, FIG12B schematically illustrates an example timing diagram for an image processing scenario according to some embodiments of the present disclosure. As shown in FIG12 , as an example of the image management system 1210 described above, the TMS system may include an image storage such as a TMS image server, a file storage such as a TMS XML server, and a first interface such as a TMS interface; the ADC system, as the image classification system 1220 described above, may include a second interface such as an ADC gateway, a database such as an ADC database (DB), a shared storage such as shared storage, a task queue such as a message queue (MQ), a task management device such as an ADC backend, and an image classification model such as an AI algorithm system. In addition, the entire image processing process also involves image acquisition equipment such as an AOI and manual judgment equipment such as a JPC device, where the JPC device may be an integrated part of the TMS system. Exemplarily, the message queue MQ may be RabbitMQ or another type of MQ. Optionally, depending on the specific application scenario requirements, the TMS system and ADC system may also adopt a system structure different from that shown in FIG12B .

[0117] As shown in Figure 12B, after the ADC system is started or restarted, it can send a connection establishment message, such as a CE message, to the TMS system via the ADC gateway. The TMS system can receive this CE message via the TMS interface and send an ACK message back to the ADC system. Thus, the two can synchronously establish an information exchange connection and begin monitoring each other's messages. Subsequently, the TMS system can send a connection test message, such as an LT message, to the ADC system. Upon receiving the LT message, the ADC system can send an ACK message back (this process is not shown in Figure 12), thereby confirming that the message was successfully delivered to the ADC system. When a user launches a product model through the ADC system client, the product model's status information can be stored in the ADC database and synchronized to the TMS system via the aforementioned switch synchronization message. Next, when the AOI device generates an image of a product and uploads it to the TMS image server, the TMS system can determine the status of the control switch for the corresponding product, that is, whether the corresponding model is available in the ADC system. If the control switch indicates that the corresponding model is available in the ADC system, the TMS system can send an image allocation message, such as a GS message, to the ADC system to allocate the image to the ADC system. The ADC system can respond with an Ack message to confirm the GS message and, based on the product information in the GS message, write the Glass information to the ADC database (here, assuming the product is Glass, i.e., a glass substrate screen) while also obtaining the XML file name from the GS message. The ADC system can then download the XML file from the TMS XML server based on the preset path and the obtained XML file name, store it in shared storage, and parse the downloaded XML file to obtain the image storage path. The ADC system can then download the image file from the TMS image server based on the obtained image storage path and store it in shared storage. Image information, such as the image's location in shared storage and the corresponding product information, can be written to the ADC database. The ADC gateway can then provide feedback indicating the image download is complete. Upon receiving this feedback, the ADC gateway can write a Glass task to the MQ. The Glass task can be a task that specifies the classification and recognition of images of a single Glass or a batch of Glasses. The ADC backend can then consume the Glass tasks from the MQ on a first-in-first-out basis and send them to the AI ​​algorithm system. The AI ​​algorithm system can call the corresponding AI model to detect images in the Glass task and feedback the detection results, such as the determined image category and confidence level, to the ADC backend. The ADC backend can write the results to the MQ and then to the ADC database. The ADC database can then store the automatic classification results of each image for subsequent query and model evaluation.Subsequently, an XML file storing the automatic classification results can be generated and uploaded to the TMS XML server. The ADC system can send the file name of the XML file to the TMS system through a classification feedback message such as a GE message, so that the TMS system can obtain the XML file according to the file name and obtain the automatic classification results by parsing the XML file. When the ADC system receives an Ack message for the GE message from the TMS system, it can be considered that the Glass task has been completed, and the Glass task status can be modified in the ADC DB to end the task. After obtaining the automatic classification results of the image, the TMS system can compare the confidence of the image classification of each image determined by the ADC system with the preset confidence threshold to determine whether to adopt the image classification determined by the ADC system. For images whose ADC results are not adopted, the TMS can assign them to the JPC for manual review and obtain the manual review results (OP results) from the JPC as the final classification results. The re-inspection result can be sent to the ADC system via a re-inspection result message such as a GL message. The ADC system can feed back an Ack message to confirm the GL message and write the OP result involved into the ADC DB. Thus, the ADC system can store the classification results of all assigned images in the ADC DB, including automatic classification results and manual classification results. These data can be used for subsequent model evaluation and training optimization.

[0118] The above solution ensures the stability of inter-system connection establishment. Message interaction between the ADC system and the TMS system can be implemented based on a stable TCP connection, and the reliability, accuracy, and integrity of message delivery can be guaranteed through repeated ACK confirmations. Furthermore, the above solution implements a customized information interaction design for image classification scenarios, especially for automatic classification of product defects. This solution fully utilizes the data characteristics of different data, helping to address issues such as data loss that may occur during data transmission. This helps ensure the feasibility and reliability of AI technology in actual production scenarios, and opens up possibilities for its application in actual sound field scenarios.

[0119] FIG13 schematically illustrates an example block diagram of an information interaction apparatus 1300 for an image classification system according to some embodiments of the present disclosure. The information interaction apparatus 1300 may be deployed, for example, in the image classification system 140 shown in FIG1 . As shown in FIG13 , the information interaction apparatus 1300 may include a first sending module 1310 , a receiving module 1320 , and a second sending module 1330 .

[0120] Specifically, the first sending module 1310 can be configured to: send a connection establishment message to the image management system, the connection establishment message including information indicating the identity of the image classification system; the receiving module 1320 can be configured to: establish an information interaction connection with the image management system in response to receiving a confirmation of the connection establishment message from the image management system; the second sending module 1330 can be configured to: send a confirmation of the connection test message to the image management system in response to receiving a connection test message from the image management system.

[0121] FIG14 schematically illustrates an example block diagram of an information interaction apparatus 1400 for an image management system according to some embodiments of the present disclosure. The information interaction apparatus 1400 may be deployed, for example, in the image management system 130 shown in FIG1 . As shown in FIG14 , the information interaction apparatus 1400 may include a first sending module 1410, a second sending module 1420, and a receiving module 1430.

[0122] Specifically, the first sending module 1410 can be configured to: in response to receiving a connection establishment message from the image classification system, send a confirmation of the connection establishment message to the image classification system to establish an information interaction connection with the image classification system, wherein the connection establishment message includes information indicating the identity of the image classification system; the second sending module 1420 can be configured to: send a connection test message to the image classification system; the receiving module 1430 can be configured to: receive a confirmation of the connection test message from the image classification system.

[0123] It should be understood that the above-mentioned devices 1300 and 1400 can be implemented in software, hardware, or a combination of software and hardware. Multiple different modules can be implemented in the same software or hardware structure, or one module can be implemented by multiple different software or hardware structures.

[0124] Furthermore, the apparatuses 1300 and 1400 can be used to implement the methods 200 and 1100 described above, the details of which have been described above and will not be repeated here for the sake of brevity. The apparatuses 1300 and 1400 can have the same features and advantages as those described with respect to the aforementioned methods.

[0125] FIG15 schematically illustrates an example block diagram of a computing device 1500 according to some embodiments of the present disclosure. For example, it may represent the image management system 120 or the image classification system 140 in FIG1 . Alternatively, the image management system 120 or the image classification system 140 may be a system including multiple computing devices such as computing device 1500, where each computing device may perform different functions.

[0126] As shown, the example computing device 1500 includes a processing system 1501, one or more computer-readable media 1502, and one or more I / O interfaces 1503 that are communicatively coupled to each other. Although not shown, the computing device 1500 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus can include any one or a combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures, or may also include control and data lines.

[0127] Processing system 1501 represents functionality that performs one or more operations using hardware. Thus, processing system 1501 is illustrated as including hardware elements 1504 that can be configured as processors, functional blocks, and the like. This can include implementing application-specific integrated circuits (ASICs) or other logic devices formed using one or more semiconductors in hardware. Hardware elements 1504 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor can be comprised of (a plurality of) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor-executable instructions can be electronically executable instructions.

[0128] The computer-readable medium 1502 is illustrated as including a memory / storage device 1505. The memory / storage device 1505 represents a memory / storage device associated with one or more computer-readable media. The memory / storage device 1505 may include volatile storage media (such as random access memory (RAM)) and / or non-volatile storage media (such as read-only memory (ROM), flash memory, optical disk, magnetic disk, etc.). The memory / storage device 1505 may include fixed media (e.g., RAM, ROM, fixed hard drive, etc.) and removable media (e.g., flash memory, removable hard drive, optical disk, etc.). Exemplarily, the memory / storage device 1505 can be used to store the image files, XML files, automatic classification results, manual classification results, etc. mentioned in the above embodiments. The computer-readable medium 1502 can be configured in various other ways as further described below.

[0129] One or more input / output interfaces 1503 represent functionality that allows a user to enter commands and information into the computing device 1500 and also allows the information to be presented to the user and / or sent to other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, a touch input device (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., which can use visible or invisible wavelengths (such as infrared frequencies) to detect motion that does not involve touch as gestures), a network card, a receiver, and the like. Examples of output devices include a display device (e.g., a monitor or projector), a speaker, a printer, a tactile response device, a network card, a transmitter, and the like. Exemplarily, in the embodiments described above, the input device can allow the user to perform various interactive operations, such as inputting manual classification results, performing product model online and offline operations, and the output device can allow the user to view image files, and the like.

[0130] Computing device 1500 also includes an information exchange application 1506. Application 1506 for controlling online conference participation rights can be stored as computer program instructions in memory / storage device 1505. Application 1506 for controlling online conference participation rights, together with processing system 1501 and the like, can implement all the functions of the various modules of apparatus 1300 or 1400 described with respect to FIG. 13 or 14 .

[0131] Various techniques may be described herein in the general context of software, hardware, elements, or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. As used herein, the terms "module," "function," etc. generally refer to software, firmware, hardware, or a combination thereof. A feature of the techniques described herein is that they are platform-independent, meaning that these techniques can be implemented on a variety of computing platforms with a variety of processors.

[0132] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include various media accessible by computing device 1500. By way of example and not limitation, computer-readable media may include "computer-readable storage media" and "computer-readable signal media."

[0133] As opposed to a mere signal transmission, carrier wave, or signal itself, "computer-readable storage media" refers to media and / or devices, and / or tangible storage devices, capable of persistently storing information. Thus, computer-readable storage media refers to non-signal-bearing media. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-executable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage devices, hard disks, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or other storage devices, tangible media, or articles of manufacture suitable for storing desired information and accessible by a computer.

[0134] "Computer-readable signal media" refers to signal-bearing media configured to transmit instructions to the hardware of computing device 1500, such as via a network. Signal media typically embodies computer-executable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, data signal, or other transport mechanism. Signal media also includes any information delivery media. By way of example, and not limitation, signal media include wired media such as a wired network or direct connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0135] As previously mentioned, hardware elements 1504 and computer-readable medium 1502 represent instructions, modules, programmable device logic and / or fixed device logic implemented in hardware form, which can be used to implement at least some aspects of the technology described herein in some embodiments. Hardware elements can include other implementations in integrated circuits or systems on a chip, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs) and silicon or components of other hardware devices. In this context, hardware elements can be used as processing equipment for executing program tasks defined by the instructions, modules and / or logic embodied by the hardware elements, and hardware devices for storing instructions for execution, such as the computer-readable storage media previously described.

[0136] The aforementioned combinations may also be used to implement the various techniques and modules described herein. Thus, software, hardware, or program modules and other program modules may be implemented as one or more instructions and / or logic on some form of computer-readable storage medium and / or embodied by one or more hardware elements 1504. Computing device 1500 may be configured to implement specific instructions and / or functions corresponding to software and / or hardware modules. Thus, for example, by using a computer-readable storage medium and / or hardware elements 1504 of a processing system, a module may be implemented as a module executable by computing device 1500 as software, at least in part, in hardware. Instructions and / or functions may be executed / operable by, for example, one or more computing devices 1500 and / or processing system 1501 to implement the techniques, modules, and examples described herein.

[0137] The techniques described herein may be supported by these various configurations of computing device 1500 and are not limited to the specific examples of the techniques described herein.

[0138] It should be understood that, for the sake of clarity, embodiments of the present disclosure have been described with reference to different functional units. However, it will be apparent that, without departing from the present disclosure, the functionality of each functional unit can be implemented in a single unit, in multiple units, or as a part of other functional units. For example, the functionality described as being performed by a single unit can be performed by multiple different units. Therefore, reference to a specific functional unit is only considered as a reference to the appropriate unit for providing the described functionality, rather than indicating a strict logical or physical structure or organization. Therefore, the present disclosure can be implemented in a single unit, or can be physically and functionally distributed between different units and circuits.

[0139] The present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon. When the computer-executable instructions are executed, the information interaction method for an image classification system or an image management system is implemented.

[0140] The present disclosure provides a computer program product or computer program, which includes computer-executable instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computing device to perform the information interaction method for an image classification system or image management system provided in the various embodiments described above.

[0141] Variations on the disclosed embodiments will be understood and effected by those skilled in the art in practicing the claimed subject matter by studying the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. An information interaction method for an image classification system, wherein: The image classification system is configured to identify an image assigned by an image management system to determine a category of the assigned image, the image management system is configured to receive an image from an image acquisition device and assign the received image, the method comprising: sending a connection establishment message to the image management system, the connection establishment message including information indicating an identity of the image classification system; In response to receiving a confirmation of the connection establishment message from the image management system, establishing an information exchange connection with the image management system; In response to receiving the connection test message from the image management system, sending an acknowledgement of the connection test message to the image management system.

2. The method according to claim 1, further comprising: receiving a product launch operation, wherein the product launch operation is used to launch a target model corresponding to a target product; A first switch synchronization message is sent to the image management system, wherein the first switch synchronization message includes product information related to the target product and a switch status indicating that the target model has been online.

3. The method according to claim 1, further comprising: receiving an image allocation message from the image management system, the image allocation message being used to allocate a target image related to the target product to the image classification system, wherein the image allocation message is sent in response to a target model related to the target product being in an available state; Based on the image allocation message, acquiring the target image; inputting the target image into the target model; Based on the output of the target model, an image classification result is fed back to the image management system, wherein the image classification result includes at least one of the following items: an image category, a confidence level corresponding to the image category, and information indicating whether to adopt the classification result.

4. The method according to claim 3, wherein: The image allocation message includes information related to a storage path of a target image of the target product, and acquiring the target image based on the image allocation message includes: Based on the image allocation message, acquiring a storage path of the target image; The target image is read from the image memory according to the storage path.

5. The method according to claim 4, wherein: The image allocation message includes a file name of an image path file, and the acquiring the storage path of the target image based on the image allocation message includes: According to the file name of the image path file and the first preset path, the image path file is acquired, wherein the image path file includes the storage path of the target image; The storage path of the target image is obtained by parsing the image path file.

6. The method according to claim 3, wherein: Feeding back the image classification result to the image management system includes: Writing the image classification result into a classification result file via a second preset path; An image classification feedback message is sent to the image management system, wherein the image classification feedback message includes the file name of the classification result file.

7. The method according to claim 1, further comprising: Receiving a product offline operation, wherein the product online operation is used to launch a target model corresponding to a target product; A second switch synchronization message is sent to the image management system, wherein the second switch synchronization message includes product information related to the target product and a switch status indicating that the target model has been offline.

8. The method according to claim 3, further comprising: In response to receiving a review result message from the image management system, the review result message including information related to a manual review result of the target image, the manual review result is stored for evaluation and / or iterative training of the target model.

9. The method according to claims 1 to 8, wherein: Each of the connection establishment message, the connection test message, the first switch synchronization message, the second switch synchronization message, the image allocation message, the image classification feedback message, and the re-inspection result message includes a corresponding message header, and the message header includes one or more of the following items: message node type, message node number, message name, response type, number of text items, and message number.

10. An information interaction method for an image management system, wherein: The image management system is configured to receive images from an image acquisition device and distribute the received images, and the method includes: In response to receiving a connection establishment message from the image classification system, The class system sends a confirmation of the connection establishment message to establish an information exchange connection with the image classification system, wherein the connection establishment message includes information indicating the identity of the image classification system, and the image classification system is configured to identify the image assigned by the image management system to determine the category of the assigned image; Sending a connection test message to the image classification system; An acknowledgement of the connection test message is received from the image classification system.

11. The method according to claim 10, further comprising: In response to receiving a first switch synchronization message from the image classification system, the first switch synchronization message including product information related to the target product and a switch status indicating that the target model is online, the availability of the target model related to the target product is updated to an available status.

12. The method according to claim 10, further comprising: In response to receiving a second switch synchronization message from the image classification system, the second switch synchronization message includes product information related to the target product and a switch status indicating that the target model has been offline, the availability of the target model related to the target product is updated to a non-available status.

13. The method according to claim 10, further comprising: In response to receiving a target image associated with the target product from an image acquisition device and the target model associated with the target product being in an available state, assigning the target image to the image classification system; An image classification result fed back by the image classification system is obtained, wherein the image classification result includes at least one of the following items: an image category, a confidence level corresponding to the image category, and information indicating whether to adopt the classification result.

14. The method according to claim 13, wherein: The assigning the target image to the image classification system comprises: Writing the storage path of the target image into an image path file via a first preset path; An image allocation message is sent to the image classification system, wherein the image allocation message includes the file name of the image path file.

15. The method according to claim 13, wherein: The obtaining of the image classification result fed back by the image classification system comprises: Receive an image classification feedback message from the image classification system, the image classification The feedback message includes the file name of the classification result file, and the classification result file includes the image classification result; Acquire the classification result file according to the file name of the classification result file and the second preset path; The image classification result is obtained by parsing the classification result file.

16. The method according to claim 13, further comprising: In response to the confidence corresponding to the image category being greater than a preset confidence threshold, a classification result of the target image is determined based on the image category.

17. The method according to claim 13, further comprising: In response to a confidence level corresponding to the image category being less than a preset confidence threshold, presenting the target image to a user; receiving a manual review result from the user; Based on the manual review result, a classification result of the target image is determined.

18. The method according to claim 17, further comprising: A re-inspection result message is sent to the image classification system, wherein the re-inspection result message includes information related to a manual re-inspection result of the target image.

19. The method according to claim 10, further comprising: In response to receiving a target image associated with the target product from an image acquisition device, and the target model associated with the target product is in an unavailable state, presenting the target image to a user; receiving a manual classification result from the user; Based on the manual classification result, a classification result of the target image is determined.

20. The method according to claims 10 to 19, wherein: Each of the connection establishment message, the connection test message, the first switch synchronization message, the second switch synchronization message, the image allocation message, the image classification feedback message, and the re-inspection result message includes a corresponding message header, and the message header includes one or more of the following items: message node type, message node number, message name, response type, number of text items, and message number.

21. An information interaction device for an image classification system, wherein: The image classification system is configured to identify images assigned by an image management system to determine the category of the assigned images, and the image management system is configured to receive images from an image acquisition device and store and assign the received images. The method includes: A first sending module is configured to: send a connection establishment message to the image management system, wherein the connection establishment message includes information indicating the identity of the image classification system; A receiving module, configured to: establish an information interaction connection with the image management system in response to receiving a confirmation of the connection establishment message from the image management system; The second sending module is configured to: in response to receiving a connection test message from the image management system, send a confirmation of the connection test message to the image management system.

22. An information interaction device for an image management system, wherein: The image management system is configured to receive images from an image acquisition device, and store and distribute the received images. The method includes: a first sending module configured to: in response to receiving a connection establishment message from an image classification system, send a confirmation of the connection establishment message to the image classification system to establish an information exchange connection with the image classification system, wherein the connection establishment message includes information indicating the identity of the image classification system, and the image classification system is configured to identify the image assigned by the image management system to determine the category of the assigned image; A second sending module is configured to: send a connection test message to the image classification system; The receiving module is configured to receive a confirmation of the connection test message from the image classification system.

23. An image processing system, comprising an image management system and an image classification system, in, The image management system comprises: a first interface configured to: receive a connection establishment message from the image classification system, send a confirmation of the connection establishment message and a connection test message to the image classification system, receive an image from an image acquisition device, and distribute the received image, wherein the connection establishment message includes information indicating an identity of the image classification system; an image memory configured to store images from the image acquisition device, and Wherein, the image classification system comprises: a second interface configured to: send the connection establishment message to the image management system, receive a confirmation of the connection establishment message and the connection test message from the image management system, and send a confirmation of the connection test message to the image management system; The image classification model is configured to: classify the images assigned by the image management system Row recognition to determine the category to which the image is assigned.

24. The system of claim 23, wherein: The first interface of the image management system is further configured to: receive a first switch synchronization message from the image classification system, the first switch synchronization message including product information related to the target product and a switch status indicating that the target model is online, and update the availability of the target model related to the target product to an available status according to the first switch synchronization message, and, Wherein, the image classification system further includes: The user interface is configured to: receive a product launch operation, wherein the product launch operation is used to launch a target model corresponding to a target product; The database is configured to: update the switch state of the target model corresponding to the target product to online according to the product online operation, The second interface is further configured to: in response to the switch status of the target model corresponding to the target product in the database being updated to online, send the first switch synchronization message to the image management system.

25. The system of claim 24, wherein: The image management system also includes: The file storage is configured to store an image path file, wherein the image path file includes a storage path of a target image of the target product. The first interface is further configured to: in response to the target model associated with the target product being in an available state, send an image allocation message to the image classification system, wherein the image allocation message includes the file name of the image path file, and Wherein, the image classification system further includes: A shared memory configured to: store an image path file downloaded from the file memory of the image management system, and store a target image downloaded from the image memory of the image management system; Among them, the second interface is also configured to: receive the image allocation message, download the image path file from the file storage according to the file name of the image path file, obtain the storage path of the target image by parsing the image path file, and download the target image from the image storage according to the storage path.

26. The system of claim 25, wherein: The image classification system further comprises: A task queue, configured to store image classification tasks created based on the target image; The task management device is configured to: send the image classification tasks in the task queue to the image classification model, and receive the image classification results from the image classification model; The database is further configured to store image information related to the target image and the image classification result. The second interface is further configured to: upload a classification result file to the file storage, and send an image classification feedback message to the image management system, wherein the classification result file is generated based on the image classification result, and the image classification feedback message includes a file name of the classification result file, And, wherein, the first interface in the image management system is further configured to: receive the image classification feedback message, obtain the image classification result based on the image classification feedback message, and send confirmation of the image classification feedback message to the image classification system.

27. The system of claim 26, wherein: The image management system further includes a manual determination device, and the first interface is further configured to: in response to a confidence level corresponding to the image category being less than a preset confidence threshold, present the target image to a user through the manual determination device, receive a manual re-inspection result from the user, and send a re-inspection result message to the image classification system, wherein the re-inspection result message includes information related to the manual re-inspection result of the target image, And wherein, the second interface in the image classification system is further configured to: receive the re-inspection result message, store the manual re-inspection result to the database based on the re-inspection result message, and send confirmation of the re-inspection result message to the image management system.

28. A computing device comprising: a memory configured to store computer-executable instructions; A processor configured to perform the method according to any one of claims 1 to 20 when the computer executable instructions are executed by the processor.

29. A computer-readable storage medium storing computer-executable instructions, which, when executed, perform the method according to any one of claims 1 to 20.

30. A computer program product comprising computer instructions which, when executed by a processor, implement the steps of the method of any one of claims 1 to 20.