Aipc edge device and data processing method, apparatus and storage medium thereof

Abstract

The present disclosure provides an AIPC edge device and a data processing method, device and storage medium thereof. The method comprises: obtaining a plurality of to-be-detected image data collected by a production line device based on a local data link; performing deduplication verification on the plurality of to-be-detected image data to perform idempotent queuing on the plurality of to-be-detected image data; performing asynchronous batch processing on the plurality of to-be-detected image data in the queue according to the task priority of the queue to obtain at least one task object group; calling a target defect detection model based on real-time monitoring data of a beat and computing power index; performing inference on each task object in the at least one task object group based on the target defect detection model to obtain an inference result of whether each to-be-detected product has a defect; and outputting the inference result of each to-be-detected product in an idempotent write-back and breakpoint resume manner. The method can ensure stable operation of the defect detection system in a long-time high-load scenario.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of data processing technology for Artificial Intelligence Personal Computers (AIPCs), and more particularly to AIPC edge devices and their data processing methods, apparatus, and storage media. Background Technology

[0002] When deploying intelligent inspection on an Automated Optical Inspection (AOI) production line, data typically enters the inference server from the equipment side or network-attached storage (NAS) in the data center. Due to the large data volume and strict cycle time requirements, the system needs to ensure throughput and latency under limited computing power. At the same time, the on-site environment has issues such as network jitter, equipment restarts, disk I / O fluctuations, and temperature changes, which can easily lead to instability in the inference service.

[0003] Therefore, there is an urgent need to propose a system that can operate stably under long-term high-load scenarios. Summary of the Invention

[0004] This disclosure provides an AIPC edge device and its data processing method, apparatus and storage medium, which can ensure the stable operation of the defect detection system under long-term high-load scenarios.

[0005] In a first aspect, this disclosure provides a data processing method for an AIPC edge device, wherein a defect detection system is deployed on the AIPC edge device, and the data processing method includes: Multiple image data sets to be inspected are acquired from production line equipment via a local data link, where each image data set corresponds to a product to be inspected. The multiple image data sets are deduplicated and then enqueued idempotently. Asynchronous batch processing is performed on the multiple image data sets in the queue according to the task priority of the queue, resulting in at least one task object group. Each task object group includes at least one task object, and each task object corresponds to a product to be inspected. A target defect detection model is invoked based on real-time monitoring data of cycle time and computing power indicators. The cycle time and computing power indicators include at least one of latency, queue length, GPU utilization, memory usage, and memory temperature. Inference is performed on each task object in the at least one task object group based on the target defect detection model to obtain an inference result indicating whether each product to be inspected has a defect. The inference results for each product to be inspected are output using idempotent write-back and breakpoint resume methods.

[0006] In some embodiments of this disclosure, the asynchronous batch processing of multiple image data to be detected in the queue according to the task priority of the queue to obtain at least one task object group includes: Extract the region of interest of each image data to be detected in the queue as a task object; merge task objects of the same size into a first task object group; divide the task objects corresponding to the real-time detection task in the first task object group into a second task object group, and divide the task objects corresponding to the offline statistics task in the first task object group into a third task object group; dequeue the second task object group before the third task object group.

[0007] In some embodiments of this disclosure, the invocation of the target defect detection model based on real-time monitoring data of cycle time and computing power indicators includes: When the real-time monitoring data does not meet the model degradation conditions, the multi-expert semantic detection model is invoked; when the real-time monitoring data meets the model degradation conditions, the multi-expert semantic detection model is downgraded so that the downgraded model can be invoked.

[0008] In some embodiments of this disclosure, the downgrading process of the multi-expert semantic detection model includes at least one of the following processes: reducing the number of multi-scale pruning operations of the multi-expert semantic detection model, reducing the number of prompt word templates of the multi-expert semantic detection model, downgrading the multi-expert semantic detection model to a single general model, and switching the multi-expert semantic detection model to a lightweight student model.

[0009] In some embodiments of this disclosure, the reasoning based on the target defect detection model for each task object in the at least one group of task objects includes: When the first target task object appears for the i-th time in the first target task object group, the semantic vector of the first target task object is cached; where i is an integer greater than the first preset number, the first target task object is any one of the task objects in all the second task object groups and all the third task object groups, and the first target task object group is the task object group to which the first target task object is located.

[0010] And / or, when the target sub-region in the second target task object appears for the jth time in the second target task object group, the semantic vector of the target sub-region is cached; where j is an integer greater than a second preset number, the second target task object is any one of the task objects in all second task object groups and all third task object groups, the second target task object group is the task object group to which the second target task object is located, and the target sub-region is any one of the multiple sub-regions of the second target task object.

[0011] In some embodiments of this disclosure, the step of performing deduplication verification on the plurality of image data to be detected, in order to perform idempotent queuing on the plurality of image data to be detected, includes: The identifier of the currently acquired image data to be detected is matched with the identifier of the image data to be detected in the queue to determine whether the currently acquired image data to be detected has been enqueued; when it is determined that the currently acquired image data to be detected has been enqueued, the currently acquired image data to be detected is ignored to avoid the same image data to be detected being enqueued repeatedly.

[0012] In some embodiments of this disclosure, acquiring multiple image data to be detected collected by production line equipment based on a local data link includes: The plurality of image data to be tested can be read directly from the production line equipment; or, the plurality of image data to be tested can be read from the NAS; or, the plurality of image data to be tested can be received from the production line equipment based on an interface protocol.

[0013] Secondly, this disclosure provides a data processing apparatus for an AIPC edge device, wherein a defect detection system is deployed on the AIPC edge device, and the data processing apparatus includes: The system comprises the following modules: an acquisition module for acquiring multiple image data sets to be inspected from production line equipment via a local data link, where each image data set corresponds to a product to be inspected; a verification module for deduplicating the multiple image data sets to be inspected and enqueuing them idempotently; an asynchronous batch processing module for asynchronously batch processing the multiple image data sets in the queue according to the task priority of the queue to obtain at least one task object group, each task object group including at least one task object, and each task object corresponding to a product to be inspected; a calling module for calling the target defect detection model based on real-time monitoring data of cycle time and computing power indicators, where the cycle time and computing power indicators include at least one of latency, queue length, GPU utilization, video memory usage, and video memory temperature; an inference module for inferring each task object in the at least one task object group based on the target defect detection model to obtain an inference result indicating whether each product to be inspected has a defect; and an output module for outputting the inference results of each product to be inspected using idempotent write-back and breakpoint resume methods.

[0014] Thirdly, this disclosure provides an AIPC edge device, including a processor for executing a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the steps of any of the methods provided in the first aspect.

[0015] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the steps of any of the methods provided in the first aspect.

[0016] In this disclosed technical solution, multiple image data to be inspected are acquired from production line equipment based on a local data link. The multiple image data to be inspected are deduplicated and then enqueued idempotently. The multiple image data to be inspected in the queue are then asynchronously batch-processed according to the task priority of the queue to obtain at least one task object group. Based on real-time monitoring data of cycle time and computing power indicators, a target defect detection model is invoked. Inference is performed on each task object in the at least one task object group based on the target defect detection model to obtain the inference result of whether each product to be inspected has a defect. Based on idempotent write-back and breakpoint resume, the inference result of each product to be inspected is output. This ensures the long-term stable operation of the defect detection system under abnormal on-site conditions and guarantees the data throughput and data latency of the defect detection system under limited computing power, thereby ensuring the stable operation of the defect detection system under long-term high-load scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. It should be understood that the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure, wherein: Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of the present disclosure.

[0018] Figure 2 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure.

[0019] Figure 3 This is a schematic diagram of the structure of an AIPC edge device provided in an embodiment of the present disclosure. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are also within the scope of protection of this disclosure.

[0021] In this disclosure, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of the phrase "embodiment" in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this disclosure can be combined with other embodiments.

[0022] Furthermore, the terms "first," "second," etc., in the specification, claims, or the accompanying drawings are used to distinguish different objects rather than to describe a specific order, and may explicitly or implicitly include one or more of the features.

[0023] In the description of this disclosure, unless otherwise stated, "multiple" and "at least two" mean two or more (including two), and similarly, "multiple groups" and "at least two groups" mean two or more (including two groups).

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0025] The production line equipment provided in this disclosure can be AOI equipment and / or review equipment on a manufacturing production line. Both AOI equipment and review equipment can acquire images of the product to be inspected in order to obtain image data to be inspected. The product to be inspected includes, but is not limited to, display panels, polarizers, glass substrates, and wafers.

[0026] The AIPC edge device disclosed herein is equipped with a defect detection system. It acquires multiple image data sets to be inspected from production line equipment via a local data link, performs deduplication verification on these image data sets, and then enqueues them idempotently. The system asynchronously batch processes these image data sets according to their task priorities to obtain at least one task object group. Based on real-time monitoring data of cycle time and computing power, it calls a target defect detection model and performs inference on each task object in the at least one task object group to obtain an inference result indicating whether each product to be inspected has a defect. Using idempotent write-back and breakpoint resume methods, it outputs the inference results for each product to be inspected. This system can ensure long-term stable operation of the defect detection system even under abnormal on-site conditions, and can also guarantee data throughput and latency under limited computing power, thus ensuring stable operation of the defect detection system under long-term high-load scenarios.

[0027] The technical solutions provided in this disclosure are described in detail below with reference to several specific embodiments.

[0028] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this disclosure, applied to the defect detection system deployed on an AIPC edge device provided in this disclosure, such as... Figure 1 As shown, the specific steps of the data processing method include: S101 acquires multiple image data to be detected collected by production line equipment based on the local data link.

[0029] For example, each image data to be detected corresponds to a product to be detected. The image data to be detected can be grayscale data or multi-channel image data. This disclosure does not impose any specific restrictions on this.

[0030] The production line equipment is equipped with an image acquisition device, which can capture images of the products to be inspected and obtain images to be inspected. The production line equipment generates images to be inspected based on the images to be inspected and stores multiple images to be inspected locally or to a NAS.

[0031] A specific description of one possible implementation of S101 is as follows: The defect detection system can directly read multiple image data to be inspected from the production line equipment.

[0032] The following is a detailed description of another possible implementation when executing S101: The defect detection system can read multiple image data to be inspected from the NAS.

[0033] The following is a detailed description of another possible implementation method when executing S101: The defect detection system can receive multiple image data to be detected sent by the AIPC edge device based on the interface protocol.

[0034] S102, perform deduplication verification on multiple image data to be detected, and enqueue the multiple image data to be detected in an idempotent manner.

[0035] For example, when the production line equipment generates image data to be inspected, it generates a unique identifier for each image data to be inspected. The identifier may include at least one of batch, timestamp, file hash and serial number. Therefore, the defect detection system can obtain multiple image data to be inspected and the identifier corresponding to each image data to be inspected.

[0036] During the execution of S102, firstly, the identifier of the currently acquired image data to be detected is matched with the identifier of the image data to be detected in the queue to determine whether the currently acquired image data to be detected has been enqueued; then, when it is determined that the currently acquired image data to be detected has been enqueued, the currently acquired image data to be detected is ignored to avoid the same image data to be enqueued repeatedly; when it is determined that the currently acquired image data to be detected has not been enqueued, the currently acquired image data to be detected is enqueued.

[0037] For example, the currently acquired image data to be detected is designated as the first image data to be detected, and the image data to be detected in the queue is designated as the second image data to be detected. The first image data to be detected is identified as the first identifier, and the second image data to be detected is identified as the second identifier. The second identifier can be written into the queue or into a specific area for storing the second identifier.

[0038] The first identifier is compared with the second identifiers of all the second image data to be detected in the queue one by one to determine whether the first identifier is included in all the second identifiers. If the first identifier is included in all the second identifiers, it is determined that the first image data to be detected has been enqueued and is ignored. If the first identifier is not included in all the second identifiers, it is determined that the first image data to be detected has not been enqueued and is enqueued.

[0039] In this way, multiple identical image data to be detected generated due to network jitter will not be queued repeatedly, thus achieving idempotent queuing. This ensures that data will not be transmitted repeatedly, improving the stability of the defect detection system during data transmission and ensuring the long-term stable operation of the defect detection system.

[0040] S103, perform asynchronous batch processing on multiple image data to be detected in the queue according to the task priority of the queue, and obtain at least one task object group.

[0041] For example, each task object group includes at least one task object, and each task object corresponds to a product to be detected. The task object can be a Region of Interest (ROI). Task priority can be the priority of the task type; for example, task types include real-time detection tasks and offline statistical tasks, with real-time detection tasks having a higher priority than offline statistical tasks.

[0042] A specific description of one possible implementation of S103 is as follows: S1031, extract the region of interest for each image data to be detected in the queue as a task object.

[0043] For example, a defect detection system includes a Graphics Processing Unit (GPU). Since defect detection of a product to be inspected often involves detecting anomalies in specific regions of the image data to be inspected, such as defect detection of a liquid crystal display panel which involves detecting the image data of the display area, the ROI is the image data of the display area. Therefore, the GPU needs to first extract the ROI of each image data to be inspected in the queue and treat the ROI of each image data to be inspected as a task object.

[0044] S1032, merge task objects of the same size into a first task object group.

[0045] For example, the GPU divides all ROIs of the same size in the image to be detected into a first task object group, thereby obtaining at least one first task object group. Each first task object group includes at least one ROI. Therefore, the ROIs in the same first task object group have the same size, while the ROIs in different first task object groups have different sizes, thereby improving GPU utilization.

[0046] S1033, divide the task objects corresponding to the real-time detection tasks in the first task object group into the second task object group, and divide the task objects corresponding to the offline statistics tasks in the first task object group into the third task object group.

[0047] For example, the GPU divides all ROIs corresponding to the real-time detection tasks in each first task object group into a second task object group, thereby obtaining at least one second task object group, wherein each second task object group includes at least one ROI. At the same time, it divides all ROIs corresponding to the offline statistics tasks in each first task object group into a third task object group, thereby obtaining at least one third task object group, wherein each third task object group includes at least one ROI.

[0048] S1034, the second task object group is dequeued before the third task object group.

[0049] For example, since the real-time detection task has a higher priority than the offline statistics task, the GPU dequeues the second task object group before the third task object group to achieve asynchronous batch processing.

[0050] In this way, by extracting the region of interest of each image data to be detected in the queue as a task object, merging task objects of the same size into a first task object group, dividing the task objects corresponding to the real-time detection tasks in the first task object group into a second task object group, dividing the task objects corresponding to the offline statistics tasks in the first task object group into a third task object group, and dequeuing the second task object group before the third task object group, asynchronous batch processing can be achieved, thereby improving the data throughput of the defect detection system and ensuring that the defect detection system can operate under high load scenarios.

[0051] S104 invokes the target defect detection model based on real-time monitoring data of cycle time and computing power indicators.

[0052] For example, the clock speed and computing power metrics include at least one of latency, queue length, GPU utilization, memory usage, and memory temperature. The defect detection system can monitor the clock speed and computing power metrics in real time.

[0053] The target defect detection model can be any one of multiple defect detection models, including a multi-expert semantic detection model, a single general model, and / or a lightweight student model. The multi-expert semantic detection model is the default target defect detection model, while the single general model and the lightweight student model can be understood as the target defect detection models when the model degradation strategy is triggered. Obviously, the computational cost of the single general model and the lightweight student model is less than that of the multi-expert semantic detection model.

[0054] During the execution of S104, firstly, the defect detection system determines whether the model degradation conditions are met based on real-time monitoring data of the cycle time and computing power indicators. If the cycle time and computing power indicators include latency time, display space occupied, and display temperature, then the model degradation conditions can be at least one of the following: P95 latency time exceeds the time threshold, display space occupied exceeds the space threshold, and display temperature exceeds the temperature threshold.

[0055] Secondly, when real-time monitoring data does not meet the model downgrade conditions, the defect detection system calls the default target defect detection model, i.e., the multi-expert semantic detection model. When real-time monitoring data meets the model downgrade conditions, the defect detection system downgrades the multi-expert semantic detection model to call the downgraded model. The downgrade process for the multi-expert semantic detection model includes at least one of the following: reducing the number of multi-scale pruning operations in the multi-expert semantic detection model, reducing the number of prompt word templates in the multi-expert semantic detection model, downgrading the multi-expert semantic detection model to a single general model, and switching the multi-expert semantic detection model to a lightweight student model.

[0056] For example, when performing the step of the above defect detection system to downgrade the multi-expert semantic detection model, if there are multiple model downgrade conditions, the defect detection system can further downgrade the multi-expert semantic detection model to different degrees based on the number of model downgrade conditions met, so as to adapt to the different cycle time and computing power indicators of the defect detection system.

[0057] If there are three model degradation conditions, and the degradation conditions are P95 delay time exceeding the time threshold, display space exceeding the space threshold, and display temperature exceeding the temperature threshold, when the real-time monitoring data meets one of the model degradation conditions, the degradation processing of the multi-expert semantic detection model can include any one of the following: reducing the number of multi-scale prunings of the multi-expert semantic detection model, reducing the number of prompt word templates of the multi-expert semantic detection model, downgrading the multi-expert semantic detection model to a single general model, and switching the multi-expert semantic detection model to a lightweight student model.

[0058] When real-time monitoring data meets both model downgrade conditions, downgrading the multi-expert semantic detection model can include any two of the following: reducing the number of multi-scale pruning operations and reducing the number of prompt word templates in the multi-expert semantic detection model; downgrading the multi-expert semantic detection model to a single general-purpose model; and switching the multi-expert semantic detection model to a lightweight student model. For example, reducing the number of multi-scale pruning operations and prompt word templates in the multi-expert semantic detection model; or reducing the number of multi-scale pruning operations and downgrading the multi-expert semantic detection model to a single general-purpose model; or reducing the number of prompt word templates and downgrading the multi-expert semantic detection model to a single general-purpose model; or reducing the number of multi-scale pruning operations and switching the multi-expert semantic detection model to a lightweight student model; or reducing the number of prompt word templates and switching the multi-expert semantic detection model to a lightweight student model; or downgrading the multi-expert semantic detection model to a single general-purpose model and switching to a lightweight student model; or downgrading the multi-expert semantic detection model to a single general-purpose model and switching to a lightweight student model.

[0059] When real-time monitoring data meets the three model downgrade conditions, downgrading the multi-expert semantic detection model can include any three of the following: reducing the number of multi-scale clippings and prompt word templates in the multi-expert semantic detection model; reducing the number of prompt word templates in the multi-expert semantic detection model; downgrading the multi-expert semantic detection model to a single general model; and switching the multi-expert semantic detection model to a lightweight student model. For example, reducing the number of multi-scale clippings and prompt word templates in the multi-expert semantic detection model and downgrading it to a single general model; or reducing the number of multi-scale clippings and prompt word templates in the multi-expert semantic detection model and switching it to a lightweight student model; or reducing the number of multi-scale clippings in the multi-expert semantic detection model, downgrading it to a single general model, and switching it to a lightweight student model; or reducing the number of prompt word templates in the multi-expert semantic detection model, downgrading it to a single general model, and switching it to a lightweight student model.

[0060] In this way, by calling the target defect detection model based on real-time monitoring data of cycle time and computing power indicators, the target defect detection model can be adaptively adjusted based on cycle time and computing power indicators, which can reduce the latency of the defect detection system and achieve controllable cost and stable output under cycle time constraints.

[0061] S105, based on the target defect detection model, reason about each task object in at least one task object group to obtain the reasoning result of whether each product to be detected has a defect.

[0062] For example, all ROIs in the same task object group can be simultaneously input into the target defect detection model. The target defect detection model can simultaneously perform anomaly detection on all ROIs in the same task object group, thereby simultaneously inferring whether multiple products to be detected have defects, and the inference result is either defective or not defective.

[0063] S106, based on idempotent write-back and breakpoint resume, outputs the inference results of each product to be tested.

[0064] For example, based on the above embodiments, since each image data to be detected corresponds to a unique identifier and each image data to be detected corresponds to a product to be detected, the inference result of each product to be detected corresponds to an identifier.

[0065] By validating the inference results with identifiers, the inference results can be output in an idempotent write-back manner to ensure that the inference results are not duplicated when multiple inference results for the same product under test are generated due to network jitter. Furthermore, when a breakpoint occurs in the inference results, the inference results can be resumed based on the identifier to automatically restore the normal and stable output of the inference results when a breakpoint occurs due to network jitter.

[0066] In this way, by using idempotent write-back and breakpoint resume to output the inference results of each product to be tested, the normal and stable output of the inference results can be guaranteed even under abnormal conditions, thereby ensuring the long-term stable operation of the defect detection system.

[0067] In summary, by acquiring multiple image data to be inspected from production line equipment via a local data link, deduplicating and verifying these multiple image data, and then enqueuing them idempotently, the system asynchronously batch-processes the multiple image data in the queue according to the task priority, resulting in at least one task object group. Based on real-time monitoring data of cycle time and computing power indicators, the system calls the target defect detection model and performs inference on each task object in the at least one task object group based on the target defect detection model, obtaining the inference result of whether each product to be inspected has a defect. Based on idempotent write-back and breakpoint resume, the inference result of each product to be inspected is output. This system can ensure the long-term stable operation of the defect detection system under abnormal field conditions, and can also guarantee the data throughput and data latency of the defect detection system under limited computing power, thereby ensuring the stable operation of the defect detection system under long-term high-load scenarios.

[0068] In some embodiments, a specific description of a possible implementation of S105 is as follows: S201, when the first target task object appears for the i-th time in the first target task object group, cache the semantic vector of the first target task object.

[0069] For example, i is an integer greater than the first preset number m1, the first target task object is any one of the task objects in all the second task object groups and all the third task object groups, and the first target task object group is the task object group to which the first target task object is located.

[0070] When a task object in a second task object group / third task object group appears for the i-th time, it is considered that the probability of the task object continuing to appear in its task object group is relatively high. The semantic vector of the task object can be cached to cache and reuse the semantic vector of repeated task objects, which can reduce the overhead of repeated encoding.

[0071] As a specific description of another possible implementation when executing S105, the following is an example: S201', when the target sub-region in the second target task object appears for the jth time in the second target task object group, cache the semantic vector of the target sub-region.

[0072] For example, j is an integer greater than the second preset number, the second target task object is any one of the task objects in all second task object groups and all third task object groups, and the second target task object group is the task object group to which the second target task object belongs. After each task object is segmented, multiple sub-regions can be formed, and the target sub-region is any one of the multiple sub-regions of the second target task object.

[0073] When a sub-region in a second task object group / third task object group appears for the jth time, it is considered that the probability of the sub-region continuing to appear in its task object group is relatively high. The semantic vector of the sub-region can be cached to cache and reuse the semantic vector of the repeated sub-region, which can reduce the overhead of repeated encoding.

[0074] As a specific description of another possible implementation when executing S105, it may include the above-mentioned S201 and S201', so as to cache and reuse the semantic vectors of repeated task objects and repeated sub-regions, which can reduce the overhead of repeated encoding.

[0075] In some embodiments, after performing S106, the method further includes generating a system log, wherein the system log includes: model version, routing path, expert set, model output score, threshold and inference results, to provide traceable information and support rapid location of anomalies.

[0076] This disclosure also provides a data processing apparatus for use in the defect detection system deployed on an AIPC edge device provided in this disclosure. Figure 2 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of the present disclosure, as shown below. Figure 2 As shown, the data processing device includes: The acquisition module 110 is used to acquire multiple image data to be inspected collected by production line equipment based on the local data link, wherein each image data to be inspected corresponds to a product to be inspected.

[0077] The verification module 120 is used to perform deduplication verification on multiple image data to be detected, so as to enqueue the multiple image data to be detected in an idempotent manner.

[0078] The asynchronous batch processing module 130 is used to asynchronously batch process multiple image data to be detected in the queue according to the task priority of the queue, so as to obtain at least one task object group. Each task object group includes at least one task object, and each task object corresponds to one product to be detected.

[0079] Module 140 is invoked to call the target defect detection model based on real-time monitoring data of clock speed and computing power indicators; clock speed and computing power indicators include at least one of latency, queue length, GPU utilization, video memory usage, and video memory temperature.

[0080] The reasoning module 150 is used to reason about each task object in at least one task object group based on the target defect detection model, and to obtain the reasoning result of whether each product to be detected has a defect.

[0081] Output module 160 is used to output the inference results of each product under test based on idempotent write-back and breakpoint resume.

[0082] In some embodiments, the asynchronous batch processing module 130 is further configured to extract the region of interest of each image data to be detected in the queue as a task object; merge task objects of the same size into a first task object group; divide the task objects corresponding to the real-time detection task in the first task object group into a second task object group; divide the task objects corresponding to the offline statistics task in the first task object group into a third task object group; and dequeue the second task object group before the third task object group.

[0083] In some embodiments, the calling module 140 is further configured to call the multi-expert semantic detection model when the real-time monitoring data does not meet the model degradation conditions; and to perform degradation processing on the multi-expert semantic detection model when the real-time monitoring data meets the model degradation conditions, so as to call the downgraded model.

[0084] In some embodiments, module 140 is invoked to further implement at least one of the following processes: reducing the number of multi-scale prunings in the multi-expert semantic detection model, reducing the number of prompt word templates in the multi-expert semantic detection model, downgrading the multi-expert semantic detection model to a single general model, and switching the multi-expert semantic detection model to a lightweight student model.

[0085] In some embodiments, the inference module 150 is further configured to cache the semantic vector of the first target task object when the first target task object appears for the i-th time in the first target task object group; wherein, i is an integer greater than a first preset number, the first target task object is any one of the task objects in all the second task object groups and all the third task object groups, and the first target task object group is the task object group in which the first target task object is located.

[0086] And / or, when the target sub-region in the second target task object appears for the jth time in the second target task object group, the semantic vector of the target sub-region is cached; where j is an integer greater than the second preset number, the second target task object is any one of the task objects in all second task object groups and all third task object groups, the second target task object group is the task object group to which the second target task object is located, and the target sub-region is any one of the multiple sub-regions of the second target task object.

[0087] In some embodiments, the verification module 120 is further configured to match the identifier of the currently acquired image data to be detected with the identifier of the image data to be detected in the queue to determine whether the currently acquired image data to be detected has been enqueued; when it is determined that the currently acquired image data to be detected has been enqueued, the currently acquired image data to be detected is ignored to avoid the same image data to be detected being enqueued repeatedly.

[0088] In some embodiments, the acquisition module 110 is further configured to directly read multiple image data to be detected from the production line equipment; or, read multiple image data to be detected from the NAS; or, receive multiple image data to be detected sent by the production line equipment based on an interface protocol.

[0089] The apparatus provided in this disclosure is used to perform the steps provided in any of the above method embodiments, has the functional modules corresponding to the method embodiments, and has the beneficial effects of the method embodiments, which will not be repeated here.

[0090] Figure 3 This is a schematic diagram of the structure of an AIPC edge device provided in this disclosure. Figure 3 A block diagram of an exemplary AIPC edge device suitable for implementing embodiments of the present disclosure is shown. Figure 3 The AIPC edge device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0091] like Figure 3 As shown, the AIPC edge device 12 is presented in the form of a general-purpose computing device. The components of the AIPC edge device 12 may include, but are not limited to: one or more processors 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processor 16).

[0092] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0093] AIPC edge device 12 typically includes a variety of computer system readable media. These media can be any media that can be accessed by AIPC edge device 12, including volatile and non-volatile media, removable and non-removable media.

[0094] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. AIPC edge device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (commonly referred to as "hard disk drives"). Disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disk drives for reading and writing to removable non-volatile optical disks (e.g., CD-ROMs, DVD-ROMs, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of this disclosure.

[0095] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0096] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the steps of the above-described method embodiments.

[0097] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0098] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0099] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0100] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or wide area network (WAN) domain—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0101] This disclosure also provides a computer program product that, when run on a computer, causes the computer to perform the steps of the above-described method embodiments.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses and methods according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. Unless otherwise expressly indicated by the context, the singular form of words used herein and in the appended claims includes the plural form, and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Similarly, the terms “comprising” and “including” are to be interpreted as including rather than exclusively. Likewise, the terms “including” and “or” should be interpreted as including unless such interpretation is expressly prohibited herein. Where the term “example” is used herein, the “example” is merely exemplary and illustrative, and should not be considered exclusive or extensive.

[0103] Several embodiments of this disclosure have been described in detail above. However, it is obvious that those skilled in the art can make various modifications and variations to the embodiments of this disclosure without departing from the spirit and scope of this disclosure. The scope of protection of this disclosure is defined by the appended claims.

Claims

1. A data processing method for an AIPC edge device, characterized in that, The AIPC edge device is equipped with a defect detection system, and the method includes: Multiple images to be inspected are acquired from production line equipment based on local data links, where each image corresponds to a product to be inspected. The multiple image data to be detected are deduplicated and then enqueued idempotently. According to the task priority of the queue, multiple image data to be detected in the queue are asynchronously batch processed to obtain at least one task object group. Each task object group includes at least one task object, and each task object corresponds to a product to be detected. Based on real-time monitoring data of clock speed and computing power indicators, the target defect detection model is invoked; the clock speed and computing power indicators include at least one of latency, queue length, GPU utilization, video memory usage, and video memory temperature; Based on the target defect detection model, reasoning is performed on each task object in the at least one task object group to obtain the reasoning result of whether each product to be detected has a defect; Based on idempotent write-back and breakpoint resume, the inference results of each product under test are output.

2. The method according to claim 1, characterized in that, The step of asynchronously batch processing multiple image data to be detected in the queue according to the task priority of the queue to obtain at least one task object group includes: Extract the region of interest (ROI) of each image data to be detected in the queue as a task object; Merge task objects of the same size into a first task object group; The task objects corresponding to the real-time detection tasks in the first task object group are divided into the second task object group, and the task objects corresponding to the offline statistics tasks in the first task object group are divided into the third task object group. The second task object group is dequeued before the third task object group.

3. The method according to claim 1, characterized in that, The real-time monitoring data based on cycle time and computing power indicators, and the invocation of the target defect detection model, include: When the real-time monitoring data does not meet the model degradation conditions, the multi-expert semantic detection model is invoked. When the real-time monitoring data meets the model downgrade conditions, the multi-expert semantic detection model is downgraded so that the downgraded model can be invoked.

4. The method according to claim 3, characterized in that, The downgrading process of the multi-expert semantic detection model includes at least one of the following processes: reducing the number of multi-scale pruning operations in the multi-expert semantic detection model, reducing the number of prompt word templates in the multi-expert semantic detection model, downgrading the multi-expert semantic detection model to a single general model, and switching the multi-expert semantic detection model to a lightweight student model.

5. The method according to claim 2, characterized in that, The reasoning based on the target defect detection model for each task object in the at least one group of task objects includes: When the first target task object appears for the i-th time in the first target task object group, the semantic vector of the first target task object is cached; where i is an integer greater than a first preset number, the first target task object is any task object in all second task object groups and all third task object groups, and the first target task object group is the task object group to which the first target task object belongs; and / or, When the target sub-region in the second target task object appears for the jth time in the second target task object group, the semantic vector of the target sub-region is cached; where j is an integer greater than the second preset number, the second target task object is any one of the task objects in all second task object groups and all third task object groups, the second target task object group is the task object group to which the second target task object is located, and the target sub-region is any one of the multiple sub-regions of the second target task object.

6. The method according to claim 1, characterized in that, The step of performing deduplication verification on the plurality of image data to be detected, and then performing idempotent queuing on the plurality of image data to be detected, includes: The identifier of the currently acquired image data to be detected is matched with the identifier of the image data to be detected in the queue to determine whether the currently acquired image data to be detected has been enqueued. When it is determined that the currently acquired image data to be detected has been enqueued, the currently acquired image data to be detected is ignored to avoid the same image data to be detected being enqueued repeatedly.

7. The method according to claim 1, characterized in that, The acquisition of multiple image data to be detected from production line equipment based on the local data link includes: The plurality of image data to be detected are read directly from the production line equipment; or... Read the multiple images to be detected from the NAS; or, The system receives the multiple image data to be detected sent by the production line equipment based on the interface protocol.

8. A data processing apparatus for an AIPC edge device, characterized in that, A defect detection system is deployed on the AIPC edge device, and the device includes: The acquisition module is used to acquire multiple image data to be inspected collected by production line equipment based on the local data link, wherein each image data corresponds to a product to be inspected; The verification module is used to perform deduplication verification on the multiple image data to be detected, and to enqueue the multiple image data to be detected idempotently; The asynchronous batch processing module is used to asynchronously batch process multiple image data to be detected in the queue according to the task priority of the queue, to obtain at least one task object group, each task object group includes at least one task object, and each task object corresponds to a product to be detected. The module is used to invoke the target defect detection model based on real-time monitoring data of the clock speed and computing power indicators; the clock speed and computing power indicators include at least one of latency, queue length, GPU utilization, video memory usage, and video memory temperature. The reasoning module is used to reason about each task object in the at least one task object group based on the target defect detection model, and to obtain a reasoning result on whether each product to be detected has a defect. The output module is used to output the inference results of each product under test based on idempotent write-back and breakpoint resume.

9. An AIPC edge device, characterized in that, The method includes a processor for executing a computer program stored in a memory, which, when executed by the processor, implements the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it performs the steps of the method according to any one of claims 1-7.

Images