Image data processing method, device, computer equipment and storage medium

Abstract

The present application relates to an image data processing method, device, computer equipment and storage medium. The method comprises: obtaining an image to be processed from a distributed memory; obtaining an image processing task for processing the image to be processed; determining a target processor type and target resource requirements according to the image processing task; determining a target processor node that matches the target processor type and target resource requirements from processor nodes, the processor nodes including a central processing unit CPU node and a graphics processing unit GPU node; and assigning the image processing task to the target processor node. The method is based on a combination of distributed memory and parallel processing architecture, which can effectively improve the processing efficiency of image data and meet the real-time requirements of image processing.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an image data processing method, device, computer equipment and storage medium. Background Art

[0002] Automatic optical (visual) inspection technology for surface defects based on optical image sensing has become an important means of surface defect detection. It has the advantages of automation, non-contact, fast speed, high precision and high stability. It has developed rapidly in precision manufacturing and assembly industries such as mobile phones, LCD panels, silicon wafers, printed circuit boards and other fields.

[0003] Products in the above fields are usually large-format products. During the inspection process, the object to be inspected needs to be imaged multiple times, which can easily lead to high false alarm rates caused by local deformation of the object to be inspected, as well as problems such as small defects and sub-pixel defects that are difficult to detect stably. When using real-time visual inspection technology to inspect large-format products, it is necessary to convert the image of the object to be inspected into a large-format image through optical sensors, motion devices, and signal processing devices, and then perform image stitching, defect detection, image calibration, image matching, and other processing based on the large-format image.

[0004] At present, real-time visual inspection often uses GPU (Graphics Processing Unit) computing acceleration modules to achieve parallel computing of images, or increases the capacity of memory to solve the computing bottleneck problem. It is difficult to apply to the processing of large-format images due to the difficulties of large width, high precision, and large capacity. Therefore, there is an urgent need for a method that can detect large-format images more efficiently. Summary of the invention

[0005] Based on this, it is necessary to provide an image data processing method, device, computer equipment and storage medium that can efficiently detect large-format images in response to the above technical problems.

[0006] In a first aspect, an embodiment of the present application provides an image data processing method, the method comprising:

[0007] Acquire an image processing task for processing the image to be processed, wherein the image to be processed is stored in a distributed memory;

[0008] Determining a target processor type and target resource requirements according to the image processing task;

[0009] Determine a target processor node that matches the target processor type and the target resource requirement from processor nodes, wherein the processor nodes include a central processing unit (CPU) node and a graphics processing unit (GPU) node;

[0010] The image processing task is assigned to the target processor node.

[0011] In one of the embodiments, determining a target processor node from the processor nodes that matches the target processor type and the target resource requirement includes:

[0012] Poll the processor node according to the registration information corresponding to the processor node to obtain the remaining processing resources corresponding to the processor node;

[0013] Determining a processor node that matches the target processor type;

[0014] Among the matched processor nodes, a processor node whose remaining processing resources meet the target resource requirement is used as the target processor node.

[0015] In one embodiment, determining a target processor type and a target resource requirement according to the image processing task includes:

[0016] The target processor type and the target resource requirement are determined by mapping the image processing task with the processor type and the resource requirement, wherein

[0017] When the image processing task includes a parallel processing task, the target processor type corresponding to the parallel processing task is a GPU;

[0018] When the image processing task includes a logic analysis processing task, the target processor type corresponding to the logic analysis processing task is a CPU.

[0019] In one embodiment, the method further comprises:

[0020] For the parallel processing task, the data to be processed in parallel in the to-be-processed image is transferred from the distributed memory to the GPU memory, and the data in the GPU memory is processed by the target GPU node;

[0021] For the logic analysis and processing task, the data requiring logic analysis and processing in the image to be processed is processed by the target CPU node.

[0022] In one embodiment, the method for determining the image to be processed includes:

[0023] Obtaining the original image and the image performance threshold;

[0024] Detecting the original image to obtain an image performance value of the original image;

[0025] Determining that the image to be processed is an original image whose image performance value exceeds the image performance threshold;

[0026] The image to be processed is saved in the distributed memory.

[0027] In one embodiment, the method further comprises:

[0028] The image to be processed and the original image whose image performance value does not exceed the image performance threshold are compressed respectively and saved in a distributed storage system.

[0029] In one embodiment, the method further comprises:

[0030] In the process of processing the image to be processed, a newly added image to be processed is obtained from the incremental database for use in the image processing task.

[0031] In one embodiment, the method further comprises:

[0032] Acquiring image attribute information of the image to be processed;

[0033] Determine a reference processing mode matching the image processing task and the image attribute information by associating a knowledge base, the reference processing mode including a reference processor type and a corresponding reference resource requirement;

[0034] adjusting the target processor type and the target resource requirement according to the reference processor type and the reference resource requirement;

[0035] The determining, from the processor nodes, a target processor node that matches the target processor type and the target resource requirement comprises:

[0036] A target processor node matching the adjusted target processor type and the target resource requirement is determined from the processor nodes.

[0037] In one embodiment, the method further comprises:

[0038] The associated knowledge base is updated according to the image processing task and the image attribute information, and the corresponding adjusted target processor type and the target resource requirement.

[0039] In a second aspect, an embodiment of the present application provides an image data processing device, the device comprising:

[0040] A task acquisition module, used for acquiring an image processing task for processing the image to be processed, wherein the image to be processed is stored in a distributed memory;

[0041] A demand determination module, used to determine a target processor type and target resource requirements according to the image processing task;

[0042] A node determination module is used to determine a target processor node that matches the target processor type and the target resource requirement from processor nodes, wherein the processor nodes include a central processing unit (CPU) node and a graphics processing unit (GPU) node;

[0043] A task allocation module is used to allocate the image processing task to the target processor node.

[0044] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in any one of the first aspects are implemented.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects above are implemented.

[0046] The above-mentioned image data processing method, device, computer equipment and storage medium adopt a combination of distributed memory and parallel processing architecture to load the image to be processed directly into the distributed memory, which can better meet the real-time requirements of image processing and alleviate data congestion. By registering the processor nodes separately, it is convenient to manage information such as the type and processing resources of the processing nodes, which helps to quickly determine the matching target processor node according to the target processor type and target processing resources corresponding to the image processing task. After determining the target processor node, the image processing task is processed based on the parallel processing architecture, which can effectively improve the processing efficiency of image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A diagram showing an application environment of an image data processing method in an embodiment;

[0048] Figure 2 is a schematic flow chart of an image data processing method in one embodiment;

[0049] Figure 3 A schematic diagram of a flow chart of a step of determining a target processor node in an embodiment;

[0050] Figure 4 is a schematic diagram of an improved distributed service framework in one embodiment;

[0051] Figure 5 is a schematic flow chart of a step of determining an image to be processed in one embodiment;

[0052] Figure 6 A schematic flow chart of the steps of optimizing a target processor node by applying an associated knowledge base in one embodiment;

[0053] Figure 7 is a schematic flow chart of an image data processing method in one embodiment;

[0054] Figure 8 is a schematic flow chart of an image data processing method in one embodiment;

[0055] Fig. 9 is a structural block diagram of an image data processing device in one embodiment;

[0056] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The image data processing method provided in this application can be applied to Figure 1 The application environment shown in FIG. The application environment includes an image acquisition device 110, a terminal 120, a manager 130 and a server 140, wherein:

[0059] The image acquisition device 110 may be connected to the terminal 120 via a video line or the like, and may be a video detection camera group consisting of at least one camera.

[0060] The terminal 120 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The terminal 120 may be used to configure image processing tasks, display image processing results, load images to be processed into distributed memory, and the like.

[0061] The manager 130 and the terminal 120 are connected via a network. The manager 130 may be a management server in a distributed service framework, and may be used to configure and manage data of the server 140, for example, unified naming service, state synchronization service, cluster management, management of distributed application configuration items, etc. The distributed service framework may be implemented based on ZooKeeper (a distributed, open source distributed application coordination service) and the like.

[0062] The server 140 and the manager 130 are connected via a network, which can be implemented by an independent server or a server cluster consisting of multiple servers. Each server in the server 140 can be regarded as a server node. Each server can include a CPU and (or) a GPU. During registration, the distributed service framework registers the server node and each processor node of the CPU and GPU. At the same time, information such as the processing resource type (such as CPU processing, GPU processing) and the corresponding processing resources (such as capacity data) is added to each processor node, so as to facilitate the management of each processor node.

[0063] Specifically, the image acquisition device 110 can be used to image the object to be detected to obtain an image to be processed, and send the obtained image to be processed to the terminal 120. The manager 130 obtains the configured image processing task through the terminal 120, and determines the target processor type and target resource requirements corresponding to the image processing task. The manager 130 can also obtain the processor type and resource information of each processor node in the server 140. Based on the obtained processor type and resource information, the target processor node that can process the image processing task is determined. The manager 130 assigns the image processing task to the target processor node so that the target processor node obtains the corresponding data from the distributed memory and executes the image processing task.

[0064] In one embodiment, Figure 2 As shown, a method for processing image data is provided, which is applied to Figure 1 The manager 130 in FIG. 1 is taken as an example to illustrate, and the following steps are included:

[0065] Step S210, obtaining an image processing task for processing an image to be processed, where the image to be processed is stored in a distributed memory.

[0066] Distributed memory can be a memory resource pool formed by combining multiple memory modules. The parallel process of distributed memory computing is completed by multiple processes (multi-threaded execution); each process has its own memory space. In distributed memory, these processes are distributed in multiple computers, multiple processors and / or multiple cores, together forming a parallel program.

[0067] Image processing tasks may be tasks such as image stitching, calibration, recognition, and test result analysis. Different detection tasks (such as micro defect detection, sub-pixel defect detection, etc.) may correspond to different image processing tasks, and each detection task may correspond to at least one image processing task. The correspondence between detection tasks and image processing tasks may be pre-configured in the terminal or server so that the terminal can directly query and obtain the image processing tasks to be processed based on the correspondence.

[0068] Specifically, in traditional technology, the processing task is usually identified first; then the data is stored in the memory or GPU graphics card according to the identification result of the processing task, and the data in the memory is processed by the CPU or the GPU is processed by the GPU. However, in the case of real-time processing of image data, the data scanning speed is often much faster than the image processing speed. Therefore, in some embodiments of the present application, after the terminal obtains the image to be detected, it can directly load the image to be detected into the distributed memory, which improves the data processing efficiency and reduces data blocking.

[0069] Step S220, determining the target processor type and target resource requirement according to the image processing task.

[0070] Among them, processor types include GPU and CPU. As the computing and control core of the computer system, the CPU is the final execution unit for information processing and program running. The advantages of the central processing unit CPU are its strong scheduling, management, and coordination capabilities, and computing power is second. The graphics processing unit GPU, also known as the display core, visual processor, and display chip, is a microprocessor that specializes in image and graphics related computing on personal computers, workstations, game consoles, and some mobile devices (such as tablets, smart phones, etc.). The graphics processing unit GPU provides the infrastructure for multi-core parallel computing, and the number of cores is very large, which can support parallel computing of large amounts of data. Parallel computing is a relative concept to serial computing. It is an algorithm that can execute multiple instructions at a time, which can increase the computing speed and solve large and complex computing problems by expanding the scale of problem solving.

[0071] Among them, resource requirements can be characterized by computing parameters, which may include parameters such as the number of processors, the number of processes, processor capacity, and execution time.

[0072] Specifically, a mapping relationship between image processing tasks, processor types and resource requirements can be established in advance. After the manager obtains the image processing task, the target processor type and target resource requirement corresponding to the image processing task to be processed currently are queried from the mapping relationship.

[0073] Step S230, determining a target processor node that matches the target processor type and the target resource requirement from the processor nodes, where the processor nodes include a central processing unit CPU node and a graphics processing unit GPU node.

[0074] Among them, when the type of the processor node is consistent with the target processor type, and the processing resources of the processor node are greater than the target processing resources, it can be said that they are matched; or, when the processor node meets the pre-set matching requirements, it can be said that they are matched. Exemplarily, if the type of a node A in the processor node is the same as the target processor type, and the resources that can be provided for task processing are greater than the resources required for the image processing task, then the node A can be used as the target processing node. If there are multiple nodes that meet the requirements of the image processing task, the target processing node can include multiple nodes.

[0075] Step S240: Allocate the image processing task to the target processor node.

[0076] Specifically, since all CPU and GPU processor nodes have been registered during registration, the manager can obtain the processor type and processing resources of each processor node through a polling mechanism or the like. The manager compares the processor type and processing resources of each processor node with the target processor type and target resource requirements corresponding to the image processing task. The processor node that meets the target processor type and target resource requirements is used as the target processor node. The obtained target processor node can be a single mode that only includes GPU nodes or CPU nodes, or a mixed mode that includes both GPU nodes and CPU nodes, depending on the specific image processing task. The manager assigns the image processing task to the target processor node so that the target processor node starts the corresponding process and obtains the required data from the distributed memory for processing.

[0077] In the above-mentioned image data processing method, a combination of distributed memory and parallel processing architecture is adopted to load the image to be processed directly into the distributed memory, which can better meet the real-time requirements of image processing and alleviate data blocking. By registering the processor node separately, it is convenient to manage the type of processing node and processing resources and other information, which helps to quickly determine the matching target processor node according to the target processor type and target processing resources corresponding to the image processing task; after determining the target processor node, the image processing task is processed based on the parallel processing architecture, which can effectively improve the processing efficiency of image data.

[0078] In one embodiment, Figure 3 As shown, step S240, determining a target processor node that matches the target processor type and the target resource requirement from the processor nodes, includes:

[0079] Step S310: poll the processor node according to the registration information corresponding to the processor node to obtain the remaining processing resources corresponding to the processor node.

[0080] Step S320: determine a processor node that matches the target processor type.

[0081] Step S330: Among the matched processor nodes, the processor node whose remaining processing resources meet the target resource requirement is used as the target processor node.

[0082] Specifically, the manager can obtain the processor type and remaining processing resources of each processor node through a polling mechanism or the like. The manager compares the processor type of each processor node with the target processor type corresponding to the image processing task, and determines the processor node that matches the target processor type. Then, the remaining processing resources of the matching processor node are compared with the target resource requirements, and the processor node whose remaining processing resources meet the target resource requirements is used as the target processor node.

[0083] In a specific embodiment, the distributed service framework is implemented based on an improved ZooKeeper. The improved Zookeeper may include a registration center, a round-robin mechanism, and a load balancing mechanism, etc., which can be used to perform resource management and load balancing management on processor nodes. Figure 4 As shown in the figure, the improved Zookeeper registers all CPU and GPU service nodes (processor nodes). At the same time, for each processor node, the computing resource type (CPU computing, GPU computing) and the corresponding capacity data can also be added.

[0084] The round-robin mechanism can be used to obtain the processing resource status of each type of processor node, including identifying the remaining processing capacity of each CPU node and GPU node, the execution status of computing tasks, the remaining time, etc., so as to provide a reference basis for the load balancing mechanism to allocate tasks.

[0085] The load balancing mechanism can be used for task allocation and management. Resource calling rules can be formulated in advance and task scheduling can be performed based on the resource calling rules.

[0086] Furthermore, resource call rules can be expressed as a mapping relationship between image processing tasks and processor types and processing resources. Exemplarily, for parallel processing tasks in image processing tasks, such as image stitching, filtering, calibration, registration, etc., a mapping relationship between parallel processing tasks and GPU types can be established; for logical analysis processing tasks or tasks that cannot be processed in parallel, such as detection result analysis, distributed processing result fusion, time series data processing, etc., a mapping relationship between logical analysis processing tasks and CPU types can be established. The load balancing mechanism can query the target processor type and target resource requirements corresponding to each image processing task in the current detection task based on the established mapping relationship.

[0087] Furthermore, if there are parallel processing tasks in the image processing tasks, after determining the target GPU node corresponding to the parallel processing tasks in the above manner, the data to be processed in parallel in the image to be processed can be transferred from the distributed memory to the GPU video memory (video memory is a running memory that can be used separately by the GPU node), and the data in the GPU video memory can be processed by the target GPU node. If there are logical analysis processing tasks in the image processing tasks, after determining the target CPU node corresponding to the logical analysis processing tasks in the above manner, the data to be processed by logical analysis can be obtained from the distributed memory by the target CPU node for processing.

[0088] In this embodiment, by adopting an improved distributed service framework for resource management and load balancing management, the allocation of processor nodes and processor resources is made more reasonable; by loading the image data to be processed into distributed memory in real time, a parallel processing architecture combining distributed memory computing (i.e., CPU node processing) and GPU computing (i.e., GPU node processing) is adopted, which can better meet the real-time requirements and achieve a more efficient parallel processing effect.

[0089] In one embodiment, Figure 5 As shown, a determination of an image to be processed is described. The process of determining the object to be processed can be implemented by a terminal or a server. In this embodiment, implementation by a terminal is used as an example for description, and can be implemented by the following steps:

[0090] Step S510, obtaining an original image and an image performance threshold.

[0091] The original image may refer to an image that has not been processed in any way. The image performance threshold may be used to measure the optimal performance of the image, including but not limited to grayscale level, timestamp, image size, and classification rate. Different image performance and corresponding image performance thresholds may be set for different detection tasks.

[0092] Specifically, the terminal may be configured with a correspondence between the detection task and the image performance. After the terminal acquires the original image captured by the image acquisition device in real time, the image performance and the corresponding image performance threshold are queried from the correspondence between the detection task and the image performance.

[0093] Step S520: Detect the original image to obtain an image performance value of the original image.

[0094] Step S530: determine that the image to be processed is an original image whose image performance value exceeds an image performance threshold.

[0095] Step S540, saving the image to be processed into the distributed memory.

[0096] The detection of the original image can be achieved by pre-deployed image detection logic and / or a sensing device set in the terminal. The image detection logic may include a detection algorithm based on deep learning, a detection method based on image features (features may include grayscale, texture, etc.), etc. The sensing device may include a dynamic range test card, a clarity test card, a color test card, a color difference test card, and other devices.

[0097] Specifically, one or more detection methods corresponding to each image performance can be configured in the terminal. For each original image, the terminal adopts a detection method corresponding to the image performance to obtain a corresponding image performance value. The actually detected image performance value is compared with the corresponding image performance threshold. If the image performance value of the original image exceeds the image performance threshold, the original image is used as an image to be processed (referred to as a high-value image). The image to be processed is saved in a distributed memory. Furthermore, there may be multiple image performances to be detected, so the image to be processed may be an original image whose image performance values ​​all exceed the corresponding image performance threshold.

[0098] In this embodiment, by pre-detecting the collected original image, it can be ensured that the image to be processed entering the image processing link has high validity, thereby reducing the pressure of distributed memory computing and helping to improve processing efficiency.

[0099] In one embodiment, the method further includes: compressing the image to be processed and the original image whose image performance value does not exceed the image performance threshold respectively, and saving them in the distributed storage system.

[0100] Specifically, a storage strategy may also be pre-configured in the terminal. After the image to be processed is determined from the original image through the above embodiment, the image to be processed and other images in the original image except the image to be processed (referred to as low-value images, i.e., original images whose performance values ​​do not exceed the image performance threshold) may be compressed and stored in a pre-deployed distributed storage system.

[0101] Furthermore, during the execution of the image processing task, the terminal may receive an image for the image processing task (referred to as a delayed unprocessed image), or may still receive an image for the image processing task after the image processing task is completed (referred to as a delayed processed image). For example, when multiple cameras are collecting data at the same time, some cameras may be delayed or fail; or additional images need to be added when performing the image processing task. In order to enable the terminal to accurately distinguish such images, image attribute information such as time attributes, device attributes, and supplementary tags may be carried in the image, so that the terminal can distinguish the purpose of the image according to the image attribute information. In a specific implementation, after obtaining the original image, the terminal reads the image attribute information of the original image. The image attribute information is compared with the index information returned by the distributed storage system or distributed memory. If the original image currently received is judged to be delayed according to the time attribute, and the image processing task is judged to be completed according to the time period in the index information, the original image currently obtained is compressed and saved to the distributed storage system for subsequent use when needed.

[0102] Furthermore, if it is judged based on the time attribute that the currently received original image has been delayed, but it is judged based on the time period in the index information that the image processing task has not been completed; or the image attribute information of the original image contains supplementary tags, the currently acquired original image can be directly stored in the pre-deployed incremental database for the manager to use for task scheduling.

[0103] In this embodiment, by deploying a storage strategy: the original image is divided into categories such as high-value images, low-value images, time-delayed unprocessed images, and time-delayed processed images. The high-value images, low-value images, time-delayed unprocessed images, and time-delayed processed images are directly compressed and stored in a distributed storage system; for high-value images, while data compression and storage are performed, the data is also loaded into a distributed memory for task scheduling; for time-delayed unprocessed images, they can also be stored in a real-time incremental database and further used for task scheduling. Through the above storage strategy, the problems of data redundancy and huge data volume caused by multiple imaging can be alleviated, and the effect of saving memory can be achieved while real-time processing.

[0104] In one embodiment, Figure 6 As shown, the method also includes:

[0105] Step S610: obtaining image attribute information of the image to be processed.

[0106] Step S620: determining a reference processing mode that matches the image processing task and the image attribute information through an associated knowledge base, wherein the reference processing mode includes a reference processor type and a corresponding reference resource requirement.

[0107] Step S630: adjusting the target processor type and the target resource requirement according to the reference processor type and the reference resource requirement.

[0108] The image attribute information may also include the name, identification, capacity, type, and other information of the image. The associated knowledge base may be generated by training the image size, processing efficiency, and accuracy indicators based on the detection records of multiple historical detection tasks (including processing process records, detection results, etc.). The trained associated knowledge base may be used to characterize the association between the image processing scene (including image attribute information and image processing tasks, etc.) and the processing mode.

[0109] Specifically, the manager can take image processing tasks and image attribute information as input, and automatically match reference processing modes from the associated knowledge base. The reference processing mode includes reference processor types and corresponding reference resource requirements. The manager uses the reference processing mode as a guide (refer to Figure 4 The reference processing mode is compared with the target processor type and target resource requirement obtained through the load balancing mechanism, and the target processor type and target resource requirement are adjusted to obtain a processor type and processing resource that are more suitable for the image processing task. Then, a target processor node that matches the adjusted target processor type and target resource requirement is determined from the processor nodes.

[0110] In this embodiment, by constructing an associative knowledge base of image processing scenarios and processing modes, better processor types and processing resources are matched for different image processing tasks according to the training results of the associative knowledge base, so that suitable processor nodes can be scheduled for image processing tasks, which helps to further improve the accuracy of parallel processing and improve computing efficiency.

[0111] In one embodiment, the associated knowledge base can be iteratively updated as the number of detection tasks increases. In a specific implementation, after the detection task is completed, the associated knowledge base is updated based on the image processing task, image attribute information, the corresponding adjusted target processor type and target resource requirements, and processing results. In this embodiment, by iteratively updating the associated knowledge base, the associated knowledge base can continuously learn new knowledge, thereby ensuring the accuracy of the associated knowledge base.

[0112] In one embodiment, Figure 7 As shown, an image data processing method is provided, and the method is applied to an image data processing system as an example for explanation. The image data processing system may include an image acquisition device, a terminal, a manager, and a server cluster, wherein:

[0113] The image acquisition device may include at least one camera, which may be a device in machine vision detection equipment such as an optical system, a mechanical transmission platform, and a motion system. For smaller objects to be detected, a single-camera image acquisition device may be used; for larger objects to be detected, a multi-camera image acquisition device may be used. The terminal is provided with an image acquisition card, which may be used to acquire images taken by the image acquisition device to form an image set.

[0114] When using multiple cameras, the obtained images may need to be stitched. Configure the image stitching strategy in the terminal or server in advance, and automatically determine whether the images need to be stitched according to the image stitching strategy. For example, if it is pre-configured that images taken by multiple cameras need to be stitched, the terminal can stitch the images according to the device information carried in the image. If it is pre-configured to determine whether stitching is needed based on the edge features of a single-camera image acquisition device, then when the terminal recognizes that the image is displaced or incomplete, it can call the shooting results of the surrounding cameras according to the specific position of the camera that took the image, and perform image stitching.

[0115] like Figure 7 As shown (can be synchronized with reference Figure 8 The image data processing method comprises the following steps:

[0116] Step S702: The image acquisition device photographs the object to be detected to obtain an original image.

[0117] Step S704, the image acquisition device transmits the original image to the terminal, and the terminal detects the original image and divides the original image into a high-value image (image to be processed) and a low-value image. The process of the terminal detecting the original image can refer to the above embodiment and will not be elaborated here.

[0118] Step S706: The terminal compresses the high-value image and the low-value image respectively.

[0119] Step S708: Save the compressed image to the distributed storage system.

[0120] Step S710: The terminal stores the image to be processed in the distributed memory.

[0121] Step S712, the manager performs task scheduling and load balancing based on the polling mechanism and the load balancing machine. Specifically, the manager obtains the image processing task for processing the image to be processed. The image processing task includes parallel processing tasks, such as image stitching, filtering, calibration, registration, etc.; and (or), logical analysis processing tasks, such as detection result analysis, distributed processing result fusion, time series data processing, etc. The manager determines the target processor type and target resource requirements required for the image processing task based on the polling mechanism and the load balancing mechanism. Determine the matching target processor node according to the target processor type and the target resource requirements.

[0122] Step S714, obtain a reference processing mode that matches the image processing task through the associated knowledge base. The specific process of obtaining the reference processing mode can refer to the above embodiment, and will not be elaborated here. According to the reference processing mode, adjust the target processor type and target resource requirements.

[0123] Step S716: the manager distributes the logic analysis processing task to the target CPU node, and processes the data to be logic analyzed in the distributed memory of the target CPU node.

[0124] Step S718: the manager distributes the logic analysis processing task to the target GPU node, transfers the data to be processed in parallel from the distributed memory to the GPU memory, and processes the data in the GPU memory through the target GPU node.

[0125] Step S720: The terminal obtains the time-delay processed image or the time-delay unprocessed image, compresses the time-delay processed image and the time-delay unprocessed image respectively, and stores them in the distributed storage system.

[0126] Step S722: The terminal stores the delayed unprocessed image in an incremental database for use in scheduling the image processing task.

[0127] Step S724, integrating the processing results of the target CPU node and the target GPU node.

[0128] Step S726: Update the associated knowledge base. The implementation method of updating the associated knowledge base can refer to the above embodiment, and will not be elaborated here.

[0129] In one embodiment, a specific image data processing method is provided, wherein:

[0130] Image data processing methods are applied to single-camera visual inspection equipment;

[0131] The object to be inspected is the glass substrate in the 6th generation OLED (Organic Light-Emitting Diode) linear array camera module. The size of the glass substrate is 1500mm (millimeter) * 1850mm, and the required inspection accuracy is 2μm (micrometer). The inspection task can be achieved through the following process:

[0132] If a single camera is a 16K pixel (K=1024, 16K=16×1024) linear array camera, and a scan is only 32mm, then 47 linear array cameras are required to scan a 1500mm*1850mm area, and the image data volume generated in a single scan is 646GByte (gigabyte). In order to better detect the object to be inspected, it is usually necessary to image the large-format object to be inspected multiple times, so the data volume of the image set formed will be much larger than 646GByte.

[0133] The original images in the original image set are detected, and the detection items include the grayscale level, timestamp, image size and resolution of the image. Different detection items can be configured for different detection tasks. According to the image performance threshold corresponding to the detection task, a high-value image (image to be processed) and a low-value image are obtained from the original image. In this embodiment, the image performance threshold value is a threshold value related to the detection accuracy of 2μm, then the original image below the relevant threshold value can be compressed and saved in the database. Due to the large amount of image data, distributed storage can be used for data storage. For the original image above the relevant threshold, data compression is also performed and saved in the database; at the same time, the data is loaded into the distributed memory.

[0134] Furthermore, during the execution of the image processing task, additional images may need to be taken, and the additional images can be directly stored in the incremental database for task scheduling. After the image processing task is completed, new images may still be received, and the received images can be compressed and saved in the database.

[0135] Since 47 linear array cameras are used to scan the entire format, different image processing tasks will be used for different inspection tasks. For example, for inspection tasks that require overall recognition or position layout, image stitching processing tasks are required, and 47 images of different positions need to be stitched; for defect detection tasks, image calibration, registration, or abnormal point detection processing tasks can be performed. The supporting algorithms and processing logic of image processing tasks can be pre-configured, so that the manager can allocate image processing tasks to target processor nodes based on polling mechanisms, load balancing mechanisms, and associated knowledge bases, and then execute image processing tasks through target processor nodes according to supporting algorithms and processing logic.

[0136] After the detection task is completed, the associated knowledge base may be updated according to the execution status of the detection task to ensure the accuracy of the associated knowledge base.

[0137] It should be understood that, although the various steps in the above flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above flowchart may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0138] In one embodiment, Fig. 9 As shown, an image data processing device 900 is provided, comprising: a task acquisition module 901, a demand determination module 902, a node determination module 903 and a task allocation module 904, wherein:

[0139] The task acquisition module 901 is used to acquire the image processing task for processing the image to be processed; the demand determination module 902 is used to determine the target processor type and the target resource demand according to the image processing task; the node determination module 903 is used to determine the target processor node that matches the target processor type and the target resource demand from the processor nodes, and the processor nodes include the central processing unit CPU node and the graphics processing unit GPU node; the task allocation module 904 is used to allocate the image processing task to the target processor node.

[0140] In one embodiment, the node determination module 903 includes: a polling unit, which is used to poll the processor node according to the registration information corresponding to the processor node to obtain the remaining processing resources corresponding to the processor node; a node type determination unit, which is used to determine the processor node matching the target processor type; and a node determination unit, which is used to select the processor node whose remaining processing resources meet the target resource requirements among the matching processor nodes as the target processor node.

[0141] In one embodiment, the demand determination module 902 is used to determine the target processor type and target resource requirements through the mapping relationship between the image processing task and the processor type and resource requirements, wherein when the image processing task includes a parallel processing task, the target processor type corresponding to the parallel processing task is GPU; when the image processing task includes a logic analysis processing task, the target processor type corresponding to the logic analysis processing task is CPU.

[0142] In one embodiment, the device 900 also includes: a first processing module, which is used to transfer the data that needs to be processed in parallel in the image to be processed from the distributed memory to the GPU video memory for parallel processing tasks, and process the data in the GPU video memory through the target GPU node; a second processing module is used to process the data that needs to be processed in parallel in the image to be processed through the target CPU node for logical analysis processing tasks.

[0143] In one embodiment, a method for determining an image to be processed includes: obtaining an original image and an image performance threshold; detecting the original image to obtain an image performance value of the original image; determining that the image to be processed is an original image whose image performance value exceeds the image performance threshold; and saving the image to be processed into a distributed memory.

[0144] In one embodiment, the image to be processed and the original image whose image performance value does not exceed the image performance threshold are compressed respectively and stored in a distributed storage system.

[0145] In one embodiment, the apparatus 900 further includes: an image acquisition module, which is used to acquire a newly added image to be processed from the incremental data during the process of processing the image to be processed, so as to be used for the image processing task.

[0146] In one embodiment, the device 900 also includes: an attribute acquisition module, which is used to acquire image attribute information of the image to be processed; a matching module, which is used to determine a reference processing mode that matches the image processing task and the image attribute information through an associated knowledge base, and the reference processing mode includes a reference processor type and a corresponding reference resource requirement; an adjustment module, which is used to adjust the target processor type and the target resource requirement according to the reference processor type and the reference resource requirement; a node determination module 903, which is specifically used to determine a target processor node from the processor nodes that matches the adjusted target processor type and the target resource requirement.

[0147] In one embodiment, the device further comprises: a knowledge base updating module, which is used to update the associated knowledge base according to the image processing task and image attribute information, and the corresponding adjusted target processor type and target resource requirement.

[0148] For the specific definition of the image data processing device, please refer to the definition of the image data processing method above, which will not be repeated here. Each module in the above-mentioned image data processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0149] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image data processing method is implemented.

[0150] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0151] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0152] An image processing task for processing an image to be processed is obtained, where the image to be processed is stored in a distributed memory; a target processor type and a target resource requirement are determined according to the image processing task; a target processor node matching the target processor type and the target resource requirement is determined from processor nodes, where the processor nodes include a central processing unit (CPU) node and a graphics processing unit (GPU) node; and the image processing task is allocated to the target processor node.

[0153] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0154] Poll the processor nodes according to the registration information corresponding to the processor nodes to obtain the remaining processing resources corresponding to the processor nodes; determine the processor nodes that match the target processor type; and use the processor nodes whose remaining processing resources meet the target resource requirements among the matching processor nodes as the target processor nodes.

[0155] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0156] The target processor type and target resource requirements are determined through the mapping relationship between the image processing task, the processor type and the resource requirements. When the image processing task includes a parallel processing task, the target processor type corresponding to the parallel processing task is GPU; when the image processing task includes a logic analysis processing task, the target processor type corresponding to the logic analysis processing task is CPU.

[0157] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0158] For parallel processing tasks, the data that need to be processed in parallel in the image to be processed is transferred from the distributed memory to the GPU video memory, and the data in the GPU video memory is processed by the target GPU node; for logical analysis processing tasks, the data that need to be processed in the image to be processed is processed by the target CPU node.

[0159] In one embodiment, the method for determining the image to be processed includes:

[0160] Obtain an original image and an image performance threshold; detect the original image to obtain an image performance value of the original image; determine that the image to be processed is an original image whose image performance value exceeds the image performance threshold; and save the image to be processed into a distributed memory.

[0161] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0162] The image to be processed and the original image whose image performance value does not exceed the image performance threshold are compressed respectively and saved in the distributed storage system.

[0163] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0164] In the process of processing the image to be processed, newly added images to be processed are obtained from the incremental database for use in the image processing task.

[0165] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0166] Acquire image attribute information of the image to be processed; determine a reference processing mode that matches the image processing task and the image attribute information through an associated knowledge base, the reference processing mode including a reference processor type and a corresponding reference resource requirement; adjust the target processor type and the target resource requirement according to the reference processor type and the reference resource requirement; determine a target processor node that matches the adjusted target processor type and the target resource requirement from the processor nodes.

[0167] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0168] The associated knowledge base is updated according to the image processing task and image attribute information, as well as the corresponding adjusted target processor type and target resource requirements.

[0169] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0170] An image processing task for processing an image to be processed is obtained, where the image to be processed is stored in a distributed memory; a target processor type and a target resource requirement are determined according to the image processing task; a target processor node matching the target processor type and the target resource requirement is determined from processor nodes, where the processor nodes include a central processing unit (CPU) node and a graphics processing unit (GPU) node; and the image processing task is allocated to the target processor node.

[0171] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0172] Poll the processor nodes according to the registration information corresponding to the processor nodes to obtain the remaining processing resources corresponding to the processor nodes; determine the processor nodes that match the target processor type; and use the processor nodes whose remaining processing resources meet the target resource requirements among the matching processor nodes as the target processor nodes.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0174] The target processor type and target resource requirements are determined through the mapping relationship between the image processing task, the processor type and the resource requirements. When the image processing task includes a parallel processing task, the target processor type corresponding to the parallel processing task is GPU; when the image processing task includes a logic analysis processing task, the target processor type corresponding to the logic analysis processing task is CPU.

[0175] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0176] For parallel processing tasks, the data that need to be processed in parallel in the image to be processed is transferred from the distributed memory to the GPU video memory, and the data in the GPU video memory is processed by the target GPU node; for logical analysis processing tasks, the data that need to be processed in the image to be processed is processed by the target CPU node.

[0177] In one embodiment, the method for determining the image to be processed includes:

[0178] Obtain an original image and an image performance threshold; detect the original image to obtain an image performance value of the original image; determine that the image to be processed is an original image whose image performance value exceeds the image performance threshold; and save the image to be processed into a distributed memory.

[0179] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0180] The image to be processed and the original image whose image performance value does not exceed the image performance threshold are compressed respectively and saved in the distributed storage system.

[0181] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0182] In the process of processing the image to be processed, newly added images to be processed are obtained from the incremental database for use in the image processing task.

[0183] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0184] Acquire image attribute information of the image to be processed; determine a reference processing mode that matches the image processing task and the image attribute information through an associated knowledge base, the reference processing mode including a reference processor type and a corresponding reference resource requirement; adjust the target processor type and the target resource requirement according to the reference processor type and the reference resource requirement; determine a target processor node that matches the adjusted target processor type and the target resource requirement from the processor nodes.

[0185] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0186] The associated knowledge base is updated according to the image processing task and image attribute information, as well as the corresponding adjusted target processor type and target resource requirements.

[0187] It should be noted that the above-mentioned devices, computer equipment, storage media, etc. may also include other implementation methods according to the description of the method embodiment, and the specific implementation methods may refer to the description of the relevant method embodiment. At the same time, the new embodiments composed of the mutual combination of the features between the various methods and device, equipment, and server embodiments still belong to the scope of implementation covered by this disclosure, and will not be described one by one here.

[0188] It is understandable that the various embodiments of the above methods, devices, computer equipment or storage media in this specification are described in a progressive manner, and the same / similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For relevant parts, refer to the description of the relevant method embodiments.

[0189] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0190] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0191] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for processing image data, characterized in that: The method comprises: Acquire an image processing task for processing an image to be processed, wherein the image to be processed is stored in a distributed memory; Determining a target processor type and target resource requirements according to the image processing task; A reference processing mode matching the image processing task is obtained through an associated knowledge base, wherein the associated knowledge base is generated by training the image size, processing efficiency, and accuracy indicators based on the detection records of multiple historical detection tasks, and the associated knowledge base obtained through training is used to characterize the association between the image processing scenario and the processing mode; adjusting the target processor type and the target resource requirement according to the reference processing mode; Determine a target processor node that matches the adjusted target processor type and the target resource requirement from processor nodes, the processor nodes including a central processing unit (CPU) node and a graphics processing unit (GPU) node; Allocating the image processing task to the target processor node; Wherein, the method for determining the image to be processed includes: Obtaining the original image and the image performance threshold; Detecting the original image to obtain an image performance value of the original image; Determining that the image to be processed is an original image whose image performance value exceeds the image performance threshold; The image to be processed is saved in the distributed memory.

2. The method according to claim 1, characterized in that The determining, from the processor nodes, a target processor node that matches the adjusted target processor type and the target resource requirement comprises: Poll the processor node according to the registration information corresponding to the processor node to obtain the remaining processing resources corresponding to the processor node; Determining a processor node that matches the adjusted target processor type; Among the matched processor nodes, a processor node whose remaining processing resources meet the adjusted target resource requirement is used as the target processor node.

3. The method according to claim 1, characterized in that The determining of the target processor type and target resource requirements according to the image processing task includes: The target processor type and the target resource requirement are determined by mapping the image processing task with the processor type and the resource requirement, wherein When the image processing task includes a parallel processing task, the target processor type corresponding to the parallel processing task is a GPU; When the image processing task includes a logic analysis processing task, the target processor type corresponding to the logic analysis processing task is a CPU.

4. The method according to claim 1, characterized in that The method further comprises: The image to be processed and the original image whose image performance value does not exceed the image performance threshold are compressed respectively and saved in a distributed storage system.

5. The method according to claim 1, characterized in that The method further comprises: In the process of processing the image to be processed, a newly added image to be processed is obtained from the incremental database for use in the image processing task.

6. The method according to claim 1, characterized in that The method further comprises: Acquiring image attribute information of the image to be processed; The step of obtaining a reference processing mode matching the image processing task through an associated knowledge base, and adjusting the target processor type and the target resource requirement according to the reference processing mode, comprises: Determine, through the associated knowledge base, a reference processing mode that matches the image processing task and the image attribute information, wherein the reference processing mode includes a reference processor type and a corresponding reference resource requirement; The target processor type and the target resource requirement are adjusted according to the reference processor type and the reference resource requirement.

7. An image data processing device, characterized in that: The device comprises: A task acquisition module, used to acquire an image processing task for processing an image to be processed, wherein the image to be processed is stored in a distributed memory; A demand determination module, used to determine a target processor type and target resource requirements according to the image processing task; A matching module, used to obtain a reference processing mode matching the image processing task through an associated knowledge base, wherein the associated knowledge base is generated by training the image size, processing efficiency, and accuracy indicators based on the detection records of multiple historical detection tasks, and the associated knowledge base obtained through training is used to characterize the association between the image processing scene and the processing mode; An adjustment module, configured to adjust the target processor type and the target resource requirement according to the reference processing mode; A node determination module is used to determine a target processor node that matches the adjusted target processor type and the target resource requirement from processor nodes, wherein the processor nodes include a central processing unit (CPU) node and a graphics processing unit (GPU) node; A task allocation module, used for allocating the image processing task to the target processor node; Wherein, the method for determining the image to be processed includes: Obtaining the original image and the image performance threshold; Detecting the original image to obtain an image performance value of the original image; Determining that the image to be processed is an original image whose image performance value exceeds the image performance threshold; The image to be processed is saved in the distributed memory.

8. The device according to claim 7, characterized in that The node determination module comprises: A polling unit, configured to poll the processor node according to the registration information corresponding to the processor node, and obtain the remaining processing resources corresponding to the processor node; A node type determination unit, configured to determine a processor node that matches the adjusted target processor type; The node determination unit is used to select, among the matched processor nodes, a processor node whose remaining processing resources meet the adjusted target resource requirement as the target processor node.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images