Production line visual processing method, device, equipment, storage medium and program product

Abstract

The present application discloses a production line visual processing method, device, equipment, storage medium and program product. The method includes: in response to receiving image data, creating an image processing task corresponding to the image data; allocating a target thread through a preset thread pool to perform the image processing task; in the process of the target thread performing the image processing task, when the target operator object is needed, obtaining an idle target operator object from the preset object pool; when the target operator object is obtained, the image data is processed through the target operator object. According to the embodiment of the present application, when visual processing is performed on image data, there is no need to frequently create and destroy threads and operator objects, thereby reducing the time required for visual processing, reducing resource overhead, reducing the occurrence of resource exhaustion problems, and improving visual processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of batteries, and in particular to a production line visual processing method, device, equipment, storage medium and program product. Background Art

[0002] In the battery manufacturing process, multiple image acquisition devices in a workstation usually need to collect images of the products in the workstation in parallel, and transmit the collected image data to the industrial computer, so that the industrial computer can perform visual processing such as front-end and back-end process defect detection and image analysis on multiple sets of image data. When the industrial computer performs visual processing on multiple sets of image data, in order to improve processing efficiency and improve the accuracy of detection results, it usually creates a visual processing task corresponding to each set of image data and runs multiple visual processing tasks in parallel.

[0003] When an industrial computer processes multiple visual processing tasks in parallel, because each visual processing task requires a certain amount of resources, multiple tasks will compete for the limited hardware resources of the industrial computer, such as memory, processor time, and I / O ports, which can easily lead to a shortage or exhaustion of system resources and low visual processing efficiency.

[0004] Therefore, how to reduce resource overhead in multi-task parallel visual processing scenarios and improve visual processing efficiency has become a difficult problem in the industry. Summary of the invention

[0005] The present application provides a production line visual processing method, device, equipment, storage medium and program product, which can reduce resource overhead in visual processing multi-task parallel scenarios and improve visual processing efficiency.

[0006] In a first aspect, the present application provides a production line visual processing method, including: in response to receiving image data, creating an image processing task corresponding to the image data, the image processing task is used to instruct visual processing of the image data based on a target algorithm configuration file, wherein the target algorithm configuration file is an algorithm configuration file corresponding to the image data loaded in the visual processing application, the algorithm configuration file is used to describe the steps of visual processing and the operator tools used in each step; a target thread is allocated through a preset thread pool to execute the image processing task; in the process of the target thread executing the image processing task, when the target operator object is needed, an idle target operator object is obtained from the preset object pool, wherein the target operator object is any operator object corresponding to the operator tool described in the target algorithm configuration file; when the target operator object is obtained, the image data is processed through the target operator object.

[0007] The technical solution provided by the embodiment of the present application is that the visual processing application pre-creates a thread pool and an object pool. When receiving image data, a target thread can be directly allocated from the thread pool to execute the image processing task corresponding to the image data, and in the process of executing the image processing task, the target operator object required to be used is directly obtained from the object pool to process the image data. In this way, when visually processing image data, there is no need to frequently create and destroy threads and operator objects, thereby reducing the time required for visual processing and reducing resource overhead, thereby improving visual processing efficiency and reducing the occurrence of resource exhaustion problems.

[0008] In some or more embodiments of the present application, a target thread is allocated through a preset thread pool to perform an image processing task, including: determining whether a first number of threads running in the thread pool is less than a thread pool threshold, the thread pool threshold being the maximum number of threads that can run in parallel in the thread pool; when the first number is less than the thread pool threshold, allocating a target thread from the idle threads in the thread pool to perform the image processing task; when the first number is greater than or equal to the thread pool threshold, the thread pool returns to executing the step of determining whether the first number of threads running in the thread pool is less than the thread pool threshold.

[0009] Through the above technical solution, the number of threads running in parallel in the thread pool can be controlled and managed, avoiding system crashes or instability caused by too many threads competing for resources, which helps to enhance the stability and reliability of visual processing applications.

[0010] In some or more embodiments of the present application, the method further includes: after completing the processing of the image data by the target operator object obtained from the object pool, putting the target operator object back into the object pool through a pool management tool.

[0011] The above solution can avoid frequent creation and destruction of operator objects and reduce resource consumption.

[0012] In some or more embodiments of the present application, the method further includes: creating a target operator object if the target operator object is not acquired; and processing the image data through the target operator object after the target operator object is created.

[0013] Through the above technical solution, the interruption of the image processing task caused by the inability to obtain the target operator object from the object pool is avoided, thereby improving the task processing efficiency.

[0014] In some or more embodiments of the present application, the method further includes: initializing the object pool based on an algorithm configuration file and a thread pool threshold loaded in the visual processing application.

[0015] In some or more embodiments of the present application, an object pool is initialized based on an algorithm configuration file and a thread pool threshold loaded in a visual processing application, including: determining the operator objects and the number of each type of operator objects contained in the algorithm configuration file loaded in the visual processing application; determining the initial number of each type of operator objects based on the number of each type of operator objects contained in the algorithm configuration file and the thread pool threshold; pre-building each type of operator object based on the initial number of each type of operator object; and placing the pre-built operator objects into the object pool to initialize the object pool.

[0016] By initializing the object pool through the above technical solution, the success rate of obtaining idle target operator objects from the object pool can be improved, and hardware resources can be more fully utilized.

[0017] In some or more embodiments of the present application, the method also includes: obtaining the system load of the visual processing application; when the system load is greater than a first load threshold, creating an operator object corresponding to the operator tool described in the algorithm configuration file loaded in the visual processing application, and placing the created operator object into an object pool; when the system load is less than a second load threshold, destroying the operator objects in the object pool that meet the destruction conditions, wherein the second load threshold is less than or equal to the first load threshold.

[0018] Through the above technical solution, the number of operator objects in the object pool is automatically adjusted dynamically according to the system load, which can ensure that there are enough objects available in high-concurrency scenarios and that too many resources are not wasted to maintain redundant objects under low load.

[0019] In some or more embodiments of the present application, the destruction condition includes at least one of the following: the length of time that the operator object has not been used is greater than a length threshold; the number of operator objects of the same type is greater than a number threshold.

[0020] Through the above technical solution, by destroying operator objects that have not been used for a long time and / or an excessive number of operator objects, resource occupation can be reduced and resource utilization can be improved.

[0021] In some or more embodiments of the present application, image data is processed through a target operator object, including: when the target operator object belongs to an operator object combination corresponding to a mutex lock, inputting input data corresponding to the target operator object into the mutex lock; determining whether a second number of operator objects running in the operator object combination is less than an object threshold, wherein the object threshold is used to indicate a maximum number of operator objects that can run in parallel in the operator object combination; when the second number is greater than or equal to the object threshold, returning to execute the step of determining the second number of operator objects running in the operator object combination; when the second number is less than the object threshold, inputting the input data into the target operator object, so as to process the image data through the target operator object.

[0022] Through the above technical solution, by setting a mutex lock, the pressure of parallel operation is limited and weakened, and the mutex lock restriction is continued within the execution task. The optimization granularity is finer to achieve the purpose of smooth operation under the maximum resource utilization limit in the parallel scenario.

[0023] In some or more embodiments of the present application, determining whether the second number of operator objects running in the operator object combination is less than the object threshold includes: determining whether the semaphore corresponding to the mutex lock is 0, the initial value of the semaphore is the object threshold, and the semaphore is reduced by one for each operator object running in the operator object combination; when the semaphore corresponding to the mutex lock is greater than 0, determining that the second number of operator objects running in the operator object combination is less than the object threshold.

[0024] Through the above technical solution, it is possible to quickly and accurately determine whether the second number of operator objects running in parallel in the operator object combination exceeds the object threshold based on the semaphore.

[0025] In some or more embodiments of the present application, the method also includes: determining whether the identifier of the target operator object contains the lock name of the mutex lock; when the identifier of the target operator object contains the lock name of the mutex lock, determining that the target operator object belongs to the operator object combination corresponding to the mutex lock, the operator object combination corresponding to the mutex lock is a set of operator objects whose identifiers contain the lock name; the operator object combination includes at least one of the following: different operator objects corresponding to the same algorithm configuration file; different operator objects corresponding to different algorithm configuration files.

[0026] Through the above technical solution, by associating the lock name of the mutex lock with the operator object, the locking operation of the operator object is implemented, which is universal and can be applied to various parallel scenarios.

[0027] In a second aspect, the present application provides a visual processing device, comprising: a task creation module, for creating an image processing task corresponding to the image data in response to receiving image data, the image processing task comprising image data and a target algorithm configuration file, wherein the target algorithm configuration file is an algorithm configuration file corresponding to the image data in a visual processing application, the algorithm configuration file being used to describe the steps of visual processing and the operator tools used in each step; a thread allocation module, for allocating a target thread through a preset thread pool to perform the image processing task; an object pool management module, for obtaining an idle target operator object from a preset object pool when the target thread needs to use the target operator object during the execution of the image processing task, the target operator object being any operator object corresponding to the operator tool described in the target algorithm configuration file; a task execution module, for processing the image data through the target operator object when the target operator object is obtained.

[0028] In a third aspect, the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the visual processing method as described in the first aspect is implemented.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium having computer program instructions stored thereon, and the computer program instructions, when executed by a processor, implement the visual processing method as described in the first aspect.

[0030] In a fifth aspect, the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device can perform the visual processing method described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The features, advantages and technical effects of exemplary embodiments of the present application will be described below with reference to the accompanying drawings.

[0032] Figure 1 A flowchart of a visual processing method provided in some embodiments of the present application;

[0033] Figure 2 A schematic diagram of PipeLine provided for some embodiments of the present application;

[0034] Figure 3 A flowchart of a visual processing method provided for some other embodiments of the present application;

[0035] Figure 4 A flowchart of a visual processing method provided in some embodiments of the present application;

[0036] Figure 5A flowchart of a visual processing method provided in some embodiments of the present application;

[0037] Figure 6 A flowchart of a visual processing method provided in some embodiments of the present application;

[0038] Figure 7 A flowchart of a visual processing method provided in some embodiments of the present application;

[0039] Figure 8 A schematic diagram of a shell of a mutex lock provided in some embodiments of the present application;

[0040] Fig. 9 A schematic diagram of locking a mutex lock provided in some embodiments of the present application;

[0041] Fig.10 A flowchart of a visual processing method provided in some embodiments of the present application;

[0042] Fig.11 (a) is a schematic diagram of a processing logic of a mutual exclusion lock provided in some embodiments of the present application;

[0043] Fig.11 (b) A schematic diagram of a processing logic of a mutual exclusion lock provided in some other embodiments of the present application;

[0044] Fig.12 A flowchart of a visual processing method provided in some embodiments of the present application;

[0045] Fig.13 A schematic diagram of a visual processing device provided in some embodiments of the present application;

[0046] Fig.14 A schematic diagram of an electronic device provided for some embodiments of the present application.

[0047] In the drawings, the drawings are not necessarily drawn to scale. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0050] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0051] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0052] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0053] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0054] In the description of the embodiments of the present application, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0055] To facilitate understanding, the professional terms involved in the embodiments of the present application are first explained.

[0056] Operator object: An operator is a function or method that performs some operation or transformation on data. An operator object is a specific object obtained by encapsulating and instantiating the operator function.

[0057] Operator Tools: A collection of related methods, functions, or classes used to implement specific operator functions. It provides a general mechanism and function for implementing various operator operations. It is a collection of functions for implementing various data operations and transformations.

[0058] Pipeline: It is an inspection process that connects multiple processing steps in series to achieve the architecture mode of step-by-step processing and conversion of data. The configuration or definition of Pipeline is represented as an algorithm configuration file in XML format, which describes its structure, steps, parameters and other information.

[0059] Thread pool: is a mechanism for managing threads. It maintains a certain number of pre-created threads. When a task needs to be executed, the thread pool will allocate an idle thread from the pool to handle the task. After the task is completed, the thread will not be destroyed, but returned to the pool to wait for the next task.

[0060] Object pooling: is a technology for managing objects. It creates a certain number of objects in advance and stores them in a pool. When an object needs to be used, an available object is obtained from the pool and returned to the pool after use, rather than being destroyed directly.

[0061] Mutex: A synchronization mechanism used in multi-threaded or multi-process programming. Its main purpose is to ensure that at any given time, only one or a few threads or processes can access the shared resources or critical code segments protected by the mutex, thereby avoiding data inconsistency, race conditions, and other problems caused by multiple threads or processes reading and writing to shared resources at the same time.

[0062] At present, in the battery manufacturing process, when performing visual processing such as defect detection and image analysis in the front and back processes, image acquisition is performed in parallel through multiple image acquisition devices, so as to obtain multiple sets of image data corresponding to the multiple image acquisition devices. The multiple sets of image data are transmitted to the industrial computer, and the industrial computer will process the multiple sets of image data in parallel in a multi-threaded manner through the visual processing application installed therein. During processing, not only is it necessary to create threads corresponding to each set of image data separately, but each thread also needs to create corresponding operator tools when executing. Therefore, each set of image data needs to occupy a large amount of resources when processing. This leads to the preemption of system resources when processing multiple sets of image data in parallel, resulting in system resource shortages, anomalies, and even crashes.

[0063] Therefore, how to design a resource utilization strategy to ensure that visual processing applications do not crash due to resource exhaustion in parallel visual processing scenarios, and to maximize the use of hardware resources to complete parallel scenario task processing, has become a technical problem that needs to be solved urgently.

[0064] In order to solve the above technical problems, the embodiments of the present application provide a visual processing method, device, equipment, storage medium and program product. By designing a thread pool to establish thread scheduling problems and an object pool to avoid repeated construction and destruction of operator tools, the resources required for each set of image data during visual processing can be reduced. In this way, the total resources required for parallel processing of multiple sets of image data can be reduced, thereby reducing the problem of resource exhaustion and improving the working efficiency of industrial computers.

[0065] The technical solution provided in the embodiment of the present application can be applied to the scenario of visual processing of multiple groups of image data in parallel, and its execution subject can be an industrial computer. Since the processing method for each group of image data is consistent in the scenario of parallel visual processing, for the sake of ease of description, the embodiment of the present application takes the processing of one group of image data as an example for description.

[0066] See also Figure 1 , is a flow chart of a visual processing method provided in an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps S11-S14, which are described in detail below.

[0067] S11. In response to receiving image data that needs to be processed, creating an image processing task corresponding to the image data, wherein the image processing task is used to instruct visual processing of the image data based on a target algorithm configuration file.

[0068] Here, the target algorithm configuration file is an algorithm configuration file corresponding to the image data loaded in the visual processing application. The algorithm configuration file is used to describe the steps of visual processing and the operator tools used in each step.

[0069] In some embodiments of the present application, a process configuration platform is pre-installed in an industrial computer. The process configuration platform includes multiple operator tool options, each of which corresponds to an operator tool, and different operator tool options correspond to different operator tools. The operator tool options include but are not limited to traditional operator tools such as Gaussian filtering, median filtering, and affine transformation, and options corresponding to AI operator tools obtained through similar deep algorithm training. Based on this, users can, based on the needs of visual processing, arrange and combine the operator tool options in the process configuration platform to form one or more such options. Figure 2The PipeLine shown in the figure describes the structure, steps, parameters and other information of the PipeLine to obtain the algorithm configuration file corresponding to the PipeLine. The algorithm configuration file describes the steps of the PipeLine and the operator tools used in each step, and stores the algorithm configuration file in a third-party database. Based on this, when the visual processing application pre-installed in the industrial computer is started, the algorithm configuration file corresponding to the PipeLine to be used can be called from the third-party database, and one or more algorithm configuration files can be loaded into the memory corresponding to the visual processing application. In this way, the visual processing application can perform visual processing on the received image data through the loaded one or more PipeLines.

[0070] In some embodiments of the present application, the image acquisition device in the production line can transmit the acquired image data to the visual processing application in the industrial computer. After receiving the image data, the visual processing application determines the PipeLine used to process the image data, and uses the PipeLine as the PipeLine corresponding to the image data, and uses the algorithm configuration file corresponding to the PipeLine as the target algorithm configuration file corresponding to the image data, thereby creating an image processing task corresponding to the image data based on the image data and its corresponding target algorithm configuration file. Among them, the image processing task is used to indicate that the image data is processed according to the PipeLine corresponding to the image data. As can be seen from the above content, the algorithm configuration file describes the steps of the PipeLine and the operator tools used in each step. In this way, the steps for processing the image data and the operator tools used in each step can be determined based on the image processing task.

[0071] In some embodiments of the present application, image data is acquired by an image acquisition device on a battery production manufacturing line. Multiple image acquisition devices are usually installed on the production line, and different image acquisition devices are installed at different positions to collect different image data. Different image data may require different image processing. For example, in the battery assembly process, it is necessary to perform appearance defect detection and lug weld mark detection on the battery. Among them, in order to realize appearance defect detection, the surface image data of the battery surface is collected by an image acquisition device for collecting battery surface images. In order to realize lug weld mark detection, the lug image data of the battery lug is collected by an image acquisition device for collecting battery lug images. When performing visual processing, surface defect detection is performed on the surface image data, and lug weld mark detection is performed on the lug image data. Different detection algorithms are used for surface defect detection and lug weld mark detection, and therefore, different detection processes PipeLine correspond to different detection processes. It can be seen that when visual processing applications perform visual processing on different image data, the Pipelines used may be different. In view of this, in order to accurately and quickly determine the Pipeline corresponding to the image data, the image acquisition device can add a process identifier to the image data it acquires, and the process identifier is used to indicate the Pipeline corresponding to the image data, that is, to indicate which Pipeline is used to process the image data. Based on this, the visual processing application can determine the Pipeline corresponding to the image data based on the process identifier in the image data, and then use the algorithm configuration file of the Pipeline as the target algorithm configuration file corresponding to the image data. Among them, the process identifier added by the image acquisition device to the image data can be determined according to the location and / or process of the image acquisition device. For example, in the production line corresponding to the battery pack assembly process, there is a first image acquisition device for acquiring the top image of the battery pack and a second image acquisition device for acquiring the side image of the battery pack. Since the top structure and the side structure of the battery pack are different, when performing front-end and back-end process defect detection on the battery pack, it is necessary to use Pipeline1 corresponding to the top of the battery pack to process the image data acquired by the first image acquisition device, and use Pipeline2 corresponding to the side of the battery pack to process the image data acquired by the second image acquisition device. Based on this, the first image acquisition device can add a process identifier corresponding to Pipeline1 to the image data it acquires, and the second image acquisition device can add a process identifier corresponding to Pipeline2 to the image data it acquires.

[0072] S12. Allocate a target thread through a preset thread pool to execute the image processing task.

[0073] In some embodiments of the present application, considering that in actual application, there will be multiple image processing tasks corresponding to multiple groups of image data that need to be executed, therefore, in order to improve the efficiency of task execution, it is also necessary to create multiple threads to execute these tasks in parallel. Based on this, in this embodiment, the visual processing application can create a thread pool based on the thread specification file in advance after startup and before receiving the image data. The thread pool specification file is a document or file that describes the configuration and parameters related to the thread pool. The thread pool specification file may include the following key information: the size of the thread pool, that is, the upper limit of the number of threads in the thread pool. The thread pool threshold, that is, the maximum number of threads that can run in parallel in the thread pool. The configuration of the task queue, such as the type of task queue (such as unbounded queue, bounded queue), the capacity of the queue, etc. The strategy for thread creation and destruction, including rules such as when to create a new thread and when to destroy an idle thread. Task rejection strategy, when the thread pool is full and the task queue is full, it is impossible to accept new tasks. The priority setting of the thread determines the priority level of the thread in the thread pool. Other related parameters, such as the idle time threshold of the thread, the name prefix of the thread, etc. Through such a specification file, the running characteristics and behaviors of the thread pool can be clearly defined, so that developers can correctly configure and use the thread pool in programming, and improve the performance and resource utilization of the system. The thread pool created based on the thread pool specification file can include multiple threads and a task queue. The task queue is used to store image processing tasks to be executed. Ultimately, the threads in the thread pool execute the tasks in the encapsulated task queue.

[0074] In some embodiments of the present application, the visual processing application includes a component or class TaskRunner for executing tasks. Based on this, the visual processing application can assign the generated image processing task to TaskRunner, and then TaskRunner obtains a target thread from the thread pool to execute the image processing task assigned to it, thereby realizing visual processing of image data.

[0075] By creating a thread pool and using the threads in the thread pool to perform image processing tasks, system resources can be effectively utilized and the overhead caused by frequent creation and destruction of threads can be avoided, thereby improving the efficiency and response speed of task processing.

[0076] S13. When the target thread is executing the image processing task and the target operator object is needed, an idle target operator object is obtained from a preset object pool.

[0077] Here, the target operator object is any operator object corresponding to the operator tool described in the target algorithm configuration file.

[0078] In some embodiments of the present application, each Pipeline includes one or more operator tools. When performing an image processing task, it is usually necessary to create the operator tools used in the Pipeline corresponding to the image processing task and generate the corresponding operator objects based on the created operator tools. The operator object is an abstract representation of a specific operator. It defines the properties, behaviors and operation methods of the operator. The operator object can be regarded as a template or blueprint that describes the characteristics and functions of a specific type of operator. For example, an edge detection operator object may have parameters such as detection algorithm type (such as Sobel operator, Canny operator, etc.), threshold setting and other attributes, as well as methods for performing edge detection operations. Considering that creating operator tools and operator objects when performing image processing tasks requires more resources, in order to reduce resource usage, in this embodiment, after the visual processing application is started, before receiving image data, an object pool is first created, and the object pool includes multiple created operator objects. In this way, when a certain type of target operator object is used in the process of the target thread performing the image processing task, the target thread can directly take out an idle target operator object from the object pool without real-time creation.

[0079] In some embodiments of the present application, the visual processing application includes a pool management tool PoolManager. In the object pool, PoolManager is responsible for creating and initializing the object pool. It determines the initial number of objects in the object pool, the type of object (such as operator object), and the configuration parameters of the object. During the operation of the entire software system, PoolManager continuously monitors the status of the object pool to ensure the normal operation of the object pool. When there is a request to obtain an object from the object pool, PoolManager coordinates resource allocation. It quickly locates an available object and provides it to the requester. Based on this, when the target thread needs to use a certain type of target operator object, it can obtain an idle target operator object of this type from the object pool through PoolManager.

[0080] By effectively managing the creation and destruction of objects, PoolManager can significantly improve system performance. It avoids the overhead of frequently creating new objects, such as memory allocation, object initialization, etc. At the same time, fast object allocation also reduces the waiting time of requests and improves the responsiveness of the system. For example, in a high-concurrency application scenario, PoolManager can ensure that multiple requests can quickly obtain the required objects without affecting the overall performance of the system due to delays in object creation.

[0081] S14. When the target operator object is obtained, the image data is processed by using the target operator object.

[0082] In some embodiments of the present application, after the target thread obtains the target operator object from the object pool, TaskRunner performs corresponding business operations through the obtained target operator object to realize the processing of image data. Among them, the relevant operations performed by the target operator object are related to the type of the target operator object, and the type of the target operator object includes but is not limited to Gaussian filtering, median filtering, affine transformation, deep algorithm processing, etc.

[0083] The technical solution provided by the embodiment of the present application is that the visual processing application pre-creates a thread pool and an object pool. When receiving image data, a target thread can be directly allocated from the thread pool to execute the image processing task corresponding to the image data, and in the process of executing the image processing task, the target operator object required to be used is directly obtained from the object pool to process the image data. In this way, when visually processing image data, there is no need to frequently create and destroy threads and operator objects, thereby reducing the time required for visual processing and reducing resource overhead, thereby improving visual processing efficiency and reducing the occurrence of resource exhaustion problems.

[0084] In some embodiments, Figure 3 As shown, in the above step S12, a target thread is allocated through a preset thread pool to execute the image processing task, which may include the following steps S121-S123.

[0085] S121. Determine whether a first number of threads running in the thread pool is less than a thread pool threshold.

[0086] Here, the thread pool threshold is the maximum number of threads that can run in parallel in the thread pool. The thread pool threshold can be preset according to the hardware resources of the industrial computer. For example, when the hardware resources are large, the thread pool threshold can be set to a large value, and when the hardware resources are small, the thread pool threshold can be set to a small value.

[0087] S122. When the first number is less than the thread pool threshold, a target thread is allocated from the idle threads in the thread pool to execute the image processing task.

[0088] In some embodiments of the present application, when the first number is less than the thread pool threshold, it means that the number of threads currently running in parallel in the thread pool has not reached the set upper limit, and the visual processing application still has sufficient resources for the image processing task to run. Therefore, at this time, a target thread can be directly allocated to the image processing task from the idle threads, so that the image processing task can be processed without waiting.

[0089] S123. When the first number is greater than or equal to the thread pool threshold, return to the step of determining whether the first number of threads running in the thread pool is less than the thread pool threshold.

[0090] In some embodiments of the present application, when the first number is greater than or equal to the thread pool threshold, it means that the number of threads currently running in parallel in the thread pool has reached the set upper limit. If the target thread is continuously assigned to the image processing task, the resources of the visual processing application may be exhausted, resulting in a system crash. Therefore, the target thread is not assigned to the image processing task at this time, but the above step S121 is returned to execute, and the target thread is assigned to the image processing task after the first number is less than the thread pool threshold. In this way, the crash problem caused by multi-threaded contention for resources can be avoided.

[0091] In some embodiments of the present application, when the first number is greater than or equal to the thread pool threshold, the image processing task can be placed in the task queue of the thread pool and wait for execution. After the target thread is assigned to the image processing task, the target thread extracts the image processing task from the task queue and executes it.

[0092] Through the above technical solution, the number of threads running in parallel in the thread pool can be controlled and managed, avoiding system crashes or instability caused by too many threads competing for resources, which helps to enhance the stability and reliability of visual processing applications.

[0093] In some embodiments, Figure 4 As shown, after the above step S14, the following step S15 may also be performed:

[0094] S15. After the image data is processed by the target operator object obtained from the object pool, the target operator object is put back into the object pool through the pool management tool.

[0095] In some embodiments of the present application, after the target operator object is obtained from the object pool through the PoolManager, after the image data is processed through the target operator object, that is, after the target operator object is used up, the target operator object is not directly destroyed, but the PoolManager is notified to put the target operator object back into the object pool for subsequent use. When an operator object is returned to the object pool after use, the PoolManager is responsible for receiving and processing the object, and may perform some cleanup or reset operations for the next allocation and use.

[0096] The above solution can avoid frequent creation and destruction of operator objects and reduce resource consumption.

[0097] In some embodiments, considering that the object pool contains a limited number of operator objects, when a target operator object is needed, it may be impossible to obtain the target operator object from the object pool due to reasons such as the target operator object in the object pool being occupied. Figure 5As shown, after the above step S13, the following steps S16-S17 may also be performed.

[0098] S16. If the target operator object is not obtained, create a target operator object.

[0099] In some embodiments of the present application, when a target operator object of a certain type is used during the execution of an image processing task by a target thread, the target thread will take out an idle target operator object from the object pool. If there is no idle target operator object in the object pool, the target operator object cannot be obtained from the object pool. At this time, in order to enable the target thread to continue to execute the image processing process, the operator object can be dynamically increased according to the specific operator usage, and a new target operator object can be created through the PoolManager.

[0100] S17. After the target operator object is created, the image data is processed through the target operator object.

[0101] In some embodiments of the present application, after completing the creation of the target operator object, the target thread may perform corresponding operations through the created target operator object to process the image data, thereby ensuring the smooth execution of the image processing task.

[0102] Through the above technical solution, the interruption of the image processing task caused by the inability to obtain the target operator object from the object pool is avoided, thereby improving the task processing efficiency.

[0103] In some embodiments, after creating the target operator object, the following steps may also be performed:

[0104] After the created target operator object is used up, the target operator object is destroyed.

[0105] Through the above technical solution, the total number of operator objects in the object pool can be limited to avoid memory leaks or insufficient resources caused by unlimited creation of operator objects.

[0106] In some embodiments, in order to avoid the situation where the target operator object cannot be obtained from the object pool as much as possible, Figure 6 As shown, before obtaining an idle target operator object from the preset object pool, the following step S10 may be performed first:

[0107] S10. Initialize the object pool based on the algorithm profile and thread pool threshold loaded in the vision processing application.

[0108] In some embodiments of the present application, as described above, when the visual processing application is started, one or more algorithm configuration files can be loaded from a third-party database into the memory corresponding to the visual processing application. Among them, one algorithm configuration file corresponds to one or more operator tools, and different algorithm configuration files can correspond to different operator tools. In order to improve the success rate of obtaining idle target operator objects from the object pool and to make better use of hardware resources, after the visual processing application is started, the visual processing application can first initialize the object pool based on the algorithm configuration file loaded in its memory and the thread pool threshold to ensure that the object pool contains at least one operator object of each type of operator tool corresponding to the algorithm configuration file.

[0109] In some embodiments of the present application, the PoolManager can be used to initialize the object pool. When initializing the object pool, determine the operator objects and the number of each type of operator objects contained in the algorithm configuration file loaded in the visual processing application; determine the initial number of each type of operator objects based on the number of each type of operator objects contained in the algorithm configuration file and the thread pool threshold; pre-build each type of operator object based on the initial number of each type of operator object; put the pre-built operator objects into the object pool to initialize the object pool. Among them, the initial number of each type of operator objects is greater than or equal to 1. In order to reduce resource usage, the initial number of each type of operator objects can be less than or equal to the total number of operator objects of this type in the loaded algorithm configuration file.

[0110] In some embodiments of the present application, the hardware resources of the industrial computer can be combined to determine the initial number of each type of operator object that meets the engineering requirements by numerical optimization. For example, N algorithm configuration files are loaded in the visual processing application, and the N algorithm configuration files correspond to N pipelines. The N pipelines contain M types of operator objects. The total number of each type of operator object in the M types of operator objects in the N pipelines is determined respectively. For each type of operator object, the interval consisting of 1 and the total number corresponding to it is used as the number value interval of the type of operator object. In this way, M number value intervals corresponding to the M types of operator objects can be obtained, and multiple number combinations are obtained based on the M number value intervals. Each number combination includes a number corresponding to each type of operator object. Then, the numerical optimization method is used to try multiple number combinations, predict the resources occupied by the object pool after initialization based on each number combination, and predict the work efficiency of visual processing based on the initialized object pool, so as to determine the optimal number combination that meets the engineering needs, and use the number corresponding to each type of operator object in the optimal number combination as the initial number of each type of operator object. Among them, the engineering needs to be set according to the actual situation.

[0111] Through the above technical solution, the object pool is initialized based on the algorithm configuration file in the visual processing application and the preset thread pool specification file, which can improve the success rate of obtaining idle target operator objects from the object pool and make better use of hardware resources.

[0112] In some embodiments, Figure 7 As shown, after the above step S10, the following steps S71-S73 may also be performed.

[0113] S71. Obtaining the system load of the visual processing application;

[0114] S72. When the system load is greater than the first load threshold, create an operator object corresponding to the operator tool described in the algorithm configuration file loaded in the visual processing application, and put the created operator object into the object pool;

[0115] S73. When the system load is less than a second load threshold, destroy the operator objects in the object pool that meet the destruction conditions, wherein the second load threshold is less than or equal to the first load threshold.

[0116] Here, the first load threshold and the second load threshold can be preset according to actual conditions.

[0117] In some embodiments of the present application, the system load of the visual processing application refers to the measurement of the working pressure or resource usage of the system in a specific time period. The system load can be measured from multiple aspects, including the following common aspects: CPU usage: indicates the proportion of time the CPU is used to process tasks. If the CPU usage is always high, it means that the system has a large number of computing tasks to process during this time period. Memory usage: reflects the size of the memory occupied by the application. When the memory usage is close to the total physical memory of the system, it may cause insufficient memory problems and affect system performance. Disk I / O activity: includes the frequency and amount of disk read and write operations. Frequent disk I / O may cause the system to respond slowly. When a large number of image processing tasks are processed at the same time in the visual processing system, it will cause high CPU usage and a large number of disk I / O operations, thereby increasing the system load. Therefore, the size of the system load can reflect the number of image processing tasks executed in the visual processing system. When the system load is large, the image processing tasks executed in the visual processing system are more, and more operator objects need to be used. When the system load is small, the image processing tasks executed in the visual processing system are fewer, and fewer operator objects need to be used. Based on this, when the system load of the visual processing application is greater than the first load threshold, in order to improve the visual processing efficiency, an operator object corresponding to the operator tool described in the algorithm configuration file loaded in the visual processing application is created, and the created operator object is placed in the object pool, so that when the operator object is needed, it can be directly obtained from the object pool. When the system load of the visual processing application is less than the second load threshold, in order to reduce the resources occupied by the object pool, the operator objects in the object pool that meet the destruction conditions are destroyed.

[0118] In some embodiments of the present application, PoolManager can monitor the usage of the object pool and collect various performance indicators, such as the average usage time of the object, the vacancy rate of the object pool, etc. Based on these indicators, it can dynamically adjust the size and configuration of the object pool to adapt to different load conditions. If it is found that the objects in the object pool are often in a full load state, PoolManager can increase the size of the object pool to meet more requests. Conversely, if the objects in the object pool are idle for a long time, PoolManager can appropriately reduce the size of the object pool to save system resources. In the absence of an effective management mechanism, a large number of object creation requests may cause system resources to be exhausted, causing the system to crash. PoolManager can effectively prevent this from happening by limiting the size of the object pool and allocating resources reasonably. For example, when the system faces a large number of sudden requests, PoolManager can ensure that the object pool will not expand indefinitely, thereby avoiding problems such as memory overflow.

[0119] Through the above technical solution, the number of operator objects in the object pool is automatically adjusted dynamically according to the system load, which can ensure that there are enough objects available in high-concurrency scenarios and that too many resources are not wasted to maintain redundant objects under low load.

[0120] In some embodiments, the destruction condition includes at least one of the following:

[0121] The duration for which the operator object has not been used is greater than the duration threshold.

[0122] The number of operator objects of the same type is greater than the quantity threshold.

[0123] Here, both the duration threshold and the quantity threshold can be preset according to actual conditions.

[0124] In some embodiments of the present application, when the system load of the visual processing application is less than a second load threshold, the length of time that each operator object in the object pool has not been used can be determined, and the length of time that each operator object has not been used can be compared with the length of time threshold. When the length of time that the operator object has not been used is greater than the length of time threshold, the operator object can be considered to be an unnecessary operator object, and thus it can be deleted from the object pool, thereby reducing resource occupancy and improving resource utilization.

[0125] In other embodiments of the present application, when the system load of the visual processing application is less than the second load threshold, the number of each type of operator objects contained in the object pool can be determined, and the number of each type of operator objects can be compared with the number threshold respectively. If the number of a certain type of operator objects is greater than the number threshold, it is considered that the number of operator objects of this type is large. At this time, the number of operator objects of this type can be reduced to reduce the number of operator objects of this type to less than or equal to the number threshold. In this way, sharing between multiple operator objects can be achieved as much as possible, rather than storing each operator object separately, thereby reducing resource occupation and improving resource utilization.

[0126] In some embodiments, in order to further limit and weaken the pressure of parallel operation and achieve smooth operation under the maximum resource utilization limit in parallel scenarios, after creating the Pipeline, you can also add a mutex lock to the operator object to limit the operator tool's grabbing of system resources in parallel scenarios through the mutex lock.

[0127] In some embodiments of the present application, after creating a Pipeline, the user can evaluate the hardware resources of the industrial computer, and then determine the operator object combination that may lead to system resource exhaustion based on the evaluation results, and then perform locking operations on the operator objects belonging to the operator object combination in the Pipeline, thereby limiting the number of operator objects that can run simultaneously in the operator object combination, and avoiding resource exhaustion caused by the simultaneous operation of all operator objects in the operator object combination.

[0128] In some embodiments of the present application, Figure 8 As shown, when locking the operator objects in the operator object combination, the mutex lock can be used to enclose the outer shell of each operator object in the operator object combination, so that the input data of the operator object is first input to the mutex lock, and when the number of operator objects running in parallel in the mutex lock is less than the object threshold, the mutex lock then inputs the input data to the operator object for processing, and finally transmits the output data of the operator object to the mutex lock for output. Among them, the object threshold is used to indicate the maximum number of operator objects that can run in parallel in the operator object combination corresponding to the lock tool.

[0129] Generally, in visual processing scenarios, when a pipeline is running or multiple identical or different pipelines are running in parallel, different operator objects may run at the same time. Multiple different operator objects running at the same time will compete for resources, which may lead to resource exhaustion and system crash. In view of this, in order to avoid resource exhaustion caused by resource competition among multiple operator objects, the combination of these multiple operator objects that may be in the same parallel scenario can be used as an operator object combination, and the operator objects in the operator object combination can be mutually exclusive through a mutex lock to limit the number of operator objects running in parallel in the operator object combination, thereby avoiding resource exhaustion.

[0130] As an example, Fig. 9 As shown in the figure, Pipeline1 and Pipeline2 are two different Pipelines, where Q, E, A1, B, A2, C1, A3, C2, T, R, and V all represent operator objects. Fig. 9As shown, in Pipeline1 and Pipeline2, operator objects A1, B, A2, C1, A3 and C2 may run at the same time, among which A1, A2, A3 operator objects are Gaussian filters, B operator object belongs to the free transformation operator instance, C1, C2 operator objects belong to median filters. When these 6 operator objects run at the same time, they compete for resources, which may lead to resource exhaustion. Therefore, these 6 operator objects are used as an operator object combination corresponding to a mutex lock, among which each operator object corresponds to different inputs, outputs and various attributes. Mutex locks can be shelled for each operator object through parameter configuration, and different operator objects can be bound to the same mutex lock through the lock name. In this way, when the operator object runs, lock judgment and mutual exclusive operation can be performed based on the lock name. For example, the operator object combination corresponding to the mutex lock Lk07 includes three operator objects: Tool1, Tool2, and Tool3. Then, Tool1, Tool2, and Tool3 can be bound to the mutex lock Lk07, and recorded as Lk07_ Tool1, Lk07_ Tool2, and Lk07_ Tool3, respectively.

[0131] The above method of setting up a mutex lock is universal. By associating an operator object with a lock name, different operator objects can be arbitrarily combined to share the mutex lock tool. This applies not only to different operator objects under the same Pipeline, but also to different operator objects in different Pipelines. It supports any combination of operator objects. In this way, in a parallel scenario, the operator objects can be restricted from competing for system resources, thereby ensuring smooth operation of the system with maximum resource utilization.

[0132] Based on the above, if Fig.10 As shown, when the image data is processed by the target operator object in the above step S14 or S17, the following steps S101-S104 may be included.

[0133] S101. When the target operator object belongs to the operator object combination corresponding to the mutex lock, input the input data corresponding to the target operator object into the mutex lock.

[0134] In some embodiments of the present application, it can be seen from the above content that the locking operation of the operator object is implemented by associating the operator object with the lock name of the mutex lock. Therefore, when the image data is processed through the target operator object, it can be determined whether the target operator object belongs to the operator object combination corresponding to the mutex lock by judging whether the lock name is included in the identifier of the target operator object. When the lock name is included in the identifier of the target operator object, it is determined that the target operator object belongs to the operator object combination corresponding to the mutex lock, and the set of all operator objects associated with the lock name is used as the operator object combination corresponding to the mutex lock. When the lock name is not included in the identifier of the target operator object, it is determined that the target operator object does not belong to the operator object combination corresponding to the mutex lock.

[0135] In some embodiments of the present application, when it is determined that the target operator object belongs to the operator object combination corresponding to the mutex, the input data corresponding to the target operator object is first input into the mutex.

[0136] S102. Determine whether a second number of operator objects running in the operator object combination is less than an object threshold, where the object threshold is used to indicate a maximum number of operator objects that can run in parallel in the operator object combination.

[0137] Here, the object threshold can be set according to the size of hardware resources and / or the size of resources occupied by each operator object in the operator object combination. Different object thresholds correspond to different processing logics.

[0138] As an example, see Fig.11 , take the operator object combination corresponding to the mutex lock Lk07 including Tool1, Tool2 and Tool3 as an example, where Tool1, Tool2 and Tool3 are all bound to the mutex lock Lk07, and are recorded as Lk07_ Tool1, Lk07_ Tool2 and Lk07_ Tool3 respectively. Assuming that the object threshold is 1, Lk07_ Tool1, Lk07_ Tool2 and Lk07_ Tool3 are executed serially, for example Fig.11 As shown in (a), the execution order is that the operator object that grabs the mutex lock first is executed first. Assuming the object threshold is 2, the execution order of Lk07_ Tool1, Lk07_ Tool2 and Lk07_ Tool3 is that the two operator objects that grab the mutex lock first are executed first. When the executed operator object releases the resources, the remaining operator objects compete for the lock resources for execution. For example Fig.11 (b). It should be noted that Fig.11 (a) and Fig.11 The execution order in (b) is only an example and does not constitute a limitation on the execution order.

[0139] In some embodiments of the present application, the second number of the running operator objects can be recorded by an atomic variable. Atomic variables can ensure that operations on a single variable are atomic, avoiding data inconsistency problems in a multi-threaded or concurrent environment.

[0140] S103. When the second number is greater than or equal to the object threshold, return to the step of determining the second number of operator objects running in the operator object combination.

[0141] In some embodiments of the present application, when the second number is greater than or equal to the object threshold, it means that the number of operator objects being executed in the operator object combination has reached the set upper limit value. If the target operator object is executed again, the target operator object may compete for resources, resulting in resource exhaustion. Therefore, in order to avoid resource exhaustion, at this time, the target operator object is not executed first, but the process returns to execute the above step S102, and waits until the second number is less than the object threshold before executing the target operator object.

[0142] S104. When the second number is less than the object threshold, input the input data into the target operator object, so as to process the image data through the target operator object.

[0143] In some embodiments of the present application, when the second number is less than the object threshold, it means that the number of operator objects being executed in the operator object combination has not reached the set upper limit value, which means that there are currently sufficient resources to execute the target operator object. Therefore, at this time, the mutex lock can transfer the input data of the target operator object to the target operator object, thereby executing the corresponding business operation through the target operator object.

[0144] Through the above technical solution, by setting a mutex lock, the pressure of parallel operation is limited and weakened, and the mutex lock restriction is continued within the execution task. The optimization granularity is finer to achieve the purpose of smooth operation under the maximum resource utilization limit in the parallel scenario.

[0145] In some embodiments, it may be determined based on the semaphore whether the second number of the operator objects being run in the operator object combination is less than the object threshold. Specifically, Fig.12 As shown, in the above step S102, it can be determined through the following steps S1021-S1023 whether the second number of the operating operator objects in the operator object combination is less than the object threshold.

[0146] S1021. Determine whether the semaphore corresponding to the mutex lock is 0. The initial value of the semaphore is the object threshold. Each time an operator object is run in the operator object combination, the semaphore decreases by one.

[0147] S1022. When the semaphore corresponding to the mutex lock is greater than 0, determine that the second number of the running operator objects in the operator object combination is less than the object threshold.

[0148] S1023. When the semaphore corresponding to the mutex lock is less than or equal to 0, determine whether the second number of the running operator objects in the operator object combination is greater than or equal to the object threshold.

[0149] In some embodiments of the present application, a semaphore is used to control the access of multiple threads to shared resources. The semaphore has an initial value, which is the operator threshold. When an operator object in the operator object combination corresponding to the mutex lock applies for resources, the semaphore is reduced by 1, and when an operator object in the operator object combination releases resources, the semaphore is increased by 1. In this way, using a semaphore to control the number of operator objects that can be run at the same time in the operator object combination can achieve synchronization and mutual exclusion access of resources. Based on this, when the target thread wants to run the target operator object, it can be determined whether the semaphore is greater than 0. If it is greater than 0, it is determined that the second number of operator objects running in the operator object combination is less than the object threshold, and the target operator object can be run, and the semaphore is reduced by 1. If it is not greater than 0, it is determined that the second number of operator objects running in the operator object combination is greater than or equal to the object threshold, and the target operator object cannot be run.

[0150] Based on the production line visual processing method provided in the above embodiment, the present application also provides a specific implementation of the production line visual processing device. Please refer to the following embodiment.

[0151] See also Fig.13 , a schematic diagram of a production line visual processing device provided in an embodiment of the present application, such as Fig.13 As shown, the device 1300 includes the following modules:

[0152] The task creation module 1301 is used to create an image processing task corresponding to the image data in response to receiving the image data, wherein the image processing task is used to instruct to perform visual processing on the image data based on a target algorithm configuration file, wherein the target algorithm configuration file is an algorithm configuration file corresponding to the image data loaded in the visual processing application, and the algorithm configuration file is used to describe the steps of the visual processing and the operator tools used in each step;

[0153] The thread allocation module 1302 is used to allocate a target thread to execute the image processing task through a preset thread pool;

[0154] The object pool management module 1303 is used to obtain an idle target operator object from a preset object pool when a target operator object is needed during the process of the target thread executing the image processing task. The target operator object is any operator object corresponding to the operator tool described in the target algorithm configuration file.

[0155] The task execution module 1304 is used to process the image data through the target operator object when the target operator object is acquired.

[0156] The technical solution provided by the embodiment of the present application is that the visual processing application pre-creates a thread pool and an object pool. When receiving image data, a target thread can be directly allocated from the thread pool to execute the image processing task corresponding to the image data, and in the process of executing the image processing task, the target operator object required to be used is directly obtained from the object pool to process the image data. In this way, when visually processing image data, there is no need to frequently create and destroy threads and operator objects, thereby reducing the time required for visual processing and reducing resource overhead, thereby improving visual processing efficiency and reducing the occurrence of resource exhaustion problems.

[0157] In some embodiments, the thread allocation module 1302 is specifically configured to:

[0158] Determine whether a first number of threads running in the thread pool is less than a thread pool threshold, where the thread pool threshold is a maximum number of threads that can run in parallel in the thread pool;

[0159] When the first number is less than the thread pool threshold, a target thread is allocated from idle threads in the thread pool to execute the image processing task;

[0160] In the case where the first number is greater than or equal to the thread pool threshold, the step of returning to the thread pool to determine whether the first number of threads running in the thread pool is less than the thread pool threshold is executed.

[0161] Through the above technical solution, the number of threads running in parallel in the thread pool can be controlled and managed, avoiding system crashes or instability caused by too many threads competing for resources, which helps to enhance the stability and reliability of visual processing applications.

[0162] In some embodiments, the object pool management module 1303 is further used to:

[0163] After the image data is processed by the target operator object obtained from the object pool, the target operator object is put back into the object pool through the pool management tool.

[0164] The above solution can avoid frequent creation and destruction of operator objects and reduce resource consumption.

[0165] In some embodiments, the object pool management module 1303 is further used to:

[0166] After obtaining an idle target operator object from the object pool, if the target operator object is not obtained, create the target operator object;

[0167] The task execution module 1304 is also used to process the image data through the target operator object after the target operator object is created.

[0168] Through the above technical solution, the interruption of the image processing task caused by the inability to obtain the target operator object from the object pool is avoided, thereby improving the task processing efficiency.

[0169] In some embodiments, the object pool management module 1303 is further used to:

[0170] Initializes the object pool based on the algorithm profile and thread pool threshold loaded in the vision processing application.

[0171] In some embodiments, the object pool management module 1303 is used to:

[0172] Determine the operator objects and the number of each type of operator objects contained in the algorithm configuration file loaded in the visual processing application; determine the initial number of each type of operator objects based on the number of each type of operator objects contained in the algorithm configuration file and the thread pool threshold; pre-build each type of operator object based on the initial number of each type of operator object; put the pre-built operator objects into the object pool to initialize the object pool.

[0173] By initializing the object pool through the above technical solution, the success rate of obtaining idle target operator objects from the object pool can be improved, and hardware resources can be more fully utilized.

[0174] In some embodiments, the object pool management module 1303 is further used to:

[0175] After initializing the object pool based on the algorithm configuration file in the vision processing application and the thread pool specification file for the preset, the system load of the vision processing application is obtained;

[0176] When the system load is greater than a first load threshold, an operator object corresponding to an operator tool described in an algorithm configuration file loaded in the visual processing application is created, and the created operator object is placed in an object pool;

[0177] When the system load is less than a second load threshold, operator objects in the object pool that meet the destruction condition are destroyed, wherein the second load threshold is less than or equal to the first load threshold.

[0178] Through the above technical solution, the number of operator objects in the object pool is automatically adjusted dynamically according to the system load, which can ensure that there are enough objects available in high-concurrency scenarios and that too many resources are not wasted to maintain redundant objects under low load.

[0179] In some embodiments, the destruction condition includes at least one of the following:

[0180] The duration for which the operator object has not been used is greater than the duration threshold.

[0181] The number of operator objects of the same type is greater than the quantity threshold.

[0182] Through the above technical solution, by destroying operator objects that have not been used for a long time and / or an excessive number of operator objects, resource occupation can be reduced and resource utilization can be improved.

[0183] In some embodiments, the task execution module 1304 is specifically configured to:

[0184] In the case where the target operator object belongs to the operator object combination corresponding to the mutex lock, inputting the input data corresponding to the target operator object into the mutex lock;

[0185] Determine whether a second number of operator objects running in the operator object combination is less than an object threshold, where the object threshold is used to indicate a maximum number of operator objects that can run in parallel in the operator object combination;

[0186] If the second number is greater than or equal to the object threshold, returning to the step of determining the second number of operator objects running in the operator object combination;

[0187] When the second number is less than the object threshold, the input data is input into the target operator object so as to process the image data through the target operator object.

[0188] Through the above technical solution, by setting a mutex lock, the pressure of parallel operation is limited and weakened, and the mutex lock restriction is continued within the execution task. The optimization granularity is finer to achieve the purpose of smooth operation under the maximum resource utilization limit in the parallel scenario.

[0189] In some embodiments, the task execution module 1304 is specifically configured to:

[0190] Determine whether the semaphore corresponding to the mutex lock is 0. The initial value of the semaphore is the object threshold. Each time an operator object is run in the operator object combination, the semaphore decreases by one.

[0191] When the semaphore corresponding to the mutex lock is greater than 0, it is determined that the second number of the running operator objects in the operator object combination is less than the object threshold.

[0192] Through the above technical solution, it is possible to quickly and accurately determine whether the second number of operator objects running in parallel in the operator object combination exceeds the object threshold based on the semaphore.

[0193] In some embodiments, the task execution module 1304 is further configured to:

[0194] Before inputting the input data corresponding to the target operator object into the mutex lock, determine whether the identifier of the target operator object contains the lock name of the mutex lock;

[0195] When the identifier of the target operator object includes the lock name of the mutex lock, it is determined that the target operator object belongs to the operator object combination corresponding to the mutex lock, and the operator object combination corresponding to the mutex lock is a set of operator objects whose identifiers include the lock name;

[0196] The operator object combination includes at least one of the following:

[0197] Different operator objects corresponding to the same algorithm configuration file;

[0198] Different algorithm configuration files correspond to different operator objects.

[0199] Through the above technical solution, by associating the lock name of the mutex lock with the operator object, the locking operation of the operator object is implemented, which is universal and can be applied to various parallel scenarios.

[0200] The visual processing device provided in the embodiment of the present application can achieve Figures 1 to 12 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0201] Fig.14 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0202] The electronic device 1400 may include a processor 1401 and a memory 1402 storing computer program instructions.

[0203] Specifically, the processor 1401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0204] The memory 1402 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 1402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 1402 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 1402 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 1402 is a non-volatile solid-state memory.

[0205] The memory 1402 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory 1402 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in any one of the visual processing methods in the above embodiments.

[0206] The processor 1401 implements any one of the visual processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 1402.

[0207] In one example, the electronic device 1400 may further include a communication interface 1403 and a bus 1410. Fig.14 As shown, the processor 1401, the memory 1402, and the communication interface 1403 are connected via a bus 1410 and communicate with each other.

[0208] The communication interface 1403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0209] Bus 1410 includes hardware, software or both, and couples the components of online data traffic billing equipment to each other. For example, but not limitation, the bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnect (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. Where appropriate, bus 1410 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0210] In addition, in combination with the visual processing method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the visual processing methods in the above embodiment is implemented.

[0211] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements any one of the visual processing methods in the above embodiments.

[0212] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0213] The functional blocks shown in the structural block diagram described above can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier. "Machine-readable medium" may include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0214] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0215] Aspects of the present disclosure are described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / actions specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0216] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A production line visual processing method, characterized in that: include: In response to receiving the image data, creating an image processing task corresponding to the image data, wherein the image processing task is used to instruct visual processing of the image data based on a target algorithm configuration file, wherein the target algorithm configuration file is an algorithm configuration file corresponding to the image data loaded in a visual processing application, and the algorithm configuration file is used to describe the steps of visual processing and the operator tools used in each step; Allocating a target thread through a preset thread pool to execute the image processing task; In the process of the target thread executing the image processing task, when a target operator object is needed, an idle target operator object is obtained from a preset object pool, wherein the target operator object is any operator object corresponding to the operator tool described in the target algorithm configuration file; When the target operator object is acquired, the image data is processed by using the target operator object; The processing of the image data by using the target operator object includes: In a case where the target operator object belongs to the operator object combination corresponding to the mutex lock, inputting input data corresponding to the target operator object into the mutex lock; Determining whether a second number of the operator objects being run in the operator object combination is less than an object threshold, wherein the object threshold is used to indicate a maximum number of operator objects that can be run in parallel in the operator object combination; In the case where the second number is greater than or equal to the object threshold, returning to execute the step of determining whether the second number of the operator objects being run in the operator object combination is less than the object threshold; In a case where the second number is less than an object threshold, the input data is input into the target operator object so as to process the image data through the target operator object.

2. The method according to claim 1, characterized in that The step of allocating a target thread through a preset thread pool to execute the image processing task includes: Determine whether a first number of threads running in the thread pool is less than a thread pool threshold, where the thread pool threshold is a maximum number of threads that can run in parallel in the thread pool; When the first number is less than the thread pool threshold, allocating a target thread from idle threads in the thread pool to execute the image processing task; In the case where the first number is greater than or equal to the thread pool threshold, the step of returning to determine whether the first number of threads running in the thread pool is less than the thread pool threshold is executed.

3. The method according to any one of claims 1-2, characterized in that: The method further comprises: After the image data is processed by the target operator object obtained from the object pool, the target operator object is put back into the object pool through a pool management tool.

4. The method according to claim 1, characterized in that: The method further comprises: If the target operator object is not obtained, create the target operator object; After the target operator object is created, the image data is processed by the target operator object.

5. The method according to claim 2, characterized in that: The method further comprises: The object pool is initialized based on the algorithm configuration file loaded in the visual processing application and the thread pool threshold.

6. The method according to claim 5, characterized in that Initializing the object pool based on the algorithm configuration file loaded in the visual processing application and the thread pool threshold comprises: Determining operator objects and the number of each type of operator objects contained in an algorithm configuration file loaded in the visual processing application; Determine the initial number of each type of operator objects according to the number of each type of operator objects contained in the algorithm configuration file and the thread pool threshold; Pre-build each type of operator object based on the initial number of each type of operator objects; Put the pre-built operator object into the object pool to initialize the object pool.

7. The method according to any one of claims 5-6, characterized in that: The method further comprises: Obtaining a system load of the visual processing application; When the system load is greater than a first load threshold, create an operator object corresponding to an operator tool described in an algorithm configuration file loaded in the visual processing application, and put the created operator object into the object pool; When the system load is less than a second load threshold, destroying operator objects in the object pool that meet the destruction condition, wherein the second load threshold is less than or equal to the first load threshold.

8. The method according to claim 7, characterized in that The destruction conditions include at least one of the following: The duration for which the operator object has not been used is greater than the duration threshold. The number of operator objects of the same type is greater than the quantity threshold.

9. The method according to claim 1, characterized in that: The determining whether the second number of the operator objects being run in the operator object combination is less than the object threshold includes: Determine whether the semaphore corresponding to the mutex lock is 0, the initial value of the semaphore is the object threshold, and the semaphore decreases by one each time an operator object in the operator object combination is run; When the semaphore corresponding to the mutex lock is greater than 0, it is determined that the second number of the running operator objects in the operator object combination is less than the object threshold.

10. The method according to claim 1, characterized in that The method further comprises: Determine whether the identifier of the target operator object contains the lock name of the mutex lock; In a case where the identifier of the target operator object includes the lock name of the mutex lock, determining that the target operator object belongs to the operator object combination corresponding to the mutex lock, where the operator object combination corresponding to the mutex lock is a set of operator objects whose identifiers include the lock name; The operator object combination includes at least one of the following: Different operator objects corresponding to the same algorithm configuration file; Different algorithm configuration files correspond to different operator objects.

11. A production line visual processing device, characterized in that: include: a task creation module, configured to create an image processing task corresponding to the image data in response to receiving the image data, wherein the image processing task is used to instruct visual processing of the image data based on a target algorithm configuration file, wherein the target algorithm configuration file is an algorithm configuration file corresponding to the image data loaded in a visual processing application, and the algorithm configuration file is used to describe the steps of visual processing and the operator tools used in each step; A thread allocation module, used for allocating a target thread through a preset thread pool to execute the image processing task; An object pool management module, used for the target thread to obtain an idle target operator object from a preset object pool when the target operator object is needed during the execution of the image processing task, wherein the target operator object is any operator object corresponding to the operator tool described in the target algorithm configuration file; A task execution module, configured to process the image data through the target operator object when the target operator object is acquired; The task execution module is specifically used for: In a case where the target operator object belongs to the operator object combination corresponding to the mutex lock, inputting input data corresponding to the target operator object into the mutex lock; Determining whether a second number of the operator objects being run in the operator object combination is less than an object threshold, wherein the object threshold is used to indicate a maximum number of operator objects that can be run in parallel in the operator object combination; In the case where the second number is greater than or equal to the object threshold, returning to execute the step of determining whether the second number of the operator objects being run in the operator object combination is less than the object threshold; In a case where the second number is less than an object threshold, the input data is input into the target operator object so as to process the image data through the target operator object.

12. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the production line vision processing method as described in any one of claims 1-10 is implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the production line vision processing method according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is enabled to execute the production line vision processing method as described in any one of claims 1 to 10.

Images