Efficient algorithm Pipeline method based on AI open platform

Abstract

The invention relates to an efficient algorithm Pipeline method based on an AI open platform. The efficient algorithm Pipeline method comprises the following steps: S1, establishing a structure of an algorithm Pipeline; s2, establishing a structure of an algorithm Pipeline based on the step S1, and performing implementation and function design on a plurality of independent component links; and S3, on the basis of the step S2, performing parallel computing optimization on each independent component link. The system is composed of a plurality of configurable component links, and each link executes a specific data processing task by mounting different processing functions, so that the processing speed of the whole system is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of algorithm application, and relates to an algorithm pipeline method, and in particular to an efficient algorithm pipeline method based on an AI open platform. Background Art

[0002] As AI application scenarios become increasingly complex and diverse, the requirements for algorithm application technology are also increasing. Existing algorithm application technology can no longer meet actual needs. Many existing algorithm processing pipelines may be too dependent on a single algorithm or data processing requirements in a specific field.

[0003] This reliance on a single algorithm may result in low efficiency when processing complex data structures or multi-step analysis tasks. Some pipelines fail to fully utilize parallel computing technology when processing large-scale data, resulting in suboptimal processing speed.

[0004] In addition, most pipelines determine a static algorithm execution order during design and cannot be dynamically adjusted based on the characteristics of actual data and task requirements. Therefore, the efficiency varies greatly under different data conditions.

[0005] Some pipelines also lack adaptability to complex tasks and different data types, and cannot effectively respond to real-time data processing and diverse analysis needs.

[0006] Therefore, the present invention proposes an efficient algorithm Pipeline method based on an AI open platform, which can solve the above problems.

[0007] After searching, no public documents of the prior art identical or similar to the present invention were found. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and propose an efficient algorithm Pipeline method based on an AI open platform, which consists of multiple configurable component links. Each link performs a specific data processing task by mounting different processing functions, thereby improving the processing speed of the overall system.

[0009] The present invention solves the practical problem by adopting the following technical solutions:

[0010] An efficient algorithm pipeline method based on an AI open platform includes the following steps:

[0011] S1: Establish the structure of the algorithm pipeline;

[0012] S2: Based on step S1, the structure of the algorithm pipeline is established, and multiple independent component links are implemented and functionally designed;

[0013] S3. Based on step S2, parallel computing optimization is performed on each independent component link.

[0014] Moreover, the specific method of step S1 is:

[0015] The algorithm pipeline is composed of multiple independent component links, each link is responsible for performing a specific data processing task;

[0016] Its overall structure includes:

[0017] Input module: responsible for receiving data from the external data source and passing it to the first link.

[0018] Each link: Each link is an independent processing component with its own processing function and task execution capability. Each link can specify its specific tasks and functions through configuration files or programming interfaces.

[0019] Moreover, the specific method of step S2 is:

[0020] Split the execution steps into multiple independent "links", each link implements a specific function;

[0021] Each link consists of three parts: thread, communication mailbox and processing function;

[0022] Independent threads enable each link to run independently without being affected by other functions in the system;

[0023] The communication mailbox is used to provide a mechanism for link to interact with the outside world. It can receive external commands or parameters and respond to execute corresponding tasks.

[0024] At the same time, the link's processing tasks are implemented through processing functions, and processing functions can be configured through callback registration. This allows the same link to register different processing functions as needed to perform different tasks.

[0025] Moreover, the specific method of step S3 is:

[0026] Pipeline takes advantage of the parallel performance of modern computing resources, and each link independently utilizes multi-core processors or distributed computing environments.

[0027] Advantages and beneficial effects of the present invention:

[0028] The present invention provides an efficient algorithm pipeline method based on an AI open platform. Through componentized design and a flexible task switching mechanism, it effectively improves the efficiency of data processing and the flexibility of application, and has significant market competitive advantages and broad application prospects. It has flexibility and configurability, and can dynamically adjust and configure the execution order and functions of each link according to actual needs, and adapt to different data types and processing scenarios; high efficiency improves the efficiency and speed of data processing through parallel computing technology and task optimization, especially when processing large-scale data; and wide application, it is suitable for a variety of artificial intelligence fields, such as image recognition, speech processing, natural language processing, etc., and can quickly build and deploy various complex algorithm applications, which are specifically reflected in:

[0029] (1) Modular design: Pipeline consists of multiple independent components (links), each of which is responsible for a specific data processing task. Each link is an independent module with clear responsibilities and processing capabilities, making the entire Pipeline highly modular. This design makes the system structure clear and easy to maintain and expand.

[0030] (2) Flexible task switching: By mounting different processing functions, each link can dynamically switch tasks according to actual needs. This flexibility not only supports different data processing requirements, but also allows for rapid adjustment and optimization of algorithms, improving the adaptability and availability of the Pipeline.

[0031] (3) Dynamic adjustment and configuration: Allowing users to adjust the execution order and functions of each link according to actual needs, so that the system can flexibly adapt to different data types and processing scenarios, improving the system's adaptability and user customization capabilities.

[0032] (4) Parallel computing technology: Parallel computing can accelerate data processing, significantly improving processing speed and efficiency when processing large-scale data, and effectively reducing processing time.

[0033] (5) Task optimization mechanism: Optimize the task execution process, reduce the waste of computing resources, and improve the overall system performance, so that the algorithm pipeline can complete tasks more efficiently in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the Pipeline structure of the algorithm of the present invention. DETAILED DESCRIPTION

[0035] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0036] An efficient algorithm pipeline method based on an AI open platform includes the following steps:

[0037] S1: Establish the structure of the algorithm pipeline;

[0038] The specific method of step S1 is:

[0039] The algorithm pipeline is composed of multiple independent components (called links), each of which is responsible for performing a specific data processing task;

[0040] Its overall structure includes:

[0041] Input module: responsible for receiving data from external data sources (such as files, network interfaces, etc.) and passing it to the first link.

[0042] Each link: Each link is an independent processing component with its own processing function and task execution capability. Each link can specify its specific tasks and functions through configuration files or programming interfaces.

[0043] In this embodiment, as shown in Figure 1, the algorithm Pipeline includes:

[0044] Data preprocessing link: responsible for preliminary processing of input data, such as data cleaning, format conversion, etc.

[0045] Feature extractionlink: Use specific algorithms to extract key features from processed data.

[0046] Model traininglink: Train the machine learning model based on the extracted features.

[0047] Model evaluation link: Evaluate and tune the trained model.

[0048] Result output link: Output the final processing result to a specified destination, such as a database, file system, or network service.

[0049] Output module: responsible for outputting or feeding back the final processing results to users or other systems.

[0050] S2: Based on step S1, the structure of the algorithm pipeline is established, and multiple independent component links are implemented and functionally designed;

[0051] The specific method of step S2 is:

[0052] In the actual application scenario of the algorithm, we can split the execution steps into multiple independent "links", each link implements a specific function.

[0053] Each link consists of three parts: thread, communication mailbox and processing function.

[0054] Independent threads enable each link to run independently without being affected by other functions in the system, thereby concentrating resources on processing tasks more efficiently.

[0055] The communication mailbox is used to provide a mechanism for link to interact with the outside world. It can receive external commands or parameters and respond to execute corresponding tasks, which simplifies communication and synchronization problems in a multi-threaded environment.

[0056] At the same time, the link's processing tasks are implemented through processing functions, and processing functions can be configured through callback registration. This allows the same link to register different processing functions as needed to perform different tasks.

[0057] In this embodiment, the algorithm Pipeline consists of a data link composed of multiple links, each of which has an independent thread and processing function. The multi-threaded concurrent mechanism is used to process different tasks, realizing parallel processing and asynchronous execution of different modules within the algorithm; the link processing function can be customized according to the specific application scenario and algorithm, and adapted to the characteristics and data structure of different algorithms, thereby improving execution efficiency. For example, in the human detection algorithm, the face detection function and the body detection function can be flexibly switched to realize the detection of faces and bodies respectively.

[0058] Traditional algorithm pipelines usually execute all steps in series, which leads to low execution efficiency. For example, in the face recognition algorithm, background modeling, face detection, attribute detection, and snapshot are each assumed to take 50 milliseconds, and it takes a total of 200 milliseconds to execute all steps in series. By linking to build an asynchronously processed Pipelink, these four steps can be set as independent links. In theory, the execution time of all steps can be controlled between 50 milliseconds and 60 milliseconds, greatly improving execution efficiency.

[0059] In addition, traditional algorithms can usually only be executed in a fixed order (such as background modeling → face detection → attribute detection → snapshot). When the execution order needs to be adjusted (for example: background modeling → face detection → snapshot → attribute detection), the entire code often needs to be re-implemented, wasting development efficiency. However, using the asynchronous processing Pipelink built with link, you only need to simply exchange the execution functions of the corresponding links of attribute detection and snapshot to quickly adjust the algorithm Pipeline, which greatly simplifies the development process and improves development efficiency.

[0060] Finally, when the data structure or process in a certain step changes, the traditional algorithm may need to implement all the processes from scratch. However, with the asynchronous processing Pipelink built through link, you only need to re-implement the execution function of the link corresponding to the attribute detection adapted to the new data structure, which also simplifies the development process and improves development efficiency.

[0061] S3: Parallel computing optimization for each independent component link

[0062] The specific method of step S3 is:

[0063] In order to achieve efficient data processing and computing, the Pipeline utilizes the parallel performance of modern computing resources. Each link can independently utilize a multi-core processor or a distributed computing environment to achieve optimal processing speed and efficiency.

[0064] The present invention is further described below by means of specific examples:

[0065] The algorithm Pipeline of the present invention is not only suitable for face detection, but also can be applied to multiple fields such as image recognition, speech processing, natural language processing, etc. Its key advantages include flexible task configuration, efficient parallel processing capability, and the ability to quickly build and deploy, which can significantly improve the execution efficiency and flexibility of complex data processing tasks.

[0066] Step 1: Structure of the Algorithm Pipeline

[0067] Input module: receives raw data from external data sources.

[0068] Link1: Data preprocessing link: Perform data cleaning and format conversion.

[0069] Link2: Feature extraction link: Extract key features from the preprocessed data.

[0070] Link3: Model traininglink: Train the machine learning model based on the extracted features.

[0071] Link4: Model evaluation link: Evaluate and tune the trained model.

[0072] Link5: Result output link: Output the final processing result to the database, file system or network service.

[0073] Output module: Feedback the final results to users or other systems.

[0074] Step 2: Link implementation and function design

[0075] Parallel execution support: Each link is executed in an independent thread, supporting parallel processing.

[0076] Task scheduling and coordination: Data exchange and task coordination are achieved through message queues or event-driven methods.

[0077] Flexible task switching: dynamically mount different processing functions or algorithms to adapt to different processing requirements.

[0078] State management and error handling: Implement state management mechanisms and error handling strategies to ensure the stability of the Pipeline under abnormal circumstances.

[0079] S3: Application Example: Face Detection Algorithm Pipeline

[0080] To illustrate the implementation of the Pipeline more specifically, let's take the face detection algorithm as an example:

[0081] Video frame input link: receives continuous video frame data.

[0082] Face detection link: Use classic face detection algorithms (such as Haar cascade detectors or deep learning models) to perform face detection on each frame.

[0083] Feature extraction link: Extract features of detected faces, such as facial feature points, expressions, etc.

[0084] Attribute calculation link: Calculate additional attributes of the face, such as gender, age, etc.

[0085] Result output link: Store the processed face data in the database or transmit it to other application systems.

[0086] The video frame input link is responsible for receiving video frames at a fixed frequency and sending these frames to the face detection link for processing. The face detection link performs face detection at a rate of 12 frames per second, and the detected face results are passed to the feature extraction link. The feature extraction link extracts features for each face based on the face detection results, and fuses the feature results with the face detection results, and then sends the fused results to the attribute calculation link.

[0087] The attribute calculation link uses the feature results and face detection results to calculate the face attributes, and merges all the results and sends them to the result output link. The result output link finally sends the final algorithm results to the business platform for subsequent business processing.

[0088] In the whole process, all links are processed asynchronously, and the subsequent execution is driven by the message sent by the previous link. When the face detection link sends the current detection result to the feature extraction link, it can immediately start the detection of the next frame without waiting for the completion of feature extraction. In this way, the concurrent processing of the face detection link and the feature extraction link is realized, which significantly improves the overall efficiency.

[0089] Step 4: Parallel Computing Optimization

[0090] Multi-core processor: Each link independently uses a multi-core processor for calculation.

[0091] Distributed computing environment: Supports distributed computing to improve processing speed and efficiency.

[0092] Application areas and advantages

[0093] Applicable fields: image recognition, speech processing, natural language processing, etc.

[0094] Advantages: flexible task configuration, efficient parallel processing, fast construction and deployment.

[0095] The innovation of the present invention lies in:

[0096] 1. Componentized design: Pipeline consists of multiple independent links, each of which has specific processing capabilities and task execution functions. Each link can be regarded as a primitive component, including threads and communication mailboxes, which can run independently and perform a single clear task.

[0097] Componentized design realizes multi-task concurrent processing through componentization and multi-threaded concurrent mechanism:

[0098] Independence and reusability: Each link runs as an independent component and can handle a single clear task. This design promotes the reuse and replacement of components, so that different processing flows can be achieved by simply replacing or adjusting the link, thereby reducing the workload of repeated development.

[0099] Threads and communication mailboxes: Each link has built-in threads and communication mailboxes, which can independently execute tasks and exchange data with other links. This design not only supports efficient parallel computing, but also simplifies communication and synchronization issues in a multi-threaded environment.

[0100] 2. Flexible task switching: Each link can be mounted with different processing functions, so the execution task can be dynamically switched according to actual needs. This flexibility enables Pipeline to adapt to different data characteristics and processing requirements, effectively improving processing efficiency and task execution speed.

[0101] Flexible task switching: By fast and dynamic switching of processing functions, different tasks can be quickly switched and executed, reducing the time consumption of system scheduling switching:

[0102] 1) Dynamic adjustment: Different processing functions can be dynamically mounted and switched according to actual needs. This flexibility allows the system to make real-time adjustments based on the characteristics of the data or the complexity of the processing, thereby optimizing processing efficiency. For example, when faced with different types of data, the appropriate processing function can be quickly selected to perform a specific task.

[0103] 2) Efficient task scheduling: Through flexible task switching, tasks can be effectively scheduled and processed, reducing processing time and resource waste, and improving the processing speed of the overall system.

[0104] 3. Quick construction and application: Through simple link construction, you can quickly build a complex algorithm application Pipeline. For example, a face detection algorithm can be composed of multiple links such as frame sending, detection, image pushing, attributes, etc. Through the combination of these links, you can quickly build a complete face detection algorithm application.

[0105] Rapid construction and application, through componentization and process standardization, the derivation and iteration of different algorithm processes can be quickly realized:

[0106] 1) Simplify the construction process: Complex algorithm applications can be quickly built by combining links. For example, a face detection algorithm can be combined through multiple links such as frame sending, detection, image pushing, and attributes. This process simplifies the construction of complex algorithms, lowers the development threshold, and can quickly adapt to new application requirements.

[0107] 2) Flexible combination capability: Links with different functions can be combined according to actual needs and application scenarios to meet complex functional requirements, thereby improving the flexibility and scalability of the system.

[0108] 4. Parallel computing optimization: Pipeline design supports parallel computing technology, which can make full use of multi-core processors and distributed computing resources to achieve optimal processing speed and efficiency when processing large-scale data.

[0109] Parallel computing optimization, fully utilizing the multi-threaded concurrent processing technology of the Linux system to achieve concurrent processing of multiple tasks:

[0110] 1) Make full use of computing resources: Pipeline design supports parallel computing technology and can effectively utilize multi-core processors and distributed computing resources. When processing large-scale data, Pipeline can distribute tasks to different processing units, thereby achieving higher processing speed and efficiency.

[0111] 2) Improve processing efficiency: Through parallel computing and optimized data flow processing, Pipeline can greatly improve the efficiency of data processing, especially when it is necessary to process large amounts of data or complex tasks, it can significantly shorten the processing time.

[0112] These advantages together ensure that the Pipeline of the present invention can maintain high efficiency and flexibility when processing complex data tasks, adapt to various application requirements, and make full use of computing resources.

[0113] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation modes. Any other implementation modes derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. An efficient algorithm pipeline method based on an AI open platform, characterized by: The following steps are involved: S1: Establish the structure of the algorithm pipeline; S2: Based on step S1, the structure of the algorithm pipeline is established, and multiple independent component links are implemented and functionally designed; S3. Based on step S2, parallel computing optimization is performed on each independent component link.

2. According to claim 1, an efficient algorithm pipeline method based on an AI open platform is characterized in that: The specific method of step S1 is: The algorithm pipeline is composed of multiple independent component links, each link is responsible for performing a specific data processing task; Its overall structure includes: Input module: responsible for receiving data from external data sources and passing it to the first link; Each link: Each link is an independent processing component with its own processing function and task execution capability. Each link can specify its specific tasks and functions through configuration files or programming interfaces.

3. According to claim 1, an efficient algorithm pipeline method based on an AI open platform is characterized in that: The specific method of step S2 is: Split the execution steps into multiple independent "links", each link implements a specific function; Each link consists of three parts: thread, communication mailbox and processing function; Independent threads enable each link to run independently without being affected by other functions in the system; The communication mailbox is used to provide a mechanism for link to interact with the outside world. It can receive external commands or parameters and respond to execute corresponding tasks. At the same time, the link's processing tasks are implemented through processing functions, and processing functions can be configured through callback registration. This allows the same link to register different processing functions as needed to perform different tasks.

4. According to claim 1, an efficient algorithm pipeline method based on an AI open platform is characterized in that: The specific method of step S3 is: Pipeline takes advantage of the parallel performance of modern computing resources, and each link independently utilizes multi-core processors or distributed computing environments.

Images