General System for Computer Vision Algorithm Inference

By designing a general-purpose computer vision algorithm inference system, the cross-chip compatibility problem was solved, achieving high-performance computing and low-cost expansion, supporting multiple artificial intelligence chips, and improving system energy efficiency and development efficiency.

CN117875427BActive Publication Date: 2026-11-17HANGZHOU ARCVIDEO TECHNOLOGY CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311751083.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-19
Publication Date
2026-11-17
Estimated Expiration
2043-12-19

AI Technical Summary

Technical Problem

Existing computer vision algorithm inference systems cannot adapt to various artificial intelligence chips, resulting in limited chip types, inability to fully utilize computing power, and high development costs.

Method used

A general-purpose computer vision algorithm inference system was designed, including a client, a server, an algorithm model conversion tool, an algorithm model performance optimization tool, and an algorithm model library. Cross-chip algorithm inference is achieved through API interfaces and communication modules, and secondary development and encapsulation are performed using a general interface module for the algorithm inference engine.

Benefits of technology

It achieves high-performance computing across AI chips, reduces development costs, improves system energy efficiency, supports expansion to new chips, is flexible in adaptation, and has low communication latency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117875427B_ABST
    Figure CN117875427B_ABST
Patent Text Reader

Abstract

This invention discloses a general system for computer vision algorithm inference, comprising a client, a server, an algorithm model conversion tool, an algorithm model performance optimization tool, and an algorithm model library. The client includes an API interface, a first communication module, an algorithm command request module, and an algorithm inference request module. The server includes a second communication module, a status monitoring module, an algorithm model management module, an algorithm command execution module, an algorithm inference execution module, an algorithm inference engine general interface module, and an algorithm inference engine implementation module. The API interface defines an algorithm initialization interface, an algorithm deinitialization interface, and an algorithm inference interface. The first and second communication modules are used to enable communication between the client process and the server process, establishing algorithm command channels and algorithm inference channels. The algorithm command request module is used for client algorithm command encapsulation and response result parsing; the algorithm commands include algorithm initialization and algorithm deinitialization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a general system for computer vision algorithm inference. Background Technology

[0002] With the rapid development of artificial intelligence technology, the market demands an increasing number of algorithm applications, especially computer vision algorithms. While algorithm development is crucial in algorithm application projects, efficiently running algorithms on AI chips is also a vital aspect of project implementation. Algorithm inference systems can fully utilize the computing power of AI chips to maximize algorithm execution, thereby reducing enterprise costs and helping companies fulfill their social responsibilities regarding energy conservation and environmental protection.

[0003] Existing computer vision algorithm inference systems are all based on one or more algorithm inference engines. These systems have a strong coupling relationship between the algorithm inference engine and the artificial intelligence chip, and vice versa. Based on these reasons, existing algorithm inference systems face the following problems: (1) limited support for certain types or models of artificial intelligence chips; (2) inability to fully utilize the computing power of artificial intelligence chips; and (3) inability to expand support for new artificial intelligence chips. Currently, the artificial intelligence chip market is becoming increasingly competitive. Traditional chip manufacturers are constantly innovating their products, and domestically produced chips are also booming. More types and models of chips will emerge in the future. For enterprises, in order to use increasingly diverse artificial intelligence chips, they have to develop and maintain new algorithm inference systems, but these systems can only be adapted to one or a few types of chips, resulting in high development costs. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention provides a general system for computer vision algorithm inference, which supports high-performance computing of computer vision algorithms on different artificial intelligence chips.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A general-purpose computer vision algorithm inference system includes a client, a server, an algorithm model conversion tool, an algorithm model performance optimization tool, and an algorithm model library.

[0007] The client includes an API interface, a first communication module, an algorithm command request module, and an algorithm inference request module; the server includes a second communication module, a status monitoring module, an algorithm model management module, an algorithm command execution module, an algorithm inference execution module, an algorithm inference engine general interface module, and an algorithm inference engine implementation module.

[0008] The API interface defines the algorithm initialization interface, the algorithm deinitialization interface, and the algorithm inference interface;

[0009] The first and second communication modules are used to enable communication between the client process and the server process, and to establish an algorithm command channel and an algorithm inference channel.

[0010] The algorithm command request module is used for encapsulating client algorithm commands and parsing response results. Algorithm commands include algorithm initialization and algorithm deinitialization. The algorithm inference request module is used for encapsulating client algorithm inference tasks and parsing algorithm inference task results.

[0011] The status monitoring module is used to monitor the server status and the abnormal status handling mechanism; the algorithm model management module is used for algorithm model library management; the algorithm command execution module is used for server-side algorithm command parsing, execution of algorithm commands, and encapsulation of algorithm command results, including algorithm initialization and algorithm deinitialization; the algorithm inference execution module is used for algorithm inference task parsing, algorithm inference task scheduling, execution of algorithm inference tasks, and encapsulation of algorithm inference task results.

[0012] The general interface module of the algorithm inference engine is used to define the input data interface, output data interface, algorithm initialization interface, algorithm deinitialization interface, and algorithm inference interface. The input data interface is used to convert input data from system data format to chip data format and input the data into the chip. The output data interface is used to acquire chip output data and convert the output data from chip data format to system data format. The algorithm initialization interface is used to initialize the algorithm context and request hardware and software resources. The algorithm deinitialization interface is used to deinitialize the algorithm context and release resources. The algorithm inference interface is used to perform algorithm inference calculations.

[0013] The algorithm inference engine implementation module is used for secondary development and encapsulation based on the general interface definition and the artificial intelligence chip API;

[0014] The algorithm model conversion tool is used to perform secondary development and encapsulation using the artificial intelligence chip API, converting the trained computer vision algorithm model from its original format to a format supported by the artificial intelligence chip.

[0015] The algorithm model performance optimization tool is used for secondary development and encapsulation using artificial intelligence chip APIs, and optimizes the performance of the converted model based on performance requirements.

[0016] The algorithm model library is used to store optimized algorithm models.

[0017] In one possible implementation, it is further used to: perform an algorithm initialization process.

[0018] In one possible implementation, the algorithm initialization process includes:

[0019] The application starts using the algorithm and uses the client's algorithm initialization interface, with the parameter being the algorithm's unique identifier; internally, the interface sends the algorithm initialization request to the server through the algorithm command channel established by the first communication module.

[0020] The server receives the algorithm command request through the algorithm command channel of the second communication module. The algorithm command execution module parses the command as algorithm initialization and begins to execute the algorithm initialization. Based on the unique algorithm identifier in the request parameters, the algorithm model management module loads the algorithm model, the algorithm inference task scheduling module creates the corresponding algorithm's task queue and worker thread, and the algorithm inference engine general interface module initializes the algorithm interface to complete the initialization of the algorithm context. After completion, it returns a message to the client indicating that the algorithm initialization execution was successful.

[0021] The client received a message that the initialization algorithm was executed successfully.

[0022] In one possible implementation, it is further used to: perform the algorithm reasoning process.

[0023] In one possible implementation, the execution algorithm reasoning process includes:

[0024] After the application successfully initializes the algorithm, it uses the client's algorithm inference interface with parameters including the algorithm's unique identifier and input data. Internally, the interface sends the algorithm inference request to the server through the algorithm inference channel established by the first communication module.

[0025] The server receives the algorithm inference request through the algorithm inference channel of the second communication module. The algorithm inference execution module parses the request and, based on the algorithm's unique identifier, allocates inference tasks to different task queues through scheduling. The worker thread of the algorithm inference task execution module retrieves tasks from the task queues. The input data interface of the algorithm inference engine's general interface module converts the input data of the inference task from the system data format to the chip data format and inputs it into the chip. Then, the algorithm inference interface of the algorithm inference engine's general interface module executes the algorithm inference calculation. Through the output data interface, it retrieves the output data from the chip and converts it from the chip data format to the system data format. After completion, it returns a message to the client indicating that the algorithm inference task has been successfully executed and the calculation result is displayed.

[0026] The client received the initialization algorithm inference success and the calculation result.

[0027] In one possible implementation, it is further used to: perform an algorithm deinitialization process.

[0028] In one possible implementation, the algorithm deinitialization process includes:

[0029] When the application finishes using the algorithm, it uses the algorithm deinitialization interface of the system client. The interface encapsulates the algorithm deinitialization command request and sends it to the server through the algorithm command channel of the first communication module. The parameter is the unique identifier of the algorithm.

[0030] The server receives the algorithm command request through the algorithm command channel of the second communication module. The algorithm command execution module parses the command as an algorithm deinitialization and begins the algorithm deinitialization process. Based on the unique algorithm identifier in the request parameters, the algorithm model management module unloads the algorithm model, the algorithm inference task scheduling module destroys the corresponding algorithm's task queue and worker thread, and the algorithm inference engine general interface module's deinitialization algorithm interface completes the deinitialization of the algorithm context. After completion, a message is returned to the client indicating that the algorithm deinitialization was successful.

[0031] After the client receives a successful deinitialization algorithm execution, it sends a disconnect request for the algorithm command channel and the algorithm inference channel, thus disconnecting the algorithm command channel and the algorithm inference channel.

[0032] The server receives a request to disconnect the algorithm command channel and the algorithm inference channel, disconnects the algorithm command channel and the algorithm inference channel, and destroys the algorithm command channel and the algorithm inference channel.

[0033] The present invention has the following beneficial effects:

[0034] (1) The general system for computer vision algorithm inference in this embodiment of the invention is not limited to the types and models of artificial intelligence chips and can be extended.

[0035] (2) Faster adaptation to new artificial intelligence chips. The general computer vision algorithm inference system of this embodiment only needs to develop a small number of extension modules. Compared with developing a new algorithm inference system, it reduces development costs and improves the efficiency of algorithm application projects.

[0036] (3) The general system for computer vision algorithm inference in this embodiment of the invention achieves powerful chip expansion capabilities while giving full play to the computing power of artificial intelligence chips, improving the system energy efficiency ratio, reducing hardware costs, and achieving the goal of energy saving and environmental protection.

[0037] (4) The general system for computer vision algorithm inference in this embodiment of the invention uses a modular design, which makes the development of each module simpler and more flexible.

[0038] (5) The communication module of the general system for computer vision algorithm inference in this embodiment of the invention uses shared memory technology, which ensures low latency in communication between the client and the server, and is suitable for large-scale image data transmission of computer vision algorithms.

[0039] (6) The general system for computer vision algorithm inference in this embodiment of the invention is designed with a general interface module and an extensible module for algorithm inference engine. This design achieves the goal of adapting to new artificial intelligence chips and saving development costs to the greatest extent.

[0040] (7) The algorithm inference engine implementation module, algorithm model conversion tool, and algorithm model performance optimization tool are encapsulated based on the AI ​​chip API (Application Programming Interface). The encapsulation layer achieves universal interface and fully leverages the capabilities of the AI ​​chip. Attached Figure Description

[0041] Figure 1 This is a block diagram illustrating the principle of a general-purpose computer vision algorithm inference system according to an embodiment of the present invention.

[0042] Figure 2 This is a flowchart illustrating the algorithm initialization process in a general system for computer vision algorithm inference according to another embodiment of the present invention.

[0043] Figure 3 This is a flowchart illustrating the algorithm inference process in a general system for computer vision algorithm inference according to another embodiment of the present invention.

[0044] Figure 4 This is a flowchart of the algorithm deinitialization process in a general system for computer vision algorithm inference, which is another embodiment of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] See Figure 1The diagram shows a principle block diagram of a general computer vision algorithm inference system according to an embodiment of the present invention. It includes a client, a server, an algorithm model conversion tool, an algorithm model performance optimization tool, and an algorithm model library. The client includes an API interface, a first communication module, an algorithm command request module, and an algorithm inference request module. The server includes a second communication module, a status monitoring module, an algorithm model management module, an algorithm command execution module, an algorithm inference execution module, an algorithm inference engine general interface module, and an algorithm inference engine implementation module. The API interface defines an algorithm initialization interface, an algorithm deinitialization interface, and an algorithm inference interface. The first and second communication modules are used to implement communication between the client process and the server process, establishing algorithm command channels and algorithm inference channels. The algorithm command request module is used for client algorithm command encapsulation and response result parsing; the algorithm commands include algorithm initialization and algorithm deinitialization. The algorithm inference request module is used for client algorithm inference task encapsulation and algorithm inference task result parsing. The status monitoring module is used to monitor the server status and implement an abnormal status handling mechanism. The algorithm model management module is used for algorithm model library management. The algorithm command execution module is used for server algorithm command parsing, execution, and result encapsulation; the algorithm commands include algorithm initialization and algorithm deinitialization. The algorithm inference execution module is used for parsing, scheduling, executing, and encapsulating the results of algorithm inference tasks. The general interface module for the algorithm inference engine defines the input data interface, output data interface, algorithm initialization interface, algorithm deinitialization interface, and algorithm inference interface. The input data interface converts input data from system data format to chip data format and inputs the data into the chip. The output data interface acquires chip output data and converts it back to system data format. The algorithm initialization interface initializes the algorithm context and requests hardware and software resources. The algorithm deinitialization interface deinitializes the algorithm context and releases resources. The algorithm inference interface performs algorithm inference computation. The algorithm inference engine implementation module uses the general interface definition and AI chip API for secondary development and encapsulation. The algorithm model conversion tool uses the AI ​​chip API for secondary development and encapsulation, converting trained computer vision algorithm models from their original format to a format supported by the AI ​​chip. The algorithm model performance optimization tool uses the AI ​​chip API for secondary development and encapsulation, optimizing the converted model based on performance requirements. The algorithm model library stores optimized algorithm models.

[0047] Using the general-purpose computer vision algorithm inference system configured above, a pre-trained algorithm model is acquired. An algorithm model conversion tool is then used to convert the original model format to a format supported by the target AI chip. After conversion, performance optimization tools are used for performance optimization. Based on performance requirements such as throughput, latency, and accuracy, the parameters of the performance optimization tool are adjusted, ultimately converting the model to the target chip-supported format. The optimized model is then placed into the algorithm model library, with a unique algorithm identifier and path provided through an algorithm configuration file, and managed by the server-side model management module. The client provides the model to application developers as a static or dynamic library for easy integration into the program. The first and second communication modules use shared memory technology to achieve low-latency communication between the client and server processes.

[0048] Another embodiment of the present invention provides a general system for computer vision algorithm inference, further used to execute the algorithm initialization process. See also Figure 2 The algorithm initialization process includes:

[0049] The application starts using the algorithm and uses the client's algorithm initialization interface, with the parameter being the algorithm's unique identifier; internally, the interface sends the algorithm initialization request to the server through the algorithm command channel established by the first communication module.

[0050] The server receives the algorithm command request through the algorithm command channel of the second communication module. The algorithm command execution module parses the command as algorithm initialization and begins to execute the algorithm initialization. Based on the unique algorithm identifier in the request parameters, the algorithm model management module loads the algorithm model, the algorithm inference task scheduling module creates the corresponding algorithm's task queue and worker thread, and the algorithm inference engine general interface module initializes the algorithm interface to complete the initialization of the algorithm context. After completion, it returns a message to the client indicating that the algorithm initialization execution was successful.

[0051] The client received a message that the initialization algorithm was executed successfully.

[0052] Another embodiment of the present invention provides a general system for computer vision algorithm inference, further used for: executing the algorithm inference process. See also Figure 3 The algorithm reasoning process includes:

[0053] After the application successfully initializes the algorithm, it uses the client's algorithm inference interface with parameters including the algorithm's unique identifier and input data. Internally, the interface sends the algorithm inference request to the server through the algorithm inference channel established by the first communication module.

[0054] The server receives the algorithm inference request through the algorithm inference channel of the second communication module. The algorithm inference execution module parses the request and, based on the algorithm's unique identifier, allocates inference tasks to different task queues through scheduling. The worker thread of the algorithm inference task execution module retrieves tasks from the task queues. The input data interface of the algorithm inference engine's general interface module converts the input data of the inference task from the system data format to the chip data format and inputs it into the chip. Then, the algorithm inference interface of the algorithm inference engine's general interface module executes the algorithm inference calculation. Through the output data interface, it retrieves the output data from the chip and converts it from the chip data format to the system data format. After completion, it returns a message to the client indicating that the algorithm inference task has been successfully executed and the calculation result is displayed.

[0055] The client received the initialization algorithm inference success and the calculation result.

[0056] Another embodiment of the present invention provides a general system for computer vision algorithm inference, further used for: performing the algorithm deinitialization process. See also Figure 4 The deinitialization process of the algorithm includes:

[0057] When the application finishes using the algorithm, it uses the algorithm deinitialization interface of the system client. The interface encapsulates the algorithm deinitialization command request and sends it to the server through the algorithm command channel of the first communication module. The parameter is the unique identifier of the algorithm.

[0058] The server receives the algorithm command request through the algorithm command channel of the second communication module. The algorithm command execution module parses the command as an algorithm deinitialization and begins the algorithm deinitialization process. Based on the unique algorithm identifier in the request parameters, the algorithm model management module unloads the algorithm model, the algorithm inference task scheduling module destroys the corresponding algorithm's task queue and worker thread, and the algorithm inference engine general interface module's deinitialization algorithm interface completes the deinitialization of the algorithm context. After completion, a message is returned to the client indicating that the algorithm deinitialization was successful.

[0059] After the client receives a successful deinitialization algorithm execution, it sends a disconnect request for the algorithm command channel and the algorithm inference channel, thus disconnecting the algorithm command channel and the algorithm inference channel.

[0060] The server receives a request to disconnect the algorithm command channel and the algorithm inference channel, disconnects the algorithm command channel and the algorithm inference channel, and destroys the algorithm command channel and the algorithm inference channel.

[0061] The general-purpose computer vision algorithm inference system described above includes a general interface module and an extensible module for the algorithm inference engine. This design achieves the goal of adapting to new artificial intelligence chips while minimizing development costs. The algorithm inference engine implementation module, algorithm model conversion tool, and algorithm model performance optimization tool are encapsulated through secondary development based on the artificial intelligence chip API (Application Programming Interface). The encapsulation layer achieves a universal interface and fully leverages the capabilities of the artificial intelligence chip.

[0062] It should be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made without departing from the spirit and scope of the invention as defined by the appended claims.

Claims

1. A computer vision algorithm inference general system, characterized in that, This includes client-side components, server-side components, algorithm model conversion tools, algorithm model performance optimization tools, and algorithm model libraries. The client includes an API interface, a first communication module, an algorithm command request module, and an algorithm inference request module; the server includes a second communication module, a status monitoring module, an algorithm model management module, an algorithm command execution module, an algorithm inference execution module, an algorithm inference engine general interface module, and an algorithm inference engine implementation module. The API interface defines the algorithm initialization interface, the algorithm deinitialization interface, and the algorithm inference interface; The first and second communication modules are used to enable communication between the client process and the server process, and to establish an algorithm command channel and an algorithm inference channel. The algorithm command request module is used for encapsulating client algorithm commands and parsing response results. Algorithm commands include algorithm initialization and algorithm deinitialization. The algorithm inference request module is used for encapsulating client algorithm inference tasks and parsing algorithm inference task results. The status monitoring module is used to monitor the server status and the abnormal status handling mechanism; the algorithm model management module is used for algorithm model library management; the algorithm command execution module is used for server-side algorithm command parsing, execution of algorithm commands, and encapsulation of algorithm command results, including algorithm initialization and algorithm deinitialization; the algorithm inference execution module is used for algorithm inference task parsing, algorithm inference task scheduling, execution of algorithm inference tasks, and encapsulation of algorithm inference task results. The general interface module of the algorithm inference engine is used to define the input data interface, output data interface, algorithm initialization interface, algorithm deinitialization interface, and algorithm inference interface. The input data interface is used to convert the input data from the system data format to the chip data format and input the data into the chip. The output data interface is used to acquire the chip output data and convert the output data from the chip data format to the system data format. The algorithm initialization interface is used to initialize the algorithm context and request software and hardware resources. The algorithm deinitialization interface is used to deinitialize the algorithm context and release resources; The algorithm inference interface is used to implement algorithm inference calculations; The algorithm inference engine implementation module is used for secondary development and encapsulation based on the general interface definition and the artificial intelligence chip API; The algorithm model conversion tool is used to perform secondary development and encapsulation using the artificial intelligence chip API, converting the trained computer vision algorithm model from its original format to a format supported by the artificial intelligence chip. The algorithm model performance optimization tool is used for secondary development and encapsulation using artificial intelligence chip APIs, and optimizes the performance of the converted model based on performance requirements. The algorithm model library is used to store optimized algorithm models; Further used to perform the algorithm initialization process, including: The application starts using the algorithm and uses the client's algorithm initialization interface, with the parameter being the algorithm's unique identifier; internally, the interface sends the algorithm initialization request to the server through the algorithm command channel established by the first communication module. The server receives the algorithm command request through the algorithm command channel of the second communication module. The algorithm command execution module parses the command as algorithm initialization and begins to execute the algorithm initialization. Based on the unique algorithm identifier in the request parameters, the algorithm model management module loads the algorithm model, the algorithm inference task scheduling module creates the corresponding algorithm's task queue and worker thread, and the algorithm inference engine general interface module initializes the algorithm interface to complete the initialization of the algorithm context. After completion, the server returns a successful algorithm initialization to the client. The client received a message indicating that the initialization algorithm executed successfully. Further used to perform the algorithmic reasoning process, including: After the application successfully initializes the algorithm, it uses the client's algorithm inference interface with parameters including the algorithm's unique identifier and input data. Internally, the interface sends the algorithm inference request to the server through the algorithm inference channel established by the first communication module. The server receives the algorithm inference request through the algorithm inference channel of the second communication module. The algorithm inference execution module parses the algorithm inference request and, based on the unique algorithm identifier, allocates inference tasks to different task queues through scheduling. The worker thread of the algorithm inference task execution module retrieves tasks from the task queues. The input data interface of the algorithm inference engine general interface module converts the input data of the inference task from the system data format to the chip data format and inputs it into the chip. Then, the algorithm inference interface of the algorithm inference engine general interface module executes the algorithm inference calculation. Through the output data interface, the server retrieves the output data from the chip and converts the output data from the chip data format to the system data format. After completion, the server returns the successful execution of the algorithm inference task and the calculation result to the client. The client received the initialization algorithm inference success and the calculation result.

2. The computer vision algorithm inference general system of claim 1, wherein, Further used for: executing the algorithm deinitialization process.

3. The computer vision algorithm inference general system of claim 2, wherein, The deinitialization process of the execution algorithm includes: When the application finishes using the algorithm, it uses the algorithm deinitialization interface of the system client. The interface encapsulates the algorithm deinitialization command request and sends it to the server through the algorithm command channel of the first communication module. The parameter is the unique identifier of the algorithm. The server receives the algorithm command request through the algorithm command channel of the second communication module. The algorithm command execution module parses the command as an algorithm deinitialization and begins to execute the algorithm deinitialization. Based on the unique algorithm identifier in the request parameters, the algorithm model management module unloads the algorithm model, the algorithm inference task scheduling module destroys the corresponding algorithm's task queue and worker thread, and the algorithm inference engine general interface module deinitializes the algorithm interface to complete the deinitialization of the algorithm context. After completion, the server returns a message to the client that the algorithm deinitialization was successful. After the client receives a successful deinitialization algorithm execution, it sends a disconnect request for the algorithm command channel and the algorithm inference channel, thus disconnecting the algorithm command channel and the algorithm inference channel. The server receives a request to disconnect the algorithm command channel and the algorithm inference channel, disconnects the algorithm command channel and the algorithm inference channel, and destroys the algorithm command channel and the algorithm inference channel.

Citation Information

Patent Citations

  • Method and device for executing ciphertext calculation instruction and ciphertext calculation system

    CN113343285A

  • Cross-platform model reasoning method and system, storage medium and equipment

    CN113986234A