Multi-model based ai resource management method, device and equipment and storage medium

Abstract

The application discloses a kind of based on multi-model AI resource management method, device and equipment and storage medium, according to the enabling condition of each detection model, management AI passage;Create linked list stack;When the number of nodes in the linked list stack is less than the preset threshold, and there is AI passage, a frame of picture data is obtained, saved as node, and the node is inserted into the linked list stack by head insertion method;According to the use condition of each node corresponding to the picture data in the linked list stack by each detection model, adjust the reference count value of each node;According to the reference count value of each node in the linked list stack, each node and its corresponding picture data are managed.The application can reduce unnecessary frame waiting time consumption, reduce the time consumption of overall detection process.

Description

Technical Field

[0001] This invention relates to the field of resource management technology, and in particular to an AI resource management method, apparatus, device, and storage medium based on multiple models. Background Technology

[0002] Currently, when performing AI detection on images, each AI detection model typically uses the YUV image data output by the hardware SOC as input independently before performing the detection.

[0003] However, when multiple detection models are involved (e.g., pedestrian detection, smoking and phone call detection, face detection, etc.), and each model runs independently, significant competition arises when acquiring YUV data. Since the original YUV data is directly obtained from the SDK (Software Development Kit) and each model acquires and releases data independently, image data cannot be shared. After one detection model acquires a frame, the next detection model can only wait for the next frame. This leads to competition between models and increases processing time. Furthermore, each detection model needs to release resources after completing its detection, which also adds to processing time. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an AI resource management method, apparatus, device and storage medium based on multiple models, which can reduce the time consumption of the overall detection process.

[0005] In a first aspect, the present invention provides an AI resource management method based on multiple models, comprising:

[0006] Manage the AI ​​channel based on the enabling status of each detection model;

[0007] Create a linked list stack;

[0008] When the number of nodes in the linked list stack is less than a preset threshold and an AI channel exists, a frame of image data is acquired, saved as a node, and the node is inserted into the linked list stack using the head insertion method.

[0009] Based on the usage of image data corresponding to each node in the linked list stack by each detection model, adjust the reference count value of each node;

[0010] Based on the reference count of each node in the linked list stack, manage each node and its corresponding image data.

[0011] Secondly, the present invention also provides an AI resource management device based on multiple models, comprising:

[0012] The first management module is used to manage the AI ​​channel based on the enabling status of each detection model;

[0013] Create a module for creating linked list stacks;

[0014] The insertion module is used to acquire a frame of image data, save it as a node, and insert the node into the linked list stack using the head insertion method when the number of nodes in the linked list stack is less than a preset threshold and an AI channel exists.

[0015] The adjustment module is used to adjust the reference count value of each node according to the usage of image data corresponding to each node in the linked list stack by each detection model.

[0016] The second management module is used to manage each node and its corresponding image data based on the reference count value of each node in the linked list stack.

[0017] Thirdly, the present invention also provides an electronic device, the electronic device comprising:

[0018] One or more processors;

[0019] Storage device for storing one or more programs;

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement a multi-model-based AI resource management method as provided in the first aspect.

[0021] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multi-model-based AI resource management method as provided in the first aspect.

[0022] The beneficial effects of this invention are as follows: By creating or deleting AI channels according to the enabling status of the detection model, resources can be saved as much as possible; by using a linked list stack, the latest acquired image data is saved at the top of the stack, and each detection model obtains data from the top of the stack, thus ensuring that the image data obtained by the detection model is up-to-date; furthermore, each detection model only obtains the image data corresponding to the node, and the node remains in the linked list stack, thereby reducing data contention, achieving data sharing, and reducing frame waiting time; by adding a reference count flag to each node, the reference status of the image data corresponding to each node can be easily recorded, and the nodes in the linked list stack and their corresponding image data can be managed according to the reference count value, separating the resource release time from the detection time, without consuming additional time in the detection stage, thereby reducing the overall detection process time. Attached Figure Description

[0023] Figure 1 This is a flowchart of a multi-model-based AI resource management method according to Embodiment 1 of the present invention.

[0024] Figure 2 This is a diagram illustrating the CPU consumption of the existing solution;

[0025] Figure 3 This is a schematic diagram illustrating CPU consumption in Embodiment 1 of the present invention;

[0026] Figure 4 A schematic diagram of the structure of an AI resource management device based on multiple models provided by the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation

[0028] To explain the technical content, objectives, and effects of the present invention in detail, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0029] Please see Figure 1 A multi-model-based AI resource management method includes:

[0030] Manage the AI ​​channel based on the enabling status of each detection model;

[0031] Create a linked list stack;

[0032] When the number of nodes in the linked list stack is less than a preset threshold and an AI channel exists, a frame of image data is acquired, saved as a node, and the node is inserted into the linked list stack using the head insertion method.

[0033] Based on the usage of image data corresponding to each node in the linked list stack by each detection model, adjust the reference count value of each node;

[0034] Based on the reference count of each node in the linked list stack, manage each node and its corresponding image data.

[0035] As can be seen from the above description, the beneficial effects of the present invention are: it can reduce data contention, realize resource sharing, and reduce unnecessary frame waiting time; it can separate resource release time from detection time, so as not to consume additional time in the detection stage, thereby reducing the overall detection process time.

[0036] Furthermore, the management of AI channels based on the enabling status of each detection model includes:

[0037] Obtain the enabling status of each detection model;

[0038] If at least one detection model is enabled, an AI channel is created, and the resolution of the AI ​​channel is determined based on the maximum resolution required by each detection model.

[0039] If all detection models are disabled, then the AI ​​channel is deleted.

[0040] As described above, resources can be saved as much as possible by creating or deleting AI channels based on the enabling status of the detection model.

[0041] Furthermore, after determining the resolution of the AI ​​channel based on the maximum resolution required by each detection model, the process further includes:

[0042] If the maximum resolution changes, the AI ​​channel is recreated, and the resolution of the recreated AI channel is determined based on the changed maximum resolution.

[0043] As described above, by dynamically adjusting the resolution of the AI ​​channel, high-resolution images can be provided to the required detection model.

[0044] Furthermore, adjusting the reference count value of each node based on the usage of image data corresponding to each node in the linked list stack by each detection model specifically involves:

[0045] If a detection model obtains image data corresponding to a node in the linked list stack, the reference count value of that node is increased by a preset value.

[0046] If the detection model releases the image data corresponding to the node, the reference count value of the node will be reduced by a preset value.

[0047] As described above, by adding a reference count flag to each node, it is possible to conveniently record the reference status of the image data corresponding to each node.

[0048] Furthermore, the step of managing each node and its corresponding image data based on the reference count value of each node in the linked list stack specifically involves:

[0049] Traverse each node in the linked list stack, delete nodes with a reference count of zero that are not the first node, and release the image data corresponding to the deleted nodes.

[0050] As described above, by managing the nodes in the linked list stack and their corresponding image data based on the reference count, the resource release time and detection time can be separated, eliminating the need to consume additional time in the detection phase, thereby reducing the overall detection process time.

[0051] The present invention also provides an AI resource management device based on multiple models, comprising:

[0052] The first management module is used to manage the AI ​​channel based on the enabling status of each detection model;

[0053] Create a module for creating linked list stacks;

[0054] The insertion module is used to acquire a frame of image data, save it as a node, and insert the node into the linked list stack using the head insertion method when the number of nodes in the linked list stack is less than a preset threshold and an AI channel exists.

[0055] The adjustment module is used to adjust the reference count value of each node according to the usage of image data corresponding to each node in the linked list stack by each detection model.

[0056] The second management module is used to manage each node and its corresponding image data based on the reference count value of each node in the linked list stack.

[0057] Furthermore, the first management module includes:

[0058] The acquisition unit is used to acquire the enabling status of each detection model;

[0059] The first creation unit is configured to create an AI channel if at least one detection model is enabled, and determine the resolution of the AI ​​channel based on the maximum resolution required by each detection model.

[0060] The deletion unit is used to delete the AI ​​channel if all detection models are disabled.

[0061] The second creation unit is used to recreate the AI ​​channel if the maximum resolution changes, and to determine the resolution of the recreated AI channel based on the changed maximum resolution.

[0062] Furthermore, the adjustment module includes:

[0063] An additional unit is added to increment the reference count of a node by a preset value if a detection model acquires image data corresponding to a node in the linked list stack.

[0064] The reduction unit is used to reduce the reference count value of a node by a preset value if the detection model releases the image data corresponding to the node.

[0065] The second management module is specifically used to traverse each node in the linked list stack, delete nodes with a reference count of zero and that are not the first node, and release the image data corresponding to the deleted nodes.

[0066] The present invention also provides an electronic device, the electronic device comprising:

[0067] One or more processors;

[0068] Storage device for storing one or more programs;

[0069] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described above.

[0070] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0071] Example 1

[0072] Please refer to Figure 1-3 The first embodiment of the present invention is: an AI resource management method based on multiple models, which can be applied to image-based AI detection scenarios.

[0073] In this embodiment, AI resource management mainly includes AI channel management and YUV image data management.

[0074] AI channel management primarily controls the creation, modification, and deletion of AI channels. Specifically, it detects in real-time whether an AI channel is needed and creates one if so. Then, it monitors the maximum resolution required by all detection models in real-time; if this changes, it dynamically adjusts the resolution of the current AI channel to provide high-resolution images to the required detection models. For example, face recognition requires 1080P in long-distance recognition scenarios. When all detection models stop, the AI ​​channel is shut down to save resources (e.g., for a 400W device, this effectively reduces frame drops at the VPE, a module in the Mstar339G chip; when overall resources are high, this module may malfunction, leading to dropped video frames and reduced image continuity).

[0075] YUV image data management primarily involves managing image data, including acquiring and saving image data, deleting outdated image data, and providing external acquisition and release interfaces. Specifically, to ensure the AI ​​detection models acquire the most up-to-date image data, this embodiment uses a stack-like approach, placing acquired image data at the top each time, with each AI detection model retrieving image data from the top. Furthermore, the stack implementation uses a doubly linked circular list provided by the kernel. To reduce data contention, an image data sharing mechanism is implemented, marking the nodes corresponding to the image data acquired by each AI detection model with a reference count (ref_cnt). Additionally, to separate resource release time from detection time, this embodiment traverses all nodes in the linked list stack before acquiring a frame of image data from the SDK, deleting nodes with a reference count of 0 and releasing the corresponding buffer resources.

[0076] like Figure 1 As shown, the method in this embodiment includes the following steps:

[0077] S1: Manage the AI ​​channel based on the enabling status of each detection model.

[0078] Specifically, the configuration information for the current intelligent analysis is obtained, mainly the enabling status of each AI detection model. If a detection model is enabled, an AI channel is created (createAiChn), and the resolution of the AI ​​channel is determined based on the maximum resolution required by each detection model. Alternatively, the resolution of the AI ​​channel can be determined only based on the maximum resolution required by the detection models that are enabled. If all detection models are disabled, the AI ​​channel is deleted (destroyAiChn).

[0079] Furthermore, if the maximum resolution changes, the AI ​​channel is recreated, and the resolution of the recreated AI channel is determined based on the changed maximum resolution. That is, if the required resolution changes, `recreateAiChn()` is used to recreate the channel.

[0080] S2: Create a linked list stack. In this embodiment, the stack is implemented using a doubly linked circular list provided by the kernel.

[0081] S3: Query the number of nodes in the current linked list stack. If the number of nodes in the linked list stack is less than the preset threshold and there is an AI channel (i.e., an AI channel has been created), then obtain a frame of image data from the SDK, save it as a node, and insert the node into the linked list stack using the head insertion method, i.e., insert it into the top of the linked list stack using the head insertion method of linked list operations.

[0082] S4: Adjust the reference count value of each node according to the usage of image data corresponding to each node in the linked list stack by each detection model.

[0083] Specifically, if a detection model acquires image data corresponding to a node in the linked list stack, it increments the reference count of that node by a preset value; the initial value of the reference count is 0. If the detection model releases the image data corresponding to that node, it decrements the reference count of that node by a preset value.

[0084] In this embodiment, to ensure that the detection model obtains the latest image data, it retrieves data from the top of the linked list stack whenever needed. Specifically, it retrieves the first node in the current linked list stack, copies the image data and handle corresponding to that first node, and increments the reference count of that first node. In other words, the node remains in the linked list stack during data retrieval and has not been removed from it.

[0085] When the detection model releases image data, the reference count of the corresponding node is decremented by one according to the handle.

[0086] S5: Manage each node and its corresponding image data based on the reference count value of each node in the linked list stack.

[0087] Specifically, each node in the linked list stack is traversed, nodes with a reference count of zero and not being the first node are deleted, and the image data corresponding to the deleted nodes is released.

[0088] Since referenced nodes cannot be deleted, and the first node (i.e., the first node) used to provide information to the detection model cannot be deleted either, nodes with a reference count of 0 and that are not the first node are deleted. Simultaneously, the addresses of the nodes to be released are returned, and then the actual image data is released. This release step takes some time, and since the preceding process is locked, placing this release step after the lock effectively reduces the impact of the lock on performance.

[0089] For the original scheme where each AI detection model independently uses YUV image data for detection, if all detection models (including pedestrian detection and attribute classification, smoking and phone detection, and face detection) are enabled, the CPU consumption is as follows: Figure 2 As shown. However, if the method of this embodiment is used, the CPU consumption is as follows: Figure 3 As shown. From Figure 2-3 As can be seen, in the original solution, CPU consumption was around 60%, while after using the method of this embodiment, CPU consumption was reduced to around 42%, effectively reducing resource consumption.

[0090] This embodiment can resolve the contention for YUV resources during multi-model runtime, achieve resource sharing to eliminate unnecessary frame waiting time, and provide YUV resource management, preparing the resources required for AI detection in other threads to reduce the overall process time; at the same time, it manages the image size of YUV data, pays attention to the resolution required by each model, adjusts the size of the data source in a timely manner, and performs independent scaling processing on the required size of each model.

[0091] Example 2

[0092] Please refer to Figure 4 Embodiment two of the present invention is: a multi-model-based AI resource management device, which can execute the multi-model-based AI resource management method provided in Embodiment one of the present invention, and has the corresponding functional modules and beneficial effects of the method execution. This device can be implemented by software and / or hardware, and specifically includes:

[0093] The first management module 201 is used to manage the AI ​​channel based on the enabling status of each detection model;

[0094] Create module 202 to create a linked list stack;

[0095] The insertion module 203 is used to acquire a frame of image data, save it as a node, and insert the node into the linked list stack using the head insertion method when the number of nodes in the linked list stack is less than a preset threshold and an AI channel exists.

[0096] The adjustment module 204 is used to adjust the reference count value of each node according to the usage of image data corresponding to each node in the linked list stack by each detection model.

[0097] The second management module 205 is used to manage each node and its corresponding image data according to the reference count value of each node in the linked list stack.

[0098] In an optional implementation, the first management module 201 includes:

[0099] The acquisition unit is used to acquire the enabling status of each detection model;

[0100] The first creation unit is configured to create an AI channel if at least one detection model is enabled, and determine the resolution of the AI ​​channel based on the maximum resolution required by each detection model.

[0101] The deletion unit is used to delete the AI ​​channel if all detection models are disabled.

[0102] In an optional implementation, the first management module 201 further includes:

[0103] The second creation unit is used to recreate the AI ​​channel if the maximum resolution changes, and to determine the resolution of the recreated AI channel based on the changed maximum resolution.

[0104] In an optional implementation, the adjustment module 204 includes:

[0105] An additional unit is added to increment the reference count of a node by a preset value if a detection model acquires image data corresponding to a node in the linked list stack.

[0106] The reduction unit is used to reduce the reference count value of a node by a preset value if the detection model releases the image data corresponding to the node.

[0107] In an optional implementation, the second management module 205 is specifically used to traverse each node in the linked list stack, delete nodes with a reference count of zero and that are not the first node, and release the image data corresponding to the deleted nodes.

[0108] Example 3

[0109] Please refer to Figure 5 Embodiment 3 of the present invention is: an electronic device, the electronic device comprising:

[0110] One or more processors 301;

[0111] Storage device 302 is used to store one or more programs;

[0112] When the one or more programs are executed by the one or more processors 301, the one or more processors 301 implement the various processes in the multi-model-based AI resource management method embodiment described above, and achieve the same technical effect. To avoid repetition, this will not be repeated here.

[0113] Example 4

[0114] This embodiment is a computer-readable storage medium corresponding to the above embodiments, on which a computer program is stored. When the program is executed by a processor, it implements the various steps of the multi-model-based AI resource management method in the above embodiments and can achieve the same technical effect, which will not be repeated here.

[0115] In summary, the AI ​​resource management method, apparatus, device, and storage medium provided by this invention, based on multiple models, can save resources as much as possible by creating or deleting AI channels according to the enabling status of detection models. By using a linked list stack, the latest acquired image data is stored at the top of the stack, and each detection model obtains data from the top of the stack, thus ensuring that the image data obtained by the detection model is up-to-date. Furthermore, each detection model only obtains the image data corresponding to a node, and the node remains in the linked list stack, thereby reducing data contention, achieving data sharing, and reducing frame waiting time. By adding a reference count flag to each node, the reference status of the image data corresponding to each node can be easily recorded, and the nodes in the linked list stack and their corresponding image data can be managed according to the reference count value. This separates the resource release time from the detection time, eliminating the need to consume additional time in the detection phase, thereby reducing the overall detection process time.

[0116] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A multi-model-based AI resource management method, characterized in that, include: Manage the AI ​​channel based on the enabling status of each detection model; Create a linked list stack; When the number of nodes in the linked list stack is less than a preset threshold and there is an AI channel, a frame of image data is obtained from the SDK, saved as a node, and the node is inserted into the linked list stack using the head insertion method. Based on the usage of image data corresponding to each node in the linked list stack by each detection model, adjust the reference count value of each node; Based on the reference count value of each node in the linked list stack, manage each node and its corresponding image data; The management of AI channels based on the enabling status of each detection model includes: Obtain the enabling status of each detection model; If at least one detection model is enabled, an AI channel is created, and the resolution of the AI ​​channel is determined based on the maximum resolution required by each detection model. If all detection models are disabled, then delete the AI ​​channel; The step of adjusting the reference count value of each node based on the usage of image data corresponding to each node in the linked list stack by each detection model is as follows: If a detection model obtains image data corresponding to a node in the linked list stack, the reference count value of that node is increased by a preset value. If the detection model releases the image data corresponding to the node, the reference count value of the node will be reduced by a preset value.

2. The AI ​​resource management method based on multiple models according to claim 1, characterized in that, After determining the resolution of the AI ​​channel based on the maximum resolution required by each detection model, the process further includes: If the maximum resolution changes, the AI ​​channel is recreated, and the resolution of the recreated AI channel is determined based on the changed maximum resolution.

3. The AI ​​resource management method based on multiple models according to claim 1, characterized in that, The step of managing each node and its corresponding image data based on the reference count value of each node in the linked list stack is as follows: Traverse each node in the linked list stack, delete nodes with a reference count of zero that are not the first node, and release the image data corresponding to the deleted nodes.

4. A multi-model-based AI resource management device, characterized in that, include: The first management module is used to manage the AI ​​channel based on the enabling status of each detection model; Create a module for creating linked list stacks; The insertion module is used to obtain a frame of image data from the SDK, save it as a node, and insert the node into the linked list stack using the head insertion method when the number of nodes in the linked list stack is less than a preset threshold and an AI channel exists. The adjustment module is used to adjust the reference count value of each node according to the usage of image data corresponding to each node in the linked list stack by each detection model. The second management module is used to manage each node and its corresponding image data based on the reference count value of each node in the linked list stack. The first management module includes: The acquisition unit is used to acquire the enabling status of each detection model; The first creation unit is configured to create an AI channel if at least one detection model is enabled, and determine the resolution of the AI ​​channel based on the maximum resolution required by each detection model. The deletion unit is used to delete the AI ​​channel if all detection models are disabled. The adjustment module includes: An additional unit is added to increment the reference count of a node by a preset value if a detection model acquires image data corresponding to a node in the linked list stack. The reduction unit is used to reduce the reference count value of a node by a preset value if the detection model releases the image data corresponding to the node.

5. The AI ​​resource management device based on multiple models according to claim 4, characterized in that, The first management module further includes: The second creation unit is used to recreate the AI ​​channel if the maximum resolution changes, and to determine the resolution of the recreated AI channel based on the changed maximum resolution.

6. The AI ​​resource management device based on multiple models according to claim 4, characterized in that, The second management module is specifically used to traverse each node in the linked list stack, delete nodes with a reference count of zero and that are not the first node, and release the image data corresponding to the deleted nodes.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-3.

Images