Image processing flow arrangement method and device, and electronic equipment

By initializing the container and generating corresponding subprocesses, the problem of low efficiency in image processing flow orchestration is solved, the deployment efficiency and resource utilization of image processing flow are improved, and maintenance work is simplified.

CN117437526BActive Publication Date: 2024-11-08GRG BANKING EQUIPMENT CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202311287674.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-07
Publication Date
2024-11-08
Estimated Expiration
2043-10-07

AI Technical Summary

Technical Problem

In the prior art, the orchestration efficiency of the image processing flow is low, resulting in low deployment efficiency of the image processing module.

Method used

By reading the orchestration files of multiple image processing tasks, the container is initialized, and a sub-process corresponding to the multiple image processing tasks is generated. Each sub-process includes at least one initialized container, allowing at least two image processing tasks to multiplex one container. , determine the processing order and copying of the container.

Benefits of technology

It improves the generation efficiency of the image processing process, saves processing resources of electronic devices, simplifies the maintenance of the image processing process, and improves the practicality of the image processing process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117437526B_ABST
    Figure CN117437526B_ABST
Patent Text Reader

Abstract

The present application discloses a method and device for generating an image processing flow, and an electronic device, and relates to the field of algorithm deployment technology. The electronic device can initialize each preset container after reading multiple containers recorded in an arrangement file for multiple image processing tasks. Each image processing task is completed by at least one initialized container. Then, the electronic device can generate an image processing flow including multiple sub-flows corresponding to the multiple image processing tasks, and each sub-flow includes at least one initialized container. In this way, compared with the method of implementing algorithm deployment through manual coding, the generation efficiency of the image processing flow can be effectively improved, thereby improving the efficiency of deploying the image processing flow in the electronic device.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of algorithm deployment technology, and in particular to an image processing process arrangement method and device, and electronic equipment. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, AI technology can now be used to perform image processing tasks in various application scenarios, such as identifying vehicles in surveillance videos.

[0003] Before an electronic device uses AI technology to complete each image processing task, it is necessary to deploy an image processing module for completing the image processing task in the electronic device. Since multiple image processing modules are usually required to complete each image processing task, the deployment of the image processing module also includes: arranging multiple image processing modules into a processing flow so that the electronic device can process data according to the processing flow to complete the task.

[0004] In the related art, multiple image processing modules can be arranged into a processing flow by manual coding to complete the deployment of multiple image processing modules. However, in the related art, the efficiency of the arranged processing flow is low. Summary of the invention

[0005] The present application provides a method and device for arranging an image processing flow, and an electronic device, which can solve the problem of low efficiency of the arranged processing flow in the related art. The technical solution is as follows:

[0006] In one aspect, a method for generating an image processing flow is provided, the method comprising:

[0007] Reading an arrangement file for a plurality of image processing tasks, wherein the arrangement file records a plurality of containers;

[0008] Initializing each of the preset containers, wherein each of the image processing tasks is completed by at least one of the initialized containers, and at least two of the image processing tasks reuse at least one of the initialized containers;

[0009] An image processing process is generated, wherein the image processing process includes sub-processes corresponding to the multiple image processing tasks one by one, and each sub-process includes: a corresponding at least one initialized container.

[0010] Optionally, the number of the containers of the target image processing task among the multiple image processing tasks is at least two;

[0011] The sub-process corresponding to the target image processing task includes: at least two initialized containers arranged in a processing order.

[0012] Optionally, the arrangement file further records the next-level container of each container under each image processing task, and the next-level container is used to receive the processing result of the container; before generating the image processing process, the method further includes:

[0013] For the target image processing task, a processing order of at least two containers of the target image processing task is determined based on next-level containers of each container under the target image processing task.

[0014] Optionally, the arrangement file further records indication information for whether each of the containers is copied into multiple containers; if the indication information of any container among the multiple containers is used to indicate that the any container is copied into multiple containers, then after initializing the preset containers, the method further includes:

[0015] The initialized any container is copied into multiple containers, wherein the previous level container of the multiple containers is the same, and the next level container of the multiple containers is the same;

[0016] Wherein, any one of the containers is used to receive the processing result of the previous level container.

[0017] Optionally, each of the containers includes at least one engine, and each of the image processing tasks is completed by an engine in at least one of the containers; and initializing each of the preset containers includes:

[0018] For each of the containers, the engine in the container is initialized.

[0019] Optionally, each of the sub-processes is represented by a mapping relationship; any of the mapping relationships records an identifier of the image processing task corresponding to the sub-process and an identifier of at least one initialized container arranged in the processing order.

[0020] Optionally, after the image processing flow is generated, the method further includes:

[0021] Get the image to be processed;

[0022] The image is processed using a target sub-process among the multiple sub-processes.

[0023] In another aspect, a device for generating an image processing flow is provided, the device comprising:

[0024] A reading module, used for reading an arrangement file for a plurality of image processing tasks, wherein the arrangement file records a plurality of containers;

[0025] An initialization module, used to initialize each of the preset containers, wherein each of the image processing tasks is completed by at least one of the initialized containers, and at least two of the image processing tasks reuse at least one of the initialized containers;

[0026] A generating module is used to generate an image processing flow, wherein the image processing flow includes sub-flows corresponding to the multiple image processing tasks one by one, and each sub-flow includes: a corresponding at least one initialized container.

[0027] Optionally, the number of the containers of the target image processing task among the multiple image processing tasks is at least two;

[0028] The sub-process corresponding to the target image processing task includes: at least two initialized containers arranged in a processing order.

[0029] On the other hand, an electronic device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for generating an image processing flow as described in the above aspects when executing the computer program.

[0030] On the other hand, a computer program product comprising instructions is provided, and when the computer program product is run on the computer, the computer is enabled to execute the method for generating the image processing flow described in the above aspect.

[0031] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored to implement the method for generating the image processing flow described in the above aspects.

[0032] The beneficial effects of the technical solution provided by this application include at least:

[0033] The embodiments of the present application provide a method and device for generating an image processing flow, and an electronic device, which can initialize each preset container after reading multiple containers recorded in an arrangement file for multiple image processing tasks. Each image processing task is completed by at least one initialized container. Then, the electronic device can generate an image processing flow including multiple sub-flows corresponding to the multiple image processing tasks, and each sub-flow includes at least one initialized container. In this way, compared with the method of implementing algorithm deployment through manual coding, the generation efficiency of the image processing flow can be effectively improved, thereby improving the efficiency of deploying the image processing flow in the electronic device.

[0034] Furthermore, since the image processing process includes multiple sub-processes corresponding to multiple image processing tasks, that is, the image processing process can complete multiple different image processing tasks, the image processing process is more practical. In addition, since at least two image processing tasks can reuse at least one initialized container, the processing resources of the electronic device can be effectively saved, and the maintenance of the image processing process can be simplified in the later stage.

[0035] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a method for generating an image processing flow provided in an embodiment of the present application;

[0037] Figure 2 It is another method for arranging an image processing flow provided in an embodiment of the present application;

[0038] Figure 3 is a schematic diagram of an image processing flow provided in an embodiment of the present application;

[0039] Figure 4 It is a schematic diagram of a sub-process provided in an embodiment of the present application;

[0040] Figure 5 It is a structural schematic diagram of a device for generating an image processing flow provided in an embodiment of the present application;

[0041] Figure 6 It is a structural schematic diagram of another image processing flow generation device provided in an embodiment of the present application;

[0042] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0044] The embodiment of the present application provides a method for arranging an image processing flow, which can be applied to an electronic device. Optionally, the electronic device can be a mobile terminal, a server, or a camera. The mobile terminal can be a mobile phone or a tablet computer. The server can be a single server, or a server cluster composed of several servers, or a cloud computing service center. Figure 1 , the method comprising:

[0045] Step 101: Read the arrangement files for multiple image processing tasks.

[0046] The arrangement file records the identifiers of multiple containers. The identifier of each container can uniquely identify the container among the multiple containers. For example, the identifier of each container can be a code of the container. The code can be the order of arrangement of the container among the multiple containers.

[0047] In the embodiment of the present application, the arrangement file may be pre-stored in the electronic device. For example, a developer may pre-write an arrangement file according to a plurality of image processing tasks to be completed by the electronic device, and input the arrangement file into the electronic device.

[0048] Step 102: Initialize each preset container.

[0049] Each of the multiple image processing tasks is completed by at least one initialized container. At least two image processing tasks reuse at least one initialized container. That is, in an embodiment of the present application, at least one initialized container can be used to execute at least two image processing tasks. In this way, for at least two different image processing tasks, if the two image processing tasks need to use the same container, the electronic device only needs to initialize the container once, thereby effectively saving the processing resources of the electronic device.

[0050] In the embodiment of the present application, each container may include at least one preset engine. For each container in the plurality of containers, the electronic device may initialize the engine in the container to initialize the container.

[0051] Each engine is a packaged basic functional module that can realize a certain function. Initialization of the engine may at least include: instantiation of the engine. Each initialized engine can be used as an image processing module.

[0052] It is understandable that if a container includes multiple engines, the multiple engines can process data in sequence, that is, the multiple engines process data serially. And the multiple engines share a workspace. In this way, the phenomenon of multiple engines competing for computing resources can be avoided.

[0053] Step 103: Generate an image processing flow.

[0054] The image processing process includes a plurality of sub-processes corresponding to a plurality of image processing tasks. Each sub-process includes: at least one corresponding initialized container. The at least one corresponding initialized container refers to: at least one initialized container used to complete the image processing task corresponding to the sub-process.

[0055] In summary, an embodiment of the present application provides a method for generating an image processing flow, and an electronic device can initialize each preset container after reading multiple containers recorded in an arrangement file for multiple image processing tasks. Each image processing task is completed by at least one initialized container. Then, the electronic device can generate an image processing flow including multiple sub-flows corresponding to the multiple image processing tasks, and each sub-flow includes at least one initialized container. In this way, compared with the method of implementing algorithm deployment through manual coding, the generation efficiency of the image processing flow can be effectively improved, thereby improving the efficiency of deploying the image processing flow in the electronic device.

[0056] Furthermore, since the image processing process includes multiple sub-processes corresponding to multiple image processing tasks, that is, the image processing process can complete multiple different image processing tasks, the image processing process is more practical. In addition, since at least two image processing tasks can reuse at least one initialized container, the processing resources of the electronic device can be effectively saved, and the maintenance of the image processing process can be simplified in the later stage.

[0057] Figure 2 FIG. 1 is a schematic diagram of another method for generating an image processing flow provided in an embodiment of the present application, which method can be applied to electronic devices. Figure 2 , the method may include:

[0058] Step 201: Read the arrangement files for multiple image processing tasks.

[0059] Each image processing task may be a task implemented using AI technology, such as a vehicle recognition task and / or a pedestrian recognition task.

[0060] After the electronic device reads the arrangement file, it can parse and obtain the description information of the arrangement file. The description information may include: the identifier of each container in the multiple containers. The identifier of each container can be used to uniquely identify the container in the multiple containers. For example, the identifier of each container can be the code of the container.

[0061] In the embodiment of the present application, the arrangement file may be pre-stored in the electronic device. For example, the developer may pre-write the arrangement file according to the multiple image processing tasks to be completed by the electronic device by manually filling in the configuration parameters or by dragging and connecting in the design interface, and input the arrangement file into the electronic device.

[0062] Optionally, the format of the orchestration file can be protobuf, json, or xml. Among them, the protobuf format file can automatically check the parameter name and type of the orchestration file, helping developers avoid syntax errors in configuration.

[0063] Step 202: Initialize each preset container.

[0064] Each image processing task is completed by at least one initialized container. For example, a target image processing task among multiple image processing tasks can be completed by at least two initialized containers.

[0065] At least two image processing tasks reuse at least one initialized container. That is, in the embodiment of the present application, one initialized container can be used to execute at least two image processing tasks.

[0066] In an embodiment of the present application, for each container, the electronic device may initialize the engine in the container to initialize the container. The engine may be pre-configured by the electronic device, and correspondingly, the container is pre-configured by the electronic device. Each engine may process data by calling a model. The model may be a deep learning model or a traditional machine learning model. In addition, the engine may also implement business logic.

[0067] It is understandable that the description information of the arrangement file may also record: the identifier of at least one engine included in each container. For each container, the electronic device may call the engine from a pre-configured engine library based on the identifier of each engine in the container, and initialize the engine to initialize the container. The identifier of each engine may be used to uniquely identify the engine among multiple engines. For example, the identifier of each engine may be the name of the engine, or the code of the engine.

[0068] If a container includes multiple engines, the multiple engines can process data in sequence, that is, multiple engines process data serially. And the multiple engines share a workspace. In this way, it is possible to avoid the phenomenon of multiple engines competing for computing resources.

[0069] In an embodiment of the present application, the description information may include configuration information of each container and configuration information of each engine. The configuration information of each container may include: an identifier of the container. The configuration information of each engine may include: a name of the engine, a type of the engine, and configuration parameters of the engine under at least one container. The electronic device may obtain the identifier of each engine in at least one engine included in each container based on the configuration information of each container and the configuration information of each engine.

[0070] For example, an orchestration parser may be run in the electronic device, and the orchestration parser may parse description information of the orchestration file to determine, according to at least one container under the engine, an identifier of at least one engine included in each container.

[0071] It is understandable that the electronic device can initialize the engine based on the configuration parameters of the engine. The initialization of the engine includes: instantiation of the engine, assignment of values ​​to the engine, and loading of the engine plug-in.

[0072] Optionally, an instantiated functional component may be running in the electronic device, and the instantiated functional component may instantiate the engine based on configuration parameters of the engine in the container.

[0073] It should be noted that if the engine is a model inference engine, during the process of initializing the engine, it is also possible to apply for a shared workspace and a shared storage space, and load the engine into the memory of the electronic device.

[0074] It is understandable that the description information of the arrangement file also records the configuration information of each image processing task in the multiple image processing tasks. The configuration information of each image processing task may include: the identification of the image processing task, the configuration parameters of the storage resources pre-applied for the image processing task, the user-defined dynamic library (customLib) information that needs to be loaded, the supported data types, and the total number of data processing channels. Before initializing the engine, the electronic device can also apply for and build a storage resource pool based on the configuration parameters of the storage resource, so that the engine can apply for and reuse storage resources in the storage resource pool, so as to avoid affecting the performance of the electronic device due to the frequent application and release of storage resources.

[0075] The identifier of the image processing task may be the name of the image processing task, or the code of the image processing task. The code may be the order in which the image processing task is arranged in a plurality of image processing tasks. The storage resources may include: memory storage resources and video memory storage resources. The configuration parameters of the storage resources may include the size of the memory storage resources required to be applied for, and the size of the video memory storage resources.

[0076] The user-defined dynamic library may include: dynamic library path and parameters. The electronic device may load the dynamic library, so that the custom engine and the preset engine can be used simultaneously. The data type may be one of the following types: image and video, etc. The total number of data processing channels may be used to indicate whether the image processing task supports concurrent processing of multiple data channels. If the total number is greater than 1, it indicates that the image processing task supports concurrent processing of multiple data channels.

[0077] It is understandable that, before instantiating the engine, the electronic device may also create an object resource pool and instantiate the object resource pool for use by the instantiated engine.

[0078] In an embodiment of the present application, the engines in the engine library may include: preset engines and custom engines customized by developers. The preset engine may be a model inference engine for deep learning or machine learning, or it may be an artificial intelligence algorithm engine such as target tracking, target capture, or it may be an engine that can implement specific application business logic. The preset engine may not specify a specific algorithm scenario. Developers can configure different parameters for the preset engine so that the preset engine performs different algorithm tasks. It can be seen that the preset engine has strong versatility. For example, for a tracking engine based on image matching or feature matching, developers can configure the matching type (such as image or feature) to adapt the tracking engine to different application scenarios.

[0079] A custom engine is a custom engine (also called a customized engine) that is customized by the developer and registered in the engine library. When the engine in the pre-installed engine library in the electronic device cannot implement a certain image processing task, the developer can define and register a custom engine according to the preset rules to implement the image processing task. In this process, the developer can customize the processing functions of the engine (such as pre-processing functions, post-processing functions, and result output functions) and the types of data that can be processed.

[0080] Among them, the post-processing function needs to adapt to the developer's customized training framework, and the post-processing function can parse and export the reasoning results of the customized engine. The type of data needs to meet the algorithm prior information input by the developer and meet the requirements of outputting the results of non-basic data structures. For example, in the security prevention analysis engine, different intrusion rules and prevention areas can be set. In this way, a security prevention analysis engine can be applied to different security prevention application scenarios.

[0081] The result output function can return the processing result of the engine to the upper application layer. Since at least two image processing tasks in the embodiment of the present application may reuse an initialized container, that is, the engine included in the initialized container may need to execute at least two image processing tasks, the engine needs to set different output paths to support the output of data of different image processing tasks.

[0082] In the process of customizing the engine, developers can also customize plug-ins (also called probes) for the engine. The plug-in is a functional component of the engine, which is used to facilitate developers to view the intermediate processing results output by the custom engine during the development stage of the custom engine, and to debug the algorithm of the custom engine based on the intermediate processing results. Or, it is convenient for developers to insert algorithm business logic.

[0083] It is understandable that the electronic device may provide a register group, which may be used to implement the registration function of the engine. The register group may include an engine register, a plug-in register, a processing function register (such as a post-processing function register) and a data object pool register. Developers may use the register to develop and register an engine according to the definition of the engine, and the registered engine may be called.

[0084] Optionally, the model reasoning engine may include: a general model reasoning engine and a dedicated model reasoning engine. The configuration information of the model reasoning engine may also include: an engine version, a weight file path of the called model, and a parameter configuration file path. Among them, the weight file path is the storage path of the weight file of the model, and the parameter configuration file path is the storage path of the parameter configuration file of the model. The model reasoning engine can read the weight file of the model through the weight file path, and can read the parameter configuration file of the model through the parameter configuration file path, and load the model based on the read weight file and parameter configuration file to reason about the data.

[0085] Among them, the parameters of the model are recorded in the parameter configuration file, and the parameters may include: the input layer information and output layer information of the model. It is understandable that if the model is a widely used image classification model or detection algorithm model, the parameters may also include: the type of at least one (such as multiple) targets that the model can distinguish (or detect), and the threshold corresponding to each target. The threshold can determine the accuracy of the model in identifying the corresponding target. For example, the higher the threshold is set, the higher the accuracy of the model in identifying the corresponding target from the image to be processed.

[0086] For example, if a model is a human, motor vehicle and non-motor vehicle target detection model, the model can identify three different types of targets, such as pedestrians, motor vehicles and non-motor vehicles. The threshold corresponding to pedestrians can be set to 0.5, and the threshold corresponding to vehicles can be set to 0.65.

[0087] It is understandable that when developers are arranging files, if they need to replace the model called by the model inference engine, they can modify the name of the model; if they need to replace the version of the model, they can modify the configuration parameters of the version. In this way, code development for model docking can be avoided.

[0088] Optionally, the container can support single container mode or container group mode. Container group mode means: the same container is copied into multiple, providing multiple concurrent and independently processed channels for data, which is suitable for application scenarios where multiple channels of data are processed concurrently. It can be seen that container group mode can support concurrent processing of images (i.e., multiple channels of data are processed simultaneously). For example, in the task of identifying the target object in each frame of multiple frames without tracking the target object, the multiple frames can be divided into multiple groups, and each group of images can be processed using one channel.

[0089] In addition, container groups can also achieve the effect of isolating multiple channels. For example, in a task that requires target tracking, since target tracking requires the previous and next frame information of the video, when tracking multiple channels of video at the same time, the multiple channels of video need to be isolated from each other to avoid the problem of target tracking errors.

[0090] In an embodiment of the present application, the configuration information of each container may further include: indication information for indicating whether the container needs to be copied into multiple copies. If the electronic device determines that any of the multiple containers needs to be copied into multiple copies based on the indication information, the initialized container may be copied into multiple copies. The upper level containers of the multiple initialized containers are all the same, and the lower level containers of the multiple initialized containers are also the same. It can be seen that the data received by any of the multiple initialized containers comes from the same container, and the processing results of any of the multiple initialized containers are all input into the same container.

[0091] Optionally, the configuration information may include: a group field, and the field value of the group field may be used to indicate whether the container needs to be replicated into multiple copies, and if it needs to be replicated into multiple copies, the total number of required copies.

[0092] Since the initialized container can be copied to obtain a container group, data can be processed concurrently. Therefore, when the amount of data calculation is large, data processing efficiency can be improved by splitting the data so that each container included in the container group processes the data.

[0093] Step 203: Generate an image processing flow.

[0094] The image processing process includes a plurality of sub-processes corresponding to a plurality of image processing tasks. Each sub-process includes: at least one corresponding initialized container, that is, at least one initialized container used to complete the corresponding image processing task.

[0095] Optionally, the multiple image processing tasks may include a target image processing task, and the target image processing task is completed by at least two initialized containers. The sub-process corresponding to the target image processing task may include: at least two initialized containers arranged in a processing order. The processing order refers to: the order in which at least two containers process data in the process of completing the image processing task.

[0096] Optionally, there may be multiple target image processing tasks.

[0097] In an embodiment of the present application, the configuration information of each container may further include: an identifier of a next-level container of the container. The next-level container is used to receive the processing result of the container. It is understandable that, for each container, if the container has a next-level container, the configuration information may include: an identifier of a next-level container of the container. The number of the next-level containers may be at least one. At least one next-level container corresponds to at least one image processing task executed by the container. Each next-level container is used to receive the processing result after the container executes a corresponding image processing task.

[0098] Based on this, for the target image processing task, the electronic device may determine the processing order of at least two containers of the target image processing task based on the next level container of each container under the target image processing task.

[0099] In the embodiment of the present application, the image processing process can be represented by multiple mapping relationships corresponding to multiple sub-processes. Each mapping relationship records: the identifier of the corresponding image processing task and the identifier of at least one initialized container used to complete the image processing task.

[0100] For example, assume that multiple image processing tasks include: obtaining structured information of one or two of the three categories of targets: motor vehicles (such as cars), non-motor vehicles, and pedestrians. The structured information includes: the category of the object (i.e., the motor vehicle, non-motor vehicle, and pedestrian mentioned above), and the attribute information of the object (e.g., for motor vehicles, the attribute information may include the color of the vehicle and whether there is a license plate, etc.). For example, for surveillance videos on highways, obtain structured information of motor vehicles; for surveillance videos at pedestrian shopping streets or school gates, obtain structured information of pedestrians and non-motor vehicles; for surveillance videos on urban roads, obtain structured information of motor vehicles, non-motor vehicles, and pedestrians.

[0101] Since obtaining the structured information of motor vehicles, non-motor vehicles and pedestrians is three different image processing tasks, we can get If one algorithm flow supports one image processing task, seven algorithm flows need to be developed, deployed and maintained. However, if the algorithm flow arrangement method provided by the embodiment of the present application is used, one algorithm flow can meet the needs of seven algorithm tasks. The following is a detailed description:

[0102] In the configuration file compilation stage, the developer can record the motor vehicle, non-motor vehicle and pedestrian as A, B and C respectively. Assume that the multiple image processing tasks include seven image processing tasks: obtaining structured information of A, B, C, AB, AC, BC and ABC, and the identifications of the seven image processing tasks are respectively: mode 1 to mode 7. Among them, obtaining the structured information of AB means: obtaining the structured information of A and the structured information of B, and obtaining the structured information of ABC means: obtaining the structured information of A, the structured information of B and the structured information of C.

[0103] Assume that in the process of obtaining the structured information of A, B, C, AB, AC, BC and ABC, a total of 4 engines are required. The 4 engines include: a target detection engine, a motor vehicle (ie A) attribute analysis engine, a non-motor vehicle (ie B) attribute analysis engine, and a pedestrian (ie C) attribute analysis engine. Assuming that each container includes an engine, the developer can get five containers, which are: target detection container Container_Detect, motor vehicle attribute analysis container Container_Analysis_A, non-motor vehicle attribute analysis container Container_Analysis_B, pedestrian attribute analysis container Container_Analysis_C, and output container Container_Output. Among them. The output container is also the process end container, and all image processing tasks must be configured.

[0104] Then, for each image processing task, the developer can determine at least one container that completes the image processing task. And in the case where the number of at least one container of a certain image processing task is at least two, the developer also needs to determine the processing order of at least two containers. The developer can then configure the completion arrangement file according to the container that completes the image processing task and the processing order of the container. The arrangement file may include:

[0105] mode{

[0106] name: "A"

[0107] id:1

[0108] Type: "image"

[0109] }

[0110] …

[0111] mode{

[0112] name:“ABC”

[0113] id:7

[0114] Type: "image"

[0115] }

[0116] Configuration information of container 0:

[0117] Container{

[0118] name: "Detect_A_B_C"

[0119] Unique_id: 0

[0120] output{

[0121] mode_id:1

[0122] Containere_id:1

[0123] }

[0124] …

[0125] output{

[0126] mode_id:4

[0127] Containere_id:1

[0128] }

[0129] }

[0130] Engine configuration information

[0131] engine

[0132] {

[0133] name "Detect_A_B_C"

[0134] type: "infer"

[0135] filter

[0136] {

[0137] primary: true

[0138] }

[0139] model_param

[0140] {

[0141] mame "Detect_A_B_C"

[0142] …

[0143] }

[0144] }. It can be seen that the orchestration file may include: configuration information of the image processing task, configuration information of the container, and configuration information of the engine.

[0145] In the generation stage of the image processing flow, the electronic device can read the description information of the arrangement file and initialize the engine in each container. Then, for image processing tasks that require at least two containers to complete, the electronic device can obtain the processing order of at least two initialized containers in the image processing task according to the identifier of the next-level container of each container in the arrangement file, and then generate a task-container mapping relationship for each image processing task. As shown in Table 1 according to the arrangement file described above, the mapping relationship between each task and container generated by the electronic device can be shown in Table 1. As can be seen from Table 1, the sub-process corresponding to the image processing task of obtaining the structured information of A is: (Container_Detect)->(Container_Analysis_A)->(Container_Output). The sub-process corresponding to the image processing task of obtaining the structured information of ABC is: (Container_Detect)->(Container_Analysis_A)->(Container_Analysis_B)->(Container_Analysis_C)->(Container_Output).

[0146] Table 1

[0147]

[0148] According to the above mapping relationship, the electronic device can obtain Figure 3 The image processing flow shown in Figure 1 is as follows. Figure 3It can be seen that the image processing process can include 7 sub-processes corresponding to the 7 image processing tasks. Among them, the sub-process for obtaining the structured information of the motor vehicle includes: container Container_Detect, container Container_Analysis_A, and container Container_Output. At this time, the processing order of the containers is Container_Detect->Container_Analysis_A->Container_Output. That is, container Container_Analysis_A is the next level container of container Container_Detect, and container Container_Output is the next level container of container Container_Analysis_A.

[0149] The sub-process for obtaining structured information of motor vehicles, non-motor vehicles and pedestrians includes: container Container_Detect, container Container_Analysis_A, container Container_Analysis_B, container Container_Analysis_C, and container Container_Output. At this time, the processing order of the containers is Container_Detect->Container_Analysis_A->Container_Analysis_B->Container_Analysis_C->Container_Output. That is, container Container_Analysis_A is the next level container of container Container_Detect, container Container_Analysis_B is the next level container of container Container_Analysis_A, container Container_Analysis_C is the next level container of container Container_Analysis_B, and container Container_Output is the next level container of container Container_Analysis_C.

[0150] And from Figure 3 It can also be seen that the sub-process for obtaining the structured information of motor vehicles and the sub-process for obtaining the structured information of motor vehicles and non-motor vehicles reuse the container Container_Analysis_A. The sub-process for obtaining the structured information of motor vehicles and pedestrians and the sub-process for obtaining the structured information of non-motor vehicles and pedestrians reuse the container Container_Analysis_C.

[0151] Since one image processing flow provided in the embodiment of the present application can meet the requirements of seven image processing tasks, the development cost, application cost and maintenance cost can be reduced to a large extent. In addition, the method provided in the embodiment of the present application can flexibly combine multiple engines to generate the image processing flow.

[0152] For example, assume that a sub-process includes: initialized container 0 to initialized container 2, initialized container 1 needs to be copied into 3, and the upper level container of initialized container 1 is container 0, and the lower level container is container 2. Then the sub-process obtained by the electronic device can be as follows Figure 4 shown.

[0153] from Figure 4 It can be seen that the container group obtained by copying the initialized container 1 includes: initialized container 1-1, initialized container 1-2 and initialized container 1-3. The upper level container of initialized container 1-1 to initialized container 1-3 is initialized container 0, and the lower level container is initialized container 2. In addition, see Figure 4 , the processing data of the initialized container 1-1 to the initialized container 1-3 all come from the data queue of the container group.

[0154] In the embodiment of the present application, an orchestration flow visualization tool may also be run in the electronic device. The visualization tool may read the orchestration file and generate a wireframe flow chart based on the orchestration file for developers to view.

[0155] Step 204: Acquire the image to be processed.

[0156] The image may be read locally by the electronic device, downloaded from a network, or sent to the electronic device by a sending device.

[0157] Optionally, the number of the images may be one or more. In the case where the number of images is multiple, the multiple images may be multiple frames of images in a video. That is, in the embodiment of the present application, the object processed by the image processing flow may be a video (such as a real-time video stream or an offline video stream).

[0158] In an embodiment of the present application, a preprocessing module is running in the electronic device. The preprocessing module can preprocess the image to be processed and input the preprocessed image into the image processing process. Among them, the process of preprocessing the image by the preprocessing module can include: decoding the image, formatting the decoded image, generating structured data from the format-converted image, and adding pattern information to the structured data to obtain a structured arrangement data object. Since the structured data can be stored in the form of a unified two-dimensional table, it is convenient to connect different engines with a unified interface.

[0159] The structured data may include basic attribute information such as pixels of an image (such as a video frame), width and height of the image, and pixel arrangement type of the image. The basic attribute information is the basic information required for the engine to process data. The structured arrangement data object may include attribute information such as the structured data, data mode information, data structure type of the inference result of the preset engine, and data structure type of the output result of the custom engine.

[0160] The data mode information is necessary information for the electronic device to select a target sub-process for the image to be processed. The mode information is used to indicate the identification of the image processing task that can achieve the processing purpose of the image. The data structure type of the inference result of the model inference engine (such as the target detection engine, the classification engine, and the feature recognition engine) is a general structure type.

[0161] It is understandable that the format of the decoded image can be a format that is easier to process by the hardware of the layout device. For example, the format of the image before decoding can be RGB format, and the format of the decoded image can be BGR format or NV12 format.

[0162] It can also be understood that the process of the electronic device generating a structured arrangement data object may include: detecting whether the current object pool is empty, if the current object pool is empty, creating an object, and assigning a value to the object based on the mode information and the structured data to obtain the structured arrangement data object. If the current object pool is not empty, obtaining an object from the object pool for resetting, and assigning a value to the reset object based on the mode information and the structured data to obtain the structured arrangement data object.

[0163] Optionally, the preprocessing module may be a data handle. The data handle is independent of the image processing process. The data handle may be set with different thread blocking modes to provide synchronous or asynchronous processing for the data.

[0164] Step 205: Process the image according to the target sub-process in the multiple sub-processes.

[0165] Among them, using the target sub-process to process the image can achieve the processing purpose of the image.

[0166] The embodiment of the present application takes the case where the target sub-process includes multiple containers as an example to exemplarily illustrate the process of an electronic device processing an image according to the target sub-process:

[0167] The electronic device may first input the preprocessed image into the first initialized container in the target sub-process, and then input the processing result of the first initialized container into the second initialized container, and so on, until the processing result of the last initialized container of the target sub-process is obtained, and then it can be determined that the processing of the image is completed.

[0168] It is understandable that after the last initialized container outputs the processing result, the electronic device can return the processing result to the application layer in the manner specified by the orchestration file. For example, the processing results of each initialized container can be output in the form of a structured orchestration data object. After returning the processing result to the application layer, the system object pool of the electronic device can recycle the structured orchestration data object, and the process of the structured orchestration data object can be terminated.

[0169] In the embodiment of the present application, each initialized container has a data queue, which can be used to receive data to be processed (such as the processing result of the container initialized at the previous level or the data sent by the preprocessing module). The initialized container can obtain data in batches from the data queue as the input of the initialized container. In this way, the processing speed of the initialized container can be improved.

[0170] The preprocessing module of the electronic device can push the structured arrangement data object generated based on the image to the data queue of the first initialized container. The first initialized container can batch obtain the structured arrangement data object from the data queue and process the structured arrangement data object.

[0171] Optionally, each initialized container may also include a data filter. The data filter is a component of the engine, which is used to filter the processing objects of the current engine and filter out the data objects that the engine does not need. The filter object of the data filter is the processing result of the previous initialized container of the current initialized container (the processing can be transmitted in the form of a data object), or a structured arrangement data object sent by the preprocessing module.

[0172] The data filter can obtain data objects (such as structured orchestration data objects) in batches from the data queue of the initialized container, and filter out specified data objects from the obtained multiple data objects according to the filter parameters set in the orchestration file. Then, the data filter can push the filtered data objects to the engine in the initialized container for the engine to process the data objects.

[0173] For example, when the screening object is the processing result of the previous level initialized container of the current initialized container, the data filter can filter out the specified data object according to the filtering parameter and the object type or object name of the data object, wherein the object type and object name are generated by the engine in the previous level initialized container.

[0174] It is understandable that the order of the steps of the method for generating the image processing flow provided in the embodiment of the present application can be adjusted appropriately, and the steps can be increased or decreased accordingly according to the situation. For example, step 203 can be deleted according to the situation; or, step 203 can be executed before step 202; or, step 205 and step 206 can also be deleted according to the situation. Any technician familiar with the technical field can easily think of a method of change within the technical scope disclosed in this application, which should be covered within the scope of protection of this application, so it will not be repeated.

[0175] In summary, an embodiment of the present application provides a method for generating an image processing flow, and an electronic device can initialize each preset container after reading multiple containers recorded in an arrangement file for multiple image processing tasks. Each image processing task is completed by at least one initialized container. Then, the electronic device can generate an image processing flow including multiple sub-flows corresponding to the multiple image processing tasks, and each sub-flow includes at least one initialized container. In this way, compared with the method of implementing algorithm deployment through manual coding, the generation efficiency of the image processing flow can be effectively improved, thereby improving the efficiency of deploying the image processing flow in the electronic device.

[0176] Furthermore, since the image processing process includes multiple sub-processes corresponding to multiple image processing tasks, that is, the image processing process can complete multiple different image processing tasks, the image processing process is more practical. In addition, since at least two image processing tasks can reuse at least one initialized container, the processing resources of the electronic device can be effectively saved, and the maintenance of the image processing process can be simplified in the later stage.

[0177] Figure 5is a schematic diagram of the structure of a device for generating an image processing flow provided in an embodiment of the present application. The device can be used to execute the method for generating an image processing flow provided in the above method embodiment. Figure 5 , the device 300 comprises:

[0178] The reading module 301 is used to read the arrangement file for multiple image processing tasks, where multiple containers are recorded in the arrangement file.

[0179] The initialization module 302 is used to initialize each preset container, wherein each image processing task is completed by at least one initialized container, and at least two image processing tasks reuse at least one initialized container.

[0180] The generation module 303 is used to generate an image processing flow, the image processing flow includes sub-flows corresponding to a plurality of image processing tasks one by one, and each sub-flow includes: at least one corresponding initialized container.

[0181] Optionally, the number of containers of the target image processing task among the multiple image processing tasks is at least two;

[0182] The sub-process corresponding to the target image processing task includes: at least two initialized containers arranged in a processing order.

[0183] Optionally, the arrangement file also records the next-level container of each container under each image processing task, and the next-level container is used to receive the processing result of the container. Figure 6 , the apparatus 300 may further include:

[0184] The determination module 304 is used to determine, for the target image processing task, a processing order of at least two containers of the target image processing task based on the next-level containers of each container under the target image processing task.

[0185] Optionally, the arrangement file also records indication information for whether each container is copied into multiple copies. The device 300 may also include:

[0186] A copy module 305 is used to copy any initialized container into multiple containers, wherein the containers at the previous level of the multiple containers are the same, and the containers at the next level of the multiple containers are the same;

[0187] Among them, any container is used to receive the processing result of the previous level container.

[0188] Optionally, each container includes at least one engine, and each image processing task is completed by an engine in at least one container. The initialization module 302 can be used to:

[0189] For each container, initialize the engine in the container.

[0190] Optionally, each sub-process is represented by a mapping relationship; any mapping relationship records the identifier of the image processing task corresponding to the sub-process and the identifier of at least one initialized container arranged in a processing order.

[0191] Optional, such as Figure 6 As shown, the device 300 may also include:

[0192] The acquisition module 306 is used to acquire the image to be processed.

[0193] The processing module 307 is used to process the image using a target sub-process among the multiple sub-processes.

[0194] In summary, an embodiment of the present application provides a device for generating an image processing flow, which can initialize each preset container after reading multiple containers recorded in an arrangement file for multiple image processing tasks. Each image processing task is completed by at least one initialized container. Then, the electronic device can generate an image processing flow including multiple sub-flows corresponding to the multiple image processing tasks, and each sub-flow includes at least one initialized container. In this way, compared with the method of implementing algorithm deployment through manual coding, the generation efficiency of the image processing flow can be effectively improved, thereby improving the efficiency of deploying the image processing flow in the electronic device.

[0195] Furthermore, since the image processing process includes multiple sub-processes corresponding to multiple image processing tasks, that is, the image processing process can complete multiple different image processing tasks, the image processing process is more practical. In addition, since at least two image processing tasks can reuse at least one initialized container, the processing resources of the electronic device can be effectively saved, and the maintenance of the image processing process can be simplified in the later stage.

[0196] Figure 7 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the controller 400 includes: a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, such as through a bus 402. Optionally, the controller 400 may also include a transceiver 404. It should be noted that in actual applications, the transceiver 404 is not limited to one, and the structure of the controller 400 does not constitute a limitation on the embodiments of the present application.

[0197] Processor 401 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 401 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0198] The bus 402 may include a path to transmit information between the above components. The bus 402 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0199] The memory 403 is used to store a computer program corresponding to the method for generating the image processing flow in the above embodiment of the present application, and the computer program is controlled and executed by the processor 401. The processor 401 is used to execute the computer program stored in the memory 403 to implement the contents shown in the above method embodiment.

[0200] The embodiment of the present application provides a computer program product including instructions, and when the computer program product is run on a computer, the computer executes the method for generating an image processing flow provided in the above method embodiment. For example, Figure 1 or Figure 2 The method shown.

[0201] The embodiment of the present application provides a computer program product including instructions, and when the computer program product is run on a computer, the computer is caused to execute the method for generating an image processing flow provided in the above method embodiment. For example, Figure 1 or Figure 2 The method shown.

[0202] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0203] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0204] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0205] In the description of the present application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0206] In the present application, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0207] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for generating an image processing flow, characterized in that: The method comprises: Reading an arrangement file for a plurality of image processing tasks, wherein the arrangement file records a plurality of containers; Initializing each of the preset containers, wherein each of the image processing tasks is completed by at least one of the initialized containers, and at least two of the image processing tasks reuse at least one of the initialized containers; Generate an image processing process, wherein the image processing process includes sub-processes corresponding to the plurality of image processing tasks one by one, and each sub-process includes: a corresponding at least one initialized container; The container supports a container group mode, the container group mode supports concurrent processing of multiple data paths, and the arrangement file further records indication information for indicating whether each of the containers is copied into multiple containers; if the indication information of any container among the multiple containers is used to indicate that the any container is copied into multiple containers, then after initializing the preset containers, the method further includes: The initialized any container is copied into multiple containers, wherein the previous level containers of the multiple containers are the same, and the next level containers of the multiple containers are the same, so as to obtain a container group; Wherein, any one of the containers is used to receive the processing result of the previous level container.

2. The method according to claim 1, characterized in that The number of the containers of the target image processing task among the multiple image processing tasks is at least two; The sub-process corresponding to the target image processing task includes: at least two initialized containers arranged in a processing order.

3. The method according to claim 2, characterized in that The arrangement file also records the next-level container of each container under each image processing task, and the next-level container is used to receive the processing result of the container; Before generating the image processing flow, the method further includes: For the target image processing task, a processing order of at least two containers of the target image processing task is determined based on next-level containers of each container under the target image processing task.

4. The method according to any one of claims 1 to 3, characterized in that: Each of the containers includes at least one engine, and each of the image processing tasks is completed by an engine in at least one of the containers; The initializing each of the preset containers includes: For each of the containers, initializing the engine in the container; Initializing the engine includes: instantiating the engine, assigning values ​​to the engine, and loading a plug-in of the engine.

5. The method according to claim 4, characterized in that In the case that there are plural engines, the engines process data in sequence.

6. The method according to any one of claims 1 to 3, characterized in that: Each of the sub-processes is represented by a mapping relationship; any of the mapping relationships records an identifier of the image processing task corresponding to the sub-process and an identifier of at least one initialized container.

7. The method according to any one of claims 1 to 3, characterized in that: After the image processing flow is generated, the method further includes: Obtain the image to be processed; The image is processed using a target sub-process among the plurality of sub-processes.

8. A device for generating an image processing flow, characterized in that: The device comprises: A reading module, used for reading an arrangement file for a plurality of image processing tasks, wherein the arrangement file records a plurality of containers; An initialization module, used to initialize each of the preset containers, wherein each of the image processing tasks is completed by at least one of the initialized containers, and at least two of the image processing tasks reuse at least one of the initialized containers; A generating module, configured to generate an image processing flow, wherein the image processing flow includes sub-flows corresponding to the plurality of image processing tasks one by one, and each sub-flow includes: a corresponding at least one initialized container; The container supports a container group mode, the container group mode supports concurrent processing of multiple data paths, and the arrangement file further records indication information for indicating whether each container is copied into multiple copies; the device further includes: a copy module, configured to, if the indication information of any container among the multiple containers is used to indicate that the any container is to be copied into multiple containers, copy the initialized any container into multiple containers, the previous level containers of the multiple containers are the same, and the next level containers of the multiple containers are the same, so as to obtain a container group; Wherein, any one of the containers is used to receive the processing result of the previous level container.

9. An electronic device, characterized in that: The electronic device comprises: a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Virtual machine batch creation method and device, electronic equipment and storage medium

    CN114327761A

  • Flow processing method and system and electronic equipment

    CN115033349A

  • Algorithm pipeline arrangement method and device, electronic equipment and storage medium

    CN115167865A

  • Visual service orchestration system based on micro-service architecture

    CN115964036A

  • Containerized DevOps method, system and equipment capable of performing task arrangement

    CN116339749A