DSL-based visual application programming system, electronic equipment and storage medium

Abstract

The invention discloses a DSL (Digital Subscriber Line)-based visual application programming system, electronic equipment and a storage medium. The system comprises a hardware stack, a compiler, a DSL and a software stack, the hardware stack is used for providing a hardware stack containing a register transfer level source code, so that a developer performs a calculation task; the compiler and the DSL are used for providing the compiler for realizing the DSL so as to improve the development efficiency; and the software stack is used for providing a visual algorithm and an artificial intelligence algorithm so as to support a user to call. According to the method, by simplifying the development process, optimizing the calculation efficiency, improving the flexibility and reducing the power consumption, the development efficiency and the operation performance of visual processing and manual reasoning application are remarkably enhanced, and the method has wide application prospects and remarkable technical advantages.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and programming, and particularly relates to a vision application programming system, an electronic device, and a storage medium based on DSL. Background Art

[0002] With the wide application of artificial intelligence and computer vision technologies, the demand for image processing, object detection, and feature extraction in various industries is increasing continuously. However, most traditional computing methods rely on general-purpose programming languages and frameworks, making the development and optimization processes cumbersome and requiring high skills of developers. Especially in the scenarios of embedded systems and edge devices, with limited resources, more efficient vision processing methods are needed to meet the requirements of real-time performance and power consumption.

[0003] With the continuous increase in the number of intelligent devices, the Internet of Things (IoT) and edge computing are becoming new application trends. According to statistics, the number of global IoT connections is growing exponentially and is expected to exceed 29 billion by 2027. Edge computing has gradually attracted attention because it is close to the data source, can process and respond quickly, and reduces the dependence on remote cloud services. Therefore, efficiently deploying and running vision processing functions on edge devices has become an urgent problem to be solved.

[0004] In vision application development, the current solutions available for accelerating image processing and vision tasks mainly include: Traditional vision programming frameworks have been widely used in the field of computer vision. They encapsulate image processing and AI tasks as nodes, enabling developers to build vision application processes in a modular way. However, these frameworks still rely on general-purpose programming languages and complex programming interfaces, requiring high professional skills of developers and being difficult to meet the requirements in resource-constrained embedded environments.

[0005] Domain Specific Language (DSL) has been applied in fields such as finance and data analysis and can optimize the programming process for specific tasks. Although in the field of computer vision, the programming method based on DSL is still in the initial exploration stage, some studies have begun to attempt to combine DSL with vision applications to simplify the development work and reduce the dependence on underlying knowledge.

[0006] Although the above technical solutions play an important role in vision application development, in the scenarios of embedded and edge computing, these solutions have the following deficiencies in several aspects:

[0007] High programming complexity: Traditional vision programming frameworks usually require developers to master a large amount of underlying knowledge and manually optimize each algorithm. For example, to implement complex vision tasks in traditional vision programming frameworks, developers need to deeply understand image processing algorithms and data structure optimization, which has a steep learning curve for non-professional developers.

[0008] Lack of real-time processing ability: For vision applications with high real-time requirements, such as autonomous driving and security monitoring, the performance of existing general vision programming frameworks on embedded devices often fails to meet expectations. Embedded devices have limited computing power, and the encapsulation of general interfaces and functional modules in traditional frameworks brings additional performance overhead, affecting the real-time processing efficiency of tasks.

[0009] Insufficient scalability and flexibility: DSL is less applied in the vision field, lacking a dedicated language adapted to vision tasks, which limits its promotion in specific scenarios. At the same time, existing DSLs lack sufficient modular support and are difficult to quickly adapt to different vision application requirements. Summary of the Invention

[0010] To solve the above problems existing in the prior art, the present invention provides a vision application programming system, an electronic device, and a storage medium based on DSL.

[0011] The technical problems to be solved by the present invention are realized through the following technical solutions:

[0012] In a first aspect, the present invention provides a vision application programming system based on DSL, the system comprising: a hardware stack, a compiler and DSL, and a software stack;

[0013] The hardware stack is used to provide a hardware stack containing register transfer level source code to enable developers to perform computing tasks;

[0014] The compiler and DSL are used to provide a compiler implementing DSL to improve development efficiency;

[0015] The software stack is used to provide vision algorithms and artificial intelligence algorithms to support users for calling.

[0016] Optionally, the compiler and DSL include: tensor classes and multiple graph nodes;

[0017] The tensor classes are used to encapsulate tensor data as the transmission data between the graph nodes;

[0018] The multiple graph nodes are used to encapsulate various image processing functions to process the tensor data.

[0019] Optionally, the tensor classes include: input tensors, intermediate tensors, and output tensors;

[0020] The input tensors are used as the initial input data of the first graph node among the multiple graph nodes;

[0021] The intermediate tensors are used as the data exchanged between the multiple graph nodes;

[0022] The output tensor is used to store the final processing result obtained after passing through the multiple graph nodes.

[0023] Optionally, the processing functions corresponding to the tensor class include a constructor, a main operation function, and an auxiliary acquisition function.

[0024] Optionally, the processing functions corresponding to the multiple graph nodes include interface functions.

[0025] Optionally, the image processing functions encapsulated in the multiple graph nodes include Canny edge detection algorithm, Color-Reshape, Gaussian filter, Harris-based corner detection, optical flow algorithm-based motion detection, and image size scaling.

[0026] According to a second aspect of the embodiments of the present disclosure, there is provided an electronic device, including: a processor; a memory for storing processor-executable instructions;

[0027] Wherein, the processor is configured to: execute the executable instructions to implement the content of the DSL-based visual application programming system according to any one of the embodiments in the above first aspect.

[0028] According to a third aspect of the embodiments of the present invention, there is provided a computer-readable storage medium, on which computer program instructions are stored, and when the program instructions are executed by a processor, the content of the DSL-based visual application programming system according to the first aspect of the present invention is implemented.

[0029] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0030] In the above technical solution, the hardware stack provides a hardware stack containing complete Register Transfer Level (RTL) source code, which supports portability between multiple hardware platforms. Developers can use this set of hardware designs to achieve flexible and efficient computing tasks without having to design the hardware architecture from scratch; the compiler and DSL provide a powerful compiler that supports the implementation of the necessary DSL. Through the DSL, users can shield the complexity of the underlying hardware and focus on high-level logic. This design enables software engineers to apply the same software to hardware with different capacities and characteristics without having to deeply understand the details of the underlying hardware, thus greatly improving development efficiency and software portability; the software stack incorporates many common vision and artificial intelligence algorithms and supports direct user calls. In addition, the stack provides native support for frameworks, allowing users to efficiently run artificial intelligence workloads without having to retrain the model, making it suitable for rapid deployment and iteration on edge intelligent devices. By simplifying the development process, optimizing computing efficiency, enhancing flexibility, and reducing power consumption, it significantly improves the development efficiency and running performance of vision processing and artificial reasoning applications, with broad application prospects and significant technical advantages.

[0031] The following will further elaborate on the present invention in conjunction with the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic structural diagram of a DSL-based visual application programming system provided by an embodiment of the present invention;

[0033] Figure 2 is a schematic diagram of the operation process of a compiler and DSL proposed by an embodiment of the present invention;

[0034] Figure 3 is a block diagram of an electronic device for a DSL-based visual application programming system shown according to an exemplary embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] The following further describes the present invention in detail in conjunction with specific embodiments, but the implementation manners of the present invention are not limited thereto.

[0036] Figure 1 is a schematic structural diagram of a DSL-based visual application programming system provided by an embodiment of the present invention, as Figure 1 shown, the system includes: a hardware stack, a compiler and DSL, and a software stack;

[0037] The hardware stack is used to provide a hardware stack containing Register Transfer Level source code for developers to perform computing tasks.

[0038] It is understandable that the hardware stack provides a hardware stack containing the complete Register Transfer Level (RTL) source code, which supports portability between multiple hardware platforms. Developers can utilize this set of hardware designs to achieve flexible and efficient computing tasks without having to design the hardware architecture from scratch.

[0039] A compiler and DSL, used to provide a compiler for implementing the DSL to improve development efficiency.

[0040] It is understandable that the compiler and DSL provide a powerful compiler that supports implementing the necessary DSL. Through the DSL, users can shield the complexity of the underlying hardware and focus on high-level logic in development. This design enables software engineers to apply the same software to hardware with different capacities and characteristics without having to deeply understand the details of the underlying hardware, just by recompiling, thus greatly improving development efficiency and software portability.

[0041] A software stack, used to provide vision algorithms and artificial intelligence algorithms to support user calls.

[0042] It is understandable that the software stack has built-in many common vision and artificial intelligence algorithms to support direct user calls. In addition, the stack provides native support for the framework, allowing users to efficiently run artificial intelligence workloads without having to retrain the model, which is suitable for rapid deployment and iteration on edge intelligent devices. By simplifying the development process, optimizing computing efficiency, enhancing flexibility, and reducing power consumption, it significantly improves the development efficiency and running performance of vision processing and artificial reasoning applications, with broad application prospects and significant technical advantages.

[0043] It is worth mentioning that the above-mentioned hardware stack, compiler and DSL, and software stack together constitute a closed-loop development ecosystem, seamlessly connecting from software to hardware, forming an easy-to-use, efficient, and highly flexible architecture. Developers can utilize the hardware capabilities of the hardware acceleration system to accelerate computing tasks, and can also quickly implement various complex artificial intelligence and computer vision applications with the help of the compiler and software stack.

[0044] Optionally, the compiler and DSL include: tensor classes and multiple graph nodes;

[0045] Tensor classes, used to encapsulate tensor data as the transmission data between graph nodes;

[0046] Multiple graph nodes, used to encapsulate various image processing functions to process tensor data.

[0047] Optionally, the tensor classes include: input tensors, intermediate tensors, and output tensors;

[0048] An input tensor, used as the initial input data for the first graph node among multiple graph nodes;

[0049] An intermediate tensor, used as the data exchanged between multiple graph nodes;

[0050] An output tensor, used to store the final processing result obtained after passing through multiple graph nodes.

[0051] It can be understood that Figure 2 is a schematic diagram of a compiler and a DSL operation process proposed in an embodiment of the present invention. As Figure 2 shown, during the execution process, the input tensor is gradually passed to the graph nodes, and each node processes the data and then passes the result to the input tensor of the next node. The intermediate tensor is used to save the processing results of the corresponding graph nodes until the final result tensor is output after all nodes have been processed. Through this design of the graph structure, the present invention can efficiently process visual tasks and support parallel or pipeline processing of multiple visual applications.

[0052] Optionally, the processing functions corresponding to the tensor class include a constructor, a main operation function, and an auxiliary acquisition function.

[0053] It can be understood that the tensor class encapsulates a tensor data object, used to transfer data between graph nodes, and provides multiple construction methods and operation functions to support the creation, cloning, alias reference, and basic attribute acquisition of tensors. The following is the programming specification and related function description of the tensor class.

[0054] For example, 1. Constructor:

[0055] TZL(): The default constructor, without any initialization.

[0056] TZL(

[0057] TDataType_dataType,

[0058] TFormat_fmt,

[0059] TObjType objType,

[0060] std::vector <int>&dim,

[0061] void* shm): This constructor is used to initialize the tensor.

[0062] Parameter description:

[0063]

[0064]

[0065] 2. Main operation functions:

[0066] 2.1 Create: Called when using the default constructor to initialize the tensor object.

[0067] TST Create(

[0068] TDataType _dataType,

[0069] TFormat _fmt,

[0070] TObjType _objType,

[0071] std::vector <int>&dim,

[0072] SHARED_MEM_shm = 0);

[0073] Input Similar to the TZL() function Output TST, TSTOk: Creation successful. TSTFail: Creation failed.

[0074] 2.2 Clone: Creates a clone copy of another tensor and allocates a new memory block to store the same content.

[0075] TST Clone(T* other);

[0076]

[0077] 2.3 Alias(T* other): Initializes this tensor object as a reference to another tensor. The current tensor does not own the data content.

[0078] TST Alias(T* other);

[0079]

[0080] 2.4 Alias(void* _shm): References the data content of the tensor as the specified memory block without owning the memory block.

[0081] TST Alias(void* _shm);

[0082]

[0083] 2.5 CreateWithBitmap: Initializes the tensor according to the specified 24-bit bitmap file, suitable for loading image data.

[0084] TST CreateWithBitmap(const char* bmpFile, TFormat fmt = TFormatSplit);

[0085]

[0086] 3. Auxiliary access functions:

[0087] 3.1 TDataType GetDataType(): Returns the data type of the tensor.

[0088]

[0089] 3.2 TFormat GetFormat(): Returns the data layout format of the tensor.

[0090]

[0091] 3.3TObjType GetObjType(): Returns the object type of the tensor.

[0092]

[0093] 3.4std::vector <int>*GetDimension(): Returns the list of dimensions of the tensor.

[0094]

[0095] 3.5 int GetDimension(int _idx): Returns the size of the tensor in a specific dimension.

[0096]

[0097] 3.6 void* GetBuf(): Returns the address of the tensor data buffer.

[0098] Output void*, Address pointing to the tensor data buffer for direct access to the tensor data content.

[0099] 3.7 int GetBufLen(): Returns the total length of the tensor data buffer.

[0100] Output int, Represents the total number of bytes in the tensor data buffer.

[0101] 3.8 static size_t GetTSize(std::vector <int>&shape): Computes and returns the number of elements in a tensor along a specified dimension.

[0102] Input shape: std::vector <int>&, a vector containing the sizes of each dimension of the tensor. < / int> Output size_t, Specifies the total number of tensor elements in a dimension.

[0103] 4. Data Types and Formats:

[0104] 4.1 TDataType: Supports the following data types.

[0105] TDataTypeInt8 Signed 8-bit integer TDataTypeUint8 Unsigned 8-bit integer TDataTypeInt16 Signed 16-bit integer TDataTypeUint16 Unsigned 16-bit integer

[0106] 4.2 TFormat: Data layout format.

[0107]

[0108]

[0109] 4.3 TObjType: Tensor object type.

[0110]

[0111] Optionally, the image processing functions encapsulated in multiple graph nodes include Canny edge detection algorithm, Color-Reshape, Gaussian filter, Harris-based corner detection, motion detection based on optical flow algorithm, and image size scaling.

[0112] Optionally, the processing functions corresponding to multiple graph nodes include interface functions.

[0113] It can be understood that the graph node class is a template class, and the virtual functions in it are implemented by derived classes. The objects of the graph node class serve as the execution units in the graph, and various tensor acceleration functions are implemented through derived classes. The visual application acceleration functions are implemented through the encapsulation of graph nodes in derived classes.

[0114] For example, the following are the main interfaces and implementation specifications of the graph node class:

[0115] 1. Interface Functions:

[0116] 1.1 GN(): Default constructor, does not perform any initialization.

[0117] 1.2 TST Verify(): This is a virtual function implemented by derived classes.

[0118] Used to verify the integrity of the graph node and complete necessary initialization before starting execution.

[0119] Output TST, TSTOk: Verification successful. TSTFail: Verification failed.

[0120] 1.3TST Execute(int queue, bool stepMode): This is a virtual function implemented by derived classes.

[0121] Execute tasks related to this node.

[0122]

[0123] 1.4uint32_t GetJobId(int queue): Generate a unique job ID for tensor program execution.

[0124]

[0125] Optionally, the image processing functions encapsulated in multiple graph nodes include Canny edge detection algorithm, Color-Reshape, Gaussian filter, Harris-based corner detection, motion detection based on optical flow algorithm, and image size scaling.

[0126] In one implementation, the Canny edge detection algorithm includes:

[0127] 1. Constructor:

[0128] GNCanny(T* input, T* output)

[0129]

[0130]

[0131] 2. Create method:

[0132] TST Create(T* input, T* output): Used to initialize graph nodes after using the default constructor.

[0133] The parameters are the same as those of the constructor (input and output).

[0134] 3. SetThreshold method:

[0135] void STS(int_loTS, int_hiTS): Set the threshold for edge detection.

[0136]

[0137] 4. GetThreshold method:

[0138] void GTS(int*_loTS, int*_hiTS): Return the current threshold for edge detection.

[0139] Output _loTS: Current low threshold. _hiTS: Current high threshold.

[0140] Color-Reshape (Color Space Conversion and Tensor Reshaping), including:

[0141] 1. Constructor:

[0142]

[0143]

[0144]

[0145] 2. Create method:

[0146]

[0147] Used to initialize graph nodes after using the default constructor.

[0148] The parameters are the same as those of the constructor.

[0149] Gaussian filter, including:

[0150] 1. Constructor:

[0151] GNGaussian(T*input, T*output).

[0152] Input Input tensor for applying Gaussian filtering. Output Output tensor after blurring.

[0153] 2. Create method:

[0154] TST Create(T*input, T*output): Used to initialize graph nodes after using the default constructor.

[0155] The parameters are the same as those of the constructor (input and output).

[0156] 3. SetSigma method:

[0157] void SetSigma(float _sigma): Sets the sigma value (standard deviation) of the Gaussian filter, used to control the degree of blurring.

[0158] 4. GetSigma method:

[0159] float GetSigma(): Returns the sigma value of the current Gaussian filter.

[0160] Motion detection based on optical flow algorithm, including:

[0161] 1. Constructor:

[0162]

[0163]

[0164] 2. Create method:

[0165]

[0166]

[0167] Used to initialize the graph node after using the default constructor.

[0168] The parameters are the same as those of the constructor.

[0169] Image size scaling, including:

[0170] 1. Constructor:

[0171] GNResize(T* input, T* output, int w, int h)

[0172]

[0173] 2. Create method:

[0174] TST Create(T* input, T* output, int w, int h);

[0175] Used to initialize the graph node after using the default constructor.

[0176] The parameters are the same as those of the constructor.

[0177] It is worth mentioning that the present invention supports loading the TensorFlow Lite model as a graph node, so as to implement the artificial intelligence inference function in visual applications. Through the TFNN node, the user can load the model in TensorFlow Lite format (such as object detection, classification, etc.) and directly run the inference task on the edge device, which is applicable to visual application scenarios that require artificial intelligence recognition. The TFNN node can transfer data through input and output tensors, enabling seamless connection between artificial intelligence inference and other visual processing nodes.

[0178] For example, the programming specification of the TFNN node is as follows:

[0179] 1. Create method:

[0180] TST Create(const char*fname,T*_input,int numOutput,...): Loads a TensorFlow Lite model and prepares it for inference.

[0181]

[0182]

[0183] 2. Load method:

[0184] TST Load(const char*fname,T*_input,int numOutput,...): Loads and initializes the model.

[0185] The parameters are the same as those of the Create method.

[0186] 3. Unload method:

[0187] TST Unload(): Unloads and closes the current TensorFlow Lite model.

[0188] The graph structure programming framework of the present invention supports parallel execution of multiple graph instances and performs dynamic task scheduling through the step mode. Through the step mode, graph nodes can be executed step by step, allowing tasks of multiple graphs to run alternately, avoiding the problem of a single task occupying too many resources, and is suitable for multi-task processing requirements in edge computing scenarios. At the same time, the present invention supports allocating computing resources according to task priorities to optimize system resource utilization. The tensor class of the present invention provides rich interfaces to support operations such as tensor creation, initialization, cloning, and reference, allowing developers to flexibly configure the type, format, and dimension of tensors. In particular, the tensor class supports the function of initializing tensors from bitmap files, and can directly load 24-bit BMP images as input tensors for visual processing. This efficient tensor management method not only supports seamless transfer of visual data between graph nodes, but also can optimize memory usage and reduce the power consumption of data transmission.

[0189] Figure 3 is a block diagram of an electronic device for a DSL-based visual application programming system shown according to an exemplary embodiment, as Figure 3 shown, including a processor 301, a communication interface 302, a memory 303, and a communication bus 304. Among them, the processor 301, the communication interface 302, and the memory 303 complete mutual communication through the communication bus 304,

[0190] The memory is used to store computer programs;

[0191] When the processor is used to execute the program stored in the memory, it implements the content of any of the above DSL-based visual application programming systems according to the embodiments of the present invention.

[0192] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0193] The communication interface is used for communication between the above electronic device and other devices.

[0194] The memory may include a Random Access Memory (RAM), and may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0195] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0196] The method provided by the embodiments of the present invention can be applied to an electronic device. Specifically, the electronic device may be: a desktop computer, a portable computer, a smart mobile terminal, a server, etc. There is no limitation here, and any electronic device that can implement the present invention belongs to the protection scope of the present invention.

[0197] In another exemplary embodiment, the embodiments of the present invention also provide a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, it implements the content of any of the above DSL-based visual application programming systems according to the embodiments of the present invention.

[0198] For the apparatus / electronic device / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, please refer to the partial description of the method embodiments.

[0199] It should be noted that the electronic device and storage medium of the embodiments of the present invention are respectively the electronic device and storage medium applying the above DSL-based visual application programming system. All embodiments of the above DSL-based visual application programming system are applicable to the electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0200] The above are only the preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.< / int> < / int> < / int> < / int>

Claims

1. A visual application programming system based on DSL, characterized in that: The system includes: a hardware stack, a compiler, a DSL and a software stack; The hardware stack is used to provide a hardware stack including register transfer level source code to enable developers to perform computing tasks; The compiler and DSL are used to provide a compiler that implements the DSL to improve development efficiency; The software stack is used to provide visual algorithms and artificial intelligence algorithms to support user calls.

2. The DSL-based visual application programming system according to claim 1, characterized in that: The compiler and DSL include: a tensor class and a plurality of graph nodes; The tensor class is used to encapsulate tensor data to serve as transmission data between the graph nodes; The multiple graph nodes are used to encapsulate various image processing functions to process the tensor data.

3. The DSL-based visual application programming system according to claim 2, characterized in that: The tensor class includes: input tensor, intermediate tensor and output tensor; The input tensor is used as initial input data of a first graph node among the multiple graph nodes; The intermediate tensor is used as data exchanged between the multiple graph nodes; The output tensor is used to store the final processing result obtained after passing through the multiple graph nodes.

4. The DSL-based visual application programming system according to claim 2, characterized in that: The processing functions corresponding to the tensor class include a constructor, a main operation function and an auxiliary acquisition function.

5. The DSL-based visual application programming system according to claim 2, characterized in that: The processing functions corresponding to the multiple graph nodes include interface functions.

6. The DSL-based visual application programming system according to claim 2, characterized in that: The image processing functions encapsulated in the multiple graph nodes include Canny edge detection algorithm, Color-Reshape, Gaussian filter, Harris corner detection, optical flow algorithm-based motion detection and image size scaling.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to: execute the executable instructions to implement the content of the DSL-based visual application programming system according to any one of claims 1 to 6.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the contents of the DSL-based visual application programming system according to any one of claims 1 to 6 are implemented.

Images