Multi-model reasoning process processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a multi-model reasoning process processing method and device, electronic equipment and a storage medium, and can be widely applied to the technical field of model processing. According to the application, after the preset domain characteristic language information representing the preset implementation process is constructed, the preset domain characteristic language information is analyzed to obtain the first abstract syntax tree, and then the first abstract syntax tree is optimized to obtain the second abstract syntax tree; then, after a data structure representing multiple models and an abstract interface for calculating an operation instruction are constructed, an instruction sequence is generated according to the second abstract syntax tree and the abstract interface, and the instruction sequence corresponding to the service information is run through the target back-end architecture; therefore, a complex model calling function can be realized without setting a configuration file and modifying a source code, so that the model calling accuracy and the model operation stability can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of model processing technology, and in particular to a method and device for processing a multi-model reasoning process, an electronic device, and a storage medium. Background Art

[0002] In related technologies, the multi-model reasoning architecture defines the reasoning sequence of multiple models through configuration files, or directly defines the calling functions of multiple models in the code, so as to obtain a complete implementation process including data processing, multi-model serial or parallel, model reasoning, accuracy evaluation, deployment, etc. Using configuration files for definition can only simply define the execution order of multiple models, and cannot implement complex model calling functions; while defining directly in the code, because it is necessary to configure several parameters, it is easy to make mistakes in the definition, which leads to the failure of multi-model calling. In addition, the definition process needs to be operated in the underlying code, which may reduce the stability of model operation.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a processing method and device, electronic device and storage medium for a multi-model reasoning process, which can realize complex model calling functions and improve model calling accuracy and model operation stability.

[0005] To achieve the above purpose, an embodiment of the present application provides a method for processing a multi-model reasoning process, the method comprising the following steps:

[0006] Constructing preset domain characteristic language information, wherein the preset domain characteristic language information is used to represent a preset implementation process, wherein the preset implementation process includes a connection relationship between multiple models;

[0007] Parsing the preset domain-specific language information to obtain a first abstract syntax tree;

[0008] Optimizing the first abstract syntax tree to obtain a second abstract syntax tree;

[0009] Constructing an abstract interface, wherein the abstract interface is used to represent data structures and computing operation instructions of multiple models;

[0010] Generate an instruction sequence according to the second abstract syntax tree and the abstract interface;

[0011] The instruction sequence corresponding to the business information is executed through the target backend architecture.

[0012] In some embodiments, the step of constructing preset domain-specific language information includes:

[0013] Obtain business information;

[0014] Constructing preset domain-specific language information according to the business information, wherein the preset domain-specific language information includes input variables, output variables, intermediate variables, calculation instructions, global variables or input variable sequences;

[0015] The input variables and the output variables are used to represent the input and output information of the preset implementation process;

[0016] The intermediate variables are used to characterize the information flow between models;

[0017] The calculation instructions are used to represent the reasoning information or operation information of multiple models in the preset implementation process;

[0018] The global variable is used to control the resource consumption state of the preset implementation process;

[0019] The input variable sequence is used to represent the parallel processing data of multiple models in the preset implementation process.

[0020] In some embodiments, the parsing of the preset domain-specific language information to obtain a first abstract syntax tree includes:

[0021] Performing lexical analysis on the preset domain characteristic language information to obtain a token sequence;

[0022] Performing grammatical analysis on the flag sequence to obtain a preset report statement expression;

[0023] The preset report statement expression is combined into an abstract syntax tree as the first abstract syntax tree.

[0024] In some embodiments, optimizing the first abstract syntax tree to obtain a second abstract syntax tree includes:

[0025] Obtain several optimization sub-algorithms to be selected;

[0026] Selecting at least one optimization sub-algorithm to be selected from the plurality of optimization sub-algorithms to be selected as a target optimization sub-algorithm;

[0027] Combining all the target optimization sub-algorithms into a target optimization algorithm;

[0028] The first abstract syntax tree is optimized by using the target optimization algorithm to obtain a second abstract syntax tree.

[0029] In some embodiments, optimizing the first abstract syntax tree by using the target optimization algorithm to obtain a second abstract syntax tree includes:

[0030] Performing type judgment on each module or each code in the first abstract syntax tree to obtain a type judgment result;

[0031] According to the type judgment result, matching the target optimization sub-algorithm corresponding to each module or each code in the target optimization algorithm;

[0032] The first abstract syntax tree is optimized according to the matching result to obtain a second abstract syntax tree.

[0033] In some embodiments, generating an instruction sequence according to the second abstract syntax tree and the abstract interface includes:

[0034] Compiling the second abstract syntax tree and calling the abstract interface;

[0035] When the abstract interface exists and the data structure corresponding to the abstract interface matches the data structure of the second abstract syntax tree, the instruction sequence is generated according to the compilation result.

[0036] In some embodiments, the executing the instruction sequence corresponding to the business information through the target backend architecture includes:

[0037] Building the first virtual operating machine;

[0038] Dynamically loading the target backend architecture in the first virtual operating machine according to the business information to obtain a second virtual operating machine;

[0039] The instruction sequence corresponding to the business information is executed by the second virtual execution machine.

[0040] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a processing device for a multi-model reasoning process, the device comprising:

[0041] The first module is used to construct preset domain characteristic language information, wherein the preset domain characteristic language information is used to represent a preset implementation process, and the preset implementation process includes a connection relationship between multiple models;

[0042] The second module is used to parse the preset domain characteristic language information to obtain a first abstract syntax tree;

[0043] A third module is used to optimize the first abstract syntax tree to obtain a second abstract syntax tree;

[0044] The fourth module is used to construct an abstract interface, wherein the abstract interface is used to represent the data structure and computing operation instructions of multiple models;

[0045] A fifth module, configured to generate an instruction sequence according to the second abstract syntax tree and the abstract interface;

[0046] The sixth module is used to run the instruction sequence corresponding to the business information through the target backend architecture.

[0047] To achieve the above object, another aspect of an embodiment of the present application provides an electronic device, including:

[0048] at least one processor;

[0049] at least one memory for storing at least one program;

[0050] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0051] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0052] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and device for processing a multi-model reasoning process, an electronic device and a storage medium. The scheme constructs preset domain-specific language information that represents a preset implementation process, parses the preset domain-specific language information to obtain a first abstract syntax tree, optimizes the first abstract syntax tree to obtain a second abstract syntax tree, and then constructs an abstract interface representing data structures and computing operation instructions of multiple models. Then, an instruction sequence is generated according to the second abstract syntax tree and the abstract interface, and then the instruction sequence corresponding to the business information is run through the target back-end architecture, thereby realizing complex model calling functions without setting configuration files and modifying source code, thereby effectively improving model calling accuracy and model operation stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flowchart of a method for processing a multi-model reasoning process provided in an embodiment of the present application;

[0054] Figure 2 yes Figure 1 Flow chart of step S110 in FIG.

[0055] Figure 3 yes Figure 1 Flow chart of step S120 in FIG.

[0056] Figure 4 yes Figure 1 Flow chart of step S130 in FIG.

[0057] Figure 5 yes Figure 4 Flow chart of step S440 in FIG.

[0058] Figure 6 This is a schematic diagram of offline compilation provided in an embodiment of the present application;

[0059] Figure 7 yes Figure 1 Flow chart of step S150 in FIG.

[0060] Figure 8 It is an application flow chart of the processing method of the multi-model reasoning process provided in the embodiment of the present application;

[0061] Fig. 9 It is a structural diagram of a processing device for a multi-model reasoning process provided in an embodiment of the present application;

[0062] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application, they are only examples of devices and methods consistent with some aspects of the embodiments of the present application.

[0064] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0065] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0067] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0068] Pipeline: In the field of machine learning, Pipeline refers to the complete implementation process including data preprocessing, multi-model serial or parallel, model reasoning, accuracy evaluation, and deployment.

[0069] DSL (Domain-Specific Language): A language that is specific to a particular domain or problem set. It usually has a smaller vocabulary and grammar rules and focuses on solving specific types of problems.

[0070] Dead node: A node that does not participate in calculations, i.e., an invalid node.

[0071] AST (Abstract Syntax Tree): AST is a tree data structure used in computer science to represent the structure of source code. It represents the syntax structure of the program in the form of a tree graph, where each node represents a construct in the program.

[0072] CSE (Common Subexpression Elimination): Identify and remove the same expression that is calculated multiple times, calculate it only once and store the result.

[0073] PASS: PASS in the compiler refers to a certain stage or step in the compilation process, such as an independent stage of code or specified conversion or optimization. Removing it will not affect the operation process of the entire compiler.

[0074] Backend: This application mainly refers to the inference backend. Each chip or accelerated computing library can be regarded as an independent backend with the most basic functions of inference operation.

[0075] Subview: Logical view of data.

[0076] Frame: refers to a frame, which is a static image played continuously, or refers to a unit in the execution process.

[0077] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0078] Image processing, also known as image processing, is a technology that uses computers to process images to achieve the desired results. The main contents of image processing technology include three parts: image compression, enhancement and restoration, and matching description and recognition. Common processing includes image digitization, image coding, image enhancement, image restoration, image segmentation and image analysis. Image processing is the act of using computers to process image information to meet human visual psychology or application needs. It is widely used and is mostly used in surveying and mapping, atmospheric science, astronomy, image beautification, and improving image recognition.

[0079] Smart city refers to the use of various information technologies or innovative concepts to connect and integrate urban systems and services to improve the efficiency of resource utilization, optimize urban management and services, and improve the quality of life of citizens.

[0080] In the related technologies, in the fields of artificial intelligence, image processing or smart cities, a multi-model reasoning architecture is set up to call each other between multiple models, thereby realizing the data processing process and improving the efficiency of data processing. At present, the multi-model reasoning architecture mainly defines the reasoning sequence of multiple models through configuration files, and some reasoning architectures realize the mutual calling process between multiple models by directly defining calling functions in the code. The complete implementation process defined by the configuration file can only realize the simple execution sequence of multiple models, and cannot realize the complex model calling process; the complete implementation process defined in the code needs to configure several parameters, which is easy to cause definition errors, and then cause the failure of multi-model calls, and the definition process needs to be operated in the underlying code, which may reduce the stability of model operation.

[0081] In view of this, an embodiment of the present application provides a processing method and device for a multi-model reasoning process, an electronic device and a storage medium. After constructing a standard representation of the preset implementation process and preset domain characteristic language information, the present application defines the connection relationship between multiple models in the preset implementation process, so that subsequent calls do not need to modify the underlying code and do not need to set the configuration file; then, after parsing the preset domain characteristic language information to obtain a first abstract syntax tree, the first abstract syntax tree is optimized to obtain a second abstract syntax tree, thereby improving the processing efficiency of subsequent reasoning processes; then, after constructing an abstract interface representing the data structure and calculation operation instructions of multiple models, an instruction sequence is generated according to the second abstract syntax tree and the abstract interface, and then the instruction sequence corresponding to the business information is run through the target back-end architecture, which can realize the complex calling function between multiple models corresponding to the business scenario, and improve the model calling accuracy and model operation stability.

[0082] The processing method of the multi-model reasoning process provided in the embodiment of the present application relates to the field of model processing technology. The processing method of the multi-model reasoning process provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in the blockchain network; the software can be an application of the processing method for implementing the multi-model reasoning process, etc., but is not limited to the above form.

[0083] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0084] Figure 1 is an optional flowchart of a method for processing a multi-model reasoning process provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0085] Step S110: constructing preset domain characteristic language information, wherein the preset domain characteristic language information is used to represent a preset implementation process, and the preset implementation process includes a connection relationship between multiple models;

[0086] Step S120, parsing the preset domain characteristic language information to obtain a first abstract syntax tree;

[0087] Step S130: Optimize the first abstract syntax tree to obtain a second abstract syntax tree;

[0088] Step S140: construct an abstract interface, wherein the abstract interface is used to represent the data structures and computing operation instructions of multiple models;

[0089] Step S150: Generate an instruction sequence according to the second abstract syntax tree and the abstract interface;

[0090] Step S160: Run the instruction sequence corresponding to the business information through the target backend architecture.

[0091] In the embodiments of the present application, Figure 2 As shown, the process of constructing the preset domain characteristic language information includes but is not limited to the following steps:

[0092] Step S210: Obtaining business information;

[0093] Step S220: Construct preset domain-specific language information according to the business information.

[0094] It can be understood that business information can be business scenario information. For example, in traffic and security scenarios, business information can be image recognition processing information, and the corresponding preset domain characteristic language information is constructed based on the information, so that the preset domain characteristic language information can effectively define the data interaction process when multiple models in the application scenario process image information. The preset domain characteristic language information in the embodiment of the present application may include but is not limited to input variables, output variables, intermediate variables, calculation instructions, global variables or input variable sequences. Specifically, input variables and output variables are used to characterize the input and output information of the preset implementation process; intermediate variables are used to characterize the information flow between models; calculation instructions are used to characterize the reasoning information or operation information of multiple models in the preset implementation process; global variables are used to control the resource overhead state of the preset implementation process; and input variable sequences are used to characterize the parallel processing data of multiple models in the preset implementation process.

[0095] It is understandable that the preset implementation process of the embodiment of the present application can be a pipeline in the field of machine learning, and pipeline refers to a complete process such as data preprocessing, multi-model serial or parallel, model reasoning, accuracy assessment, and deployment. The preset domain-specific language information can be DSL language information, or it can be other specific language information that can achieve the effect of the embodiment of the present application. Exemplarily, taking DSL language information as the preset domain-specific language information and pipeline as the preset implementation process as an example, the format of the domain-specific language information in the embodiment of the present application is shown in Table 1:

[0096] Table 1

[0097]

[0098] From the content of Table 1, the definition of each information in the preset domain characteristic language information is as follows:

[0099] Input variables: label1Frame image{input=True};

[0100] Intermediate variable: label2Frame tmp_image{};

[0101] Calculation instruction: label3 Buffer output = Det(image){threshold = 0.5, roi = [50, 50, 100, 100]};

[0102] Output variable: label5 Subview output1 = output {output = True};

[0103] Global variables: label6 Frame g_image{};

[0104] Input variable sequence: label7 Frames images{input=True}.

[0105] Based on the above definitions, we can know that the input variables and output variables define the input and output of the entire pipeline, and are identified by input=True. The input variables and output variables can also be accompanied by requirements such as image resolution; intermediate variables represent the information flow transmitted between models, that is, there is no input or output attribute as an identifier, and the intermediate variables will be destroyed after their life cycle; calculation instructions represent specific operations, including model reasoning and some calculations that need to be done on the CPU. The operation can also be accompanied by some operations, such as the number of concurrent operations of the node, threshold, and input ROI area processed; output variables represent variables that need to be passed to the business system as the output of the pipeline; global variables indicate that the variable is valid during the entire pipeline operation process, which is used to prevent the performance overhead caused by the frequent application and release of certain variables, thereby controlling the resource overhead status of the preset implementation process; the input variable sequence is used to represent the batch processing of data when multiple models are processed in parallel. A group of data is input at a time, and the group of data is processed in parallel through multiple models or multiple modules.

[0106] In the embodiments of the present application, Figure 3 As shown, the process of parsing the preset domain characteristic language information to obtain the first abstract syntax tree includes but is not limited to the following steps:

[0107] Step S310, performing lexical analysis on the preset domain characteristic language information to obtain a token sequence;

[0108] Step S320, parse the tag sequence to obtain a preset report statement expression;

[0109] Step S330: Combining the preset report statement expression into an abstract syntax tree as a first abstract syntax tree.

[0110] It is understandable that the parsing process of this embodiment can be parsed by a pre-set compiler. Among them, functional modules such as a lexical analyzer and a syntax analyzer can be set in the compiler to process the preset domain-specific language information through these functional modules, and then obtain the first abstract syntax tree. Specifically, this embodiment obtains a tokens sequence (flag sequence) after analyzing the preset domain-specific language information in the script through a lexical analyzer, and then parses the tokens sequence into statements expressions (preset report statements) through a syntax analyzer, and then all statements expressions are combined into an abstract syntax tree (AST) as the first abstract syntax tree, so as to facilitate subsequent analysis calls.

[0111] In the embodiment of the present application, since there may be code overlap, invalid code, etc. in the first abstract syntax tree, the embodiment optimizes the first abstract syntax tree by setting a corresponding optimization algorithm to obtain a second abstract syntax tree, thereby further improving the efficiency of subsequent code execution. It is understandable that, if Figure 4 As shown, the process of optimizing the first abstract syntax tree to obtain the second abstract syntax tree includes but is not limited to the following steps:

[0112] Step S410, obtaining a number of optimization sub-algorithms to be selected;

[0113] Step S420, selecting at least one optimization sub-algorithm to be selected from a plurality of optimization sub-algorithms to be selected as a target optimization sub-algorithm;

[0114] Step S430, combining all target optimization sub-algorithms into a target optimization algorithm;

[0115] Step S440: Optimize the first abstract syntax tree using a target optimization algorithm to obtain a second abstract syntax tree.

[0116] It is understandable that the optimization sub-algorithms to be selected include but are not limited to constant folding, dead code elimination, loop optimization, common sub-code elimination (CSE) or Tensor reuse. Among them, the constant folding method means that offline calculations can be performed first to reduce the computing consumption at runtime; the dead code elimination method means that the code that never participates in the calculation is directly eliminated; the loop optimization method means that for modules or models that are executed multiple times, data parallelism or model parallelism can be used to improve the reasoning efficiency; the common sub-code elimination (CSE) method means that for calculations with the same input data and computing units, the calculation results are cached, and the cached data is directly called when needed next time to reduce the amount of calculation; the Tensor reuse method means that Tensors (processors) of the same size and data type that appear in different places can be merged into one processor to prevent repeated memory application and release processes.

[0117] Specifically, this embodiment can select the constant folding method as the target optimization sub-algorithm, or the dead code elimination method and the loop optimization method as the target optimization sub-algorithm, or the loop optimization method, the common sub-code elimination method (CSE) and the Tensor reuse method as the target optimization sub-algorithm. It can be understood that each optimization sub-algorithm to be selected can be called multiple times, or can be called by different application scenarios at the same time, and the specific calling process can be set according to the actual situation. After determining all the target optimization sub-algorithms, this embodiment combines all the determined target optimization sub-algorithms into a target optimization algorithm, so as to optimize the first abstract syntax tree through the target optimization algorithm to obtain the second abstract syntax tree.

[0118] It is understandable that if Figure 5 As shown, the process of optimizing the first abstract syntax tree by the target optimization algorithm to obtain the second abstract syntax tree includes but is not limited to the following steps:

[0119] Step S510: Perform type judgment on each module or each code in the first abstract syntax tree to obtain a type judgment result;

[0120] Step S520: According to the type judgment result, match the target optimization sub-algorithm corresponding to each module or each code in the target optimization algorithm;

[0121] Step S530: Optimize the first abstract syntax tree according to the matching result to obtain a second abstract syntax tree.

[0122] Specifically, several codes can form a module. Since the modules or codes required for each business application scenario are different, the types of codes or models in the first abstract syntax tree in different business application scenarios will be different. For example, when the business application scenario is a traffic vehicle recognition scenario, the module used for sound recognition in the first abstract syntax tree belongs to a dead code type; when the business application scenario is a bank security system, the vehicle identification code in the first abstract syntax tree belongs to a dead code type. In the actual application process, an application scenario will call multiple model codes for processing. In order to improve the data processing effect, this embodiment judges the types of multiple model codes or the modules composed of the codes respectively, and then matches the target optimization sub-algorithm corresponding to each module or each code in the target optimization algorithm according to the type judgment result, so that the code or module in the first abstract syntax tree can be optimized by the corresponding target optimization sub-algorithm to improve the accuracy of the optimization result.

[0123] In some embodiments, Figure 1 Steps S110 to S130 may be compiled offline. Figure 6As shown, the modules involved in the offline compilation process include but are not limited to a source program writing module, a compiler front end, an optimization pass module, a compiler back end, and an output module.

[0124] Specifically, the source program writing module is used to write pipeline source programs in DSL language to describe a preset implementation process to form preset domain-specific language information, and the preset domain-specific language information is sent to the compiler front end.

[0125] The compiler front end is used to parse the preset domain-specific language information into a token stream, convert the token stream into statements expressions (all statement expressions form the first abstract syntax tree), and then send all statement expressions to the optimization pass module.

[0126] The optimization pass module is used to optimize all statement expressions. Specifically, constant folding, dead node elimination, data parallelism, and model parallelism can be used for optimization. Each optimization algorithm can be used repeatedly or only once. After completing the optimization of all statement expressions, the optimized statement expressions are sent to the compiler backend.

[0127] The compiler backend is used to convert the optimized statements expression into AST (second abstract syntax tree), thus forming a set of highly optimized abstract syntax trees, and translate the optimized abstract syntax trees into instruction sequences, and then send all instruction sequences to the output module.

[0128] The output module is used to save all instruction sequences into instruction sequence files for subsequent use.

[0129] In an embodiment of the present application, since the second abstract syntax does not contain all scenario operation functions or calculation functions, in order to accurately complete the corresponding functions when the instruction sequence is called in the business scenario, this embodiment also constructs an abstract interface of data structures and calculation operation instructions that represent multiple models, and these abstract interfaces are formed by a series of virtual instructions. Among them, the virtual instructions define the data structure and the calculation operation function. Specifically, the data structure includes but is not limited to Frame, Buffer or Subview; the calculation operation function includes but is not limited to yoloDet to implement the yolo detection function, faceCheck to implement the face verification function, etc. The abstract interface constructed in this embodiment serves as the base for instruction execution, and is used to provide a set of all operation instructions required to execute instructions. Specifically, the instruction consists of an opcode and an operand, and some instructions are shown in Table 2:

[0130] Table 2

[0131]

[0132]

[0133] After completing the construction of the abstract interface, the embodiment of the present application, such as Figure 7 As shown, the process of generating an instruction sequence according to the second abstract syntax tree and the abstract interface includes but is not limited to the following steps:

[0134] Step S710: compile the second abstract syntax tree and call the abstract interface;

[0135] Step S720: When the abstract interface exists and the data structure corresponding to the abstract interface matches the data structure of the second abstract syntax tree, an instruction sequence is generated according to the compilation result.

[0136] It is understandable that in this embodiment, the abstract interface may be called while compiling the second abstract syntax tree into an instruction sequence, and it is determined whether the abstract interface exists or whether the data structure of the abstract interface matches the data structure of the second abstract syntax tree. If the abstract interface does not exist or the data structure does not match, a compilation error reminder is given, thereby realizing offline verification of the preset domain characteristic language information. When the abstract interface exists and the data structure matches, an instruction sequence corresponding to the second abstract syntax tree is generated.

[0137] In the embodiment of the present application, after obtaining the instruction sequence, it is also possible to construct the first virtual operating machine, and when it is necessary to run the execution sequence in the corresponding business scenario, dynamically load the target backend architecture in the first virtual operating machine according to the business information to obtain the second virtual operating machine, and then run the instruction sequence corresponding to the business information through the second virtual operating machine, so as to effectively realize the functions required by the current business scenario. It is understandable that the backend structure of the embodiment of the present application includes but is not limited to onnxruntime, tensorrt, ncnn, cambricon or atlas.

[0138] For example, Figure 8As shown, taking the intelligent manufacturing scenario of a vehicle production workshop as an example, when intelligent monitoring of the vehicle production workshop is required, the corresponding target backend architecture is selected from the backends and dynamically loaded into the instruction executor, and the target instruction sequence is selected from the pipline instruction sequence obtained by offline compilation and dynamically loaded into the instruction executor (equivalent to the second virtual running machine). When the video stream or image stream of the vehicle production workshop is obtained, the video stream or image stream is parsed into skeleton data corresponding to the data structure of the target backend architecture, and the skeleton data is sent to the instruction executor. The target instruction sequence is run through the instruction executor to analyze the skeleton data to obtain the real-time working dynamics of the vehicle production workshop; when it is determined that there is a danger in the vehicle production workshop, the corresponding alarm information is output, and the aforementioned processing process is performed after waiting for the input of the next frame of data.

[0139] From the above content, it can be seen that the method of the embodiment of the present application can effectively improve the accuracy of model calling and the stability of model operation, thereby improving the efficiency and accuracy of application scenario data processing.

[0140] Reference Fig. 9 , the embodiment of the present application improves a processing device for a multi-model reasoning process, the device comprising:

[0141] The first module 910 is used to construct preset domain characteristic language information, wherein the preset domain characteristic language information is used to represent a preset implementation process, and the preset implementation process includes a connection relationship between multiple models;

[0142] The second module 920 is used to parse the preset domain characteristic language information to obtain a first abstract syntax tree;

[0143] The third module 930 is used to optimize the first abstract syntax tree to obtain a second abstract syntax tree;

[0144] A fourth module 940 is used to construct an abstract interface, wherein the abstract interface is used to represent data structures and computing operation instructions of multiple models;

[0145] A fifth module 950 is used to generate an instruction sequence according to the second abstract syntax tree and the abstract interface;

[0146] The sixth module 960 is used to run the instruction sequence corresponding to the business information through the target backend architecture.

[0147] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0148] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the processing method of the above multi-model reasoning process when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0149] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0150] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0151] The processor 1010 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0152] The memory 1020 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1020 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1020, and the processor 1010 calls and executes the processing method of the multi-model reasoning process of the embodiment of this application;

[0153] Input / output interface 1030, used to implement information input and output;

[0154] The communication interface 1040 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0155] Bus 1050 , which transmits information between various components of the device (e.g., processor 1010 , memory 1020 , input / output interface 1030 , and communication interface 1040 );

[0156] The processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 are connected to each other in communication within the device via the bus 1050 .

[0157] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the processing method of the above-mentioned multi-model reasoning process.

[0158] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0159] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0160] The embodiments of the present application provide a method and device for processing a multi-model reasoning process, an electronic device, and a storage medium. The method constructs DSL language information to characterize a pipeline implementation process, and defines the connection relationship between multiple models in the pipeline implementation process. This eliminates the need to set a configuration file or define a calling relationship in the source code, thereby realizing complex model calling functions and effectively improving the model calling accuracy and model operation stability. The abstract syntax tree corresponding to the DSL language information is optimized, thereby effectively reducing the data handling cost. At the same time, the pipeline implementation process can limit the model to perform parallel data processing, thereby effectively improving the execution efficiency of the pipeline and further improving the data processing efficiency.

[0161] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0162] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0163] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0165] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0166] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0167] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0168] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0171] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A method for processing a multi-model reasoning process, characterized in that: The method comprises the following steps: Constructing preset domain characteristic language information, wherein the preset domain characteristic language information is used to represent a preset implementation process, wherein the preset implementation process includes a connection relationship between multiple models; Parsing the preset domain-specific language information to obtain a first abstract syntax tree; Optimizing the first abstract syntax tree to obtain a second abstract syntax tree; Constructing an abstract interface, wherein the abstract interface is used to represent data structures and computing operation instructions of multiple models; Generate an instruction sequence according to the second abstract syntax tree and the abstract interface; The instruction sequence corresponding to the business information is executed through the target backend architecture.

2. The method according to claim 1, characterized in that: The step of constructing the preset domain characteristic language information includes: Obtain business information; Constructing preset domain-specific language information according to the business information, wherein the preset domain-specific language information includes input variables, output variables, intermediate variables, calculation instructions, global variables or input variable sequences; The input variables and the output variables are used to represent the input and output information of the preset implementation process; The intermediate variables are used to characterize the information flow between models; The calculation instructions are used to represent the reasoning information or operation information of multiple models in the preset implementation process; The global variable is used to control the resource consumption state of the preset implementation process; The input variable sequence is used to represent the parallel processing data of multiple models in the preset implementation process.

3. The method according to claim 1, characterized in that The step of parsing the preset domain-specific language information to obtain a first abstract syntax tree includes: Performing lexical analysis on the preset domain characteristic language information to obtain a token sequence; Performing grammatical analysis on the flag sequence to obtain a preset report statement expression; The preset report statement expression is combined into an abstract syntax tree as the first abstract syntax tree.

4. The method according to claim 1, characterized in that: The step of optimizing the first abstract syntax tree to obtain a second abstract syntax tree includes: Obtain several optimization sub-algorithms to be selected; Selecting at least one optimization sub-algorithm to be selected from the plurality of optimization sub-algorithms to be selected as a target optimization sub-algorithm; Combining all the target optimization sub-algorithms into a target optimization algorithm; The first abstract syntax tree is optimized by using the target optimization algorithm to obtain a second abstract syntax tree.

5. The method according to claim 4, characterized in that The step of optimizing the first abstract syntax tree by using the target optimization algorithm to obtain a second abstract syntax tree includes: Performing type judgment on each module or each code in the first abstract syntax tree to obtain a type judgment result; According to the type judgment result, matching the target optimization sub-algorithm corresponding to each module or each code in the target optimization algorithm; The first abstract syntax tree is optimized according to the matching result to obtain a second abstract syntax tree.

6. The method according to claim 1, characterized in that The generating an instruction sequence according to the second abstract syntax tree and the abstract interface comprises: Compiling the second abstract syntax tree and calling the abstract interface; When the abstract interface exists and the data structure corresponding to the abstract interface matches the data structure of the second abstract syntax tree, the instruction sequence is generated according to the compilation result.

7. The method according to claim 2, characterized in that: The step of running the instruction sequence corresponding to the business information through the target backend architecture includes: Building the first virtual operating machine; Dynamically loading the target backend architecture in the first virtual operating machine according to the business information to obtain a second virtual operating machine; The instruction sequence corresponding to the business information is executed by the second virtual execution machine.

8. A processing device for a multi-model reasoning process, characterized in that: The device comprises: The first module is used to construct preset domain characteristic language information, wherein the preset domain characteristic language information is used to represent a preset implementation process, and the preset implementation process includes a connection relationship between multiple models; The second module is used to parse the preset domain characteristic language information to obtain a first abstract syntax tree; A third module is used to optimize the first abstract syntax tree to obtain a second abstract syntax tree; A fourth module is used to construct an abstract interface, wherein the abstract interface is used to represent data structures and computing operation instructions of multiple models; A fifth module, configured to generate an instruction sequence according to the second abstract syntax tree and the abstract interface; The sixth module is used to run the instruction sequence corresponding to the business information through the target backend architecture.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images