AI reasoning method, system, electronic device, readable storage medium and product

Abstract

The present application discloses an AI reasoning method, system, electronic device, readable storage medium and product, wherein the AI ​​reasoning method comprises: receiving an AI combination reasoning service request sent by a client, wherein the AI ​​combination reasoning service request comprises at least one AI combination reasoning requirement; generating a first model combination reasoning request corresponding to each of the AI ​​combination reasoning requirements; for each of the first model combination reasoning requests, respectively executing: sending the first model combination reasoning request to the corresponding distribution container sequence to request to call at least one AI model corresponding to the distribution container sequence for model reasoning, and obtaining the model combination reasoning result corresponding to the first model combination reasoning request; feeding back each of the model combination reasoning results to the client. The present application solves the technical problem of low efficiency of client requesting AI reasoning service in the prior art.

Description

Technical Field

[0001] The present application relates to the field of model deployment technology, and in particular to an AI reasoning method, system, electronic device, readable storage medium and product. Background Art

[0002] With the continuous development of artificial intelligence technology, the application of AI models is becoming more and more extensive. After building the AI ​​model, it is usually necessary to deploy the AI ​​model through a specific method so that the client can request the AI ​​reasoning service. At present, the server usually deploys the AI ​​reasoning service through tfserving. However, tfserving can only expose one http service to the outside during deployment. If the client needs multiple AI models for combined reasoning to get the final result, it is necessary to write a lot of code at the business layer and make multiple http service calls. The efficiency of the client requesting the AI ​​reasoning service is low. Summary of the invention

[0003] The main purpose of this application is to provide an AI reasoning method, system, electronic device, readable storage medium and product, aiming to solve the technical problem of low efficiency of client requesting AI reasoning services in the prior art.

[0004] To achieve the above objectives, the present application provides an AI reasoning method, which includes:

[0005] Receiving an AI combined reasoning service request sent by a client, wherein the AI ​​combined reasoning service request includes at least one AI combined reasoning requirement; generating a first model combined reasoning request corresponding to each of the AI ​​combined reasoning requirements;

[0006] For each of the first model combination reasoning requests, respectively execute: sending the first model combination reasoning request to the corresponding distribution container sequence to request calling at least one AI model corresponding to the distribution container sequence to perform model reasoning, and obtaining the model combination reasoning result corresponding to the first model combination reasoning request;

[0007] The inference results of each model combination are fed back to the client.

[0008] Optionally, the distribution container sequence includes at least one distribution container arranged in sequence, and the step of sending the first model combination inference request to the corresponding distribution container sequence to request calling at least one AI model corresponding to the distribution container sequence for model inference, and obtaining the model combination inference result corresponding to the first model combination inference request includes:

[0009] The first model combination reasoning request is transmitted to the distribution container ranked first in the distribution container sequence by sequentially calling the module through the AI ​​service, wherein the first model combination reasoning request includes a tracking identifier and a sample to be inferred;

[0010] Calling the corresponding AI model through the first-ranked distribution container to perform model inference on the sample to be inferred, and obtaining a model inference result;

[0011] According to the tracking identifier, the next distribution container in the distribution container sequence is matched, and a second model combination inference request carrying the model inference result and the tracking identifier is passed to the next distribution container. The corresponding AI model is called through the next distribution container to perform model inference on the model inference result and generate a new model inference result, until the next distribution container is not matched, and the latest model inference result is used as the model combination inference result.

[0012] Optionally, the step of matching the next distribution container in the distribution container sequence according to the tracking identifier, transmitting the second model combination inference request carrying the model inference result and the tracking identifier to the next distribution container, calling the corresponding AI model through the next distribution container to perform model inference on the model inference result and generate a new model inference result, until the next distribution container is not matched, and using the latest model inference result as the model combination inference result includes:

[0013] Matching a next distribution container in the distribution container sequence according to the tracking identifier;

[0014] If the matching fails, the latest model inference result is used as the model combination inference result;

[0015] If the match is successful, a second model combination inference request carrying the model inference result and the tracking identifier is generated; and the second model combination inference request is passed to the next distribution container, and the corresponding AI model is called through the next distribution container to perform model inference on the model inference result to obtain a new model inference result, and return to the execution step: according to the tracking identifier, match the next distribution container in the distribution container sequence.

[0016] Optionally, the steps of calling the corresponding AI model for model inference by the distribution container to obtain the model inference result include:

[0017] Forwarding the sample to be inferred in the first model combination inference request or the model inference result in the second model combination inference request from the distribution container to the AI ​​model corresponding to the distribution container;

[0018] Using the AI ​​model corresponding to the distribution container, model reasoning is performed on the sample to be inferred or the model reasoning result, the model reasoning result of the current AI model is obtained and fed back to the distribution container.

[0019] Optionally, the step of feeding back the inference results of each model combination to the client includes:

[0020] Integrate the model combination reasoning results fed back by each distribution container sequence into a service request response result;

[0021] The service request response result is fed back to the client.

[0022] Optionally, before the step of generating the first model combination reasoning request corresponding to each of the AI ​​combination reasoning requirements, the AI ​​reasoning method further includes:

[0023] Generate a corresponding module identifier according to the AI ​​combination reasoning requirement information in the AI ​​combination reasoning service request;

[0024] According to the module identifier, query whether there is an AI service calling module in sequence corresponding to the AI ​​combined reasoning service request;

[0025] If so, the AI ​​combined reasoning service request is passed to the AI ​​service sequentially calling module;

[0026] If it does not exist, deploy the AI ​​combined reasoning service corresponding to the AI ​​combined reasoning service request, and pass the AI ​​combined reasoning service request to the AI ​​service in the AI ​​combined reasoning service to call the module in sequence.

[0027] The present application also provides an AI reasoning system, which includes a service layer, an application layer, and a single AI service layer, wherein:

[0028] The service layer deploys at least one AI service sequential calling module, wherein the AI ​​service sequential calling module is used to generate a first model combination reasoning request corresponding to each AI combination reasoning requirement in the AI ​​combination reasoning service request sent by the client, and feed back the model combination reasoning results generated in response to each of the first model combination reasoning requests to the client;

[0029] The application layer deploys at least one distribution container sequence, wherein the distribution container sequence is used to receive the first model combination reasoning request transmitted by the AI ​​service calling the module in sequence, and perform model reasoning by calling the corresponding AI models, and respond to generate model combination reasoning results corresponding to each of the first model combination reasoning requests;

[0030] The single AI service layer deploys at least one AI model, wherein the AI ​​model is used for model reasoning.

[0031] Optionally, the application layer includes a registration center, a container allocation module and a construction center, wherein:

[0032] The registration center is used to register the AI ​​combined reasoning service and the AI ​​service sequentially calling modules;

[0033] The container allocation module is used to allocate distribution containers to each of the AI ​​models in sequence to form a distribution container sequence corresponding to each of the AI ​​models;

[0034] The construction center is used to package the AI ​​model files in the single AI service layer into a mirror image with external service functions.

[0035] Optionally, the AI ​​service calls the modules in sequence including:

[0036] A receiving and generating unit, configured to receive an AI combined reasoning service request sent by a client, wherein the AI ​​combined reasoning service request includes at least one AI combined reasoning requirement; and generate a first model combined reasoning request corresponding to each of the AI ​​combined reasoning requirements;

[0037] A request calling unit, configured to execute, for each of the first model combination reasoning requests, respectively: sending the first model combination reasoning request to the corresponding distribution container sequence to request calling at least one AI model corresponding to the distribution container sequence to perform model reasoning, and obtaining a model combination reasoning result corresponding to the first model combination reasoning request;

[0038] A feedback unit is used to feed back the inference results of each model combination to the client.

[0039] Optionally, the distribution container sequence includes at least one distribution container arranged in sequence, and the request calling unit is further used for:

[0040] The first model combination reasoning request is transmitted to the distribution container ranked first in the distribution container sequence by sequentially calling the module through the AI ​​service, wherein the first model combination reasoning request includes a tracking identifier and a sample to be inferred;

[0041] Calling the corresponding AI model through the first-ranked distribution container to perform model inference on the sample to be inferred, and obtaining a model inference result;

[0042] According to the tracking identifier, the next distribution container in the distribution container sequence is matched, and a second model combination inference request carrying the model inference result and the tracking identifier is passed to the next distribution container. The corresponding AI model is called through the next distribution container to perform model inference on the model inference result and generate a new model inference result, until the next distribution container is not matched, and the latest model inference result is used as the model combination inference result.

[0043] Optionally, the request calling unit is further used for:

[0044] Matching a next distribution container in the distribution container sequence according to the tracking identifier;

[0045] If the matching fails, the latest model inference result is used as the model combination inference result;

[0046] If the match is successful, a second model combination inference request carrying the model inference result and the tracking identifier is generated; and the second model combination inference request is passed to the next distribution container, and the corresponding AI model is called through the next distribution container to perform model inference on the model inference result to obtain a new model inference result, and return to the execution step: according to the tracking identifier, match the next distribution container in the distribution container sequence.

[0047] Optionally, the request calling unit is further used for:

[0048] Forwarding the sample to be inferred in the first model combination inference request or the model inference result in the second model combination inference request from the distribution container to the AI ​​model corresponding to the distribution container;

[0049] Using the AI ​​model corresponding to the distribution container, model reasoning is performed on the sample to be inferred or the model reasoning result, the model reasoning result of the current AI model is obtained and fed back to the distribution container.

[0050] Optionally, the feedback unit is further used for:

[0051] Integrate the model combination reasoning results fed back by each distribution container sequence into a service request response result;

[0052] The service request response result is fed back to the client.

[0053] Optionally, the AI ​​reasoning system further includes:

[0054] A module identification generating unit, configured to generate a corresponding module identification according to the AI ​​combination reasoning requirement information in the AI ​​combination reasoning service request;

[0055] A query unit is used to query whether there is an AI service sequentially calling module corresponding to the AI ​​combined reasoning service request based on the module identifier; if so, pass the AI ​​combined reasoning service request to the AI ​​service sequentially calling module; if not, deploy the AI ​​combined reasoning service corresponding to the AI ​​combined reasoning service request, and pass the AI ​​combined reasoning service request to the AI ​​service sequentially calling module in the AI ​​combined reasoning service.

[0056] The present application also provides an electronic device, which is a physical device, and includes: a memory, a processor, and a program of the AI ​​reasoning method stored in the memory and executable on the processor. When the program of the AI ​​reasoning method is executed by the processor, the steps of the AI ​​reasoning method as described above can be implemented.

[0057] The present application also provides a readable storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a program for implementing the AI ​​reasoning method. When the program of the AI ​​reasoning method is executed by a processor, the steps of the AI ​​reasoning method as described above are implemented.

[0058] The present application also provides a product, which is a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the AI ​​reasoning method as described above are implemented.

[0059] The present application provides an AI reasoning method, system, electronic device, readable storage medium and product, that is, the present application first receives an AI combination reasoning service request sent by a client, wherein the AI ​​combination reasoning service request includes at least one AI combination reasoning requirement; generates a first model combination reasoning request corresponding to each of the AI ​​combination reasoning requirements, and then executes for each of the first model combination reasoning requests respectively: sends the first model combination reasoning request to the corresponding distribution container sequence to request to call at least one AI model corresponding to the distribution container sequence for model reasoning, obtains the model combination reasoning result corresponding to the first model combination reasoning request, and then realizes calling each AI model for model reasoning according to the distribution container sequence. The purpose of combined reasoning is to feed back the combined reasoning results of each model to the client, so as to achieve the goal of responding to the client's request only once, that is, completing the AI ​​model combined reasoning service for the client. That is, when the client needs multiple AI models for combined reasoning, it only needs to call the model reasoning service once to complete the request for the AI ​​combined reasoning service, thereby overcoming the problem that tfserving can only expose one http service to the outside world during deployment. If the client needs multiple AI models for combined reasoning to obtain the final result, it is necessary to write a lot of code at the business layer and make multiple http service calls, which reduces the efficiency of the client requesting AI reasoning services. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0062] Figure 1 This is a flowchart of the first embodiment of the AI ​​reasoning method of this application;

[0063] Figure 2 This is a flowchart of the second embodiment of the AI ​​reasoning method of the present application;

[0064] Figure 3 A request lifecycle diagram for AI reasoning in the AI ​​reasoning method of this application;

[0065] Figure 4 This is a system framework diagram of the AI ​​reasoning system described in the AI ​​reasoning method of this application;

[0066] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the AI ​​reasoning method in the embodiment of the present application.

[0067] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0068] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0069] Embodiment 1

[0070] The present application embodiment provides an AI reasoning method applied to a data sender. In the first embodiment of the AI ​​reasoning method of the present application, refer to Figure 1 , the AI ​​reasoning method includes:

[0071] Step S10, receiving an AI combination reasoning service request sent by a client, wherein the AI ​​combination reasoning service request includes at least one AI combination reasoning requirement; generating a first model combination reasoning request corresponding to each of the AI ​​combination reasoning requirements;

[0072] Step S20, for each of the first model combination reasoning requests, respectively execute: sending the first model combination reasoning request to the corresponding distribution container sequence to request calling at least one AI model corresponding to the distribution container sequence to perform model reasoning, and obtaining the model combination reasoning result corresponding to the first model combination reasoning request;

[0073] Step S30, feeding back the inference results of each model combination to the client.

[0074] After receiving the AI ​​combination reasoning service request sent by the client, the embodiment of the present application can generate at least one AI combination reasoning request based on at least one AI combination reasoning requirement in the AI ​​combination reasoning service request, and then can call the corresponding at least one AI model to perform model combination reasoning according to the distribution container sequence corresponding to each AI combination reasoning request, and obtain the model combination reasoning result corresponding to each first model combination reasoning request, and then feed back each model combination reasoning result to the client, so as to achieve only one response to the client's request, that is, to complete the AI ​​model combination reasoning service for the client, that is, when the client needs multiple AI models for combined reasoning, it only needs to call the model reasoning service once to complete the request for the AI ​​combination reasoning service, instead of the client sending multiple requests to the service provider to directly call the corresponding AI models for model reasoning, thereby achieving model combination reasoning, so as to improve the efficiency of the client requesting AI reasoning service.

[0075] In this embodiment, it should be noted that the AI ​​reasoning method is applied to an AI reasoning system, wherein an AI service sequential calling module is deployed in the AI ​​reasoning system, and the AI ​​service sequential calling module is an AI customized linear communication model, which is used to provide customized AI combined reasoning services for the client. The client sends an AI combined reasoning service request to the AI ​​service sequential calling module once, and the AI ​​service sequential calling module can feedback the final model reasoning result. For example, assuming that when the client performs image recognition on a face image, it currently needs to call the feature extraction model and the classification model in sequence for face recognition, and then needs to send two http service requests to the server to call the feature extraction model and the classification model respectively for model reasoning, so as to obtain the final face recognition result. In the embodiment of the present application, the client only needs to send an http service to the AI ​​service sequential calling module once, and the AI ​​service sequential calling module can automatically call the feature extraction model and the classification model in sequence to obtain the final face recognition result, and feedback to the client.

[0076] In the specific implementation, step S10, receiving an AI combination reasoning service request sent by the client, the AI ​​combination reasoning service request includes at least one AI combination reasoning requirement; generating a first model combination reasoning request corresponding to each of the AI ​​combination reasoning requirements, specifically including: receiving the AI ​​combination reasoning service request sent by the client through the AI ​​service in sequence calling the module, and parsing the AI ​​combination reasoning service request to obtain at least one AI combination reasoning requirement, and generating a corresponding first model combination reasoning request for each of the AI ​​combination reasoning requirements, wherein the AI ​​combination reasoning requirement is a requirement for calling at least one AI model to perform model reasoning and generate a final model prediction result. For example, assuming that the client simultaneously requires face recognition and iris recognition of a face image, face recognition is an AI combination reasoning requirement. When performing face recognition, it is necessary to call a face feature extraction model and a face feature classification model for model reasoning to finally generate a face recognition result, while iris recognition is another AI combination reasoning requirement. When performing iris recognition, it is necessary to call an iris feature extraction model and an iris feature classification model for model reasoning to finally generate an iris recognition result.

[0077] The step of generating a corresponding first model combination reasoning request for each AI combination reasoning requirement includes:

[0078] Obtain the sample to be inferred in the AI ​​combined reasoning service request, and generate a tracking identifier corresponding to the AI ​​combined reasoning requirement based on the AI ​​model that needs to be called by the AI ​​combined reasoning requirement and the order in which the AI ​​models are called, and then encapsulate the tracking identifier corresponding to the AI ​​combined reasoning requirement and the corresponding sample to be inferred as an http service request to obtain the first model combined reasoning request corresponding to the AI ​​combined reasoning requirement, wherein the sample to be inferred can be a pixel matrix corresponding to an image and a user feature matrix corresponding to a user portrait, etc.

[0079] Wherein, before the step of generating the first model combination reasoning request corresponding to each of the AI ​​combination reasoning requirements, the AI ​​reasoning method further includes:

[0080] Step A10, generating a corresponding module identifier according to the AI ​​combination reasoning requirement information in the AI ​​combination reasoning service request;

[0081] In this embodiment, it should be noted that the AI ​​combined reasoning service request is an http service request for requesting AI combined reasoning service, and the AI ​​combined reasoning service request includes at least a sample to be inferred and AI combined reasoning requirement information, wherein the AI ​​combined reasoning requirement information can be the type of AI combined reasoning required by the client, or the identifier of the AI ​​combined reasoning required by the client, etc. For example, assuming that the type of AI combined reasoning required by the client can be a face recognition type, indicating that the client needs to perform face recognition on the sample to be inferred, and assuming that the identifier of the AI ​​combined reasoning required by the client is code A, then code A indicates that the client needs to perform iris recognition on the inference target, etc.

[0082] Generate a corresponding module identifier based on the AI ​​combined reasoning requirement information in the AI ​​combined reasoning service request. Specifically, determine the model call information based on the AI ​​combined reasoning requirement information in the AI ​​combined reasoning service request, and then generate a module identifier corresponding to the model call information.

[0083] The step of determining the model calling information according to the AI ​​combined reasoning requirement information in the AI ​​combined reasoning service request includes:

[0084] According to the AI ​​combined reasoning requirement information, query the AI ​​models required to be called by the sample to be inferred and the model calling order of calling each AI model, and then use each AI model and the model calling order as the model calling information.

[0085] The step of generating a module identifier corresponding to the model calling information includes:

[0086] In the model calling order, the model identification sequence corresponding to each AI model is spliced ​​into a vector to obtain the module identification. For example, assuming that each AI model includes model A, model B, model C and model D, for an AI combined reasoning requirement information in the AI ​​combined reasoning service request, the model calling order is A first, then B, and finally C. For another AI combined reasoning requirement information in the AI ​​combined reasoning service request, the model calling order is A first, then D. The model identification corresponding to model A is a, the model identification corresponding to model B is b, the model identification corresponding to model C is c, and the model identification corresponding to model D is d. The module identification can be set to (a, b, c, a, d), or it can be set to a matrix composed of a vector (a, b, c) and a vector (a, d).

[0087] Alternatively, based on a preset mapping relationship between the model calling information and the module identifier, the module identifier corresponding to the model calling information is determined.

[0088] In addition, it should be noted that one of the AI ​​combined reasoning requirement information corresponds to a tracking identifier, and the module identifier can be composed of the tracking identifiers corresponding to the respective AI combined reasoning requirement information, for example, directly concatenating the respective tracking identifiers into a module identifier, or taking a vector composed of the first bit values ​​of the respective tracking identifiers as the module identifier, etc.

[0089] Step A20, querying whether there is an AI service sequentially calling module corresponding to the AI ​​combined reasoning service request according to the module identifier;

[0090] In this embodiment, based on the module identifier, it is queried whether there is an AI service sequentially calling module corresponding to the AI ​​combined reasoning service request. Specifically, using the module identifier as an index, it is queried whether there is an AI service sequentially calling module corresponding to the AI ​​combined reasoning service request.

[0091] Step A30: if it exists, passing the AI ​​combined reasoning service request to the AI ​​service sequential calling module;

[0092] Step A40: If it does not exist, deploy the AI ​​combined reasoning service corresponding to the AI ​​combined reasoning service request, and pass the AI ​​combined reasoning service request to the AI ​​service in the AI ​​combined reasoning service to call the module in sequence.

[0093] In this embodiment, if there is an AI service sequentially calling module corresponding to the AI ​​combination reasoning service request, the AI ​​combination reasoning service request and the tracking identifier are passed to the AI ​​service sequentially calling module, so that the step of generating the first model combination reasoning request corresponding to each AI combination reasoning requirement is performed through the AI ​​service sequentially calling module; if there is no AI service sequentially calling module corresponding to the AI ​​combination reasoning service request, it is proved that the AI ​​combination reasoning service request is a new requirement, and then the AI ​​combination reasoning service corresponding to the AI ​​combination reasoning service request is deployed, and the AI ​​combination reasoning service request is passed to the newly deployed AI service sequentially calling module, so that the step of generating the first model combination reasoning request corresponding to each AI combination reasoning requirement is performed through the newly deployed AI service sequentially calling module. Then, the purpose of identifying whether the AI ​​combination reasoning service request is a new requirement is achieved, so that when the AI ​​combination reasoning service request is not a new requirement, a customized AI combination reasoning service can be directly provided to the client, and when the AI ​​combination reasoning service request is a new requirement, the corresponding AI combination reasoning service can be identified and deployed in time to perform model combination reasoning.

[0094] In this embodiment, for step S20, for each of the first model combination reasoning requests, the following is executed respectively: the first model combination reasoning request is sent to the corresponding distribution container sequence to request to call at least one AI model corresponding to the distribution container sequence for model reasoning, and obtain the model combination reasoning result corresponding to the first model combination reasoning request; it should be noted that one AI combination reasoning requirement corresponds to one first model combination reasoning request, one distribution container sequence corresponds to one first model combination reasoning request, the arrangement order of the distribution container sequence is consistent with the model calling order corresponding to the AI ​​combination reasoning requirement, the distribution container sequence includes at least one distribution container arranged in sequence, the distribution container is used to call the corresponding AI model with specific functions for model reasoning, and can also be used to manage the output results of the AI ​​model, wherein the AI ​​model with specific functions can be a face feature extraction model, a face feature classification model, an iris feature extraction model, and an iris feature classification model, etc.

[0095] For each of the first model combination reasoning requests, respectively, the following steps are performed: sending the first model combination reasoning request to the corresponding distribution container sequence to request calling at least one AI model corresponding to the distribution container sequence for model reasoning, and obtaining the model combination reasoning result corresponding to the first model combination reasoning request. Specifically, the number of the first model combination reasoning requests is at least 1, and for each of the first model combination reasoning requests, the following steps are performed:

[0096] The AI ​​service sequentially calls the module to send the first model combination reasoning request to the first model combination reasoning request and the corresponding distribution container sequence, and then the distribution container sequence sequentially calls the AI ​​models corresponding to each distribution container in the distribution container sequence according to the tracking identifier in the first model combination reasoning request to perform model reasoning on the samples to be inferred in the first model combination reasoning request to obtain the model combination reasoning result.

[0097] Furthermore, step S30, feeding back the inference results of each model combination to the client, specifically includes:

[0098] Step S31, integrating the model combination reasoning results fed back by each distribution container sequence into a service request response result by sequentially calling the module through the AI ​​service;

[0099] In this embodiment, it should be noted that the first model combination inference request can be an http service request, and the request response result is the response result of the http service request. Each model combination inference result is generated by the AI ​​model called by the last-ranked distribution container in the corresponding distribution container sequence.

[0100] The model combination reasoning results fed back by each of the distribution container sequences are integrated into a service request response result. Specifically, the following steps are performed for each of the model combination reasoning results: according to the arrangement order of each distribution container in the distribution container sequence, the model combination reasoning result is fed back layer by layer from the last-ranked distribution container to the first-ranked distribution container, and then the first-ranked distribution container feeds back the model combination reasoning result to the AAI service sequential calling module, and then the model combination reasoning result sent by the first-ranked distribution container in each distribution container sequence is received through the AI ​​service sequential calling module, and each model combination reasoning result is integrated into a service request response result through the AI ​​service sequential calling module.

[0101] The step of integrating the model combination reasoning results fed back by each distribution container sequence into a service request response result further includes:

[0102] According to the preset aggregation rules, the AI ​​service sequentially calls the modules to aggregate the inference results of each model combination to obtain a service request response result, wherein the preset aggregation rules may be averaging, weighted summing, and weighted averaging, etc. For example, assuming that the inference results of each model combination are A and B, and the preset aggregation rule is averaging, wherein A is the probability of determining that the user is himself through face recognition, with a specific value of 0.8, and B is the probability of determining that the user is himself through iris recognition, with a specific value of 0.7, then the service request response result is the probability of determining that the user is himself, with a specific value of 0.75.

[0103] The step of integrating the model combination reasoning results fed back by each distribution container sequence into a service request response result further includes:

[0104] The AI ​​service calls the modules in sequence to splice the inference results of each model combination to obtain a service request response result, and feeds back the service request response result to the client. For example, assuming that the inference results of each model combination are face recognition classification label X, iris recognition classification label Y, and gait feature recognition classification label Z, the service request response result is a vector (X, Y, Z).

[0105] Step S32: The AI ​​service calls the modules in sequence to feed back the service request response result to the client.

[0106] In this embodiment, the service request response result is fed back to the client as the request response result of the AI ​​combined reasoning service request.

[0107] The embodiments of the present application provide an AI reasoning method, system, electronic device, readable storage medium and product. That is, the embodiments of the present application first receive an AI combination reasoning service request sent by a client, wherein the AI ​​combination reasoning service request includes at least one AI combination reasoning requirement; generate a first model combination reasoning request corresponding to each of the AI ​​combination reasoning requirements, and then execute for each of the first model combination reasoning requests respectively: send the first model combination reasoning request to the corresponding distribution container sequence to request to call at least one AI model corresponding to the distribution container sequence for model reasoning, and obtain the model combination reasoning result corresponding to the first model combination reasoning request, thereby realizing calling each AI model according to the distribution container sequence respectively. The purpose of combined reasoning of the model is to further feed back the combined reasoning results of each model to the client, thereby realizing the response to the client's request only once, that is, completing the AI ​​model combined reasoning service for the client. That is, when the client needs multiple AI models for combined reasoning, it only needs to call the model reasoning service once to complete the request for the AI ​​combined reasoning service, thereby overcoming the technical defect that tfserving can only expose one http service to the outside during deployment. If the client needs multiple AI models for combined reasoning to obtain the final result, it is necessary to write a lot of code in the business layer and make multiple http service calls, and the efficiency of the client requesting AI reasoning service is low.

[0108] Embodiment 2

[0109] Further, refer to Figure 2 Based on the first embodiment of the present application, in another embodiment of the present application, the distribution container sequence includes at least one distribution container arranged in sequence. In this embodiment, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be described in detail later.

[0110] Step S20, for each of the first model combination reasoning requests, is performed separately: the step of sending the first model combination reasoning request to the corresponding distribution container sequence to request to call at least one AI model corresponding to the distribution container sequence for model reasoning, and obtaining the model combination reasoning result corresponding to the first model combination reasoning request includes:

[0111] Step B10, calling the module in sequence through the AI ​​service to pass the first model combination reasoning request to the distribution container ranked first in the distribution container sequence, wherein the first model combination reasoning request includes a tracking identifier and a sample to be inferred;

[0112] Step B20, calling the corresponding AI model through the first-ranked distribution container to perform model inference on the sample to be inferred, to obtain a model inference result;

[0113] Step B30, according to the tracking identifier, match the next distribution container in the distribution container sequence, pass the second model combination inference request carrying the model inference result and the tracking identifier to the next distribution container, call the corresponding AI model through the next distribution container to perform model inference on the model inference result and generate a new model inference result, until the next distribution container is not matched, and use the latest model inference result as the model combination inference result.

[0114] The embodiment of the present application achieves the purpose of performing model combination reasoning on the inference sample based on the tracking identifier and the distribution container sequence. The AI ​​service sequentially calling module only needs to send a service request to the container distribution sequence once to obtain the model combination reasoning result, thereby avoiding the situation where the AI ​​service sequentially calling module calls the AI ​​model multiple times, and improving the efficiency of model combination reasoning through the AI ​​service sequentially calling module.

[0115] In this embodiment, it should be noted that the first model combination reasoning request can be an http service request including a tracking identifier and a sample to be inferred, and the tracking identifier is a sequence composed of at least one model identifier. The arrangement order of the model identifiers is consistent with the arrangement order of the distribution containers. One model identifier corresponds to a distribution container, which corresponds to an AI model with a specific function. According to the first-ranked model identifier, the distribution container to which the first model combination reasoning request needs to be transmitted can be determined. For example, assuming that the tracking identifier is a sequence ABC, the first model combination reasoning request needs to be passed by the AI ​​service to the distribution container corresponding to the model identifier A in sequence. Further, the distribution container corresponding to the model identifier A can pass the second model combination reasoning request to the distribution container corresponding to the model identifier B, and the distribution container corresponding to the model identifier B can pass the new second model combination reasoning request to the distribution container corresponding to the model identifier C.

[0116] Among them, step B10, through the AI ​​service sequentially calling the module to pass the first model combination inference request to the distribution container ranked first in the distribution container sequence, wherein the first model combination inference request includes a tracking identifier and a sample to be inferred, specifically includes: calling the model identifier ranked first in the tracking identifiers through the AI ​​service sequentially calling the module, and passing the first model combination inference request to the distribution container corresponding to the first model identifier, wherein the first model combination inference request includes a tracking identifier and a sample to be inferred.

[0117] Step B20, calling the corresponding AI model through the first-ranked distribution container to perform model reasoning on the sample to be inferred, and obtaining the model reasoning result. The specific implementation method is: forwarding the sample to be inferred in the first model combination reasoning request to the corresponding AI model through the first-ranked distribution container, and then performing model reasoning on the sample to be inferred through the AI ​​model to obtain the model reasoning result.

[0118] Furthermore, in step B20, the step of calling the corresponding AI model for model inference by the distribution container to obtain the model inference result includes:

[0119] Step C10: forwarding the sample to be inferred in the first model combination inference request or the model inference result in the second model combination inference request from the distribution container to the AI ​​model corresponding to the distribution container;

[0120] In this embodiment, when the distribution container is the first distribution container in the distribution container sequence, the sample to be inferred in the corresponding first model combination inference request is passed to the corresponding AI model through the distribution container; when the distribution container is not the first distribution container in the distribution container sequence, the model inference result in the corresponding second model combination inference request is passed to the corresponding AI model through the distribution container.

[0121] Step C20, using the AI ​​model corresponding to the distribution container, perform model reasoning on the sample to be inferred or the model reasoning result, obtain the model reasoning result of the current AI model and feed it back to the distribution container.

[0122] In this embodiment, the AI ​​model corresponding to the distribution container is used to perform model reasoning on the sample to be inferred or the model reasoning result to obtain the current model reasoning result, and the current model reasoning result is fed back to the corresponding distribution container.

[0123] In this embodiment, step B30, according to the tracking identifier, matches the next distribution container in the distribution container sequence, passes the second model combination inference request carrying the model inference result and the tracking identifier to the next distribution container, calls the corresponding AI model through the next distribution container to perform model inference on the model inference result and generate a new model inference result, until the next distribution container is not matched, and uses the latest model inference result as the model combination inference result, specifically including: according to the next model identifier in the tracking identifier, matches the next distribution container in the distribution container sequence through the first-order distribution container, and generates a second model combination inference request carrying the model inference result and the tracking identifier, and passes the second model combination inference request to the next distribution container, so as to call the corresponding AI model through the next distribution container to perform model inference on the model inference result and generate a new model inference result, until the next distribution container is not matched, and uses the latest model inference result as the model combination inference result.

[0124] It should be noted that the model identifier in the tracking identifier can only be called once and can only be called in sequence. For example, if the tracking identifier is (A, B), only model identifier A can be called for the first time and only model identifier B can be called for the second time.

[0125] Among them, according to the next model identifier in the tracking identifier, the first distribution container is matched to the next distribution container in the distribution container sequence, and a second model combination reasoning request carrying the model reasoning result and the tracking identifier is generated, and the second model combination reasoning request is transmitted to the next distribution container, so as to call the corresponding AI model through the next distribution container to perform model reasoning on the model reasoning result and generate a new model reasoning result, until the next distribution container is not matched, the step of using the latest model reasoning result as the model combination reasoning result includes:

[0126] The next model identifier is called in the tracking identifier. If the calling of the next model identifier in the tracking identifier is successful, a second model combination inference request carrying the model inference result and the tracking identifier is generated, and the second model combination inference request is passed to the next distribution container to call the AI ​​model corresponding to the next distribution container, and the model inference result is continued to be performed on the model inference result to obtain the next model inference result, and the execution step is returned: the next model identifier is called in the tracking identifier. If the calling of the model identifier in the tracking identifier fails, the latest model inference result is used as the model combination inference result.

[0127] Among them, step B30, matching the next distribution container in the distribution container sequence according to the tracking identifier, transmitting the second model combination inference request carrying the model inference result and the tracking identifier to the next distribution container, calling the corresponding AI model through the next distribution container to perform model inference on the model inference result and generate a new model inference result, until the next distribution container is not matched, the step of using the latest model inference result as the model combination inference result includes:

[0128] Step B31, matching the next distribution container in the distribution container sequence according to the tracking identifier;

[0129] In this embodiment, the next distribution container in the distribution container sequence is matched according to the tracking identifier. Specifically, the next distribution container in the distribution container sequence is matched by calling the model identifier in the tracking identifier.

[0130] The next distribution container in the distribution container sequence is matched by calling the model identifier in the tracking identifier. Specifically, if the model identifier in the tracking identifier is called, if the call is successful, the next distribution container in the distribution container sequence is matched successfully; if the call fails, the next distribution container in the distribution container sequence is matched unsuccessfully.

[0131] Step B32: if the matching fails, the latest model inference result is used as the model combination inference result;

[0132] Step B34, if the match is successful, a second model combination inference request carrying the model inference result and the tracking identifier is generated; and the second model combination inference request is passed to the next distribution container, and the corresponding AI model is called through the next distribution container to perform model inference on the model inference result to obtain a new model inference result, and return to the execution step: according to the tracking identifier, match the next distribution container in the distribution container sequence.

[0133] In this embodiment, if the match fails, it proves that the model combination reasoning process is over, and then the latest model reasoning result is used as the model combination reasoning result, and the model combination reasoning result is gradually fed back from the current distribution container to the first-ranked distribution container, and then fed back from the first-ranked distribution container to the AI ​​service calling module in sequence; if the match is successful, a second model combination reasoning request carrying the model reasoning result and the tracking identifier is generated, and then the second model combination reasoning request is passed to the next distribution container, and then the model reasoning result in the second model combination reasoning request is passed to the AI ​​model corresponding to the next distribution container through the next distribution container, so as to further perform model reasoning on the model reasoning result through the AI ​​model corresponding to the next distribution container to obtain a new model reasoning result, and return to the execution step: according to the tracking identifier, match the next distribution container in the distribution container sequence, wherein, since the AI ​​model sequence is not constructed in the embodiment of the present application, the same AI model can be called by multiple distribution containers, and each AI model can be arbitrarily combined to realize model combination reasoning, avoiding the situation where the AI ​​model is occupied due to the formation of each AI model into a fixed AI model sequence, and the AI ​​model can be reused in multiple model combination reasoning services, thereby improving the utilization efficiency of the AI ​​model.

[0134] like Figure 3 The figure shows a request lifecycle diagram for AI reasoning in an embodiment of the present application, wherein http is an http service request or a response feedback corresponding to an http request service, AI Pipline1 is the AI ​​service calling modules in sequence, trace id is the tracing identifier, AI1, AI2 and AI3 are all AI models, and Dispatch Sidecar is the distribution container.

[0135] The embodiment of the present application discloses a model combination reasoning method, that is, firstly, the first model combination reasoning request is transmitted to the distribution container ranked first in the distribution container sequence through the AI ​​service sequential calling module, wherein the first model combination reasoning request includes a tracking identifier and a sample to be inferred, and then the corresponding AI model is called by the first distribution container to perform model reasoning on the sample to be inferred to obtain a model reasoning result, and then according to the tracking identifier, the next distribution container in the distribution container sequence is matched, and a second model combination reasoning request carrying the model reasoning result and the tracking identifier is transmitted to the next distribution container, and the corresponding AI model is called by the next distribution container to perform model reasoning on the model reasoning result and generate a new model reasoning result, until the next distribution container is not matched, the latest model reasoning result is used as the model combination reasoning result, so as to achieve the purpose of performing combined reasoning on the sample to be inferred according to the tracking identifier and the distribution container sequence, and the AI ​​service sequential calling module only needs to send a service request to the container distribution sequence once to obtain the model combination reasoning result, thereby avoiding the situation where the AI ​​service sequential calling module calls the AI ​​model multiple times, and improving the efficiency of model combination reasoning through the AI ​​service sequential calling module.

[0136] Embodiment 3

[0137] Corresponding to the above-mentioned embodiment 1 and embodiment 2, the embodiment of the present application further provides an AI reasoning system, including a service layer, an application layer and a single AI service layer, wherein:

[0138] The service layer deploys at least one AI service sequential calling module, wherein the AI ​​service sequential calling module is used to generate a first model combination reasoning request corresponding to each AI combination reasoning requirement in the AI ​​combination reasoning service request sent by the client, and feed back the model combination reasoning results generated in response to each of the first model combination reasoning requests to the client;

[0139] The application layer deploys at least one distribution container sequence, wherein the distribution container sequence is used to receive the first model combination reasoning request transmitted by the AI ​​service calling the module in sequence, and perform model reasoning by calling the corresponding AI models, and respond to generate model combination reasoning results corresponding to each of the first model combination reasoning requests;

[0140] The single AI service layer deploys at least one AI model, wherein the AI ​​model is used for model reasoning.

[0141] The application layer includes a registration center, a container allocation module and a construction center, wherein:

[0142] The registration center is used to register the AI ​​combined reasoning service and the AI ​​service sequentially calling modules;

[0143] The container allocation module is used to allocate distribution containers to each of the AI ​​models in sequence to form a distribution container sequence corresponding to each of the AI ​​models;

[0144] The construction center is used to package the AI ​​model files in the single AI service layer into a mirror image with external service functions.

[0145] Optionally, the AI ​​service calls the modules in sequence including:

[0146] A receiving and generating unit, configured to receive an AI combined reasoning service request sent by a client, wherein the AI ​​combined reasoning service request includes at least one AI combined reasoning requirement; and generate a first model combined reasoning request corresponding to each of the AI ​​combined reasoning requirements;

[0147] A request calling unit, configured to execute, for each of the first model combination reasoning requests, respectively: sending the first model combination reasoning request to the corresponding distribution container sequence to request calling at least one AI model corresponding to the distribution container sequence to perform model reasoning, and obtaining a model combination reasoning result corresponding to the first model combination reasoning request;

[0148] A feedback unit is used to feed back the inference results of each model combination to the client.

[0149] Optionally, the distribution container sequence includes at least one distribution container arranged in sequence, and the request calling unit is further used for:

[0150] The first model combination reasoning request is transmitted to the distribution container ranked first in the distribution container sequence by sequentially calling the module through the AI ​​service, wherein the first model combination reasoning request includes a tracking identifier and a sample to be inferred;

[0151] Calling the corresponding AI model through the first-ranked distribution container to perform model inference on the sample to be inferred, and obtaining a model inference result;

[0152] According to the tracking identifier, the next distribution container in the distribution container sequence is matched, and a second model combination inference request carrying the model inference result and the tracking identifier is passed to the next distribution container. The corresponding AI model is called through the next distribution container to perform model inference on the model inference result and generate a new model inference result, until the next distribution container is not matched, and the latest model inference result is used as the model combination inference result.

[0153] Optionally, the request calling unit is further used for:

[0154] Matching a next distribution container in the distribution container sequence according to the tracking identifier;

[0155] If the matching fails, the latest model inference result is used as the model combination inference result;

[0156] If the match is successful, a second model combination inference request carrying the model inference result and the tracking identifier is generated; and the second model combination inference request is passed to the next distribution container, and the corresponding AI model is called through the next distribution container to perform model inference on the model inference result to obtain a new model inference result, and return to the execution step: according to the tracking identifier, match the next distribution container in the distribution container sequence.

[0157] Optionally, the request calling unit is further used for:

[0158] Forwarding the sample to be inferred in the first model combination inference request or the model inference result in the second model combination inference request from the distribution container to the AI ​​model corresponding to the distribution container;

[0159] Using the AI ​​model corresponding to the distribution container, model reasoning is performed on the sample to be inferred or the model reasoning result, the model reasoning result of the current AI model is obtained and fed back to the distribution container.

[0160] Optionally, the feedback unit is further used for:

[0161] Integrate the model combination reasoning results fed back by each distribution container sequence into a service request response result;

[0162] The service request response result is fed back to the client.

[0163] Optionally, the AI ​​reasoning system further includes:

[0164] A module identification generating unit, configured to generate a corresponding module identification according to the AI ​​combination reasoning requirement information in the AI ​​combination reasoning service request;

[0165] A query unit is used to query whether there is an AI service sequentially calling module corresponding to the AI ​​combined reasoning service request based on the module identifier; if so, pass the AI ​​combined reasoning service request to the AI ​​service sequentially calling module; if not, deploy the AI ​​combined reasoning service corresponding to the AI ​​combined reasoning service request, and pass the AI ​​combined reasoning service request to the AI ​​service sequentially calling module in the AI ​​combined reasoning service.

[0166] Specifically, Figure 4The figure shows the system framework diagram of the AI ​​reasoning system. Among them, the AI ​​Serving Pipeline is the AI ​​service calling modules in sequence.

[0167] The first layer is the service layer, which provides each AI service to call the module in sequence. By calling the AI ​​service to call the module in sequence of an AI reasoning service, the entire AI service can call the combined reasoning service of all the predetermined AIs in the module in sequence to obtain the final model combination reasoning result.

[0168] The second layer is the application layer, which includes a registration center, a container allocation module, a construction center, and a service visualization page. Among them, the construction center packages the AI ​​model files in the next layer of the single AI service layer into an image of an external service function (RPC service or HTTP service), and the AI ​​reasoning service can be deployed and registered through the registration center. When using tfserving (or other methods) to deploy an AI reasoning service, a distribution container (Dispatch SideCar) will be automatically started on the side of this service, and this container will distribute service requests, process context input and output, and jump to the next AI reasoning service. The registration center can also be used to register AI services to call modules in sequence. AI services calling modules in sequence are equivalent to the combination of multiple AI reasoning services, which exist in the form of Pipeline (pipeline or pipeline), where Pipeline is a declarative application for Pipeline registration to the registration center through a configuration file (such as yaml format). The configuration file includes but is not limited to: Pipeline service routing, Pipeline name, AI reasoning services used, the order of AI reasoning services, and context input and output format processing code. All configurations will be updated to the dispatch container (Dispatch SideCar) next to each AI inference service. The service visualization page (ServingDashboard) provides a visual interface for services, providing a browse of the Pipeline and the AI ​​inference service, as well as monitoring of the current load and number of replicas of these AI inference services.

[0169] The third layer is the AI ​​reasoning service, which is also the minimized AI reasoning unit we can use, that is, the AI ​​model.

[0170] The fourth layer is the underlying application, which mainly uses basic services such as Kubernetes, Docker container warehouse and redis.

[0171] Among them, in the embodiments of the present application, for the contents that are the same as or corresponding to Embodiment 1 or Embodiment 2, please refer to the above introduction and will not be repeated later.

[0172] The AI ​​reasoning system of the present invention adopts the AI ​​reasoning method provided in the above-mentioned embodiment 1 to solve the technical problem of low efficiency of client requesting AI reasoning service. Compared with the prior art, the beneficial effects of the AI ​​reasoning system provided in the embodiment of the present invention are the same as the beneficial effects of the AI ​​reasoning method provided in the above-mentioned embodiment 1, and other technical features in the system are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0173] Embodiment 4

[0174] An embodiment of the present invention provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the AI ​​reasoning method in the above-mentioned embodiment one.

[0175] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0176] like Figure 5 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device are also stored. The processing device, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0177] Typically, the following systems can be connected to the I / O interface: input devices including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices including, for example, a magnetic tape, a hard disk, etc.; and communication devices. The communication device can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0178] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0179] The electronic device provided by the present invention adopts the AI ​​reasoning method in the above-mentioned embodiment 1 or embodiment 2 to solve the technical problem of low efficiency of the client requesting AI reasoning service. Compared with the prior art, the beneficial effects of the electronic device provided by the embodiment of the present invention are the same as the beneficial effects of the AI ​​reasoning method provided by the above-mentioned embodiment 1, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0180] It should be understood that the various parts of the present disclosure can be implemented with hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0181] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0182] Embodiment 5

[0183] This embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the AI ​​reasoning method in the above-mentioned embodiment 1.

[0184] The computer-readable storage medium provided in the embodiment of the present invention may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0185] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0186] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device: obtains at least two Internet Protocol addresses; sends a node evaluation request including the at least two Internet Protocol addresses to a node evaluation device, wherein the node evaluation device selects an Internet Protocol address from the at least two Internet Protocol addresses and returns it; receives the Internet Protocol address returned by the node evaluation device; wherein the obtained Internet Protocol address indicates an edge node in a content distribution network.

[0187] Alternatively, the computer-readable storage medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device: receives a node evaluation request including at least two Internet Protocol addresses; selects an Internet Protocol address from the at least two Internet Protocol addresses; and returns the selected Internet Protocol address; wherein the received Internet Protocol address indicates an edge node in a content distribution network.

[0188] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0189] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0190] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0191] The computer-readable storage medium provided by the present invention stores computer-readable program instructions for executing the above-mentioned AI reasoning method, which solves the technical problem of low efficiency of requesting AI reasoning services by the client. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the embodiment of the present invention are the same as the beneficial effects of the AI ​​reasoning method provided by the above-mentioned embodiment 1 or embodiment 2, and will not be repeated here.

[0192] Embodiment 6

[0193] The present application also provides a product, which is a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the AI ​​reasoning method as described above are implemented.

[0194] The computer program product provided by this application solves the technical problem of low efficiency of client requesting AI reasoning service. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present invention are the same as the beneficial effects of the AI ​​reasoning method provided by the above-mentioned embodiment 1 or embodiment 2, which will not be repeated here.

[0195] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the present application.

Claims

1. An AI reasoning method, characterized in that: The AI ​​reasoning method includes: Receiving an AI combination reasoning service request sent by a client, wherein the AI ​​combination reasoning service request includes multiple AI combination reasoning requirements; generating a corresponding module identifier according to the AI ​​combination reasoning requirement information in the AI ​​combination reasoning service request; According to the module identifier, query whether there is an AI service calling module in sequence corresponding to the AI ​​combined reasoning service request; If so, the AI ​​combined reasoning service request is passed to the AI ​​service sequentially calling module; If it does not exist, deploy the AI ​​combined reasoning service corresponding to the AI ​​combined reasoning service request, and pass the AI ​​combined reasoning service request to the AI ​​service in the AI ​​combined reasoning service to call the modules in sequence; Generate a first model combination reasoning request corresponding to each of the AI ​​combination reasoning requirements; For each of the first model combination reasoning requests, respectively execute: sending the first model combination reasoning request to the corresponding distribution container sequence to request calling at least one AI model corresponding to the distribution container sequence to perform model reasoning, and obtaining the model combination reasoning result corresponding to the first model combination reasoning request; The inference results of each model combination are fed back to the client.

2. The AI ​​reasoning method according to claim 1, characterized in that: The distribution container sequence includes at least one distribution container arranged in sequence, and the step of sending the first model combination inference request to the corresponding distribution container sequence to request calling at least one AI model corresponding to the distribution container sequence for model inference, and obtaining the model combination inference result corresponding to the first model combination inference request includes: The first model combination reasoning request is transmitted to the distribution container ranked first in the distribution container sequence by sequentially calling the module through the AI ​​service, wherein the first model combination reasoning request includes a tracking identifier and a sample to be inferred; Calling the corresponding AI model through the first-ranked distribution container to perform model inference on the sample to be inferred, and obtaining a model inference result; According to the tracking identifier, the next distribution container in the distribution container sequence is matched, and a second model combination inference request carrying the model inference result and the tracking identifier is passed to the next distribution container. The corresponding AI model is called through the next distribution container to perform model inference on the model inference result and generate a new model inference result, until the next distribution container is not matched, and the latest model inference result is used as the model combination inference result.

3. The AI ​​reasoning method according to claim 2, characterized in that: The step of matching the next distribution container in the distribution container sequence according to the tracking identifier, transmitting the second model combination inference request carrying the model inference result and the tracking identifier to the next distribution container, calling the corresponding AI model through the next distribution container to perform model inference on the model inference result and generate a new model inference result, until the next distribution container is not matched, and using the latest model inference result as the model combination inference result includes: Matching a next distribution container in the distribution container sequence according to the tracking identifier; If the matching fails, the latest model inference result is used as the model combination inference result; If the match is successful, a second model combination inference request carrying the model inference result and the tracking identifier is generated; and the second model combination inference request is passed to the next distribution container, and the corresponding AI model is called through the next distribution container to perform model inference on the model inference result to obtain a new model inference result, and return to the execution step: according to the tracking identifier, match the next distribution container in the distribution container sequence.

4. The AI ​​reasoning method according to claim 1 or 2, characterized in that: The steps of calling the corresponding AI model for model inference by distributing the container and obtaining the model inference result include: Forwarding the sample to be inferred in the first model combination inference request or the model inference result in the second model combination inference request from the distribution container to the AI ​​model corresponding to the distribution container; Using the AI ​​model corresponding to the distribution container, model reasoning is performed on the sample to be inferred or the model reasoning result, the model reasoning result of the current AI model is obtained and fed back to the distribution container.

5. The AI ​​reasoning method according to claim 1, characterized in that: The step of feeding back the inference results of each model combination to the client comprises: Integrate the model combination reasoning results fed back by each distribution container sequence into a service request response result; The service request response result is fed back to the client.

6. An AI reasoning system, characterized in that: It includes service layer, application layer and single AI service layer, among which: The service layer deploys at least one AI service sequential calling module, wherein the AI ​​service sequential calling module is used to generate a first model combination reasoning request corresponding to each AI combination reasoning requirement in the AI ​​combination reasoning service request sent by the client, and feed back the model combination reasoning results generated in response to each of the first model combination reasoning requests to the client; The application layer deploys at least one distribution container sequence, wherein the distribution container sequence is used to receive the first model combination reasoning request transmitted by the AI ​​service calling the module in sequence, and perform model reasoning by calling the corresponding AI models, and respond to generate model combination reasoning results corresponding to each of the first model combination reasoning requests; The single AI service layer deploys at least one AI model, wherein the AI ​​model is used for model reasoning; The AI ​​service calls the modules in sequence including: A receiving and generating unit, configured to receive an AI combined reasoning service request sent by a client, wherein the AI ​​combined reasoning service request includes multiple AI combined reasoning requirements; and generate a first model combined reasoning request corresponding to each of the AI ​​combined reasoning requirements; A request calling unit, configured to execute, for each of the first model combination reasoning requests, respectively: sending the first model combination reasoning request to the corresponding distribution container sequence to request calling at least one AI model corresponding to the distribution container sequence to perform model reasoning, and obtaining a model combination reasoning result corresponding to the first model combination reasoning request; A feedback unit, used for feeding back the inference results of each model combination to the client; The AI ​​reasoning system also includes: A module identification generating unit, configured to generate a corresponding module identification according to the AI ​​combination reasoning requirement information in the AI ​​combination reasoning service request; A query unit is used to query whether there is an AI service sequentially calling module corresponding to the AI ​​combined reasoning service request based on the module identifier; if so, pass the AI ​​combined reasoning service request to the AI ​​service sequentially calling module; if not, deploy the AI ​​combined reasoning service corresponding to the AI ​​combined reasoning service request, and pass the AI ​​combined reasoning service request to the AI ​​service sequentially calling module in the AI ​​combined reasoning service.

7. The AI ​​reasoning system according to claim 6, characterized in that: The application layer includes a registration center, a container allocation module and a construction center, wherein: The registration center is used to register the AI ​​combined reasoning service and the AI ​​service sequentially calling modules; The container allocation module is used to allocate distribution containers to each of the AI ​​models in sequence to form a distribution container sequence corresponding to each of the AI ​​models; The construction center is used to package the AI ​​model files in the single AI service layer into a mirror image with external service functions.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the AI ​​reasoning method according to any one of claims 1 to 5.

9. A readable storage medium, characterized in that: The readable storage medium stores a program for implementing the AI ​​reasoning method, and the program for implementing the AI ​​reasoning method is executed by a processor to implement the steps of the AI ​​reasoning method as claimed in any one of claims 1 to 5.

Images