A model reasoning method, device and electronic device for audio and video

Abstract

The present application provides a model reasoning method, device and electronic device for audio and video, wherein the method includes: obtaining audio and video data to be inferred; the audio and video data to be inferred includes multiple frames of data; inputting the audio and video data to be inferred into a first-layer decision model to obtain a first output result; the first-layer decision model is obtained by training a pre-constructed first-layer intelligent agent with valid data and negative sample data; if the first output result indicates that multiple inference cards need to be used in parallel to process the audio and video data to be inferred, the audio and video data to be inferred is input into a second-layer decision model to obtain a second output result; through the second output result, at least two inference cards are determined to perform inference on the audio and video data to be inferred; at least two inference cards perform inference on the audio and video data to be inferred to obtain the inferred results for each frame of data.

Description

Technical Field

[0001] The present application relates to the field of large model reasoning technology, and in particular to a model reasoning method, device and electronic equipment for audio and video. Background Art

[0002] At present, with the rapid development of AI technology, more and more technologies can be implemented with models. Among them, large models have shown strong generalization capabilities in various model processing tasks such as machine translation, text generation, code development, and intelligent question and answer. At the same time, the number of parameters of large models has increased from billions to hundreds of billions and trillions. The current large model reasoning is to load the large model parameters into the inference card memory, and deploy the model as an inference service through the inference engine, so that a large amount of storage resources and computing resources are required in the inference process, which greatly increases the cost of computing and storage resources. Summary of the invention

[0003] The present application provides a model reasoning method, device and electronic device for audio and video, and the related technical solutions are as follows:

[0004] In a first aspect, an embodiment of the present application provides a model reasoning method for audio and video, the method comprising:

[0005] Acquire audio and video data to be inferred; the audio and video data to be inferred includes multiple frames of data;

[0006] The audio and video data to be inferred are input into a first-layer decision model to obtain a first output result; the first-layer decision model is obtained by training a pre-constructed first-layer agent with valid data and negative sample data; the valid data and the negative sample data are data determined by comparing the throughput rates obtained by inferring test samples with multiple inference card combinations; each of the inference card combinations includes at least one inference card, and the number of inference cards included in each of the inference card combinations is different;

[0007] If the first output result indicates that multiple inference cards need to be used in parallel to process the audio and video data to be inferred, the audio and video data to be inferred is input into the second-layer decision model to obtain a second output result; the second-layer decision model is obtained by training the pre-constructed second-layer intelligent agent with the valid data;

[0008] Determine at least two inference cards for inferring the audio and video data to be inferred based on the second output result;

[0009] The at least two inference cards infer the audio and video data to be inferred to obtain inferred results for each frame of data.

[0010] In one possible implementation manner, before obtaining the audio and video data to be inferred, the method further includes:

[0011] Acquire a set of multiple intelligent solutions, each of which includes at least one large model;

[0012] Each of the intelligent solutions is deployed on a hardware device including a plurality of inference cards, and each of the intelligent solutions is sequentially run on the hardware device to obtain the valid data and the negative sample data; each of the inference cards is loaded with a large model corresponding to any intelligent solution in the set.

[0013] In one possible implementation manner, after deploying each of the intelligent solutions on a hardware device including a plurality of inference cards, the method further includes:

[0014] When running any intelligent solution in the set on the hardware device, the test sample is sent to the plurality of inference card combinations by increasing the number of inference cards one by one, and the inference cards in each of the inference card combinations perform inference, thereby obtaining the throughput of each of the inference card combinations;

[0015] Comparing the throughput rates of the respective inference card combinations to obtain the valid data and the negative sample data;

[0016] The first-layer agent is trained by using the valid data and the negative sample data to obtain the first-layer decision model;

[0017] The second-layer agent is trained using the valid data to obtain the second-layer decision model.

[0018] In one possible implementation, comparing the throughput of each of the inference card combinations to obtain the valid data and the negative sample data includes:

[0019] Determine the throughput rate of the current inference card combination and the throughput rate of the previous inference card combination; the current inference card combination includes the inference cards in the previous inference card combination and a newly added inference card;

[0020] If the throughput rate of the current inference card combination is less than the throughput rate of the previous inference card combination, then saving the data corresponding to the current inference card combination to the negative sample data;

[0021] If the throughput rate of the current inference card combination is greater than the throughput rate of the previous inference card combination, the data corresponding to the current inference card combination is saved to the valid data; the valid data includes each inference card in the current inference card combination and the utilization rate of each inference card.

[0022] In one possible implementation manner, before training the first-layer agent by using the valid data and the negative sample data to obtain the first-layer decision model, the method further includes:

[0023] Based on the idea of ​​ensemble learning, the first layer of intelligent agents is pre-constructed; the first layer of intelligent agents includes various categories of machine learning algorithms, and various categories of machine learning algorithms are assigned corresponding weights.

[0024] In one possible implementation manner, before training the second-layer agent with the valid data to obtain the second-layer decision model, the method further includes:

[0025] The second-layer agent is pre-constructed, and the second-layer agent is composed of a plurality of differentiated neural networks.

[0026] In one possible implementation manner, after the at least two inference cards infer the audio and video data to be inferred and obtain the inference result, the method further includes:

[0027] According to the time sequence of each frame data in the multiple frames of data, the inferred data for each frame data is output.

[0028] In a second aspect, an embodiment of the present application provides a model reasoning device for audio and video, including:

[0029] An acquisition unit, used to acquire audio and video data to be inferred; the audio and video data to be inferred includes multiple frames of data;

[0030] The first output unit is used to input the audio and video data to be inferred into the first-layer decision model to obtain a first output result; the first-layer decision model is obtained by training a pre-constructed first-layer intelligent agent with valid data and negative sample data; the valid data and the negative sample data are data determined by comparing the throughput obtained by inferring the test samples with multiple inference card combinations; each of the inference card combinations includes at least one inference card, and the number of inference cards included in each of the inference card combinations is different;

[0031] A second output unit is used for inputting the audio and video data to be inferred into a second-layer decision model to obtain a second output result if the first output result indicates that multiple inference cards need to be used in parallel to process the audio and video data to be inferred; the second-layer decision model is obtained by training a pre-constructed second-layer intelligent agent with the valid data;

[0032] A decision unit, configured to determine at least two inference cards for inferring the audio and video data to be inferred based on the second output result;

[0033] The inference unit is used to infer the audio and video data to be inferred by the at least two inference cards to obtain the inferred results for each frame of data.

[0034] In a possible implementation, before the acquisition unit acquires the audio and video data to be inferred, the device further includes a control unit, and the control unit is specifically configured to:

[0035] Acquire a set of multiple intelligent solutions, each of which includes at least one large model;

[0036] Each of the intelligent solutions is deployed on a hardware device including a plurality of inference cards, and each of the intelligent solutions is sequentially run on the hardware device to obtain the valid data and the negative sample data; each of the inference cards is loaded with a large model corresponding to any intelligent solution in the set.

[0037] In a possible implementation, after the control unit deploys each of the intelligent solutions on a hardware device including a plurality of inference cards, the apparatus further includes a processing unit, wherein the processing unit is specifically configured to:

[0038] When running any intelligent solution in the set on the hardware device, the test sample is sent to the plurality of inference card combinations by increasing the number of inference cards one by one, and the inference cards in each of the inference card combinations perform inference, thereby obtaining the throughput of each of the inference card combinations;

[0039] Comparing the throughput rates of the respective inference card combinations to obtain the valid data and the negative sample data;

[0040] The first-layer agent is trained by using the valid data and the negative sample data to obtain the first-layer decision model;

[0041] The second-layer agent is trained using the valid data to obtain the second-layer decision model.

[0042] In a possible implementation manner, the processing unit is specifically configured to:

[0043] Determine the throughput rate of the current inference card combination and the throughput rate of the previous inference card combination; the current inference card combination includes the inference cards in the previous inference card combination and a newly added inference card;

[0044] If the throughput rate of the current inference card combination is less than the throughput rate of the previous inference card combination, then saving the data corresponding to the current inference card combination to the negative sample data;

[0045] If the throughput rate of the current inference card combination is greater than the throughput rate of the previous inference card combination, the data corresponding to the current inference card combination is saved to the valid data; the valid data includes each inference card in the current inference card combination and the utilization rate of each inference card.

[0046] In a possible implementation, before the processing unit trains the first-layer agent by using the valid data and the negative sample data to obtain the first-layer decision model, the apparatus further includes a first construction unit, wherein the first construction unit is specifically configured to:

[0047] Based on the idea of ​​ensemble learning, the first layer of intelligent agents is pre-constructed; the first layer of intelligent agents includes various categories of machine learning algorithms, and various categories of machine learning algorithms are assigned corresponding weights.

[0048] In a possible implementation, before the processing unit trains the second-layer agent through the valid data to obtain the second-layer decision model, the device further includes a second construction unit, wherein the second construction unit is specifically configured to:

[0049] The second-layer agent is pre-constructed, and the second-layer agent is composed of a plurality of differentiated neural networks.

[0050] In a possible implementation, after the inference unit uses the at least two inference cards to infer the audio and video data to be inferred and obtains the inferenced result, the device further includes a third output unit, and the third output unit is specifically used to:

[0051] According to the time sequence of each frame data in the multiple frames of data, the inferred data for each frame data is output.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0053] Memory, used to store computer programs;

[0054] The processor is used to implement the method steps described in any one of the above items when executing the computer program stored in the memory.

[0055] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the method steps described in any one of the above items.

[0056] The beneficial effects of the embodiments of the present application are as follows:

[0057] The embodiment of the present application provides a model reasoning method, device and electronic device for audio and video. First, the audio and video data to be inferred is obtained, and the audio and video data to be inferred includes multiple frames of data; then, the audio and video data to be inferred is input into the first-layer decision model to obtain a first output result; wherein the first-layer decision model is obtained by training a pre-constructed first-layer intelligent agent with valid data and negative sample data; the valid data and negative sample data are data determined by comparing the throughput obtained by reasoning a test sample with a combination of multiple inference cards; each inference card combination includes at least one inference card; wherein at least one inference card can be one or more, which is not limited here. Since the valid data and negative sample data are data determined by comparing the throughput obtained by reasoning a test sample with a combination of multiple inference cards, the valid data and negative sample data can be used as valid training data for training the pre-constructed first-layer intelligence in the future, thereby ensuring the diversity of the first-layer decision model obtained through training.

[0058] In addition, if the first output result indicates that multiple inference cards need to be used in parallel to process the audio and video data to be inferred, the audio and video data to be inferred is input into the second-layer decision model to obtain a second output result; further, at least two inference cards for inference of the audio and video data to be inferred are determined by the second output result; then, the at least two inference cards are used to infer the audio and video data to be inferred, thereby obtaining the inferred result for each frame of data. Among them, the second-layer decision model is obtained by training the pre-constructed second-layer intelligent agent with valid data. In other words, whether to input the audio and video data to be inferred into the second-layer decision model depends on the output result of the first-layer decision model, thereby avoiding invalid calculations and reducing the overall time consumption of the solution. In particular, when multiple inference cards are used for inference, the utilization efficiency of multiple inference cards is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flow chart of one of the methods of model reasoning for audio and video provided in an embodiment of the present application;

[0060] Figure 2 For Figure 1 One of the method flow charts before step S101;

[0061] Figure 3 For Figure 2 A flow chart of one of the methods after step S202;

[0062] Figure 4 for Figure 3A method flow chart of step S302;

[0063] Figure 5 A flow chart of a method for obtaining training data based on a model inference method for audio and video provided in an embodiment of the present application;

[0064] Figure 6 A flow chart of one method for reasoning on audio and video data based on the model reasoning method for audio and video provided in an embodiment of the present application;

[0065] Figure 7 A schematic diagram of a structure of a model reasoning device for audio and video provided in an embodiment of the present application;

[0066] Figure 8 A schematic diagram of a structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0068] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0069] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0070] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0071] Before introducing a model inference method for audio and video provided in an embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced in detail.

[0072] In practical applications, large models can be used in the audio and video field. However, intelligent audio and video solutions often require higher asynchronous task concurrency capabilities to achieve higher frame rates in order to meet the needs of intelligent solutions.

[0073] The current methods for optimizing large model reasoning include data parallelism, tensor parallelism, and pipeline parallelism. Among them, pipeline parallelism requires dividing each layer of the large model into multiple pipeline stages, and different inference cards are stored in different layers to form pipeline reasoning; however, this method requires communication between different stages, and its impact is time-consuming. Tensor parallelism is to split the weights of certain layers in the model, and different inference cards retain some weights, which also requires multi-card communication. Data parallelism is to split the data and save the same model parameters on different inference cards, but how to adaptively use multiple inference cards and data timing synchronization are still relatively difficult problems. Therefore, it is necessary to optimize model reasoning for the audio and video field in order to reduce the model data storage cost and computing cost while taking into account the reasoning effect of audio and video intelligent solutions.

[0074] In view of this, the present application provides a model reasoning method, device and electronic device for audio and video, which are used to optimize model reasoning in the audio and video field and improve the reasoning effect of audio and video intelligent solutions.

[0075] Please refer to Figure 1 As shown, an embodiment of the present application provides a model reasoning method for audio and video, the method comprising:

[0076] S101: Acquire audio and video data to be inferred; the audio and video data to be inferred includes multiple frames of data;

[0077] In the specific implementation process, first, the audio and video data to be inferred is obtained. Among them, the audio and video data to be inferred includes multiple frames of data. In practical applications, the specific number of frames of multiple frames of data can be set according to actual needs, which is not limited here. In addition, the audio and video data to be inferred can be audio data or video data, which is not limited here. Exemplarily, the acquired audio and video data can be data from an image acquisition unit such as a camera, or data from other devices such as a mobile phone or a computer.

[0078] S102: Input the audio and video data to be inferred into the first-layer decision model to obtain a first output result; the first-layer decision model is obtained by training a pre-constructed first-layer agent with valid data and negative sample data; the valid data and the negative sample data are data determined by comparing the throughput obtained by inferring the test samples with multiple inference card combinations; each of the inference card combinations includes at least one inference card, and the number of inference cards included in each of the inference card combinations is different;

[0079] In the specific implementation process, after obtaining the audio and video data to be inferred, the audio and video data to be inferred are input into the first-layer decision model, so that the first-layer decision model outputs the first output result. Wherein, the first output result is used to indicate whether it is necessary to use multiple inference cards to process the audio and video data to be inferred in parallel; the first-layer decision model is a model obtained by training the pre-constructed first-layer agent through valid data and negative sample data; and the valid data and negative sample data are data determined by comparing the throughput obtained by the test sample inference of multiple inference card combinations; wherein, each inference card combination includes at least one inference card, and the at least one inference card can be one, or two or more, which is not limited here; and the number of inference cards included in each inference card combination is different. Exemplarily, the multiple inference card combinations are A, B and C, wherein A includes inference card a1, B includes inference cards b1 and b2, and C includes inference cards c1, c2 and c3. In this way, the valid data and negative sample data can be used as valid training data to train the pre-constructed first-layer agent, thereby obtaining the first-layer decision model. Since the first-level decision model is based on the valid data and negative sample data determined by comparing the throughput obtained by reasoning with test samples based on multiple inference card combinations, the model obtained after training the first-level intelligent agent ensures the accuracy of the subsequent first-level decision model in processing the inferred audio and video data.

[0080] S103: If the first output result indicates that multiple inference cards need to be used in parallel to process the audio and video data to be inferred, the audio and video data to be inferred is input into the second-layer decision model to obtain a second output result; the second-layer decision model is obtained by training a pre-constructed second-layer intelligent agent with the valid data;

[0081] In the specific implementation process, if the first output result indicates that multiple inference cards need to be used in parallel to process the audio and video data to be inferred, the audio and video data to be inferred is input into the second-layer decision model, so that the second output result is output through the second-layer decision model; the second output result is used to indicate the specific number of inference cards used, so that the audio and video data to be inferred are processed by the corresponding number of inference cards. Among them, the second-layer decision model is obtained by training the pre-constructed second-layer intelligent agent with valid data. In the specific implementation process, whether to continue the decision of the second-layer decision model mainly depends on the first output result of the first-layer decision model. For example, if the first output result indicates that a single inference card is needed to process the audio and video data to be inferred, there is no need to input the audio and video data to be inferred into the second-layer decision model. Only the single inference card is needed to process the audio and video data to be inferred, thereby reducing invalid calculations and reducing the overall time consumption of the solution.

[0082] S104: Determine at least two inference cards for inferring the audio and video data to be inferred based on the second output result;

[0083] In the specific implementation process, after the second-layer decision model outputs the second output result, at least two inference cards for inference on the audio and video data to be inferred can be determined by the second output result. Exemplarily, the at least two inference cards can be two, or three or more, which is not limited here.

[0084] S105: The at least two inference cards infer the audio and video data to be inferred to obtain inference results for each frame of data.

[0085] During the specific implementation process, after at least two inference cards for inferring the audio and video data to be inferred are determined through the second-level decision model, the at least two inference cards can be used to infer the audio and video data to be inferred, thereby obtaining the inferred results for each frame of data in the audio and video data to be inferred.

[0086] In the embodiments of the present application, Figure 2 As shown, before step S101: obtaining the audio and video data to be inferred, the method further includes:

[0087] S201: Acquire a set of multiple intelligent solutions, each of which includes at least one large model;

[0088] S202: deploying each of the intelligent solutions onto a hardware device comprising a plurality of inference cards, and running each of the intelligent solutions in sequence on the hardware device to obtain the valid data and the negative sample data; each of the inference cards is loaded with a large model corresponding to any intelligent solution in the set.

[0089] In the specific implementation process, the specific implementation process of step S201 to step S202 is as follows:

[0090] Before obtaining the audio and video data to be inferred, obtain a set including multiple intelligent solutions. Exemplarily, the model reasoning method for audio and video provided by the present application is applicable to a variety of scenarios, each scenario includes a variety of intelligent services, and each intelligent service includes multiple intelligent solutions; it can be obtained by obtaining intelligent solutions corresponding to different intelligent services in different scenarios, thereby obtaining a set including multiple intelligent solutions. Exemplarily, in behavioral scenarios, intelligent services include gathering, fighting, etc., and related intelligent solutions can be counting people, counting animals, etc. Exemplarily, in traffic scenarios, intelligent services include violations, traffic accidents, etc., and related intelligent solutions can be running light tests, line pressure tests, collision tests, etc. Of course, in actual applications, there can also be other types of scenarios and corresponding intelligent services and corresponding intelligent solutions, which are not limited here. In addition, each intelligent solution includes at least one large model, and at least one large model can be one, or one or more, which are not limited here.

[0091] In one exemplary embodiment, the obtained set including multiple intelligent solutions may be as shown in the following formula:

[0092] S1={solu1,solu2,…,solu N}

[0093] Wherein, N is an integer greater than 2, S1 represents a set of multiple intelligent solutions, solu1 represents the first intelligent solution in the set, solu2 represents the second intelligent solution in the set, ..., solu N represents the Nth intelligent solution in the set, where solu N It can be shown as the following formula:

[0094] solu N ={alg1, alg2,…, alg M}

[0095] Wherein, M is an integer greater than 2, M represents the total number of large models included in the Nth intelligent solution, alg1 represents the first large model in the Nth intelligent solution, alg2 represents the second large model in the Nth intelligent solution, ..., alg M Represents the Mth large model in the Nth intelligent solution.

[0096] After obtaining a set of multiple intelligent solutions, each intelligent solution in the set is deployed on a hardware device including multiple inference cards, and each intelligent solution is sequentially run on the hardware device, thereby providing a guarantee for subsequent acquisition of valid data and negative sample data. Exemplarily, the hardware device includes K inference cards, and the set of inference cards included in the hardware device can be as shown in the following formula:

[0097] PU={c1,c2,……,c K}

[0098] Where K is an integer greater than 2, PU represents the set of reasoning cards, c1 represents the first reasoning card, c2 represents the second reasoning card, ..., c K Represents the Kth inference card.

[0099] Taking the first smart solution (solu1) mentioned above as an example, solu1 can be deployed on c1, c2, ..., c K On; with the Nth smart solution mentioned above (solu N ) as an example, we can use solu N Deployed on c1, c2, ..., c K By analogy, the smart solutions in set S1 are deployed on c1, c2, ..., c K In this way, each intelligent solution can be run sequentially on the hardware device. In addition, each inference card is loaded with all the large models in any intelligent solution in set S1.

[0100] In the embodiments of the present application, Figure 3 As shown, after step S202: deploying each of the intelligent solutions on a hardware device including a plurality of inference cards, the method further includes:

[0101] S301: When running any intelligent solution in the set on the hardware device, the test sample is sent to the plurality of inference card combinations by increasing the number of inference cards one by one, and the inference cards in each of the inference card combinations perform inference to obtain the throughput of each of the inference card combinations;

[0102] S302: comparing the throughput of each of the inference card combinations to obtain the valid data and the negative sample data;

[0103] S303: training the first-layer agent by using the valid data and the negative sample data to obtain the first-layer decision model;

[0104] S304: Train the second-layer agent using the valid data to obtain the second-layer decision model.

[0105] In the specific implementation process, the specific implementation process of step S301 to step S304 is as follows:

[0106] After each intelligent solution in a set including multiple intelligent solutions is deployed on a hardware device including multiple inference cards, especially when any intelligent solution in the set is run on the hardware device, the throughput of each inference card combination can be obtained by increasing the number of inference cards one by one, sending test samples to multiple inference card combinations, and having the inference cards in each inference card combination perform inference.

[0107] Still taking the aforementioned set S1 as an example, select the intelligent solution solu1 in the set S1 and run it on the hardware device; by increasing the number of inference cards one by one until all the inference cards are used, the test samples are sent to multiple inference card combinations and processed by the inference cards in the corresponding combinations; illustratively, still taking the deployment of K inference cards on the hardware device as an example, the multiple inference card combinations include a combination consisting of a single inference card, a combination consisting of two inference cards, a combination consisting of three inference cards, ..., a combination consisting of K inference cards, and the test samples are sent to the single inference card processing, two inference cards processing, ..., K inference cards processing corresponding to the combination in turn.

[0108] Exemplarily, the audio and video data set corresponding to the test sample is as follows in time sequence:

[0109] img={pic1,pic2,…,pic J}

[0110] Where J is an integer greater than 2, img represents the audio and video data set corresponding to the test sample, pic1 represents the first frame of data, pic2 represents the second frame of data, ..., pic J Represents the J-th frame data.

[0111] Record the throughput of the intelligent solution solu1 at each time under each inference card combination, as shown below:

[0112] inout={algt1, algt2,…, algtU}

[0113] Among them, inout represents the real-time throughput rate of the intelligent solution solu1 under one of the inference card combinations; algt1 represents the throughput rate of the intelligent solution solu1 at time 1, algt2 represents the throughput rate of the intelligent solution solu1 at time 2, ..., algtU represents the throughput rate of the intelligent solution solu1 at time U.

[0114] Based on the same processing process, the audio and video data sets corresponding to the test samples can be input into each intelligent solution in set S1 except the intelligent solution solu1, and inferred by the inference cards in each inference card combination, so as to obtain the real-time throughput rate of running each intelligent solution in set S1 in sequence and processing the input test samples by each inference card combination.

[0115] Then, the throughput of each inference card combination is compared to obtain valid data and negative sample data; the specific process of obtaining valid data and negative sample data can refer to the description of the relevant part below, which will not be described in detail here. Subsequently, the first-layer intelligent agent can be trained with valid data and negative sample data to obtain the first-layer decision model. The second-layer intelligent agent can also be trained with valid data to obtain the second-layer decision model. The relevant training process can refer to the description of the relevant part below, which will not be described in detail here.

[0116] In the embodiments of the present application, Figure 4 As shown, step S302: comparing the throughput of each of the inference card combinations to obtain the valid data and the negative sample data, including:

[0117] S401: Determine the throughput rate of the current inference card combination and the throughput rate of the previous inference card combination; the current inference card combination includes the inference cards in the previous inference card combination and a newly added inference card;

[0118] S402: If the throughput rate of the current inference card combination is less than the throughput rate of the previous inference card combination, then saving the data corresponding to the current inference card combination to the negative sample data;

[0119] S403: If the throughput rate of the current inference card combination is greater than the throughput rate of the previous inference card combination, the data corresponding to the current inference card combination is saved to the valid data; the valid data includes each inference card in the current inference card combination and the utilization rate of each inference card.

[0120] In the specific implementation process, the specific implementation process of step S401 to step S403 is as follows:

[0121] Still taking the intelligent solution solu1 in set S1 as an example, first, determine the throughput of the current inference card combination and the throughput of the previous inference card combination; for example, the previous inference card combination only includes inference card a, and the current inference card combination includes two inference cards including inference card a and inference card b; the throughput under the current inference card combination is recorded as algtRT2, and the throughput under the previous inference card combination is recorded as algtRT.

[0122] Compare the throughput rate of the current inference card combination with the throughput rate of the previous inference card combination. If the throughput rate of the current inference card combination is less than that of the previous inference card combination, for example, algtRT2<algtRT, then save the data corresponding to the current inference card combination to the negative sample data (recorded as D1); if the throughput rate of the current inference card combination is greater than that of the previous inference card combination, for example, algtRT2>algtRT, then save the data corresponding to the current inference card combination to the valid data (recorded as D).

[0123] It should be noted that when the frames are sent, each frame of the audio and video data set is sent to the same inference card for processing, and the initial throughput rate (recorded as T0) is recorded, and the current throughput rate (recorded as algtRT1) is recorded in real time; further, the data in the audio and video data set corresponding to the test sample is evenly divided into two inference cards for processing, and the throughput rate (recorded as algtRT2) is recorded in real time; if algtRT2 < algtRT1, it indicates that only the single inference card is needed to infer the test sample to achieve a higher throughput rate, and the data related to the single inference card can be saved to the negative sample data (recorded as D1); if algtRT2 > algtRT1, it indicates that the two inference cards are needed to infer the test sample to achieve a higher throughput rate, and the current data related to the two inference cards can be saved to the valid data (recorded as D); wherein, the saved current data can be as follows:

[0124] data = {c1, c2, env1, env2}

[0125] Among them, data represents the current data, c1 represents the first inference card for processing the test sample, c2 represents the second inference card for processing the test sample, env1 represents the utilization rate of the first inference card, and env2 represents the utilization rate of the second inference card.

[0126] Furthermore, the data in the audio and video data set corresponding to the test sample is evenly divided into three inference cards for processing, and the current throughput is recorded in real time, which is recorded as algtRT3. If algtRT3<algtRT2, it means that only two inference cards are needed to infer the test sample to achieve a higher throughput, and the data related to these three inference cards can be saved to the negative sample data D1; if algtRT3>algtRT2, it means that these three inference cards are needed to infer the test sample, which can achieve a higher throughput than two inference cards. In this case, the current data related to these three inference cards can be saved to the valid data D. Based on the same processing process, the data in the audio and video data set corresponding to the test sample is evenly divided into K inference cards for processing in turn, and the real-time throughput under each inference card combination is recorded, and the corresponding valid data D and negative sample data D1 are recorded, thereby completing the recording and saving of the valid data D and negative sample data D1 of the first intelligent solution in set S1 under all inference card combinations. Based on the same processing process, the processing of all intelligent solutions in set S1 is completed in turn, and all valid data is recorded, as shown below:

[0127] D={data1, data2,..., dataM}

[0128] Among them, data1 represents the valid data saved by the first intelligent solution solu1 in set S1, data2 represents the valid data saved by the second intelligent solution solu2 in set S1, ..., dataM represents the valid data saved by the Mth intelligent solution solu in set S1. M Save valid data.

[0129] Combine the following Figure 5 The method flow chart shown provides a detailed explanation of the process of obtaining training data, wherein the obtained valid data D and negative sample data D1 are used to train the first layer of intelligent agents, and the obtained valid data D is used to train the second layer of intelligent agents.

[0130] s1: deploy intelligent solutions to hardware devices;

[0131] s2: count the number of inference cards in the hardware device;

[0132] s3: load the model of the intelligent solution to each inference card;

[0133] s4: input audio and video data;

[0134] s5: records the initial throughput rate T0 of the current intelligent solution;

[0135] Execute in sequence:

[0136] s6: single inference card processing; specifically, a single inference card processes the audio and video data;

[0137] s7: two inference cards for processing; specifically, the audio and video data are evenly divided into two inference cards, and are processed by the corresponding inference cards respectively;

[0138] By gradually increasing the number of inference cards, sequentially until step s8;

[0139] s8: K inference cards process; specifically, the audio and video data are evenly divided into K inference cards in sequence, and processed by the corresponding inference cards respectively;

[0140] s9: Determine whether the current throughput is greater than T0; specifically, determine whether the current throughput under each reasoning card combination is greater than the throughput of the previous reasoning card combination; for the convenience of drawing and illustrating, the throughput of the previous reasoning card combination is uniformly recorded as T0;

[0141] If yes, execute s10;

[0142] s10: save data to D;

[0143] If no, execute s11;

[0144] s11: save data to D1;

[0145] s12: whether to traverse all solutions; specifically, determine whether to traverse all intelligent solutions in set S1;

[0146] If yes, execute s13;

[0147] s13: complete the acquisition of training data;

[0148] If not, re-execute s1 until the training data is obtained.

[0149] In the embodiment of the present application, before step S303: training the first-layer agent by using the valid data and the negative sample data to obtain the first-layer decision model, the method further includes:

[0150] Based on the idea of ​​ensemble learning, the first layer of intelligent agents is pre-constructed; the first layer of intelligent agents includes various categories of machine learning algorithms, and various categories of machine learning algorithms are assigned corresponding weights.

[0151] In the specific implementation process, it is necessary to pre-build multi-card concurrent agents including the first-layer agents and the second-layer agents. For the first-layer agents, it can be constructed based on the idea of ​​ensemble learning and by machine learning algorithms of Y categories. Exemplarily, Y is an integer greater than 2, for example, 2≤Y≤8; of course, the specific value of Y can be set according to the actual application needs, which is not limited here. Exemplarily, the machine learning algorithms of Y categories can be linear regression, decision tree, random forest, support vector machine, etc. In addition, corresponding weights are assigned to various categories of machine learning algorithms. Accordingly, the output after the first-layer agent is as follows:

[0152]

[0153] Among them, OUT represents the final output result of the first-layer agent, out1 represents the output result of the first machine learning algorithm, w1 represents the weight corresponding to the output result of the first machine learning algorithm, out2 represents the output result of the second machine learning algorithm, w2 represents the weight corresponding to the output result of the second machine learning algorithm, ..., outY represents the output result of the Y-th machine learning algorithm, and wY represents the weight corresponding to the output result of the Y-th machine learning algorithm. In practical applications, the output result of each machine learning algorithm in the first-layer agent is multiplied by the corresponding weight, and the mode is finally taken. Then, the product after taking the mode is used as the final output result of the first-layer agent.

[0154] It should be noted that based on the idea of ​​ensemble learning, by building a diverse first-layer intelligent agent and assigning weights, the mode of all outputs is used as the final output, which improves the stability of the first-layer intelligent agent. In addition, the weight can be positively fed back based on the final output to ensure that the weight of the corresponding machine learning algorithm with correct prediction is increased, thereby improving the accuracy of the first-layer intelligent agent. In practical applications, the first-layer intelligent agent is trained with valid data and negative sample data, which improves the accuracy of the first-layer decision model obtained through training.

[0155] In the embodiment of the present application, before step S304: training the second-layer agent with the valid data to obtain the second-layer decision model, the method further includes:

[0156] The second-layer agent is pre-constructed, and the second-layer agent is composed of a plurality of differentiated neural networks.

[0157] In the specific implementation process, the pre-built multi-card concurrent intelligent agent includes not only the first-layer intelligent agent, but also the second-layer intelligent agent. The second-layer intelligent agent is composed of multiple neural networks with differences, wherein the multiple neural networks included in the second-layer intelligent agent can be two, or two or more, which is not limited here. In this way, the second-layer intelligent agent has good nonlinear fitting ability and prosperity ability. Exemplarily, each neural network has differences in the number of neural nodes, the number of neural layers, etc.

[0158] For the second layer agent, the loss function used can be as follows:

[0159]

[0160] in, F represents the loss function, Indicates the label that the i-th sample belongs to category j, a represents the number of samples, b represents the number of categories, It represents the probability that the i-th sample is predicted to be of category j.

[0161] In practical applications, in the process of training the second-layer agent with valid data, the relevant loss function can be used to determine the relevant parameters of each neural network, so as to obtain the second-layer decision model after training. In addition, in the specific implementation process, the decision of whether to perform the second-layer decision model depends on the output result of the first-layer decision model, which can reduce invalid calculations to a certain extent and reduce the overall time consumption of the solution.

[0162] In the specific implementation process, the valid data D and negative sample data D1 obtained above can be used to train the pre-built multi-card concurrent agent. Specifically, the labels of the data in D1 are all marked with -1, and the labels of the data in D are all marked with 1, and they are sent as input to the first-layer agent for training. After the training is completed, the data in D is sent as input to the second-layer agent for training, and the specific result of dividing the audio and video data evenly into several inference cards for inference is output.

[0163] In the embodiment of the present application, in step S105: after the at least two inference cards infer the audio and video data to be inferred and obtain the inferred results for each frame of data, the method further includes:

[0164] According to the time sequence of each frame data in the multiple frames of data, the inferred data for each frame data is output.

[0165] In the specific implementation process, after at least two inference cards infer the audio and video data to be inferred and obtain the inferred results for each frame of data, timing management is added to ensure the timing of the output results and the validity of the output results. Specifically, the inferred data for each frame of data can be output according to the time sequence of multiple frames of data in the audio and video data to be inferred, thereby ensuring the accuracy of the model inference for audio and video.

[0166] It should be noted that the model reasoning method for audio and video provided in the embodiment of the present application can be adaptively applied to scenarios with different types of reasoning cards, different numbers of reasoning cards, and different solutions, and can also be applied to reasoning card devices with lower computing power, thereby lowering the hardware use threshold for large models and improving the operating performance of large model solutions. Under the premise of ensuring the effect of large models, the throughput of the overall solution is improved and the cost of using large model solutions is reduced. In this way, the model reasoning method provided in the embodiment of the present application can be adaptively, highly stable, and highly accurate applied to the field of audio and video based on large models.

[0167] Combine the following Figure 6 The method flow chart shown in the figure provides a detailed explanation of the process of reasoning the audio and video data to be inferred according to the model reasoning method for audio and video provided in the embodiment of the present application:

[0168] s21: Get audio and video data;

[0169] s22: load the first-layer decision model;

[0170] s23: Obtain the first output result of the first-layer decision model;

[0171] s24: Determine whether to send to the second-layer decision model; specifically, based on the first output result, determine whether it is necessary to perform multi-card concurrent processing on the audio and video data;

[0172] If yes, execute s25;

[0173] s25: loading the second-layer decision model; accordingly, inputting the audio and video data into the second-layer decision model;

[0174] s26: obtaining a second output result output by the second-layer decision model;

[0175] s27: based on the second output result, determine a plurality of inference cards for inferring the audio and video data;

[0176] s28: perform inter-frame synchronization on each frame of data processed by multiple inference cards;

[0177] If no, execute s29;

[0178] s29: No need to send audio and video data to the second-layer decision model.

[0179] Based on the same inventive concept, Figure 7 As shown, the embodiment of the present application also provides a model reasoning device for audio and video, including:

[0180] An acquisition unit 10 is used to acquire audio and video data to be inferred; the audio and video data to be inferred includes multiple frames of data;

[0181] The first output unit 20 is used to input the audio and video data to be inferred into the first-layer decision model to obtain a first output result; the first-layer decision model is obtained by training the pre-constructed first-layer agent with valid data and negative sample data; the valid data and the negative sample data are data determined by comparing the throughput obtained by inferring the test samples with multiple inference card combinations; each of the inference card combinations includes at least one inference card, and the number of inference cards included in each of the inference card combinations is different;

[0182] The second output unit 30 is used for inputting the audio and video data to be inferred into the second-layer decision model to obtain a second output result if the first output result indicates that multiple inference cards need to be used in parallel to process the audio and video data to be inferred; the second-layer decision model is obtained by training the pre-constructed second-layer intelligent agent with the valid data;

[0183] A decision unit 40, configured to determine at least two inference cards for inferring the audio and video data to be inferred based on the second output result;

[0184] The inference unit 50 is used to infer the audio and video data to be inferred by the at least two inference cards to obtain the inferred results for each frame of data.

[0185] In the embodiment of the present application, before the acquisition unit 10 acquires the audio and video data to be inferred, the device further includes a control unit, which is specifically configured to:

[0186] Acquire a set of multiple smart solutions, each of which includes at least one large model;

[0187] Each of the intelligent solutions is deployed on a hardware device including a plurality of inference cards, and each of the intelligent solutions is sequentially run on the hardware device to obtain the valid data and the negative sample data; each of the inference cards is loaded with a large model corresponding to any intelligent solution in the set.

[0188] In the embodiment of the present application, after the control unit deploys each of the intelligent solutions to a hardware device including a plurality of inference cards, the apparatus further includes a processing unit, and the processing unit is specifically configured to:

[0189] When running any intelligent solution in the set on the hardware device, the test sample is sent to the plurality of inference card combinations by increasing the number of inference cards one by one, and the inference cards in each of the inference card combinations perform inference, thereby obtaining the throughput of each of the inference card combinations;

[0190] Comparing the throughput rates of the respective inference card combinations to obtain the valid data and the negative sample data;

[0191] The first-layer agent is trained by using the valid data and the negative sample data to obtain the first-layer decision model;

[0192] The second-layer agent is trained using the valid data to obtain the second-layer decision model.

[0193] In the embodiment of the present application, the processing unit is specifically used for:

[0194] Determine the throughput rate of the current inference card combination and the throughput rate of the previous inference card combination; the current inference card combination includes the inference cards in the previous inference card combination and a newly added inference card;

[0195] If the throughput rate of the current inference card combination is less than the throughput rate of the previous inference card combination, then saving the data corresponding to the current inference card combination to the negative sample data;

[0196] If the throughput rate of the current inference card combination is greater than the throughput rate of the previous inference card combination, the data corresponding to the current inference card combination is saved to the valid data; the valid data includes each inference card in the current inference card combination and the utilization rate of each inference card.

[0197] In the embodiment of the present application, before the processing unit trains the first-layer agent by using the valid data and the negative sample data to obtain the first-layer decision model, the device further includes a first construction unit, and the first construction unit is specifically used to:

[0198] Based on the idea of ​​ensemble learning, the first layer of intelligent agents is pre-constructed; the first layer of intelligent agents includes various categories of machine learning algorithms, and various categories of machine learning algorithms are assigned corresponding weights.

[0199] In the embodiment of the present application, before the processing unit trains the second-layer agent through the valid data to obtain the second-layer decision model, the device further includes a second construction unit, and the second construction unit is specifically used to:

[0200] The second-layer agent is pre-constructed, and the second-layer agent is composed of a plurality of differentiated neural networks.

[0201] In the embodiment of the present application, after the inference unit uses the at least two inference cards to infer the audio and video data to be inferred and obtains the inferenced result, the device further includes a third output unit, and the third output unit is specifically used to:

[0202] According to the time sequence of each frame data in the multiple frames of data, the inferred data for each frame data is output.

[0203] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the electronic device can implement the functions of the aforementioned model reasoning method for audio and video, referring to Figure 8 , the electronic device comprises:

[0204] Memory 100, used for storing computer programs;

[0205] The processor 200 is used to implement the steps of the model reasoning method for audio and video as described in any of the above items when executing the computer program stored in the memory.

[0206] The specific connection medium between the processor 200 and the memory 100 is not limited in the embodiment of the present application. Figure 8 In the example, the processor 200 and the memory 100 are connected via the bus 300. The bus 300 is connected to the memory 100 via the bus 300. Figure 8 The bus 300 is represented by a bold line, and the connection between other components is only for schematic illustration and is not intended to be limiting. The bus 300 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 200 can also be called a controller, and there is no limitation on the name.

[0207] In the embodiment of the present application, the memory 100 stores instructions that can be executed by the processor 200. When the processor 200 executes the computer program stored in the memory 100, the model inference method for audio and video discussed above can be executed. The processor 200 can implement Figure 7 The functions of each module are shown in the figure.

[0208] Among them, the processor 200 is the control center of the device, and can use various interfaces and lines to connect various parts of the entire control device, and through running or executing computer programs stored in the memory 100 and calling data stored in the memory 100, the various functions of the device and process data, thereby monitoring the device as a whole.

[0209] In one possible design, the processor 200 may include one or more processing units, and the processor 200 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. In some embodiments, the processor 200 and the memory 100 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on separate chips.

[0210] The memory 100, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 100 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. Alternatively, the memory 100 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 100 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0211] By designing and programming the processor 200, the code corresponding to the model reasoning method for audio and video introduced in the above embodiment can be fixed into the chip, so that the chip can execute the steps of the model reasoning method for audio and video discussed above when running. How to design and program the processor 200 is a technology well known to those skilled in the art and will not be described in detail here.

[0212] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and the computer program is used to enable a computer to execute the steps of the model inference method for audio and video discussed above.

[0213] In some possible implementations, various aspects of the model reasoning method for audio and video provided in the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on an apparatus, the program code is used to enable the control device to execute the steps of the model reasoning method for audio and video according to various exemplary implementations of the present application described above in this specification.

[0214] The embodiment of the present application provides a model reasoning method, device and electronic device for audio and video. First, the audio and video data to be inferred is obtained, and the audio and video data to be inferred includes multiple frames of data; then, the audio and video data to be inferred is input into the first-layer decision model to obtain a first output result; wherein the first-layer decision model is obtained by training a pre-constructed first-layer intelligent agent with valid data and negative sample data; the valid data and negative sample data are data determined by comparing the throughput obtained by reasoning a test sample with a combination of multiple inference cards; each inference card combination includes at least one inference card; wherein at least one inference card can be one or more, which is not limited here. Since the valid data and negative sample data are data determined by comparing the throughput obtained by reasoning a test sample with a combination of multiple inference cards, the valid data and negative sample data can be used as valid training data for training the pre-constructed first-layer intelligence in the future, thereby ensuring the diversity of the first-layer decision model obtained through training.

[0215] In addition, if the first output result indicates that multiple inference cards need to be used in parallel to process the audio and video data to be inferred, the audio and video data to be inferred is input into the second-layer decision model to obtain a second output result; further, at least two inference cards for inference of the audio and video data to be inferred are determined by the second output result; then, the at least two inference cards are used to infer the audio and video data to be inferred, thereby obtaining the inferred result for each frame of data. Among them, the second-layer decision model is obtained by training the pre-constructed second-layer intelligent agent with valid data. In other words, whether to input the audio and video data to be inferred into the second-layer decision model depends on the output result of the first-layer decision model, thereby avoiding invalid calculations and reducing the overall time consumption of the solution. In particular, when multiple inference cards are used for inference, the utilization efficiency of multiple inference cards is improved.

[0216] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0217] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0218] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0219] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0220] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A model reasoning method for audio and video, characterized in that: The method comprises: Acquire audio and video data to be inferred; the audio and video data to be inferred includes multiple frames of data; The audio and video data to be inferred are input into a first-layer decision model to obtain a first output result; the first-layer decision model is obtained by training a pre-constructed first-layer agent with valid data and negative sample data; the valid data and the negative sample data are data determined by comparing the throughput rates obtained by inferring test samples with multiple inference card combinations; each of the inference card combinations includes at least one inference card, and the number of inference cards included in each of the inference card combinations is different; If the first output result indicates that multiple inference cards need to be used in parallel to process the audio and video data to be inferred, the audio and video data to be inferred is input into the second-layer decision model to obtain a second output result; the second-layer decision model is obtained by training the pre-constructed second-layer intelligent agent with the valid data; Determine at least two inference cards for inferring the audio and video data to be inferred based on the second output result; The at least two inference cards infer the audio and video data to be inferred to obtain inference results for each frame of data; The specific process of obtaining the valid data and the negative sample data includes: Determine the throughput rate of the current inference card combination and the throughput rate of the previous inference card combination; the current inference card combination includes the inference cards in the previous inference card combination and a newly added inference card; If the throughput rate of the current inference card combination is less than the throughput rate of the previous inference card combination, then saving the data corresponding to the current inference card combination to the negative sample data; If the throughput rate of the current inference card combination is greater than the throughput rate of the previous inference card combination, the data corresponding to the current inference card combination is saved to the valid data; the valid data includes each inference card in the current inference card combination and the utilization rate of each inference card.

2. The method according to claim 1, characterized in that Before obtaining the audio and video data to be inferred, the method further includes: Acquire a set of multiple intelligent solutions, each of which includes at least one large model; Each of the intelligent solutions is deployed on a hardware device including a plurality of inference cards, and each of the intelligent solutions is sequentially run on the hardware device to obtain the valid data and the negative sample data; each of the inference cards is loaded with a large model corresponding to any intelligent solution in the set.

3. The method according to claim 2, characterized in that After deploying each of the intelligent solutions on a hardware device including a plurality of inference cards, the method further includes: When running any intelligent solution in the set on the hardware device, the test sample is sent to the plurality of inference card combinations by increasing the number of inference cards one by one, and the inference cards in each of the inference card combinations perform inference, thereby obtaining the throughput of each of the inference card combinations; Comparing the throughput rates of the respective inference card combinations to obtain the valid data and the negative sample data; The first-layer agent is trained by using the valid data and the negative sample data to obtain the first-layer decision model; The second-layer agent is trained using the valid data to obtain the second-layer decision model.

4. The method according to claim 3, characterized in that Before training the first-layer agent by using the valid data and the negative sample data to obtain the first-layer decision model, the method further includes: Based on the idea of ​​ensemble learning, the first layer of intelligent agents is pre-constructed; the first layer of intelligent agents includes various categories of machine learning algorithms, and various categories of machine learning algorithms are assigned corresponding weights.

5. The method according to claim 3, characterized in that Before training the second-layer agent by using the valid data to obtain the second-layer decision model, the method further includes: The second-layer agent is pre-constructed, and the second-layer agent is composed of a plurality of differentiated neural networks.

6. The method according to any one of claims 1 to 5, characterized in that: After the at least two inference cards infer the audio and video data to be inferred and obtain the inferenced result, the method further includes: According to the time sequence of each frame data in the multiple frames of data, the inferred data for each frame data is output.

7. A model reasoning device for audio and video, characterized in that: include: An acquisition unit, used to acquire audio and video data to be inferred; The audio and video data to be inferred includes multiple frames of data; A first output unit, used for inputting the audio and video data to be inferred into a first-layer decision model to obtain a first output result; The first-layer decision model is obtained by training a pre-constructed first-layer agent with valid data and negative sample data; the valid data and the negative sample data are data determined by comparing the throughput rates obtained by reasoning test samples with multiple inference card combinations; each of the inference card combinations includes at least one inference card, and the number of inference cards included in each of the inference card combinations is different; A second output unit is used for inputting the audio and video data to be inferred into a second-layer decision model to obtain a second output result if the first output result indicates that multiple inference cards need to be used in parallel to process the audio and video data to be inferred; the second-layer decision model is obtained by training a pre-constructed second-layer intelligent agent with the valid data; A decision unit, configured to determine at least two inference cards for inferring the audio and video data to be inferred based on the second output result; An inference unit, configured to use the at least two inference cards to infer the audio and video data to be inferred, and obtain inference results for each frame of data; The device further includes a processing unit, and the processing unit is specifically configured to: Determine the throughput rate of the current inference card combination and the throughput rate of the previous inference card combination; the current inference card combination includes the inference cards in the previous inference card combination and a newly added inference card; If the throughput rate of the current inference card combination is less than the throughput rate of the previous inference card combination, then saving the data corresponding to the current inference card combination to the negative sample data; If the throughput rate of the current inference card combination is greater than the throughput rate of the previous inference card combination, the data corresponding to the current inference card combination is saved to the valid data; the valid data includes each inference card in the current inference card combination and the utilization rate of each inference card.

8. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the method steps as described in any one of claims 1 to 6 when executing the computer program stored in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to enable a computer to execute the method steps according to any one of claims 1 to 6.

Images