Multi-model operation strategy generation method, device and equipment and readable storage medium
By responding to model scheduling operations in intelligent devices, matching target video frames and models, and generating strategic solutions, the problem of reduced operational efficiency caused by competition in multiple models is solved, and more efficient hardware resource utilization and user experience improvement is achieved.
Patent Information
- Application Number
- CN202411781836.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2025-03-18
AI Technical Summary
In smart devices, due to limited hardware resources, resource competition problems are prone to occur when deploying multiple models, resulting in reduced model operation efficiency and poor user experience.
By responding to model scheduling operations, the target video frames to be processed are determined, and each video frame is modeled. Combined with the current computing resources, model calculation costs and dependencies, a strategy plan for scheduling each target model is generated.
By filtering the target models with operation requirements, avoid redundant model operation requirements, make full use of hardware resources, improve the efficiency and accuracy of model applications, and improve the user experience.
Smart Images
Figure CN119649131A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and more specifically, to a method, device, equipment and readable storage medium for generating a multi-model operation strategy. Background Art
[0002] In today's era of rapid development of digitalization and intelligence, artificial intelligence technology can be applied to all aspects of life. Various smart devices often need to run multiple models with different functions to meet diverse application needs. However, the limited hardware resources of smart devices themselves lead to hardware resource competition problems when deploying multiple models, which further leads to reduced model operation efficiency and poor user experience. Therefore, how to make full use of limited hardware resources to run various models to improve user experience has become the focus of people's attention. Summary of the invention
[0003] In view of this, the present application provides a multi-model operation strategy generation method, device, equipment and readable storage medium, which are used to solve the shortcomings of the prior art that multiple model resource competition leads to poor model operation efficiency.
[0004] In order to achieve the above objectives, the proposed solution is as follows:
[0005] A multi-model operation strategy generation method, comprising:
[0006] In response to the model scheduling operation, one or more target video frames to be processed are determined;
[0007] Perform model matching on each target video frame to determine all target models corresponding to the target video frame;
[0008] Based on the current computing resources, the computing cost of each target model and the dependencies between each target model, a strategy for scheduling each target model is generated.
[0009] Optionally, the response model scheduling operation determines more than one target video frame to be processed, including:
[0010] In response to the model scheduling operation, video frames are extracted for all video streams that need to be processed, and the extracted video frames are filtered to obtain one or more target video frames after the filtering.
[0011] Optionally, performing model matching on each target video frame to determine all target models corresponding to the target video frame includes:
[0012] The trained classification model is used to evaluate the matching degree between each target video frame and different models, and the model whose matching degree exceeds the preset matching threshold is used as the target model of the corresponding target video frame.
[0013] Optionally, the training process of the classification model includes:
[0014] Obtain multiple training video frames, where the annotation label of each training video frame is used to indicate all training models corresponding to the training video frame;
[0015] Using each training video frame in turn to train the original classification model to obtain a multi-model matching score predicted by the classification model;
[0016] The loss value is calculated based on the multi-model matching scores and annotation labels corresponding to the same training video frame, and the parameters of the original classification model are adjusted with reference to the loss value until the preset training stop condition is met to obtain a trained classification model.
[0017] Optionally, the strategy for scheduling each target model is generated by combining the current computing resources, the computing cost of each target model and the dependency relationship between each target model, including:
[0018] Determine the frequency of running each target model;
[0019] Based on the running frequency, computing cost and dependency of each target model, combined with the current computing resources, a strategy scheme is generated for scheduling each target model in different time periods.
[0020] Optionally, the generating of a strategy for scheduling each target model in different time periods based on the running frequency, computing cost and dependency of each target model in combination with current computing resources includes:
[0021] Using the trained scheduling model, according to the running frequency, computing cost and dependency of each target model, combined with the current computing resources, a strategy for scheduling each target model in different time periods is generated.
[0022] Optionally, the training process of the scheduling model includes:
[0023] Obtain several model scheduling training samples, each of which includes a model scheduling plan and the training computing resources corresponding to the model scheduling plan, the training operation frequency of the scheduling model, the training computing cost, and the training dependency relationship between the scheduling models. Each model scheduling plan records the running time period of the corresponding scheduling model;
[0024] The original scheduling model is trained using each model scheduling training sample in turn until the original scheduling model meets the preset conditions, thereby obtaining a trained scheduling model.
[0025] A multi-model operation strategy generating device, comprising:
[0026] A determination module, for responding to the model scheduling operation and determining one or more target video frames to be processed;
[0027] A matching module is used to perform model matching on each target video frame to determine all target models corresponding to the target video frame;
[0028] The generation module is used to generate a strategy for scheduling each target model by combining the current computing resources, the computing cost of each target model, and the dependencies between each target model.
[0029] A multi-model operation strategy generation device includes a memory and a processor;
[0030] The memory is used to store programs;
[0031] The processor is used to execute the program to implement the various steps of the above-mentioned multi-model operation strategy generation method.
[0032] A readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, each step of the above-mentioned multi-model operation strategy generation method is implemented.
[0033] It can be seen from the above technical solutions that the multi-model operation strategy generation method provided by the present application can respond to the model scheduling operation and determine one or more target video frames to be processed; perform model matching on each target video frame to determine all target models corresponding to the target video frame; based on this, the present application can screen the model corresponding to the target video frame from multiple models of the device, determine the target model with operation requirements by whether it corresponds to the target video frame, remove redundant model operation requirements, and avoid wasting hardware resources; then, the current computing power resources, the computing cost of each target model and the dependency relationship between each target model can be combined to generate a strategy for scheduling each target model; based on this, the present application can generate a strategy from multiple angles such as computing power resources, model dependencies and model operation costs. Through the analysis of dependencies, it is possible to avoid the situation in which the dependent items in the strategy are run before the dependent items, further avoiding the waste of computing power resources, and by combining the current computing power resources and model operation costs, determine the models that can be run simultaneously, and further make full use of hardware resources. It can be seen that the present application can use the target video frames to screen the target models that have running requirements, and comprehensively consider the computing resources, model running costs and model dependencies to generate a scheduling model strategy, make full use of limited computing resources, achieve high efficiency and accuracy of model application, and improve the processing effect and efficiency in related scenarios, thereby further improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0035] Figure 1 A flow chart of a method for generating a multi-model operation strategy disclosed in an embodiment of the present application;
[0036] Figure 2 A structural block diagram of a multi-model operation strategy generating device disclosed in an embodiment of the present application;
[0037] Figure 3 This is a hardware structure block diagram of a multi-model operation strategy generation device disclosed in an embodiment of the present application. DETAILED DESCRIPTION
[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0039] The embodiment of the present application provides a multi-model operation strategy generation method, which can be applied to various artificial intelligence systems or model intelligent management systems, and can also be applied to various computer terminals or intelligent terminals. The execution subject can be a processor or server of the computer terminal or intelligent terminal. The method flow chart of the method is as follows: Figure 1 As shown, specifically including:
[0040] Step S1, in response to a model scheduling operation, determining one or more target video frames to be processed.
[0041] Specifically, it can respond to any operation for scheduling the model; for example, it can respond to the operation of calling the model to analyze the video stream, and it can also respond to the operation of calling the model to analyze the monitoring image.
[0042] The target video frame that needs to be processed by the model can be determined.
[0043] Among them, the target video frame can be subjected to pre-processing operations such as screening and sharpening.
[0044] In response to the model scheduling operation, one or more target video frames can be extracted from the video frame pool.
[0045] Step S2: perform model matching on each target video frame to determine all target models corresponding to the target video frame.
[0046] Specifically, the target model may be an image processing model that can be used to parse any one or more target video frames.
[0047] Each target video frame may correspond to zero or more target models.
[0048] Each target model may correspond to one or more target video frames.
[0049] For each target video frame, a model that can parse the target video frame may be used as a target model.
[0050] A model that can parse any one or more target video frames can be used as a target model.
[0051] Model matching can be performed on each target video frame in a variety of ways.
[0052] For example, each target video frame may be input into a trained classification model in turn, and the trained classification model may be used to perform model matching on each target video frame.
[0053] It is also possible to determine the type of each target video frame and determine whether there is a model of the corresponding type. If so, a matching relationship is established between the target video frames and the models of the same type.
[0054] Step S3: Generate a strategy for scheduling each target model based on the current computing resources, the computing cost of each target model, and the dependencies between each target model.
[0055] Specifically, the current computing power resources may be hardware resources such as memory access capability, CPU performance, and GPU computing power.
[0056] Each computational cost can be used to indicate the computing power, number of parameters, running time required to parse video frames, and other resources required for model operation of the corresponding target model.
[0057] The dependency relationship between target models can be used to indicate the mutually exploitable information between target models.
[0058] There are many ways to generate a strategy for scheduling each target model by combining the current computing resources, the computing cost of each target model, and the dependencies between each target model.
[0059] For example, a trained scheduling model can be obtained, and the current computing resources, the computing costs of each target model, and the dependencies between the target models can be input into the scheduling model to obtain the strategy plan output by the scheduling model.
[0060] The current computing resources can also be quantified to obtain quantified resources; based on the computational cost of each target model, the running time and resource consumption of each target model can be calculated; the running priority of each target model can be determined according to the dependency relationship between each target model; and a strategy plan can be generated based on the quantified resources, the running time of each target model, the resource consumption of each target model and the running priority of each target model.
[0061] It can be seen from the above technical scheme that the multi-model operation strategy generation method provided by the embodiment of the present application can respond to the model scheduling operation and determine one or more target video frames to be processed; perform model matching on each target video frame to determine all target models corresponding to the target video frame; based on this, the present application can screen the model corresponding to the target video frame from multiple models of the device, determine the target model with operation requirements by whether it corresponds to the target video frame, remove redundant model operation requirements, and avoid wasting hardware resources; then, the current computing power resources, the computing cost of each target model and the dependency relationship between each target model can be combined to generate a strategy for scheduling each target model; based on this, the present application can generate a strategy from multiple angles such as computing power resources, model dependencies and model operation costs. Through the analysis of dependencies, it is avoided that the dependent items in the strategy are run before the dependent items, further avoiding the waste of computing power resources, and by combining the current computing power resources and model operation costs, the models that can be run simultaneously are determined to further make full use of hardware resources. It can be seen that the present application can use the target video frames to screen the target models that have running requirements, and comprehensively consider the computing resources, model running costs and model dependencies to generate a scheduling model strategy, make full use of limited computing resources, achieve high efficiency and accuracy of model application, and improve the processing effect and efficiency in related scenarios, thereby further improving the user experience.
[0062] In some embodiments of the present application, the process of step S1, responding to the model scheduling operation and determining one or more target video frames to be processed is described in detail, and the steps are as follows:
[0063] S10, in response to the model scheduling operation, extracting video frames from all video streams that need to be processed, screening the extracted video frames, and obtaining one or more target video frames after screening.
[0064] Specifically, in response to the model scheduling operation, the video frames can be extracted for each video stream that needs to be processed by the model, and each video frame obtained after the extraction can be screened to remove blurred frames, low-quality frames and duplicate frames, and the video frames obtained after screening can be used as target video frames.
[0065] Among them, each video frame obtained after extraction can be screened in a variety of ways.
[0066] For example, features can be extracted from each video frame, and based on the extracted features, a multi-dimensional contribution value evaluation can be performed on each video frame; the sum of the multi-dimensional contribution values corresponding to the same video frame is calculated to obtain the total score of each video frame, and video frames whose total scores are lower than a preset score threshold are removed.
[0067] Among them, the multi-dimensional contribution value evaluation can include any of the following: feature information amount, fuzziness, clarity and feature similarity, etc.
[0068] For another example, a screening model may be used to screen each video frame; wherein the screening model is trained using a plurality of training frames labeled with screening results.
[0069] It can be seen from the above technical solution that this embodiment provides an optional method for determining the target video frame. Through the above method, the video frames extracted from the video stream can be screened, and the screened video frames can be used as target video frames. Through the above method, blurred frames, low-quality frames and duplicate frames can be removed, and the processing amount of each target model can be simplified. There is no need to input the video stream into each target model, and each video frame in the video stream is processed using the target model. Each target model only needs to process the matching and screened target video frames, which further reduces the operating pressure of the target model and further reduces the operating requirements of the target model, thereby reducing the hardware resource pressure in the process of intelligent analysis of the video stream.
[0070] In some embodiments of the present application, step S2, performing model matching on each target video frame to determine all target models corresponding to the target video frame, is described in detail, and the steps are as follows:
[0071] S20. Using the trained classification model, evaluate the matching degree between each target video frame and different models, and use the model whose matching degree exceeds a preset matching threshold as the target model of the corresponding target video frame.
[0072] Specifically, it can be determined whether a trained classification model exists;
[0073] If not, the original classification model can be trained to obtain a trained classification model;
[0074] If so, the trained classification model can be directly retrieved.
[0075] Each target video frame is input into a trained classification model, feature information of each target video frame is extracted using the classification model, and matching results of each target video frame are predicted based on the feature information of each target video frame.
[0076] Each matching result can be used to indicate the matching degree between the corresponding target video frame and different models.
[0077] A model whose matching degree exceeds a preset matching threshold may be selected from various models as a target model corresponding to the target video frame.
[0078] The matching threshold may be determined in advance based on the parameter values of each model.
[0079] It can be seen from the above technical solution that this embodiment provides an optional method for model matching each target video frame and determining all target models corresponding to the target video frame. Through the above method, the model matching of the target video frame can be further achieved by training the classification model. Through the above method, the matching process can be unified and standardized with the help of the classification model to ensure the uniformity of the matching standard and avoid the situation where manual classification causes matching errors due to differences in subjective judgment standards; at the same time, the trained classification model can learn the complex mapping relationship between the feature information of each target video frame and each model. With the help of the trained classification model, it is more accurate to identify the subtle feature differences in the video frame, further improving the reliability and accuracy of the matching results.
[0080] At the same time, the classification model has efficient computing power and can perform model matching on massive video frames in a short time, thus accelerating the efficiency of multi-model operation strategy generation. The classification model has high matching efficiency and can perform model matching on target video frames in real time, meeting the real-time generation of strategy solutions.
[0081] In some embodiments of the present application, considering that the classification model used for model matching is trained, a training process of the classification model can be added. Next, the training process of the classification model will be described in detail, and the steps are as follows:
[0082] S21. Acquire multiple training video frames, where the annotation label of each training video frame is used to indicate all training models corresponding to the training video frame.
[0083] Specifically, a plurality of training video frames including annotated labels may be constructed, wherein each annotated label may record a training model identifier matched with the corresponding training video frame.
[0084] Each training video frame can be captured from different video streams.
[0085] The annotation labels of different training video frames may be the same or different, and the annotation labels of different training video frames may record the same training model identifier.
[0086] It should be noted that the training model is an image processing model corresponding to the training video frame, and the target model is an image processing model corresponding to the target video frame. The same image processing model can be a training model or a target model. For example, when training a classification model, the image processing model can be a training model, and in the process of generating a strategy solution, the image processing model can be a target model.
[0087] The first model set is a subset of the second model set, wherein the first model set is composed of all target models, and the second model set is composed of all training models of each training video frame.
[0088] The third model set is a subset of the first model set, wherein the third model set is composed of all training models corresponding to any training video frame.
[0089] S22. Use each training video frame to train the original classification model in turn to obtain a multi-model matching score predicted by the classification model.
[0090] Specifically, each training video frame may be used in turn to train the original classification model, and a multi-model matching score corresponding to each training video frame predicted by the classification model may be obtained.
[0091] S23. Calculate the loss value based on the multi-model matching scores and annotation labels corresponding to the same training video frame, and adjust the parameters of the original classification model with reference to the loss value until the preset training stop condition is met, thereby obtaining a trained classification model.
[0092] Specifically, the similarity between the multi-model matching score of each training video frame and its corresponding annotation label can be calculated, and the parameters of the original classification model can be adjusted with reference to the similarity until multiple consecutive similarities exceed a preset similarity threshold, thereby obtaining a trained classification model.
[0093] It can be seen from the above technical solution that this embodiment provides an optional method for training a classification model. Through the above method, multiple training video frames are integrated to train the classification model, so that the classification model learns the mapping rules between various feature data in the video frames and the training model, thereby further avoiding subjective judgment errors, reducing manual input, and further improving the accuracy and reliability of the classification model.
[0094] In some embodiments of the present application, step S3, combining the current computing resources, the computing costs of each target model and the dependencies between each target model, to generate a strategy for scheduling each target model is described in detail, and the steps are as follows:
[0095] S30: Determine the running frequency of each target model.
[0096] Specifically, the calling frequency of each target model in different time periods or different business scenarios can be determined.
[0097] S31. Based on the running frequency, computing cost and dependency of each target model and combined with the current computing resources, generate a strategy for scheduling each target model in different time periods.
[0098] Specifically, we can comprehensively consider the calling frequency, computing resource requirements, operation priority and equipment hardware resources of different target models in different time periods or different business scenarios, and build a strategy for scheduling each target model in different time periods.
[0099] It can be seen from the above technical solution that this embodiment provides an optional method for generating a strategy scheme for scheduling each target model. Through the above method, the model running frequency, model calculation cost, model dependency and computing power resources can be comprehensively considered to construct a strategy scheme. Therefore, dynamic allocation of resources can be achieved to avoid idle resources while giving priority to dependent target models, optimizing the execution order of target models and improving resource utilization.
[0100] In some embodiments of the present application, step S31, based on the running frequency, computing cost and dependency of each target model, combined with the current computing resources, generates a strategy for scheduling each target model in different time periods, and the steps are as follows:
[0101] S310. Using the trained scheduling model, according to the running frequency, computing cost and dependency of each target model, combined with the current computing resources, generate a strategy for scheduling each target model in different time periods.
[0102] Specifically, the running frequency, computing cost, dependency and current computing resources of each target model can be input into the trained scheduling model to obtain the strategy plan output by the scheduling model.
[0103] The strategy plan may record the operation time period of each target model.
[0104] Among them, one or more target models can be run in the same time period.
[0105] The same target model can be run at different time periods.
[0106] The total running time of different target models can be the same or different.
[0107] The runtime of the target model may be proportional to the number and computational cost of the corresponding target video frames.
[0108] Different target models can be scheduled in different time periods based on strategic plans.
[0109] It can be seen from the above technical solution that this embodiment provides a method of integrating the running frequency, computing cost, dependency and current computing resources of each target model to generate an optional method for scheduling each target model in different time periods. The above method can improve the accuracy of resource matching and achieve optimal resource utilization.
[0110] In some embodiments of the present application, considering that the scheduling model used to generate the policy solution is trained, the training process of the scheduling model can be added. Next, the training process of the scheduling model will be described in detail, and the steps are as follows:
[0111] S31. Obtain several model scheduling training samples, each model scheduling training sample includes a model scheduling plan and the training computing resources corresponding to the model scheduling plan, the training operation frequency of the scheduling model, the training computing cost and the training dependency relationship between the scheduling models. Each model scheduling plan records the running time period of the corresponding scheduling model.
[0112] Specifically, multiple model scheduling samples may be constructed.
[0113] Among them, each model scheduling training sample can record the training computing resources, the training operation frequency of the scheduling model, the training computing cost, the training dependency relationship between the scheduling models, and the model scheduling plan matching the above information.
[0114] Each model scheduling plan may record the running time period of the corresponding scheduling model and the running sequence of each scheduling model.
[0115] S32. Use each model scheduling training sample in turn to train the original scheduling model until the original scheduling model meets the preset conditions, thereby obtaining a trained scheduling model.
[0116] Specifically, each model scheduling training sample is input into the original scheduling model in turn to obtain the training scheduling solution predicted by the original scheduling model;
[0117] Calculate the loss value of the original scheduling model based on the model scheduling scheme and the training scheduling scheme corresponding to the same model scheduling training sample;
[0118] Referring to the loss value of the original scheduling model, the parameters of the original scheduling model are adjusted until the original scheduling model converges to obtain a trained scheduling model.
[0119] It can be seen from the above technical solution that this embodiment provides an optional method for training a scheduling model. Through the above method, the scheduling model can be trained based on a large number of model scheduling training samples, ensuring the rationality of the scheduling scheme generated by the scheduling model, ensuring the orderly execution of each target model, and making full use of hardware resources.
[0120] Next, we will combine Figure 2 The multi-model operation strategy generation device provided in the present application is introduced in detail. The multi-model operation strategy generation device provided below can be compared with the multi-model operation strategy generation method provided above.
[0121] See also Figure 2 It can be found that the multi-model operation strategy generation device may include:
[0122] A determination module 10, for determining one or more target video frames to be processed in response to the model scheduling operation;
[0123] A matching module 20 is used to perform model matching on each target video frame to determine all target models corresponding to the target video frame;
[0124] The generation module 30 is used to generate a strategy plan for scheduling each target model by combining the current computing resources, the computing cost of each target model and the dependency relationship between each target model.
[0125] Further, the determination module may include:
[0126] The video frame extraction unit is used to respond to the model scheduling operation, extract video frames from all video streams that need to be processed, filter the extracted video frames, and obtain one or more target video frames after filtering.
[0127] Furthermore, the matching module may include:
[0128] The matching evaluation unit is used to evaluate the matching degree between each target video frame and different models by using the trained classification model, and take the model whose matching degree exceeds a preset matching threshold as the target model of the corresponding target video frame.
[0129] Furthermore, the multi-model operation strategy generating device may also include:
[0130] A training video frame acquisition unit, used to acquire multiple training video frames, wherein the annotation label of each training video frame is used to indicate all training models corresponding to the training video frame;
[0131] A multi-model matching score prediction unit, used to train the original classification model using each training video frame in turn to obtain a multi-model matching score predicted by the classification model;
[0132] The loss value calculation unit is used to calculate the loss value based on the multi-model matching scores and annotation labels corresponding to the same training video frame, and adjust the parameters of the original classification model with reference to the loss value until the preset training stop condition is met, thereby obtaining a trained classification model.
[0133] Furthermore, the generation module may include:
[0134] An operation frequency determination unit, used to determine the operation frequency of each target model;
[0135] The strategy generation unit is used to generate a strategy for scheduling each target model in different time periods based on the running frequency, computing cost and dependency of each target model and the current computing resources.
[0136] Furthermore, the strategy scheme generating unit may include:
[0137] The scheduling model utilization component is used to utilize the trained scheduling model to generate a strategy for scheduling each target model in different time periods based on the running frequency, computing cost and dependency of each target model and the current computing resources.
[0138] Furthermore, the multi-model operation strategy generating device may also include:
[0139] The scheduling training sample acquisition component is used to obtain a number of model scheduling training samples. Each model scheduling training sample includes a model scheduling plan and the training computing resources corresponding to the model scheduling plan, the training operation frequency of the scheduling model, the training computing cost, and the training dependency relationship between the scheduling models. Each model scheduling plan records the operation time period of the corresponding scheduling model.
[0140] The scheduling model training component is used to use each model scheduling training sample in turn to train the original scheduling model until the original scheduling model meets the preset conditions, thereby obtaining a trained scheduling model.
[0141] The multi-model operation strategy generation device provided in the embodiment of the present application can be applied to multi-model operation strategy generation devices, such as PC terminals, cloud platforms, servers and server clusters. Figure 3 The hardware structure diagram of the device is shown in Figure 3 ,The hardware structure of the multi-model operation strategy generation device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;
[0142] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;
[0143] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;
[0144] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory;
[0145] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:
[0146] In response to the model scheduling operation, one or more target video frames to be processed are determined;
[0147] Perform model matching on each target video frame to determine all target models corresponding to the target video frame;
[0148] Based on the current computing resources, the computing cost of each target model and the dependencies between each target model, a strategy for scheduling each target model is generated.
[0149] Optionally, the detailed functions and extended functions of the program may refer to the above description.
[0150] The embodiment of the present application further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:
[0151] In response to the model scheduling operation, one or more target video frames to be processed are determined;
[0152] Perform model matching on each target video frame to determine all target models corresponding to the target video frame;
[0153] Based on the current computing resources, the computing cost of each target model and the dependencies between each target model, a strategy for scheduling each target model is generated.
[0154] Optionally, the detailed functions and extended functions of the program may refer to the above description.
[0155] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0156] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0157] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. The various embodiments of the present application can be combined with each other. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for generating a multi-model operation strategy, characterized in that: include: In response to the model scheduling operation, one or more target video frames to be processed are determined; Perform model matching on each target video frame to determine all target models corresponding to the target video frame; Based on the current computing resources, the computing cost of each target model and the dependencies between each target model, a strategy for scheduling each target model is generated.
2. The multi-model operation strategy generation method according to claim 1, characterized in that: The response model scheduling operation determines one or more target video frames to be processed, including: In response to the model scheduling operation, video frames are extracted for all video streams that need to be processed, and the extracted video frames are filtered to obtain one or more target video frames after the filtering.
3. The method for generating a multi-model operation strategy according to claim 1, characterized in that: The performing model matching on each target video frame to determine all target models corresponding to the target video frame includes: The trained classification model is used to evaluate the matching degree between each target video frame and different models, and the model whose matching degree exceeds the preset matching threshold is used as the target model of the corresponding target video frame.
4. The method for generating a multi-model operation strategy according to claim 3, characterized in that: The training process of the classification model includes: Obtain multiple training video frames, where the annotation label of each training video frame is used to indicate all training models corresponding to the training video frame; Using each training video frame in turn to train the original classification model to obtain a multi-model matching score predicted by the classification model; The loss value is calculated based on the multi-model matching scores and annotation labels corresponding to the same training video frame, and the parameters of the original classification model are adjusted with reference to the loss value until the preset training stop condition is met to obtain a trained classification model.
5. The method for generating a multi-model operation strategy according to claim 1, characterized in that: The strategy scheme for scheduling each target model is generated by combining the current computing resources, the computing cost of each target model and the dependency relationship between each target model, including: Determine the frequency of running each target model; Based on the running frequency, computing cost and dependency of each target model, combined with the current computing resources, a strategy scheme is generated for scheduling each target model in different time periods.
6. The method for generating a multi-model operation strategy according to claim 5, characterized in that: The strategy scheme for scheduling each target model in different time periods is generated based on the running frequency, computing cost and dependency of each target model and the current computing resources, including: Using the trained scheduling model, according to the running frequency, computing cost and dependency of each target model, combined with the current computing resources, a strategy for scheduling each target model in different time periods is generated.
7. The method for generating a multi-model operation strategy according to claim 6, characterized in that: The training process of the scheduling model includes: Obtain several model scheduling training samples, each of which includes a model scheduling plan and the training computing resources corresponding to the model scheduling plan, the training operation frequency of the scheduling model, the training computing cost, and the training dependency relationship between the scheduling models. Each model scheduling plan records the running time period of the corresponding scheduling model; The original scheduling model is trained using each model scheduling training sample in turn until the original scheduling model meets the preset conditions, thereby obtaining a trained scheduling model.
8. A multi-model operation strategy generation device, characterized in that: include: A determination module, for responding to the model scheduling operation and determining one or more target video frames to be processed; A matching module is used to perform model matching on each target video frame to determine all target models corresponding to the target video frame; The generation module is used to generate a strategy plan for scheduling each target model by combining the current computing resources, the computing cost of each target model and the dependencies between each target model.
9. A multi-model operation strategy generation device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the multi-model operation strategy generation method as described in any one of claims 1-7.
10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the multi-model operation strategy generation method as described in any one of claims 1 to 7 is implemented.