Model call thread scheduling method and device

Abstract

The invention discloses a scheduling method and device for a model calling thread, and relates to the technical field of servers. The method comprises the steps of obtaining scheduling configuration information corresponding to a plurality of tasks in a task list; according to the scheduling configuration information corresponding to the multiple tasks, image list information of model calling threads corresponding to the multiple models is generated, and the image list information comprises an image of at least one video stream and a thread scheduling sequence corresponding to the image; starting model calling threads corresponding to the plurality of models, and obtaining a to-be-processed target image and a thread scheduling sequence corresponding to the target image from image list information corresponding to the model calling threads; and according to the thread scheduling sequence corresponding to the target image, calling a model calling thread of the at least one model to process the target image, and obtaining an image processing result corresponding to the target image. The method can improve the task processing efficiency of the server.

Description

Technical Field

[0001] This application relates to the field of server technology, and in particular to a method and device for scheduling model call threads. Background Technology

[0002] Currently, artificial intelligence (AI) models are increasingly widely used in fields such as video analysis, image recognition, and intelligent security. Servers often need to process multiple high-resolution video streams simultaneously and call multiple AI models to perform real-time reasoning and analysis on the video content.

[0003] For example, in intelligent security systems, front-end cameras may need to identify people using facial detection models, determine abnormal behavior using behavior analysis models, and extract vehicle information using license plate recognition models. In this case, the model calling threads of multiple AI models need to be called in series or in parallel according to the task logic.

[0004] However, in the task processing of related technologies, due to the existence of a global interpreter lock mechanism, the model calling threads of multiple AI models cannot be executed in parallel, resulting in low task processing efficiency and server resource constraints and high response latency in high-concurrency video stream processing scenarios. Summary of the Invention

[0005] This application provides a method and device for scheduling model call threads, so as to at least solve the problems of scarce computing resources and high response latency in related technologies.

[0006] On the one hand, this application provides a scheduling method for model invocation threads, including:

[0007] Obtain the scheduling configuration information corresponding to each of the multiple tasks in the task list. The scheduling configuration information includes the video stream identifier of at least one video stream that needs to be processed, the model identifier of at least one model that needs to be called, and the scheduling order of at least one model.

[0008] Based on the scheduling configuration information corresponding to each of the multiple tasks, generate image list information for the model calling threads corresponding to each of the multiple models. The image list information includes images from at least one video stream and the thread scheduling order corresponding to the images.

[0009] Start the model calling thread corresponding to each of the multiple models, and obtain the target image to be processed and the thread scheduling order corresponding to the target image from the image list information corresponding to the model calling thread;

[0010] Based on the thread scheduling order corresponding to the target image, at least one model's model calling thread is invoked to process the target image and obtain the image processing result corresponding to the target image.

[0011] On the other hand, this application provides a scheduling device for model calling threads, including:

[0012] The first acquisition unit is used to acquire the scheduling configuration information corresponding to each of the multiple tasks in the task list. The scheduling configuration information includes the video stream identifier of at least one video stream that needs to be processed, the model identifier of at least one model that needs to be called, and the scheduling order of at least one model.

[0013] The generation unit is used to generate image list information for the model calling threads corresponding to multiple models based on the scheduling configuration information corresponding to multiple tasks. The image list information includes images from at least one video stream and the thread scheduling order corresponding to the images.

[0014] The second acquisition unit is used to start the model calling thread corresponding to each of the multiple models, and to obtain the target image to be processed and the thread scheduling order corresponding to the target image from the image list information corresponding to the model calling thread.

[0015] The calling unit is used to call at least one model calling thread to process the target image according to the thread scheduling order corresponding to the target image, and obtain the image processing result corresponding to the target image.

[0016] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the thread scheduling method of any of the above models when executing the computer program.

[0017] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the scheduling method for calling threads in any of the above models.

[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of calling the thread scheduling method of any of the above models.

[0019] This application provides a method and device for scheduling model invocation threads. The method includes: obtaining scheduling configuration information corresponding to multiple tasks in a task list, the scheduling configuration information including video stream identifiers of at least one video stream to be processed, model identifiers of at least one model to be invoked, and scheduling order of at least one model; generating image list information for model invocation threads corresponding to multiple models based on the scheduling configuration information corresponding to the multiple tasks, the image list information including images of at least one video stream and the thread scheduling order corresponding to the images; starting model invocation threads corresponding to multiple models, obtaining target images to be processed and the thread scheduling order corresponding to the target images from the image list information corresponding to the model invocation threads; and invoking the model invocation threads of at least one model to process the target images according to the thread scheduling order corresponding to the target images, thereby obtaining the image processing result corresponding to the target images. In the embodiments of this application, by utilizing the image list information of the model invocation threads to achieve dynamic binding between video streams and model invocation threads, the overall scheduling task of multiple models is transformed into the scheduling task of a single model invocation thread. Through the parallel execution of model invocation threads, the parallel execution of serial and parallel tasks is achieved, thereby meeting the algorithm requirements of different tasks and improving the efficiency of task processing. Attached Figure Description

[0020] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 The flow of the model call thread scheduling method provided in the embodiments of this application Figure 1 ;

[0022] Figure 2 A schematic diagram of the scheduling method for model invocation threads provided in the embodiments of this application. Figure 1 ;

[0023] Figure 3 A schematic diagram of the scheduling method for model invocation threads provided in the embodiments of this application. Figure 2 ;

[0024] Figure 4 A schematic diagram of the scheduling method for model invocation threads provided in the embodiments of this application. Figure 3 ;

[0025] Figure 5 A schematic diagram of the structure of the scheduling device for the model calling thread provided in the embodiments of this application;

[0026] Figure 6A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0028] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0029] Currently, artificial intelligence (AI) models are increasingly widely used in fields such as video analytics, image recognition, and smart security. Servers often need to process multiple high-resolution video streams simultaneously and call multiple AI models to perform real-time reasoning and analysis on the video content. For example, in a smart security system, front-end cameras may need to use face detection models to identify people, use behavior analysis models to determine abnormal behavior, and use license plate recognition models to extract vehicle information.

[0030] In real-world production scenarios, on-site deployments require access to a large amount of video stream data and the inference and computation of numerous AI models. When a single service receives dozens or even hundreds of video streams, each requiring processing by different AI algorithms, the scheduling of video and model algorithms becomes crucial. This requires ensuring smooth business operations while making rational use of hardware resources and avoiding waste.

[0031] However, due to the global interpreter lock mechanism in traditional calls, multithreading cannot truly achieve parallel execution of multiple threads when processing intensive tasks, resulting in server resource constraints and high response latency in high-concurrency video stream processing scenarios.

[0032] Therefore, how to achieve parallel execution of multiple threads to improve task processing efficiency is a technical problem that urgently needs to be solved.

[0033] To address the aforementioned technical issues, this application proposes a scheduling method for model call threads. By flexibly connecting and calling AI models and video streams, it achieves the fusion of serial and parallel AI models, which can maximize the utilization of existing server resources and accelerate task processing speed.

[0034] Optionally, the specific steps include: First, obtaining the scheduling configuration information corresponding to each of the multiple tasks in the task list; based on the scheduling configuration information corresponding to each of the multiple tasks, generating image list information for each of the multiple models' corresponding model calling threads, wherein the image list information includes images from at least one video stream and the thread scheduling order corresponding to the images. Then, starting the model calling threads corresponding to each of the multiple models, obtaining the target image to be processed and the thread scheduling order corresponding to the target image from the image list information corresponding to the model calling thread. Finally, based on the thread scheduling order corresponding to the target image, calling the model calling thread of at least one model to process the target image, obtaining the image processing result corresponding to the target image.

[0035] In this embodiment, by utilizing the image list information of the model calling thread to achieve dynamic binding between the video stream and the model calling thread, the overall scheduling task of multiple models is transformed into the scheduling task of a single model calling thread. Through the parallel execution of the model calling thread, the parallel execution of serial and parallel tasks is achieved, thereby meeting the algorithm requirements of different tasks and improving the efficiency of task processing.

[0036] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] The specific application environment architecture or specific hardware architecture on which the execution of the model's call thread scheduling method depends is described here.

[0038] Figure 1 The flow of the model call thread scheduling method provided in the embodiments of this application Figure 1 In this model, the entity that executes the thread scheduling method can be a server. For example... Figure 1 As shown, the scheduling methods for calling threads in this model include:

[0039] S101. Obtain the scheduling configuration information corresponding to each of the multiple tasks in the task list. The scheduling configuration information includes the video stream identifier of at least one video stream to be processed, the model identifier of at least one model to be called, and the scheduling order of at least one model.

[0040] In this embodiment of the disclosure, the task can be a task that invokes at least one AI model to perform inference analysis on a video stream. Examples include face recognition tasks and behavior analysis tasks. Optionally, the types of the above tasks can include: parallel tasks that invoke one model and serial tasks that invoke multiple models.

[0041] In some embodiments, such as Figure 2 As shown, the scheduling configuration information for each task in the task list can be obtained from the configuration file. The specific steps may include: starting the main program; reading and parsing the configuration file; and obtaining the video stream identifier list, AI model list, and task list.

[0042] Optionally, the video stream identifier list includes video stream identifiers corresponding to multiple video streams. The AI ​​model list includes model identifiers corresponding to multiple AI models. The task list includes scheduling configuration information corresponding to each of multiple tasks.

[0043] S102. Based on the scheduling configuration information corresponding to each of the multiple tasks, generate image list information for each of the multiple models corresponding to the model calling threads. The image list information includes images from at least one video stream and the thread scheduling order corresponding to the images.

[0044] In some embodiments, image list information of model calling threads corresponding to multiple models is generated based on the scheduling configuration information corresponding to each of the multiple tasks, including the following steps (1) to (3):

[0045] (1) For each task, based on the scheduling configuration information, obtain the image of the video stream that the task needs to process according to the video stream identifier of at least one video stream that needs to be processed in the scheduling configuration information.

[0046] Optionally, this step includes: obtaining the image of the video stream that the task needs to process from the correspondence between the image of the video stream and the video stream identifier of the video stream, based on the video stream identifier of at least one video stream that needs to be processed in the scheduling configuration information.

[0047] In some embodiments, the mapping between images of a video stream and video stream identifiers can be stored using a decoded image list. Optionally, the method may include:

[0048] The decoding module is initialized, and the number of decoding threads to be started is determined based on the number of video stream identifiers of at least one video stream. The decoding threads parse at least one video stream at a preset period to obtain the images of each video stream. A correspondence is created between the images of the video streams and the video stream identifiers of the video streams. The correspondence is stored in the decoded image list according to the memory address information corresponding to the video streams. The memory address information corresponding to multiple video streams is consecutive address information.

[0049] For example, such as Figure 3 As shown, during the initialization of the decoding module, the number of decoding threads can be determined based on the number of video stream identifiers in the video stream identifier list. A preset memory space is configured for each video stream, and the storage addresses of each image are consecutive, thus obtaining a list of decoded images.

[0050] It should be noted that in some embodiments, the video stream identifier can be obtained by parsing the configuration file, and the decoding module allocates a 1080*1920 memory space for each video stream to store the decoded images in real time. Specifically, the 1080*1920 memory space is configured to store one image corresponding to the video stream, enabling real-time overwriting of the decoded images.

[0051] In this embodiment, the decoding thread writes the image corresponding to each video stream to its allocated memory space after decoding, and the decoded images are overwritten in real time. The model calling thread can read from the memory space in real time. Thus, each model calling thread, according to a set reading interval, can achieve automatic frame extraction of video images, further improving task processing efficiency.

[0052] (2) For the images of the video stream that the task needs to process, determine the thread scheduling order corresponding to the image based on the scheduling order of at least one model in the scheduling configuration information corresponding to the task.

[0053] (3) Based on the images of the video stream that each task needs to process and the thread scheduling order corresponding to the images, generate image list information of the model calling thread corresponding to each of the multiple models.

[0054] In some embodiments, this step may include: obtaining the model calling thread corresponding to the at least one model that the task needs to call based on the model identifier of the at least one model that needs to be called in the scheduling configuration information corresponding to each task; determining the first target model calling thread to be called from the model calling threads corresponding to the at least one model based on the thread scheduling order corresponding to the image; and generating an image list information corresponding to the target model calling thread based on the images of the video stream that the task needs to process and the thread scheduling order corresponding to the images.

[0055] Optionally, based on the images of the video stream to be processed by the task and the thread scheduling order corresponding to the images, image list information corresponding to the target model calling thread is generated, including: determining the task type corresponding to the task based on the images of the video stream to be processed by the task and the thread scheduling order corresponding to the images; if the task type is a parallel task where one model needs to be called, then the images of the video stream to be processed by the task and the thread scheduling order corresponding to the images are added to the first image list to obtain the first image list information corresponding to the target model calling thread; if the task type is a serial task where multiple models need to be called, then the images of the video stream to be processed by the task and the thread scheduling order corresponding to the images are added to the second image list to obtain the second image list information corresponding to the target model calling thread.

[0056] The first image list information is used to store the images corresponding to parallel tasks, and the second image list information is used to store the images corresponding to serial tasks.

[0057] In this embodiment of the disclosure, by inserting the images of the video stream that the task needs to process into the parallel image list or the serial image list according to the task type corresponding to the task, the fusion of serial and parallel tasks is realized. Then, the images of the video stream that the serial and parallel tasks need to process can be executed in parallel by calling threads through the model, thus improving the efficiency of task processing.

[0058] In some embodiments, the model list is read to generate a thread function, a parallel input image list, a concatenated input image list, and an image queue for each model. The thread function is also the model scheduling thread, the parallel input image list is the first image list information, and the concatenated input image list is the second image list information. The image queue is a third image list information used to store intermediate images corresponding to the concatenated task.

[0059] Optionally, such as Figure 4 As shown, the configuration file model information is obtained, a thread is started for each model to call model inference, the algorithm information is read according to the video input of each algorithm and the model information called (single model or multiple concatenated models), the input image list and output queue in each model thread are filled, and the image list information of the model calling thread corresponding to multiple models is generated.

[0060] For example, such as Figure 3 As shown, this step may include:

[0061] (1) Read the task list and obtain the video stream identifier and algorithm model information of at least one video stream that needs to be processed for each task.

[0062] Optionally, the video stream identifier of the video stream can be represented as a video ID. The algorithm model information includes the model identifier of at least one model to be invoked and the scheduling order of at least one model.

[0063] (2) Determine whether each model is connected in series. If it is, jump to step (3); otherwise, jump to step (5).

[0064] (3) Obtain the image corresponding to the video stream identifier and insert the image one by one into the corresponding concatenated input image list of the first model concatenated by the task.

[0065] (4) Based on the model IDs of the tasks, insert them into the thread scheduling order corresponding to each image in the list of connected input images in order, and jump to step (6).

[0066] (5) Obtain the image corresponding to the video stream identifier and insert the image into the parallel input image list one by one.

[0067] (6) Determine if all tasks have been traversed. If so, start all model thread functions; otherwise, jump to step (2).

[0068] S103. Start the model calling thread corresponding to each of the multiple models, and obtain the target image to be processed and the thread scheduling order corresponding to the target image from the image list information corresponding to the model calling thread.

[0069] In some embodiments, such as Figure 4 As shown, the scheduling image list information can be polled to obtain the target image to be processed and the thread scheduling order corresponding to the target image.

[0070] Optionally, the image list information includes a first image list for storing images corresponding to parallel tasks, a second image list for storing images corresponding to serial tasks, and a third image list for storing intermediate images corresponding to serial tasks. Accordingly, the target image to be processed and the thread scheduling order corresponding to the target image are obtained from the image list information corresponding to the model calling thread, including: polling and scheduling the first image list information, the second image list information, and the third image list information to obtain the target image to be processed and the thread scheduling order corresponding to the target image.

[0071] S104. According to the thread scheduling order corresponding to the target image, call the model calling thread of at least one model to process the target image and obtain the image processing result corresponding to the target image.

[0072] In some embodiments, according to the thread scheduling order corresponding to the target image, at least one model calling thread is invoked to process the target image and obtain the image processing result corresponding to the target image, including: if the thread scheduling order corresponding to the target image includes a model calling thread, then the model calling thread of the invoked model processes the target image and obtains the image processing result corresponding to the target image.

[0073] In other embodiments, according to the thread scheduling order corresponding to the target image, at least one model calling thread is invoked to process the target image to obtain the image processing result corresponding to the target image. This includes: if the thread scheduling order corresponding to the target image includes the thread scheduling order of multiple model calling threads, then determining the first invoked target model calling thread; invoking the target model calling thread to process the target image to obtain the intermediate image corresponding to the target image; and invoking the next model calling thread to process the intermediate image processing result according to the thread scheduling order corresponding to the target image, until the invocation of multiple model calling threads ends, thereby obtaining the intermediate image processing result corresponding to the target image.

[0074] Optionally, based on the thread scheduling order corresponding to the target image, the next model calling thread is invoked to process the intermediate image processing result, including: determining the next model calling thread based on the thread scheduling order corresponding to the target image; adding the intermediate image corresponding to the target image and the thread scheduling order to the third image list corresponding to the next model calling thread, the third image list being used to store the intermediate images of the concatenated tasks that require calling multiple models; and retrieving the intermediate image corresponding to the target image from the third image list through the next model calling thread and processing the intermediate image corresponding to the target image.

[0075] For example, such as Figure 3 As shown, this step may include:

[0076] (1) Start all model thread functions.

[0077] (2) Each model thread function polls and traverses the list of parallel input images, the list of concatenated input images, and the image queue.

[0078] (3) Obtain the current image, call the inference interface, and perform image detection.

[0079] (4) Obtain the image corresponding to the current polling, and determine whether the current image is in the parallel input image list. If yes, jump to step (5); otherwise, jump to step (6).

[0080] (5) Obtain the inference result of the model and jump to step (8).

[0081] (6) Obtain the inference result of the model and get the ID of the next inference model in the thread scheduling order (e.g., modellist list) corresponding to the current image.

[0082] (7) Determine whether the obtained next serial model ID is empty. If yes, jump to step (8); otherwise, jump to step (9).

[0083] (8) Send the inference results to the alarm thread and jump to step (4).

[0084] (9) Add the inference result image to the image queue of the model corresponding to the next concatenated model ID, and jump to step (4).

[0085] In this embodiment, the configuration file is first read to obtain video stream information, AI model information, and detailed algorithm configuration information. Then, the video decoding model is initialized, decoding each video image and updating it to the image storage list in real time. Next, the AI ​​model information is traversed, generating a thread function, a parallel image input list, a concatenated image input list, and an image input queue for each model. Finally, based on the algorithm configuration information, the parallel image input list, concatenated image input list, and image queue of the model are updated. In this way, the information for each algorithm is distributed across different AI models, and multiple models collaborate to complete the business logic required by the algorithm. Once the above operations are completed, all model threads are started. Each model thread runs independently without interference, enabling both parallel and concatenated calls to the AI ​​models.

[0086] This application provides a method for scheduling model invocation threads: obtaining scheduling configuration information for each of multiple tasks in a task list, the scheduling configuration information including the video stream identifier of at least one video stream to be processed, the model identifier of at least one model to be invoked, and the scheduling order of at least one model; generating image list information for each model invocation thread based on the scheduling configuration information for each of the multiple tasks, the image list information including images from at least one video stream and the thread scheduling order corresponding to the images; starting the model invocation threads corresponding to each of the multiple models, obtaining the target image to be processed and the thread scheduling order corresponding to the target image from the image list information corresponding to the model invocation threads; and invoking the model invocation thread of at least one model to process the target image according to the thread scheduling order corresponding to the target image, thereby obtaining the image processing result corresponding to the target image. In this embodiment, by utilizing the image list information of the model invocation threads to achieve dynamic binding between video streams and model invocation threads, the overall scheduling task of multiple models is transformed into the scheduling task of a single model invocation thread. Through the parallel execution of the model invocation threads, the parallel execution of serial and parallel tasks is achieved, thereby meeting the algorithm requirements of different tasks and improving the efficiency of task processing.

[0087] It should be noted that, in this embodiment, the video decoding module can decode different videos and update the video images to the memory image list and system shared memory in real time. The system shared memory can be accessed and used by third-party systems. The AI ​​model serial-parallel strategy scheduling module, as the core functional module of the algorithm scheduling engine, implements the scheduling, execution, and inference management of all models and videos to meet the logical requirements of different algorithms. Optionally, the algorithm scheduling engine can interact with the front-end interface through a database and back-end services, and achieve information exchange and business logic collaboration with third-party systems through system shared memory and third-party interaction models.

[0088] Figure 5 This is a schematic diagram of the structure of the scheduling device for the model calling thread provided in an embodiment of this application. Figure 5 As shown, the device includes:

[0089] The first acquisition unit 501 is used to acquire the scheduling configuration information corresponding to each of the multiple tasks in the task list. The scheduling configuration information includes the video stream identifier of at least one video stream that needs to be processed, the model identifier of at least one model that needs to be called, and the scheduling order of at least one model.

[0090] The generation unit 502 is used to generate image list information for the model calling threads corresponding to multiple models based on the scheduling configuration information corresponding to multiple tasks. The image list information includes images from at least one video stream and the thread scheduling order corresponding to the images.

[0091] The second acquisition unit 503 is used to start the model calling thread corresponding to each of the multiple models, and to obtain the target image to be processed and the thread scheduling order corresponding to the target image from the image list information corresponding to the model calling thread.

[0092] Calling unit 504 is used to call at least one model calling thread to process the target image according to the thread scheduling order corresponding to the target image, and obtain the image processing result corresponding to the target image.

[0093] In some embodiments, the generation unit 502 generates image list information for model calling threads corresponding to multiple models based on the scheduling configuration information corresponding to each of the multiple tasks. This includes: for each task, obtaining the image of the video stream to be processed by the task based on the video stream identifier of at least one video stream to be processed in the scheduling configuration information; for the image of the video stream to be processed by the task, determining the thread scheduling order corresponding to the image based on the scheduling order of at least one model in the scheduling configuration information corresponding to the task; and generating image list information for model calling threads corresponding to multiple models based on the image of the video stream to be processed by each task and the thread scheduling order corresponding to the image.

[0094] In some embodiments, the generation unit 502 generates image list information for model calling threads corresponding to multiple models based on the images of the video stream to be processed by each task and the thread scheduling order corresponding to the images. This includes: obtaining the model calling thread corresponding to at least one model to be called by the task based on the model identifier of at least one model to be called in the scheduling configuration information corresponding to each task; determining the first target model calling thread to be called from the model calling threads corresponding to at least one model based on the thread scheduling order corresponding to the images; and generating image list information corresponding to the target model calling thread based on the images of the video stream to be processed by the task and the thread scheduling order corresponding to the images.

[0095] In some embodiments, the generation unit 502 generates image list information corresponding to the target model calling thread based on the images of the video stream to be processed by the task and the thread scheduling order corresponding to the images. This includes: determining the task type corresponding to the task based on the images of the video stream to be processed by the task and the thread scheduling order corresponding to the images; if the task type is a parallel task where one model needs to be called, then adding the images of the video stream to be processed by the task and the thread scheduling order corresponding to the images to the first image list to obtain the first image list information corresponding to the target model calling thread; if the task type is a serial task where multiple models need to be called, then adding the images of the video stream to be processed by the task and the thread scheduling order corresponding to the images to the second image list to obtain the second image list information corresponding to the target model calling thread; wherein, the first image list information is used to store images corresponding to the parallel task, and the second image list information is used to store images corresponding to the serial task.

[0096] In some embodiments, the device further includes: a storage unit; the storage unit is used to initialize the decoding module, determine the number of decoding threads to be started based on the number of video stream identifiers of at least one video stream; parse at least one video stream by the decoding threads at a preset period to obtain images of each video stream; create a correspondence between the images of the video streams and the video stream identifiers of the video streams, and store the correspondence in a decoded image list based on the memory address information corresponding to the video streams; wherein the memory address information corresponding to multiple video streams is consecutive address information.

[0097] In some embodiments, the calling unit 504 calls at least one model calling thread to process the target image according to the thread scheduling order corresponding to the target image, and obtains the image processing result corresponding to the target image, including: if the thread scheduling order corresponding to the target image includes a model calling thread, then the model calling thread of the model is called to process the target image and obtain the image processing result corresponding to the target image.

[0098] In some embodiments, the calling unit 504 calls at least one model calling thread to process the target image according to the thread scheduling order corresponding to the target image, and obtains the image processing result corresponding to the target image, including: if the thread scheduling order corresponding to the target image includes the thread scheduling order of multiple model calling threads, then determining the first called target model calling thread; calling the target model calling thread to process the target image to obtain the intermediate image corresponding to the target image; and calling the next model calling thread to process the intermediate image processing result according to the thread scheduling order corresponding to the target image, until the calling of multiple model calling threads ends, and obtaining the intermediate image processing result corresponding to the target image.

[0099] In some embodiments, the calling unit 504 calls the next model calling thread to process the intermediate image processing result according to the thread scheduling order corresponding to the target image, including: determining the next model calling thread according to the thread scheduling order corresponding to the target image; adding the intermediate image corresponding to the target image and the thread scheduling order to the third image list corresponding to the next model calling thread, the third image list being used to store intermediate images of a serial task that requires calling multiple models; and obtaining the intermediate image corresponding to the target image from the third image list through the next model calling thread and processing the intermediate image corresponding to the target image.

[0100] In some embodiments, the image list information includes a first image list for storing images corresponding to parallel tasks, a second image list for storing images corresponding to serial tasks, and a third image list for storing intermediate images corresponding to serial tasks. Accordingly, the second acquisition unit 503 acquires the target image to be processed and the thread scheduling order corresponding to the target image from the image list information corresponding to the model calling thread, including: polling and scheduling the first image list information, the second image list information, and the third image list information to acquire the target image to be processed and the thread scheduling order corresponding to the target image.

[0101] This application provides a scheduling device for model calling threads. By utilizing the image list information of the model calling threads, the video stream is dynamically bound to the model calling threads, transforming the overall scheduling task of multiple models into the scheduling task of a single model calling thread. Through the parallel execution of the model calling threads, the parallel execution of serial and parallel tasks is realized, thereby meeting the algorithm requirements of different tasks and improving the efficiency of task processing.

[0102] For a description of the features of the model call thread scheduling device provided in this application, please refer to the relevant description of the model call thread scheduling method in the corresponding embodiment, which will not be repeated here.

[0103] Figure 6 A schematic diagram of the structure of the electronic device provided in this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the electronic device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus.

[0104] In the specific implementation process, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to execute the above-described model call thread scheduling method embodiment.

[0105] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0106] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0107] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0108] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0109] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the above-described embodiments of the thread scheduling method for calling any of the models.

[0110] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0111] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in the above-described model call thread scheduling method embodiments.

[0112] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model call thread scheduling method embodiments.

[0113] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] The foregoing has provided a detailed description of a method and device for scheduling model call threads provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for scheduling model-invoking threads, characterized in that, include: Obtain the scheduling configuration information corresponding to each of the multiple tasks in the task list. The scheduling configuration information includes the video stream identifier of at least one video stream that needs to be processed, the model identifier of at least one model that needs to be called, and the scheduling order of the at least one model. Based on the scheduling configuration information corresponding to each of the multiple tasks, an image list information for each model calling thread is generated, wherein the image list information includes at least one image from a video stream and the thread scheduling order corresponding to the image. Start the model calling thread corresponding to each of the multiple models, and obtain the target image to be processed and the thread scheduling order corresponding to the target image from the image list information corresponding to the model calling thread; According to the thread scheduling order corresponding to the target image, at least one model's model call thread is invoked to process the target image and obtain the image processing result corresponding to the target image.

2. The scheduling method according to claim 1, characterized in that, The step of generating image list information for each model's corresponding model calling thread based on the scheduling configuration information corresponding to each of the multiple tasks includes: For each task, based on the scheduling configuration information, the image of the video stream to be processed by the task is obtained according to the video stream identifier of at least one video stream to be processed in the scheduling configuration information. For the images in the video stream that the task needs to process, the thread scheduling order corresponding to the image is determined according to the scheduling order of at least one model in the scheduling configuration information corresponding to the task. Based on the images of the video streams that each task needs to process and the thread scheduling order corresponding to the images, generate image list information for the model calling threads corresponding to multiple models.

3. The scheduling method according to claim 2, characterized in that, The step of generating image list information for each model's corresponding model calling thread based on the images of the video streams to be processed by each task and the thread scheduling order corresponding to the images includes: Based on the model identifier of at least one model that needs to be called in the scheduling configuration information corresponding to each task, obtain the model calling thread corresponding to at least one model that needs to be called by the task; Based on the thread scheduling order corresponding to the image, determine the first target model calling thread from the model calling threads corresponding to the at least one model; Based on the images in the video stream that the task needs to process and the thread scheduling order corresponding to the images, generate the image list information corresponding to the target model calling thread.

4. The scheduling method according to claim 3, characterized in that, The step of generating the image list information corresponding to the target model calling thread based on the images of the video stream to be processed by the task and the thread scheduling order corresponding to the images includes: The task type corresponding to the task is determined based on the images of the video stream that the task needs to process and the thread scheduling order corresponding to the images; If the task type is a parallel task that requires calling a model, then the images of the video stream that the task needs to process and the thread scheduling order corresponding to the images are added to the first image list to obtain the first image list information corresponding to the target model calling thread; If the task type is a series task involving multiple models to be called, then the images of the video stream to be processed by the task and the thread scheduling order corresponding to the images are added to the second image list to obtain the second image list information corresponding to the target model calling thread; The first image list information is used to store images corresponding to parallel tasks, and the second image list information is used to store images corresponding to serial tasks.

5. The scheduling method according to claim 2, characterized in that, Before obtaining the image of the video stream to be processed by the task based on the video stream identifier of at least one video stream to be processed in the scheduling configuration information, the method further includes: Initialize the decoding module and determine the number of decoding threads to start based on the number of video stream identifiers of at least one video stream. The decoding thread parses at least one video stream at a preset period to obtain images of each video stream; A correspondence is created between the images of the video stream and the video stream identifiers of the video stream. The correspondence is then stored in the decoded image list based on the memory address information corresponding to the video stream. The memory address information corresponding to multiple video streams is consecutive address information.

6. The scheduling method according to claim 1, characterized in that, The step of calling at least one model's model invocation thread to process the target image according to the thread scheduling order corresponding to the target image, and obtaining the image processing result corresponding to the target image, includes: If the thread scheduling order corresponding to the target image includes a model calling thread of a model, then the model calling thread of the model is called to process the target image and obtain the image processing result corresponding to the target image.

7. The scheduling method according to claim 1, characterized in that, The step of calling at least one model's model invocation thread to process the target image according to the thread scheduling order corresponding to the target image, and obtaining the image processing result corresponding to the target image, includes: If the thread scheduling order corresponding to the target image includes the thread scheduling order of the model calling threads of multiple models, then the target model calling thread of the first call is determined; The target model is invoked to process the target image using a calling thread to obtain an intermediate image corresponding to the target image; According to the thread scheduling order corresponding to the target image, the next model calling thread is called to process the intermediate image processing result until the model calling threads of multiple models are finished, and the intermediate image processing result corresponding to the target image is obtained.

8. The scheduling method according to claim 7, characterized in that, The step of calling the next model calling thread to process the intermediate image processing result according to the thread scheduling order corresponding to the target image includes: The next model calling thread is determined based on the thread scheduling order corresponding to the target image; The intermediate image corresponding to the target image and the thread scheduling order are added to the third image list corresponding to the next model calling thread. The third image list is used to store the intermediate images of the serial task that needs to call multiple models. The next model call thread retrieves the intermediate image corresponding to the target image from the third image list and processes the intermediate image corresponding to the target image.

9. The scheduling method according to claim 1, characterized in that, The image list information includes a first image list for storing images corresponding to parallel tasks, a second image list for storing images corresponding to serial tasks, and a third image list for storing intermediate images corresponding to serial tasks. Accordingly, obtaining the target image to be processed and the thread scheduling order corresponding to the target image from the image list information corresponding to the model calling thread includes: The first image list information, the second image list information, and the third image list information are polled and scheduled to obtain the target image to be processed and the thread scheduling order corresponding to the target image.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the scheduling method for the model calling thread as described in any one of claims 1 to 9 when executing the computer program.

Images