A method, system, terminal and medium for supporting multitasking

Abstract

The application discloses a kind of support multitasking processing method, system, terminal and medium, method includes: obtaining video data, video data is preprocessed, and the first target image frame set and the second target image frame set of the same timestamp are obtained;Acquire prompt switching instruction and task switching instruction, and respectively determine the prompt type corresponding to prompt switching instruction and the task type corresponding to task switching instruction;First target image frame set is inferred based on prompt type and task type, and the inference result of first target image frame is obtained;According to the inference result of first target image frame and second target image frame set, task result is obtained.The application can be based on prompt switching instruction and task switching instruction to realize the switching of inference prompt type and task type, so that the detection of target object is not only limited to single task, flexible switching of task can be realized, to facilitate to meet the detection demand under complex scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and in particular to a multi-task processing method and system, a terminal and a medium. BACKGROUND

[0002] The existing target detection system is designed only for a single task or a specific category and a specific task type, and cannot be extended to different task types or flexibly switched among different tasks. Therefore, in the face of complex and variable scenes and diverse business needs, the single detection capability is insufficient to meet the growing needs of users in complex scenarios and business development. In addition, the existing target detection system basically uses video data of the same resolution for reasoning and display, and does not distinguish between reasoning and display video data. Especially when high-resolution video data is used for reasoning, it not only affects the reasoning speed, but also wastes computing resources. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a multi-task processing method, system, terminal and medium to solve the problem that the existing target detection system is designed only for a single task or a specific category and a specific task type, and cannot be extended to different task types or flexibly switched among different tasks, and also has the problems of low reasoning efficiency and waste of computing resources.

[0004] To solve the above technical problems, the technical solution adopted by the present application is as follows:

[0005] In a first aspect, the present application provides a multi-task processing method, wherein the method comprises:

[0006] Obtaining video data, pre-processing the video data to obtain a first target image frame set and a second target image frame set with the same timestamp, the first target image frame set and the second target image frame set each containing at least one target image frame;

[0007] Obtaining a prompt switching instruction and a task switching instruction, and determining a prompt type corresponding to the prompt switching instruction and a task type corresponding to the task switching instruction, respectively;

[0008] Reasoning the first target image frame set based on the prompt type and the task type to obtain a reasoning result of the first target image frame;

[0009] Obtaining a task result according to the reasoning result of the first target image frame and the second target image frame set.

[0010] In an implementation manner, the video data is preprocessed to obtain target image frames, including:

[0011] When the video data is two video streams with same video content and different resolutions, the two video streams are processed according to video timestamps of the two video streams to determine a first image frame set and a second image frame set with same timestamps respectively.

[0012] The first image frame set and the second image frame set are denoised to obtain a first target image frame set and a second target image frame set.

[0013] In an implementation manner, the two video streams are processed according to video timestamps of the two video streams to determine a first image frame set and a second image frame set with same timestamps respectively, including:

[0014] When the video timestamps of the two video streams are inconsistent, all video stream pairs of the two video streams and video timestamps corresponding to all video stream pairs are cached to obtain cache data of all video stream pairs.

[0015] All cache data pairs with same video timestamps are determined from the cache data of all video stream pairs.

[0016] A cache data pair with the latest timestamp is determined from each cache data pair in the all cache data pairs as the first image frame set and the second image frame set respectively.

[0017] In an implementation manner, the first target image frame set is inferred based on the prompt type and the task type to obtain an inference result of the first target image frame, including:

[0018] Based on the prompt type, an inference model corresponding to the prompt type is determined.

[0019] The first target image frame set is inferred by the inference model based on the task type to obtain an inference result corresponding to the task type.

[0020] In an implementation manner, based on the prompt type, an inference model corresponding to the prompt type is determined, including:

[0021] If the prompt type is a first prompt or a third prompt, the inference model is determined as a first model.

[0022] If the prompt type is a second prompt, the inference model is determined as a second model.

[0023] In an implementation manner, the first target image frame set is inferred by the inference model based on the task type, to obtain an inference result corresponding to the task type, which comprises:

[0024] If the task type is a first task type, the first target image frame set is inferred by the inference model to obtain a detection box of a target object and a confidence of the target object;

[0025] If the task type is a second task type, the first target image frame set is inferred by the inference model to obtain mask information of the target object;

[0026] If the task type is a third task type, the first target image frame set is inferred by the inference model to obtain image features of the target object, and the image features and a first task prompt are input into a third model to obtain any one or more of text information of the target object, category information of the target object, or description information of the target object;

[0027] If the task type is a fourth task type, the first target image frame set is inferred by the inference model to obtain image features of the target object, and the image features and a second task prompt are input into the third model to obtain answer information of the target object;

[0028] If the task type is a fifth task type, the first target image frame set is inferred by the inference model to obtain category information of the target object and image features of the target object, and the category information of the target object and the image features of the target object are input into a decoder to obtain a set of two-dimensional coordinates of human key points or three-dimensional coordinates of human key points.

[0029] In an implementation manner, the first target image frame set is inferred based on the prompt type and the task type, to obtain an inference result of the first target image frame, which further comprises:

[0030] The prompt of a current target image frame in the first target image frame set is obtained according to a prompt of a previous target image frame in the first target image frame set;

[0031] The current target image frame is inferred based on the prompt of the current target image frame and the task type, to obtain an inference result of the current target image frame.

[0032] In an implementation manner, the prompt of the current target image frame in the first target image frame set is obtained according to the prompt of the previous target image frame in the first target image frame set, which comprises:

[0033] If the prompt of the previous target image frame is not modified, the previously cached prompt of the previous target image frame is taken as the prompt of the current target image frame;

[0034] If the prompt of the last target image frame is modified, the modified prompt is used as the prompt of the current target image frame.

[0035] If the prompt of the last target image frame is not modified, the pre-cached prompt of the last target image frame is used as the prompt of the current target image frame.

[0036] If the prompt of the last target image frame is modified, the modified prompt is used as the prompt of the current target image frame.

[0037] In an implementation manner, the current target image frame is inferred based on the prompt of the current target image frame and the task type, to obtain an inference result of the current target image frame, and the method further includes:

[0038] If the prompt type is the third prompt, a preset general file is used as a prompt feature to infer the first target image frame set.

[0039] In an implementation manner, if the task type is the third task type, the first target image frame set obtains an image feature of a target object through the inference model, and the image feature and the first task prompt obtain any one or more of text information of the target object, category information of the target object, or description information of the target object through a third model, including:

[0040] When the task type is the third sub-task type and the prompt type is the third prompt, the first target image frame set obtains an image feature of a target object through the inference model, and the image feature and the first task prompt obtain category information of the target object through a third model.

[0041] Based on the category information of the target object, an updated target object prompt feature is obtained.

[0042] The updated target object prompt feature and the first target image frame set are used to obtain an updated image feature of the target object through the inference model.

[0043] The updated image feature of the target object and the third sub-task type are used to obtain updated category information of the target object through the third model.

[0044] In an implementation manner, the method further includes:

[0045] A preset first confidence threshold is obtained, and the first confidence threshold includes a confidence threshold of a detection frame of the target object and / or a confidence threshold of category information of the target object.

[0046] screen the bounding box of the target object and / or the category information of the target object based on the first confidence threshold.

[0047] In an implementation manner, after the bounding box of the target object and / or the category information of the target object is screened based on the first confidence threshold, the method further includes:

[0048] matching the bounding box of the target object in the current target image frame with the bounding box of the target object in the previous target image frame to obtain a matching score;

[0049] if the matching score is greater than a preset matching threshold, retaining the bounding box of the target object in the current target image frame.

[0050] In an implementation manner, after the bounding box of the target object in the current target image frame is matched with the bounding box of the target object in the previous target image frame to obtain a matching score, the method further includes:

[0051] if the matching score of the current target image frame and the previous target image frame is less than the matching threshold and the task types are the same, determining a difference value between the matching score and the matching threshold;

[0052] if the difference value is within a preset range, taking the inference result of the previous target image frame as the inference result of the current target image frame.

[0053] In an implementation manner, the task result is obtained according to the inference result of the first target image frame and the second target image frame set, including:

[0054] dimension information of each first target image frame in the first target image frame set and each second target image frame in the second target image frame set is acquired respectively;

[0055] for each first target image frame, first coordinate information of a bounding box of a target object in the first target image frame is obtained based on the first target image frame through an inference model, and second coordinate information of the bounding box of the target object in the corresponding second target image frame is determined based on the first coordinate information and the dimension information of the corresponding second target image frame;

[0056] the inference result is displayed on the corresponding second target image frame according to the second coordinate information.

[0057] In a second aspect, an embodiment of the present application further provides a multi-task processing system, wherein the system includes:

[0058] The video processing module is configured to acquire video data, pre-process the video data, and obtain a first target image frame set and a second target image frame set with the same timestamp, wherein the first target image frame set and the second target image frame set each contain at least one target image frame.

[0059] The task prompt control module is configured to acquire a prompt switching instruction and a task switching instruction, and determine a prompt type corresponding to the prompt switching instruction and a task type corresponding to the task switching instruction, respectively.

[0060] The model inference module is configured to infer the first target image frame set based on the prompt type and the task type, and obtain an inference result of the first target image frame.

[0061] The task result output module is configured to obtain a task result based on the inference result of the first target image frame and the second target image frame set.

[0062] In a third aspect, an embodiment of the present application further provides a terminal, wherein the terminal comprises a memory, a processor, and a support multi-task processing program stored in the memory and executable on the processor, and the processor implements the steps of the support multi-task processing method of any one of the above solutions when executing the support multi-task processing program.

[0063] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a support multi-task processing program, and the support multi-task processing program implements the steps of the support multi-task processing method of any one of the above solutions when executed by a processor.

[0064] Beneficial effects: Compared with the prior art, the present application provides a multi-task processing support method. First, video data is acquired, and the video data is preprocessed to obtain a first target image frame set and a second target image frame set with the same timestamp, wherein the first target image frame set and the second target image frame set each contain at least one target image frame. Then, a prompt switching instruction and a task switching instruction are acquired, and a prompt type corresponding to the prompt switching instruction and a task type corresponding to the task switching instruction are determined respectively. Next, the first target image frame set is inferred based on the prompt type and the task type to obtain an inference result of the first target image frame. Finally, a task result is obtained according to the inference result of the first target image frame and the second target image frame set. The present application switches the prompt type and the task type of inference by receiving the prompt switching instruction and the task switching instruction, so that the inference of the target object is not limited to a single task, and flexible switching of tasks can be realized, thereby facilitating detection requirements in complex scenarios. In addition, the inference of the target object is performed on the first target image frame set, and the display and other processing of the inference result are performed on the second target image frame set. In actual application, the first target image frame set is a low-resolution image frame set, and the second target image frame set is a high-resolution image frame set. Therefore, the present application displays the inference result obtained by inferring the target object on the low-resolution image frame set on the high-resolution image frame set, which can effectively improve the inference speed and inference effect and save computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 The flowchart of the preferred embodiment of the multi-task processing support method provided by the embodiment of the present application.

[0066] Figure 2 The specific application scenario flowchart of the multi-task processing support method provided by the embodiment of the present application.

[0067] Figure 3 The architecture schematic diagram of the multi-task processing support system provided by the embodiment of the present application.

[0068] Figure 4 The principle block diagram of the terminal provided by the embodiment of the present application. DETAILED DESCRIPTION

[0069] To make the purpose, technical scheme and effect of the present application clearer and more explicit, the present application is further described in detail below with reference to the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0070] The flowcharts shown in the drawings are merely illustrative and do not necessarily include all contents and operations or steps, nor are they necessarily executed in the order described. For example, some operations or steps can be further decomposed, combined or partially merged, so the actual execution order can be changed according to actual conditions.

[0071] It should be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0072] It should be understood that, in order to facilitate clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the terms "first", "second", etc. are used to distinguish the same or similar items with basically the same function and role. For example, the first control information and the second control information are only used to distinguish different control information, and do not limit the order.

[0073] It can be understood by those skilled in the art that the terms "first", "second", etc. do not limit the quantity and execution order, and the terms "first", "second", etc. also do not necessarily mean different.

[0074] It should also be understood that the term "and / or" used in the specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0075] The existing detection system is difficult to realize flexible extension in the face of different scenes and diversified businesses. When encountering special scenes or specific business needs, it cannot be effectively adapted through convenient adjustment, and usually can only resort to customized development, which is a high-cost and long-cycle solution. In this way, not only the time, manpower and capital investment are greatly increased, but also the efficiency and response speed of business promotion are seriously affected. Therefore, the embodiment provides a multi-task processing support method. The multi-task processing support method based on the embodiment can switch between different task types, thereby facilitating expansion to more application scenarios. Specifically, the embodiment first acquires video data, pre-processes the video data, and obtains a first target image frame set and a second target image frame set with the same timestamp. The first target image frame set and the second target image frame set each contain at least one target image frame. Then, a prompt switching instruction and a task switching instruction are acquired, and a prompt type corresponding to the prompt switching instruction and a task type corresponding to the task switching instruction are determined. Next, the first target image frame set is inferred based on the prompt type and the task type, and an inference result of the first target image frame is obtained. Finally, a task result is obtained according to the inference result of the first target image frame and the second target image frame set. As can be seen, the embodiment can switch the prompt type and the task type by receiving the prompt switching instruction and the task switching instruction, so that the detection of the target object is not limited to a single task, and flexible switching of tasks can be realized. In addition, the inference of the target object in the embodiment is performed on the first target image frame set, and the display and other processing of the inference result are performed on the second target image frame set. In actual application, the first target image frame set is a low-resolution image frame set, and the second target image frame set is a high-resolution image frame set. Therefore, the inference result obtained by inferring the target object on the low-resolution image frame set is displayed on the high-resolution image frame set, which can effectively improve the inference speed and inference effect and save computing resources.

[0076] The multi-task processing support method of the embodiment can be applied to a terminal, which can be a computer, a smart television, a mobile phone, or other intelligent product terminals. As shown in Figure 1 The multi-task processing support method of the embodiment includes the following steps:

[0077] In step S100, video data is acquired, the video data is pre-processed, and a first target image frame set and a second target image frame set with the same timestamp are obtained. The first target image frame set and the second target image frame set each contain at least one target image frame.

[0078] In combination with Figure 2As shown, the embodiment can obtain video data based on the video acquisition module collecting videos captured by different types of cameras, including but not limited to USB cameras, network cameras, etc. In this embodiment, the video acquisition module supports different formats of video protocols and different ways of camera input. After obtaining the video data, the embodiment can preprocess the video data based on the video preprocessing module. The purpose of preprocessing is to improve the accuracy of subsequent processing of target objects and improve the efficiency of task processing.

[0079] In a specific application, the video data of the embodiment includes multiple video streams, and the following describes two video streams. When the collected video data is two video streams with the same video content and different resolutions, the embodiment can process the two video streams according to the video timestamps of the two video streams to determine a first image frame set and a second image frame set with the same timestamp, respectively. The two video streams of the embodiment are a low-resolution video stream and a high-resolution video stream, wherein the low-resolution video stream is used for inference, and the high-resolution video stream is used to combine the inference result of the low-resolution video stream to obtain a task result. Moreover, the two video streams of the embodiment each carry a video timestamp, and when the video timestamps corresponding to the two video streams are inconsistent, it indicates that the two video streams are not synchronized at this time. In order to improve the efficiency of model inference in the subsequent step, the two video streams need to be time-synchronized. Specifically, the embodiment caches all video stream pairs of the two video streams and the video timestamps corresponding to all video stream pairs. Specifically, a queue caching method can be used to obtain the cache data of all video stream pairs. Then, all video stream pairs are determined from all video streams, each video stream pair consisting of a high-resolution video and a low-resolution video having a corresponding relationship; all cache data pairs with the same video timestamp are determined from the cache data of all video stream pairs. At this time, all cache data pairs with the same video timestamp obtained include several image frame pairs with the same timestamp. Then, the cache data pair with the latest timestamp is determined from each cache data pair in all cache data pairs, that is, the image frame pair with the same timestamp and the latest timestamp in each cache data pair is determined, each image frame pair consisting of a high-resolution image frame and a low-resolution image frame having a corresponding relationship. Finally, the low-resolution image frames and high-resolution image frames in all image frame pairs with the same timestamp and the latest timestamp are taken as the first image frame set and the second image frame set, respectively.

[0080] Further, to improve the processing speed and accuracy in the subsequent inference process, the obtained first image frame set and second image frame set are denoised, such as using a Gaussian denoising method, to obtain a first target image frame set and a second target image frame set, so as to obtain the first target image frame set and the second target image frame set. In this embodiment, the first target image frame set and the second target image frame set are image frame sets with the same content and different resolutions, wherein the first target image frame set is all denoised low-resolution image frames, and the second target image frame set is all denoised high-resolution image frames. The first target image frame set is used for model inference of the target object in the subsequent steps, and the second target image frame set is used for displaying the inference result, which can effectively improve the inference speed and inference effect and save computing resources.

[0081] In another implementation manner, if the video time stamps corresponding to the two video streams are consistent, it indicates that the two video streams are synchronized, and at this time, the image frames of the two video streams are respectively denoised and respectively used as the first target image frame set and the second target image frame set. Through the time synchronization processing and denoising processing of the video data, the accuracy of the subsequent processing of the target object is improved, and the efficiency of the target object task processing and model inference is improved.

[0082] In step S200, a prompt switching instruction and a task switching instruction are obtained, and a prompt type corresponding to the prompt switching instruction and a task type corresponding to the task switching instruction are determined.

[0083] In this embodiment, a task control module is pre-set, which is used to output an instruction that can be used to control the process of model inference. The instruction includes a task switching instruction or a prompt switching instruction, which is combined with the prompt type and the task type to control the process of model inference. Figure 2The model inference module is used to perform model inference on the target image frames. The task switching instruction is used to control the inference model in the model inference module to switch between different task types, so as to perform different tasks. The task types include any one or more of a first task type, a second task type, a third task type, a fourth task type, and a fifth task type. For example, the first task type is a detection task, the second task type is a segmentation task, the third task type is a multi-modal task, the fourth task type is a question and answer task, and the fifth task type is a human pose estimation task or a hand pose estimation task. The third task type includes any one or more of an optical character recognition task (OCR task), a referring object classification task (ROC task), and a caption detection task. The prompt switching instruction is used to control the inference model in the model inference module to switch between different prompt types. The prompt types include any one or more of a first prompt (such as a text prompt), a second prompt (an image prompt), and a third prompt (no prompt). The content of the prompt in the embodiment includes two types: one is to prompt the target object to be detected, and the other is to prompt the task type to be processed by the model inference module. The embodiment switches the prompt type and the task type by receiving the prompt switching instruction and the task switching instruction, so that the detection of the target object is not limited to a single task, and the flexible switching of the task can be realized, thereby facilitating the detection requirements in complex scenes.

[0084] In step S300, the first target image frame set is inferred based on the prompt type and the task type, and an inference result of the first target image frame is obtained.

[0085] After the task control module issues an instruction to the model inference module, the model inference module infers the first target image frame set according to the determined prompt type and task type. Of course, in the embodiment, there can be not only one prompt type and one task type, but also multiple prompt types and multiple task types, so that the first target image frame set can be inferred for different tasks at the same time. Moreover, since the first target image frame set is a low-resolution image frame set, inferring the low-resolution image frame set can effectively improve the inference speed and save computing resources.

[0086] Further, in combination with Figure 2As shown, the model inference module of the embodiment comprises a prompt feature inference and cache module and an image inference module. In an actual inference process, the prompt of a current target image frame in the first target image frame set can be obtained according to the prompt of a previous target image frame in the first target image frame set, and then the current target image frame is inferred based on the prompt of the current target image frame and the task type to obtain an inference result of the current target image frame.

[0087] Specifically, when performing inference, the first target image frame set has multiple image frames, and when the prompt type is a first prompt (such as a text prompt) or a second prompt (such as an image prompt), the prompt feature inference and cache module caches the first prompt or the second prompt of the previous target image frame.

[0088] When performing inference, if the prompt type of the current target image frame is the first prompt or the second prompt, it is first determined whether the first prompt or the second prompt of the previous target image frame is modified, that is, whether the prompt of the current target image frame is the same as the prompt of the previous target image frame. If they are the same, it means that the prompt of the previous target image frame is not modified, and the prompt of the previous target image frame pre-cached by the prompt feature inference and cache module can be used as the prompt of the current target image frame, thereby accelerating the inference process. If the prompt of the current target image frame is not the same as the prompt of the previous target image frame, it means that the prompt of the previous target image frame is modified, and the modified prompt can be used as the prompt of the current target image frame. Therefore, the modified prompt feature can be re-extracted, and then the re-extracted prompt feature (text prompt feature or image prompt feature) is used to infer the current target image frame.

[0089] If the prompt type is a third prompt (such as no prompt), a preset universal file (such as a universal token file) is used as the prompt feature of each first target image frame set to infer the first target image frame set.

[0090] Further, in combination with Figure 2In an actual reasoning process, the image feature reasoning module in this embodiment can reason on the first target image frame set based on the prompt type and the task type. In this embodiment, different prompt types use different reasoning models. Specifically, if the input prompt type is the first prompt (such as a text prompt) or the third prompt (no prompt), it is determined that the reasoning model is a first model (such as a Grounding DINO model). The first model can reason on the first target image frame set according to the task type, and output the corresponding reasoning result. In actual application, for the third prompt, the first model combines a universal file (such as a universal token file) to implement reasoning. If the input prompt type is the second prompt (such as an image prompt), it is determined that the reasoning model is a second model (such as a T-Rex model). The second model can reason on the first target image frame set according to the task type, and output the corresponding reasoning result.

[0091] Specifically, in reasoning, the tasks performed by the reasoning model are different for different task types, and the output reasoning results are also different. If the input task type is a first task type (such as a detection task), the first target image frame set is output by the reasoning model (such as a Grounding DINO model) to output the detection frame of the target object and the confidence of the target object. The target object in this embodiment is an object in the first target image frame set, such as a fruit, an animal, or a building. If the input task type is a second task type (such as a segmentation task), the first target image frame set is output by the reasoning model (such as a Grounding DINO model) to output the mask information of the target object.

[0092] If the input task type is a third task type (such as any one or more of an optical character recognition task, an object classification task, and a description information detection task), the first target image frame set is output by the reasoning model (such as a Grounding DINO model) to output the feature information in the detection frame of the target object, which is the image feature of the target object. Then, the image feature and the first task prompt corresponding to the third task type are spliced and input to a third model (such as an OPT model) to obtain any one or more of the text information of the target object, the category information of the target object, or the description information of the target object.

[0093] If the input task type is a fourth task type (such as a QA task), the first target image frame set is output by the reasoning model (such as a Grounding DINO model) to output the image feature of the target object, and then the image feature and the second task prompt (such as a question prompt) corresponding to the fourth task type are spliced and input to the third model (such as an OPT model) to obtain the answer information.

[0094] If the input task type is the fifth task type, the first target image frame set can output the category information of the target object and the image feature of the target object through the inference model (such as the Grounding DINO model), and then the category information of the target object and the image feature of the target object are input into a preset decoder (such as a Transformer decoder) to obtain a set of two-dimensional coordinates of human key points or three-dimensional coordinates of human key points, that is, human pose estimation is realized. In another implementation manner, the embodiment can also output two-dimensional coordinates of hand key points and three-dimensional coordinates of hand key points, so as to realize hand pose estimation. As can be seen, the embodiment can be based on the diversification of the prompt type and the diversification of the task type, so as to realize the switching between different prompts and different tasks, meet the demand of supporting multi-task processing, and be beneficial to expanding the application scenarios.

[0095] Further, in an implementation manner, when the task type is the third sub-task type task (ROC task) and the prompt type is the third prompt (that is, no prompt), the inference result obtained by the first inference can be re-input into the inference model as a prompt feature for inference, so as to realize secondary inference. Specifically, when the task type is the ROC task and the prompt type is no prompt, the inference model outputs feature information in the detection frame of the target object according to the first target image frame set and the universal file (such as the Universal token file), and the feature information is the image feature of the target object. At this time, the image feature is spliced with the first task prompt of the ROC task and then input into a preset third model (such as an OPT model) to obtain the category information of the target object, that is, the first inference is completed. Then, the category information of the target object is re-input into the prompt feature inference and caching module to obtain the updated target object prompt feature. The updated target object prompt feature and the first target image frame set are re-input into the inference model (such as the Grounding DINO model) to output the updated image feature of the target object, and then the updated image feature of the target object is spliced with the first task prompt of the ROC task and then input into the third model (such as the OPT model) again to obtain the updated category information of the target object, that is, the second inference is completed. The embodiment is subjected to twice inference, which is helpful to realize the correction of the inference result obtained by the first inference and improve the accuracy of inference.

[0096] Further, in combination with Figure 2As shown, the embodiment further includes a result post-processing module, which can be used for filtering and smoothing the bounding box of the target object in the inference result, and further improves the accuracy of the inference result. Specifically, the embodiment first acquires a preset first confidence threshold, which includes a confidence threshold of the bounding box of the target object and / or a confidence threshold of the category information of the target object. Then, the bounding box of the target object and / or the category information of the target object are filtered based on the first confidence threshold, and the bounding box of the target object and the category information of the target object with a confidence greater than the first confidence threshold are retained. In another implementation manner, the embodiment further matches the bounding box of the target object of the current target image frame with the bounding box of the target object of the previous target image frame (for example, Hungarian matching is adopted), and obtains a matching score. If the matching score is greater than a preset matching threshold, it indicates that the bounding box of the target object of the current target image frame is close to the bounding box of the target object of the previous target image frame, and the bounding box of the target object of the current target image frame can be retained.

[0097] If the matching score of the current target image frame and the previous target image frame is less than the matching threshold and the task type is the same, a difference between the matching score and the matching threshold is determined. If the difference is within a preset range, it indicates that the inference result of the current target image frame is similar to but different from the inference result of the previous target image frame, and the inference result of the previous target image frame can be used as the inference result of the current target image frame, thereby ensuring the stability of the inference model output result.

[0098] Step S400: obtaining the task result according to the inference result of the first target image frame and the second target image frame set.

[0099] Further, the embodiment further provides a display control module and a result display module, the display control module is used to issue instructions to the display module, if multiple tasks simultaneously obtain results through reasoning, the content specifically required to be displayed in the display module is selected. The result display module can be used to display and process the reasoning result processed by the result post-processing module, and display locally or remotely through network transmission. Specifically, when the reasoning result is displayed and processed, the embodiment respectively obtains the dimension information (such as image size) of each second target image frame set of the first target image frame set and the second target image frame set. Then, for each first target image frame in the first target image frame set, the first coordinate information of the detection box of the target object in the first target image frame is obtained based on the first target image frame through the inference model, and the second coordinate information of the detection box of the target object in the second target image frame is determined based on the first coordinate information and the dimension information of the corresponding second target image frame. Specifically, the first coordinate information of the detection box of the target object in the first target image frame can be multiplied by the image size of the corresponding second target image frame to obtain the second coordinate information of the detection box of the target object in the second target image frame. Finally, the reasoning result is displayed on the corresponding second target image frame according to the second coordinate information, so as to realize the display of the reasoning result and obtain the task result.

[0100] In summary, the embodiment realizes the switching of the prompt type and the task type by receiving the prompt switching instruction and the task switching instruction, so that the detection of the target object is not limited to a single task, and flexible switching of the task can be realized, thereby facilitating the detection requirement in a complex scene. In addition, the embodiment displays the reasoning result obtained by reasoning the target object on the low-resolution image frame set on the high-resolution image frame set, which can effectively improve the reasoning speed and the reasoning effect, and save the computing resources.

[0101] Based on the above-mentioned embodiment, the application further provides a multi-task processing system, such as Figure 3As shown, the system includes: a video processing module 10, a task prompting control module 20, a model inference module 30, and a task result output module 40. Specifically, the video processing module 10 is used to acquire video data, preprocess the video data to obtain a first target image frame set and a second target image frame set with the same timestamp, where both the first and second target image frame sets contain at least one target image frame. The task prompting control module 20 is used to acquire prompt switching instructions and task switching instructions, and determine the prompt type corresponding to the prompt switching instruction and the task type corresponding to the task switching instruction, respectively. The model inference module 30 is used to infer the first target image frame set based on the prompt type and the task type to obtain the inference result of the first target image frame. The task result output module 40 is used to obtain the task result based on the inference result of the first target image frame and the second target image frame set.

[0102] The working principle of each module in the multi-tasking system supported in this embodiment is the same as that of each step in the above method embodiment, and will not be repeated here.

[0103] The modules in the aforementioned multitasking system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the terminal in hardware form or independent of it, or stored in the terminal's memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0104] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 4 As shown. The terminal may include one or more processors 100 ( Figure 4 (Only one is shown in the image), a memory 101, and a computer program 102 stored in the memory 101 and executable on one or more processors 100, such as a sleep analysis program based on multi-sensor data. When one or more processors 100 execute the computer program 102, they can implement the various steps in the sleep analysis method embodiment based on multi-sensor data. Alternatively, when one or more processors 100 execute the computer program 102, they can implement the functions of various modules / units in the sleep analysis system embodiment based on multi-sensor data, which is not limited here.

[0105] In one embodiment, the processor 100 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0106] In one embodiment, the memory 101 can be an internal storage unit of the electronic device, such as a hard disk or a memory of the electronic device. The memory 101 can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 101 can include both the internal storage unit and the external storage device of the electronic device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 can also be used to temporarily store data that has been output or will be output.

[0107] Those skilled in the art can understand that, Figure 4 The block diagram shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. The specific terminal can include more or less components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, operating database or other medium used in the embodiments of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0109] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for supporting multi-task processing, characterized in that, The method includes: Acquire video data, preprocess the video data to obtain a first target image frame set and a second target image frame set with the same timestamp, wherein both the first target image frame set and the second target image frame set contain at least one target image frame; Obtain the prompt switching instruction and the task switching instruction, and determine the prompt type corresponding to the prompt switching instruction and the task type corresponding to the task switching instruction, respectively; Based on the prompt type and the task type, inference is performed on the first target image frame set to obtain the inference result of the first target image frame; The task result is obtained based on the inference result of the first target image frame and the second target image frame set.

2. The method for supporting multi-task processing according to claim 1, characterized in that, The video data is preprocessed to obtain the target image frame, including: When the video data consists of two video streams with the same video content but different resolutions, the two video streams are processed according to their video timestamps to determine a first image frame set and a second image frame set with the same timestamp. The first image frame set and the second image frame set are denoised to obtain the first target image frame set and the second target image frame set.

3. The method for supporting multi-task processing according to claim 2, characterized in that, The two video streams are processed according to their timestamps to determine a first set of image frames and a second set of image frames with the same timestamp, including: When the video timestamps of the two video streams are inconsistent, all video stream pairs of the two video streams and the video timestamps corresponding to all video stream pairs are cached to obtain cached data of all video stream pairs. Identify all cached data pairs with the same video timestamp from the cached data of all video stream pairs; From each of the cached data pairs, determine the cached data pair with the latest timestamp, and use it as the first image frame set and the second image frame set, respectively.

4. The method for supporting multitasking processing according to any one of claims 1-3, characterized in that, Based on the prompt type and the task type, inference is performed on the first target image frame set to obtain the inference result of the first target image frame, including: Based on the prompt type, determine the inference model corresponding to the prompt type; Based on the task type, the inference model is used to infer the first target image frame set to obtain the inference result corresponding to the task type.

5. The method for supporting multi-task processing according to claim 4, characterized in that, Based on the prompt type, determine the inference model corresponding to the prompt type, including: If the prompt type is a first prompt or a third prompt, then the inference model is determined to be a first model; If the prompt type is a second prompt, then the reasoning model is determined to be a second model.

6. The method for supporting multitasking processing according to claim 5, characterized in that, Based on the task type, the inference model is used to infer the first target image frame set to obtain an inference result corresponding to the task type, including: If the task type is the first task type, then the first target image frame set obtains the detection box and confidence of the target object through the inference model, and the target object is the object in the first target image frame set; If the task type is the second task type, then the first target image frame set obtains the mask information of the target object through the inference model; If the task type is the third task type, the first target image frame set obtains the image features of the target object through the inference model. The image features and the first task prompt obtain any one or more of the text information, category information, or description information of the target object through the third model. If the task type is the fourth task type, then the first target image frame set obtains the image features of the target object through the inference model, and the image features and the second task prompt obtain the target object's answer information through the third model; If the task type is the fifth task type, the first target image frame set obtains the category information and image features of the target object through the inference model. The category information and image features of the target object are then decoded to obtain a set of two-dimensional coordinates or three-dimensional coordinates of human key points.

7. The method for supporting multi-task processing according to claim 6, characterized in that, Based on the prompt type and the task type, inference is performed on the first target image frame set to obtain the inference result of the first target image frame, which further includes: The prompt for the current target image frame in the first target image frame set is obtained based on the prompt of the previous target image frame in the first target image frame set. Based on the prompts of the current target image frame and the task type, inference is performed on the current target image frame to obtain the inference result of the current target image frame.

8. The method for supporting multitasking processing according to claim 7, characterized in that, The prompt for the current target image frame in the first target image frame set is obtained based on the prompt of the previous target image frame in the first target image frame set, including: If the prompt of the previous target image frame is not modified, the prompt of the previous target image frame that is pre-cached will be used as the prompt of the current target image frame. If the prompt for the previous target image frame is modified, the modified prompt will be used as the prompt for the current target image frame.

9. The method for supporting multitasking processing according to claim 7, characterized in that, Based on the prompt type and task type of the current target image frame, inference is performed on the current target image frame to obtain the inference result of the current target image frame, which also includes: If the prompt type is a third prompt, then a preset general file is used as a prompt feature to infer the first target image frame set.

10. The method for supporting multi-task processing according to claim 6, characterized in that, If the task type is a third task type, then the first target image frame set obtains the image features of the target object through the inference model. These image features, along with the first task prompt, are obtained through the third model using any one or more of the following: textual information of the target object, category information of the target object, or descriptive information of the target object, including: When the task type is the third sub-task type and the prompt type is the third prompt, the first target image frame set obtains the image features of the target object through the inference model, and the image features and the first task prompt obtain the target object category information through the third model; Based on the category information of the target object, the updated target object prompt features are obtained; The updated target object cue features and the first target image frame set are used by the inference model to obtain the updated image features of the target object; The updated image features of the target object and the type of the third subtask are used by the third model to obtain the updated category information of the target object.

11. The method for supporting multitasking processing according to any one of claims 7-9, characterized in that, The method further includes: Obtain a preset first confidence threshold, which includes the confidence threshold of the detection box of the target object and / or the confidence threshold of the category information of the target object; The detection bounding box and / or category information of the target object are filtered based on the first confidence threshold.

12. The method for supporting multi-task processing according to claim 11, characterized in that, After filtering the detection bounding box and / or category information of the target object based on the first confidence threshold, the method further includes: The detection bounding boxes of the target objects in the current target image frame are matched with the detection bounding boxes of the target objects in the previous target image frame to obtain a matching score; If the matching score is greater than the preset matching threshold, the detection box of the target object in the current target image frame is retained.

13. The method for supporting multitasking processing according to claim 12, characterized in that, After matching the detection bounding boxes of the target objects in the current target image frame with the detection bounding boxes of the target objects in the previous target image frame to obtain a matching score, the method further includes: If the matching score between the current target image frame and the previous target image frame is less than the matching threshold and the task types are the same, then the difference between the matching score and the matching threshold is determined. If the difference is within a preset range, the inference result of the previous target image frame is used as the inference result of the current target image frame.

14. The method for supporting multitasking processing according to claim 1 or 13, characterized in that, Based on the inference result of the first target image frame and the second target image frame set, the task result is obtained, including: The dimensional information of each first target image frame in the first target image frame set and each second target image frame in the second target image frame set are obtained respectively; For each first target image frame, the first coordinate information of the detection box of the target object is obtained through an inference model based on the first target image frame, and the second coordinate information of the detection box of the target object in the second target image frame is determined based on the first coordinate information and the dimension information of the corresponding second target image frame. The reasoning result is displayed on the corresponding second target image frame based on the second coordinate information.

15. A multi-tasking system, characterized in that, The system includes: The video processing module is used to acquire video data, preprocess the video data, and obtain a first target image frame set and a second target image frame set with the same timestamp. Both the first target image frame set and the second target image frame set contain at least one target image frame. The task prompt control module is used to acquire prompt switching instructions and task switching instructions, and to determine the prompt type corresponding to the prompt switching instructions and the task type corresponding to the task switching instructions, respectively. The model inference module is used to infer the first target image frame set based on the prompt type and the task type to obtain the inference result of the first target image frame; The task result output module is used to obtain the task result based on the inference result of the first target image frame and the second target image frame set.

16. A terminal, characterized in that, The terminal includes a memory, a processor, and a multitasking support program stored in the memory and executable on the processor. When the processor executes the multitasking support program, it implements the steps of the multitasking support method as described in any one of claims 1-13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multitasking program, which, when executed by a processor, implements the steps of the multitasking method as described in any one of claims 1-13.

Images