Method and system supporting multitask processing, terminal and medium

Abstract

The invention discloses a multi-task supporting processing method and system, a terminal and a medium, and the method comprises the steps: obtaining video data, carrying out the preprocessing of the video data, and obtaining a first target image frame set and a second target image frame set which have the same timestamp; obtaining a prompt switching instruction and a task switching instruction, and respectively determining a prompt type corresponding to the prompt switching instruction and a task type corresponding to the task switching instruction; reasoning the first target image frame set based on the prompt type and the task type to obtain a reasoning result of the first target image frame; and obtaining a task result according to the reasoning result of the first target image frame and the second target image frame set. According to the invention, switching of the inferred prompt type and task type can be realized based on the prompt switching instruction and the task switching instruction, so that detection of the target object is not limited to a single task, flexible switching of tasks can be realized, and detection requirements in a complex scene can be met conveniently.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a method, system, terminal and medium supporting multi-task processing. Background Art

[0002] Existing target detection systems are only designed for a single task or a specific category and a specific task type. They cannot be expanded to different task types, nor can they be flexibly switched between different tasks. Therefore, in the face of complex and ever-changing scenarios and diverse business needs, the ability of a single detection has become insufficient and difficult to meet the growing and diverse needs of users in complex scenarios and during business operations. In addition, existing target detection systems basically use video data of the same resolution for reasoning and display, and do not distinguish between the video data for reasoning and display. In particular, when high-resolution video data is used for reasoning, it will not only affect the reasoning speed, but also waste computing resources. Summary of the invention

[0003] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a method, system, terminal and medium supporting multi-task processing are provided, aiming to solve the problems that the existing target detection system is only designed for a single task or a specific category and a specific task type, cannot be expanded to different task types, cannot be flexibly switched between different tasks, and still has problems such as low reasoning efficiency and waste of computing resources.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0005] In a first aspect, the present invention provides a method for supporting multi-tasking, wherein the method comprises:

[0006] Acquire video data, and preprocess the video data to obtain a first target image frame set and a second target image frame set having the same timestamp, wherein the first target image frame set and the second target image frame set each include at least one target image frame;

[0007] Acquire a prompt switching instruction and a task switching instruction, and respectively determine a prompt type corresponding to the prompt switching instruction and a task type corresponding to the task switching instruction;

[0008] Reasoning the first target image frame set based on the prompt type and the task type to obtain a reasoning result of the first target image frame;

[0009] A task result is obtained according to the inference result of the first target image frame and the second target image frame set.

[0010] In one implementation, preprocessing the video data to obtain a target image frame includes:

[0011] When the video data is two video streams with the same video content and different resolutions, the two video streams are processed according to the video timestamps of the two video streams to respectively determine a first image frame set and a second image frame set with the same timestamp;

[0012] The first image frame set and the second image frame set are denoised to obtain the first target image frame set and the second target image frame set.

[0013] In one implementation, the two video streams are processed according to the video timestamps of the two video streams to respectively determine a first image frame set and a second image frame set having the same timestamps, including:

[0014] When the video timestamps of the two video streams are inconsistent, all video stream pairs of the two video streams and the video timestamps corresponding to all video stream pairs are cached to obtain cached data of all video stream pairs;

[0015] Determine all cached data pairs with the same video timestamp from the cached data of all video stream pairs;

[0016] From each of the cached data pairs among all the cached data pairs, a cached data pair with the latest timestamp is determined as the first image frame set and the second image frame set, respectively.

[0017] In one implementation, reasoning the first target image frame set based on the prompt type and the task type to obtain a reasoning result of the first target image frame includes:

[0018] Based on the prompt type, determining a reasoning model corresponding to the prompt type;

[0019] Based on the task type, the first target image frame set is inferred through the inference model to obtain an inference result corresponding to the task type.

[0020] In one implementation, based on the prompt type, determining the reasoning model corresponding to the prompt type includes:

[0021] If the prompt type is the first prompt or the third prompt, determining that the inference model is the first model;

[0022] If the prompt type is the second prompt, the inference model is determined to be the second model.

[0023] In one implementation, reasoning the first target image frame set based on the task type using the reasoning model to obtain a reasoning result corresponding to the task type includes:

[0024] If the task type is the first task type, the first target image frame set obtains a detection frame of the target object and a confidence level of the target object through the inference model;

[0025] If the task type is the second task type, the first target image frame set obtains mask information of the target object through the inference model;

[0026] If the task type is the third task type, the first target image frame set obtains image features of the target object through the inference model, and the image features and the first task prompt obtain any one or more of text information of the target object, category information of the target object, or description information of the target object through the third model;

[0027] If the task type is the fourth task type, the first target image frame set obtains the image features of the target object through the inference model, and the image features and the second task prompt obtain the target object to obtain answer information through the third model;

[0028] If the task type is the fifth task type, the first target image frame set obtains the category information of the target object and the image features of the target object through the inference model, and the category information of the target object and the image features of the target object are passed through the decoder to obtain a set of two-dimensional coordinates of human body key points or three-dimensional coordinates of human body key points.

[0029] In one implementation, reasoning the first target image frame set based on the prompt type and the task type to obtain a reasoning result of the first target image frame further includes:

[0030] Obtaining a prompt of a current target image frame in the first target image frame set according to a prompt of a previous target image frame in the first target image frame set;

[0031] The current target image frame is inferred based on the prompt of the current target image frame and the task type to obtain an inference result of the current target image frame.

[0032] In one implementation, obtaining a prompt of a current target image frame in the first target image frame set according to a prompt of a previous target image frame in the first target image frame set includes:

[0033] If the prompt of the previous target image frame is not modified, the pre-cached prompt of the previous target image frame is used as the prompt of the current target image frame;

[0034] If the prompt of the previous target image frame is modified, the modified prompt is used as the prompt of the current target image frame.

[0035] If the prompt of the previous target image frame is not modified, the pre-cached prompt of the previous target image frame is used as the prompt of the current target image frame;

[0036] If the prompt of the previous target image frame is modified, the modified prompt is used as the prompt of the current target image frame.

[0037] In one implementation, reasoning the current target image frame based on the prompt of the current target image frame and the task type to obtain the reasoning result of the current target image frame further includes:

[0038] If the prompt type is the third prompt, a preset universal file is used as a prompt feature to perform reasoning on the first target image frame set.

[0039] In one implementation, if the task type is a third task type, the first target image frame set obtains image features of a target object through the inference model, and the image features and the first task prompt obtain any one or more of text information of the target object, category information of the target object, or description information of the target object through the third model, including:

[0040] When the task type is the third subtask type and the prompt type is the third prompt, the first target image frame set obtains image features of the target object through the inference model, and the image features and the first task prompt obtain target object category information through the third model;

[0041] Based on the category information of the target object, obtaining an updated target object prompt feature;

[0042] The updated target object prompt features and the first target image frame set are passed through the inference model to obtain updated image features of the target object;

[0043] The updated image features of the target object and the third subtask type are used through the third model to obtain updated category information of the target object.

[0044] In one implementation, the method further includes:

[0045] Acquire a preset first confidence threshold, where the first confidence threshold includes a confidence threshold of a detection frame of the target object and / or a confidence threshold of category information of the target object;

[0046] The detection frame of the target object and / or the category information of the target object are filtered based on the first confidence threshold.

[0047] In one implementation, after screening the detection frame of the target object and / or the category information of the target object based on the first confidence threshold, the method further includes:

[0048] Matching a detection frame of a target object in the current target image frame with a detection frame of a target object in a previous target image frame to obtain a matching score;

[0049] If the matching score is greater than a preset matching threshold, the detection frame of the target object of the current target image frame is retained.

[0050] In one implementation, after matching the detection frame of the target object in the current target image frame with the detection frame of the target object in the previous target image frame to obtain a matching score, the method further includes:

[0051] If the matching score between the current target image frame and the previous target image frame is less than the matching threshold and the task types are the same, determining the difference between the matching score and the matching threshold;

[0052] If the difference is within a preset range, the inference result of the previous target image frame is used as the inference result of the current target image frame.

[0053] In one implementation, obtaining the task result according to the inference result of the first target image frame and the second target image frame set includes:

[0054] Respectively acquiring dimension information of each first target image frame in the first target image frame set and each second target image frame in the second target image frame set;

[0055] For each of the first target image frames, first coordinate information of a detection frame of the target object is obtained through an inference model based on the first target image frame, and second coordinate information of the detection frame of the target object in the second target object frame is determined based on the first coordinate information and dimension information of the corresponding second target image frame;

[0056] The inference result is displayed on a corresponding second target image frame according to the second coordinate information.

[0057] In a second aspect, an embodiment of the present invention further provides a system for supporting multi-tasking processing, wherein the system comprises:

[0058] A video processing module, used for acquiring video data, preprocessing the video data, and obtaining a first target image frame set and a second target image frame set with the same timestamp, wherein the first target image frame set and the second target image frame set both contain at least one target image frame;

[0059] A task prompt control module, used to obtain a prompt switching instruction and a task switching instruction, and respectively determine a prompt type corresponding to the prompt switching instruction and a task type corresponding to the task switching instruction;

[0060] A model reasoning module, used for reasoning the first target image frame set based on the prompt type and the task type to obtain a reasoning result of the first target image frame;

[0061] The task result output module is used to obtain the task result according to the inference result of the first target image frame and the second target image frame set.

[0062] In a third aspect, an embodiment of the present invention further provides a terminal, wherein the terminal includes a memory, a processor, and a multi-tasking support program stored in the memory and executable on the processor, and when the processor executes the multi-tasking support program, the steps of the multi-tasking support method of any one of the above-mentioned schemes are implemented.

[0063] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a program supporting multi-tasking is stored on the computer-readable storage medium, and when the program supporting multi-tasking is executed by a processor, the steps of the method supporting multi-tasking described in any one of the above-mentioned schemes are implemented.

[0064] Beneficial effects: Compared with the prior art, the present invention provides a method for supporting multi-task processing. The present invention first obtains video data, pre-processes the video data, and obtains a first target image frame set and a second target image frame set with the same timestamp, wherein the first target image frame set and the second target image frame set each contain at least one target image frame. Then, a prompt switching instruction and a task switching instruction are obtained, and the prompt type corresponding to the prompt switching instruction and the task type corresponding to the task switching instruction are determined respectively. Next, the first target image frame set is inferred based on the prompt type and the task type to obtain the inference result of the first target image frame. Finally, the task result is obtained based on the inference result of the first target image frame and the second target image frame set. The present invention switches the prompt type and the task type of inference by receiving the prompt switching instruction and the task switching instruction, so that the inference of the target object is not limited to a single task, and flexible switching of tasks can be achieved, thereby facilitating the detection requirements in complex scenes. In addition, the present invention uses the first target image frame set to infer the target object, and the display and other processing of the inference results are performed in the second target image frame set. In actual application, the first target image frame set is a low-resolution image frame set, and the second target image frame set is a high-resolution image frame set. Therefore, the present invention displays the inference results obtained by inferring the target object on the low-resolution image frame set on the high-resolution image frame set, which can effectively improve the inference speed and inference effect and save computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 The present invention provides a flowchart of a preferred embodiment of a method for supporting multi-tasking.

[0066] Figure 2 A flowchart of a specific application scenario of the method for supporting multi-tasking provided by an embodiment of the present invention.

[0067] Figure 3 A schematic diagram of the architecture of a system supporting multi-tasking processing provided by an embodiment of the present invention.

[0068] Figure 4 A functional block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0070] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations or steps, nor must they be executed in the order described. For example, some operations or steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0071] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0072] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish between identical or similar items with substantially identical functions and effects. For example, the first control information and the second control information are only used to distinguish different control information, and their order is not limited.

[0073] Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit the differences.

[0074] It should be further understood that the term “and / or” used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0075] Existing detection systems are difficult to achieve flexible expansion and extension when facing different scenarios and diversified businesses. Whenever encountering special scenarios or specific business requirements, it is impossible to effectively adapt through convenient adjustments, and usually only resort to customized development, a high-cost, long-cycle solution. In this way, not only the investment in time, manpower and funds is greatly increased, but also the efficiency and response speed of business advancement are seriously affected. To this end, this embodiment provides a method for supporting multi-task processing. Based on the method for supporting multi-task processing in this embodiment, switching between different task types can be achieved, which is conducive to expanding to more application scenarios. Specifically, this embodiment first obtains video data, pre-processes the video data, and obtains a first target image frame set and a second target image frame set with the same timestamp, and the first target image frame set and the second target image frame set both contain at least one target image frame. Then, a prompt switching instruction and a task switching instruction are obtained, and the prompt type corresponding to the prompt switching instruction and the task type corresponding to the task switching instruction are determined respectively. Then, the first target image frame set is inferred based on the prompt type and the task type to obtain the inference result of the first target image frame. Finally, the task result is obtained according to the reasoning result of the first target image frame and the second target image frame set. It can be seen that the present embodiment can realize the switching of the prompt type and the task type by receiving the prompt switching instruction and the task switching instruction, so that the detection of the target object is not limited to a single task, and the flexible switching of tasks can be realized. In addition, the reasoning of the target object in the present embodiment uses the first target image frame set, and the display of the reasoning result and other processing are in the second target image frame set. In actual application, the first target image frame set is a low-resolution image frame set, and the second target image frame set is a high-resolution image frame set. Therefore, the present embodiment will display the reasoning result obtained by reasoning the target object on the low-resolution image frame set on the high-resolution image frame set, which can effectively improve the reasoning speed and reasoning effect and save computing resources.

[0076] The method for supporting multitasking in this embodiment can be applied to a terminal, which can be a computer, a smart TV, a mobile phone or other intelligent product terminal. Figure 1 As shown in , the method for supporting multi-tasking in this embodiment includes the following steps:

[0077] Step S100: Acquire video data, pre-process the video data to obtain a first target image frame set and a second target image frame set with the same timestamp, wherein the first target image frame set and the second target image frame set each include at least one target image frame.

[0078] Combination Figure 2As shown, this embodiment can acquire video data by acquiring videos captured by different types of cameras based on the video acquisition module, and the cameras include but are not limited to USB cameras, network cameras, etc. In this embodiment, the video acquisition module supports video protocols of different formats and also supports camera inputs of different ways. After acquiring the video data, this embodiment can pre-process the video data based on the video pre-processing module. The purpose of the pre-processing is to improve the accuracy of the processing of the target object in the subsequent steps and to improve the efficiency of the task processing.

[0079] In specific applications, the video data of this embodiment includes multiple video streams, and the following is explained with two video streams. When the collected video data is two video streams with the same video content and different resolutions, this embodiment can process the two video streams according to the video timestamps of the two video streams, and respectively determine the first image frame set and the second image frame set with the same timestamp. The two video streams of this embodiment are a low-resolution video stream and a high-resolution video stream, wherein the low-resolution video stream is used for reasoning, and the high-resolution video stream is used to combine with the reasoning result of the low-resolution video stream to obtain the task result. In addition, the two video streams of this embodiment carry video timestamps respectively. When the video timestamps corresponding to the two video streams are inconsistent, it means that the two video streams are not synchronized at this time. In order to improve the efficiency of model reasoning in subsequent steps, it is necessary to perform time synchronization processing on the two asynchronous video streams. Specifically, this embodiment caches all video stream pairs of the two video streams and the video timestamps corresponding to all video stream pairs. Specifically, a queue cache method can be used to obtain cached data of all video stream pairs. Next, all video stream pairs are determined from all video streams, each video stream pair consists of a high-resolution video and a low-resolution video with a corresponding relationship; all cached data pairs with the same video timestamp are determined from the cached data of all video stream pairs, and all cached data pairs with the same video timestamp obtained at this time include several image frame pairs with the same timestamp, and then the cached data pair with the latest timestamp is determined from each cached data pair in all the cached data pairs, that is, the image frame pair with the same timestamp and the latest timestamp in each cached data pair is determined, and each image frame pair consists of a high-resolution image frame and a low-resolution image frame with a corresponding relationship, and finally the low-resolution image frames and the high-resolution image frames in all the image frame pairs with the same timestamp and the latest timestamp are respectively used as the first image frame set and the second image frame set.

[0080] Furthermore, in order to improve the processing speed and accuracy in the subsequent reasoning process, the present embodiment denoises the obtained first image frame set and the second image frame set, for example, by using Gaussian denoising to obtain the first target image frame set and the second target image frame set, thereby obtaining the first target image frame set and the second target image frame set. In the present embodiment, the first target image frame set and the second target image frame set are sets of image frames with the same content and different resolutions, wherein the first target image frame set is all denoised low-resolution image frames, and the second target image frame set is all denoised high-resolution image frames. The first target image frame set is used for model reasoning of the target object in the subsequent steps, and the second target image frame set is used to display the reasoning results, which can effectively improve the reasoning speed and reasoning effect and save computing resources.

[0081] In another implementation, if the video timestamps corresponding to the two video streams are consistent, it means that the two video streams are synchronized. At this time, the image frames of the two video streams are denoised respectively and used as the first target image frame set and the second target image frame set respectively. This embodiment is conducive to improving the accuracy of target object processing in subsequent steps and improving the efficiency of target object task processing and model reasoning by performing time synchronization and denoising on video data.

[0082] Step S200: Acquire a prompt switching instruction and a task switching instruction, and respectively determine a prompt type corresponding to the prompt switching instruction and a task type corresponding to the task switching instruction.

[0083] In this embodiment, a task control module is pre-set, and the task control module is used to output instructions, and the instructions can be used to control the process of model reasoning. The instructions include task switching instructions or prompt switching instructions, combined with Figure 2, this embodiment uses a model inference module to perform model inference on the target image frame. The task switching instruction is used to control the inference model in the model inference module to switch between different task types, so as to perform different tasks. The task type includes any one or more of the first task type, the second task type, the third task type, the fourth task type, and the fifth task type. For example, the first task type is a detection task, the second task type is a segmentation task, the third task type is a multimodal task, the fourth task type is a question-and-answer task, and the fifth task type is a human posture estimation task or a hand posture estimation task. The third task type includes: any one or more of an optical character recognition task (OCR (Optical Character Recognition) task), an object classification task (ROC (Referring Object Classification) task) and a description information (Caption) detection task. The prompt switching instruction is used to control the inference model in the model inference module to switch between different prompt types, and the prompt type includes any one or more of the first prompt (such as a text prompt), the second prompt (image prompt) and the third prompt (no prompt). There are two types of prompt content in this embodiment: one is to prompt the target object to be detected, and the other is to prompt the task type that the model inference module needs to process. This embodiment switches the prompt type and the task type by receiving the prompt switching instruction and the task switching instruction, so that the detection of the target object is not limited to a single task, and flexible switching of tasks can be achieved, thereby facilitating the detection needs in complex scenarios.

[0084] Step S300: Inferring the first target image frame set based on the prompt type and the task type to obtain an inference result of the first target image frame.

[0085] When the task control module sends instructions to the model reasoning module, the model reasoning module will reason on the first target image frame set according to the determined prompt type and task type. Of course, in this embodiment, the prompt type and task type may be determined to be more than one, and multiple prompt types may exist at the same time, and multiple task types may exist at the same time, so that the first target image frame set can be reasoned for different tasks at the same time. In addition, since the first target image frame set is a low-resolution image frame set, reasoning on the low-resolution image frame set can effectively improve the reasoning speed and save computing resources.

[0086] Furthermore, combined with Figure 2As shown, the model reasoning module of this embodiment includes a prompt feature reasoning and cache module and an image reasoning module. In the actual reasoning process, this embodiment can obtain the prompt of the current target image frame in the first target image frame set according to the prompt of the previous target image frame in the first target image frame set, and then reason the current target image frame based on the prompt and task type of the current target image frame to obtain the reasoning result of the current target image frame.

[0087] Specifically, when performing inference, the first target image frame set has multiple image frames. When the prompt type is a first prompt (such as a text prompt) or a second prompt (such as an image prompt), the prompt feature inference and cache module will cache the first prompt or the second prompt of the previous target image frame.

[0088] When performing reasoning, if the prompt type of the current target image frame is the first prompt or the second prompt, first determine whether the first prompt or the second prompt of the previous target image frame is modified, that is, determine whether the prompt of the current target image frame is the same as the prompt of the previous target image frame. If they are the same, it means that the prompt of the previous target image frame has not been modified. At this time, the prompt of the previous target image frame pre-cached by the prompt feature reasoning and cache module can be used as the prompt of the current target image frame, thereby accelerating the reasoning process. If the prompt of the current target image frame is different from the prompt of the previous target image frame, it means that the prompt of the previous target image frame has been modified. At this time, the modified prompt can be used as the prompt of the current target image frame, so the modified prompt feature can be re-extracted, and then the re-extracted prompt feature (text prompt feature or image prompt feature) can be used to reason about the current target image frame.

[0089] If the prompt type is the third prompt (such as no prompt), a preset universal file (such as a universal token file) is used as a prompt feature of each first target image frame set to infer the first target image frame set.

[0090] Furthermore, combined with Figure 2In the actual reasoning process, the image feature reasoning module of this embodiment can reason about the first target image frame set based on the prompt type and the task type. In this embodiment, different prompt types use different reasoning models. Specifically, if the input prompt type is the first prompt (such as a text prompt) or the third prompt (no prompt), the reasoning model is determined to be the first model (such as the Grounding DINO model), and the first model can reason about the first target image frame set according to the task type and output the corresponding reasoning result. In actual application, for the third prompt, the first model will be combined with a universal file (such as a universal token file) to achieve reasoning. If the input prompt type is the second prompt (such as an image prompt), the reasoning model is determined to be the second model (such as the T-Rex model), and the second model can reason about the first target image frame set according to the task type and output the corresponding reasoning result.

[0091] Specifically, during reasoning, for different task types, the tasks performed by the reasoning model are different, and the output reasoning results are also different. If the input task type is the first task type (such as a detection task), the first target image frame set will output the detection box of the target object and the confidence of the target object through the reasoning model (such as the Grounding DINO model). The target object of this embodiment is the object in the first target image frame set, such as fruits, animals, or buildings. If the input task type is the second task type (such as a segmentation task), the first target image frame set outputs the mask information (mask) of the target object through the reasoning model (such as the Grounding DINO model).

[0092] If the input task type is a third task type (such as any one or more of an optical character recognition task, an object classification task, and a description information detection task), the first target image frame set can output feature information in the detection frame of the target object through an inference model (such as a Grounding DINO model), and the feature information is the image feature of the target object. Then, this embodiment splices the image feature with the first task prompt corresponding to the third task type and inputs it into a preset third model (such as an OPT model), thereby obtaining any one or more of the text information of the target object, the category information of the target object, or the description information of the target object.

[0093] If the input task type is the fourth task type (such as a QA task), the first target image frame set can output the image features of the target object through an inference model (such as a Grounding DINO model), and then concatenate the image features with the second task prompts (such as question prompts) corresponding to the fourth task type and input them into the third model (such as an OPT model) to obtain answer information.

[0094] If the input task type is the fifth task type, then the first target image frame set can output the category information of the target object and the image features of the target object through the inference model (such as the Grounding DINO model), and then input the category information of the target object and the image features of the target object into a preset decoder (such as a Transformer decoder) to obtain a set of two-dimensional coordinates of human key points or three-dimensional coordinates of human key points, thereby realizing human posture estimation. In another implementation method, the present embodiment can also output two-dimensional coordinates of hand key points and three-dimensional coordinates of hand key points to realize hand posture estimation. It can be seen that the present embodiment can realize switching between different prompts and different tasks based on the diversification of prompt types and task types, meet the needs of supporting multi-tasking, and is conducive to expanding application scenarios.

[0095] Further, in one implementation, in this embodiment, when the task type is a third subtask type task (ROC task) and the prompt type is a third prompt (i.e., no prompt), the inference result obtained by the first inference can be re-input into the inference model as a prompt feature for inference, thereby realizing secondary inference. Specifically, when the task type is a ROC task and the prompt type is no prompt, the inference model outputs the feature information of the detection frame of the target object according to the first target image frame set and the universal file (such as the Universaltoken file), and the feature information is the image feature of the target object. At this time, the image feature is spliced ​​with the first task prompt of the ROC task and input into the preset third model (such as the OPT model) to obtain the category information of the target object, that is, the first inference is completed. Then, the category information of the target object is re-input into the prompt feature inference and cache module to obtain the updated target object prompt feature. The updated target object prompt features and the first target image frame set are re-input into the reasoning model (such as the Grounding DINO model), and the updated image features of the target object are output. Then, the updated image features of the target object are spliced ​​with the first task prompt of the ROC task and then input into the third model (such as the OPT model) again to obtain the updated category information of the target object, that is, the second reasoning is completed. After two reasonings, this embodiment helps to correct the reasoning results obtained by the first reasoning and improve the accuracy of the reasoning.

[0096] Furthermore, combined with Figure 2As shown, the present embodiment also includes a result post-processing module, which can be used to screen and smooth the detection frame of the target object in the reasoning result, so as to further improve the accuracy of the reasoning result. Specifically, the present embodiment first obtains a preset first confidence threshold, and the first confidence threshold includes a confidence threshold of the detection frame of the target object and / or a confidence threshold of the category information of the target object. Then, based on the first confidence threshold, the detection frame of the target object and / or the category information of the target object are screened, and the detection frame of the target object and the category information of the target object whose confidence is greater than the first confidence threshold are retained. In another implementation, the present embodiment also matches the detection frame of the target object of the current target image frame with the detection frame of the target object of the previous target image frame (for example, using Hungarian matching) to obtain a matching score. If the matching score is greater than the preset matching threshold, it means that the detection frame of the target object of the current target image frame is close to the detection frame of the target object of the previous target image frame, and the detection frame of the target object of the current target image frame can be retained.

[0097] If the matching score between the current target image frame and the previous target image frame is less than the matching threshold and the task type is the same, the difference between the matching score and the matching threshold is determined. If the difference is within a preset range, it means that the reasoning result of the current target image frame is similar to but different from the reasoning result of the previous target image frame. In this case, the reasoning result of the previous target image frame can be used as the reasoning result of the current target image frame, thereby ensuring the stability of the output result of the reasoning model.

[0098] Step S400: Obtain the task result according to the inference result of the first target image frame and the second target image frame set.

[0099] Furthermore, the present embodiment further provides a display control module and a result display module, and the display control module is used to issue instructions to the display module. If multiple tasks are inferred to obtain results at the same time, the specific content to be displayed in the display module is selected. The result display module can be used to display the inference results processed by the result post-processing module, and display them locally or remotely after being transmitted through the network. Specifically, when displaying the inference results, the present embodiment obtains the dimension information (such as image size) of each second target image frame set of the first target image frame set and the second target image frame set respectively. Then, for each first target image frame in the first target image frame set, the first coordinate information of the detection frame of the target object in the first target image frame is obtained through the inference model based on the first target image frame, and the second coordinate information of the detection frame of the target object in the second target image frame is determined based on the first coordinate information and the dimension information of the corresponding second target image frame. Specifically, the first coordinate information of the detection frame of the target object in the first target image frame can be multiplied by the image size of the corresponding second target image frame to obtain the second coordinate information of the detection frame of the target object in the second target image frame. Finally, the inference result is displayed on the corresponding second target image frame according to the second coordinate information, thereby realizing the display of the inference result and obtaining the task result.

[0100] In summary, this embodiment realizes the switching of prompt type and task type by receiving prompt switching instructions and task switching instructions, so that the detection of target objects is not limited to a single task, and flexible switching of tasks can be realized, so as to meet the detection needs in complex scenes. In addition, this embodiment displays the inference results obtained by inferring the target object on a low-resolution image frame set on a high-resolution image frame set, which can effectively improve the inference speed and inference effect and save computing resources.

[0101] Based on the above embodiments, the present invention also provides a system supporting multi-tasking processing, such as Figure 3As shown in , the system includes: a video processing module 10, a task prompt control module 20, a model reasoning module 30 and a task result output module 40. Specifically, the video processing module 10 is used to obtain video data, pre-process the video data, and obtain a first target image frame set and a second target image frame set with the same timestamp, and the first target image frame set and the second target image frame set each contain at least one target image frame. The task prompt control module 20 is used to obtain a prompt switching instruction and a task switching instruction, and respectively determine the prompt type corresponding to the prompt switching instruction and the task type corresponding to the task switching instruction. The model reasoning module 30 is used to reason the first target image frame set based on the prompt type and the task type to obtain the reasoning result of the first target image frame. The task result output module 40 is used to obtain the task result based on the reasoning result of the first target image frame and the second target image frame set.

[0102] The working principles of each module in the multi-tasking processing supporting system of this embodiment are the same as the principles of each step in the above method embodiment, and will not be repeated here.

[0103] Each module in the above-mentioned multi-tasking processing supporting system can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the terminal in the form of hardware, or can be stored in the memory in the terminal in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0104] Based on the above embodiment, the present invention further provides a terminal, the principle block diagram of the terminal can be as follows: Figure 4 The terminal may include one or more processors 100 ( Figure 4 Only one is shown in the figure), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a sleep analysis program based on multi-sensor data. When one or more processors 100 execute the computer program 102, the various steps in the embodiment of the sleep analysis method based on multi-sensor data can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of various modules / units in the embodiment of the sleep analysis system based on multi-sensor data can be implemented, which is not limited here.

[0105] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0106] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0107] Those skilled in the art will understand that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0108] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double operational data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for supporting multitasking, characterized in that: The method comprises: Acquire video data, and preprocess the video data to obtain a first target image frame set and a second target image frame set having the same timestamp, wherein the first target image frame set and the second target image frame set each include at least one target image frame; Acquire a prompt switching instruction and a task switching instruction, and respectively determine a prompt type corresponding to the prompt switching instruction and a task type corresponding to the task switching instruction; Reasoning the first target image frame set based on the prompt type and the task type to obtain a reasoning result of the first target image frame; A task result is obtained according to the inference result of the first target image frame and the second target image frame set.

2. The method for supporting multitasking according to claim 1, characterized in that: Preprocessing the video data to obtain a target image frame includes: When the video data is two video streams with the same video content and different resolutions, the two video streams are processed according to the video timestamps of the two video streams to respectively determine a first image frame set and a second image frame set with the same timestamp; The first image frame set and the second image frame set are denoised to obtain the first target image frame set and the second target image frame set.

3. The method for supporting multitasking according to claim 2, characterized in that: Processing the two video streams according to the video timestamps of the two video streams to respectively determine a first image frame set and a second image frame set having the same timestamps, including: When the video timestamps of the two video streams are inconsistent, all video stream pairs of the two video streams and the video timestamps corresponding to all video stream pairs are cached to obtain cached data of all video stream pairs; Determine all cached data pairs with the same video timestamp from the cached data of all video stream pairs; From each of the cached data pairs among all the cached data pairs, determine the cached data pair with the latest timestamp, and use them as the first image frame set and the second image frame set, respectively.

4. The method for supporting multitasking according to any one of claims 1 to 3, characterized in that: Reasoning the first target image frame set based on the prompt type and the task type to obtain a reasoning result of the first target image frame includes: Based on the prompt type, determining a reasoning model corresponding to the prompt type; Based on the task type, the first target image frame set is inferred through the inference model to obtain an inference result corresponding to the task type.

5. The method for supporting multitasking according to claim 4, characterized in that: Based on the prompt type, determining a reasoning model corresponding to the prompt type includes: If the prompt type is the first prompt or the third prompt, determining that the inference model is the first model; If the prompt type is the second prompt, the inference model is determined to be the second model.

6. The method for supporting multitasking according to claim 5, characterized in that: Reasoning the first target image frame set based on the task type through the reasoning model to obtain a reasoning result corresponding to the task type, including: If the task type is the first task type, the first target image frame set obtains a detection frame of the target object and a confidence level of the target object through the inference model; If the task type is the second task type, the first target image frame set obtains mask information of the target object through the inference model; If the task type is the third task type, the first target image frame set obtains image features of the target object through the inference model, and the image features and the first task prompt obtain any one or more of text information of the target object, category information of the target object, or description information of the target object through the third model; If the task type is the fourth task type, the first target image frame set obtains the image features of the target object through the inference model, and the image features and the second task prompt obtain the target object to obtain answer information through the third model; If the task type is the fifth task type, the first target image frame set obtains the category information of the target object and the image features of the target object through the inference model, and the category information of the target object and the image features of the target object are passed through the decoder to obtain a set of two-dimensional coordinates of human body key points or three-dimensional coordinates of human body key points.

7. The method for supporting multitasking according to claim 6, characterized in that: Reasoning the first target image frame set based on the prompt type and the task type to obtain a reasoning result of the first target image frame, further comprising: Obtaining a prompt of a current target image frame in the first target image frame set according to a prompt of a previous target image frame in the first target image frame set; The current target image frame is inferred based on the prompt of the current target image frame and the task type to obtain an inference result of the current target image frame.

8. The method for supporting multitasking according to claim 7, characterized in that: Obtaining a prompt of a current target image frame in the first target image frame set according to a prompt of a previous target image frame in the first target image frame set, comprising: If the prompt of the previous target image frame is not modified, the pre-cached prompt of the previous target image frame is used as the prompt of the current target image frame; If the prompt of the previous target image frame is modified, the modified prompt is used as the prompt of the current target image frame.

9. The method for supporting multitasking according to claim 7, characterized in that: Reasoning the current target image frame based on the prompt type and the task type of the current target image frame to obtain a reasoning result of the current target image frame, further comprising: If the prompt type is the third prompt, a preset universal file is used as a prompt feature to perform reasoning on the first target image frame set.

10. The method for supporting multitasking according to claim 6, characterized in that: If the task type is the third task type, the first target image frame set obtains the image features of the target object through the inference model, and the image features and the first task prompt obtain any one or more of the text information of the target object, the category information of the target object, or the description information of the target object through the third model, including: When the task type is the third subtask type and the prompt type is the third prompt, the first target image frame set obtains image features of the target object through the inference model, and the image features and the first task prompt obtain target object category information through the third model; Based on the category information of the target object, obtaining an updated target object prompt feature; The updated target object prompt features and the first target image frame set are passed through the inference model to obtain updated image features of the target object; The updated image features of the target object and the third subtask type are used through the third model to obtain updated category information of the target object.

11. The method for supporting multitasking according to any one of claims 6 to 10, characterized in that: The method further comprises: Acquire a preset first confidence threshold, where the first confidence threshold includes a confidence threshold of a detection frame of the target object and / or a confidence threshold of category information of the target object; The detection frame of the target object and / or the category information of the target object are filtered based on the first confidence threshold.

12. The method for supporting multitasking according to claim 11, characterized in that: After filtering the detection frame of the target object and / or the category information of the target object based on the first confidence threshold, the method further includes: Matching a detection frame of a target object in the current target image frame with a detection frame of a target object in a previous target image frame to obtain a matching score; If the matching score is greater than a preset matching threshold, the detection frame of the target object of the current target image frame is retained.

13. The method for supporting multitasking according to claim 12, characterized in that: After matching the detection frame of the target object of the current target image frame with the detection frame of the target object of the previous target image frame to obtain a matching score, the method further includes: If the matching score between the current target image frame and the previous target image frame is less than the matching threshold and the task types are the same, determining the difference between the matching score and the matching threshold; If the difference is within a preset range, the inference result of the previous target image frame is used as the inference result of the current target image frame.

14. The method for supporting multitasking according to claim 1 or 13, characterized in that: Obtaining the task result according to the inference result of the first target image frame and the second target image frame set includes: Respectively acquiring dimension information of each first target image frame in the first target image frame set and each second target image frame in the second target image frame set; For each of the first target image frames, first coordinate information of a detection frame of the target object is obtained through an inference model based on the first target image frame, and second coordinate information of the detection frame of the target object in the second target object frame is determined based on the first coordinate information and dimension information of the corresponding second target image frame; The inference result is displayed on a corresponding second target image frame according to the second coordinate information.

15. A system supporting multi-tasking, characterized in that: The system comprises: A video processing module, used for acquiring video data, preprocessing the video data, and obtaining a first target image frame set and a second target image frame set with the same timestamp, wherein the first target image frame set and the second target image frame set both contain at least one target image frame; A task prompt control module, used to obtain a prompt switching instruction and a task switching instruction, and respectively determine a prompt type corresponding to the prompt switching instruction and a task type corresponding to the task switching instruction; A model reasoning module, used for reasoning the first target image frame set based on the prompt type and the task type to obtain a reasoning result of the first target image frame; The task result output module is used to obtain the task result according to the inference result of the first target image frame and the second target image frame set.

16. A terminal, characterized in that: The terminal includes a memory, a processor, and a multi-tasking support program stored in the memory and executable on the processor. When the processor executes the multi-tasking support program, the steps of the multi-tasking support method as described in any one of claims 1 to 13 are implemented.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multi-tasking support program, and when the multi-tasking support program is executed by the processor, the steps of the multi-tasking support method according to any one of claims 1 to 13 are implemented.

Images