Target detection method and device, equipment and medium

Abstract

The embodiment of the invention provides a target detection method and device, equipment and a medium, and is used for solving the problems that time is consumed, tasks with high real-time requirements cannot be processed, a model trained for a specific scene can only recognize targets of limited categories, and the adaptability to a new scene is lacked in the prior art. The electronic equipment determines each target sub-task described by the task in the detection instruction and the association relationship between the target sub-tasks based on the LLM, and determines the target model corresponding to each target sub-task, so that the process does not need to train the model again, and the target detection speed can be improved. And the electronic equipment determines the calling sequence of each target model based on the association relationship of each sub-task, so that target detection can be accurately and effectively carried out, detection can be realized based on task description when a target detection demand exists, the method can effectively adapt to a rapidly changing industrial environment, and the adaptability to a new scene is enhanced.

Description

Technical Field

[0001] The present application relates to the field of model technology, and in particular to a target detection method, device, equipment and medium. Background Art

[0002] In the industrial world, target detection tasks are a key link in ensuring production safety and efficiency. For example, on construction sites, safety helmet hazard detection is an important measure to prevent accidents. In the power industry, monitoring whether power workers in substations are wearing safety protection equipment correctly is also a necessary means to ensure operational safety. With the development of technology, these tasks increasingly rely on machine learning and deep learning models, such as the target detection (You Only Look Once, YOLO) model, which can quickly and accurately identify target objects.

[0003] However, with the continuous emergence of new scenarios, models that can recognize multiple scenario combinations are needed. Traditional solutions usually require retraining the model, but retraining the model is time-consuming, making it impossible to handle tasks with high real-time requirements. In addition, models trained for specific scenarios can often only recognize a limited number of categories of targets and lack the ability to adapt to new scenarios. Summary of the invention

[0004] The embodiments of the present application provide a target detection method, apparatus, device and medium for solving the problems in the related technology that it is time-consuming and cannot process tasks with high real-time requirements, and that the models trained for specific scenarios can often only recognize a limited number of categories of targets and lack the ability to adapt to new scenarios.

[0005] In a first aspect, an embodiment of the present application provides a target detection method, the method comprising:

[0006] Obtain the task description carried in the received detection instruction;

[0007] Inputting the task description into a first large language model (LLM), obtaining each target subtask split by the first LLM and the association relationship between each target subtask; wherein the association relationship includes an execution order and a series-parallel relationship;

[0008] According to the pre-saved correspondence between subtasks and models, the target models corresponding to each target subtask are obtained respectively, and the calling order of each obtained target model is determined based on the association relationship; each target model is called based on the calling order to execute the task to be detected.

[0009] In a second aspect, an embodiment of the present application further provides a target detection device, the device comprising:

[0010] An acquisition module, used to acquire a task description carried in a received detection instruction; input the task description into a first LLM, and acquire each target subtask split by the first LLM and an association relationship between each target subtask; wherein the association relationship includes an execution order and a series-parallel relationship;

[0011] A processing module is used to obtain the target model corresponding to each target subtask according to the pre-saved correspondence between the subtask and the model, determine the calling order of each acquired target model based on the association relationship; and call each target model based on the calling order to execute the task to be detected.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0013] A memory for storing program instructions;

[0014] The processor is used to call the program instructions stored in the memory and execute the steps included in any one of the above-mentioned target detection methods according to the obtained program instructions.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes a method for target detection as described in any one of the above items.

[0016] In an embodiment of the present application, the electronic device obtains the task description carried in the received detection instruction; inputs the task description into the first LLM, obtains each target subtask split by the first LLM and the association relationship between each target subtask; wherein the association relationship includes the execution order and the series-parallel relationship; according to the correspondence relationship between the pre-saved subtasks and the models, respectively obtains the target model corresponding to each target subtask, and determines the call order of each target model obtained based on the association relationship; calls each target model based on the call order to execute the task to be detected. Since in an embodiment of the present application, the electronic device determines the association relationship between each target subtask and each target subtask in the task description in the detection instruction based on the LLM, and determines the target model corresponding to each target subtask, the process does not need to retrain the model, and can improve the speed of target detection. And the electronic device determines the call order of each target model based on the association relationship of each subtask, so that target detection can be performed accurately and effectively. When there is a need for target detection, detection can be achieved based on the task description, which can effectively adapt to the rapidly changing industrial environment and enhance the adaptability to new scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A process diagram of a target detection method provided in an embodiment of the present application;

[0019] Figure 2 A detailed process diagram of target detection provided in an embodiment of the present application;

[0020] Figure 3 A structural diagram of a target detection device provided in an embodiment of the present application;

[0021] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0023] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0024] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0025] The words "connected", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects, and do not represent a specific ordering of objects.

[0026] In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0027] In order to identify multiple categories of targets and improve the ability to adapt to new scenarios, the embodiments of the present application provide a target detection method, device, equipment and medium.

[0028] The target detection method includes: an electronic device obtains a task description carried in a received detection instruction; inputs the task description into a first LLM, obtains each target subtask split by the first LLM and the association relationship between each target subtask; wherein the association relationship includes an execution order and a series-parallel relationship; according to a pre-saved correspondence between subtasks and models, obtains the target model corresponding to each target subtask respectively, and determines the calling order of each acquired target model based on the association relationship; and calls each target model based on the calling order to execute the task to be detected.

[0029] Embodiment 1:

[0030] Figure 1 A process diagram of a target detection method provided in an embodiment of the present application, the process includes the following steps:

[0031] S101: Obtaining a task description carried in a received detection instruction.

[0032] The target detection method provided in the embodiment of the present application is applied to an electronic device, which may be a smart device such as a PC or a server.

[0033] In order to accurately and effectively perform target detection, when there is a need for target detection, the user can enter a task description on the preset page of the device he is using or the preset device, and click a preset button. For example, by clicking the "Detect" button, the electronic device can receive a detection instruction. After receiving the detection instruction, the electronic device can obtain the task description carried in the received detection instruction. For example, the task description can be "detect whether workers wearing work clothes and safety helmets are working at high altitude on ladders. When there are people working at high altitude, but no one is holding the ladder to ensure safety, output an alarm message."

[0034] S102: Input the task description into the first LLM, and obtain each target subtask split by the first LLM and the association relationship between each target subtask; wherein the association relationship includes the execution order and the series-parallel relationship.

[0035] In order to accurately and effectively perform target detection, the electronic device can input the acquired task description into the first LLM and obtain the output of the first LLM, and the output of the first LLM is each split target subtask and the association relationship between each target subtask. Among them, the association relationship includes the execution order and the series-parallel relationship. The execution order can also be called a dependency relationship.

[0036] Among them, the first LLM can first analyze the user needs in detail through task description, clarify the specific content and goals of the needs, and realize the refinement of the needs. For example, the task description of "detecting whether the staff wearing work clothes and safety helmets are working at high altitude on the ladder, and outputting alarm information when there are people working at high altitude but no one is holding the ladder to ensure safety" can be refined into the following needs: detecting work clothes and safety helmets, detecting ladders and people to determine whether people are working at high altitude on the ladder, determining whether someone is holding the ladder to ensure safety, and outputting alarm information; based on the identified specific content and goals, the tasks are disassembled and multiple large tasks are decomposed into a series of subtasks. For example: Task 1: Detect work clothes and helmets; Subtask 1.1: Image preprocessing; Subtask 1.2: Work clothes detection; Subtask 1.3: Helmet detection; Task 2: Detect ladders and people; Subtask 2.1: Image preprocessing; Subtask 2.2: Ladder detection; Subtask 2.3: Personnel detection; Task 3: Determine whether there are people working at high altitude on ladders; Subtask 3.1: Personnel position analysis; Subtask 3.2: Ladder position analysis; Subtask 3.3: High-altitude work judgment; Task 4: Determine whether there are people holding the ladder to ensure safety; Subtask 4.1: Relative position analysis of personnel and ladder; Subtask 4.2: Safe ladder judgment; Task 5: Output alarm information; Subtask 5.1: Alarm condition judgment; Subtask 5.2: Alarm information generation; Subtask 5.3: Alarm information output. And the first LLM determines the dependency and series-parallel relationship between each target subtask,

[0037] For example, each target subtask in Task 1 and each target subtask in Task 2 can be executed in parallel because they process different detection targets. The first subtask in Task 3 depends on the results of the last subtask in Task 1 and the last subtask in Task 2, and needs to be performed after the person and the ladder are detected. The first subtask in Task 4 depends on the results of the last subtask in Task 2, and needs to be performed after the ladder is detected. The first subtask in Task 5 depends on the results of the last subtask in Task 3 and the last subtask in Task 4, and needs to be performed after the high-altitude operation and safety escalator conditions are determined.

[0038] S103: According to the pre-saved correspondence between subtasks and models, respectively obtain the target model corresponding to each target subtask, determine the calling order of each acquired target model based on the association relationship; call each target model based on the calling order to execute the task to be detected.

[0039] In order to perform target detection accurately and effectively, the electronic device pre-stores the correspondence between subtasks and models locally, where the correspondence between subtasks and models can be saved in a structured model document. The electronic device can obtain the target model corresponding to each target subtask based on the pre-stored correspondence between subtasks and models.

[0040] In one possible implementation, a target subtask may correspond to multiple models. The electronic device can determine the target model that meets the detection requirement in each model corresponding to the target subtask based on the detection requirement carried in the received detection instruction. For example, if the detection requirement is that the detection speed is lower than a threshold, then the model with the fastest detection speed in each model corresponding to the target subtask can be obtained as the target model. If there is no detection requirement in the detection instruction, then a model can be randomly determined as the target model from the multiple models corresponding to the target subtask.

[0041] After obtaining the target model corresponding to each target subtask, the electronic device can further determine their calling order based on the association between these models. This order is determined based on the logical dependencies and process requirements between the subtasks to ensure that the models can be called in the correct order to complete the entire task to be detected.

[0042] The method provided in the embodiment of the present application can reduce manual intervention, reduce costs, and improve the response speed to new scenarios by automating this process. In addition, this method can also improve the generalization ability of the model, so that it can adapt to changing industrial environments and needs. In this way, existing model resources can be used more effectively, rather than training the model from scratch for each new scenario, thereby achieving optimal configuration and utilization of resources.

[0043] Since in the embodiment of the present application, the electronic device determines each target subtask and the association between each target subtask in the task description in the detection instruction based on LLM, and determines the target model corresponding to each target subtask, this process does not require retraining the model, and the speed of target detection can be improved. And the electronic device determines the calling order of each target model based on the association between each subtask, so that target detection can be performed accurately and effectively. When there is a need for target detection, detection can be achieved based on the task description, which can effectively adapt to the rapidly changing industrial environment and enhance the adaptability to new scenarios.

[0044] Embodiment 2:

[0045] In order to accurately and effectively determine the target model, based on the above embodiment, in the embodiment of the present application, after obtaining each target subtask split by the first LLM and the association relationship between each target subtask, and before respectively obtaining the target model corresponding to each target subtask according to the pre-saved correspondence between the subtask and the model, the method further includes:

[0046] Input the task description, each target subtask and the association relationship between each target subtask into the second LLM, and obtain the result of whether the second LLM outputs a reasonable result; wherein a reasonable result indicates that each target subtask and the association relationship between each target subtask are reasonable, and an unreasonable result indicates that there is an unreasonable target subtask in each target subtask or there is an unreasonable association relationship between each target subtask;

[0047] If the result output by the second LLM is reasonable, the subsequent step of obtaining the target model corresponding to each target subtask is performed according to the pre-saved correspondence between the subtask and the model.

[0048] Since each target subtask split by the first LLM or the association relationship between each target subtask may be unreasonable, if each target subtask split by the first LLM or the association relationship between each target subtask is unreasonable, the target model cannot be accurately and effectively determined subsequently, and the task to be detected cannot be executed. Based on this, the electronic device can determine whether each target subtask and the association relationship between each target subtask is reasonable based on the second LLM.

[0049] Specifically, the electronic device can input the task description, each target subtask, and the direct association relationship between each target subtask into the second LLM, and obtain the result of whether the second LLM outputs a reasonable result. Among them, a reasonable result means that each target subtask and the association relationship between each target subtask are reasonable, and an unreasonable result means that there is an unreasonable target subtask in each target subtask, or there is an unreasonable association relationship between the association relationships of each target subtask.

[0050] If the result of the second LLM output is reasonable, it can be determined that each target subtask currently divided and the association relationship between each target subtask are reasonable. The electronic device can execute subsequent steps to obtain the target model corresponding to each target subtask according to the pre-saved correspondence between the subtasks and models.

[0051] Embodiment 3:

[0052] In order to accurately and effectively determine the target model, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0053] If the result output by the second LLM is unreasonable, obtaining unreasonable description information output by the second LLM;

[0054] Input the description information, the task description, each target subtask and the association relationship between each target subtask into a third LLM, and obtain the association relationship between each subtask and each subtask output by the third LLM;

[0055] The association relationship between each subtask and each subtask output by the third LLM is used to update the association relationship between each target subtask and each target subtask.

[0056] In order to accurately and effectively determine the target model, if the result output by the second LLM is unreasonable, the second LLM will also provide additional unreasonable description information, which is used to explain the unreasonable reason. For example, the description information may be the unreasonableness of the determined subtask 1.1.

[0057] In order to improve the accuracy of target model determination, the electronic device can input the description information, the task description in the received detection instruction, each target subtask and the relationship between each target subtask into a third LLM, and obtain the output of the third LLM. The output of the third LLM is each subtask after correction and the relationship between each subtask.

[0058] The electronic device uses the association relationship between each subtask and each subtask output by the third LLM to update the association relationship between each target subtask and each target subtask.

[0059] In an embodiment of the present application, the second LLM is used to detect whether each target subtask of the plan and the relationship between each target subtask are reasonable and whether they can meet the task requirements corresponding to the task description. Moreover, based on the third LLM, each split target subtask and the relationship between each target subtask are further optimized to improve the accuracy of the target model determination.

[0060] Figure 2 A detailed process diagram of target detection provided in an embodiment of the present application includes the following steps:

[0061] S201: Obtain the task description carried in the received detection instruction.

[0062] S202: Input the task description into the first LLM, and obtain each target subtask split by the first LLM and the association relationship between each target subtask.

[0063] S203: Input the task description, each target subtask and the association relationship between each target subtask into the second LLM.

[0064] S204: Determine whether the result output by the second LLM is reasonable, if so, execute S205, if not, execute S207.

[0065] S205: Pre-store the correspondence between subtasks and models, obtain the target model corresponding to each target subtask, and execute S206.

[0066] S206: Determine the calling order of each acquired target model based on the association relationship; call each target model based on the calling order to execute the task to be detected, and end.

[0067] S207: Obtain unreasonable description information output by the second LLM.

[0068] S208: Input the description information, the task description, each target subtask and the association relationship between each target subtask into the third LLM, and obtain each subtask and the association relationship between each subtask output by the third LLM.

[0069] S209: using the association relationship between each subtask and each subtask output by the third LLM, updating the association relationship between each target subtask and each target subtask, and executing S205.

[0070] Embodiment 4:

[0071] In order to execute the task to be detected, based on the above embodiments, in the embodiment of the present application, calling each target model based on the calling sequence includes:

[0072] For each target model, obtain the interface name, input parameter, output parameter and identification type saved for the target model; input the interface name, input parameter, output parameter and identification type into the fourth LLM, and obtain the calling code of the target model output by the fourth LLM; wherein the calling code is used to realize the loading of the target model;

[0073] Input the calling sequence, the implementation code of the corresponding target model saved for each target model, and the calling code of each target model into the fifth LLM, and obtain the target implementation code of the total model composed of each target model based on the calling sequence output by the fifth LLM; wherein the implementation code of each target model is used to run and realize the recognition function of the corresponding target model;

[0074] Run the target implementation code.

[0075] In order to perform the task to be detected, the electronic device can extract the key information of each target model from the preset data storage or configuration file, including the interface name, input parameters, output parameters and recognition type of the target model, where the recognition type of the model is such as image recognition, natural language processing, etc. This information is the basis for model integration and calling.

[0076] For each target model, the electronic device can generate a call code for the target model based on the information of the target model obtained. Specifically, the electronic device can pass the obtained interface name, input parameters, output parameters and identification type as input to the fourth LLM. LLM4 intelligently generates a call code for each target model based on this information. This call code is responsible for loading the corresponding target model at runtime to ensure that the model can be correctly integrated into a larger application or process.

[0077] Moreover, after obtaining the calling code of each target model, the electronic device further organizes the specific implementation code of each target model. These implementation codes are the core of the model to perform its recognition function, and they contain all the logic and operations required for the model to run. Then, the electronic device inputs the calling order of each target model, wherein the calling order defines the execution order of each target model in the overall process, each corresponding implementation code, and the calling code of each target model, into the fifth LLM. The task of the fifth LLM is to intelligently determine the target implementation code of the total model based on this information. Specifically, the fifth LLM connects the entire link in series, and for subtasks with dependencies, generates sequential execution code according to the execution order corresponding to the dependencies. For subtasks that can be executed in parallel, multi-threading or asynchronous programming techniques are used to improve efficiency. The target implementation code of the total model follows the specified calling order, and can load and execute each target model in sequence, thereby completing the entire recognition task or data processing process.

[0078] After generating the target implementation code of the total model, the electronic device may need to perform additional verification steps to ensure the correctness and efficiency of the code. Specifically, the code-debug-recode mode can be used to automatically run tests on the sight code generated by the fifth LLM until the code passes, and finally the complete running code can be obtained.

[0079] Once the target implementation code of the overall model has been verified, the electronic device can execute the target implementation code in the appropriate operating environment. This will trigger the entire process, load and run each target model in a predefined calling order, and finally complete the entire recognition or data processing task.

[0080] It should be noted that the interface name, input parameters, output parameters and identification type saved for the target model described in the embodiment of the present application can be saved in the model document. The electronic device can structure the model document before the target detection begins, specifically using LLM to extract information from the implementation code or API of the existing model, mainly including: Interface name: clarify the name of each interface. Interface description: provide a brief description of the interface to explain its function and purpose. Input parameters: list the input parameters of each interface, including parameter name, type, description and whether it is required. Output parameters: list the output parameters of each interface, including parameter name, type, description. Identify incoming attacks: clarify the detection categories supported by the interface, such as safety helmets, ladders, etc. Example code: provide example code for each interface to show how to call the interface. Error handling: describe possible error responses and exception handling methods. Version control: record the version information of the interface to track changes. The documents of all models are structured, and the electronic equipment adds the functions of regular update and automated testing. The documents are checked and updated regularly to reflect the latest status of the model. Automated testing tools are used to ensure that the documents are consistent with the actual code and API, and unusable models or interfaces are removed.

[0081] The method provided by the embodiment of the present application realizes efficient integration of the target model.

[0082] It should be noted that in an embodiment of the present application, the electronic device can also use LLM to optimize the target implementation code of the overall model, input the target implementation code into the LLM, use LLM to analyze the target implementation code, and identify which parts are repeated calls to the same model for different category detections. For example, if it is found that the code has multiple calls to the same model for different category detections, LLM optimizes these codes, merges multiple calls into one call, and detects multiple categories at the same time by modifying the input data. For code parts that can be executed in parallel, LLM can help identify and optimize these parts to improve the execution efficiency of the program. The electronic device can deploy the optimized code to a test environment and monitor its performance in the environment to ensure the stability and performance of the code.

[0083] Embodiment 5:

[0084] In order to accurately and effectively determine the target model, based on the above embodiments, in the embodiment of the present application, after respectively obtaining the target model corresponding to each target subtask and before calling each target model based on the calling sequence, the method further includes:

[0085] If a target subtask is not included in the pre-saved correspondence between subtasks and models, the preset first multimodal large model is determined as the target model corresponding to the target subtask.

[0086] In actual scenarios, the model corresponding to a target subtask may not be saved, and the target model corresponding to the target subtask cannot be obtained at this time, resulting in the problem of being unable to execute the task to be detected. In order to solve this problem, if the correspondence between the pre-saved subtasks and models does not include a target subtask, the preset first multimodal large model is determined as the target model corresponding to the target subtask. The first multimodal large model can be GPT-4V

[0087] It should be noted that if the first multimodal large model is determined as the target model corresponding to a target subtask, the electronic device can determine the input parameters, output parameters and recognition type saved for the target subtask as the input parameters, output parameters and recognition type of the first multimodal large model, and obtain the interface name saved for the first multimodal large model, input the obtained interface name, input parameters, output parameters and recognition type into the fourth LLM, and obtain the calling code when the first multimodal large model output by the fourth LLM executes the target subtask.

[0088] Embodiment 6:

[0089] In order to accurately and effectively perform target detection, based on the above embodiments, in the embodiment of the present application, after calling each target model based on the calling sequence, the method further includes:

[0090] When executing the task to be detected, if the output result of a target model is obtained, the input of the target model and the output result are input into the second multimodal large model, and the score of the output result by the second multimodal large model is obtained; if the score is higher than a threshold, the target model is fine-tuned based on the input and the output result.

[0091] In order to accurately and effectively detect targets, electronic devices can fine-tune the target model. Specifically, when executing the task to be detected, if the electronic device obtains the output result of a certain target model, the input and output result of the target model can be input into a pre-saved second multimodal large model. The second multimodal large model is used to evaluate the output result. The electronic device can obtain the output of the second multimodal large model, which is the score for the output result. If the score is higher than the pre-saved threshold, it can be determined that the output result is relatively accurate. The electronic device can use the input and the output result as samples to fine-tune the target model. Specifically, the electronic device can input the input into the target model, obtain the actual output result of the target model, and then fine-tune and optimize the target model based on the difference between the actual output and the output result.

[0092] The method provided in the embodiment of the present application is equivalent to continuously collecting data online, reviewing the annotation results of the target model through a multimodal large model, selecting high-quality data, fine-tuning the target model of the target subtask, and subsequently using the optimized target model to replace the complex pipeline to improve efficiency.

[0093] Embodiment 7:

[0094] In order to accurately and effectively perform target detection, based on the above embodiments, in the embodiment of the present application, after obtaining each target subtask split by the first LLM and the association relationship between each target subtask, the method further includes:

[0095] Input each target subtask and the association relationship between each target subtask into the sixth LLM, and obtain a task planning graph output by the sixth LLM; wherein the node in the task planning graph is each subtask, and the connection relationship between each node is the association relationship between the corresponding target subtasks;

[0096] Generate a visual planning diagram corresponding to the task planning diagram based on a preset graphical tool; and display the visual planning diagram.

[0097] In order to facilitate the user to view and determine each target subtask and the association relationship between each target subtask, the electronic device can display a visual planning diagram that identifies each target subtask and the association relationship between each target subtask.

[0098] Specifically, the electronic device can input each acquired target subtask and the association relationship between each target subtask into the sixth LLM, and obtain the output of the sixth LLM, which is the task planning diagram. The task planning diagram is not only an intuitive display of the task structure, but also a deep revelation of the task execution process and logical relationship. Each node in the task planning diagram accurately corresponds to a target subtask, and the connection relationship corresponding to the connection line between the nodes is the association relationship between the corresponding subtasks, that is, the logical order, dependency relationship, and possible parallel processing path.

[0099] In order to visualize the task planning map, the electronic device uses a preset graphical tool to convert the task planning map into a visual planning map, wherein the preset graphical tool may be Mermaid or Graphviz. The electronic device displays the visual planning map through its own display screen or a display screen connected to it, and presents the visual planning map to the user. It can help the user quickly understand the composition and process of the entire task.

[0100] The embodiment of the present application is equivalent to providing a method for constructing an adaptive workflow for industrial visual tasks based on an intelligent agent, which is equivalent to automating the orchestration of the workflow, wherein the workflow is the target model described in the embodiment of the present application.

[0101] It should be noted that in target detection tasks in the industrial sector, such as helmet hazard detection and unmanned escalator scene monitoring, it is usually necessary to train recognition models of specific categories. These recognition models can often only recognize limited categories. For example, a helmet detection model may only recognize heads and helmets, while an escalator scene monitoring model may only recognize people and ladders. With the emergence of new scenarios, models that can recognize multi-scenario combination categories are needed, but existing methods usually require retraining models or experts to build multi-model call workflows, which is not only time-consuming but also inefficient. The technical problem that the present invention aims to solve is how to automatically build an intelligent agent that can automatically perform task decomposition, determine the dependencies between tasks, and the goals required for each task when new requirements arise. It will automatically search algorithm documents, find the optimal model, and automatically arrange calls between multiple models to form code, and finally test and deploy. Such an intelligent agent will greatly improve the flexibility and efficiency of industrial target detection tasks, reduce dependence on experts, and speed up the deployment of models in new scenarios.

[0102] The technical problem to be solved by the embodiments of the present application is to solve the following shortcomings of the prior art in target detection tasks for specific industrial scenarios:

[0103] Limited model generalization ability: In the prior art, models trained for specific scenarios can often only recognize a limited number of categories of targets and lack the ability to adapt to new scenarios. The embodiments of the present application use adaptive scene recognition technology to enable the intelligent agent to quickly adapt to new scenarios and improve the generalization ability of the model.

[0104] Low efficiency of model selection and fusion: In multi-scenario target detection, it is necessary to select the optimal model from a large number of models and fuse them. Existing technologies often rely on manual selection and fusion, which is inefficient. The embodiments of the present application use dynamic model selection and fusion technology to automatically select and fuse the optimal model, thereby improving the accuracy and robustness of detection.

[0105] Complex workflow construction: In the prior art, the calling workflow of multiple models needs to be manually constructed by experts, which is cumbersome and time-consuming. The embodiment of the present application uses automated workflow construction technology, and the intelligent agent can automatically perform task decomposition, dependency determination, target identification and model selection, simplifying the workflow construction process.

[0106] Embodiment 8:

[0107] Based on the same inventive concept, the present application embodiment provides a joint learning aggregation device, please refer to Figure 3 , the device comprises:

[0108] The acquisition module 301 is used to acquire the task description carried in the received detection instruction; input the task description into the first large language model LLM, and acquire each target subtask split by the first LLM and the association relationship between each target subtask; wherein the association relationship includes the execution order and the series-parallel relationship;

[0109] The processing module 302 is used to obtain the target model corresponding to each target subtask according to the pre-saved correspondence between the subtask and the model, and determine the calling order of each acquired target model based on the association relationship; call each target model based on the calling order to execute the task to be detected.

[0110] In a possible implementation, the processing module 302 is also used to input the task description, each target subtask and the relationship between each target subtask into the second LLM to obtain whether the result output by the second LLM is reasonable; wherein a reasonable result indicates that each target subtask and the relationship between each target subtask are reasonable, and an unreasonable result indicates that there are unreasonable target subtasks in each target subtask or there are unreasonable relationships between each target subtask; if the result output by the second LLM is reasonable, then the subsequent steps of respectively obtaining the target model corresponding to each target subtask according to the correspondence between the pre-saved subtasks and the model are executed.

[0111] In a possible implementation, the processing module 302 is also used to obtain unreasonable description information of the second LLM output if the result of the second LLM output is unreasonable; input the description information, the task description, each target subtask and the association relationship between each target subtask into the third LLM to obtain the association relationship between each subtask and each subtask output by the third LLM; use the association relationship between each subtask and each subtask output by the third LLM to update the association relationship between each target subtask and each target subtask.

[0112] In a possible implementation, the processing module 302 is specifically used to obtain, for each target model, the interface name, input parameters, output parameters and identification type saved for the target model; input the interface name, the input parameters, the output parameters and the identification type into the fourth LLM, and obtain the calling code of the target model output by the fourth LLM; wherein the calling code is used to implement the loading of the target model; input the calling sequence, the implementation code of the corresponding target model saved for each target model, and the calling code of each target model into the fifth LLM, and obtain the target implementation code of the total model composed of the calling sequence for each target model output by the fifth LLM; wherein the implementation code of each target model is used to run and implement the identification function of the corresponding target model; and run the target implementation code.

[0113] In a possible implementation, the processing module 302 is further configured to determine the preset first multimodal large model as the target model corresponding to the target subtask if a target subtask is not included in the pre-saved correspondence between subtasks and models.

[0114] In a possible implementation, the processing module 302 is also used to, when executing the task to be detected, if an output result of a target model is obtained, input the input of the target model and the output result into a second multimodal large model, and obtain a score of the output result from the second multimodal large model; if the score is higher than a threshold, fine-tune the target model based on the input and the output result.

[0115] In a possible implementation, the processing module 302 is also used to input each target subtask and the association relationship between each target subtask into the sixth LLM to obtain a task planning diagram output by the sixth LLM; wherein the nodes in the task planning diagram are each subtask, and the connection relationship between each node is the association relationship between the corresponding target subtasks; generate a visual planning diagram corresponding to the task planning diagram based on a preset graphical tool; and display the visual planning diagram.

[0116] Embodiment 9:

[0117] Based on the same inventive concept, the present application embodiment provides an electronic device, which can implement the function of the above intention update, please refer to Figure 4 The device includes a processor 401, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0118] The memory 403 stores a computer program. When the program is executed by the processor 401, the processor 401 performs the following steps:

[0119] Obtain the task description carried in the received detection instruction;

[0120] Input the task description into the first LLM, and obtain each target subtask split by the first LLM and the association relationship between each target subtask; wherein the association relationship includes the execution order and the series-parallel relationship;

[0121] According to the pre-saved correspondence between subtasks and models, the target models corresponding to each target subtask are obtained respectively, and the calling order of each obtained target model is determined based on the association relationship; each target model is called based on the calling order to execute the task to be detected.

[0122] In a possible implementation, after obtaining each target subtask split by the first LLM and the association relationship between each target subtask, and before respectively obtaining the target model corresponding to each target subtask according to the pre-saved correspondence between the subtask and the model, the method further includes:

[0123] Input the task description, each target subtask and the association relationship between each target subtask into the second LLM, and obtain the result of whether the second LLM outputs a reasonable result; wherein a reasonable result indicates that each target subtask and the association relationship between each target subtask are reasonable, and an unreasonable result indicates that there is an unreasonable target subtask in each target subtask or there is an unreasonable association relationship between each target subtask;

[0124] If the result output by the second LLM is reasonable, the subsequent step of obtaining the target model corresponding to each target subtask is performed according to the pre-saved correspondence between the subtask and the model.

[0125] In a possible implementation, the method further includes:

[0126] If the result output by the second LLM is unreasonable, obtaining unreasonable description information output by the second LLM;

[0127] Input the description information, the task description, each target subtask and the association relationship between each target subtask into a third LLM, and obtain the association relationship between each subtask and each subtask output by the third LLM;

[0128] The association relationship between each subtask and each subtask output by the third LLM is used to update the association relationship between each target subtask and each target subtask.

[0129] In a possible implementation, calling each target model based on the calling sequence includes:

[0130] For each target model, obtain the interface name, input parameter, output parameter and identification type saved for the target model; input the interface name, input parameter, output parameter and identification type into the fourth LLM, and obtain the calling code of the target model output by the fourth LLM; wherein the calling code is used to realize the loading of the target model;

[0131] Input the calling sequence, the implementation code of the corresponding target model saved for each target model, and the calling code of each target model into the fifth LLM, and obtain the target implementation code of the total model composed of each target model based on the calling sequence output by the fifth LLM; wherein the implementation code of each target model is used to run and realize the recognition function of the corresponding target model;

[0132] Run the target implementation code.

[0133] In a possible implementation, after respectively acquiring the target model corresponding to each target subtask and before calling each target model based on the calling sequence, the method further includes:

[0134] If a target subtask is not included in the pre-saved correspondence between subtasks and models, the preset first multimodal large model is determined as the target model corresponding to the target subtask.

[0135] In a possible implementation, after calling each target model based on the calling sequence, the method further includes:

[0136] When executing the task to be detected, if the output result of a target model is obtained, the input of the target model and the output result are input into the second multimodal large model, and the score of the output result by the second multimodal large model is obtained; if the score is higher than a threshold, the target model is fine-tuned based on the input and the output result.

[0137] In a possible implementation manner, after obtaining each target subtask split by the first LLM and the association relationship between each target subtask, the method further includes:

[0138] Input each target subtask and the association relationship between each target subtask into the sixth LLM, and obtain a task planning graph output by the sixth LLM; wherein the node in the task planning graph is each subtask, and the connection relationship between each node is the association relationship between the corresponding target subtasks;

[0139] Generate a visual planning diagram corresponding to the task planning diagram based on a preset graphical tool; and display the visual planning diagram.

[0140] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0141] The communication interface 402 is used for communication between the electronic device and other devices.

[0142] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0143] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0144] Embodiment 10:

[0145] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device implements the following steps when executing:

[0146] Obtain the task description carried in the received detection instruction;

[0147] Input the task description into the first LLM, and obtain each target subtask split by the first LLM and the association relationship between each target subtask; wherein the association relationship includes the execution order and the series-parallel relationship;

[0148] According to the pre-saved correspondence between subtasks and models, the target models corresponding to each target subtask are obtained respectively, and the calling order of each obtained target model is determined based on the association relationship; each target model is called based on the calling order to execute the task to be detected.

[0149] In a possible implementation, after obtaining each target subtask split by the first LLM and the association relationship between each target subtask, and before respectively obtaining the target model corresponding to each target subtask according to the pre-saved correspondence between the subtask and the model, the method further includes:

[0150] Input the task description, each target subtask and the association relationship between each target subtask into the second LLM, and obtain the result of whether the second LLM outputs a reasonable result; wherein a reasonable result indicates that each target subtask and the association relationship between each target subtask are reasonable, and an unreasonable result indicates that there is an unreasonable target subtask in each target subtask or there is an unreasonable association relationship between each target subtask;

[0151] If the result output by the second LLM is reasonable, the subsequent step of obtaining the target model corresponding to each target subtask is performed according to the pre-saved correspondence between the subtask and the model.

[0152] In a possible implementation, the method further includes:

[0153] If the result output by the second LLM is unreasonable, obtaining unreasonable description information output by the second LLM;

[0154] Input the description information, the task description, each target subtask and the association relationship between each target subtask into a third LLM, and obtain the association relationship between each subtask and each subtask output by the third LLM;

[0155] The association relationship between each subtask and each subtask output by the third LLM is used to update the association relationship between each target subtask and each target subtask.

[0156] In a possible implementation, calling each target model based on the calling sequence includes:

[0157] For each target model, obtain the interface name, input parameter, output parameter and identification type saved for the target model; input the interface name, input parameter, output parameter and identification type into the fourth LLM, and obtain the calling code of the target model output by the fourth LLM; wherein the calling code is used to realize the loading of the target model;

[0158] Input the calling sequence, the implementation code of the corresponding target model saved for each target model, and the calling code of each target model into the fifth LLM, and obtain the target implementation code of the total model composed of each target model based on the calling sequence output by the fifth LLM; wherein the implementation code of each target model is used to run and realize the recognition function of the corresponding target model;

[0159] Run the target implementation code.

[0160] In a possible implementation, after respectively acquiring the target model corresponding to each target subtask and before calling each target model based on the calling sequence, the method further includes:

[0161] If a target subtask is not included in the pre-saved correspondence between subtasks and models, the preset first multimodal large model is determined as the target model corresponding to the target subtask.

[0162] In a possible implementation, after calling each target model based on the calling sequence, the method further includes:

[0163] When executing the task to be detected, if the output result of a target model is obtained, the input of the target model and the output result are input into the second multimodal large model, and the score of the output result by the second multimodal large model is obtained; if the score is higher than a threshold, the target model is fine-tuned based on the input and the output result.

[0164] In a possible implementation manner, after obtaining each target subtask split by the first LLM and the association relationship between each target subtask, the method further includes:

[0165] Input each target subtask and the association relationship between each target subtask into the sixth LLM, and obtain a task planning graph output by the sixth LLM; wherein the node in the task planning graph is each subtask, and the connection relationship between each node is the association relationship between the corresponding target subtasks;

[0166] Generate a visual planning diagram corresponding to the task planning diagram based on a preset graphical tool; and display the visual planning diagram.

[0167] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0168] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0171] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A target detection method, characterized in that: The method comprises: Obtain the task description carried in the received detection instruction; Input the task description into a first large language model LLM, and obtain each target subtask split by the first LLM and the association relationship between each target subtask; wherein the association relationship includes an execution order and a series-parallel relationship; According to the pre-saved correspondence between subtasks and models, the target models corresponding to each target subtask are obtained respectively, and the calling order of each obtained target model is determined based on the association relationship; each target model is called based on the calling order to execute the task to be detected.

2. The method according to claim 1, characterized in that After obtaining each target subtask split by the first LLM and the association relationship between each target subtask, and before respectively obtaining the target model corresponding to each target subtask according to the pre-saved correspondence relationship between the subtask and the model, the method further includes: Input the task description, each target subtask and the association relationship between each target subtask into the second LLM, and obtain the result of whether the second LLM outputs a reasonable result; wherein a reasonable result indicates that each target subtask and the association relationship between each target subtask are reasonable, and an unreasonable result indicates that there is an unreasonable target subtask in each target subtask or there is an unreasonable association relationship between each target subtask; If the result output by the second LLM is reasonable, the subsequent step of obtaining the target model corresponding to each target subtask is performed according to the pre-saved correspondence between the subtask and the model.

3. The method according to claim 2, characterized in that The method further comprises: If the result output by the second LLM is unreasonable, obtaining unreasonable description information output by the second LLM; Input the description information, the task description, each target subtask and the association relationship between each target subtask into a third LLM, and obtain the association relationship between each subtask and each subtask output by the third LLM; The association relationship between each subtask and each subtask output by the third LLM is used to update the association relationship between each target subtask and each target subtask.

4. The method according to claim 1, characterized in that The calling each target model based on the calling sequence includes: For each target model, obtain the interface name, input parameter, output parameter and identification type saved for the target model; input the interface name, input parameter, output parameter and identification type into the fourth LLM, and obtain the calling code of the target model output by the fourth LLM; wherein the calling code is used to realize the loading of the target model; Input the calling sequence, the implementation code of the corresponding target model saved for each target model, and the calling code of each target model into the fifth LLM, and obtain the target implementation code of the total model composed of each target model based on the calling sequence output by the fifth LLM; wherein the implementation code of each target model is used to run and realize the recognition function of the corresponding target model; Run the target implementation code.

5. The method according to claim 1, characterized in that After respectively acquiring the target model corresponding to each target subtask and before calling each target model based on the calling sequence, the method further includes: If a target subtask is not included in the pre-saved correspondence between subtasks and models, the preset first multimodal large model is determined as the target model corresponding to the target subtask.

6. The method according to claim 1, characterized in that After calling each target model based on the calling sequence, the method further includes: When executing the task to be detected, if the output result of a target model is obtained, the input of the target model and the output result are input into the second multimodal large model, and the score of the output result by the second multimodal large model is obtained; if the score is higher than a threshold, the target model is fine-tuned based on the input and the output result.

7. The method according to claim 1, characterized in that After obtaining each target subtask split by the first LLM and the association relationship between each target subtask, the method further includes: Input each target subtask and the association relationship between each target subtask into the sixth LLM, and obtain a task planning graph output by the sixth LLM; wherein the node in the task planning graph is each subtask, and the connection relationship between each node is the association relationship between the corresponding target subtasks; Generate a visual planning diagram corresponding to the task planning diagram based on a preset graphical tool; and display the visual planning diagram.

8. A target detection device, characterized in that: The device comprises: An acquisition module is used to acquire a task description carried in a received detection instruction; input the task description into a first large language model LLM, and acquire each target subtask split by the first LLM and an association relationship between each target subtask; wherein the association relationship includes an execution order and a series-parallel relationship; A processing module is used to obtain the target model corresponding to each target subtask according to the pre-saved correspondence between the subtask and the model, determine the calling order of each acquired target model based on the association relationship; and call each target model based on the calling order to execute the task to be detected.

9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in the method for target detection according to any one of claims 1 to 7 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the steps included in the method for target detection according to any one of claims 1 to 7.

Images