Multi-agent intelligent video image analysis method and system

Abstract

The invention provides an intelligent video image analysis method and system under multiple agents. The method comprises the following steps: receiving image data and user natural language description; defining function descriptions of different recognition agents, wherein the recognition agents are agents packaged with various recognition cues for a large model; the method comprises the following steps: acquiring a first user cue word used for analyzing agent capability description, integrating the first user cue word with function description of different recognition agents, inputting the first user cue word, natural language description and image data into a large model, returning to the recognition agent needing to be called, analyzing the image data, iteratively reflecting an analysis result for multiple times, and giving a reliability score. Obtaining an analysis result of which the reliability score is greater than a preset credible threshold value; and converting analysis of which the reliability score is greater than a credible threshold value into natural language description, and forming a report in combination with image data. The recognition and analysis capability of the model in a complex and changeable scene is enhanced, the robustness of the model is improved, and it is ensured that high-precision performance can be kept under various conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of large models, and in particular to an intelligent video image analysis method and system under multi-agent conditions. Background Art

[0002] In today's highly digital and information-based society, video image analysis technology, as an indispensable part of many fields such as intelligent security, automated monitoring, and traffic management, is facing unprecedented challenges and opportunities. Although traditional video image analysis systems, especially recognition and detection technologies based on deep learning, have shown excellent performance in specific scenarios, they still have many limitations in practical applications.

[0003] On the one hand, the traditional methods are highly dependent on large-scale annotated data sets, which not only consumes a lot of manpower and time resources, but also seriously restricts the system's rapid deployment and ability to adapt to new scenarios. On the other hand, the generalization ability and robustness of deep learning models are insufficient. Faced with complex and changing real-world environments and customized requirements, such as changes in lighting conditions, diversity of target postures, occlusion and changes in perspective, they often show a significant decrease in recognition accuracy. In addition, the high consumption of computing resources, the "black box" effect of the model (i.e., the opacity of the decision-making process) and the high cost of updating and maintenance have also become bottlenecks that limit its widespread application.

[0004] In the field of intelligent security, the above limitations are particularly prominent, which seriously hinders the improvement of system performance and the ability to respond quickly to new threats. In the face of this series of challenges, an innovative technical solution is urgently needed that can achieve rapid adaptation, cost control, easy deployment and maintenance, and good interpretability while ensuring high-precision recognition, so as to meet the growing security needs and intelligent trends. Summary of the invention

[0005] In order to solve the technical problems raised by the background technology, the present invention proposes an intelligent video image analysis method and system under multi-agent to solve the above technical problems.

[0006] According to one aspect of the present invention, a multi-agent intelligent video image analysis method is proposed, comprising:

[0007] S1. Receive image data and a natural language description of a user, and pre-process the image data and the natural language description; wherein the image data is at least one of a video and a picture;

[0008] S2. Define the functional descriptions of different recognition agents, where the recognition agents are agents that encapsulate scene recognition, behavior recognition, emotion recognition, action recognition, portrait recognition, and object recognition prompt words for large models;

[0009] S3, obtaining a first user prompt word for analyzing the description of the ability of the agent, integrating the first user prompt word with the functional descriptions of different recognition agents, and inputting the first user prompt word into the big model together with the natural language description and the image data, using the big model to return the recognition agent to be called, and using the recognition agent to be called to analyze the image data to obtain the analysis result;

[0010] S4. Perform multiple iterations of reflection on the analysis results, assign reliability scores, and obtain analysis results whose reliability scores are greater than a preset trustworthy threshold;

[0011] S5. Convert the analysis results whose reliability scores are greater than the preset trust threshold into natural language descriptions, combine them with the image data, and form a report.

[0012] In some specific embodiments, step S1 preprocessing includes obtaining an access link to the image data, concatenating the access link to the image data with a natural language description, and providing the preprocessing output to the large model for use.

[0013] In some specific embodiments, step S3 includes:

[0014] S31, obtaining a first user prompt word for analyzing the description of the agent's capabilities;

[0015] S32, obtaining functional descriptions of different recognition agents;

[0016] S33, integrating the first user prompt word and the functional descriptions of different recognition agents;

[0017] S34, inputting the integrated prompt words, natural language description and image data into the big model, and using the big model to return a list of recognition agents that need to be called;

[0018] S35. Determine whether there is a recognition agent list. If so, traverse and call all recognition agents in the recognition agent list, summarize the recognition results of all recognition agents, and obtain the analysis results. Conduct in-depth analysis of the ability description of the recognition agent to clarify its expertise and data processing capabilities. Accurately interpret the needs expressed by the user in natural language and understand their specific needs. Based on the above analysis, intelligently call the most suitable agent to accurately analyze the provided video or picture.

[0019] In some specific embodiments, the conclusion of step S3 includes text, pictures and video clips.

[0020] In some specific embodiments, step S4 includes:

[0021] S41, obtaining a second user prompt word for reflecting on the description of the intelligent agent's ability;

[0022] S42, initializing the best reliability score to 0;

[0023] S43, inputting the second user prompt word and the analysis result into the big model, using the big model to give the analysis result a reliability score, and increasing the number of reflections once;

[0024] S44, comparing the reliability score of the analysis result with the best reliability score, if it is greater than, updating the best reliability score, and saving the analysis result;

[0025] S45. Determine whether the best reliability score is greater than the trust threshold. If so, return the analysis result whose best reliability score is greater than the trust threshold. Perform multiple iterations of reflection on the analysis results of the analysis agent, give a reliability score, and give the most reliable analysis and analysis basis based on the reliability judgment.

[0026] In some specific embodiments, step S4 includes: if the reliability score of the analysis result is less than the optimal reliability score, or the optimal reliability score is less than the trust threshold, determine whether the reflection upper limit is reached; if the reflection upper limit is reached, merge the analysis result with the image data, and input them together into the recognition agent that needs to be called for analysis, and retrieve the analysis result.

[0027] In some specific embodiments, step S4 includes:

[0028] S46, inputting the natural language description and the analysis result with the best reliability score greater than the trust threshold into the big model, and using the big model to return the target of the user's query, the target including people and events;

[0029] S47. If the object is a person, call the confidence interface to conduct confidence testing on the target and obtain information greater than a preset confidence threshold.

[0030] According to a second aspect of the present invention, a multi-agent intelligent video image analysis system is proposed, the system comprising:

[0031] A data receiving module is configured to receive image data and a natural language description of a user, and pre-process the image data and the natural language description; wherein the image data is at least one of a video and a picture;

[0032] The recognition agent module is configured to define functional descriptions of different recognition agents, where the recognition agents are agents encapsulating scene recognition, behavior recognition, emotion recognition, action recognition, portrait recognition, and object recognition prompt words for the large model;

[0033] The analysis agent module is configured to obtain a first user prompt word for describing the capabilities of the analysis agent, integrate the first user prompt word with the functional descriptions of different recognition agents, input the first user prompt word together with the natural language description and the image data into the large model, use the large model to return the recognition agent to be called, use the recognition agent to be called to analyze the image data, and obtain the analysis result;

[0034] A reflective agent module is configured to perform multiple iterations of reflection on the analysis results, assign reliability scores, and obtain analysis results whose reliability scores are greater than a preset trustworthy threshold;

[0035] The summary agent module is configured to convert the analysis results with a reliability score greater than a preset trust threshold into a natural language description, combined with the image data, to form a report.

[0036] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which one or more computer programs are stored. When the one or more computer programs are executed by a computer processor, any of the above methods is implemented.

[0037] According to a fourth aspect of the present invention, an electronic device is proposed, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0038] The following beneficial effects are achieved through the intelligent video image analysis method and system under multi-agent of the present application:

[0039] 1. Significantly accelerate the ability to adapt to new scenarios: The present invention uses innovative prompt word fine-tuning technology to greatly enhance the system's ability to quickly adapt to new scenarios and new demands, effectively shortening the time cycle from demand identification to system deployment, thereby enabling rapid response to the ever-changing security environment.

[0040] 2. Significantly reduce the dependence on data and computing resources: no longer rely on large-scale annotated data sets, reducing the time and cost of data collection and preprocessing. At the same time, the optimized model design reduces the consumption of computing resources, making it possible to deploy the system on edge devices, and improving the overall application scope and flexibility.

[0041] 3. Enhance the generalization and robustness of the model: Through the multi-agent collaborative architecture and multi-round reflective analysis mechanism, the present invention significantly enhances the model's recognition and analysis capabilities in complex and changing scenarios, improves the model's robustness, and ensures that the system can maintain high-precision performance under various conditions.

[0042] 4. Improve the transparency and explainability of decisions: The automatic generation system of logical chain reports ensures the transparency and explainability of the intelligent decision-making process, improves the credibility of the system and user satisfaction. Especially in the security field, the traceability of decisions is crucial for compliance and responsibility identification.

[0043] 5. Simplify the optimization process and reduce costs: The prompt word tuning algorithm simplifies the system optimization process, avoids lengthy training cycles, significantly reduces the technical threshold and overall cost, and makes high-quality video image analysis technology more popular and practical.

[0044] To sum up, the technical solution of this application not only improves the performance of the video image analysis system, but also greatly improves its practicality, cost-effectiveness and industry influence, providing strong support for technological progress in the fields of intelligent security and so on. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and together with the description are used to explain the principles of the present invention. Other embodiments and many expected advantages of the embodiments will be readily appreciated as they become better understood by reference to the following detailed description. Other features, objects and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings:

[0046] Figure 1 is a flow chart of a multi-agent intelligent video image analysis method according to an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of a description of the recognition agent function of an embodiment of the present application;

[0048] Figure 3 is a schematic diagram of scene prompt words of an embodiment of the present application;

[0049] Figure 4 is a flow chart of an analysis agent of an embodiment of the present application;

[0050] Figure 5 is a flow chart of a reflective agent of one embodiment of the present application;

[0051] Figure 6 is a schematic diagram of a report template of an embodiment of the present application;

[0052] Figure 7 is a schematic diagram of a multi-agent intelligent video image analysis system according to an embodiment of the present application;

[0053] Figure 8It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0054] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0055] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0056] According to an embodiment of the present application, a method for correcting the longitude and latitude of city-level large-scale cameras based on face barcodes is provided. Figure 1 is a flowchart of a multi-agent intelligent video image analysis method according to an embodiment of the present application. Figure 1 As shown, the method includes:

[0057] S1. Receive image data and a natural language description of a user, and pre-process the image data and the natural language description; wherein the image data is at least one of a video and a picture;

[0058] Specifically, step S1 preprocessing includes:

[0059] S11, obtaining an access link to the image data;

[0060] S12: splice the access link of the image data and the natural language description, and output it in the following form for use in the next step.

[0061] {

[0062] “query”: user’s natural language description;

[0063] "url": video or image access specific link;

[0064] }The preprocessed output is provided to the large model.

[0065] S2. Define the functional descriptions of different recognition agents, where the recognition agents are agents that encapsulate scene recognition, behavior recognition, emotion recognition, action recognition, portrait recognition, and object recognition prompt words for large models;

[0066] Specifically, step S2 includes:

[0067] S21. Define the identification agent function description, which is used to analyze the call identification of the agent. The specific example format is as follows: Figure 2 shown.

[0068] S22, construct each recognition agent, set each dedicated prompt word, and give the prompt word to the big model to guide it to analyze according to the prompt word content. Figure 3 shown.

[0069] S23. Encapsulate each recognition agent into a unified interface for the analysis agent to call.

[0070] S3, obtaining a first user prompt word for analyzing the description of the ability of the agent, integrating the first user prompt word with the functional descriptions of different recognition agents, and inputting the first user prompt word into the big model together with the natural language description and the image data, using the big model to return the recognition agent to be called, and using the recognition agent to be called to analyze the image data to obtain the analysis result;

[0071] Specifically, refer to Figure 4 , step S3 comprises:

[0072] S31, obtaining a first user prompt word for analyzing the description of the agent's capabilities;

[0073] S32, obtaining functional descriptions of different recognition agents;

[0074] S33, integrating the first user prompt word and the functional descriptions of different recognition agents;

[0075] S34, inputting the integrated prompt words, natural language description and image data into the big model, and using the big model to return a list of recognition agents that need to be called;

[0076] S35. Determine whether there is a list of recognition agents. If so, traverse and call all recognition agents in the list of recognition agents, summarize the recognition results of all recognition agents, and obtain the analysis results. Conduct in-depth analysis of the ability description of the recognition agent to clarify its areas of expertise and data processing capabilities. Accurately interpret the needs expressed by users in natural language and understand their specific needs. Based on the above analysis, intelligently call the most suitable agent to accurately analyze the provided video or picture. Finally, give the conclusion and analysis basis, not limited to text, pictures, and video clips.

[0077] S4. Perform multiple iterations of reflection on the analysis results, assign reliability scores, and obtain analysis results whose reliability scores are greater than a preset trustworthy threshold;

[0078] Specifically, refer to Figure 5 , step S4 comprises:

[0079] S41, obtaining a second user prompt word for reflecting on the description of the intelligent agent's ability;

[0080] S42, initializing the best reliability score to 0;

[0081] S43, inputting the second user prompt word and the analysis result into the big model, using the big model to give the analysis result a reliability score, and increasing the number of reflections once;

[0082] S44, comparing the reliability score of the analysis result with the best reliability score, if it is greater than, updating the best reliability score, and saving the analysis result;

[0083] In S43 or S44, if the reliability score of the analysis result is less than the optimal reliability score, or the optimal reliability score is less than the trust threshold, it is determined whether the reflection upper limit is reached. If the reflection upper limit is reached, the analysis result is merged with the image data and input into the recognition agent that needs to be called for analysis to retrieve the analysis result.

[0084] S45. Determine whether the best reliability score is greater than the trust threshold. If so, return the analysis result with the best reliability score greater than the trust threshold. Perform multiple iterations of reflection on the analysis results of the analysis agent, give a reliability score, and give the most reliable analysis and analysis basis based on the reliability judgment;

[0085] S46, inputting the natural language description and the analysis result with the best reliability score greater than the trust threshold into the big model, and using the big model to return the target of the user's query, the target including people and events;

[0086] S47. If the object is a person, call the confidence interface to conduct confidence testing on the target and obtain information greater than a preset confidence threshold.

[0087] S5. Convert the analysis results with reliability scores greater than the preset trust threshold into natural language descriptions, combine them with image data, and form a report. Automatically convert the analysis results into easy-to-understand natural language descriptions, combine them with relevant pictures or video clips, and form a report with clear logic and detailed content.

[0088] Specifically, step S5 includes:

[0089] S51, first define the user prompt words that summarize the description of the agent's capabilities, and define the report template. The specific report template prompt words are as follows: Figure 6 shown.

[0090] S52: Integrate the user prompt words and the report template prompt words to form a complete prompt word.

[0091] S53, transmitting the complete prompt words and the final result provided by the reflection result to the large model to obtain a logic chain report.

[0092] According to a second aspect of the present invention, Figure 7is a framework diagram of a multi-agent intelligent video image analysis system according to an embodiment of the present application. Figure 7 As shown, the system includes:

[0093] The data receiving module 701 is configured to receive image data and a natural language description of a user, and pre-process the image data and the natural language description; wherein the image data is at least one of a video and a picture;

[0094] The recognition agent module 702 is configured to define the functional descriptions of different recognition agents, where the recognition agents are agents encapsulating scene recognition, behavior recognition, emotion recognition, action recognition, portrait recognition, and object recognition prompt words for the large model;

[0095] The analysis agent module 703 is configured to obtain a first user prompt word for describing the ability of the analysis agent, integrate the first user prompt word with the functional descriptions of different recognition agents, input the first user prompt word together with the natural language description and the image data into the large model, use the large model to return the recognition agent to be called, use the recognition agent to be called to analyze the image data, and obtain the analysis result;

[0096] The reflection agent module 704 is configured to perform multiple iterations of reflection on the analysis results, assign reliability scores, and obtain analysis results whose reliability scores are greater than a preset trustworthy threshold;

[0097] The summary agent module 705 is configured to convert the analysis results with reliability scores greater than a preset trust threshold into natural language descriptions, combined with image data, to form a report.

[0098] The following beneficial effects are achieved through the intelligent video image analysis method and system under multi-agent of the present application:

[0099] 1. Significantly accelerate the ability to adapt to new scenarios: The present invention uses innovative prompt word fine-tuning technology to greatly enhance the system's ability to quickly adapt to new scenarios and new demands, effectively shortening the time cycle from demand identification to system deployment, thereby enabling rapid response to the ever-changing security environment.

[0100] 2. Significantly reduce the dependence on data and computing resources: no longer rely on large-scale annotated data sets, reducing the time and cost of data collection and preprocessing. At the same time, the optimized model design reduces the consumption of computing resources, making it possible to deploy the system on edge devices, and improving the overall application scope and flexibility.

[0101] 3. Enhance the generalization and robustness of the model: Through the multi-agent collaborative architecture and multi-round reflective analysis mechanism, the present invention significantly enhances the model's recognition and analysis capabilities in complex and changing scenarios, improves the model's robustness, and ensures that the system can maintain high-precision performance under various conditions.

[0102] 4. Improve the transparency and explainability of decisions: The automatic generation system of logical chain reports ensures the transparency and explainability of the intelligent decision-making process, improves the credibility of the system and user satisfaction. Especially in the security field, the traceability of decisions is crucial for compliance and responsibility identification.

[0103] 5. Simplify the optimization process and reduce costs: The prompt word tuning algorithm simplifies the system optimization process, avoids lengthy training cycles, significantly reduces the technical threshold and overall cost, and makes high-quality video image analysis technology more popular and practical.

[0104] To sum up, the technical solution of this application not only improves the performance of the video image analysis system, but also greatly improves its practicality, cost-effectiveness and industry influence, providing strong support for technological progress in the fields of intelligent security and so on.

[0105] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0106] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0107] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0108] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wireline, optical cable, RF, etc., or any suitable combination of the foregoing.

[0109] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0110] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] The modules involved in the embodiments of the present application may be implemented by software or by hardware.

[0112] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes: receiving image data and a natural language description of a user, and preprocessing the image data and the natural language description; wherein the image data is at least one of a video and a picture; defining functional descriptions of different recognition agents, wherein the recognition agents are agents encapsulating scene recognition, behavior recognition, emotion recognition, action recognition, portrait recognition, and object recognition prompt words for a large model; obtaining a first user prompt word for analyzing the description of the agent's capabilities, integrating the first user prompt word and the functional descriptions of different recognition agents, and inputting them into the large model together with the natural language description and the image data, using the large model to return the recognition agent to be called, and using the recognition agent to be called to analyze the image data to obtain an analysis result; performing multiple iterations of reflection on the analysis result, giving a reliability score, and obtaining an analysis result with a reliability score greater than a preset trustworthy threshold; converting the analysis result with a reliability score greater than the preset trustworthy threshold into a natural language description, and combining it with the image data to form a report.

[0113] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A multi-agent intelligent video image analysis method, characterized in that: include: S1. Receive image data and a natural language description of a user, and pre-process the image data and the natural language description; wherein the image data is at least one of a video and a picture; S2. Define functional descriptions of different recognition agents, where the recognition agents are agents that encapsulate scene recognition, behavior recognition, emotion recognition, action recognition, portrait recognition, and object recognition prompt words for large models; S3, obtaining a first user prompt word for analyzing the capability description of the agent, integrating the first user prompt word with the functional descriptions of the different recognition agents, and inputting the first user prompt word into a large model together with the natural language description and the image data, using the large model to return the recognition agent to be called, and using the recognition agent to be called to analyze the image data to obtain an analysis result; S4, performing multiple iterations of reflection on the analysis results, assigning reliability scores, and obtaining analysis results whose reliability scores are greater than a preset trustworthy threshold; S5. Convert the analysis results whose reliability scores are greater than a preset trust threshold into natural language descriptions, and combine them with the image data to form a report.

2. The method according to claim 1, characterized in that The preprocessing in step S1 includes obtaining an access link to the image data; and concatenating the access link to the image data and the natural language description.

3. The method according to claim 1, characterized in that Step S3 includes: S31, obtaining a first user prompt word for analyzing the description of the agent's capabilities; S32, obtaining functional descriptions of the different recognition agents; S33, integrating the first user prompt word and the functional descriptions of the different recognition agents; S34, inputting the integrated prompt word, the natural language description and the image data into the big model, and using the big model to return a list of recognition agents that need to be called; S35. Determine whether the recognition agent list exists. If so, traverse and call all recognition agents in the recognition agent list, summarize the recognition results of all recognition agents, and obtain the analysis results.

4. The method according to claim 1, characterized in that: The conclusion in step S3 includes text, pictures and video clips.

5. The method according to claim 1, characterized in that Step S4 includes: S41, obtaining a second user prompt word for reflecting on the description of the intelligent agent's ability; S42, initializing the best reliability score to 0; S43, inputting the second user prompt word and the analysis result into the large model, using the large model to give the analysis result a reliability score, and increasing the number of reflections once; S44, comparing the reliability score of the analysis result with the best reliability score, if the reliability score is greater than the best reliability score, updating the best reliability score and saving the analysis result; S45: Determine whether the best reliability score is greater than a credible threshold, and if so, return an analysis result indicating that the best reliability score is greater than the credible threshold.

6. The method according to claim 5, characterized in that Step S4 includes: if the reliability score of the analysis result is less than the optimal reliability score, or the optimal reliability score is less than the trust threshold, determine whether the reflection upper limit is reached; if the reflection upper limit is reached, merge the analysis result with the image data, input them together into the recognition agent that needs to be called for analysis, and reacquire the analysis result.

7. The method according to claim 5, characterized in that Step S4 includes: S46, inputting the natural language description and the analysis result of the best reliability score greater than the trust threshold into the big model, and using the big model to return the target of the user query, the target including people and events; S47: If the object is a person, call a confidence interface to conduct confidence testing on the target and obtain information greater than a preset confidence threshold.

8. An intelligent video image analysis system under multi-agent, characterized in that: The system comprises: A data receiving module, configured to receive image data and a natural language description of a user, and pre-process the image data and the natural language description; wherein the image data is at least one of a video and a picture; The recognition agent module is configured to define functional descriptions of different recognition agents, wherein the recognition agents are agents encapsulating scene recognition, behavior recognition, emotion recognition, action recognition, portrait recognition and object recognition prompt words for the large model; The analysis agent module is configured to obtain a first user prompt word for describing the capabilities of the analysis agent, integrate the first user prompt word with the functional descriptions of the different recognition agents, input the first user prompt word together with the natural language description and the image data into a large model, use the large model to return the recognition agent to be called, and use the recognition agent to be called to analyze the image data to obtain an analysis result; A reflection agent module is configured to perform multiple iterations of reflection on the analysis results, assign reliability scores, and obtain analysis results whose reliability scores are greater than a preset trustworthy threshold; The summary agent module is configured to convert the analysis results whose reliability scores are greater than a preset trust threshold into natural language descriptions, and combine them with the image data to form a report.

9. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that: When the one or more computer programs are executed by a computer processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, comprising: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

Images