Video safety supervision platform multi-modal information query control method, system and device

Abstract

The invention provides a multi-modal information query control method, system and device for a video safety supervision platform. The multi-modal information query control method for the video safety supervision platform comprises the following steps: controlling a Gorilla back-end reasoning module to receive input information sent by a user terminal from a front-end web interface in real time for a Gradio application module to obtain a reasoning result, and feeding back the reasoning result to the user terminal; controlling an API calling module to execute API calling according to a calling request corresponding to the API contained in the reasoning result, and returning a calling result; and the Gorilla back-end reasoning module generates output return in combination with the user input and the API calling result, and sends the output return to the Gradio application module. The system comprises modules corresponding to the steps of the method.

Description

Technical Field

[0001] The invention proposes a multimodal information query control method, system and device for a video safety monitoring platform, belonging to the technical field of query control. Background Art

[0002] Existing video security monitoring platform systems usually rely on users to operate through a graphical user interface (GUI) to control monitoring equipment and query alarm information. Users need to be familiar with the system's interface, buttons, and operating logic in order to effectively retrieve alarm information or issue control instructions. Although the information query processing and control system based on large language model agent technology can lower the user's usage threshold, expand the system's interaction methods, and has certain natural language interaction capabilities, it can usually only perform single-step tool calls, cannot handle complex queries, and lacks multimodal capabilities and subsequent information processing capabilities. The existing video security monitoring platform information query and control system based on large language model agent technology has the following problems:

[0003] 1. Usually only single-step tool calls can be performed, and the ability to handle complex queries is weak.

[0004] 2. Lack of multimodal information processing capabilities and can only process text information.

[0005] 3. The backend large model has fixed capabilities and generally only works with a single large model, which lacks scalability. Summary of the invention

[0006] The present invention provides a multimodal information query control method, system and device for a video safety monitoring platform to solve the above technical problems existing in the prior art. The technical solutions adopted are as follows:

[0007] A multimodal information query control method for a video safety monitoring platform, the multimodal information query control method for a video safety monitoring platform comprising:

[0008] Control the gorilla backend reasoning module to receive input information sent by the user terminal from the front-end web interface in real time for the gradio application module to obtain reasoning results, and feed back the reasoning results to the user terminal;

[0009] The control API call module executes the API call according to the call request corresponding to the API contained in the inference result, and returns the call result;

[0010] The gorilla backend reasoning module generates an output return in combination with the user input and the API call result, and sends the output return to the gradio application module.

[0011] Further, controlling the gorilla backend reasoning module to receive input information sent by the user terminal from the front-end web interface in real time for the gradio application module to obtain a reasoning result, and feeding back the reasoning result to the user terminal, includes:

[0012] The gradio application module receives input information sent by the user terminal from the front-end web interface in real time;

[0013] Calling the gorilla backend inference module to perform inference according to the input information sent by the user terminal to obtain an inference result; wherein the inference result includes but is not limited to whether to call an API, which API to call, and API parameters and the call request corresponding to the API;

[0014] The inference result feedback obtained by the gorilla backend inference module is fed back to the user terminal through the gradio application module.

[0015] Furthermore, the control API call module executes the API call according to the call request corresponding to the API included in the inference result, and returns the call result, including:

[0016] Control the gorilla backend reasoning module to send a call request corresponding to the API included in the reasoning result to the API calling module;

[0017] Controlling the API call module to execute an API call corresponding to the API call request contained in the reasoning result, and obtaining the API call result;

[0018] Feedback the API call result to the gorilla backend reasoning module.

[0019] Furthermore, the execution status of the API call process is monitored in real time, and abnormal alarms are issued, including:

[0020] Real-time monitoring of execution status parameters during the API call process, wherein the execution status parameters include response time, return data validity ratio, call failure rate and call timeout rate;

[0021] Obtaining a response time coefficient using the response time;

[0022] The response time coefficient is obtained by the following formula:

[0023]

[0024] Where Q represents the response time coefficient; n represents the number of API calls; T i represents the response time corresponding to the i-th API call; P csirepresents the overall call timeout rate corresponding to the completion time of the i-th API call; P csmax Indicates the maximum call timeout rate; T mb Indicates the standard deviation of the response time of the completed API calls at the time of the maximum call timeout rate;

[0025] comparing the response time coefficient with a preset time coefficient threshold;

[0026] When the response time coefficient exceeds a preset time coefficient threshold, the return data included in the execution running status parameters are retrieved to obtain the data validity ratio, call failure rate and call timeout rate;

[0027] The execution status determination coefficient is obtained by using the return data included in the execution status parameter as the ratio of data validity, the call failure rate and the call timeout rate combined with the response time coefficient;

[0028] The execution status determination coefficient is obtained by the following formula:

[0029]

[0030] Where R represents the execution status determination coefficient; m represents the number of unit time that the API call has experienced; P yxj Indicates the proportion of the return data corresponding to the jth unit time as data validity; B j represents the call failure rate corresponding to the j-th unit time; Q j represents the average value of the response time coefficient corresponding to the jth unit time; Q bj represents the standard deviation of the response time coefficient corresponding to the jth unit time; P csj Indicates the call timeout rate corresponding to the j-th unit time;

[0031] Comparing the execution running state determination coefficient with a preset determination coefficient threshold;

[0032] When the execution running state determination coefficient is lower than a preset determination coefficient threshold, it is determined that the execution running state in the API calling process is abnormal, and an abnormality alarm is issued.

[0033] A multimodal information query control system for a video safety monitoring platform, the multimodal information query control system for the video safety monitoring platform comprising:

[0034] The reasoning control module is used to control the gorilla backend reasoning module to receive the input information sent by the user terminal from the front-end web interface in real time for the gradio application module to obtain the reasoning result, and feed back the reasoning result to the user terminal;

[0035] An API call execution module is used to control the API call module to execute an API call according to a call request corresponding to the API contained in the inference result, and return a call result;

[0036] The output return control module is used for the gorilla backend reasoning module to generate output returns in combination with user input and API call results, and send the output returns to the gradio application module.

[0037] Furthermore, the inference control module includes:

[0038] The input information real-time receiving module is used for the gradio application module to receive the input information sent by the user terminal from the front-end web interface in real time;

[0039] An inference control execution module, used to call the gorilla backend inference module to perform inference according to the input information sent by the user terminal to obtain an inference result; wherein the inference result includes but is not limited to whether to call an API, which API to call, and API parameters and the call request corresponding to the API;

[0040] The result feedback module is used to feedback the inference result obtained by the gorilla backend inference module to the user terminal through the gradio application module.

[0041] Furthermore, the API call execution module includes:

[0042] A call request sending control module, used to control the gorilla backend reasoning module to send a call request corresponding to the API contained in the reasoning result to the API calling module;

[0043] An API call result acquisition module, used to control the call request corresponding to the API included in the reasoning result of the API call module to execute the API call and obtain the API call result;

[0044] The call result feedback module is used to feed back the API call result to the gorilla backend reasoning module.

[0045] Furthermore, the multimodal information query and control system of the video safety monitoring platform also includes:

[0046] A status parameter monitoring module is used to monitor the execution status parameters of the API call in real time, wherein the execution status parameters include response time, return data validity ratio, call failure rate and call timeout rate;

[0047] A response time coefficient acquisition module, used to acquire a response time coefficient using the response time;

[0048] The response time coefficient is obtained by the following formula:

[0049]

[0050] Where Q represents the response time coefficient; n represents the number of API calls; T i represents the response time corresponding to the i-th API call; P csi represents the overall call timeout rate corresponding to the completion time of the i-th API call; P csmax Indicates the maximum call timeout rate; T mb Indicates the standard deviation of the response time of the completed API calls at the time of the maximum call timeout rate;

[0051] A time coefficient comparison module, used for comparing the response time coefficient with a preset time coefficient threshold;

[0052] A parameter retrieving module, for retrieving the return data contained in the execution running status parameters, such as the ratio of data validity, call failure rate and call timeout rate, when the response time coefficient exceeds a preset time coefficient threshold;

[0053] An execution running state determination coefficient acquisition module is used to obtain the execution running state determination coefficient by using the return data included in the execution running state parameter as the ratio of data validity, the call failure rate and the call timeout rate combined with the response time coefficient;

[0054] The execution status determination coefficient is obtained by the following formula:

[0055]

[0056] Where R represents the execution status determination coefficient; m represents the number of unit time that the API call has experienced; P yxj Indicates the proportion of the return data corresponding to the jth unit time as data validity; B j represents the call failure rate corresponding to the j-th unit time; Q j represents the average value of the response time coefficient corresponding to the jth unit time; Q bj represents the standard deviation of the response time coefficient corresponding to the jth unit time; P csj Indicates the call timeout rate corresponding to the j-th unit time;

[0057] A determination coefficient comparison module, used for comparing the execution running state determination coefficient with a preset determination coefficient threshold;

[0058] The abnormality determination and alarm module is used to determine that the execution status during the API call process is abnormal and to issue an abnormality alarm when the execution status determination coefficient is lower than a preset determination coefficient threshold.

[0059] A multimodal information query control device for a video safety monitoring platform, comprising a user terminal, a gradio application module, a gorilla backend reasoning module and an API calling module; wherein the input information output end of the user terminal is data-connected with the input information input end of the gradio application module; the signal output end of the gradio application module is data-connected with the signal input end of the gorilla backend reasoning module, and the feedback information output end of the gorilla backend reasoning module is data-connected with the feedback signal input end of the gradio application module; the signal output end of the gorilla backend reasoning module is data-connected with the signal input end of the API calling module; the feedback information output end of the API calling module is data-connected with the feedback signal input end of the gorilla backend reasoning module.

[0060] Beneficial effects of the present invention:

[0061] The multimodal information query control method, system and device of the video safety monitoring platform proposed in the present invention can realize flexible loading of the back-end reasoning model. The back-end reasoning model can provide services through API, and unified scheduling is realized by gorilla. Based on the image processing API provided by Gorilla, the alarm image can be further analyzed, which solves the defect that the previous LLM can only analyze text information. Compared with the general LLM, this system produces fewer hallucinations when executing tool calls according to user intentions, has a higher call success rate, and improves the intelligence level of the video safety monitoring system. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flow chart of the method of the present invention;

[0063] Figure 2 is a system block diagram of the system of the present invention;

[0064] Figure 3 The schematic diagram of the device of the present invention is shown in FIG. DETAILED DESCRIPTION

[0065] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0066] The embodiment of the present invention proposes a multi-modal information query control method for a video safety monitoring platform, such as Figure 1 As shown, the multimodal information query control method of the video safety monitoring platform includes:

[0067] S1, control the gorilla backend reasoning module to receive input information sent by the user terminal from the front-end web interface in real time for the gradio application module to obtain the reasoning result, and feed back the reasoning result to the user terminal;

[0068] S2, controlling the API call module to execute the API call according to the call request corresponding to the API contained in the inference result, and returning the call result;

[0069] S3. The gorilla backend reasoning module generates an output return in combination with the user input and the API call result, and sends the output return to the gradio application module.

[0070] The working principle of the above technical solution is as follows: Gorilla is an advanced large language model (LLM) that understands and accurately calls more than 1,600 APIs by interpreting natural language queries. Gorilla selects and utilizes tools with evolving capabilities by using self-guidance and retrieval techniques. Compared with previous API calling technologies, Gorilla is more accurate and produces fewer false API call hallucinations.

[0071] Based on Gorilla, this embodiment develops an API for the actual needs of the video security platform information query and processing system, and fine-tunes Gorilla by making instructions and fine-tuning the data set according to the API document, so that it can accurately call APIs such as alarm information query and camera control. For the queried alarm information, users can use the API supported by Gorilla for further processing, such as target detection, network search, and calling a multimodal large model for analysis of the alarm image.

[0072] The system consists of the gradio application module, the Gorilla backend reasoning module, and the API execution module. The gradio application module is used to accept user input from the front-end web interface, call the Gorilla backend reasoning module for reasoning, and return the reasoning results to the user. The Gorilla backend reasoning module is responsible for reasoning the user's input, deciding whether to call, which API to call, and the API parameters, and handing the call request to the API execution module. The API execution module executes the API call according to the API call request of the Gorilla backend reasoning module, and returns the result to the Gorilla backend reasoning module. The Gorilla backend reasoning module combines the user input and the API call result to generate output and return it to the gradio application module.

[0073] The effect of the above technical solution is that the Gorilla backend reasoning module can quickly process the user input information received in real time by the gradio application module from the front-end web interface and obtain the reasoning results. This efficient processing capability ensures timely response to user queries and improves user experience. The control API call module quickly executes the corresponding API call according to the reasoning result and returns the call result. This fast API call and response capability helps to reduce user waiting time and improve the overall efficiency of the system. From user input to reasoning result feedback, to API call and output return, the entire process is processed in real time. This ensures that users can get the required information immediately and enhances the real-time and availability of the system. The Gorilla backend reasoning module uses advanced algorithms and technologies to accurately parse user input and obtain accurate reasoning results. This helps to reduce false positives and false negatives and improve the accuracy of the system. The control API call module interacts with a stable API service to ensure the reliability and stability of API calls. This helps to reduce problems caused by API failures and improve the overall reliability of the system. The Gorilla backend reasoning module combines the output return generated by the user input and the API call result to accurately reflect user needs and related information. This helps users better understand and use the information provided by the system. The technical solution adopts a modular design, so that each module can be developed and upgraded independently. This helps to reduce system maintenance costs and improve system scalability. The Gradio application module supports a variety of front-end input methods, such as text, image, voice, etc. This helps to meet the needs and preferences of different users and improve the flexibility of the system. The technical solution can be integrated and expanded with other systems or modules to adapt to different application scenarios and needs. This helps to expand the application scope of the system and improve its market competitiveness. The Gorilla back-end reasoning module uses intelligent algorithms and technologies for reasoning and decision-making, which can automatically parse user input and generate corresponding output returns. This helps to reduce manual intervention and reduce operating costs. The control API call module can automatically execute API calls based on the reasoning results and feed back the call results to the user terminal. This automated processing method improves the system's automation and work efficiency.

[0074] In summary, the multimodal information query control method of the video safety monitoring platform has demonstrated technical effects such as high efficiency and real-time performance, accuracy and reliability, scalability and flexibility, intelligence and automation in terms of performance indicators. These technical effects have jointly improved the overall performance of the system and user experience.

[0075] In one embodiment of the present invention, the gorilla backend reasoning module is controlled to receive input information sent by a user terminal from a front-end web interface in real time for a gradio application module to obtain a reasoning result, and the reasoning result is fed back to the user terminal, including:

[0076] S101, the gradio application module receives input information sent by the user terminal from the front-end web interface in real time;

[0077] S102, calling the gorilla backend reasoning module to perform reasoning according to the input information sent by the user terminal to obtain a reasoning result; wherein the reasoning result includes but is not limited to whether to call an API, which API to call, and API parameters and a call request corresponding to the API;

[0078] S103: Feedback the inference result obtained by the gorilla backend inference module to the user terminal through the gradio application module.

[0079] The working principle of the above technical solution is as follows: the Gradio application module acts as a bridge between the front-end and the back-end, and receives input information sent by the user terminal in real time from the front-end web interface. These input information may include text, images, voice and other forms, depending on the configuration of the gradio application module and the needs of the user. The gorilla back-end reasoning module is called to reason about the input information sent by the user terminal. The gorilla back-end reasoning module may contain a variety of algorithms and models for parsing and understanding user input, and reasoning according to preset rules or trained models. The reasoning results may include a variety of information, such as whether to call an API, which API to call, API parameters, and the corresponding call request of the API. This information will serve as the basis for subsequent operations. The reasoning results obtained by the gorilla back-end reasoning module are fed back to the user terminal through the gradio application module. The user can view the reasoning results through the front-end web interface and perform subsequent operations as needed.

[0080] The effect of the above technical solution is: through the real-time interaction between the Gradio application module and the front-end web interface, users can input information and view the reasoning results in real time, which improves the system's response speed and user experience. The Gorilla back-end reasoning module uses advanced algorithms and models for reasoning, which can accurately understand user input and give reasonable reasoning results, improving the intelligence level of the system. The technical solution supports multiple input forms and configurations of reasoning results, and can be flexibly adjusted according to actual needs. This helps to meet the needs of different users and application scenarios. Both the Gradio application module and the gorilla back-end reasoning module have good scalability and can be upgraded and expanded as technology develops and user needs change. This helps to maintain the advancement and competitiveness of the system. By directly feeding back the reasoning results to the user terminal, the automated processing of API calls is realized. This reduces manual intervention and improves the automation and efficiency of the system.

[0081] In summary, this technical solution provides users with efficient, intelligent and convenient query and control services through technical effects such as real-time interaction, intelligent reasoning, flexible configuration, scalability and automated processing.

[0082] In one embodiment of the present invention, the API calling module is controlled to execute the API calling according to the calling request corresponding to the API included in the inference result, and return the calling result, including:

[0083] S201, controlling the gorilla backend reasoning module to send a call request corresponding to the API included in the reasoning result to the API calling module;

[0084] S202, controlling the API call module to execute an API call corresponding to the API included in the inference result, and obtaining an API call result;

[0085] S203: Feedback the API call result to the gorilla backend reasoning module.

[0086] The working principle of the above technical solution is as follows: After the Gorilla backend reasoning module completes the reasoning, it will generate a reasoning result containing an API call request. The reasoning result is then sent to the API call module as the basis for executing the API call. After receiving the reasoning result, the API call module parses the API call request therein. According to the API call request, the API call module executes the corresponding API call and waits for the response of the API service provider. After the API service provider processes the request, it returns the API call result to the API call module. After the API call module obtains the API call result, it feeds it back to the gorilla backend reasoning module. The Gorilla backend reasoning module can perform further processing or decision-making based on the API call result.

[0087] The effect of the above technical solution is: by combining the reasoning results with the API call module, the automated processing of API calls is realized. This reduces manual intervention and improves the automation and efficiency of the system. The technical solution supports the calling and configuration of multiple APIs, and can be flexibly adjusted according to actual needs. This helps to meet the needs of different users and application scenarios. The API call module can quickly parse and execute API call requests in the reasoning results, thereby ensuring the efficient operation of the system. By feeding back the API call results to the gorilla backend reasoning module, data integration and sharing are achieved. This helps the gorilla backend reasoning module to make more accurate reasoning and decisions. The technical solution has good scalability and can be upgraded and expanded as technology develops and user needs change. For example, more API call modules can be added or the algorithms and models of the gorilla backend reasoning module can be optimized to improve the performance and intelligence level of the system.

[0088] In summary, this technical solution provides users with efficient, intelligent and convenient API calling services through technical effects such as automated processing, flexible configuration, efficient execution, data integration and scalability.

[0089] An embodiment of the present invention monitors the execution status of an API call in real time and issues an abnormal alarm, including:

[0090] Step 1: monitor the execution status parameters of the API call in real time, wherein the execution status parameters include response time, return data validity ratio, call failure rate and call timeout rate;

[0091] Step 2: Obtain a response time coefficient using the response time;

[0092] The response time coefficient is obtained by the following formula:

[0093]

[0094] Where Q represents the response time coefficient; n represents the number of API calls; T i represents the response time corresponding to the i-th API call; P csi represents the overall call timeout rate corresponding to the completion time of the i-th API call; P csmax Indicates the maximum call timeout rate; T mb Indicates the standard deviation of the response time of the completed API calls at the time of the maximum call timeout rate;

[0095] Step 3: Compare the response time coefficient with a preset time coefficient threshold;

[0096] Step 4: When the response time coefficient exceeds a preset time coefficient threshold, the return data included in the execution status parameters are retrieved to obtain the data validity ratio, call failure rate and call timeout rate;

[0097] Step 5: derive the execution status determination coefficient by using the return data included in the execution status parameter as the ratio of data validity, the call failure rate and the call timeout rate in combination with the response time coefficient;

[0098] The execution status determination coefficient is obtained by the following formula:

[0099]

[0100] Where R represents the execution status determination coefficient; m represents the number of unit time that the API call has experienced; P yxj Indicates the proportion of the return data corresponding to the jth unit time as data validity; B j represents the call failure rate corresponding to the j-th unit time; Q j represents the average value of the response time coefficient corresponding to the jth unit time; Q bj represents the standard deviation of the response time coefficient corresponding to the jth unit time; P csj Indicates the call timeout rate corresponding to the j-th unit time;

[0101] Step 6: Compare the execution running state determination coefficient with a preset determination coefficient threshold;

[0102] Step 7: When the execution status determination coefficient is lower than a preset determination coefficient threshold, it is determined that the execution status during the API call process is abnormal, and an abnormal alarm is issued.

[0103] The working principle of the above technical solution is: the execution running state parameters in the process of API calling are monitored in real time by technical means (such as API monitoring tools). These parameters can reflect the performance and stability of API calling. The response time coefficient is calculated by combining the response time data monitored, the call timeout rate and the standard deviation of the response time of the completed API call. The response time coefficient can comprehensively reflect the response speed and stability of the API call. The calculated response time coefficient is compared with the preset time coefficient threshold. If the response time coefficient exceeds the threshold, it indicates that there may be a problem with the response speed of the API call. When the response time coefficient exceeds the threshold, other execution running state parameters (data validity ratio of returned data, call failure rate and call timeout rate) are retrieved for further analysis. The execution running state determination coefficient is calculated by combining the retrieved running state parameters and response time coefficient with the statistical information within the unit time (such as average value, standard deviation, etc.). The execution running state determination coefficient can comprehensively evaluate the overall performance and stability of the API call. Determination coefficient threshold comparison and abnormal alarm: The calculated execution running state determination coefficient is compared with the preset determination coefficient threshold. If the determination coefficient is lower than the threshold, it is determined that the execution status during the API call is abnormal, and an abnormal alarm is issued.

[0104] The effect of the above technical solution is: by real-time monitoring of multiple execution status parameters, and combining the response time coefficient and the execution status determination coefficient for comprehensive evaluation, it is possible to more accurately detect abnormal situations in the API call process. The real-time monitoring and abnormal alarm mechanism can promptly discover and handle problems in the API call process, thereby avoiding further deterioration of the problem and enhancing the stability of the system. By improving the performance and stability of API calls, the user experience can be optimized and user satisfaction can be improved. The automated monitoring and alarm mechanism can reduce the workload of operation and maintenance personnel and improve operation and maintenance efficiency. This technical solution can be flexibly configured and expanded according to actual needs to adapt to API call scenarios of different scales and complexities.

[0105] On the other hand, by real-time monitoring of response time and calculating the response time coefficient, it is possible to timely find out whether the response speed of API calls meets the standard. When the response time coefficient exceeds the preset threshold, the system can respond quickly to provide a basis for further analysis and processing. This helps to optimize the performance of API calls, reduce user waiting time, and improve user experience. Monitoring the data validity ratio of returned data can ensure that the results of API calls meet expectations and improve the accuracy and reliability of data. When the data validity ratio decreases, the system can detect and alarm in time to avoid using wrong or invalid data. By monitoring the call failure rate and call timeout rate, the stability and reliability of API calls can be evaluated. When these indicators exceed the preset threshold, the system can trigger an alarm to remind the operation and maintenance personnel to intervene in time to avoid service interruption or performance degradation. Combining multiple indicators such as response time coefficient, data validity ratio, call failure rate and call timeout rate, calculating the execution running status determination coefficient can more comprehensively evaluate the overall performance and stability of API calls. This comprehensive anomaly detection capability can more accurately identify potential abnormal problems and reduce false positives and false negatives. When the execution running status determination coefficient is lower than the preset threshold, the system can automatically trigger an early warning or alarm mechanism to notify the operation and maintenance personnel to handle it in time. This helps to respond and solve problems quickly, reducing the duration and scope of service interruptions. The technical solution can be flexibly configured and expanded according to actual needs to adapt to API call scenarios of different scales and complexities. This means that as the business develops and changes, the system can continue to maintain efficient and stable operation.

[0106] In summary, the technical effects of the above technical solutions in terms of performance indicators are mainly reflected in improving the response speed, data quality, stability and reliability evaluation capabilities of API calls, as well as enhancing comprehensive anomaly detection capabilities and early warning alarm mechanisms. These technical effects help to improve the overall performance and user experience of the system while reducing operation and maintenance costs and risks. At the same time, by real-time monitoring of the execution status parameters during the API call process and making abnormal alarms, the technical solution can achieve significant technical effects in improving the accuracy of anomaly detection, enhancing system stability, optimizing user experience, improving operation and maintenance efficiency, and scalability and flexibility.

[0107] The embodiment of the present invention proposes a multi-modal information query control system for a video safety monitoring platform, such as Figure 2 As shown, the multimodal information query and control system of the video safety monitoring platform includes:

[0108] The reasoning control module is used to control the gorilla backend reasoning module to receive the input information sent by the user terminal from the front-end web interface in real time for the gradio application module to obtain the reasoning result, and feed back the reasoning result to the user terminal;

[0109] An API call execution module is used to control the API call module to execute an API call according to a call request corresponding to the API contained in the inference result, and return a call result;

[0110] The output return control module is used for the gorilla backend reasoning module to generate output returns in combination with user input and API call results, and send the output returns to the gradio application module.

[0111] The working principle of the above technical solution is as follows: Gorilla is an advanced large language model (LLM) that understands and accurately calls more than 1,600 APIs by interpreting natural language queries. Gorilla selects and utilizes tools with evolving capabilities by using self-guidance and retrieval techniques. Compared with previous API calling technologies, Gorilla is more accurate and produces fewer false API call hallucinations.

[0112] Based on Gorilla, this embodiment develops an API for the actual needs of the video security platform information query and processing system, and fine-tunes Gorilla by making instructions and fine-tuning the data set according to the API document, so that it can accurately call APIs such as alarm information query and camera control. For the queried alarm information, users can use the API supported by Gorilla for further processing, such as target detection, network search, and calling a multimodal large model for analysis of the alarm image.

[0113] The system consists of the gradio application module, the Gorilla backend reasoning module, and the API execution module. The gradio application module is used to accept user input from the front-end web interface, call the Gorilla backend reasoning module for reasoning, and return the reasoning results to the user. The Gorilla backend reasoning module is responsible for reasoning the user's input, deciding whether to call, which API to call, and the API parameters, and handing the call request to the API execution module. The API execution module executes the API call according to the API call request of the Gorilla backend reasoning module, and returns the result to the Gorilla backend reasoning module. The Gorilla backend reasoning module combines the user input and the API call result to generate output and return it to the gradio application module.

[0114] The effect of the above technical solution is that the Gorilla backend reasoning module can quickly process the user input information received in real time by the gradio application module from the front-end web interface and obtain the reasoning results. This efficient processing capability ensures timely response to user queries and improves user experience. The control API call module quickly executes the corresponding API call according to the reasoning result and returns the call result. This fast API call and response capability helps to reduce user waiting time and improve the overall efficiency of the system. From user input to reasoning result feedback, to API call and output return, the entire process is processed in real time. This ensures that users can get the required information immediately and enhances the real-time and availability of the system. The Gorilla backend reasoning module uses advanced algorithms and technologies to accurately parse user input and obtain accurate reasoning results. This helps to reduce false positives and false negatives and improve the accuracy of the system. The control API call module interacts with a stable API service to ensure the reliability and stability of API calls. This helps to reduce problems caused by API failures and improve the overall reliability of the system. The Gorilla backend reasoning module combines the output return generated by the user input and the API call result to accurately reflect user needs and related information. This helps users better understand and use the information provided by the system. The technical solution adopts a modular design, so that each module can be developed and upgraded independently. This helps to reduce system maintenance costs and improve system scalability. The Gradio application module supports a variety of front-end input methods, such as text, image, voice, etc. This helps to meet the needs and preferences of different users and improve the flexibility of the system. The technical solution can be integrated and expanded with other systems or modules to adapt to different application scenarios and needs. This helps to expand the application scope of the system and improve its market competitiveness. The Gorilla back-end reasoning module uses intelligent algorithms and technologies for reasoning and decision-making, which can automatically parse user input and generate corresponding output returns. This helps to reduce manual intervention and reduce operating costs. The control API call module can automatically execute API calls based on the reasoning results and feed back the call results to the user terminal. This automated processing method improves the system's automation and work efficiency.

[0115] In summary, the multimodal information query control method of the video safety monitoring platform has demonstrated technical effects such as high efficiency and real-time performance, accuracy and reliability, scalability and flexibility, intelligence and automation in terms of performance indicators. These technical effects have jointly improved the overall performance of the system and user experience.

[0116] In one embodiment of the present invention, the inference control module includes:

[0117] The input information real-time receiving module is used for the gradio application module to receive the input information sent by the user terminal from the front-end web interface in real time;

[0118] An inference control execution module, used to call the gorilla backend inference module to perform inference according to the input information sent by the user terminal to obtain an inference result; wherein the inference result includes but is not limited to whether to call an API, which API to call, and API parameters and the call request corresponding to the API;

[0119] The result feedback module is used to feedback the inference result obtained by the gorilla backend inference module to the user terminal through the gradio application module.

[0120] The working principle of the above technical solution is as follows: the Gradio application module acts as a bridge between the front-end and the back-end, and receives input information sent by the user terminal in real time from the front-end web interface. These input information may include text, images, voice and other forms, depending on the configuration of the gradio application module and the needs of the user. The gorilla back-end reasoning module is called to reason about the input information sent by the user terminal. The gorilla back-end reasoning module may contain a variety of algorithms and models for parsing and understanding user input, and reasoning according to preset rules or trained models. The reasoning results may include a variety of information, such as whether to call an API, which API to call, API parameters, and the corresponding call request of the API. This information will serve as the basis for subsequent operations. The reasoning results obtained by the gorilla back-end reasoning module are fed back to the user terminal through the gradio application module. The user can view the reasoning results through the front-end web interface and perform subsequent operations as needed.

[0121] The effect of the above technical solution is: through the real-time interaction between the Gradio application module and the front-end web interface, users can input information and view the reasoning results in real time, which improves the system's response speed and user experience. The Gorilla back-end reasoning module uses advanced algorithms and models for reasoning, which can accurately understand user input and give reasonable reasoning results, improving the intelligence level of the system. The technical solution supports multiple input forms and configurations of reasoning results, and can be flexibly adjusted according to actual needs. This helps to meet the needs of different users and application scenarios. Both the Gradio application module and the gorilla back-end reasoning module have good scalability and can be upgraded and expanded as technology develops and user needs change. This helps to maintain the advancement and competitiveness of the system. By directly feeding back the reasoning results to the user terminal, the automated processing of API calls is realized. This reduces manual intervention and improves the automation and efficiency of the system.

[0122] In summary, this technical solution provides users with efficient, intelligent and convenient query and control services through technical effects such as real-time interaction, intelligent reasoning, flexible configuration, scalability and automated processing.

[0123] In one embodiment of the present invention, the API call execution module includes:

[0124] A call request sending control module, used to control the gorilla backend reasoning module to send a call request corresponding to the API contained in the reasoning result to the API calling module;

[0125] An API call result acquisition module, used to control the call request corresponding to the API included in the reasoning result of the API call module to execute the API call and obtain the API call result;

[0126] The call result feedback module is used to feed back the API call result to the gorilla backend reasoning module.

[0127] The working principle of the above technical solution is as follows: After the Gorilla backend reasoning module completes the reasoning, it will generate a reasoning result containing an API call request. The reasoning result is then sent to the API call module as the basis for executing the API call. After receiving the reasoning result, the API call module parses the API call request therein. According to the API call request, the API call module executes the corresponding API call and waits for the response of the API service provider. After the API service provider processes the request, it returns the API call result to the API call module. After the API call module obtains the API call result, it feeds it back to the gorilla backend reasoning module. The Gorilla backend reasoning module can perform further processing or decision-making based on the API call result.

[0128] The effect of the above technical solution is: by combining the reasoning results with the API call module, the automated processing of API calls is realized. This reduces manual intervention and improves the automation and efficiency of the system. The technical solution supports the calling and configuration of multiple APIs, and can be flexibly adjusted according to actual needs. This helps to meet the needs of different users and application scenarios. The API call module can quickly parse and execute API call requests in the reasoning results, thereby ensuring the efficient operation of the system. By feeding back the API call results to the gorilla backend reasoning module, data integration and sharing are achieved. This helps the gorilla backend reasoning module to make more accurate reasoning and decisions. The technical solution has good scalability and can be upgraded and expanded as technology develops and user needs change. For example, more API call modules can be added or the algorithms and models of the gorilla backend reasoning module can be optimized to improve the performance and intelligence level of the system.

[0129] In summary, this technical solution provides users with efficient, intelligent and convenient API calling services through technical effects such as automated processing, flexible configuration, efficient execution, data integration and scalability.

[0130] In one embodiment of the present invention, the multimodal information query control system of the video security monitoring platform further includes:

[0131] A status parameter monitoring module is used to monitor the execution status parameters of the API call in real time, wherein the execution status parameters include response time, return data validity ratio, call failure rate and call timeout rate;

[0132] A response time coefficient acquisition module, used to acquire a response time coefficient using the response time;

[0133] The response time coefficient is obtained by the following formula:

[0134]

[0135] Where Q represents the response time coefficient; n represents the number of API calls; T i represents the response time corresponding to the i-th API call; P csi represents the overall call timeout rate corresponding to the completion time of the i-th API call; P csmax Indicates the maximum call timeout rate; T mb Indicates the standard deviation of the response time of the completed API calls at the time of the maximum call timeout rate;

[0136] A time coefficient comparison module, used for comparing the response time coefficient with a preset time coefficient threshold;

[0137] A parameter retrieving module, for retrieving the return data contained in the execution running status parameters, such as the ratio of data validity, call failure rate and call timeout rate, when the response time coefficient exceeds a preset time coefficient threshold;

[0138] An execution running state determination coefficient acquisition module is used to obtain the execution running state determination coefficient by using the return data included in the execution running state parameter as the ratio of data validity, the call failure rate and the call timeout rate combined with the response time coefficient;

[0139] The execution status determination coefficient is obtained by the following formula:

[0140]

[0141] Where R represents the execution status determination coefficient; m represents the number of unit time that the API call has experienced; P yxj Indicates the proportion of the return data corresponding to the jth unit time as data validity; B j represents the call failure rate corresponding to the j-th unit time; Q j represents the average value of the response time coefficient corresponding to the jth unit time; Q bj represents the standard deviation of the response time coefficient corresponding to the jth unit time; P csj Indicates the call timeout rate corresponding to the j-th unit time;

[0142] A determination coefficient comparison module, used for comparing the execution running state determination coefficient with a preset determination coefficient threshold;

[0143] The abnormality determination and alarm module is used to determine that the execution status during the API call process is abnormal and to issue an abnormality alarm when the execution status determination coefficient is lower than a preset determination coefficient threshold.

[0144] The working principle of the above technical solution is: the execution running state parameters in the process of API calling are monitored in real time by technical means (such as API monitoring tools). These parameters can reflect the performance and stability of API calling. The response time coefficient is calculated by combining the response time data monitored, the call timeout rate and the standard deviation of the response time of the completed API call. The response time coefficient can comprehensively reflect the response speed and stability of the API call. The calculated response time coefficient is compared with the preset time coefficient threshold. If the response time coefficient exceeds the threshold, it indicates that there may be a problem with the response speed of the API call. When the response time coefficient exceeds the threshold, other execution running state parameters (data validity ratio of returned data, call failure rate and call timeout rate) are retrieved for further analysis. The execution running state determination coefficient is calculated by combining the retrieved running state parameters and response time coefficient with the statistical information within the unit time (such as average value, standard deviation, etc.). The execution running state determination coefficient can comprehensively evaluate the overall performance and stability of the API call. Determination coefficient threshold comparison and abnormal alarm: The calculated execution running state determination coefficient is compared with the preset determination coefficient threshold. If the determination coefficient is lower than the threshold, it is determined that the execution status during the API call is abnormal, and an abnormal alarm is issued.

[0145] The effect of the above technical solution is: by real-time monitoring of multiple execution status parameters, and combining the response time coefficient and the execution status determination coefficient for comprehensive evaluation, it is possible to more accurately detect abnormal situations in the API call process. The real-time monitoring and abnormal alarm mechanism can promptly discover and handle problems in the API call process, thereby avoiding further deterioration of the problem and enhancing the stability of the system. By improving the performance and stability of API calls, the user experience can be optimized and user satisfaction can be improved. The automated monitoring and alarm mechanism can reduce the workload of operation and maintenance personnel and improve operation and maintenance efficiency. This technical solution can be flexibly configured and expanded according to actual needs to adapt to API call scenarios of different scales and complexities.

[0146] On the other hand, by real-time monitoring of response time and calculating the response time coefficient, it is possible to timely find out whether the response speed of API calls meets the standard. When the response time coefficient exceeds the preset threshold, the system can respond quickly to provide a basis for further analysis and processing. This helps to optimize the performance of API calls, reduce user waiting time, and improve user experience. Monitoring the data validity ratio of returned data can ensure that the results of API calls meet expectations and improve the accuracy and reliability of data. When the data validity ratio decreases, the system can detect and alarm in time to avoid using wrong or invalid data. By monitoring the call failure rate and call timeout rate, the stability and reliability of API calls can be evaluated. When these indicators exceed the preset threshold, the system can trigger an alarm to remind the operation and maintenance personnel to intervene in time to avoid service interruption or performance degradation. Combining multiple indicators such as response time coefficient, data validity ratio, call failure rate and call timeout rate, calculating the execution running status determination coefficient can more comprehensively evaluate the overall performance and stability of API calls. This comprehensive anomaly detection capability can more accurately identify potential abnormal problems and reduce false positives and false negatives. When the execution running status determination coefficient is lower than the preset threshold, the system can automatically trigger an early warning or alarm mechanism to notify the operation and maintenance personnel to handle it in time. This helps to respond and solve problems quickly, reducing the duration and scope of service interruptions. The technical solution can be flexibly configured and expanded according to actual needs to adapt to API call scenarios of different scales and complexities. This means that as the business develops and changes, the system can continue to maintain efficient and stable operation.

[0147] In summary, the technical effects of the above technical solutions in terms of performance indicators are mainly reflected in improving the response speed, data quality, stability and reliability evaluation capabilities of API calls, as well as enhancing comprehensive anomaly detection capabilities and early warning alarm mechanisms. These technical effects help to improve the overall performance and user experience of the system while reducing operation and maintenance costs and risks. At the same time, by real-time monitoring of the execution status parameters during the API call process and making abnormal alarms, the technical solution can achieve significant technical effects in improving the accuracy of anomaly detection, enhancing system stability, optimizing user experience, improving operation and maintenance efficiency, and scalability and flexibility.

[0148] The embodiment of the present invention proposes a multi-modal information query control device for a video safety monitoring platform, such as Figure 3 As shown, the multimodal information query control device of the video safety monitoring platform includes a user terminal, a gradio application module, a gorilla back-end reasoning module and an API calling module; wherein, the input information output end of the user terminal is data-connected with the input information input end of the gradio application module; the signal output end of the gradio application module is data-connected with the signal input end of the gorilla back-end reasoning module, and the feedback information output end of the gorilla back-end reasoning module is data-connected with the feedback signal input end of the gradio application module; the signal output end of the gorilla back-end reasoning module is data-connected with the signal input end of the API calling module; the feedback information output end of the API calling module is data-connected with the feedback signal input end of the gorilla back-end reasoning module.

[0149] The working principle of the above technical solution is as follows: Gorilla is an advanced large language model (LLM) that understands and accurately calls more than 1,600 APIs by interpreting natural language queries. Gorilla selects and utilizes tools with evolving capabilities by using self-guidance and retrieval techniques. Compared with previous API calling technologies, Gorilla is more accurate and produces fewer false API call hallucinations.

[0150] Based on Gorilla, this embodiment develops an API for the actual needs of the video security platform information query and processing system, and fine-tunes Gorilla by making instructions and fine-tuning the data set according to the API document, so that it can accurately call APIs such as alarm information query and camera control. For the queried alarm information, users can use the API supported by Gorilla for further processing, such as target detection, network search, and calling a multimodal large model for analysis of the alarm image.

[0151] The system consists of the gradio application module, the Gorilla backend reasoning module, and the API execution module. The gradio application module is used to accept user input from the front-end web interface, call the Gorilla backend reasoning module for reasoning, and return the reasoning results to the user. The Gorilla backend reasoning module is responsible for reasoning the user's input, deciding whether to call, which API to call, and the API parameters, and handing the call request to the API execution module. The API execution module executes the API call according to the API call request of the Gorilla backend reasoning module, and returns the result to the Gorilla backend reasoning module. The Gorilla backend reasoning module combines the user input and the API call result to generate output and return it to the gradio application module.

[0152] The effect of the above technical solution is that the Gorilla backend reasoning module can quickly process the user input information received in real time by the gradio application module from the front-end web interface and obtain the reasoning results. This efficient processing capability ensures timely response to user queries and improves user experience. The control API call module quickly executes the corresponding API call according to the reasoning result and returns the call result. This fast API call and response capability helps to reduce user waiting time and improve the overall efficiency of the system. From user input to reasoning result feedback, to API call and output return, the entire process is processed in real time. This ensures that users can get the required information immediately and enhances the real-time and availability of the system. The Gorilla backend reasoning module uses advanced algorithms and technologies to accurately parse user input and obtain accurate reasoning results. This helps to reduce false positives and false negatives and improve the accuracy of the system. The control API call module interacts with a stable API service to ensure the reliability and stability of API calls. This helps to reduce problems caused by API failures and improve the overall reliability of the system. The Gorilla backend reasoning module combines the output return generated by the user input and the API call result to accurately reflect user needs and related information. This helps users better understand and use the information provided by the system. The technical solution adopts a modular design, so that each module can be developed and upgraded independently. This helps to reduce system maintenance costs and improve system scalability. The Gradio application module supports a variety of front-end input methods, such as text, image, voice, etc. This helps to meet the needs and preferences of different users and improve the flexibility of the system. The technical solution can be integrated and expanded with other systems or modules to adapt to different application scenarios and needs. This helps to expand the application scope of the system and improve its market competitiveness. The Gorilla back-end reasoning module uses intelligent algorithms and technologies for reasoning and decision-making, which can automatically parse user input and generate corresponding output returns. This helps to reduce manual intervention and reduce operating costs. The control API call module can automatically execute API calls based on the reasoning results and feed back the call results to the user terminal. This automated processing method improves the system's automation and work efficiency.

[0153] In summary, the multimodal information query control method of the video safety monitoring platform has demonstrated technical effects such as high efficiency and real-time performance, accuracy and reliability, scalability and flexibility, intelligence and automation in terms of performance indicators. These technical effects have jointly improved the overall performance of the system and user experience.

[0154] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multimodal information query control method for a video safety monitoring platform, characterized in that: The multimodal information query control method of the video safety monitoring platform includes: Control the gorilla backend reasoning module to receive input information sent by the user terminal from the front-end web interface in real time for the gradio application module to obtain reasoning results, and feed back the reasoning results to the user terminal; The control API call module executes the API call according to the call request corresponding to the API contained in the inference result, and returns the call result; The gorilla backend reasoning module generates an output return in combination with the user input and the API call result, and sends the output return to the gradio application module.

2. According to claim 1, the multimodal information query control method of the video safety monitoring platform is characterized in that: Control the gorilla backend reasoning module to receive input information sent by the user terminal from the front-end web interface in real time for the gradio application module to obtain the reasoning result, and feed back the reasoning result to the user terminal, including: The gradio application module receives input information sent by the user terminal from the front-end web interface in real time; Call the gorilla backend inference module to perform inference based on the input information sent by the user terminal and obtain the inference result; The inference result feedback obtained by the gorilla backend inference module is fed back to the user terminal through the gradio application module.

3. According to claim 1, the multimodal information query control method of the video safety monitoring platform is characterized in that: The control API call module executes the API call according to the call request corresponding to the API contained in the inference result, and returns the call result, including: Control the gorilla backend reasoning module to send a call request corresponding to the API included in the reasoning result to the API calling module; Controlling the API call module to execute an API call corresponding to the API call request contained in the reasoning result, and obtaining the API call result; Feedback the API call result to the gorilla backend reasoning module.

4. The multimodal information query control method of the video safety monitoring platform according to claim 1 or 3 is characterized in that: Real-time monitoring of the execution status of the API call process and abnormal alarms, including: Real-time monitoring of execution status parameters during the API call process, wherein the execution status parameters include response time, return data validity ratio, call failure rate and call timeout rate; Obtaining a response time coefficient using the response time; The response time coefficient is obtained by the following formula: Where Q represents the response time coefficient; n represents the number of API calls; T i represents the response time corresponding to the i-th API call; P csi represents the overall call timeout rate corresponding to the completion time of the i-th API call; P csmax Indicates the maximum call timeout rate; T mb Indicates the standard deviation of the response time of the completed API calls at the time of the maximum call timeout rate; comparing the response time coefficient with a preset time coefficient threshold; When the response time coefficient exceeds a preset time coefficient threshold, the return data included in the execution running status parameters are retrieved to obtain the data validity ratio, call failure rate and call timeout rate; The execution status determination coefficient is obtained by using the return data included in the execution status parameter as the ratio of data validity, the call failure rate and the call timeout rate combined with the response time coefficient; The execution status determination coefficient is obtained by the following formula: Where R represents the execution status determination coefficient; m represents the number of unit time that the API call has experienced; P yxj Indicates the proportion of the return data corresponding to the jth unit time as data validity; B j represents the call failure rate corresponding to the j-th unit time; Q j represents the average value of the response time coefficient corresponding to the jth unit time; Q bj represents the standard deviation of the response time coefficient corresponding to the jth unit time; P csj Indicates the call timeout rate corresponding to the j-th unit time; Comparing the execution running state determination coefficient with a preset determination coefficient threshold; When the execution running state determination coefficient is lower than a preset determination coefficient threshold, it is determined that the execution running state in the API calling process is abnormal, and an abnormality alarm is issued.

5. A multi-modal information query and control system for a video safety monitoring platform, characterized in that: The multimodal information query and control system of the video safety monitoring platform includes: The reasoning control module is used to control the gorilla backend reasoning module to receive the input information sent by the user terminal from the front-end web interface in real time for the gradio application module to obtain the reasoning result, and feed back the reasoning result to the user terminal; An API call execution module is used to control the API call module to execute an API call according to a call request corresponding to the API contained in the inference result, and return a call result; The output return control module is used for the gorilla backend reasoning module to generate output returns in combination with user input and API call results, and send the output returns to the gradio application module.

6. According to claim 5, the multimodal information query and control system of the video safety monitoring platform is characterized in that: The inference control module comprises: The input information real-time receiving module is used for the gradio application module to receive the input information sent by the user terminal from the front-end web interface in real time; The inference control execution module is used to call the gorilla backend inference module to perform inference according to the input information sent by the user terminal and obtain the inference result; The result feedback module is used to feedback the inference result obtained by the gorilla backend inference module to the user terminal through the gradio application module.

7. According to claim 5, the multimodal information query and control system of the video safety monitoring platform is characterized in that: The API call execution module includes: A call request sending control module, used to control the gorilla backend reasoning module to send a call request corresponding to the API contained in the reasoning result to the API calling module; An API call result acquisition module, used to control the call request corresponding to the API included in the reasoning result of the API call module to execute the API call and obtain the API call result; The call result feedback module is used to feed back the API call result to the gorilla backend reasoning module.

8. The multimodal information query and control system of the video safety monitoring platform according to claim 5 or 7, characterized in that: The multi-modal information query and control system of the video safety monitoring platform also includes: A status parameter monitoring module is used to monitor the execution status parameters of the API call in real time, wherein the execution status parameters include response time, return data validity ratio, call failure rate and call timeout rate; A response time coefficient acquisition module, used to acquire a response time coefficient using the response time; The response time coefficient is obtained by the following formula: Where Q represents the response time coefficient; n represents the number of API calls; T i represents the response time corresponding to the i-th API call; P csi represents the overall call timeout rate corresponding to the completion time of the i-th API call; P csmax Indicates the maximum call timeout rate; T mb Indicates the standard deviation of the response time of the completed API calls at the time of the maximum call timeout rate; A time coefficient comparison module, used for comparing the response time coefficient with a preset time coefficient threshold; A parameter retrieving module, for retrieving the return data contained in the execution running status parameters, such as the ratio of data validity, call failure rate and call timeout rate, when the response time coefficient exceeds a preset time coefficient threshold; An execution running state determination coefficient acquisition module is used to obtain the execution running state determination coefficient by using the return data included in the execution running state parameter as the ratio of data validity, the call failure rate and the call timeout rate combined with the response time coefficient; The execution status determination coefficient is obtained by the following formula: Where R represents the execution status determination coefficient; m represents the number of unit time that the API call has experienced; P yxj Indicates the proportion of the return data corresponding to the jth unit time as data validity; B j represents the call failure rate corresponding to the j-th unit time; Q j represents the average value of the response time coefficient corresponding to the jth unit time; Q bj represents the standard deviation of the response time coefficient corresponding to the jth unit time; P csj Indicates the call timeout rate corresponding to the j-th unit time; A determination coefficient comparison module, used for comparing the execution running state determination coefficient with a preset determination coefficient threshold; The abnormality determination and alarm module is used to determine that the execution status during the API call process is abnormal and to issue an abnormality alarm when the execution status determination coefficient is lower than a preset determination coefficient threshold.

9. A multi-modal information query control device for a video safety monitoring platform, characterized in that: The multimodal information query control device of the video safety monitoring platform includes a user terminal, a gradio application module, a gorilla back-end reasoning module and an API calling module; wherein the input information output end of the user terminal is data-connected with the input information input end of the gradio application module; the signal output end of the gradio application module is data-connected with the signal input end of the gorilla back-end reasoning module, and the feedback information output end of the gorilla back-end reasoning module is data-connected with the feedback signal input end of the gradio application module; the signal output end of the gorilla back-end reasoning module is data-connected with the signal input end of the API calling module; the feedback information output end of the API calling module is data-connected with the feedback signal input end of the gorilla back-end reasoning module.

Images