Natural language interactive retrieval intelligent security and protection system based on large model

By adopting the natural language interaction retrieval technology of large models in the security system, analyzing video data and outputting query results, the problem of inefficient processing of video data in existing security systems is solved, and efficient video data retrieval and user-friendly interaction are achieved.

CN118939831AInactive Publication Date: 2024-11-12SHENZHEN AIMALL TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411421116.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-12
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing security systems are inefficient in processing video data, resulting in insufficient interactiveness and cannot meet the needs of large-scale video data processing.

Method used

Using a natural language interactive search intelligent security system based on large models, analyzing and processing video data through the analysis platform, obtaining common information and specific information of image frames, and converting user's query information into query instructions through the interactive platform, using large models Perform similarity calculation and output query results to simplify user operation process.

Benefits of technology

It significantly improves the speed and accuracy of video data retrieval, improves the system's operation efficiency and user experience, and enhances the system's intelligence and user-friendliness.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939831A_ABST
    Figure CN118939831A_ABST
Patent Text Reader

Abstract

The invention discloses a natural language interaction retrieval intelligent security and protection system based on a large model. The system comprises an analysis platform and an interaction platform. The analysis platform analyzes and processes the video data to obtain analysis information corresponding to the video data; the interaction platform is automatically built based on a robot process; the interaction platform is configured to convert query information of a user into a query instruction by using a large model, and obtain query content, target time and a target location of the user based on the query instruction; and the interaction platform obtains analysis information matched with the target time and the target location from the analysis platform, performs similarity calculation on the query content and the analysis information obtained through matching, and outputs a query result. According to the method, the speed and the accuracy of monitoring data retrieval can be remarkably improved, so that the requirement of large-scale video data processing can be met; meanwhile, the user operation process can be greatly simplified, the operation convenience is improved, and the system is more user-friendly and intelligent.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of intelligent security technology, and in particular to a natural language interactive retrieval intelligent security system based on a large model. Background Art

[0002] With the widespread application of video surveillance technology, traditional security systems face many challenges in data retrieval and processing. Existing systems usually rely on manual operations and basic retrieval algorithms, resulting in low efficiency and slow response when processing large amounts of video data, especially in real-time monitoring data analysis and user interaction. These problems urgently require a more efficient and intelligent solution to improve the performance of security systems. Summary of the invention

[0003] The purpose of this application is to provide a natural language interactive retrieval intelligent security system based on a large model to solve the technical problem that the existing security system in the prior art has low efficiency in processing video data, resulting in insufficient interactivity. The many technical effects that can be produced by the preferred technical solution among the many technical solutions provided in this application are described in detail below.

[0004] To achieve the above objectives, this application provides the following technical solutions: The present application provides a natural language interactive retrieval intelligent security system based on a large model, including an analysis platform and an interaction platform; the analysis platform analyzes and processes video data to obtain analysis information corresponding to the video data; wherein the video data includes multiple image frames, and the analysis information includes general information and specific information for describing the multiple image frames; the interaction platform is built based on robotic process automation; the interaction platform is configured to use the large model to convert the user's query information into a query instruction, and obtain the user's query content, target time and target location based on the query instruction; the interaction platform obtains analysis information matching the target time and the target location from the analysis platform, performs similarity calculation on the query content and the analysis information obtained after matching, and outputs the query result.

[0005] In some embodiments, the analysis platform includes an acquisition module, a detection module and an analysis module; the acquisition module is configured to acquire video data containing multiple objects; the detection module is configured to detect the video data and acquire key frames corresponding to each object; the analysis module is configured to analyze each of the key frames and acquire the general information and the specific information corresponding to the key frames.

[0006] In some embodiments, the detection module is configured to obtain objects in each image frame in the video data and generate a confidence probability and a bounding box for each object.

[0007] In some embodiments, the detection module is configured to obtain a key frame corresponding to each object based on the motion state of each object in combination with time series data, wherein the key frame is the image frame with the highest confidence probability.

[0008] In some embodiments, the analysis module is configured to perform overall picture analysis on each of the key frames to obtain a general description as the general information; the analysis module is configured to perform independent analysis on each of the bounding boxes to obtain an object description as the specific information.

[0009] In some embodiments, the analysis module is configured to vectorize the general information and the specific information respectively to obtain general vector information and specific vector information.

[0010] In some embodiments, the analysis module analyzes and processes the video data based on a plurality of different categories of model prompt words to obtain the analysis information.

[0011] In some embodiments, the interactive platform is configured to vectorize the query content, use the large model to calculate the semantic similarity between the general vector information and the specific vector information corresponding to the key frame and the vectorized query content, and filter out at least one of the key frames as the query result.

[0012] In some embodiments, the user's query information is voice information; the interactive platform is also configured to convert the voice information into text information using the large model, and obtain the query instruction based on the text information.

[0013] In some embodiments, it also includes a camera module for providing the video data; the interactive platform is configured to filter out camera operation commands from the query instructions, and obtain the video data from the camera module as the query result based on the camera operation commands, or send the video data to the analysis platform.

[0014] Implementing one of the above-mentioned technical solutions of the present application has the following advantages or beneficial effects: in the present application, by constructing an analysis platform to analyze and process the video data, general information and specific information describing the image frames of the video data can be obtained, which can significantly improve the speed and accuracy of video data, such as monitoring data retrieval, thereby meeting the needs of large-scale video data processing; at the same time, the present application converts the user's query information into query instructions by constructing an interactive platform, and outputs the query results by calculating the similarity with the analysis information, thereby improving the system's operating efficiency and user experience; in addition, the interactive platform of the present application is built based on robotic process automation, which can greatly simplify the user's operating process, improve the convenience of operation, and make the system more user-friendly and intelligent. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. It is obvious that the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 It is a structural block diagram of a natural language interactive retrieval intelligent security system based on a large model according to an embodiment of the present application; Figure 2 It is a flow chart of matching video data and query information of a smart security system based on a large model of natural language interactive retrieval in an embodiment of the present application; Figure 3 It is a flowchart of query instruction processing in an embodiment of the present application; Figure 4 It is a structural block diagram of the processing device of an embodiment of the present application.

[0016] In the figure: 1. Natural language interactive retrieval intelligent security system based on large model; 10. Analysis platform; 11. Acquisition module; 12. Detection module; 13. Analysis module; 20. Interaction platform; 30. Camera module; 4. Processing device; 40. Memory; 41. Processor. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of the present application clearer, the various exemplary embodiments to be described below will refer to the corresponding drawings, which constitute a part of the exemplary embodiments, wherein various exemplary embodiments that may be used to implement the present application are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present disclosure. It should be understood that they are only examples of processes, methods, and devices that are consistent with some aspects disclosed in the present application as detailed in the attached claims, and other embodiments may also be used, or the embodiments listed herein may be modified in structure and function without departing from the scope and essence of the present application.

[0018] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", etc. indicate the orientation or positional relationship based on the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the elements referred to must have a specific orientation, be constructed and operated in a specific orientation. The terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "multiple" means two or more. The terms "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a communication connection, a direct connection, an indirect connection through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0019] In order to illustrate the technical solution described in the present application, a specific embodiment is provided below, and only the parts related to the embodiment of the present application are shown.

[0020] The present application provides a natural language interactive retrieval intelligent security system 1 based on a large model. The natural language interactive retrieval intelligent security system 1 based on a large model involved in the embodiment of the present application can obtain the user's query information and output the corresponding query results. The natural language interactive retrieval intelligent security system 1 based on a large model can output the corresponding query results based on video data. The natural language interactive retrieval intelligent security system 1 based on a large model of the present application can obtain the user's query information, and compare the user's query information with the analysis information of the video data to obtain and output the corresponding query results.

[0021] In some embodiments, Figure 1As shown, the natural language interactive retrieval intelligent security system 1 based on a large model may include an analysis platform 10 and an interaction platform 20. The analysis platform 10 may be used to obtain analysis information of video data; the interaction platform 20 may be used to obtain query information of the user and output corresponding query results based on the analysis information.

[0022] In some embodiments, the analysis platform 10 may analyze and process the video data to obtain analysis information corresponding to the video data; wherein the video data may include multiple image frames, and the analysis information may include general information and specific information for describing the multiple image frames.

[0023] In some embodiments, the analysis platform 10 may include an acquisition module 11, a detection module 12, and an analysis module 13. In some embodiments, the acquisition module 11 may be configured to acquire video data containing multiple objects; the detection module 12 may be configured to detect the video data and acquire a key frame corresponding to each object; in some embodiments, the analysis module 13 may be configured to analyze each key frame and acquire general information and specific information corresponding to the key frame.

[0024] In some embodiments, the video data may include real-time video data and cached video data. The acquisition module 11 may acquire real-time video data from a camera or a monitoring device. The cached video data may refer to video data cached in a local device or a network device, or may be video data uploaded or retrieved by a user.

[0025] In some embodiments, the video data may include multiple image frames. The image frames may include multiple objects. The objects may include items, people, or vehicles, etc. For example, for real-time video data of a security camera, the multiple objects may refer to people, items, etc. that appear in the real-time video data.

[0026] In some embodiments, detection module 12 may be configured to obtain objects in each image frame within the video data and generate a confidence probability and a bounding box for each object.

[0027] In some embodiments, the detection module 12 may process each image frame in sequence using an object detection algorithm, detect each object in the image, and generate a confidence probability and a bounding box (BBOX) for each object. The object detection algorithm may include the following steps: feature extraction, such as extracting features using a CNN (convolutional neural network), and then generating candidate regions, such as generating candidate regions based on whether an object exists through a region proposal network (RPN), and then classifying the candidate regions to obtain a bounding box.

[0028] In some embodiments, the object detection algorithm may be one of a YOLO algorithm, an SSD (Single Stage Detector) algorithm, a Faster R-CNN (Fast Regional Convolutional Neural Network) algorithm, or a CenterNet (Center Point Network) algorithm.

[0029] In some embodiments, the detection module 12 may also utilize a deep learning model to perform real-time object detection and tracking to obtain a confidence probability and a bounding box for each object.

[0030] In some embodiments, the detection module 12 may be configured to obtain a key frame corresponding to each object based on the motion state of each object in combination with the time series data, wherein the key frame may be an image frame with the highest confidence probability.

[0031] In some embodiments, the detection module 12 may use Kalman filtering to predict the motion state of the object. The Kalman filtering may be obtained by the following steps: defining state variables, constructing motion equations based on the state variables, initializing state steps and predicting the motion state of the object based on the motion equations.

[0032] In some embodiments, the detection module 12 may use an extended Kalman filter or an unscented Kalman filter to predict the motion state of the object.

[0033] In some embodiments, each object may correspond to at least one key frame, that is, at least one image frame may be selected from the video data as a key frame. The key frame may represent the best detection moment of the object in the video data.

[0034] In some embodiments, the analysis module 13 may be configured to analyze each key frame to obtain analysis information, where the analysis information may include general information and specific information.

[0035] In some embodiments, the analysis module 13 can use a vision-language model (VLM) to analyze each key frame to obtain analysis information. The vision-language model is a deep learning model that combines computer vision and natural language processing technology, and its purpose is to understand and generate the association between images and texts. The vision-language model can include at least one of a VGG network (VGG convolutional neural network) + RNN (recurrent neural network) model and a Faster R-CNN (fast regional convolutional neural network) model + LSTM (long short-term memory network) model.

[0036] In some embodiments, the analysis module 13 can be configured to perform an overall picture analysis on each key frame and obtain a general description as general information. The general description can be text information that at least includes the overall picture information in the key frame. For example, the general description can be "This picture is an indoor scene taken by a surveillance camera, showing two people. A lady is entering from the right side of the picture, and another lady is standing behind the counter on the left. The interior decoration is simple, and the walls and floors are white. There are some items on the left counter, including drinks and office supplies." It should be noted that the above examples are only used to explain the general description and do not constitute a limitation to the embodiments of the present application.

[0037] In some embodiments, the analysis module 13 may be configured to independently analyze each bounding box and obtain an object description as specific information. The analysis module 13 may independently analyze the bounding box in each key frame and obtain multiple specific information. Independent analysis may mean that the obtained object description corresponds to the bounding box one by one. For example, the bounding box A1 in key frame A and the bounding box B1 in key frame B may be independently analyzed to obtain the object description corresponding to the bounding box A1 and the object description corresponding to the bounding box B1 as specific information.

[0038] In some embodiments, the analysis module 13 may independently analyze all or part of the bounding boxes in the key frame to obtain a plurality of specific information. In some embodiments, a key frame may include a plurality of specific information.

[0039] In some embodiments, the object description may be text information including at least the object information in the key frame. For example, the object description may be "This is a blurry surveillance video screenshot showing a person lying sideways. The person appears to be wearing a light-colored top and dark-colored pants, with hair tied in a ponytail." It should be noted that the above example is only used to explain the object description and does not constitute a limitation to the embodiments of the present application.

[0040] In some embodiments, the analysis module 13 may be configured to vectorize the general information and the specific information respectively to obtain the general vector information and the specific vector information.

[0041] In some embodiments, the analysis module 13 can use an Embedding algorithm to vectorize the general information and the specific information respectively. The Embedding algorithm can be used to map data in a high-dimensional space to a low-dimensional space while retaining the information and structure in the original data. That is, the general vector information can be a vectorized representation of the general information, and the specific vector information can be a vectorized representation of the specific information.

[0042] In some embodiments, the Embedding algorithm may be at least one of a Word2Vec algorithm (word vector algorithm), a GloVe algorithm (global vector word representation algorithm), and a Node2Vec algorithm (node ​​vector algorithm).

[0043] In some embodiments, the large model-based natural language interactive retrieval intelligent security system 1 may further include a storage module. The storage module may be configured to store general vector information, specific vector information, time information, and location information of each key frame.

[0044] In some embodiments, the time information can be obtained based on time series data. Figure 2 As shown, the general vector information, specific vector information, time information and location information of a key frame can be stored as a set of data information of the video data. That is, the storage module can store multiple sets of data information of the video data. In some embodiments, the data information related to the key frame can be obtained based on the key frame, and the corresponding key frame can also be obtained from the storage module based on the data information, such as time information and / or location information.

[0045] In some embodiments, the storage module may be a database. For example, the storage module may be a SQLite database, a HSQLDB database, a SQLCipher database, or a MemSQL database.

[0046] In some embodiments, the analysis module 13 can analyze and process the video data based on multiple different categories of model prompt words to obtain analysis information. In this case, through multiple different angles, that is, through different categories to build model prompt words during analysis, the same scene is analyzed multiple times from different angles using the visual-language model, thereby obtaining a comprehensive picture description, which can be separated from a single application scenario and realize full-content universal query in different scenarios.

[0047] In some embodiments, the interactive platform 20 may be configured to convert the user's query information into a query instruction using a large model, and obtain the user's query content, target time, and target location based on the query instruction.

[0048] In some embodiments, the user's query information may be text information. The interactive platform 20 may use the LLM large language model to convert the text information into a query instruction. The query instruction may have a preset format, that is, the interactive platform 20 may use the LLM large language model to convert the text information into a formatted query instruction. The interactive platform 20 may obtain the query instruction by extracting and formatting the text information.

[0049] In some embodiments, the user's query information may be voice information. The interactive platform 20 may also be configured to convert the voice information into text information using a large model, and obtain the query instruction based on the text information. In some embodiments, the user's voice information may be converted into text information using an automatic speech recognition (ASR) model. Selecting a high-precision ASR model can improve the accuracy of voice-to-text conversion.

[0050] In some embodiments, the interactive platform 20 may obtain analysis information matching the target time and the target location from the analysis platform 10, perform similarity calculation on the query content and the analysis information obtained through matching, and output the query result. The similarity calculation may refer to semantic similarity calculation.

[0051] In some embodiments, the interactive platform 20 can be configured to vectorize the query content, and obtain multiple key frames based on the target time and target location, and use a large model to calculate the semantic similarity between the general vector information and specific vector information corresponding to the key frames and the vectorized query content, and filter out at least one key frame as the query result.

[0052] In some embodiments, the interactive platform 20 can embed and vectorize the query content to obtain the query content that has been vectorized. The interactive platform 20 can obtain corresponding data information and key frames corresponding to the data information based on the user's query information. Specifically, the interactive platform 20 can obtain data information and key frames that are consistent with the target location and target time based on the user's target location and target time. For example, the user's target location is "company front desk", and the user's target time is "12 o'clock to 14 o'clock". The interactive platform 20 can obtain multiple key frames in the data information whose time information is 12 o'clock, 13 o'clock or 14 o'clock, and the location information is the company front desk. It should be noted that the above time information and location information are only for example.

[0053] In some embodiments, the interactive platform 20 may calculate the semantic similarity between the data information of each key frame and the query content processed by vectorization. In some embodiments, the interactive platform 20 may sort the calculation results from large to small according to the relevance, and select at least one key frame as the query result based on the relevance.

[0054] In some embodiments, the large model-based natural language interactive retrieval intelligent security system 1 may further include a camera module 30 for providing video data. The interactive platform 20 may also be configured to filter out camera operation commands from the query instructions, and obtain video data from the camera module 30 as a query result based on the camera operation command, or send the video data to the analysis platform 10.

[0055] In some embodiments, the camera module 30 may include multiple cameras for acquiring video data. The location information in the video data may be the location where the camera is located or the location photographed by the camera.

[0056] In some embodiments, the interactive platform 20 may classify the query instructions. Specifically, the interactive platform 20 may classify the query instructions into camera-related instructions and irrelevant instructions, wherein the camera-related instructions may refer to instruction information that requires obtaining video data through the camera module 30 .

[0057] In some embodiments, the interactive platform 20 can convert camera-related instructions into camera operation commands based on the large language model, such as playback or retrieval of video data, etc. The large language model can have special system prompt words, that is, the interactive platform 20 can extract camera-related instructions from the semantic level to obtain camera operation commands.

[0058] In some embodiments, the interactive platform 20 can obtain the video data of the camera module 30 in real time based on the camera operation command through the RTSP protocol. The RTSP (Real-Time Streaming Protocol) protocol is a network protocol that can be used to control the interaction between the streaming media server and the client.

[0059] In some embodiments, the interactive platform 20 can be built based on robotic process automation (RPA technology). RPA technology (Robotic Process Automation) is a technology that uses software robots to simulate and integrate human interaction in digital systems. RPA technology can be used to automate repetitive and regular tasks. For example, the interactive platform 20 can classify query instructions based on RPA technology and obtain video data from the camera module 30.

[0060] In some embodiments, the interactive platform 20 may include an AI (artificial intelligence) agent, and the AI ​​agent may be integrated through multiple large language models. Specifically, the AI ​​agent may include a first large language model, a second large language model, and a third large language model. Specifically, Figure 3 As shown, the first language model can be used to divide query instructions into camera-related instructions and irrelevant instructions, and the camera-related instructions may include camera operation instructions; the second language model can convert camera-related instructions into camera operation commands, and the camera operation commands can be obtained by formatting; the third language model can be used to generate text results.

[0061] In some embodiments, the query result may include at least one of an image result, a text result, and a voice result.

[0062] In some embodiments, the image result may be at least one key frame. In some embodiments, the text result may be a reply text generated based on the data information of the key frame and the query information of the user. For example, the reply text may be "The person in this image is a person in a white shirt who appeared at the company's front desk at 2:00 p.m. on July 24, 2024."

[0063] In some embodiments, the text result may also be a reply text generated based on the query instruction and the camera operation command. For example, the reply text may be "The surveillance video of the company's front desk at 2:00 p.m. on July 24, 2024 has been replayed."

[0064] In some embodiments, the interactive platform 20 can convert the text results into voice results through a TTS (Text-to-Speech) model. In this case, the voice feedback function can be enhanced, thereby improving the user interaction experience and the usability of the system.

[0065] In the present application, by constructing an analysis platform 10 to analyze and process the video data, and obtaining general information and specific information describing the image frames of the video data, the monitoring data, such as the speed and accuracy of monitoring data retrieval, can be significantly improved, thereby meeting the needs of large-scale video data processing; at the same time, the present application converts the user's query information into query instructions by constructing an interactive platform 20, and outputs the query results by calculating the similarity with the analysis information, which can improve the system's operating efficiency and user experience; in addition, the interactive platform 20 of the present application is built based on robotic process automation, which can greatly simplify the user's operating process, improve the convenience of operation, and make the system more user-friendly and intelligent.

[0066] The present application also includes a security interaction method, which may include the following steps: Analyze and process the video data to obtain analysis information corresponding to the video data; wherein the video data may include multiple image frames, and the analysis information may include general information and specific information for describing the multiple image frames; The user's query information is converted into a query instruction using a large model, and the user's query content, target time and target location are obtained based on the query instruction; Obtain analysis information that matches the target time and target location, perform similarity calculation on the query content and the analysis information obtained through matching, and output the query result.

[0067] Those skilled in the art can understand that all or part of the features / steps of the above-mentioned method embodiments can be implemented by methods, data processing systems or computer programs, and these features can be implemented without hardware, entirely by software, or by a combination of hardware and software. The aforementioned computer program can be stored in one or more computer-readable storage media, and the storage medium stores the computer program. When the computer program is executed (such as by a processor), the steps of the above-mentioned security interaction method embodiments are executed.

[0068] The aforementioned storage media that can store program codes include: static hard disks, solid-state hard disks, random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), optical storage devices, magnetic storage devices, flash memory, magnetic disks or optical disks and / or combinations of the above devices, that is, they can be implemented by any type of volatile or non-volatile storage device or a combination thereof.

[0069] like Figure 4 As shown, the present application also provides an embodiment of a processing device 4, including one or more processors 41 and a memory 40; wherein the memory 40 is used to store one or more computer programs, and the one or more processors 41 are used to execute one or more computer programs stored in the memory 40, so that the processor 41 executes the features / steps of the above-mentioned security interaction method embodiment.

[0070] The above is only the preferred embodiment of the present application. Those skilled in the art will appreciate that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application belong to the protection scope of the present application.

Claims

1. A natural language interactive retrieval intelligent security system based on a large model, characterized in that: Includes analysis platform and interaction platform; The analysis platform analyzes and processes the video data to obtain analysis information corresponding to the video data; wherein the video data includes a plurality of image frames, and the analysis information includes general information and specific information for describing the plurality of image frames; The interactive platform is built based on robotic process automation; the interactive platform is configured to convert the user's query information into a query instruction using the large model, and obtain the user's query content, target time and target location based on the query instruction; The interactive platform obtains analysis information matching the target time and the target location from the analysis platform, performs similarity calculation on the query content and the analysis information obtained through matching, and outputs a query result.

2. According to the large model-based natural language interactive retrieval intelligent security system of claim 1, it is characterized in that: The analysis platform includes an acquisition module, a detection module and an analysis module; The acquisition module is configured to acquire video data containing a plurality of objects; The detection module is configured to detect the video data and obtain a key frame corresponding to each object; The analysis module is configured to analyze each of the key frames to obtain the general information and the specific information corresponding to the key frame.

3. The natural language interactive retrieval intelligent security system based on a large model according to claim 2 is characterized in that: The detection module is configured to obtain objects in each image frame in the video data and generate a confidence probability and a bounding box for each object.

4. The natural language interactive retrieval intelligent security system based on a large model according to claim 3 is characterized in that: The detection module is configured to obtain a key frame corresponding to each object based on the motion state of each object in combination with time series data, wherein the key frame is an image frame with the highest confidence probability.

5. The natural language interactive retrieval intelligent security system based on a large model according to claim 3 is characterized in that: The analysis module is configured to perform overall picture analysis on each of the key frames and obtain a general description as the general information; the analysis module is configured to perform independent analysis on each of the bounding boxes and obtain an object description as the specific information.

6. The natural language interactive retrieval intelligent security system based on a large model according to claim 2 is characterized in that: The analysis module is configured to vectorize the general information and the specific information respectively to obtain general vector information and specific vector information.

7. The natural language interactive retrieval intelligent security system based on a large model according to claim 2 is characterized in that: The analysis module analyzes and processes the video data based on a plurality of different categories of model prompt words to obtain the analysis information.

8. The natural language interactive retrieval intelligent security system based on a large model according to claim 6 is characterized in that: The interactive platform is configured to vectorize the query content, use the large model to calculate the semantic similarity between the general vector information and the specific vector information corresponding to the key frame and the vectorized query content, and filter out at least one of the key frames as the query result.

9. The natural language interactive retrieval intelligent security system based on a large model according to claim 1 is characterized in that: The user's query information is voice information; the interactive platform is also configured to use the large model to convert the voice information into text information, and obtain the query instruction based on the text information.

10. The natural language interactive retrieval intelligent security system based on a large model according to claim 1 is characterized in that: It also includes a camera module for providing the video data; the interactive platform is configured to filter out camera operation commands from the query instructions, and obtain the video data from the camera module as the query result based on the camera operation commands, or send the video data to the analysis platform.

Citation Information

Patent Citations

  • Interactive video retrieval method and system based on large language model

    CN118394969A

  • Generative large model-based natural language interactive security video retrieval system and device

    CN118551077A

  • Video retrieval method and system based on multi-modal information fusion

    CN118626672A

  • Generative artificial intelligence enterprise search

    US20240202221A1

Cited By

  • Video search method by natural language by use of Vision-Language Model for video surveillance system

    KR102981845B1