A natural language interactive security video retrieval system and device based on generative large model
Through a natural language interactive security video retrieval system combining large language model and large visual model, accurate understanding of video and text and semantic conversion is achieved, solving the satisfaction of user query needs in security scenarios, improving retrieval efficiency and effect, and providing Efficient, intelligent and privacy-protected solutions.
Patent Information
- Application Number
- CN202411002019.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-07-25
AI Technical Summary
The prior art is difficult to achieve accurate understanding and semantic conversion of video (image) and text in security scenarios, resulting in the inability to effectively meet user query requirements, especially when dealing with complex semantic and syntactic structures.
Adopt natural language interactive security video retrieval system based on generative large models, combining large language models and large visual models, real-time acquisition, object detection, and attributes of video data through video acquisition, processing and analysis, semantic similarity calculation and other modules. Extract semantic matching between user queries, providing efficient information retrieval and display.
Improve the intelligence level of the video retrieval system, better understand the context, recognize polysemous words, generate more accurate semantic information, realize cross-language translation and semantic transformation, significantly improve efficiency and effect, and provide efficient, intelligent, scalable and A security video retrieval solution for privacy protection.
Smart Images

Figure CN118551077B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of large model interaction technology, and in particular to a natural language interactive security video retrieval system and device based on a generative large model. Background Art
[0002] With the continuous advancement of AI technology, large language models, large visual models, and the multimodal large models that will continue to emerge in the future not only play a key role in enhancing the flexibility and application scope of AI systems, protecting privacy issues and reducing costs, improving response speed and customized development, but also have great potential in improving user experience and solving complex problems.
[0003] The Large Visual Model (LVM) has significant advantages over traditional computer vision algorithm models. LVM uses deep learning technology to automatically learn and extract complex features from large-scale data without manually designing features, which improves the accuracy and robustness of image classification and object detection. In addition, LVM has multimodal processing capabilities and can simultaneously understand and generate visual, textual, and audio information, enabling it to perform well in cross-modal tasks. LVM uses an end-to-end learning approach, which simplifies the development process and enhances generalization capabilities.
[0004] After searching, the publication number is CN108052882A, and the invention name is: A Chinese invention of an operating method for an intelligent security monitoring system. It discloses convenient operation, timely information sharing, improved case handling efficiency, and strong practicality.
[0005] The Large Language Model (LLM) has significant advantages over traditional natural language processing (NLP) algorithm models.
[0006] LLMs are trained using deep learning techniques and large-scale data to automatically learn complex language patterns and long-distance dependencies, which makes them excellent in understanding and generating language.
[0007] In security scenarios, the combination of a large language model and a large visual model can achieve more efficient semantic matching and information query. Specifically, the large visual model can detect and identify targets in surveillance images and convert the recognition results into text information; then, the large language model can deeply understand and analyze these text information and semantically match them with user query requirements; finally, the relevant information is returned to the user based on the matching results. This combination can greatly improve the intelligence level of the natural language interactive security video retrieval system based on the generative large model and meet the query needs of users in security scenarios.
[0008] In contrast, traditional NLP algorithms usually rely on manually designed features and rules, which have limitations in dealing with complex semantic and syntactic structures. In addition, LLM's Transformer architecture captures contextual relationships through a self-attention mechanism and can generate contextually relevant and coherent text. After pre-training and fine-tuning, it can demonstrate outstanding performance in a variety of tasks, including text generation, language translation, content summarization, and code generation. To this end, a natural language interactive security video retrieval system and its device based on a generative large model are proposed. Summary of the invention
[0009] In view of this, the present invention provides a natural language interactive security video retrieval system and device based on a generative large model to solve or alleviate the technical problems existing in the prior art. Through the accurate understanding and semantic conversion of videos (images) and texts by large language models and large visual models, semantic matching of stored information and user information is performed to meet the user's query needs in security scenarios, thereby providing at least a beneficial option.
[0010] The technical solution of the embodiment of the present invention is achieved as follows:
[0011] In a first aspect, the present invention provides a natural language interactive security video retrieval system based on a generative large model, comprising:
[0012] Video acquisition module, video processing and analysis module, large language model processing module, large visual model processing module, semantic similarity calculation module, human-computer interaction and result display module, data storage and management module;
[0013] The video acquisition module is used to collect video data in real time or periodically; the video acquisition module includes a high-definition camera installed inside a building, inside an entertainment venue, and within a preset distance range, and the video acquisition module is connected to the video processing and analysis module and the data storage and management module respectively;
[0014] The video processing and analysis module is used to receive the video data transmitted by the video acquisition module, perform target detection on the video data using a deep learning algorithm, identify the human objects appearing in the video data, and extract the attribute information of the human objects; and store the attribute vectorization results obtained after data vectorization of the attribute information corresponding to each of the human objects in a vector library;
[0015] The data storage and management module is connected to the semantic similarity calculation module and the human-computer interaction and result display module respectively, and the data storage and management module includes the vector library, and the data storage and management module is used to store the video data and the attribute vectorization result corresponding to the video data;
[0016] The large language model processing module is connected to the human-computer interaction and result display module, and the large language model processing module is used to receive an input query instruction; perform semantic analysis on the query instruction to convert it into formatted data for retrieval; and perform data vectorization on the formatted data corresponding to the query instruction;
[0017] The large visual model processing module is connected to the semantic similarity calculation module and the video processing and analysis module respectively, and the semantic similarity calculation module is used to calculate the semantic similarity between the data vectorization result corresponding to the query instruction and the attribute vectorization result corresponding to the person object stored in the vector library; according to the similarity sorting result, the target video clip matching the query instruction is screened out;
[0018] The human-computer interaction and result display module includes a user interaction display interface, which is used to display the target video clip on the user interaction display interface; and respond to user operations on the user interaction display interface, and perform video operation actions corresponding to the user operations; wherein the user operations include zooming in / out images, playing / pausing videos, and exporting data.
[0019] Further preferably, the video acquisition module is used to continuously capture at least three instantaneous images from the video data acquired in real time or periodically with a preset fixed time interval t as a step length, wherein the instantaneous image is included in t b The initial image at time t m The intermediate image at time t e Node image at the moment;
[0020] The video acquisition module is used to process the instantaneous image to obtain a pixel mean matrix, and extract the attribute information of the person object based on the pixel mean matrix.
[0021] Further preferably, the video acquisition module is used to process the instantaneous image to obtain a pixel mean matrix, and extract the attribute information of the person object according to the pixel mean matrix, including:
[0022] The instantaneous image is evenly divided into cell regions with a step length of ε, and the cell regions are numbered in sequence, namely A1, A2, A3, ..., A n ;
[0023] Obtain A1-A obtained after processing each image in the instantaneous image n The pixel mean value P of the pixels in the cell area bn , P mn , P en, and according to the pixel mean P of the pixels in the cell area bn , P mn , P en Generate a pixel mean matrix, where P bn , P mn , P en The initial image, intermediate image, and node image correspond to A1-A respectively. n The pixel mean of the cell area;
[0024] According to the pixel mean matrix, a first pixel mean difference matrix ΔP between the initial image and the intermediate image is calculated, and a second pixel mean difference matrix ΔP' between the intermediate image and the node image is calculated; wherein the step length ε of the cell area is not greater than 1 / 100 of the instantaneous image size;
[0025] A motion trajectory of the human object is generated according to the first pixel mean difference matrix ΔP and the second pixel mean difference matrix ΔP′, and the attribute information includes the motion trajectory.
[0026] Further preferably, the video processing and analysis module is a large visual model LVM, and the large visual model LVM uses a Transformer-based computer vision model to perform target detection and feature extraction on the video data to obtain the attribute vectorization result;
[0027] The large language model processing module is a large language model LLM, and the large language model LLM uses a Transformer-based natural language processing model to encode the query instruction to obtain a data vectorization result of a text feature vector type.
[0028] Further preferably: the Transformer includes a self-attention mechanism, a multi-head attention mechanism and an encoder-decoder structure;
[0029] The self-attention mechanism is used to focus on the specific location of the person object in the video data with the video frame as the spatial range; wherein the video frame includes a first video frame, a second video frame, a third video frame and a fourth video frame;
[0030] The multi-head attention mechanism is used to execute multiple attention heads in parallel to perform person object recognition on the first video frame, the second video frame, the third video frame, and the fourth video frame respectively according to the specific position of the person object in the video frame, so as to obtain a target detection result;
[0031] The encoder-decoder structure is used to extract and encode the features of the target detection result corresponding to the video frame, and input the encoded signal into the decoder for character recognition to obtain the attribute vectorization result.
[0032] Further preferably, the video processing and analysis module is also used to continuously collect the video data within 24 hours, use a deep learning algorithm to perform real-time analysis on the video data, identify expected dangerous events, and the dangerous events include fire events and smoke events. When a dangerous event is detected, the time, location, and related personal information of the event are automatically recorded, and the related attributes of the dangerous event are vectorized and stored in a vector library;
[0033] The large language model processing module is also used to receive a query instruction input by a user; perform semantic analysis on the query instruction to extract key information, wherein the key information includes time, location and event type;
[0034] Vectorize the key information of the query instruction to compare it with the data in the vector library;
[0035] The semantic similarity calculation module is also used to calculate the semantic similarity between the vectorized result of the user query and the vectorized result of the dangerous event attribute stored in the vector library; according to the similarity sorting result, the dangerous event record that best matches the user query is screened out;
[0036] The human-computer interaction and result display module is also used to display the screened dangerous event records to the user in a visual form, providing a user interaction interface, allowing the user to further view the detailed information of the event, the detailed information including the complete video of the event and the identity information of the relevant personnel.
[0037] Further preferably, the natural language interactive security video retrieval system based on the generative big model also includes an early warning and notification module, which is used to automatically trigger an early warning mechanism when the dangerous event is detected, allowing users to customize early warning rules.
[0038] Further preferably: the video processing and analysis module is further connected to a privacy protection module, and the privacy protection module is used to protect the privacy of relevant attributes of characters and desensitize sensitive information when processing and analyzing video data, specifically including:
[0039] The privacy protection module is used to pre-process the video data, wherein the pre-processing includes frame extraction and key frame identification;
[0040] Detecting sensitive information in the video data using facial recognition and optical character recognition;
[0041] According to the type of the identified sensitive information, the detected face or sensitive information is blurred using Gaussian blur to obtain a desensitized video;
[0042] Post-process the desensitized video.
[0043] Further preferably, the related attributes include timestamp, location, gender, age and clothing color, and the formatted data includes time, location and clothing features.
[0044] The second aspect: A natural language interactive security video retrieval device based on a generative large model, comprising a memory and one or more processors, wherein a computer program is stored on the memory, and when the processor executes the computer program, it is used to implement the above-mentioned natural language interactive security video retrieval system based on a generative large model.
[0045] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution:
[0046] The method of large-scale model understanding and semantic transformation is adopted, which enables it to process larger-scale data in specific scenarios, capture more complex language patterns, better understand the context, identify the different meanings of polysemous words, and generate more reasonable and accurate semantic information.
[0047] In addition, large models perform particularly well in processing multilingual tasks and can effectively perform cross-language translation and semantic conversion, greatly improving the efficiency and effectiveness of various natural language processing applications;
[0048] The method of data vectorization is used to achieve information matching, so that complex original information can be converted into a high-dimensional vector representation. This representation captures more semantic information and subtle differences, thereby improving the data representation capability and matching accuracy, significantly improving the effect and efficiency, and making the processing of large-scale data more accurate and efficient.
[0049] Secondly, the natural language interactive security video retrieval system based on the generative large model combines video processing, natural language processing, semantic similarity calculation and other technologies to achieve retrieval of specific targets, including the following advantages:
[0050] Efficiency: The large visual model LVM processes video data in real time and can quickly extract relevant attributes of people.
[0051] The large language model (LLM) is used to convert user queries into formatted data and quickly find matching video clips.
[0052] Intelligence: The system can automatically analyze video data and perform target retrieval without human intervention.
[0053] By calculating semantic similarity, user queries and video data can be matched more accurately.
[0054] Scalability: The system can add more cameras and attribute extraction capabilities as needed to accommodate more complex scenarios.
[0055] More natural language processing technologies can be integrated to support richer query methods.
[0056] Ease of use: Provide a user interface to facilitate users to enter queries and view results.
[0057] The system's operating status can be easily managed through the system monitoring and maintenance module.
[0058] Privacy protection: The privacy protection module is taken into consideration in the design to ensure the security and compliance of sensitive information.
[0059] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0061] Figure 1 The overall structure diagram of a natural language interactive security video retrieval system based on a generative large model of the present invention;
[0062] Figure 2 It is a flow chart of a method for implementing a natural language interactive security video retrieval system based on a generative large model of the present invention;
[0063] Figure 3 A schematic block diagram of a computer device provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION
[0064] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.
[0065] Large Model:
[0066] Large models refer to machine learning models with large-scale parameters and complex computing structures. These models are usually built from deep neural networks and have billions or even hundreds of billions of parameters. The design purpose of large models is to improve the expressiveness and predictive performance of the model and to be able to handle more complex tasks and data. Large models are widely used in various fields, including natural language processing, computer vision, speech recognition, and recommendation systems. Large models learn complex patterns and features by training massive data, have more powerful generalization capabilities, and can make accurate predictions for unseen data. The large models used in the present invention include large language models (Large Language Model, LLM) and large vision models (Large Vision Model, LVM).
[0067] Large Language Model (LLM):
[0068] Large language models are usually natural language processing models with large-scale parameters and computing power, such as OpenAI's GPT series models. These models can be trained with a large amount of data and parameters to generate human-like text or answer questions in natural language. Large language models are widely used in natural language processing, text generation, and intelligent dialogue.
[0069] Large Visual Model (LVM):
[0070] The Big Vision Model is a machine learning model based on a neural network. Through training, the model can automatically extract useful features from a large amount of image and video data, and perform tasks such as classification, detection, and recognition. Its core principle is to use neural networks to simulate the hierarchical structure of the human brain, and extract increasingly abstract feature representations by processing input data layer by layer.
[0071] Event attributes:
[0072] Including attributes of other dangerous events (fire, overcrowding, etc.) such as the time of appearance of a person, gender, clothing, hair color, etc.; attributes include: "10 o'clock in the morning, female, young, yellow upper body, black lower body, long hair, wearing a hat, no glasses, short sleeves, shorts, no shoes, no bag, no backpack, with a handbag, no shoulder bag".
[0073] User message (Query):
[0074] It refers to the information that users want to obtain from the video stream, such as searching for people with specific physical attributes at a certain location and time: "an adult man wearing a white short-sleeved shirt and red shorts around 10 o'clock in the morning in Shenzhen."
[0075] Semantic conversion:
[0076] It means that the big model converts the information in the input video stream, input picture, and input text into the required features (which can be understood as keywords such as time, place, person, event, and event attributes) to facilitate subsequent queries.
[0077] Data vectorization (Embedding) and vector library:
[0078] Data vectorization refers to the process of converting various types of unstructured data into fixed-length digital vectors using machine learning technology. The vectorized data can be mathematically represented as a fixed-length digital list (multidimensional array). The significance of data vectorization is that it implements a computable representation method to express the semantic information of unstructured data, mapping unstructured data into a fixed-dimensional vector space, so that the machine can "understand" the inherent relationship of the data and perform operations.
[0079] Vector database is a new type of unstructured data management system, which is specially used to store and manage vectorized data. It is mainly used in scenarios related to artificial intelligence such as machine learning. Unlike traditional databases, data in vector databases is not stored in the form of tabular structured data, but embedded in a high-dimensional space in the form of vectors. This vectorized data has the ability to capture the semantic features of unstructured data (such as text, images, audio, etc.), so that vector databases can understand the relationship between different data from a semantic perspective and realize fast search and matching of unstructured data by calculating the distance between vectors.
[0080] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. Figure 1-Figure 2 shown.
[0081] Embodiment 1
[0082] A natural language interactive security video retrieval system based on a generative large model, comprising:
[0083] Video acquisition module, video processing and analysis module, large language model processing module, large visual model processing module, semantic similarity calculation module, human-computer interaction and result display module, data storage and management module;
[0084] The video acquisition module is used to collect video data in real time or periodically; the video acquisition module includes a high-definition camera installed inside a building, inside an entertainment venue, and within a preset distance range, and the video acquisition module is connected to the video processing and analysis module and the data storage and management module respectively;
[0085] The video processing and analysis module is used to receive the video data transmitted by the video acquisition module, perform target detection on the video data using a deep learning algorithm, identify the human objects appearing in the video data, and extract the attribute information of the human objects; and store the attribute vectorization results obtained after data vectorization of the attribute information corresponding to each of the human objects in a vector library;
[0086] The data storage and management module is connected to the semantic similarity calculation module and the human-computer interaction and result display module respectively, and the data storage and management module includes the vector library, and the data storage and management module is used to store the video data and the attribute vectorization result corresponding to the video data;
[0087] The large language model processing module is connected to the human-computer interaction and result display module, and the large language model processing module is used to receive an input query instruction; perform semantic analysis on the query instruction to convert it into formatted data for retrieval; and perform data vectorization on the formatted data corresponding to the query instruction;
[0088] The large visual model processing module is connected to the semantic similarity calculation module and the video processing and analysis module respectively, and the semantic similarity calculation module is used to calculate the semantic similarity between the data vectorization result corresponding to the query instruction and the attribute vectorization result corresponding to the person object stored in the vector library; according to the similarity sorting result, the target video clip matching the query instruction is screened out;
[0089] The human-computer interaction and result display module includes a user interaction display interface, which is used to display the target video clip on the user interaction display interface; and respond to user operations on the user interaction display interface, and perform video operation actions corresponding to the user operations; wherein the user operations include zooming in / out images, playing / pausing videos, and exporting data.
[0090] The related attributes include timestamp, location, gender, age and clothing color, and the formatted data includes time, location and clothing features.
[0091] In the context of the big model and the security scenario, the big visual model (LVM) and the big language model (LLM) are used to perform semantic transformation on the collected videos or pictures and user information (Query) to obtain the attributes of the relevant information. Among the attributes obtained, time and location are accurate information to facilitate subsequent queries in the original videos and pictures. Human attributes and dangerous event-related attributes are fuzzy information to facilitate subsequent generalized matching. After obtaining the attributes of the relevant information, these attributes are vectorized and stored in the vector database. Finally, the semantic similarity of the two is calculated to match the information in the video or picture that meets the user's query requirements, so as to map back to the video or picture to find the original detailed information that the user needs to query.
[0092] The video acquisition module is used to continuously capture at least three instantaneous images from the video data acquired in real time or periodically with a preset fixed time interval t as a step length, wherein the instantaneous image includes b The initial image at time t m The intermediate image at time t e Node image at the moment;
[0093] The video acquisition module is used to process the instantaneous image to obtain a pixel mean matrix, and extract the attribute information of the person object based on the pixel mean matrix.
[0094] Specifically: the video acquisition module is used to process the instantaneous image to obtain a pixel mean matrix, and extract the attribute information of the person object according to the pixel mean matrix, including:
[0095] The instantaneous image is evenly divided into cell regions with a step length of ε, and the cell regions are numbered in sequence, namely A1, A2, A3, ..., A n ;
[0096] Obtain A1-A obtained after processing each image in the instantaneous image n The pixel mean value P of the pixels in the cell area bn , P mn , P en , and according to the pixel mean P of the pixels in the cell area bn , P mn , P en Generate a pixel mean matrix, where P bn , P mn , P en The initial image, intermediate image, and node image correspond to A1-A respectively. n The pixel mean of the cell area;
[0097] According to the pixel mean matrix, a first pixel mean difference matrix ΔP between the initial image and the intermediate image is calculated, and a second pixel mean difference matrix ΔP' between the intermediate image and the node image is calculated; wherein the step length ε of the cell area is not greater than 1 / 100 of the instantaneous image size;
[0098] A motion trajectory of the human object is generated according to the first pixel mean difference matrix ΔP and the second pixel mean difference matrix ΔP′, and the attribute information includes the motion trajectory.
[0099] The instantaneous image is processed, the instantaneous image is evenly divided into cell areas with a step size of ε, and the cell areas are numbered in sequence, wherein, when the image processing module processes the instantaneous image, the step size of evenly dividing the instantaneous image into cell areas is ε, which is determined by the size of the instantaneous image, and the step size ε is not greater than 1 / 100 of the size of the instantaneous image. In this embodiment, the instantaneous image target is the video stream of the camera to extract the picture frame.
[0100] The camera records data for 24 hours a day, and the data for the whole day is extracted from the data of a certain time (such as someone passing by, a fire breaking out). In addition, if the data is extracted frame by frame or frame by frame, the storage demand for data is also very large. It is necessary to first detect events in the video (target detection, whether someone passes by or a fire breaks out), and then track the event (target tracking, where a person appears and disappears or where a fire breaks out). In the process of event tracking, the corresponding image frames are saved according to certain conditions (such as someone appears, a person is in the middle of the picture or a fire breaks out, frame by frame extraction).
[0101] Specifically in this embodiment: the video processing and analysis module is further connected to a privacy protection module, and the privacy protection module is used to protect the privacy of relevant attributes of characters and desensitize sensitive information when processing and analyzing video data, specifically including:
[0102] The privacy protection module is used to pre-process the video data, wherein the pre-processing includes frame extraction and key frame identification;
[0103] Detecting sensitive information in the video data using facial recognition and optical character recognition;
[0104] According to the type of the identified sensitive information, the detected face or sensitive information is blurred using Gaussian blur to obtain a desensitized video;
[0105] Post-process the desensitized video.
[0106] Privacy protection:
[0107] Strictly comply with privacy protection regulations to ensure the security and compliance of user data.
[0108] Use encryption technology and access control policies to protect sensitive data from unauthorized access.
[0109] Ensure compliance with relevant privacy protection regulations and standards when processing and analyzing video data.
[0110] Desensitize sensitive information to prevent data leakage.
[0111] Provide user privacy setting options to allow users to control their own data access and usage permissions.
[0112] A natural language interactive security video retrieval system based on a generative large model also includes:
[0113] System monitoring and maintenance module:
[0114] Real-time monitoring of the system's operating status, including the camera's working status, data processing speed, resource utilization, etc.
[0115] Provides system log and error reporting functions to facilitate problem troubleshooting and repair.
[0116] Maintain and upgrade the system regularly to ensure system stability and performance.
[0117] Deep learning techniques include:
[0118] Recurrent Neural Networks (RNNs): Used to process sequence data, especially natural language text.
[0119] Long Short-Term Memory Networks (LSTMs) and Gated Recurrent Units (GRUs): As improved versions of RNNs, they are better able to handle long-term dependencies.
[0120] Transformers and self-attention mechanism: Models such as BERT and GPT are based on the Transformer architecture, which captures dependencies in the input sequence through the self-attention mechanism.
[0121] Pre-trained models include:
[0122] Language model pre-training: such as BERT, RoBERTa, GPT, etc., captures common language patterns through unsupervised learning on large-scale corpora.
[0123] Multi-task pre-training: Some models, such as T5 (Text-to-Text Transfer Transformer), enhance the model's capabilities through multi-task learning.
[0124] Multi-language processing:
[0125] Multilingual pre-trained models: such as mBERT (Multilingual BERT), XLM (Cross-lingualLanguage Model), and XLM-R (XLM-Roberta), which can be pre-trained on multiple languages and support cross-language understanding and conversion.
[0126] Machine translation technology: used to translate text from one language into another, usually based on an encoder-decoder architecture such as Transformer.
[0127] Experimental Example 2
[0128] Search for men wearing white short-sleeved shirts and blue pants who appeared at Shenzhen headquarters at 10 a.m.:
[0129] Before the user's query, the Large Vision Model (LVM) has performed semantic conversion on the video collected by the Shenzhen headquarters, and obtained the relevant attributes of each person appearing in the video (time of appearance, location, gender, age, top color, trouser color, hair length, top length, trouser length, etc.), and vectorized the relevant attributes of each person and stored them in the vector library.
[0130] When the user enters the query information (query for men wearing white short-sleeved shirts and blue pants who appeared at the Shenzhen headquarters building at 10 a.m.), the large language model (LLM) performs semantic conversion on the query instruction. For example, the conversion result is: "10:00, Shenzhen headquarters, men wearing white short-sleeved shirts and blue pants". Then the conversion result is vectorized and saved in the vector library. By calculating the semantic similarity between the vectorized result of the user's query information and the vectorized result of the video extraction attribute extraction, the original information with similar semantics is found from the vectorized result of the video extraction attribute extraction. In this way, the man wearing white short-sleeved shirts and blue pants who appeared at the Shenzhen headquarters at 10 a.m. can be found from the original video, thereby achieving the user's query goal.
[0131] For example, a user:
[0132] Input: An adult man wearing a white short-sleeved shirt and red shorts at around 10am in Shenzhen;
[0133] The result obtained by LLM semantic transformation is:
[0134] [Shenzhen, 10:00, Male, Adult, White, Short-sleeve, Red, Shorts, N / A, N / A, N / A, N / A];
[0135] [Shenzhen, 10:00, the picture shows a man wearing a white short-sleeved shirt and red shorts];
[0136] After vectorization, calculate the similarity and arrange it from high to low:
[0137] [Shenzhen, 11:11, (male, young, yellow top, black bottom, long hair, hat, no glasses, short sleeves, shorts, no shoes, no bag, no backpack, has a handbag, no shoulder bag)];
[0138] [Shenzhen, 10:10, (male, young, white top, black bottom, long hair, hat, glasses, long sleeves, shorts, no shoes, with a tote bag, no backpack, with a handbag, no shoulder bag)];
[0139] [Shenzhen, 08:08, (male, young, white top, black bottom, short hair, hat, no glasses, short sleeves, long pants, no shoes, no bag, no backpack, has a handbag, no shoulder bag)];
[0140] [Shenzhen, 09:09, (male, young, white top, black bottom, short hair, hat, glasses, long sleeves, shorts, no shoes, no bag, no backpack, handbag, no shoulder bag)];
[0141] [Shenzhen, 12:12, (male, elderly, white, unknown, short hair, hat, glasses, long sleeves, long pants, no shoes, no bag, no backpack, handbag, no shoulder bag)];
[0142] Specifically in this embodiment: the video processing and analysis module is a large visual model LVM, and the large language model processing module is a large language model LLM.
[0143] Data collection and preprocessing:
[0144] Collect security-related video and image data.
[0145] Preprocess user queries (Query), such as text cleaning, word segmentation, part-of-speech tagging, etc.
[0146] Applications of Large Visual Model (LVM):
[0147] Use LVM to extract features from videos and pictures to obtain attributes related to visual content, such as time, location, and human attributes (such as clothing color, behavioral characteristics, etc.).
[0148] Precise information (such as time and location) is accurately identified through LVM's specific modules or algorithms.
[0149] Fuzzy information (such as human attributes and attributes related to dangerous events) is extracted through the generalization ability of LVM, and these attributes may contain certain uncertainty or ambiguity.
[0150] Applications of Large Language Model (LLM):
[0151] LLM is used to semantically understand user queries and convert them into representations that match the visual attributes extracted by LVM.
[0152] LLM's capabilities can process complex natural language queries and translate them into a format that can be compared to LVM attributes.
[0153] Data vectorization:
[0154] The attributes extracted by LVM and LLM are vectorized for storage and retrieval in a vector database.
[0155] Vectorization methods can be based on embedding vectors (such as Word2Vec, BERT, etc.) or feature-based (such as SIFT, HOG, etc.).
[0156] Semantic similarity calculation:
[0157] In the vector database, the semantic similarity between the vector representation of the user query (Query) and the vector representation of the video / image attribute is calculated.
[0158] The similarity calculation method can be cosine similarity, Euclidean distance, etc.
[0159] Matching and result return:
[0160] The results are sorted according to semantic similarity, and the video or picture information that best matches the user's query is returned.
[0161] If necessary, detailed frames or areas in the original video or picture can be further returned to meet the user's precise query needs.
[0162] Feedback and Optimization:
[0163] Based on user feedback and actual application results, the large visual model (LVM) and large language model (LLM) are optimized and adjusted to improve matching accuracy and efficiency.
[0164] Through the collaborative work of the above modules, the system can realize real-time monitoring and intelligent analysis of the Shenzhen headquarters video data, helping users to quickly locate and retrieve target persons, thereby improving the efficiency and accuracy of security work.
[0165] Embodiment 3
[0166] On the basis of the first embodiment, the invention further includes:
[0167] The video processing and analysis module is a large visual model LVM, and the large visual model LVM uses a Transformer-based computer vision model to perform target detection and feature extraction on the video data to obtain the attribute vectorization result;
[0168] The large language model processing module is a large language model LLM, and the large language model LLM uses a Transformer-based natural language processing model to encode the query instruction to obtain a data vectorization result of a text feature vector type.
[0169] Specifically: The Transformer includes a self-attention mechanism, a multi-head attention mechanism, and an encoder-decoder structure.
[0170] The self-attention mechanism is used to focus on the specific location of the person object in the video data with the video frame as the spatial range; wherein the video frame includes a first video frame, a second video frame, a third video frame and a fourth video frame;
[0171] The first video frame is a video of the person's head; the second video frame is a video of the person's torso; the third video frame is a video of the person's arms;
[0172] The third video frame is a video of the person's legs;
[0173] Spatial attention: The model is able to focus on the specific location of the character within the spatial range of the video frame. This is achieved by generating a spatial attention weight map, where the highlighted area corresponds to the location of the character.
[0174] Temporal attention: In addition to spatial position, the model can also pay attention to which frames in the video sequence contain key information. Temporal attention weights can help the model select important frames that contain character behaviors or event changes in consecutive video frames;
[0175] Model-based attention mechanism: In Transformer, attention recognition of characters is achieved by adding additional attention modules. These modules can dynamically generate spatial and temporal attention weights based on the characteristics of the input data.
[0176] Training process: Through a large amount of annotated surveillance video data, we learn how to generate effective attention weights. The annotated data usually includes the location information of the person (such as the bounding box) and the category information of the behavior;
[0177] The multi-head attention mechanism is used to execute multiple attention heads in parallel to perform character object recognition on the first video frame, the second video frame, the third video frame and the fourth video frame respectively according to the specific location of the character object in the video frame, so as to obtain a target detection result.
[0178] The encoder-decoder structure is used to extract and encode the features of the target detection results corresponding to the video frames, and input the encoded signals into the decoder for character recognition to obtain the attribute vectorization results. First, the video frames are extracted from the surveillance video, and necessary preprocessing is performed, such as scaling, normalization, etc., to facilitate subsequent feature extraction.
[0179] Feature extraction and encoding: Use the encoder to extract features and encode the preprocessed video frames. The encoder uses deep learning models such as Transformer to automatically learn and extract features that are useful for person recognition.
[0180] Coded signal processing: After obtaining the coded signal, it can be further processed, such as dimensionality reduction, feature selection, etc., to remove redundant information and noise and improve recognition accuracy.
[0181] Decoding and recognition: The processed encoded signal is input into the decoder, which decodes it into the original input signal or an approximate representation and performs character recognition.
[0182] The basic principle of the large model is based on the Transformer, which is a neural network architecture based on the self-attention mechanism. The Transformer model was originally proposed for machine translation tasks. The basic components of the Transformer include the self-attention mechanism and some feed-forward neural networks.
[0183] Its main components are introduced as follows:
[0184] (1) Self-Attention Mechanism: It is the core part of the Transformer model. It allows the model to focus on different positions in the input sequence at the same time when processing the input sequence and assign different weights to each position.
[0185] In the self-attention mechanism, the input sequence undergoes three linear transformations (usually called query, key, and value transformations) to obtain query (Q), key (K), and value (V) vectors. Then, the attention weights are obtained by calculating the similarity between the query vector and the key vector. Finally, the value vector is weighted and summed according to these weights to obtain the final self-attention representation. Through this mechanism, the Transformer can capture the contextual information in the sequence when processing the input sequence, rather than relying solely on a fixed-size local window.
[0186] (2) Multi-head Attention: It is an improved mechanism in Transformer. It can improve the expressiveness and representation diversity of the model by performing multiple parallel calculations on different queries, keys, and value transformations. In multi-head attention, multiple sets of query (Q), key (K), and value (V) vectors are obtained by linearly transforming the query, key, and value transformations. Then, the attention weights are calculated for each set of queries and keys, and the weights are applied to the corresponding value vectors. Finally, the outputs of each head are concatenated and linearly transformed. The multi-head attention mechanism enables the model to focus on different representation subspaces at the same time, thereby better capturing semantic information at different levels.
[0187] (3) Encoder-Decoder Architecture: The Transformer model usually consists of two parts: an encoder and a decoder. The encoder is responsible for encoding the input sequence into a continuous representation, while the decoder gradually generates the target sequence based on the encoder's output. The encoder is composed of multiple identical layers stacked together, each of which contains a self-attention mechanism and a feedforward neural network. In the encoder, the input sequence passes through the self-attention mechanism and the feedforward neural network layer by layer, and finally obtains the encoder's output representation. The decoder is also composed of multiple identical layers stacked together. In addition to the self-attention mechanism and the feedforward neural network, each layer also includes an additional attention mechanism for paying attention to the encoder's output. In the decoder, the target sequence passes through the self-attention mechanism, the encoder-decoder attention mechanism, and the feedforward neural network layer by layer, and finally generates the decoder's output sequence.
[0188] The training phase of the Transformer model consists of the encoder and decoder working together. The source sequence is encoded by the encoder to generate a feature representation, which is then passed to the decoder. The input sequence to the decoder is the target sequence shifted to the left by one position, which allows the decoder to focus on the generated part and the encoded representation of the source sequence through the attention mechanism. The goal of the decoder is to predict the next token in the target sequence. At each time step, the decoder generates a probability distribution representing the likelihood of the next token, obtained by linearly transforming and softmaxing the output of the last layer of the decoder. During training, the model uses the real tokens in the target sequence as input to the decoder and calculates the cross entropy loss between the generated probability distribution and the next token in the target sequence. Through backpropagation and optimization algorithms, the model adjusts the parameters to minimize the loss. In the inference phase, the model takes a special start symbol as the initial input and generates the next token step by step through the encoder and decoder. The decoder calculates the attention weights based on the current generated part of the sequence and the encoded representation of the source sequence and applies them to the encoded representation of the source sequence. The decoder then processes the attention-weighted encoded representation through a feedforward neural network and generates a probability distribution for the next token. The inference process usually uses a greedy search method to select the token with the highest probability as the next token to be generated. This process continues until a special termination symbol is generated or a predetermined maximum sequence length is reached. In this way, the Transformer model is able to predict and generate sequence data.
[0189] The video processing and analysis module is also used to continuously collect the video data within 24 hours, use a deep learning algorithm to perform real-time analysis on the video data, identify expected dangerous events, and the dangerous events include fire events and smoke events. When a dangerous event is detected, the time, location, and relevant personal information of the event are automatically recorded, and the relevant attributes of the dangerous event are vectorized and stored in a vector library;
[0190] The large language model processing module is also used to receive a query instruction input by a user; perform semantic analysis on the query instruction to extract key information, wherein the key information includes time, location and event type;
[0191] Vectorize the key information of the query instruction to compare it with the data in the vector library;
[0192] The semantic similarity calculation module is also used to calculate the semantic similarity between the vectorized result of the user query and the vectorized result of the dangerous event attribute stored in the vector library; according to the similarity sorting result, the dangerous event record that best matches the user query is screened out;
[0193] The human-computer interaction and result display module is also used to display the screened dangerous event records to the user in a visual form, providing a user interaction interface, allowing the user to further view the detailed information of the event, the detailed information including the complete video of the event and the identity information of the relevant personnel.
[0194] Specifically: the natural language interactive security video retrieval system based on the generative big model also includes an early warning and notification module, which is used to automatically trigger an early warning mechanism when the dangerous event is detected, allowing users to customize early warning rules;
[0195] The large language model processing module is also used to receive a query instruction input by a user, such as "query the cause of a fire in a certain entertainment venue last night"; perform semantic analysis on the query instruction to extract key information, wherein the key information includes time, location and event type;
[0196] Vectorize the key information of the query instruction to compare it with the data in the vector library;
[0197] The semantic similarity calculation module is also used to calculate the semantic similarity between the vectorized results of the user query and the vectorized results of the dangerous event attributes stored in the vector library; according to the similarity sorting results, the dangerous event records that best match the user query are screened out.
[0198] Experimental Example 4
[0199] Query the cause of the fire in a certain entertainment venue yesterday evening:
[0200] Before the user's query, the large visual model (LVM) has already performed event identification and semantic conversion on the videos collected from the entertainment venues, obtained the relevant attributes of each dangerous event (such as fire) (occurrence time, location, related people, possible causes, etc.), and vectorized the relevant attributes of each dangerous event and stored them in the vector library. When the user enters the query information (query the cause of the fire in a certain entertainment venue last night), the large language model (LLM) performs semantic conversion on the query instruction, for example, the conversion result is: "Last night, a certain entertainment venue, fire", and then the conversion result is vectorized and saved in the vector library. By calculating the semantic similarity between the vectorized result of the user's query information and the vectorized result of the video extraction attribute extraction, the original information with similar semantics is found from the vectorized result of the video extraction attribute extraction, so that the relevant information of the fire in a certain entertainment venue last night can be found from the original video, thereby achieving the user's query goal.
[0201] Embodiment 5
[0202] On the basis of the third embodiment, it further includes: an NB classifier, which calculates the posterior probability that the sample belongs to a certain category through the prior probability of some features of the sample to be classified, so as to predict the category label of the sample.
[0203] Assume that the dangerous event set samples have m categories, namely C1, C2, ..., C m , the extracted n data feature attributes are X1, X2, ..., X n Given a data sample X of unknown category, if and only if When , the classifier will predict that X belongs to the class C with the highest posterior probability i (1≤i≤m). It can be seen that:
[0204] ;
[0205] Among them, the class prior probability P(C i ) expresses the proportion of each type of sample in the dangerous event data sample. According to the law of large numbers, P(C i ) can be estimated by the frequency of occurrence of each type of sample. i ), since it involves the joint probability of all attributes of X, estimating it directly based on the frequency of sample occurrence will encounter serious difficulties.
[0206] To solve this problem, the NB classifier adopts the attribute conditional independence assumption, that is, for known categories, it is assumed that all attributes are independent of each other; in other words, it is assumed that each attribute independently affects the classification result.
[0207] Embodiment 6
[0208] The above-mentioned natural language interactive security video retrieval device based on the generative large model can be implemented in the form of a computer program. The computer program can be used in Figure 3 Runs on the computer device shown.
[0209] See also Figure 3 , Figure 3 The present invention is a schematic block diagram of a computer device provided by an embodiment of the present invention. The computer device integrates any one of the natural language interactive security video retrieval systems based on a generative large model provided by an embodiment of the present invention.
[0210] See also Figure 3 The computer device includes a processor 402 , a memory and a network interface 405 connected via a system bus 401 , wherein the memory may include a storage medium 403 and an internal memory 404 .
[0211] The storage medium 403 can store an operating system 4031 and a computer program 4032. The computer program 4032 includes program instructions, and when the program instructions are executed, the processor 402 can execute the above-mentioned natural language interactive security video retrieval system based on a generative large model.
[0212] The processor 402 is used to provide computing and control capabilities to support the operation of the entire computer device.
[0213] The internal memory 404 provides an environment for the operation of the computer program 4032 in the storage medium 403. When the computer program 4032 is executed by the processor 402, the processor 402 can execute the above-mentioned natural language interactive security video retrieval system based on the generative large model.
[0214] The network interface 405 is used to communicate with other devices over the network. Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0215] The processor 402 is used to run a computer program 4032 stored in the memory to implement a natural language interactive security video retrieval system based on a generative large model as described above.
[0216] It should be understood that in the embodiment of the present invention, the processor 402 may be a central processing unit (CPU), and the processor 402 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0217] It can be understood by those skilled in the art that all or part of the processes in the system implementing the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above-mentioned system.
[0218] Therefore, the present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes a natural language interactive security video retrieval system based on a generative large model as described above.
[0219] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.
[0220] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different systems to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0221] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and systems can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0222] The steps in the system of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0223] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to execute all or part of the steps of the system described in each embodiment of the present invention.
[0224] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A natural language interactive security video retrieval system based on a generative large model, characterized in that: include: Video acquisition module, video processing and analysis module, large language model processing module, large visual model processing module, semantic similarity calculation module, human-computer interaction and result display module, data storage and management module; The video acquisition module is used to collect video data in real time or periodically; the video acquisition module includes a high-definition camera installed inside a building, inside an entertainment venue, and within a preset distance range, and the video acquisition module is connected to the video processing and analysis module and the data storage and management module respectively; The video processing and analysis module is used to receive the video data transmitted by the video acquisition module, perform target detection on the video data using a deep learning algorithm, identify the human objects appearing in the video data, and extract the attribute information of the human objects; and store the attribute vectorization results obtained after data vectorization of the attribute information corresponding to each of the human objects in a vector library; The data storage and management module is connected to the semantic similarity calculation module and the human-computer interaction and result display module respectively, and the data storage and management module includes the vector library, and the data storage and management module is used to store the video data and the attribute vectorization result corresponding to the video data; The large language model processing module is connected to the human-computer interaction and result display module, and the large language model processing module is used to receive an input query instruction; perform semantic analysis on the query instruction to convert it into formatted data for retrieval; and perform data vectorization on the formatted data corresponding to the query instruction; The large visual model processing module is connected to the semantic similarity calculation module and the video processing and analysis module respectively, and the semantic similarity calculation module is used to calculate the semantic similarity between the data vectorization result corresponding to the query instruction and the attribute vectorization result corresponding to the person object stored in the vector library; according to the similarity sorting result, the target video clip matching the query instruction is screened out; The human-computer interaction and result display module includes a user interaction display interface, which is used to display the target video clip on the user interaction display interface; And in response to the user operation on the user interaction display interface, perform the video operation action corresponding to the user operation; wherein the user operation includes zooming in / out the image, playing / pausing the video and exporting the data; The video processing and analysis module is a large visual model LVM, and the large visual model LVM uses a Transformer-based computer vision model to perform target detection and feature extraction on the video data to obtain the attribute vectorization result; The large language model processing module is a large language model LLM, and the large language model LLM uses a natural language processing model based on Transformer to encode the query instruction to obtain a data vectorization result of a text feature vector type; the Transformer includes a self-attention mechanism, a multi-head attention mechanism and an encoder-decoder structure; The self-attention mechanism is used to focus on the specific location of the person object in the video data with the video frame as the spatial range; wherein the video frame includes a first video frame, a second video frame, a third video frame and a fourth video frame; The multi-head attention mechanism is used to execute multiple attention heads in parallel to perform person object recognition on the first video frame, the second video frame, the third video frame, and the fourth video frame respectively according to the specific position of the person object in the video frame, so as to obtain a target detection result; The encoder-decoder structure is used to extract and encode the target detection result corresponding to the video frame, and input the encoded signal into the decoder for person recognition to obtain the attribute vectorization result; The video processing and analysis module is also used to continuously collect the video data within 24 hours, use a deep learning algorithm to perform real-time analysis on the video data, identify expected dangerous events, and the dangerous events include fire events and smoke events. When a dangerous event is detected, the time, location, and relevant personal information of the event are automatically recorded, and the relevant attributes of the dangerous event are vectorized and stored in a vector library; The large language model processing module is also used to receive a query instruction input by a user; perform semantic analysis on the query instruction to extract key information, wherein the key information includes time, location and event type; Vectorize the key information of the query instruction to compare it with the data in the vector library; The semantic similarity calculation module is also used to calculate the semantic similarity between the vectorized result of the user query and the vectorized result of the dangerous event attribute stored in the vector library; according to the similarity sorting result, the dangerous event record that best matches the user query is screened out; The human-computer interaction and result display module is also used to display the screened dangerous event records to the user in a visual form, providing a user interaction interface, allowing the user to further view the detailed information of the event, the detailed information includes the complete video of the event and the identity information of the relevant personnel, the natural language interactive security video retrieval system based on the generative large model also includes an early warning and notification module, the early warning and notification module is used to automatically trigger the early warning mechanism when the dangerous event is detected, allowing the user to customize the early warning rules, the video processing and analysis module is also connected to a privacy protection module, the privacy protection module is used to protect the privacy of the relevant attributes of the person when processing and analyzing the video data, and desensitize the sensitive information, specifically including; The privacy protection module is used to pre-process the video data, wherein the pre-processing includes frame extraction and key frame identification; Detecting sensitive information in the video data using facial recognition and optical character recognition; According to the type of the identified sensitive information, the detected face or sensitive information is blurred using Gaussian blur to obtain a desensitized video; Post-process the desensitized video.
2. The natural language interactive security video retrieval system based on a generative large model according to claim 1 is characterized in that: The video acquisition module is used to continuously capture at least three instantaneous images from the video data acquired in real time or periodically with a preset fixed time interval t as a step length, wherein the instantaneous image includes b The initial image at time t m The intermediate image at time t e Node image at the moment; The video acquisition module is used to process the instantaneous image to obtain a pixel mean matrix, and extract the attribute information of the person object based on the pixel mean matrix.
3. The natural language interactive security video retrieval system based on generative large model according to claim 2 is characterized by: The video acquisition module is used to process the instantaneous image to obtain a pixel mean matrix, and extract the attribute information of the person object according to the pixel mean matrix, including: The instantaneous image is evenly divided into cell regions with a step length of ε, and the cell regions are numbered in sequence, namely A1, A2, A3, ..., A n ; Obtain A1-A obtained after processing each image in the instantaneous image n The pixel mean value P of the pixels in the cell area bn , P mn , P en , and according to the pixel mean P of the pixels in the cell area bn , P mn , P en Generate a pixel mean matrix, where P bn , P mn , P en The initial image, intermediate image, and node image correspond to A1-A respectively. n The pixel mean of the cell area; According to the pixel mean matrix, a first pixel mean difference matrix ΔP between the initial image and the intermediate image is calculated, and a second pixel mean difference matrix ΔP' between the intermediate image and the node image is calculated; wherein the step length ε of the cell area is not greater than 1 / 100 of the instantaneous image size; A motion trajectory of the human object is generated according to the first pixel mean difference matrix ΔP and the second pixel mean difference matrix ΔP′, and the attribute information includes the motion trajectory.
4. The natural language interactive security video retrieval system based on a generative large model according to claim 1, characterized in that: The related attributes include timestamp, location, gender, age and clothing color, and the formatted data includes time, location and clothing features.
5. A natural language interactive security video retrieval device based on a generative large model, characterized in that: It also includes a memory and one or more processors, wherein the memory stores a computer program, and when the processor executes the computer program, it is used to implement the natural language interactive security video retrieval system based on the generative large model as described in any one of claims 1-4.
Citation Information
Patent Citations
Operation method of intelligent security monitoring system
CN108052882A
Video abstract algorithm and system based on gated multi-head position attention mechanism
CN115002559A
Event detection method and device, equipment and storage medium
CN116824455A
Image search method and device based on large model, equipment and medium
CN117493595A
Community intelligent security system based on machine vision and implementation method thereof
CN117523443A
