Urban monitoring video retrieval method, device and equipment and medium

Abstract

The invention discloses a city monitoring video retrieval method and device, equipment and a medium, and relates to the technical field of information retrieval. The method comprises the following steps: acquiring a to-be-processed video stream, performing frame extraction on a target to-be-processed video stream to obtain a video frame image sequence, and extracting a semantic feature vector; calculating a first feature similarity between the semantic feature vector of the current video frame image and the semantic feature vector of the previous video frame image; the video frame images with the first feature similarity smaller than a preset similarity threshold value are determined as effective video frame images, and semantic feature vectors of the video frame images serve as effective semantic feature vectors and are stored in a vector database; obtaining a retrieval request of a user, and extracting a retrieval semantic vector; calculating a second feature similarity between the retrieval semantic vector and a plurality of target effective semantic feature vectors in a vector database; and selecting the first preset number of effective video frame images with the second feature similarity from high to low as a retrieval result. According to the method and the device, the city monitoring video retrieval efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information retrieval, and in particular to a city monitoring video retrieval method, device, equipment and medium. BACKGROUND

[0002] With the continuous advancement of smart city construction, city monitoring cameras have become a key infrastructure for obtaining city operation status and maintaining public safety. These devices generate massive video data. How to quickly and accurately retrieve specific video content from these unstructured data has become a technical challenge that needs to be addressed. Existing video retrieval technologies generally have long data processing links, high storage costs, and slow retrieval response when dealing with smart city-level massive concurrent video streams and performing efficient retrieval. This seriously restricts the application value of video data in emergency command, case investigation and other scenarios. In the face of city-level massive concurrent video streams, video frame processing, feature extraction and data storage are mostly processed in a serial mode. The delay from data collection to retrieval is high, which cannot meet the real-time requirements. At the same time, similar or static continuous frames in the video are not effectively filtered, resulting in a large number of redundant feature vectors being repeatedly stored. This not only greatly wastes storage resources, but also significantly increases the computational burden during subsequent retrieval. With the rapid growth of data size, the retrieval response time is prolonged with the increase of data volume, which reduces the efficiency of city monitoring video retrieval. SUMMARY

[0003] The embodiments of the present application provide a city monitoring video retrieval method, device, equipment and medium, which are used to solve the technical problem of how to improve the efficiency of city monitoring video retrieval.

[0004] In a first aspect, the embodiments of the present application provide a city monitoring video retrieval method, comprising: obtaining a plurality of channels of to-be-processed video streams, and frame extracting a target to-be-processed video stream through an independent video processing thread to obtain a video frame image sequence, the target to-be-processed video stream being any channel of the to-be-processed video stream, one target to-be-processed video stream corresponding to one independent video processing thread, and the video frame image sequence comprising a plurality of video frame images; extracting semantic feature vectors corresponding to each video frame image in time sequence through a preset cross-modal learning model; calculating a first feature similarity between a semantic feature vector of a current video frame image and a semantic feature vector of a previous video frame image, the current video frame image being any video frame image; determining the current video frame image with a first feature similarity less than a preset similarity threshold as an effective video frame image, and taking the semantic feature vector of the effective video frame image as an effective semantic feature vector; store all the valid vector data to the vector database, the valid vector data at least including a creation timestamp corresponding to the valid semantic feature vector and the valid video frame image; obtain a search request of a user, and extract a search semantic vector corresponding to the search request through the preset cross-modal learning model; calculate a second feature similarity between the search semantic vector and a plurality of target valid semantic feature vectors respectively corresponding to the second feature similarity in the vector database, the target valid semantic feature vector being a valid semantic feature vector selected based on a constraint condition of the search request; select a first preset number of valid video frame images corresponding to the second feature similarity from high to low as a search result.

[0005] Optionally, before the step of extracting the semantic feature vector corresponding to each video frame image in time sequence through the preset cross-modal learning model, the method further includes discarding all abnormal video frame images, the abnormal video frame image being used to represent a video frame image with abnormal quality, and the identification method of the abnormal video frame image including at least one of the following: calculating a Laplacian operator variance corresponding to each video frame image through a preset Laplacian variance algorithm, and taking the video frame image with the Laplacian operator variance less than a preset variance threshold as the abnormal video frame image; calculating an edge pixel ratio corresponding to each video frame image through a preset edge detection algorithm, and taking the video frame image with the edge pixel ratio less than a preset ratio threshold as the abnormal video frame image; calculating a gray mean value corresponding to each video frame image, and taking the video frame image with the gray mean value less than a preset minimum gray threshold or greater than a preset maximum gray threshold as the abnormal video frame image.

[0006] Optionally, the method further includes: extracting the semantic feature vector corresponding to each video frame image in time sequence in batches through the preset cross-modal learning model, a second preset number of semantic feature vectors corresponding to the video frame image being extracted in each batch; calculating a third feature similarity set of the semantic feature vector of a target video frame image and the semantic feature vectors of all video frame images in a previous batch, the target video frame image being any video frame image in a current batch, and the third feature similarity set including a second preset number of third feature similarities; determining the target video frame image with all the third feature similarities in the third feature similarity set being less than the preset similarity threshold as the valid video frame image.

[0007] Optionally, the retrieval request comprises a text retrieval request and an image retrieval request, and the preset cross-modal learning model comprises a text-to-vector interface and an image-to-vector interface; the method of extracting the retrieval semantic vector of the retrieval request by using the preset cross-modal learning model comprises: when the retrieval request is the text retrieval request, generating the retrieval semantic vector of the text retrieval request by using the text-to-vector interface; and when the retrieval request is the image retrieval request, generating the retrieval semantic vector of the image retrieval request by using the image-to-vector interface.

[0008] Optionally, before the calculating of the second feature similarity between the retrieval semantic vector and each of the target effective semantic feature vectors in the vector database, the method further comprises: analyzing, by using the preset cross-modal learning model, a time range requirement and a return result number of the retrieval request, and taking the return result number as the first preset number; determining a storage partition corresponding to the retrieval request based on the time range requirement, the storage partition being used to represent a partition address of the vector database for storing data in a time sequence; sending the retrieval semantic vector, the storage partition and the first preset number to a query interface of the vector database; and taking the effective semantic feature vectors stored in the storage partition as the target effective semantic feature vectors.

[0009] Optionally, the vector database comprises a hot data partition and a regular partition, and the method further comprises: based on a creation timestamp corresponding to each of the effective video frame images, storing effective video frame data with a creation duration less than a first preset duration to the hot data partition, the hot data partition being configured in a resident memory state; storing effective video frame data with the creation duration greater than or equal to the first preset duration and less than or equal to a second preset duration to the regular partition; deleting effective video frame data with the creation duration greater than the second preset duration, and deleting a storage partition corresponding to the effective video frame data with the creation duration greater than the second preset duration.

[0010] Optionally, the storing all effective vector data to the vector database comprises: creating a unique identifier corresponding to target effective vector data as a primary key, the target effective vector data being any of the effective vector data; obtaining a video stream identifier of a video stream corresponding to the target effective vector data; generating an access path string of the effective video frame image corresponding to the target effective vector data through a preset structured path; generating a similar search index corresponding to the target effective vector data through a preset clustering algorithm based on the effective semantic feature vector; and storing the primary key, the video stream identifier, the access path string, the creation timestamp, the similar search index, and the target effective vector data corresponding to the target effective vector data to the vector database according to a preset storage partition based on the creation timestamp.

[0011] In a second aspect, an embodiment of the present application provides a city monitoring video retrieval device, comprising: A video frame extraction module is configured to obtain a multi-channel video stream to be processed, and extract frames from a target video stream to be processed through an independent video processing thread to obtain a video frame image sequence, the target video frame being any channel of the video stream to be processed, one target video stream to be processed corresponding to one independent video processing thread, and the video frame image sequence comprising a plurality of video frame images. A feature extraction module is configured to extract semantic feature vectors corresponding to each video frame image in time sequence through a preset cross-modal learning model. A vector deduplication module is configured to calculate a first feature similarity between a semantic feature vector of a current video frame image and a semantic feature vector of a previous video frame image, the current video frame image being any of the video frame images; determine the current video frame image with a first feature similarity less than a preset similarity threshold as an effective video frame image, and take the semantic feature vector of the effective video frame image as an effective semantic feature vector. A vector database module is configured to store all effective vector data to a vector database, the effective vector data at least comprising the effective semantic feature vector and a creation timestamp corresponding to the effective video frame image. A semantic retrieval module is configured to obtain a retrieval request of a user, extract a retrieval semantic vector of the retrieval request through the preset cross-modal learning model, calculate a second feature similarity between the retrieval semantic vector and a plurality of target effective semantic feature vectors respectively corresponding to the vector database, and select a first preset number of effective video frame images corresponding to the second feature similarity from high to low as a retrieval result.

[0012] In a third aspect, an electronic device is provided, which includes a memory, a processor, a user interface, and a network interface. The memory is configured to store instructions. The user interface and the network interface are configured to communicate with other devices. The processor is configured to execute the instructions stored in the memory, so that the electronic device performs the urban monitoring video retrieval method described above.

[0013] In a fourth aspect, a computer readable storage medium is provided, which stores instructions. When the instructions are executed, the urban monitoring video retrieval method provided by the embodiments of the present application is implemented.

[0014] To sum up, the one or more technical solutions provided by the present application have at least the following technical effects or advantages: 1. By adopting the multi-channel independent thread parallel frame extraction and processing mode, the real-time camera input of mass concurrent video streams and the feature extraction task are decoupled and allocated to independent computing units for execution, effectively breaking the data processing bottleneck in the traditional serial processing mode, thereby reducing the overall processing delay of video data from acquisition to retrievability to seconds, meeting the real-time requirement of video retrieval for emergency command, real-time tracking and other applications in urban monitoring scenarios, and improving the efficiency of urban monitoring video retrieval.

[0015] 2. By performing dynamic redundancy filtering and quality abnormal frame screening based on cross-modal semantic features before storage, the system systematically eliminates still pictures with highly repeated content, invalid frames with blurred distortion, and image data with abnormal exposure, so that the final feature vector set stored has higher representativeness and discrimination in the semantic dimension. This not only reduces the invalid storage overhead by more than 60%, but also significantly reduces the noise interference in subsequent retrieval calculation, thereby improving the average retrieval precision by about 30% while ensuring the retrieval recall rate.

[0016] 3. By constructing a multi-level storage system combining time partition management, hot data resident memory optimization, and automatic expiration cleaning mechanism, and combining IVF_FLAT index and inner product similarity calculation, millisecond-level approximate nearest neighbor retrieval of hundreds of billions of feature vectors is realized. At the same time, based on the unified vectorization support of text and image queries by the cross-modal learning model, and the retrieval optimization of on-demand loading of specified time partitions and returning the number of results, the system can support both "image search video" and "text search video" natural interaction modes, and still maintain stable sub-second retrieval response as the data size continues to grow, significantly improving the intelligent retrieval efficiency and practicality of the city-level video monitoring system. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flowchart of the urban monitoring video retrieval method provided by the embodiments of the present application; Figure 2 is a structural schematic diagram of a city monitoring video retrieval device provided by an embodiment of the present application; Figure 3 is a structural schematic diagram of an electronic device provided by an embodiment of the present application.

[0018] Legend: 600, city monitoring video retrieval device; 601, video frame extraction module; 602, feature extraction module; 603, vector deduplication module; 604, vector database module; 605, semantic retrieval module; 606, anomaly detection module; 607, data management module; 700, electronic device; 701, processor; 702, communication bus; 703, user interface; 704, network interface; 705, memory. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without making any creative labor fall within the scope of protection of the present application.

[0020] In the description of the embodiments of the present application, the words “exemplary”, “for example”, or “for instance” are used to represent an example, illustration, or description. Any embodiment or design solution described as “exemplary”, “for example”, or “for instance” in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design solutions. Rather, the words “exemplary”, “for example”, or “for instance” are intended to present the relevant concept in a specific manner.

[0021] In the description of the embodiments of the present application, the term “first\second\third” only distinguishes similar objects, and does not represent a specific order of the objects. Understandably, “first\second\third” can be interchanged with a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0022] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit. The terms "include", "contain", "have" and their variants mean "including but not limited to", unless otherwise specifically emphasized otherwise.

[0023] In the embodiments of the present application, the relevant data collection process should be strictly in accordance with the requirements of the relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and within the scope of authorization of the laws and regulations and the personal information subject, carry out subsequent data use and processing behavior.

[0024] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by a person skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0025] In the related art, when processing massive concurrent video streams at the level of smart city and performing efficient retrieval, there are generally long data processing links, high storage costs and slow retrieval response, and the retrieval response time is prolonged with the increase of data volume, which reduces the efficiency of city monitoring video retrieval. To solve the above problems, the embodiments of the present application provide a city monitoring video retrieval method, device, equipment and medium, which effectively improves the efficiency of city monitoring video retrieval.

[0026] Figure 1 is a flowchart of a city monitoring video retrieval method provided by the embodiments of the present application.

[0027] The embodiments of the present application disclose a city monitoring video retrieval method, as shown in Figure 1 The method comprises the following steps.

[0028] S101, acquiring a plurality of channels of video streams to be processed, and performing frame extraction on a target video stream to be processed through an independent video processing thread to obtain a video frame image sequence, the target video stream to be processed being any channel of the video stream to be processed, one target video stream to be processed corresponding to one independent video processing thread, and the video frame image sequence comprising a plurality of video frame images.

[0029] Specifically, first, multiple physically or logically independent video data sources are accessed from the city monitoring network, each source forms a data channel, that is, a multi-channel video stream to be processed, then for each channel video stream as a processing object, that is, a target video stream to be processed, a dedicated, resource-isolated computing thread is dynamically created or allocated from a thread pool, which is an independent video processing thread, each such thread is independently responsible for its corresponding single video stream, decodes and captures frame images according to a preset frame extraction strategy (such as fixed time interval frame extraction or dynamic frame extraction based on scene change, etc., which is not limited here), and stores the images captured in time sequence or temporarily, and finally forms an image set arranged in time sequence, that is, a video frame image sequence, which is the basic data unit for all subsequent image processing and analysis operations.

[0030] Among them, multi-channel is used to represent multiple parallel input video data sources, each channel usually corresponds to an independent monitoring camera or an independent video stream network address. The video stream to be processed is the original continuous video data obtained from the monitoring network or other video sources, which has not been subjected to feature extraction and structured analysis, and its format is usually RTSP, RTMP, etc. Streaming media protocol format or encapsulated video file. Independent video processing thread represents an independent task execution unit in the operating system or running environment, which has a dedicated CPU time slice and memory space, and is used to process a single video stream to ensure that the video stream processing processes do not block each other. The target video stream to be processed is a single video stream instance selected from the multi-channel video stream to be processed, which is about to be or is being operated by the current processing logic. The video frame image sequence is a collection of multiple static images (frames) arranged in time sequence after frame extraction of a single video stream, each image is a snapshot of video content at a certain time.

[0031] Through the above embodiment, the traditional serial processing of video stream is innovated to a parallel processing architecture based on independent threads, so that the system can fully utilize the computing resources of multi-core CPU, realize the synchronous input and preliminary processing of massive concurrent video streams, and improve the throughput of the data acquisition end by several orders of magnitude, effectively solving the bottleneck problem of video data access in city-level monitoring scenarios, and laying a high-efficiency data foundation for subsequent real-time analysis and retrieval.

[0032] S102, extracting semantic feature vectors corresponding to each video frame image in time sequence through a preset cross-modal learning model.

[0033] Specifically, first, a pre-trained deep learning model, i.e., a preset cross-modal learning model, is loaded, which has the ability to map different modal information such as visual images and texts to a unified semantic vector space. Each video frame image is input into the model one by one or in batches according to the time sequence of the images in the video frame image sequence. The deep neural network (e.g., an encoder based on Transformer or CNN) inside the model performs multi-level abstraction and understanding on the input images, and finally generates a fixed-dimension dense numerical vector at the output layer. This vector is the semantic feature vector corresponding to the frame image, which comprehensively represents the high-level semantic information of the objects, scenes, actions and their relationships in the image. This operation is performed for each frame image in the sequence, thereby obtaining a semantic feature vector sequence that strictly corresponds to the original video frame image sequence in order and quantity.

[0034] Among them, the preset cross-modal learning model refers to a deep learning model that is pre-trained on a large amount of image-text data and may be fine-tuned for the monitoring scene. Its typical representatives are CLIP (Contrastive Language-Image Pre-training) model or its variants, which are used to encode images and texts into the same semantic space. Time sequence refers to the order arranged according to the time sequence of video frame images in their original video stream. Video frame image refers to a single static digital image extracted from a video stream, which is the basic unit of video content. Semantic feature vector refers to a fixed-length (e.g., 512-dimensional or 768-dimensional) one-dimensional floating-point number array, whose numerical distribution is determined by the deep learning model, and is used to represent the abstract semantic content contained in the input image in high dimension and densely.

[0035] On the basis of the above embodiment, as an optional embodiment, before step S102 shown in the figure, the city monitoring video retrieval method provided by the embodiment of the application further includes discarding all abnormal video frame images, and the identification method of the abnormal video frame image includes at least one of steps S201-S203, which will be described in detail below. Figure 1

[0036] S201, calculate the Laplacian operator variance corresponding to each video frame image by a preset Laplacian variance algorithm, and take the video frame image with a Laplacian operator variance less than a preset variance threshold as an abnormal video frame image.

[0037] ​Specifically, the original video frame images collected are subjected to quality preliminary screening to filter out invalid images caused by device failure, transmission problems or environmental interference, such as blurring, defocusing, etc. First, a preset Laplacian variance algorithm is applied to each video frame image to be evaluated: the algorithm first converts the original color image into a grayscale image (such as a non-gray image), then performs convolution operation on the grayscale image using a specific Laplacian convolution kernel (for example, a 3x3 kernel with a center value of 8 and surrounding values of -1) to obtain the Laplacian transform result (i.e. second derivative approximation) of the image, which reflects the edge and detail intensity information of each pixel point in the image, and then calculates the variance of all pixel values in the Laplacian transform result image. The variance value quantitatively reflects the overall sharpness and richness of the edges and textures in the image in mathematics, and a higher variance value generally indicates a clearer image with more details. The calculated Laplacian operator variance of each video frame image is compared with a predefined numerical reference, i.e. a preset variance threshold (for example, an empirical value or a threshold value obtained by statistical analysis of a large number of clear samples). If the Laplacian operator variance calculation result of a certain frame image is less than the preset variance threshold, it is determined that the frame image is of low quality due to lack of sufficient clear edges and texture details, and is marked as an abnormal video frame image to be excluded in the subsequent processing process.

[0038] The preset Laplacian variance algorithm refers to a classical digital image processing algorithm for objectively evaluating the blurring degree of an image based on image second-order differentiation (Laplacian operator) and statistical variance calculation. The Laplacian operator variance refers to the variance (Variance) of all pixel values in the output image (i.e. edge / detail intensity map) obtained by applying the Laplacian operator to the input image for convolution processing. This value is a key scalar index for measuring the overall sharpness of an image. The abnormal video frame image is used to represent a video frame that is visually blurred or has severe information loss due to lens defocusing, severe motion blurring, severe compression distortion, signal transmission failure or extreme lighting, etc., and is not suitable for subsequent semantic analysis and retrieval. The preset variance threshold refers to a predefined numerical dividing point for determining whether an image is clear and qualified, and its value is usually determined through experimental analysis or statistical analysis of a set of high-quality reference images.

[0039] S202, calculate the edge pixel ratio of each video frame image by a preset edge detection algorithm, and regard the video frame image with an edge pixel ratio less than a preset ratio threshold as an abnormal video frame image.

[0040] Specifically, for each frame of video frame image to be processed, a preset edge detection algorithm (such as Sobel, Canny or Prewitt operator) is applied for processing, which may convert a color image into a grayscale image, and then a specific gradient operator convolution kernel is used to traverse the image to calculate the gradient intensity of each pixel position in the image in the horizontal and vertical directions. By combining these gradient information, the algorithm can highlight the areas where the grayscale value changes dramatically in the image, i.e. the edge profile of the object. After processing, the algorithm generates a binary image or an edge intensity image, in which the edge pixels are marked (e.g. assigned a value of 255) and the non-edge pixels are suppressed (e.g. assigned a value of 0). Subsequently, the total number of pixels marked as edges in the resulting image is counted and divided by the total number of pixels in the image, thereby calculating the edge pixel ratio of the video frame image. This ratio quantifies the richness of the identifiable profile and texture information in the image. The calculated edge pixel ratio of each video frame image is compared with a pre-set numerical benchmark representing the minimum acceptable edge richness, i.e. the pre-set ratio threshold. If the edge pixel ratio of a certain frame image is less than the pre-set ratio threshold, it is determined that the frame image lacks sufficient structural information and identifiable profile, and is classified as a low-quality image, which is classified as an abnormal video frame image to be excluded in the subsequent feature extraction and analysis process. As another effective method for quality screening of video frame images, it aims to identify invalid frames with extremely low image content contrast and indistinguishable subject profile caused by excessive exposure, underexposure, heavy fog, dust and other environmental influences or sensor failure.

[0041] The preset edge detection algorithm refers to a classical image processing technique for identifying the boundaries of objects in digital images (i.e. areas where pixel intensity changes significantly), which is based on first or second derivative calculation of image gradients. The edge pixel ratio refers to the ratio of the number of pixels determined to belong to the edge after processing by the edge detection algorithm to the total number of pixels in the image, usually expressed in percentage or decimal form, which is a key indicator of image structural information richness and contrast. The pre-set ratio threshold is a pre-set minimum ratio value used to determine whether the image edge information is sufficient, which is usually determined based on analysis and statistics of a large number of normal scene images or experimental verification.

[0042] S203, calculate the gray mean value corresponding to each video frame image, and take the video frame image with a gray mean value less than a pre-set minimum gray threshold or greater than a pre-set maximum gray threshold as an abnormal video frame image.

[0043] Specifically, when processing each video frame image, first convert it from the original color space (such as RGB) to a grayscale image, that is, convert the red, green, and blue channel values of each pixel point into a single gray intensity value through a weighting formula (for example, the standard formula: Gray = 0.299R + 0.587G + 0.114B), after conversion, traverse the entire grayscale image, accumulate the gray values of all pixel points, then divide the sum by the total number of pixels in the image, to accurately calculate the gray mean value corresponding to the video frame image. The gray mean value is a scalar that quantitatively represents the average brightness level of the entire image. The gray mean value of each video frame image calculated is compared with two pre-set numerical references: one is a pre-set minimum gray threshold representing the acceptable minimum average brightness, and the other is a pre-set maximum gray threshold representing the acceptable maximum average brightness. If the gray mean value of a certain frame image is less than the pre-set minimum gray threshold, it is determined that the overall brightness of the image is too low, which may correspond to a black or severely underexposed scene. If its gray mean value is greater than the pre-set maximum gray threshold, it is determined that the overall brightness of the image is too high, which may correspond to a white or severely overexposed scene. Video frame images that meet either of the above conditions are classified as abnormal video frame images and are marked to be excluded from subsequent feature extraction and analysis processes. As a special detection method for light abnormality in the image quality screening process, it aims to identify invalid video frames caused by night imaging, device failure, exposure parameter error, or strong backlight, etc. which result in overall brightness being too dark or too bright.

[0044] wherein the gray mean value refers to the arithmetic mean of the gray values of all pixel points in the video frame image after conversion to a grayscale image, its numerical range is usually between 0 (pure black) and 255 (pure white), and it is the core statistical quantity for measuring the overall brightness level of the image. The pre-set minimum gray threshold refers to a pre-set lower limit gray value for judging whether the image is invalid due to low brightness, images below this value are usually considered to be underexposed or in an invalid dark state. The pre-set maximum gray threshold refers to a pre-set upper limit gray value for judging whether the image is invalid due to high brightness, images above this value are usually considered to be overexposed or in an invalid white state.

[0045] S103, calculate the first feature similarity between the semantic feature vector of the current video frame image and the semantic feature vector of the previous video frame image, the current video frame image being any video frame image.

[0046] Specifically, the video frame image sequence is traversed, for each frame in the sequence, i.e. defined as the current video frame image, its corresponding semantic feature vector is obtained, and the semantic feature vector corresponding to the immediately preceding frame in time sequence is also obtained. A preset vector similarity measurement method (such as cosine similarity, inner product, or the reciprocal of Euclidean distance) is used to calculate the similarity value between the semantic feature vectors of the two adjacent frames. This value is the first feature similarity, which measures the closeness of the current frame and the previous frame in the semantic space understood by the deep learning model. The higher the similarity value, the more similar the high-level semantic content (such as scene, object, action) contained in the two frames of images. The lower the similarity value, the more significant the change in semantic content.

[0047] wherein the current video frame image refers to a single video frame image that is being processed by the process and has a specific time position in the video frame image sequence. The previous video frame image refers to the video frame image that directly precedes the current video frame image in the time sequence of the video frame image sequence. The first feature similarity refers to a scalar value calculated by a specific mathematical formula, which quantifies the closeness or consistency of the semantic feature vector of the current video frame image and the semantic feature vector of the previous video frame image in the semantic space.

[0048] S104, determining the current video frame image with a first feature similarity less than a preset similarity threshold as an effective video frame image, and taking the semantic feature vector of the effective video frame image as an effective semantic feature vector.

[0049] Specifically, the first feature similarity value is compared with a pre-set and stored judgment criterion, i.e. the preset similarity threshold, which is a scalar value determined by experiment or experience (e.g. 0.92 or 0.95), defining the critical point at which the system determines whether the content of two frames of video is significantly different in semantic level. If the comparison result is that the first feature similarity is less than the preset similarity threshold, it is determined that the semantic content contained in the current video frame image has changed significantly enough relative to the previous frame, and the frame carries new and valuable information, so the current video frame image is marked and determined as an effective video frame image, and the semantic feature vector previously extracted corresponding to the effective video frame image is formally updated and defined as an effective semantic feature vector. This effective semantic feature vector will be sent to the subsequent data storage process as the core data unit representing the key semantic moment. Conversely, if the first feature similarity is greater than or equal to the preset similarity threshold, it indicates that the current frame and the previous frame are highly repetitive in semantics, and the current frame is determined as a redundant frame, and its corresponding semantic feature vector will be discarded and not participate in subsequent storage and retrieval.

[0050] The preset similarity threshold is a predefined numerical demarcation point for judging whether the semantic content of adjacent frames has changed significantly, is a key parameter for controlling the sensitivity of redundancy filtering, and can be experimentally set and optimized by collecting typical video data under the target scene, performing semantic similarity analysis and statistics on continuous frames with different semantic changes, and balancing the business demands for search accuracy and storage efficiency before system deployment. The effective video frame image is used to represent that the semantic content thereof has changed significantly compared to the previous frame in the time sequence, and thus is determined by the system as a non-redundant video frame image with storage and analysis value. The effective semantic feature vector is the semantic feature vector corresponding to the effective video frame image, which is selected and prepared to be stored in the database for future retrieval.

[0051] On the basis of the above embodiment, as an optional embodiment, for Figure 1 The steps S102-S104 shown can be replaced by steps S301-S303, which will be described in detail below.

[0052] S301, extracting the semantic feature vectors corresponding to the video frame images in batches in time sequence by the preset cross-modal learning model, and extracting the semantic feature vectors corresponding to the second preset number of video frame images in each batch.

[0053] Specifically, when processing the video frame image sequence arranged in time sequence, not only can each image be processed independently, but also a batch processing strategy can be used. The system first determines a batch size, i.e., a second preset number (for example, 16, 32, 64, 100, etc., which is not limited here), which represents the number of images input into the model for parallel computation at a time. The images are read in time sequence from the video frame image sequence, and when the second preset number of images is accumulated, these images are grouped into a batch. The image batch is input as a whole into the preset cross-modal learning model, which is optimized internally and can use the parallel computing capability of hardware such as a graphics processing unit (GPU) to simultaneously perform forward propagation calculation on all images in the batch. The model outputs a two-dimensional array or tensor, wherein each row corresponds to an input image in the batch, and the row data is the semantic feature vector corresponding to the image. In this way, the entire video frame image sequence is processed in time sequence and in batches, and finally the set of semantic feature vectors corresponding to all images in the sequence is efficiently obtained. The order of the output feature vectors strictly maintains the original time order of the input images.

[0054] The batch is a data organization form in which multiple data (here, multiple images) are combined into a data set for batch processing in a computing model. The second preset number is a positive integer that defines the number of video frame images in each batch, which is a key parameter for controlling parallel computing granularity and memory occupation.

[0055] S302, calculate the third feature similarity set of the semantic feature vector of the target video frame image and the semantic feature vectors of all video frame images in the previous batch, the target video frame image being any video frame image in the current batch, and the third feature similarity set including a second preset number of third feature similarities.

[0056] Specifically, the purpose is to achieve efficient redundancy detection across batches to process long-time continuous similar content that may be fragmented due to batch division. After performing batch feature extraction, each batch is processed in sequence. For the batch being processed (not the first batch), each frame in the batch is designated as a target video frame image. For each selected target video frame image, the following operations are performed: first, obtain its corresponding semantic feature vector, then obtain the semantic feature vectors of all video frame images in the immediately preceding batch (i.e., the previous batch) in the time sequence, and then calculate the similarity value between the semantic feature vector of the target video frame image and each semantic feature vector in the previous batch using a preset vector similarity measurement method (such as cosine similarity). Since the previous batch contains a second preset number of video frame images, this calculation process will produce a second preset number of similarity values, which are organized into an ordered list or array, i.e., a third feature similarity set. Each element in the set is a third feature similarity, which quantifies the closeness of the target frame in the current batch to each frame in the previous batch in the semantic space.

[0057] The target video frame image is a specific video frame image selected as a redundancy detection reference object in the batch being processed. The previous batch is a data batch processing unit containing a second preset number of video frame images and their semantic feature vectors, which is adjacent to the current batch and precedes it in the time sequence. The third feature similarity set is a data structure containing multiple (equal to the second preset number) scalar values, each of which is a third feature similarity obtained by calculating the similarity between the semantic feature vector of the target video frame image and the semantic feature vector of a specific video frame image in the previous batch. The third feature similarity is a scalar value obtained by a specific mathematical formula, which quantifies the closeness of the semantic feature vector of the target video frame image and the semantic feature vector of any video frame image in the previous batch in the semantic space.

[0058] S303. All target video frame images in the third feature similarity set whose third feature similarity is less than the preset similarity threshold are determined as valid video frame images.

[0059] Specifically, each third feature similarity value in the third feature similarity set is iterated and logically judged. The judgment criterion is: each third feature similarity in the set is compared with a pre-set scalar benchmark used to define whether the semantics are significantly different—a preset similarity threshold. It needs to be confirmed that all third feature similarities in the set are less than this preset similarity threshold. If this strict condition is met, that is, the semantic feature vector of the target video frame image is lower than the threshold in terms of similarity with each frame in the previous batch (indicating that the semantic content is not highly similar), then it is determined that: the target video frame image... The semantic content contained in the target video frame image is not only different from the previous frame, but also significantly different from all the content in the entire previous time period (the duration covered by the previous batch). Therefore, it is a key frame that carries new and independent semantic information. Based on this judgment, the target video frame image is officially marked and determined as a valid video frame image. Conversely, if any third feature similarity in the third feature similarity set is greater than or equal to the preset similarity threshold, it indicates that the semantic content of the target frame is highly repetitive with at least one frame in the previous batch. Therefore, it is judged as a redundant frame and is not retained as a valid video frame image.

[0060] The preset similarity threshold is a predefined numerical boundary point used to determine whether two semantic feature vectors represent highly similar semantic content.

[0061] For example, assume a preset similarity threshold of 0.90 and a second preset quantity of 4. In the current batch, target frame F5 has a third feature similarity set relative to the previous batch F1-F4 of [0.85, 0.82, 0.88, 0.79]. All values ​​in this set [0.85, 0.82, 0.88, 0.79] are less than 0.90. Therefore, frame F5 is determined as a valid video frame image. For target frame F6 in the same batch, its calculated set is [0.87, 0.93, 0.84, 0.80]. Since there is a value 0.93 in this set that is greater than 0.90, the condition that all values ​​are less than the threshold is not met. Therefore, frame F6 is determined to be a redundant frame and is not retained as a valid video frame image.

[0062] S105. Store all valid vector data in the vector database. Valid vector data shall include at least the valid semantic feature vectors and the creation timestamps corresponding to the valid video frame images.

[0063] Specifically, the core information associated with all the video frame images determined to be valid is encapsulated to form a structured data unit, collectively referred to as valid vector data, each valid vector data unit at least forcibly contains a semantic feature vector extracted from the valid video frame image, which represents the high-level semantic content thereof, i.e., a valid semantic feature vector, and an accurate time point marker recorded when the valid video frame image is generated in the original video stream or processed by the system, i.e., a creation timestamp, and then a connection with a dedicated vector database, which is an optimized storage system for efficient storage and retrieval of high-dimensional vector data, is established, and through the insertion interface provided by the database, the encapsulated valid vector data is written into the vector database in the form of records, in batches or one by one, and persistently, after this operation, all feature vectors representing the key semantic moments of the video content and their temporal context information are converted from the volatile memory state to the persistent data that can be queried for a long time and stably, and constitute the core data basis of the subsequent video retrieval service.

[0064] Among them, the valid vector data refers to a structured data unit which at least encapsulates the valid semantic feature vector and the creation timestamp associated with an effective video frame image, and is the basic record unit for storage and retrieval of the vector database. The creation timestamp refers to a numerical label based on standard time (such as Unix timestamp) used to identify and record the moment when the valid video frame image is generated or processed by the system. The vector database is a database management system specially designed for storing, indexing and quickly querying high-dimensional vector data.

[0065] On the basis of the above embodiment, as an optional embodiment, for Figure 1 The step S105 shown can be implemented by steps S1051-S1055, which will be described in detail below.

[0066] S1051, create a unique identifier corresponding to the target valid vector data as the primary key, and the target valid vector data is any valid vector data.

[0067] Specifically, when processing each valid vector data to be stored (referred to as target valid vector data), a unique identifier that is not repeated with any existing record in the library is first generated or allocated for the data unit, that is, the unique identifier, the generation of the identifier follows strict uniqueness rules, and various strategies can be used, such as a globally increasing sequence number, a snowflake algorithm combining a timestamp and a machine identifier, or a universally unique identifier (UUID) with strong randomness. After generating the unique identifier, it is formally designated as the primary key of the target valid vector data record in the database. The identifier will become the main basis for indexing and quickly locating the specific record within the database. The unique identifier (primary key) is bound with the valid semantic feature vector originally contained in the target valid vector data, the creation timestamp and other fields to form a complete database record with a globally unique primary key identifier, providing precise addressing capability for subsequent efficient insertion, retrieval, update or deletion operations.

[0068] Among them, the target valid vector data refers to a single valid vector data unit selected from the set of all valid vector data to be stored and currently being processed by the operation. The unique identifier refers to a string or numerical sequence generated or allocated by the system, which has absolute uniqueness and cannot be repeated within its scope (usually the entire database or a specific collection), and is used to uniquely mark a data entity. The primary key refers to one or a group of fields in a database table or collection designed to uniquely identify each record in the table, whose value must be unique and non-empty, and is the key to establishing data table index and ensuring data integrity.

[0069] S1052, obtaining a video stream identifier of a target valid vector data corresponding to the video stream to be processed.

[0070] Specifically, after generating the primary key, the generation link of the target valid vector data is traced back, and according to the associated valid video frame image, it is traced back to the original video stream that performs the frame extraction operation. From the metadata, processing context or configuration information of the video stream, a string or code that is specially used to uniquely identify the specific video stream in the system is extracted, that is, the video stream identifier. The identifier is different from the "unique identifier" used when generating the primary key, and it does not point to a single record, but to the continuous data source that generates the records.

[0071] Among them, the video stream identifier refers to a string or code that is pre-defined or allocated in the system and is used to uniquely name and distinguish different video streams to be processed.

[0072] S1053, generating an access path string of the target valid vector data corresponding to the valid video frame image through a preset structured path.

[0073] Specifically, the system locates the valid video frame image file associated with the target valid vector data. This file may already be stored in an object storage service (such as MinIO) or a distributed file system. Then, a predefined path generation rule or function is called, i.e., a preset structured path. This rule is a logical template, which usually embeds key variable information. During generation, the system dynamically fills the template with metadata related to the target valid vector data. This metadata includes, but is not limited to: the video stream identifier (vid), creation timestamp, primary key (ID) corresponding to the data, or the system's pre-configured bucket name, base URL, etc. By combining and formatting these variable values ​​according to the rules, a unique, well-structured string that can be directly used for network access or protocol reading is generated. This is the access path string, which indicates the precise location of the original image file in network storage.

[0074] The preset structured path refers to a pre-configured logical rule or string template that defines how to combine specific metadata into the final file access address. The access path string refers to a string address that conforms to a specific storage service protocol format (such as HTTP / HTTP SURL, S3 path, etc.) and is used to uniquely point to and remotely access the original file of the valid video frame image.

[0075] For example, the access path string can be generated in a format like " / object_storage_bucket / {video stream identifier} / {year} / {month} / {day} / {primary key}.mp4" (or .jpg), which facilitates efficient management and access to raw images or video clips by the storage system.

[0076] S1054. Based on the effective semantic feature vector, generate a similarity retrieval index corresponding to the target effective vector data through a preset clustering algorithm.

[0077] Specifically, for the core data unit contained in the target effective vector data, the effective semantic feature vector (a high-dimensional floating-point number vector), a pre-configured algorithm specially used for spatial division and organization of high-dimensional vectors, i.e., a preset clustering algorithm (for example, the clustering stage of K-Means or IVF), is called, and instead of being run independently for a single vector, it is run on a data set containing a large number of stored or ready-to-store feature vectors. The effective semantic feature vector of the target effective vector data is input into the clustering algorithm together with other vectors in the data set. The algorithm divides the entire vector space into several (for example, 2048) subspaces or clusters through iterative calculation, and calculates a center point for each cluster. For the target effective vector data, the algorithm calculates which cluster the vector belongs to or the distance relationship with each cluster center. The result of this process is a set of metadata or labels for accelerating the nearest neighbor search of the vector, collectively referred to as a similarity search index. The index is not a readable label, but an internal data structure (such as cluster ID in inverted list, quantization coding, etc.), so that in subsequent approximate nearest neighbor (ANN) search, the relevant cluster where the data is located can be quickly located and fine matching can be performed, greatly accelerating the retrieval process of high-dimensional vectors. The index indicates the position of the vector in the constructed index structure and is a component of the target effective vector data record, stored in the vector database together with the primary key, feature vector itself and other fields, and is used to serve subsequent fast similarity search queries.

[0078] Among them, the preset clustering algorithm refers to a pre-selected and configured machine learning algorithm for unsupervised grouping or spatial division of a large number of high-dimensional vectors, and its core goal is to construct an efficient data structure for approximate nearest neighbor search. The similarity search index refers to an internal data structure or metadata label generated by the preset clustering algorithm, which is used to accelerate similarity search for a specific vector or entire vector set.

[0079] S1055, based on the creation timestamp, the primary key, the video stream identifier, the access path string, the creation timestamp, the similarity search index and the target effective vector data corresponding to the target effective vector data are stored in the vector database according to the preset storage partition.

[0080] Specifically, after all the necessary fields (primary key, video stream identifier, access path string, creation timestamp, similarity search index) have been generated for the target effective vector data, the final storage location of the record needs to be determined according to its core time attribute, the creation timestamp. According to a predefined data organization strategy, i.e., a preset storage partition rule, the creation timestamp is analyzed. Common rules are to divide by time dimension, such as monthly, weekly, or daily creation of independent partitions. According to this rule, the corresponding partition identifier (for example, "2023-10" represents the October 2023 partition) is parsed from the creation timestamp. Subsequently, all components of the target effective vector data complete record, including its own effective semantic feature vector and other core data (i.e., the target effective vector data), as well as all associated fields (primary key, video stream identifier, access path string, creation timestamp, similarity search index) generated for it, are combined into a complete, structured data row. Finally, through a database operation interface, the data row is explicitly stored in the specific database partition specified by the partition identifier. If the partition does not exist, the system may dynamically create it according to the rules.

[0081] Among them, the preset storage partition refers to a strategy that predefines a database table or collection to be physically or logically divided into multiple smaller and more manageable units based on a certain attribute of the data item (here, time).

[0082] S106, acquire the user's search request, and extract the search semantic vector corresponding to the search request through the preset cross-modal learning model.

[0083] Specifically, the system receives and identifies user search requests. These requests are diverse, not limited to traditional text keywords, but can encompass various data types such as images, audio, video clips, and even combinations of text and images. This reflects the "cross-modal" input characteristic. After obtaining this raw, multimodal search request, the system inputs it into a pre-trained and deployed pre-defined cross-modal learning model. The core capability of this model lies in understanding and associating deep semantic relationships between different modalities of data. Through a complex neural network structure (e.g., including encoders that process different modalities of data, and an interactive module for aligning their semantics), the input, whether textual descriptions or visual images, is transformed and mapped into a unified, high-dimensional mathematical space. In this shared semantic space, content from different modalities but with similar meanings is represented as points that are close to each other. The final output of this process is to generate a unique, standardized numerical representation of the search request—a search semantic vector. This vector is an array of floating-point numbers, no longer the original pixels or characters, but a highly condensed abstract mathematical representation of the user's search intent, serving as the fundamental basis for subsequent efficient and accurate semantic matching and retrieval.

[0084] Here, a retrieval request refers to a user's instruction, entered through a human-computer interaction interface, expressing their desire to find specific content from the video library. This instruction can be in the form of natural language text or example images. A retrieval semantic vector is a fixed-dimensional numerical vector obtained by transforming the user's retrieval request (whether text or image) through the preset cross-modal learning model. It represents the quantitative expression of the request in the model's semantic space.

[0085] Based on the above embodiments, as an optional embodiment, the retrieval request includes a text retrieval request and an image retrieval request, and the preset cross-modal learning model includes a text-to-image interface and an image-to-image interface, targeting... Figure 1 The step S106 shown can be implemented through steps S1061-S1062, which will be explained in detail below.

[0086] S1061. When the retrieval request is a text retrieval request, generate the retrieval semantic vector of the text retrieval request through the text redirection interface.

[0087] Specifically, when receiving a user's search request, if the system determines that the input content of the request is composed of a character sequence, expressing a natural language text with semantic description, it will be classified as a text search request, and the part of the loaded preset cross-modal learning model responsible for processing the text modality will be called, that is, the text-to-vector interface, which is usually a text encoder subnetwork (e.g., a text encoder based on the Transformer architecture) within the model. The original text string input by the user (e.g., "find a white sedan") is sent to this interface. The text-to-vector interface first performs standardization preprocessing on the text (such as word segmentation and conversion to token ID sequences), and then performs deep context semantic understanding and encoding on the text sequence through its multi-layer neural network. Finally, a fixed-dimensional, dense floating-point number vector is generated at the output layer. This vector is the search semantic vector corresponding to the text search request, encapsulating the abstract semantic information contained in the input text description, and is in the same vector semantic space as the semantic feature vectors of the video frame images stored in the database, thereby having the comparability to directly perform similarity calculation.

[0088] Among them, the text search request refers to the type of search request that uses a natural language text string as the input carrier. The text-to-vector interface refers to the functional module or subnetwork in the preset cross-modal learning model that is specifically designed to receive text input and encode it into a semantic vector.

[0089] S1062, when the search request is an image search request, generate the search semantic vector of the image search request through the image-to-vector interface.

[0090] Specifically, when the received search request is modality parsed, if it is determined that the input content of the request is a digital image file or image data stream, it will be classified as an image search request, and the part of the loaded preset cross-modal learning model responsible for processing the image modality will be called, that is, the image-to-vector interface, which is usually a visual encoder subnetwork (e.g., an image encoder based on Vision Transformer or convolutional neural network) within the model. The image data input by the user (which may have undergone preprocessing such as size adjustment and normalization) is sent to this interface, and the pixel information of the input image is abstracted and understood layer by layer through its deep neural network, from low-level texture and edge features to high-level semantic concepts. Finally, a fixed-dimensional, dense floating-point number vector is generated at the output layer. This vector is the search semantic vector corresponding to the image search request, condensing the core visual semantic content of the query image, and completely in the same vector semantic space as the semantic feature vectors of the video frame images stored in the database, thereby enabling direct similarity comparison.

[0091] The image retrieval request refers to a retrieval request type taking a digital image file or an image data stream as an input carrier. The image-to-vector interface refers to a functional module or a subnetwork specially designed to receive image input and encode it into a semantic vector in the preset cross-modal learning model.

[0092] S107, calculate a second feature similarity corresponding to each of a plurality of target effective semantic feature vectors in the vector database, the target effective semantic feature vector being an effective semantic feature vector selected based on a constraint condition of the retrieval request.

[0093] Specifically, a retrieval semantic vector representing a user query intention is obtained, and a target data range for similarity calculation in the vector database, i.e., a plurality of target effective semantic feature vectors, is determined. These target vectors are a subset of all effective semantic feature vectors in the database, selected according to retrieval conditions such as specified time range, camera position, etc. If there are no additional conditions, it may be all vectors. A preset vector similarity measurement method compatible with the index construction (such as cosine similarity, inner product, or Euclidean distance) is used to perform independent similarity calculation between the retrieval semantic vector and each of the plurality of target effective semantic feature vectors. Each calculation outputs a scalar value, which quantitatively represents the closeness or semantic consistency of the retrieval semantic vector and a specific target effective semantic feature vector in the unified high-dimensional semantic space defined by the model. After performing this calculation for each target vector, a series of similarity values are obtained, which are collectively referred to as second feature similarities. Each second feature similarity uniquely corresponds to a target effective semantic feature vector.

[0094] The target effective semantic feature vector refers to a set of effective semantic feature vectors selected from all effective semantic feature vectors in the vector database according to specific constraint conditions (such as time and space) of the current retrieval, which need to be compared for similarity.

[0095] Through the above embodiments, the abstract "find similar content" requirement is converted into a series of objective scores (second feature similarities) that can be sorted and compared through mathematical calculations. This is not only a necessary step from "understanding intention" to "producing results" in the entire retrieval process, but also the core of determining retrieval quality. By comparing the query vector with all relevant target vectors one by one, any potential similar results are ensured not to be missed, so that the system can objectively find the video segments that best match the user's needs from a large number of candidates, thereby ensuring the accuracy and relevance of the retrieval results.

[0096] On the basis of the above embodiments, as an optional embodiment, in the step S107, Figure 1Before the step S106 shown, the urban monitoring video retrieval method provided by the embodiments of the present application further includes steps S401-S404, which are specifically described below.

[0097] S401, parse the time range requirement and the return result number of the retrieval request by the preset cross-modal learning model, and take the return result number as a first preset number.

[0098] Specifically, when receiving the retrieval request of the user, the request may not only contain the core query content (text or image), but also may be attached with structured or natural language expression constraint conditions. The preset cross-modal learning model or the integrated natural language understanding capability thereof is used to deeply analyze the complete input of the request (especially for the text request) to identify and extract the limiting information therein. First, the model needs to identify whether the user specifies the time range (for example, "this afternoon", "during the National Day in 2023", "last Wednesday") that the user hopes to search. If such a time description is identified, the model will parse and convert it into a standardized time interval parameter that the system can process, that is, the time range requirement, and identify whether the user specifies the number of results (for example, "find the top 5 most similar" or "show me 10 screenshots") that the user expects to return. If such a number description is identified, the model will parse it into a specific integer value, which is explicitly assigned as the return result number. The return result number is assigned to an internal variable or parameter, which will be used as a key threshold to determine the length of the final output list in the subsequent result sorting and truncation steps, that is, the first preset number. If the number of results is not explicitly specified in the request, the system will use a default value as the first preset number.

[0099] The time range requirement is a specific time period that the user hopes to retrieve the video content from the retrieval request, which is usually represented as a start and end timestamp. The return result number is the maximum number of similar results that the user expects the system to return from the retrieval request.

[0100] Through the above embodiments, not only the "what" (core semantics) that the user wants to find is focused on, but also the "where" (time range) and "how many" (number of results) that the user wants to find are accurately captured. By limiting the time range, the system can significantly reduce the range of similarity calculation and index loading to the database partition corresponding to the specified time period, which greatly reduces unnecessary calculation and data I / O, improves the retrieval response speed by several times, and reduces the system load. By explicitly specifying the first preset number (the return result number), the system can directly specify the Top-K parameter when retrieving the underlying vector database, so that the database engine can optimize its search process and only accurately calculate and return the required number of most similar results, avoiding the acquisition and transmission of a large number of irrelevant intermediate results, and further improving the retrieval efficiency and network bandwidth utilization.

[0101] S402, determine the storage partition corresponding to the retrieval request based on the time range requirement, the storage partition being used to represent the partition address of the data stored in the vector database in chronological order.

[0102] Specifically, the parsed and standardized time range requirement is obtained, which usually includes a start timestamp and an end timestamp, and then the given time range is analyzed by referring to the pre-set data partition strategy of the vector database (which must be consistent with the strategy of storing data according to the pre-set storage partition when the data is imported, for example, both are monthly partitioning), and all time interval units covered by the time range and meeting the partition rules are calculated and listed, for example, if the database is monthly partitioned, the time range "2023-10-25 to 2023-11-05" will cover "2023-10" and "2023-11" two months interval. Each such time interval unit corresponds to a physically or logically independent storage partition in the database. The unique identifiers (such as partition names "2023-10", "2023-11") corresponding to these partitions are determined as the storage partition set corresponding to the current retrieval request. These partition identifiers are the partition addresses pointing to a specific data subset in the database, and the subsequent retrieval operation will be limited to these partitions.

[0103] Among them, the time range requirement refers to the start and end time period of the video content specified by the user and hoped to be retrieved. The storage partition refers to the physical or logical unit divided according to a specific rule (here, time) inside the vector database, used to store and manage a data subset independently. The partition address refers to the name or identifier used to uniquely identify and locate a specific storage partition in the database.

[0104] S403, send the retrieval semantic vector, the storage partition and the first preset number to the query interface of the vector database.

[0105] Specifically, the three core retrieval elements prepared in the preceding steps are encapsulated, one is the mathematical expression representing the user's query intention, i.e. the retrieval semantic vector, one element is the physical location pointing to the data range of this retrieval, i.e. the storage partition (one or more partition identifiers), and the last element is the parameter controlling the output size of this retrieval, i.e. the first preset number. By calling the special application programming interface (API) provided by the vector database service, i.e. the query interface, the encapsulated three elements are sent to the vector database as core parameters in a structured data format (such as JSON request body or specific function parameters), triggering the retrieval calculation process of the database server. In addition, the request usually contains other necessary configuration parameters, such as the specified index name, similarity measure type (such as inner product), and index search parameters, etc., which are not limited here.

[0106] The query interface refers to the application programming interface exposed by the vector database to the outside, which is used to receive similarity search requests and return results.

[0107] Through the above embodiment, the complex retrieval intention (what to find, where to find, how much to find) is abstracted and encapsulated into instructions that can be accurately understood by the database engine, ensuring that even with city-level massive data, each retrieval can quickly obtain the most relevant and accurate results with minimal resource consumption.

[0108] S404, taking the valid semantic feature vectors stored in the storage partition as target valid semantic feature vectors.

[0109] Specifically, based on the storage partition (one or more) specified in the retrieval request sent by the upper application, the vector database system will take these storage partitions as the boundary of data access when performing queries internally. From these specified partitions, all valid semantic feature vectors (i.e. video frame feature vectors retained after redundancy filtering and screening) stored therein are read out, and this complete feature vector set limited within the partition is established as the entire object that needs to be directly compared for similarity calculation. This established set is referred to as the target valid semantic feature vector. In actual efficient retrieval implementation, the database does not read all these vectors and then compare them one by one, but uses the pre-constructed index (such as IVF index) to perform fast neighbor search within the specified partition. However, regardless of how the operation is performed, the reference data range for similarity calculation (or approximate calculation) is logically equivalent to the set of all valid semantic feature vectors stored in the storage partition.

[0110] For example, a one-time retrieval request specifies a time range of "October 1-31, 2023", which is mapped to the storage partition partition_2023_10 according to the partitioning strategy. When executing the query, the vector database logically regards all the approximately 10 million valid semantic feature vectors stored in the partition_2023_10 partition as the "target valid semantic feature vector" set of this retrieval. When the query interface receives a retrieval semantic vector, the retrieval engine of the database will quickly find the most similar several results in this set of 10 million vectors (rather than in the entire database that may contain hundreds of millions of vectors).

[0111] Through the above embodiment, the core mechanism that guarantees that the entire system can still maintain stable and predictable retrieval performance in the face of long-term operation and continuous data accumulation in business scenarios is ensured, effectively solving the scalability bottleneck problem in massive data retrieval.

[0112] S108, select the first preset number of valid video frame images corresponding to the second feature similarity from high to low as the retrieval result.

[0113] Specifically, all second feature similarity values are globally sorted in descending order. After sorting, the system selects the specified number (i.e., the first preset number) of the highest scores from the top of the sorted list according to the first preset number (for example, the user requires to return 10 results) determined in advance. Each selected second feature similarity is uniquely associated with a target valid semantic feature vector, and the feature vector accurately corresponds to an original valid video frame image record (including image access path, timestamp, camera identifier, and other metadata) stored in the database. The complete data (or key metadata such as image access path, timestamp, and camera identifier) of these associated valid video frame images is extracted and organized to form a final ordered list. This list is the final output of this retrieval request and is defined as the retrieval result, which is returned to the user end for display or subsequent processing.

[0114] The retrieval result refers to a set of valid video frame images and their associated information that best meet the user's query intent after sorting by similarity.

[0115] Through the above embodiment, it is ensured that the time to return the final result is efficient and stable regardless of the size of the underlying data, which is a key step to achieve real-time interactive search experience for the entire retrieval method.

[0116] On the basis of the above embodiment, as an optional embodiment, the urban monitoring video retrieval method provided in the embodiment of the present application further includes steps S501-S503, which are described in detail below.

[0117] S501, store the valid video frame data with a creation duration less than a first preset duration to a hot data partition based on the creation timestamp corresponding to each valid video frame image, and the hot data partition is configured in a resident memory state.

[0118] Specifically, for each to-be-stored or stored valid video frame image data, first, the creation timestamp corresponding to the data is obtained, the current time is recorded, and the time difference between the creation timestamp and the current time, i.e., the creation duration of the data, is calculated. The calculated creation duration is compared with a time threshold representing the boundary of recent data, i.e., the first preset duration (for example, 15 days). If the creation duration of a certain valid video frame data is less than the first preset duration, it is determined that the data belongs to the hot data with high-frequency access, and then the entire content of the data (including the valid semantic feature vector, the timestamp, the video stream identifier, the image access path, etc.) is stored or migrated to a specially designed and independent database partition, which is defined as the hot data partition. More importantly, at the database configuration level, a performance optimization strategy is applied to the hot data partition, i.e., the hot data partition is set in a resident memory state, which means that all indexes and vector data in the partition are preloaded and continuously retained in the physical memory (RAM) of the database server, thereby avoiding the high-latency I / O operation required for data reading in the traditional disk storage mode.

[0119] The creation duration refers to the length of time elapsed from the creation timestamp of the data to the current system time. The first preset duration refers to a pre-set time length threshold for distinguishing whether the data is recent data. The hot data partition refers to a storage partition in the vector database specially used for storing recent high-frequency access data and having an independent performance configuration. The resident memory state refers to a configuration state in which the database management system persistently retains all or core data (such as indexes and vectors) of a specified partition in the physical memory to achieve the fastest access speed.

[0120] Through the above embodiments, the storage architecture is intelligently layered, the recent data (hot data) most likely to be accessed is placed in the memory medium with the fastest access speed, real-time retrieval is realized, and the scene requirements of high time efficiency in emergency response and real-time control in urban monitoring are met. At the same time, by separating the management of hot data and historical data, the huge hardware cost caused by resident memory of all data is avoided, and the optimal balance between system performance and cost is achieved.

[0121] S502, store the valid video frame data with a creation duration greater than or equal to the first preset duration and less than or equal to a second preset duration to a regular partition.

[0122] Specifically, for each valid video frame data, after calculating its creation duration, it is not only compared with the first preset duration (e.g. 15 days), but also compared with a larger time threshold representing the boundary of historical data or archived data, i.e. the second preset duration (e.g. 90 days). The system performs a logical judgment: if the creation duration of a piece of valid video frame data is greater than or equal to the first preset duration (i.e. it is no longer “recent” data), but at the same time is less than or equal to the second preset duration (i.e. it has not yet exceeded the retention period of regular historical data), it is determined that the data belongs to “warm data” or regular access data. Subsequently, the complete record of the data is stored or migrated to a general storage partition for storing such data, i.e. a regular partition. This regular partition generally follows the basic partitioning strategy (e.g. partitioning by month), and is configured in a standard storage mode, with data generally residing on a disk or solid state drive (SSD) and loaded into memory as needed for computation. Such a partition does not have the special performance optimization of hot data partitions that are resident in memory, but is still logically and managerially separated from older data, maintaining the orderliness of data organization.

[0123] The second preset duration refers to a pre-set longer time length threshold for distinguishing regular data from expired data to be cleaned up. The regular partition refers to a standard storage partition in the vector database for storing regular accessible data with a creation duration between the first preset duration and the second preset duration.

[0124] S503, deleting valid video frame data with a creation duration greater than the second preset duration, and deleting the storage partition corresponding to the valid video frame data with a creation duration greater than the second preset duration.

[0125] Specifically, the creation duration is strictly compared with the threshold representing the longest valid retention period of data, i.e. the second preset duration (e.g. 90 days). If the result of the judgment is that the creation duration is greater than the second preset duration, two-step cleaning operations are performed: first, the data record itself is deleted from the corresponding partition in the database; second, and more importantly, since all data records in the entire storage partition (e.g. a partition representing a certain month) have exceeded the second preset duration (because the partition is organized continuously by time), it is determined that the partition has expired as a whole, and a partition-level deletion operation is further performed to completely remove the metadata, index and possibly remaining data structure of the partition from the vector database, ensuring deep cleaning of expired data and releasing the storage space, memory index and file handle system resources occupied by the data.

[0126] Through the above embodiment, the automatic closed-loop management of the data life cycle is realized, a rigid expiration elimination mechanism is established for the characteristic that the value of video monitoring data decays over time, not only the expired records in the database are deleted, but also the entire partition and external storage file associated therewith are deleted, so that the full-link resources from calculation, memory to disk are completely and cleanly recycled, the unlimited growth of the system storage cost is effectively controlled, which is crucial for the wisdom city-level application which needs to run for a long time. By regularly cleaning the oldest data partition, the total amount of indexes maintained by the system and the size of metadata managed are controlled within a relatively stable range, the performance degradation problems such as index expansion and slow maintenance operation caused by unlimited accumulation of data are avoided, and the search performance stability and maintainability of the system under long-term operation are ensured.

[0127] With reference to Figure 2 A city monitoring video retrieval device 600 is provided for the embodiment of the application, and the device comprises a video frame extraction module 601, a feature extraction module 602, a vector deduplication module 603, a vector database module 604, and a semantic retrieval module 605.

[0128] The video frame extraction module 601 is configured to acquire a plurality of channels of a to-be-processed video stream, and extract frames from a target to-be-processed video stream through an independent video processing thread to obtain a video frame image sequence, the target to-be-processed video stream is any channel of the to-be-processed video stream, one target to-be-processed video stream corresponds to one independent video processing thread, and the video frame image sequence comprises a plurality of video frame images.

[0129] The feature extraction module 602 is configured to extract semantic feature vectors corresponding to the video frame images in time sequence through a preset cross-modal learning model.

[0130] The vector deduplication module 603 is configured to calculate a first feature similarity between a semantic feature vector of a current video frame image and a semantic feature vector of a previous video frame image, the current video frame image being any video frame image; determine the current video frame image with a first feature similarity less than a preset similarity threshold as an effective video frame image, and take the semantic feature vector of the effective video frame image as an effective semantic feature vector.

[0131] The vector database module 604 is configured to store all effective vector data to a vector database, the effective vector data at least comprising a creation timestamp corresponding to the effective semantic feature vector and the effective video frame image.

[0132] The semantic retrieval module 605 is configured to acquire a retrieval request of a user, extract a retrieval semantic vector of the retrieval request by using a preset cross-modal learning model, calculate second feature similarities respectively corresponding to the retrieval semantic vector and a plurality of target effective semantic feature vectors in a vector database, and select, from high to low, a first preset number of effective video frame images corresponding to the second feature similarities as retrieval results.

[0133] On the basis of the above-mentioned embodiments, the city monitoring video retrieval device 600 further comprises an anomaly detection module 606 configured to discard all abnormal video frame images before the feature extraction module 602, the abnormal video frame images are used to represent video frame images with quality anomalies, and the identification method of the abnormal video frame images comprises at least one of the following: calculating a Laplacian operator variance corresponding to each video frame image by using a preset Laplacian variance algorithm, and taking a video frame image with a Laplacian operator variance less than a preset variance threshold as an abnormal video frame image; calculating an edge pixel ratio corresponding to each video frame image by using a preset edge detection algorithm, and taking a video frame image with an edge pixel ratio less than a preset ratio threshold as an abnormal video frame image; calculating a gray mean value corresponding to each video frame image, and taking a video frame image with a gray mean value less than a preset minimum gray threshold or greater than a preset maximum gray threshold as an abnormal video frame image.

[0134] On the basis of the above-mentioned embodiments, the feature extraction module 602 is further configured to extract semantic feature vectors corresponding to the video frame images in batches in time sequence by using a preset cross-modal learning model, and extract semantic feature vectors corresponding to a second preset number of video frame images in one batch.

[0135] On the basis of the above-mentioned embodiments, the vector deduplication module 603 is further configured to calculate a third feature similarity set of a semantic feature vector of a target video frame image and semantic feature vectors of all video frame images in a previous batch, the target video frame image being any video frame image in a current batch, and the third feature similarity set comprising a second preset number of third feature similarities; and determine, as an effective video frame image, a target video frame image with all third feature similarities in the third feature similarity set being less than a preset similarity threshold.

[0136] On the basis of the above-mentioned embodiments, the retrieval request comprises a text retrieval request and an image retrieval request, the preset cross-modal learning model comprises a text-to-vector interface and an image-to-vector interface, and the semantic retrieval module 605 is further configured to generate a retrieval semantic vector of the text retrieval request by using the text-to-vector interface when the retrieval request is the text retrieval request, and generate a retrieval semantic vector of the image retrieval request by using the image-to-vector interface when the retrieval request is the image retrieval request.

[0137] On the basis of the above-mentioned embodiments, the semantic retrieval module 605 is further configured to analyze the time range requirement, the number of returned results of the retrieval request by a preset cross-modal learning model, and take the number of returned results as a first preset number; determine the storage partition corresponding to the retrieval request based on the time range requirement, the storage partition being used to represent the partition address of the data stored in the vector database in chronological order; send the retrieval semantic vector, the storage partition, and the first preset number to the query interface of the vector database; and take the effective semantic feature vector stored in the storage partition as a target effective semantic feature vector.

[0138] On the basis of the above-mentioned embodiments, the city monitoring video retrieval device 600 further comprises a data management module 607 configured to store the effective video frame data with a creation duration less than a first preset duration to a hot data partition based on the creation timestamp corresponding to each effective video frame image, the hot data partition being configured in a resident memory state; store the effective video frame data with a creation duration greater than or equal to the first preset duration and less than or equal to a second preset duration to a regular partition; delete the effective video frame data with a creation duration greater than the second preset duration, and delete the storage partition corresponding to the effective video frame data with a creation duration greater than the second preset duration.

[0139] On the basis of the above-mentioned embodiments, the vector database module 604 is further configured to create a unique identifier corresponding to the target effective vector data as a primary key, the target effective vector data being any effective vector data; obtain the video stream identifier of the target effective vector data corresponding to the to-be-processed video stream; generate an access path string of the effective video frame image corresponding to the target effective vector data through a preset structured path; generate a similar retrieval index corresponding to the target effective vector data through a preset clustering algorithm based on the effective semantic feature vector; and store the primary key, the video stream identifier, the access path string, the creation timestamp, the similar retrieval index, and the target effective vector data corresponding to the target effective vector data to the vector database according to a preset storage partition based on the creation timestamp.

[0140] It should be noted that: the device provided in the above-mentioned embodiments is only exemplified by the division of the above-mentioned functional modules in realizing its functions, and in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above-mentioned embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0141] The embodiments of the present application provide an electronic device, as shown in the figure, the electronic device 700 can include: at least one processor 701, at least one network interface 704, a user interface 703, a memory 705, at least one communication bus 702. Figure 3 As shown in the figure, the electronic device 700 can include: at least one processor 701, at least one network interface 704, a user interface 703, a memory 705, at least one communication bus 702.

[0142] The communication bus 702 is configured to realize the connection communication between the components.

[0143] The user interface 703 can include a display screen and a camera. Optionally, the user interface 703 can further include a standard wired interface and a wireless interface.

[0144] The network interface 704 can optionally include a standard wired interface and a wireless interface (e.g., a WI-FI interface).

[0145] The processor 701 can include one or more processing cores. The processor 701 is connected to various parts of the electronic device 700 through various interfaces and lines, and performs various functions of the electronic device 700 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 705, and calling data stored in the memory 705. Optionally, the processor 701 can be implemented in at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 701 can be integrated with a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU is mainly used to process an operating system, a user interface, and an application program. The GPU is used to render and draw the content to be displayed on the display screen. The modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 701, but can be implemented by a separate chip.

[0146] The memory 705 can include a random access memory (RAM) and can also include a read-only memory (ROM). Optionally, the memory 705 includes a non-transitory computer-readable storage medium. The memory 705 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 705 can include a program storage area and a data storage area, where the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the various method embodiments described above, and the like; the data storage area can store data involved in the various method embodiments described above, and the like. The memory 705 can also optionally be at least one storage device located away from the aforementioned processor 701. As shown, the memory 705 as a computer storage medium can include an operating system, a network communication module, a user interface module, and an application program of a city monitoring video retrieval method. Figure 3

[0147] In the electronic device 700 shown, the user interface 703 is mainly used to provide an interface for user input and obtain data input by the user; and the processor 701 can be used to call the application program of the city monitoring video retrieval method stored in the memory 705, and when executed by one or more processors, causes the electronic device to perform the city monitoring video retrieval method described above. Figure 3

[0148] The embodiments of the present application provide a computer readable storage medium, which stores instructions. When executed by one or more processors, causes the city monitoring video retrieval method provided by the embodiments of the present application to be performed.

[0149] The above is only exemplary embodiments of the present disclosure, and cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the specification and practicing the true principles of the present disclosure. The present application is intended to cover any variations, uses, or adaptive changes of the present disclosure that follow the general principles of the present disclosure and include common knowledge or conventional technical means in the art that are not described in the present disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.​​

Claims

1. A method for retrieving urban surveillance video, characterized in that, include: A multi-channel video stream to be processed is acquired, and a video frame image sequence is obtained by extracting frames from the target video stream to be processed through an independent video processing thread. The target video stream to be processed is the video stream to be processed in any channel. One target video stream to be processed corresponds to one independent video processing thread. The video frame image sequence includes multiple video frame images. The semantic feature vectors corresponding to each video frame image are extracted in chronological order using a preset cross-modal learning model. Calculate the first feature similarity between the semantic feature vector of the current video frame image and the semantic feature vector of the previous video frame image, wherein the current video frame image is any of the aforementioned video frame images; The current video frame image whose first feature similarity is less than a preset similarity threshold is determined as a valid video frame image, and the semantic feature vector of the valid video frame image is used as a valid semantic feature vector; All valid vector data are stored in a vector database, and the valid vector data includes at least the creation timestamps corresponding to the valid semantic feature vectors and the valid video frame images; Obtain the user's search request, and extract the search semantic vector corresponding to the search request through the preset cross-modal learning model; Calculate the second feature similarity between the retrieved semantic vector and the second feature similarity between the retrieved semantic vector and the second feature similarity between the second feature and the second feature similarity between the second feature and the third ... A first preset number of valid video frame images corresponding to the second feature similarity are selected from high to low as the retrieval results.

2. The method according to claim 1, characterized in that, Before extracting the semantic feature vectors corresponding to each video frame image in chronological order using a preset cross-modal learning model, the method further includes discarding all abnormal video frame images, which are used to characterize video frame images with quality abnormalities, and the method for identifying abnormal video frame images includes at least one of the following: The variance of the Laplacian operator corresponding to each video frame image is calculated using a preset Laplacian variance algorithm. Video frame images whose Laplacian operator variance is less than a preset variance threshold are identified as abnormal video frame images. Using a preset edge detection algorithm, the proportion of edge pixels corresponding to each video frame image is calculated, and the video frame images whose edge pixel proportion is less than a preset proportion threshold are regarded as abnormal video frame images. Calculate the average grayscale value corresponding to each video frame image, and designate the video frame images whose average grayscale value is less than a preset minimum grayscale threshold or greater than a preset maximum grayscale threshold as the abnormal video frame images.

3. The method according to claim 1, characterized in that, The method further includes: The semantic feature vectors corresponding to each video frame image are extracted in batches according to time sequence using the preset cross-modal learning model, and a second preset number of semantic feature vectors corresponding to each video frame image are extracted in one batch. Calculate the third feature similarity set between the semantic feature vector of the target video frame image and the semantic feature vectors of all video frame images in the previous batch, wherein the target video frame image is any video frame image in the current batch, and the third feature similarity set includes the second preset number of third feature similarities; The target video frame images whose third feature similarity is less than the preset similarity threshold in the third feature similarity set are determined as the valid video frame images.

4. The method according to claim 1, characterized in that, The retrieval request includes text retrieval requests and image retrieval requests. The preset cross-modal learning model includes a text-to-text interface and an image-to-text interface. Obtaining the user's retrieval request and extracting the retrieval semantic vector of the retrieval request through the preset cross-modal learning model includes: When the search request is a text search request, the search semantic vector of the text search request is generated through the text redirection interface; When the retrieval request is an image retrieval request, the retrieval semantic vector of the image retrieval request is generated through the image steering interface.

5. The method according to claim 1, characterized in that, Before calculating the second feature similarity between the retrieved semantic vector and the multiple target valid semantic feature vectors in the vector database, the method further includes: The time range requirement and the number of returned results of the retrieval request are analyzed by the preset cross-modal learning model, and the number of returned results is used as the first preset number; Based on the time range requirement, the storage partition corresponding to the retrieval request is determined, and the storage partition is used to represent the partition address of the data stored in the vector database in chronological order; Send the retrieval semantic vector, the storage partition, and the first preset quantity to the query interface of the vector database; The effective semantic feature vectors stored in the storage partition are used as the target effective semantic feature vectors.

6. The method according to any one of claims 1-5, characterized in that, The vector database includes hot data partitions and regular partitions, and the method further includes: Based on the creation timestamp corresponding to each of the effective video frame images, the effective video frame data with a creation duration less than the first preset duration is stored in the hot data partition, and the hot data partition is configured to be resident in memory. The effective video frame data with a creation duration greater than or equal to the first preset duration and less than or equal to the second preset duration is stored in the regular partition; Delete the valid video frame data whose creation duration is longer than the second preset duration, and delete the storage partition corresponding to the valid video frame data whose creation duration is longer than the second preset duration.

7. The method according to any one of claims 1-5, characterized in that, The step of storing all valid vector data in a vector database includes: A unique identifier corresponding to the target valid vector data is created as the primary key, wherein the target valid vector data is any of the valid vector data; Obtain the video stream identifier of the video stream to be processed corresponding to the target valid vector data; The access path string of the effective video frame image corresponding to the target effective vector data is generated by a preset structured path; Based on the effective semantic feature vector, a similarity retrieval index corresponding to the target effective vector data is generated by a preset clustering algorithm; Based on the creation timestamp, the primary key, video stream identifier, access path string, creation timestamp, similarity search index, and target valid vector data corresponding to the target valid vector data are stored in the vector database according to the preset storage partition.

8. A city surveillance video retrieval device, characterized in that, include: The video frame extraction module is used to acquire multi-channel video streams to be processed, and extract frames from the target video streams to be processed through an independent video processing thread to obtain a video frame image sequence. The target video frames to be processed are the video streams to be processed from any channel. One target video stream to be processed corresponds to one independent video processing thread. The video frame image sequence includes multiple video frame images. The feature extraction module is used to extract the semantic feature vectors corresponding to each video frame image in chronological order using a preset cross-modal learning model; The vector deduplication module is used to calculate the first feature similarity between the semantic feature vector of the current video frame image and the semantic feature vector of the previous video frame image, wherein the current video frame image is any of the video frame images; the current video frame images whose first feature similarity is less than a preset similarity threshold are determined as valid video frame images, and the semantic feature vector of the valid video frame image is used as the valid semantic feature vector; The vector database module is used to store all valid vector data in the vector database. The valid vector data includes at least the creation timestamps corresponding to the valid semantic feature vectors and the valid video frame images. The semantic retrieval module is used to obtain the user's retrieval request, extract the retrieval semantic vector of the retrieval request through the preset cross-modal learning model, and calculate the second feature similarity between the retrieval semantic vector and the second feature similarity between the second feature vector and the second feature vectors corresponding to multiple target effective semantic feature vectors in the vector database. A first preset number of valid video frame images corresponding to the second feature similarity are selected from high to low as the retrieval results.

9. An electronic device, characterized in that, The device includes a memory, a processor, a user interface, and a network interface. The memory is used to store instructions. The user interface and the network interface are both used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the method as described in any one of claims 1 to 7.

Images