A real-time video structured information extraction and storage method and system

Abstract

The application provides a real-time video structured information extraction and storage method and system, and the method comprises the following steps: acquiring image frames from a video source, controlling the input image frame frame rate, and counting the frame rate; target detection and target tracking are performed on the input image frames, wherein, during target tracking, the target tracking strategy is dynamically adjusted according to the frame rate; according to the target detection and target tracking results, different types of targets are subjected to structured information extraction respectively, wherein, for the same target in different frames, the corresponding structured information is intermittently extracted; the extracted structured information is cached based on a hash table priority queue, and the structured information meeting the out-of-queue condition is persisted to a database. The application provides a real-time video structured information extraction scheme with better real-time performance and higher efficiency, and to a certain extent, solves the problems of difficult real-time video information extraction and low retrieval efficiency.

Description

Technical Field

[0001] This invention relates to the field of video structuring, and more specifically, to a method and system for real-time video structured information extraction and storage. Background Technology

[0002] With the rapid development of deep learning, neural network-based target detection and multi-target tracking technologies have become increasingly mature and widely used in video image processing. Target detection and multi-target tracking play an irreplaceable role in scenarios such as autonomous driving, border patrols, and smart cities.

[0003] Existing deep learning-based object detection technologies have achieved high accuracy levels. Networks such as YOLO and SSD can achieve detection accuracy exceeding 90% and a frame rate of 25 FPS even in environments with limited hardware, balancing detection quality and efficiency. Furthermore, in multi-object tracking, object detection-based multi-object tracking models are developing rapidly. Combining traditional motion information estimation with appearance feature matching based on convolutional neural networks effectively solves the problem of re-identification after object occlusion or loss. Based on this, real-time extraction of structured information from camera videos is feasible.

[0004] In traditional video surveillance systems, retrieving camera video information requires significant manpower and time, and accuracy cannot be guaranteed. However, after extracting and storing structured information from camera video, the source video or image can be retrieved directly from the database based on the characteristics of the object being searched, using text information—a precise and efficient method. Therefore, the extraction and storage of structured information from camera video is essential. Summary of the Invention

[0005] This invention addresses the technical problem in traditional video surveillance systems where retrieving camera video information requires significant manpower and time, and accuracy cannot be guaranteed. It provides a method and system for real-time video structured information extraction and storage.

[0006] According to a first aspect of the present invention, a method for real-time video structured information extraction and storage is provided, comprising:

[0007] Image frames are acquired from the video source, the frame rate of the input image frames is controlled, and the frame rate is counted.

[0008] Target detection and target tracking are performed on the input image frames, wherein, during target tracking, the target tracking strategy is dynamically adjusted according to the frame screening rate;

[0009] Based on the target detection and target tracking results, structured information is extracted for different types of targets. For the same target in different frames, the corresponding structured information is extracted intermittently.

[0010] A priority queue based on a hash table is used to cache extracted structured information and persist structured information that meets the dequeue conditions to the database.

[0011] According to a second aspect of the present invention, a real-time video structured information extraction and storage system is provided, comprising:

[0012] The statistics module is used to acquire image frames from the video source, control the frame rate of the input image frames, and count the frame rate of the screened frames.

[0013] The detection and tracking module is used to perform target detection and target tracking on the input image frames. During target tracking, the target tracking strategy is dynamically adjusted according to the frame screening rate.

[0014] The extraction module is used to extract structured information for different types of targets based on the target detection and target tracking results. Specifically, for the same target in different frames, the corresponding structured information is extracted intermittently.

[0015] The caching module is used for priority queues based on hash tables to cache extracted structured information and persist structured information that meets the dequeue conditions to the database.

[0016] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the processor is configured to execute a computer management program stored in the memory to implement a method for extracting and storing real-time video structured information.

[0017] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer management program is stored, wherein the computer management program, when executed by a processor, implements the steps of a method for extracting and storing real-time video structured information.

[0018] The present invention provides a method and system for real-time video structured information extraction and storage, the beneficial effects of which are as follows: addressing the shortcomings of existing security monitoring systems in quickly and accurately querying targets, the present invention designs real-time video structured information extraction and storage software, which combines target detection and feature extraction to extract information from targets in the video and store it in a database, transforming unstructured video into structured text information for easy retrieval and utilization. Attached Figure Description

[0019] Figure 1 A flowchart of a real-time video structured information extraction and storage method provided by the present invention;

[0020] Figure 2 This is a schematic diagram of a priority queue based on a hash table.

[0021] Figure 3 This is a schematic diagram illustrating the specific implementation steps of a real-time structured information extraction method based on object detection.

[0022] Figure 4 This is a schematic diagram of the structure of a real-time video structured information extraction and storage system provided by the present invention;

[0023] Figure 5 A schematic diagram of a possible hardware structure of an electronic device provided by the present invention;

[0024] Figure 6 This is a schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. Detailed Implementation

[0025] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0026] Example 1

[0027] A method for real-time video structured information extraction and storage, see [link to relevant documentation]. Figure 1 The method includes:

[0028] S1: Obtain image frames from the video source, control the frame rate of the input image frames, and count the frame rate of the screened frames.

[0029] As an example, controlling the input image frame rate and calculating the frame rate includes: performing adaptive uniform frame filtering on the video source based on the video image preprocessing module, so that the output frame rate meets the following requirements:

[0030] fps output ≤min(fps input fps max );

[0031] Among them, fps output The final output frame rate (fps) of the video image preprocessing module. input The initial frame rate (fps) of the video source. max To meet the maximum frame rate without blocking;

[0032] Statistical frame rate:

[0033]

[0034] Understandably, this involves acquiring video images from a network camera and preprocessing the video. Specifically, the method involves using user-inputted camera IP address, username, password, and other information to read the camera's output video stream via a common RTSP stream or the camera manufacturer's SDK. The video stream is then decoded into RGB image frames and passed to the next module. During this process, the output frame rate needs to be controlled according to the system's processing capacity; if necessary, frames can be dropped evenly to prevent system congestion, and the frame rate α of the output image frames is statistically analyzed.

[0035] Specifically, in this embodiment of the invention, video preprocessing can employ adaptive uniform frame filtering, including: maintaining a queue with a set threshold length for caching image frames; when an image frame obtained from the video source is added to the queue, the queue length is checked; if the queue length is less than the set threshold length, the newly obtained image frame is directly added to the cache queue; if the queue length has reached the set threshold length, the image frame at the head of the queue is first dequeued and discarded, and then the newly obtained image frame is added to the tail of the queue, so that the queue length is always less than the set threshold length, thereby achieving relatively uniform frame dropping processing while ensuring that subsequent modules are not blocked.

[0036] The specific algorithm for preprocessing camera video is as follows:

[0037]

[0038] S2, target detection and target tracking are performed on the input image frames, wherein, during target tracking, the target tracking strategy is dynamically adjusted according to the frame rate.

[0039] It is understood that target detection and target tracking are performed on image frames output from a video source. As an example, the target detection and target tracking of the input image frames includes, during target tracking, dynamically adjusting the target tracking strategy according to the frame filtering rate. This includes: performing target detection on each input image frame to identify the target detection location and target type in the image frame; and performing target tracking based on the identified target detection location and target type in each image frame, using target motion information and feature information. During target tracking, the weights of the target motion information and feature information in the target tracking strategy are dynamically adjusted according to the frame filtering rate.

[0040] The target motion information in the target tracking strategy is as follows:

[0041] d (1) (i, j) = (d j -y j ) T S i -1 (d j -y j );

[0042] Where, d j Let y be the detection position of object j. i S is the predicted position after tracking the i-th target. i This is the covariance matrix between the detected and predicted object locations;

[0043] The target feature information is as follows:

[0044]

[0045] The matching degree of target association is calculated based on the target motion information and feature information as follows:

[0046] c i，j =λd (1) (i,j)+(1-λ)d (2) (i, j);

[0047] Where c i，j This is the final weighted matching information, where λ is the weighting coefficient, which is dynamically adjusted according to the frame rate.

[0048] The weight coefficient λ can be dynamically adjusted based on the frame rate of the image frames. When there are many dropped frames, that is, the frame interval is large and the correlation of positional information between adjacent frames is weak, λ can be reduced to decrease the weight of motion information and increase the weight of feature information, and matching is mainly based on feature information. Conversely, when there are few dropped frames, the correlation of positional information between adjacent frames is strong, λ can be increased to increase the weight of feature information and decrease the weight of motion information, thus balancing the efficiency and accuracy of multi-target re-identification.

[0049] Wherein, the frame rate λ = αW of the image frame is... define -β;

[0050] Among them, W define The weights are predefined initial values, and β is a constant bias. This is further supported by the following formula:

[0051] fps output ≤min(fps input fps max );

[0052] Given that α∈[0,1], theoretically, we have:

[0053] λ∈[0, W define -β];

[0054] In boundary cases, when there are many dropped frames, the correlation of positional information between adjacent image frames is insufficient. Therefore, λ = 0 can be set, at which point c... i，j It is entirely determined by feature information.

[0055] The matching method (i.e., target tracking strategy) during target tracking can be dynamically adjusted according to the input frame rate. The specific algorithm is as follows:

[0056]

[0057] S3. Based on the target detection and target tracking results, structured information is extracted for different types of targets. For the same target in different frames, the corresponding structured information is extracted intermittently.

[0058] Understandably, after target detection and tracking are performed on the real-time input image frames according to the above method, in the target tracking process, the same ID is assigned to the same target in different image frames. That is, after target tracking, each target has its own ID. Then, the structured information of each type of target is extracted. For example, pedestrian structured information can be defined as age, gender, and clothing color, while vehicle structured information can be defined as vehicle type, body color, and license plate information. For specific information, please refer to [link / reference needed]. Figure 2 As shown, the specific algorithm flow is as follows:

[0059]

[0060] When extracting structured information for each type of target, an intermittent extraction method is used. Specifically, based on the assigned target ID, a target with the same ID is only output once per unit time by the target detection and multi-target tracking module; that is, structured information is extracted only once per unit time for the same target. Compared to a scheme that extracts structured information for all targets in every frame, this significantly reduces the input to the structured information extraction module, thus reducing its workload. Furthermore, the scheme allowing for secondary detection after a set unit time ensures that new structured information for the target can be captured promptly when it is updated, preventing the loss of information in the video. The unit time can be set based on the scene and camera resolution, meaning that for the same target in consecutive image frames, a set unit time can be set, during which the target's structured information is extracted only once.

[0061] S4 is a priority queue based on a hash table that caches extracted structured information and persists structured information that meets the dequeue conditions to the database.

[0062] It is understandable that a hash table-based priority queue is used to store the structured information of each extracted target.

[0063] As an example, the hash table-based priority queue, which caches extracted structured information and persists structured information that meets the dequeue conditions to the database, includes: maintaining a hash table-based priority queue, establishing a hash table with the target ID as the key and the combination of the target ID's structured information and the last update time as the value; constructing a priority queue for key-value pairs in the hash table with the last update time of the structured information as the weight; setting a time threshold for key-value pairs in the priority queue, and dequeuing the corresponding key-value pair when the time difference between the last update time and the current time is greater than the set time threshold, thus persisting the structured information to the database.

[0064] For a detailed structural diagram of a hash table, please refer to [link / reference]. Figure 2 A hash table is constructed using the identified target IDs as "keys." Using the identified target IDs as "keys" ensures the uniqueness and uniformity of the keys, avoiding hash collisions. Using a hash table as the storage structure ensures that the time complexity of querying and modifying structured information data is constant. Furthermore, the hash table should record the time of the last update of the data; therefore, the time difference is defined as:

[0065] Δ t =T current -T update ;

[0066] Among them, T current T represents the current time. update The time of the last update to this data. A priority queue is created using this time difference as the weight, i.e., Δ. t Larger data is placed at the front of the queue. The priority queue is updated every unit of time by adjusting the Δ value based on a set time difference threshold. t Data exceeding this threshold is dequeued and output to a relational database. A schematic diagram of this structure is shown below. Figure 3 As shown. Specifically, within each unit of time, the hash table can be traversed, and timed-out key-value pairs can be output to a persistent queue and deleted from the hash table to simulate a priority queue. The unit of time can be set according to specific factors such as database performance. The specific algorithm flow is as follows:

[0067]

[0068] Example 2

[0069] A real-time video structured information extraction and storage system, see [link to relevant documentation]. Figure 4 The system includes a statistics module 401, a detection and tracking module 402, an extraction module 403, and a caching module 404, wherein:

[0070] The statistics module 401 is used to acquire image frames from the video source, control the frame rate of the input image frames, and count the frame rate. The detection and tracking module 402 is used to perform target detection and target tracking on the input image frames, wherein, during target tracking, the target tracking strategy is dynamically adjusted according to the frame rate. The extraction module 403 is used to extract structured information for different types of targets based on the target detection and target tracking results, wherein, for the same target in different frames, the corresponding structured information is extracted intermittently. The caching module 404 is used to cache the extracted structured information based on a priority queue of a hash table, and persist the structured information that meets the dequeue conditions to the database.

[0071] It is understood that the real-time video structured information extraction and storage system provided by the present invention corresponds to the real-time video structured information extraction and storage method provided in the foregoing embodiments. The relevant technical features of the real-time video structured information extraction and storage system can be referred to the relevant technical features of the real-time video structured information extraction and storage method, and will not be repeated here.

[0072] Example 3

[0073] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 5 As shown, an embodiment of the present invention provides an electronic device 500, including a memory 510, a processor 520, and a computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 511, it implements the steps of the real-time video structured information extraction and storage method of Embodiment 1.

[0074] Example 4

[0075] Please see Figure 6 , Figure 6 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 6 As shown, this embodiment provides a computer-readable storage medium 600, on which a computer program 611 is stored. When the computer program 611 is executed by a processor, it implements the steps of the real-time video structured information extraction and storage method of Embodiment 1.

[0076] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0077] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0079] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0080] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0081] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0082] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for real-time video structured information extraction and storage, characterized in that, include: Image frames are acquired from the video source, the frame rate of the input image frames is controlled, and the frame rate is counted. Target detection and target tracking are performed on the input image frames, wherein, during target tracking, the target tracking strategy is dynamically adjusted according to the frame screening rate; Based on the target detection and target tracking results, structured information is extracted for different types of targets. For the same target in different frames, the corresponding structured information is extracted intermittently. A priority queue based on a hash table is used to cache the extracted structured information and persist the structured information that meets the dequeue conditions to the database. The control of the input image frame rate and the statistical analysis of the screening frame rate include: The video image preprocessing module performs adaptive uniform frame filtering on the video source to ensure that the output frame rate meets the following requirements: ； in, The final output frame rate of the video image preprocessing module. The initial frame rate of the video source. To meet the maximum frame rate without blocking; Statistical frame rate: ； The step of performing target detection and target tracking on the input image frames, wherein, during target tracking, the target tracking strategy is dynamically adjusted according to the frame rate, including: Perform target detection on each input image frame to identify the target detection location and target type in the image frame; Based on the target detection location and target type in each identified image frame, target tracking is performed based on target motion information and feature information. During target tracking, the weights of target motion information and feature information in the target tracking strategy are dynamically adjusted according to the frame rate.

2. The method according to claim 1, characterized in that, The adaptive uniform frame filtering process of the video source based on the video image preprocessing module includes: Maintain a queue with a set threshold length to cache image frames; When an image frame acquired from a video source is added to the queue, the queue length is checked. If the queue length is less than the set threshold length, the newly acquired image frame is directly added to the buffer queue. If the queue length has reached the set threshold length, the image frame at the head of the queue is first dequeued and discarded, and then the newly acquired image frame is added to the tail of the queue, so that the queue length is always less than the set threshold length.

3. The method according to claim 1, characterized in that, The method of dynamically adjusting the weights of target motion information and feature information in the target tracking strategy based on the frame rate during target tracking includes: The target motion information in the target tracking strategy is: ； in, For the tracked object The detection location in the previous frame image. To predict the location of the i-th target in the next frame of the image, Detecting the position of an object With predicted location The covariance matrix between them; The target feature information is as follows: ； in, R is the 128-dimensional feature vector calculated by the target detection model for the j-th tracked target in the previous frame image. i This is the set of 128-dimensional feature vectors of the i-th target computed in the next frame image. for The k-th eigenvector in the 128-dimensional eigenvector matrix; The matching degree of the target association is then: ； in This is the final weighted matching information. This is a weighting coefficient that is dynamically adjusted based on the frame rate.

4. The method according to claim 3, characterized in that, ； in, For predefined initial weight values, For constant bias, To filter frame rate; 。 5. The method according to claim 1, characterized in that, The step of intermittently extracting the corresponding structured information for the same target in different frames includes: For the same target in consecutive image frames, a unit time is set, and the target's structured information is extracted only once within the set unit time.

6. The method according to claim 1, characterized in that, The hash table-based priority queue caches extracted structured information and persists structured information that meets the dequeue conditions to the database, including: Maintain a priority queue based on a hash table, using the target ID as the key and the combination of the target ID's structured information and the last update time as the value; For key-value pairs in the hash table, a priority queue is constructed using the last update time of the structured information as the weight. For key-value pairs in the priority queue, a time threshold is set. When the time difference between the last update time and the current time is greater than the set time threshold, the corresponding key-value pair is dequeued and the structured information is persisted to the database.

7. A real-time video structured information extraction and storage system, characterized in that, include: The statistics module is used to acquire image frames from the video source, control the frame rate of the input image frames, and count the frame rate of the screened frames. The detection and tracking module is used to perform target detection and target tracking on the input image frames. During target tracking, the target tracking strategy is dynamically adjusted according to the frame screening rate. The extraction module is used to extract structured information for different types of targets based on the target detection and target tracking results. Specifically, for the same target in different frames, the corresponding structured information is extracted intermittently. The caching module is used for a priority queue based on a hash table to cache the extracted structured information and persist the structured information that meets the dequeue conditions to the database. The control of the input image frame rate and the statistical analysis of the screening frame rate include: The video image preprocessing module performs adaptive uniform frame filtering on the video source to ensure that the output frame rate meets the following requirements: ； in, The final output frame rate of the video image preprocessing module. The initial frame rate of the video source. To meet the maximum frame rate without blocking; Statistical frame rate: ； The step of performing target detection and target tracking on the input image frames, wherein, during target tracking, the target tracking strategy is dynamically adjusted according to the frame rate, including: Perform target detection on each input image frame to identify the target detection location and target type in the image frame; Based on the target detection location and target type in each identified image frame, target tracking is performed based on target motion information and feature information. During target tracking, the weights of target motion information and feature information in the target tracking strategy are dynamically adjusted according to the frame rate.

8. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the steps of the real-time video structured information extraction and storage method as described in any one of claims 1-6.

Images