Video storage management method and system for target-oriented tracking query

Abstract

The application discloses a video storage management method and system for target tracking query, relates to the technical field of multimedia information storage and query, and can accelerate the retrieval of pixels, reduce the number of pixels processed by a target recognition neural network, and thus relieve the problems of long query time and high energy consumption through adaptive video blocking and semantic indexing technology. An original video is taken to find a region of interest in each frame; the region of interest is divided into a blocking region, any region of interest falls into and only falls into one blocking region, and the original video is divided into a blocking video by using blocking region information; the blocking video is encoded, a semantic index is established, the blocking video frame and the semantic index are stored correspondingly, and an association relationship between the semantic index and a label is constructed; when querying, a query type label is received, the semantic index is determined, video frames meeting the query type are screened as query result frames, the query result frames are decoded and output, target recognition and tracking operations are performed, and a target tracking trajectory result is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multimedia information storage and query, and particularly relates to a video storage management method and system for target tracking query. BACKGROUND

[0002] With the decline in the cost of cameras, the number of monitoring cameras, dash cams and other camera equipment is rapidly increasing. On the other hand, thanks to the rapid development of deep learning, many high-precision image and video processing algorithms have emerged in recent years, making it possible to use machines to analyze potential semantic information in videos and perform query operations. Among them, target tracking query is one of the most complex mass video query tasks, and the current mainstream method is to track based on target recognition results. However, the target recognition method often works on expensive GPU devices, and the calculation speed on large-scale video data is not optimistic. Therefore, there is an urgent need for a query algorithm that can be efficiently and low-energy-consumption executed on mass video data.

[0003] The existing research on mass video target tracking query direction takes the longest time for video decoding and target recognition process. Since the time consumption of video decoding and neural network calculation is positively correlated with the number of pixels processed, when the number of pixels increases, the system processing time increases linearly. However, there may be a large number of regions in the video that are not related to the query, and decoding and processing the pixels in these regions is one of the main problems that leads to low efficiency and high energy consumption of the target tracking query system.

[0004] The inference speed of mainstream neural networks is often positively correlated with the total number of input pixels, i.e. the product of the width, height and frame number of the video. Existing technologies often reduce the number of calls to the target recognition network, thereby accelerating the query within a given error limit. From another perspective, reducing the input video frame resolution, i.e. width and height, can also speed up the processing speed of the entire query system. In other words, by excluding pixels that do not contain the target from the input of the target recognition network, the throughput of the system can be increased.

[0005] The HEVC encoder proposes the concept of video blocking, which introduces intra-frame random access points for video decoding by dividing a video segment into non-overlapping rectangles. Thanks to homomorphic stitching, the HEVC encoder and decoder can restore the blocks to complete video frames without re-encoding. Therefore, encoding the area where the target is located as a block can effectively reduce the overhead of video decoding and neural network inference.

[0006] Therefore, for the storage management of mass video targets, the application of HEVC encoder and mainstream neural networks has the problems of slow query speed and large resource overhead. SUMMARY

[0007] Therefore, the application provides a video storage management method and system for target tracking query, which can accelerate the retrieval of pixels, reduce the number of pixels processed by a target recognition neural network, and thus relieve the problems of long query time and high energy consumption through adaptive video partitioning and semantic indexing technology.

[0008] To achieve the above object, the technical scheme of the application comprises the following steps:

[0009] Taking an original video as input, finding a possible area of a target in each frame as a candidate area, collecting all candidate areas in the frames, and fusing the intersecting candidate areas to obtain a region of interest;

[0010] Partitioning the obtained region of interest to divide the region into block regions so that any region of interest falls into and only falls into one block region, dividing the original video into block videos using the block region information, and recording the block region information and whether the region of interest information is contained to jointly constitute block metadata.

[0011] Encoding the block videos, establishing semantic indexes for each frame in the block videos, storing each frame in the block videos and the semantic indexes correspondingly, and constructing an association relationship between the semantic indexes and labels.

[0012] When a query event occurs, receiving a query type label input by a user, determining a semantic index corresponding to the query type label, screening block video frames conforming to the query type as query result frames, decoding the query result frames and outputting the decoded query result frames.

[0013] Performing a target recognition operation on the output decoded query result frame data, and performing target tracking using the target recognition result to output a target tracking trajectory result.

[0014] Further, taking an original video as input, finding a possible area of a target in each frame as a candidate area, collecting all candidate areas in the frames, and fusing the intersecting candidate areas to obtain a region of interest, specifically:

[0015] Separating foreground pixels and background pixels in the original video frames using foreground detection; if there is no foreground pixel in a frame, it means that there is no candidate area in the current frame; otherwise, the foreground pixels in the same frame generate a plurality of connected domains, and the circumscribed rectangle of each connected domain is called a candidate area of the current frame;

[0016] The original video is composed of a series of continuous frames, and an I frame independently encoded divides the original video into a plurality of segments in time sequence; for each segment, the candidate areas in all frames are fused according to spatial distance, that is, the intersecting candidate areas are merged into a region of interest.

[0017] Further, the segmented video is encoded, and a semantic index is established for each frame in the segmented video, each frame in the segmented video and the semantic index are stored correspondingly, and an association relationship between the semantic index and the label is constructed; specifically:

[0018] The frame in the segmented video is input into a pre-trained neural network, the frame in the segmented video is a segmented video frame, the output layer feature of the pre-trained neural network is taken as the semantic index, each frame in the segmented video and the semantic index are stored correspondingly, and a subset is selected from the semantic index set by using a farthest point sampling algorithm, each semantic index in the subset is recorded as a representative vector; the segmented video frame corresponding to the representative vector is input into the target recognition network, and the output label is the associated label of each representative vector, so as to construct the association relationship between the semantic index and the label.

[0019] Further, when the query event occurs, the query type label input by the user is received, and the semantic index corresponding to the query type label is determined; the segmented video frame meeting the query type is screened as a query result frame, and the query result frame is decoded and output, specifically:

[0020] When the query event occurs, the video frame number and the current tracking target type label required by the user are received;

[0021] When the query event occurs, the query type specified by the user is given in the form of a label, which is recorded as a query label; the segmented block information of the segmented video frame corresponding to the video frame number is taken out from the storage area, and the segmented block region not containing the region of interest is filtered out;

[0022] The segmented video frame not containing the current tracking target type is filtered out from the segmented video frame containing the region of interest, and the cosine similarity of the semantic index corresponding to the remaining segmented video frame and all representative vectors associated with the query label is calculated; if the similarity of a semantic index and any representative vector is higher than a threshold value, it is indicated that the semantic index is related to the current query type, and the segmented video frame corresponding to the semantic index is output as a query result frame.

[0023] Another embodiment of the application further provides a video storage management system for target tracking query, which comprises an interested region acquisition module, a segmented encoding and decoding module, a semantic index management module, a segmented retrieval module and a target tracking module.

[0024] The region of interest acquisition module takes the original video as input, finds the region where the target may exist in each frame as a candidate region, collects the candidate regions in all frames, and fuses the intersecting candidate regions to obtain the region of interest; the module uses the obtained region of interest to divide the region into blocks, so that any region of interest falls into and only falls into one block region, and uses the block region information to divide the original video to obtain a block video;

[0025] The block coding module encodes the block video from the region of interest acquisition module and sends it to the storage region for storage;

[0026] The semantic index module is used to establish a semantic index for each frame in all block videos in the storage region, store each frame in the block video and the semantic index, and build an association relationship between the semantic index and the label;

[0027] The target tracking module gradually sends the video frame number and the current tracking target type label required for the query to the block retrieval module when the query occurs;

[0028] The block retrieval module is used to receive the video frame number and the current tracking target type label required for the query from the target tracking module when the query occurs; the block retrieval module actively takes out the block metadata of the block video frame corresponding to the video frame number from the storage region, filters the block regions that do not contain the region of interest, and then calls the semantic index management module to filter the block video frames that do not contain the current tracking target type from the remaining block regions; then the decoding result of the query result frame is obtained from the block coding module and returned to the target tracking module;

[0029] The semantic index module is also used to be called by the block retrieval module when the query occurs, receive the current tracking target type label, filter the block video frames that do not contain the current tracking target type from the block video frames containing the region of interest by using the association relationship between the semantic index and the label, output the block video frames related to the current tracking target type, and record them as query result frames. The query result frames are sent to the block coding module;

[0030] The block coding module decodes the query result frames from the semantic index module and sends them to the block retrieval module;

[0031] The target tracking module uses the decoded query result frames to perform target recognition operation, and uses the result of target recognition to perform target tracking, and outputs the target tracking trajectory result.

[0032] Further, the block coding module uses the HEVC protocol for encoding and decoding;

[0033] The encoding process converts each block video into a code stream file and saves it on the disk;

[0034] The decoding process reads the specified query result frame corresponding code stream file from the disk and decodes it into frame data.

[0035] Further, the region of interest acquisition module separates the foreground pixels and the background pixels in the original video frame by foreground detection; if there is no foreground pixel in a frame, it means that there is no candidate region in the current frame; otherwise, the foreground pixels in the same frame generate a plurality of connected domains, and the circumscribed rectangle of each connected domain is called a candidate region of the current frame.

[0036] The original video is composed of a series of continuous frames, and the independent coded frame (I frame) divides the original video in time sequence into a plurality of segments. For each segment, the candidate regions in all frames are fused according to spatial distance, that is, the intersecting candidate regions are merged into a region of interest.

[0037] The region of interest acquisition module is also responsible for dividing the entire video frame into a plurality of non-overlapping block regions, ensuring that any region of interest falls into and only falls into one block region. The video is divided using the block region information to obtain a block video, wherein the block region information and the information whether the region of interest is contained jointly constitute the block meta information.

[0038] Further, the semantic index module uses a pre-trained neural network, and the frames in the block video are block video frames. The output layer features of the pre-trained neural network are used as semantic indexes.

[0039] For the set of all semantic indexes corresponding to all block video frames, a subset is selected from the semantic index set by using the farthest point sampling algorithm, and each semantic index in the subset is recorded as a representative vector. By inputting the block video frame corresponding to the representative vector into the target recognition network, the label output is the associated label of each representative vector.

[0040] When a query occurs, the query type specified by the user is given in the form of a label, denoted as a query label. The semantic index module is called to filter the block video frames that do not contain the current tracking target type from the block video frames containing the region of interest. The cosine similarity between the semantic indexes corresponding to the remaining block video frames and all representative vectors associated with the query label is calculated. If the similarity between a semantic index and any one representative vector is higher than a threshold, it means that the semantic index is related to the current query type, and the block video frame corresponding to the semantic index is output as a query result frame.

[0041] Further, the target tracking module mainly includes a target recognition unit and a target tracking unit.

[0042] A target recognition unit is configured to receive a target tracking request of a user, the target tracking request comprising a video frame number of a required query and a current tracking target type label; the target recognition unit sends the target tracking request to a block retrieval module and acquires a decoded query result frame from the block retrieval module.

[0043] A target tracking unit is configured to match objects and calculate trajectories based on the decoded query result frame, internally reserve all target information of a previous frame, and determine trajectory information of a tracking target by matching the target with a target of a current frame.

[0044] Beneficial effects:

[0045] 1: The application provides a video storage management method and system for target tracking query, which adopts adaptive video block technology for block division of a video, mainly acquires a region of interest in the video through a foreground segmentation algorithm, fuses regions of interest close in interframe and intracframe distances, divides the frequency band into multiple block regions and ensures that each region of interest falls into a unique block region, and then independently encodes each block using an HEVC encoder. Layout information and block information where the region of interest is located are recorded through a database. The application adopts semantic indexing technology for video block division, removes an output layer of a common lightweight picture classification neural network, and the output feature vector of the neural network represents semantic information retained in the image to a certain extent and has a much higher operation speed than a target recognition model. The relationship between feature vectors can be strengthened through a small amount of labeled samples and a triple loss. On the other hand, the semantic indexing is clustered through a small amount of labeled samples, and block video frames with similar semantic information are associated. The semantic indexing can further filter out video blocks irrelevant to a query target during target tracking query, thereby reducing the number of pixels decoded and input into the target recognition model and accelerating the target tracking query speed. Based on the above principle, the application can accelerate the retrieval of relevant pixels, reduce the number of pixels processed by the target recognition neural network, and thereby relieve the problems of long query time and high energy consumption.

[0046] 2: The application provides a video storage management method and system for target tracking query, which adopts a video block retrieval technology based on semantic indexing. Given a tracking target type selected by a target tracking algorithm, all video blocks not containing a region of interest are first filtered out, and block video frames not meeting the conditions are then excluded according to semantic indexing clustering information. That is, the block video frames actually decoded and input into the target recognition model need to contain objects of the given tracking target type. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a schematic diagram of the overall architecture of the application;

[0048] Figure 2is a target tracking query process schematic diagram of the present application;

[0049] Figure 3 is a target tracking query flow chart of the present application;

[0050] Figure 4 is a video input, chunking and index creation flow chart of the present application. DETAILED DESCRIPTION

[0051] The present application is described in detail below with reference to the accompanying drawings and embodiments.

[0052] Embodiment 1:

[0053] The embodiment of the present application provides a video storage management method for target tracking query, comprising the following steps:

[0054] Step 1: Take the original video as input, find the possible target area in each frame as a candidate area, collect all the candidate areas in the frame, and fuse the intersecting candidate areas to obtain the region of interest. Specifically:

[0055] The foreground pixels and background pixels in the original video frame are separated by foreground detection; if there is no foreground pixel in a frame, it means that there is no candidate area in the current frame; otherwise, the foreground pixels in the same frame produce several connected domains, and the circumscribed rectangle of each connected domain is called a candidate area of the current frame;

[0056] The original video is composed of a series of continuous frames, and the independent coded frame (I frame) divides the original video into several segments in time sequence. For each segment, the candidate areas in all frames are fused according to the spatial distance, that is, the intersecting candidate areas are merged into the region of interest.

[0057] Step 2: Use the obtained region of interest to divide the block region, so that any region of interest falls into and only falls into one block region, use the block region information to divide the original video to obtain the chunked video, and record the block region information and whether the region of interest information is contained to jointly constitute the chunked metadata information.

[0058] Step 3: Encode the chunked video, and establish a semantic index for each frame in the chunked video, store each frame in the chunked video and the semantic index correspondingly, and build the association relationship between the semantic index and the label.

[0059] Specifically:

[0060] A pre-trained neural network is used, the frame in the chunked video is the chunked video frame, the chunked video frame is used as the input of the pre-trained neural network, the output layer feature of the pre-trained neural network is used as the semantic index, and each frame in the chunked video and the semantic index are stored correspondingly.

[0061] For the set of all semantic indexes corresponding to all sub-blocked video frames, a farthest point sampling algorithm is used to select a subset from the semantic index set, and each semantic index in the subset is denoted as a representative vector; by sending the sub-blocked video frame corresponding to the representative vector into the target recognition network, the output label is the associated label of each representative vector.

[0062] Thus, the association between the semantic index and the label is established.

[0063] Step 4: When the query event occurs, the query type label input by the user is received, and the semantic index corresponding to the query type label is determined; the sub-blocked video frames meeting the query type are filtered as query result frames, and the query result frames are decoded and output. Specifically:

[0064] When the query occurs, the query type specified by the user is given in the form of a label, denoted as a query label; the sub-blocked video frame corresponding to the video frame number is taken out from the storage area, and the sub-blocked region not containing the region of interest is filtered.

[0065] The sub-blocked video frame corresponding to the sub-blocked region containing the region of interest is taken, and the cosine similarity of the corresponding semantic index with all representative vectors associated with the query label is calculated; if the similarity of a semantic index with any representative vector is higher than a threshold, it means that the semantic index is related to the current query type, and the sub-blocked video frame corresponding to the semantic index is output as the query result frame.

[0066] Step 5: Perform target recognition operation on the decoded query result frame data output, and use the target recognition result to perform target tracking, and output the target tracking trajectory result.

[0067] Embodiment 2:

[0068] The embodiment provides a video storage management system for target tracking query, characterized in that it comprises a region of interest acquisition module, a sub-blocked encoding and decoding module, a semantic index management module, a sub-blocked retrieval module, and a target tracking module. Figure 1 It is a schematic diagram of the overall architecture of the massive video storage system for target tracking query of the present application. The region of interest acquisition module, the sub-blocked encoding and decoding module, the semantic index management module, and the sub-blocked retrieval module constitute a video sub-blocked management subsystem; the target tracking module is located in a target tracking subsystem.

[0069] Among them, the video sub-blocked management subsystem is used to process newly ingested videos, calculate the region of interest containing the foreground, establish the sub-blocked layout according to the region of interest, establish the semantic index for each frame of each sub-block, encode each sub-block into an independent video sequence, and selectively decode the sub-blocked frame according to the object type requested by the target tracking system;

[0070] The target tracking subsystem exposes an interface to the user to implement target tracking query service based on target recognition. The target recognition system will pass the target recognition type to the video chunk management system, and the video chunk management system will return the relevant chunk frames for target recognition and target tracking according to the target type.

[0071] The region of interest acquisition module takes the original video as input, finds the region where the target may exist in each frame as a candidate region, collects all the candidate regions in the frames, and fuses the intersecting candidate regions to obtain the region of interest; the module uses the obtained region of interest for division of the divided block region, so that any region of interest falls into and only falls into one divided block region, and uses the divided block region information to divide the original video to obtain the divided block video.

[0072] Specifically: the region of interest acquisition module separates the foreground pixels and background pixels in the original video frame using foreground detection; if there is no foreground pixel in a frame, it means that there is no candidate region in the current frame; otherwise, the foreground pixels in the same frame produce several connected domains, and the circumscribed rectangle of each connected domain is called a candidate region of the current frame; the original video is composed of a series of consecutive frames, and the independent coded frame (I frame) divides the original video in time sequence into several segments. For each segment, the candidate regions in all frames are fused according to the spatial distance, that is, the intersecting candidate regions are merged into the region of interest; the region of interest acquisition module is also responsible for dividing the entire video frame into a plurality of non-overlapping divided block regions, ensuring that any region of interest falls into and only falls into one divided block region. The divided block region information is used to divide the video to obtain the divided block video, wherein the divided block region information and the information whether the region of interest is contained jointly constitute the divided block meta information.

[0073] The divided block coding module encodes the divided block video from the region of interest acquisition module and then sends it to the storage area for storage. The divided block coding module uses the HEVC protocol for encoding and decoding; the encoding process converts each divided block video into a code stream file and saves it on the disk; the decoding process reads the code stream file corresponding to the specified query result frame from the disk and decodes it into frame data.

[0074] The semantic index module is used to establish a semantic index for each frame in all divided block videos in the storage area, associate each frame in the divided block video with the semantic index for storage, and build an association relationship between the semantic index and the label;

[0075] The target tracking module gradually sends the video frame number and the current tracking target type label of the required query to the divided block retrieval module when the query occurs.

[0076] The chunk retrieval module is configured to receive the video frame number and the current tracking target type label of the required query from the target tracking module when the query occurs; the chunk retrieval module actively takes out the chunk meta information of the chunk video frame corresponding to the video frame number from the storage area, filters the chunk regions not containing the region of interest, and then calls the semantic index management module to filter the chunk video frames not containing the current tracking target type from the remaining chunk regions; and then sends the decoding result of the query result frame obtained from the chunk coding and decoding module to the target tracking module.

[0077] The semantic index module is further configured to be called by the chunk retrieval module when the query occurs, receive the current tracking target type label, filter the chunk video frames not containing the current tracking target type from the chunk video frames containing the region of interest by using the association relationship between the semantic index and the label, output the chunk video frames related to the current tracking target type, and record the chunk video frames as query result frames, and send the query result frames to the chunk coding and decoding module.

[0078] Specifically, the semantic index module uses a pre-trained neural network, the frames in the chunk video are chunk video frames, uses the chunk video frames as the input of the pre-trained neural network, and uses the output layer features of the pre-trained neural network as the semantic index; for a set composed of all semantic indexes corresponding to all chunk video frames, a subset is selected from the semantic index set by using a farthest point sampling algorithm, and each semantic index in the subset is recorded as a representative vector; by sending the chunk video frame corresponding to the representative vector to the target recognition network, the output label is the associated label of each representative vector.

[0079] When the query occurs, the query type specified by the user is given in the form of a label, recorded as a query label, the semantic index module is called to filter the chunk video frames not containing the current tracking target type from the chunk video frames containing the region of interest, the semantic indexes corresponding to the remaining chunk video frames, and the semantic indexes corresponding to the chunk video frames are respectively calculated for the cosine similarity with all representative vectors associated with the query label; if the similarity of a semantic index with any one representative vector is higher than a threshold, it means that the semantic index is related to the current query type, and the chunk video frame corresponding to the semantic index is output as a query result frame.

[0080] The chunk coding and decoding module sends the query result frame from the semantic index module to the chunk retrieval module after decoding operation;

[0081] The target tracking module performs target recognition operation by using the decoded query result frame, and performs target tracking by using the result of target recognition, and outputs the target tracking trajectory result.

[0082] Specifically, the target tracking module mainly includes two parts of a target identification unit and a target tracking unit; the target identification unit is configured to receive a target tracking request of a user, the target tracking request containing a video frame number required to be queried and a current tracking target type label; the target identification unit sends the target tracking request to a block retrieval module and acquires a decoded query result frame from the block retrieval module;

[0083] The target tracking unit is configured to match objects and calculate a trajectory on the basis of the decoded query result frame, internally reserve all target information of a previous frame, match the target of a current frame to determine trajectory information of the tracking target.

[0084] 1) The target identification unit is mainly responsible for receiving a target tracking request of a user, asking a video block management system for a video frame, performing a target identification operation on a returned video frame, and sending a result of the target identification to a target tracking module. Among them, the unit selects appropriate video frames in a time range specified by the user for a target tracking task, continuously sends corresponding frame numbers and target type labels to the video block management system, and acquires all block frames related to the target type in the frame. The unit realizes identification of the target through a neural network model, and returns a bounding box, a category label and a confidence.

[0085] 2) The target tracking unit is mainly responsible for matching objects and calculating a trajectory on the basis of a result returned by the target identification module. The unit tries to find an optimal match of two groups of targets on results of adjacent two times of target identification. When the target identification is completely finished, the unit returns a final trajectory result to the user and ends the entire target tracking process.

[0086] Embodiment 3:

[0087] This embodiment introduces a massive video storage process based on video block and semantic indexing of the application, and further details of the application are described in combination with the drawings and embodiments, wherein Figure 3 is a target tracking query flowchart of the application, Figure 4 is a video input, block and index creation flowchart of the application.

[0088] The target tracking query process includes:

[0089] Step S11: A user initiates a target tracking query request and specifies a query object type, and a plurality of query object type labels are set in this embodiment.

[0090] Step S12: After the target tracking system receives the related request, a video frame is selected to perform a target identification operation. The target identification module sends a frame number and a target type label (converted from the query object type label) to a video block management system;

[0091] Step S13: The video chunk management system searches the video segment where the number is located and queries all chunk meta information in the segment one by one. For any chunk, if it does not contain the region of interest, it is skipped and the next chunk is checked, otherwise it goes to step S14.

[0092] Step S14: The chunk retrieval module requests the semantic index management module for the semantic index of the frame number corresponding to the chunk, and calculates the cosine similarity with all the representative vectors related to the label. If the similarity with all the related representative vectors is lower than the threshold, it means that the semantic information contained in the chunk frame is irrelevant to the related category, the chunk will be skipped, the next chunk is taken back to step S13, otherwise it goes to step S15.

[0093] Step S15: The chunk retrieval module requests the chunk codec module for the chunk frame, and the chunk codec module reads the video stream file corresponding to the chunk and decodes it into a frame image and returns it to the chunk retrieval module.

[0094] Step S16: The target recognition module receives the chunk frame returned by the chunk retrieval module, calls the target recognition neural network to recognize all targets on the chunk frame and filters the targets related to the current query type.

[0095] Step S17: At this time, the target position obtained by the target recognition module is a relative position relative to the original picture, so it is necessary to restore the target coordinates to the absolute position through the chunk position information provided by the chunk retrieval module.

[0096] Step S18: When all related chunk frames are processed, the target information of the current frame is transmitted to the target tracking module; the target tracking module retains all target information of the last frame inside, and determines the trajectory information of the related object by matching with the targets of the current frame.

[0097] The video input, chunking and index creation process includes:

[0098] Step S21: The user inputs a new video into the system.

[0099] Step S22: The region of interest acquisition module extracts foreground pixels using a foreground detection algorithm, and the circumscribed rectangle of each connected domain of foreground pixels is called a region of interest.

[0100] Step S23: The region of interest acquisition module fuses all regions of interest in a video segment according to their distances until any two regions of interest cannot be further fused.

[0101] Step S24: The region of interest acquisition module generates an N x M chunk layout, thereby ensuring that any region of interest falls into and only falls into a unique chunk.

[0102] Step S25: cutting the video frames according to the block layout, and sequentially sending all the video frames in the block containing the region of interest into the embedded network of the semantic index management module to generate a low-dimensional semantic index vector. The semantic indexes of a small number of labeled samples are clustered according to the vector distance to generate a representative vector, and the labeled information is bound to the representative vector.

[0103] Step S26: the block coding and decoding module encodes all the block main ideas into independent video stream files and saves them in the disk.

[0104] The above introduction of the target tracking query process and the video input, block and index creation process explains in detail the support of the present application to the mass video storage oriented to the target tracking query, and proves that the present application proposes a practical and effective solution in optimizing the large-scale video data target tracking query.

[0105] To sum up, the above is only a preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for video storage management for target-oriented tracking queries, characterized in that, The method comprises the following steps: Taking an original video as input, finding a possible area of a target in each frame as a candidate area, collecting all candidate areas in all frames, and fusing intersecting candidate areas to obtain a region of interest; Using the obtained region of interest to divide a block region, so that any region of interest falls into and only falls into one block region, using block region information to divide the original video to obtain a block video, and recording block region information and whether the region of interest information is contained to jointly constitute block metadata information; Encoding the block video, establishing a semantic index for each frame in the block video, storing each frame in the block video and the semantic index correspondingly, and constructing an association relationship between the semantic index and a label; When a query event occurs, receiving a query type label input by a user, determining a semantic index corresponding to the query type label, screening block video frames conforming to the query type as query result frames, decoding the query result frames, and outputting the decoded query result frames; Performing target recognition on the output decoded query result frame data, and using a result of target recognition to perform target tracking, and outputting a target tracking trajectory result.

2. The video storage management method for object-oriented tracking query according to claim 1, wherein, The taking an original video as input, finding a possible area of a target in each frame as a candidate area, collecting all candidate areas in all frames, and fusing intersecting candidate areas to obtain a region of interest is specifically: Separating foreground pixels and background pixels in the original video frame by using foreground detection; if there is no foreground pixel in a frame, it indicates that there is no candidate area in the current frame; otherwise, the foreground pixels in the same frame generate a plurality of connected domains, and the circumscribed rectangle of each connected domain is called a candidate area of the current frame; The original video is composed of a series of continuous frames, and an independent coded frame, namely an I frame, divides the original video in time sequence into a plurality of segments, and for candidate areas in all frames of each segment, the candidate areas are fused according to spatial distance, that is, intersecting candidate areas are merged into a region of interest.

3. The video storage management method for object-oriented tracking query according to claim 1 or 2, characterized in that, Encoding the block video, establishing a semantic index for each frame in the block video, storing each frame in the block video and the semantic index correspondingly, and constructing an association relationship between the semantic index and a label; Specifically: A pre-trained neural network is used, the frames in the block video are block video frames, the block video frames are used as inputs of the pre-trained neural network, features of an output layer of the pre-trained neural network are used as semantic indexes, each frame in the block video and the semantic index are stored correspondingly, a furthest point sampling algorithm is used to select a subset from a set of all semantic indexes corresponding to all block video frames, each semantic index in the subset is recorded as a representative vector, the block video frame corresponding to the representative vector is input into a target recognition network, and a label output by the target recognition network is the associated label of each representative vector, thereby constructing an association relationship between the semantic index and the label.

4. The method of claim 3, wherein, The receiving a query type label input by a user when a query event occurs, determining a semantic index corresponding to the query type label, screening block video frames conforming to the query type as query result frames, decoding the query result frames, and outputting the decoded query result frames is specifically: When the query event occurs, the video frame number and the current tracking target type label of the required query of the user input are received; When the query event occurs, the query type specified by the user is given in the form of a label, denoted as a query label; the storage area actively takes out the block metadata of the block video frame corresponding to the video frame number, and filters the block regions that do not contain the region of interest; The block video frames that do not contain the current tracking target type are filtered from the block video frames containing the region of interest, and the semantic indexes corresponding to the remaining block video frames are used to calculate the cosine similarity with all representative vectors associated with the query label, respectively; if the similarity of a semantic index with any one representative vector is higher than a threshold, it means that the semantic index is related to the current query type, and the block video frame corresponding to the semantic index is output as a query result frame.

5. A video storage management system oriented to target tracking queries, characterized in that, The system comprises an interested region acquisition module, a block encoding and decoding module, a semantic index management module, a block retrieval module, and a target tracking module; The interested region acquisition module takes an original video as input, finds the region where the target may exist in each frame as a candidate region, collects all the candidate regions in the frames, and fuses the intersecting candidate regions to obtain the interested region; the module uses the obtained interested region to divide the block regions, so that any interested region falls into and only falls into one block region, and divides the original video using the block region information to obtain block videos; The block encoding and decoding module encodes the block videos from the interested region acquisition module and sends them to the storage area for storage; The semantic index module is used to establish a semantic index for each frame in all block videos in the storage area, store each frame in the block video and the semantic index in association, and construct the association relationship between the semantic index and the label; The target tracking module gradually sends the video frame number and the current tracking target type label of the required query to the block retrieval module when the query occurs; The block retrieval module is used to receive the video frame number and the current tracking target type label of the required query from the target tracking module when the query occurs; The block retrieval module actively takes out the block metadata of the block video frame corresponding to the video frame number from the storage area, filters the block regions that do not contain the interested region, and then calls the semantic index management module to filter the block video frames that do not contain the current tracking target type from the remaining block regions; then, the block encoding and decoding module obtains the decoding result of the query result frame and returns it to the target tracking module; The semantic index module is also used to be called by the block retrieval module when the query occurs, receive the current tracking target type label, filter the block video frames that do not contain the current tracking target type from the block video frames containing the interested region, output the block video frames related to the current tracking target type, denoted as query result frames, and send the query result frames to the block encoding and decoding module; The block encoding and decoding module decodes the query result frames from the semantic index module and sends them to the block retrieval module. The target tracking module performs target recognition operation by using the decoded query result frame, and performs target tracking by using the target recognition result, and outputs a target tracking trajectory result.

6. The object-oriented tracking query-enabled video storage management system of claim 5, wherein, The block coding and decoding module uses the HEVC protocol for encoding and decoding; The encoding process converts each block video into a code stream file and saves it on the disk; The decoding process reads the code stream file corresponding to the specified query result frame from the disk and decodes it into frame data.

7. The object-oriented tracking query oriented video storage management system according to claim 5 or 6, characterized by, The region of interest acquisition module separates the foreground pixels and the background pixels in the original video frame by using foreground detection; If there is no foreground pixel in a frame, it means that there is no candidate region in the current frame; otherwise, the foreground pixels in the same frame generate a plurality of connected domains, and the circumscribed rectangle of each connected domain is called a candidate region of the current frame; The original video is composed of a series of continuous frames, and the independent coded frame (I frame) divides the original video in time sequence into a plurality of segments, and for each segment, the candidate regions in all frames are fused according to spatial distance, that is, the intersecting candidate regions are merged into a region of interest; The region of interest acquisition module is also responsible for dividing the entire video frame into a plurality of non-overlapping block regions, ensuring that any region of interest falls into and only falls into one block region, and dividing the video by using the block region information to obtain a block video, wherein the block region information and the information whether the region of interest is contained jointly constitute block metadata information.

8. The object-oriented tracking query-enabled video storage management system of claim 7, wherein, The semantic index module uses a pre-trained neural network, and the frames in the block video are block video frames, taking the block video frames as the input of the pre-trained neural network, and taking the output layer features of the pre-trained neural network as semantic indexes; For a set composed of all semantic indexes corresponding to all block video frames, a subset is selected from the semantic index set by using the farthest point sampling algorithm, and each semantic index in the subset is recorded as a representative vector; by sending the block video frame corresponding to the representative vector into the target recognition network, the output label is the associated label of each representative vector; When a query occurs, the query type specified by the user is given in the form of a label, denoted as a query label, the semantic index module is called, and the block video frames not containing the type of the current tracking target are filtered from the block video frames containing the region of interest, and the cosine similarity of the semantic indexes corresponding to the remaining block video frames and all representative vectors associated with the query label is calculated respectively; if the similarity of a semantic index and any one representative vector is higher than a threshold, it means that the semantic index is related to the current query type, and the block video frame corresponding to the semantic index is output as a query result frame.

9. The object-oriented tracking query-enabled video storage management system of claim 8, wherein, The target tracking module mainly includes a target recognition unit and a target tracking unit; The target recognition unit is configured to receive a target tracking request of a user, wherein the target tracking request comprises a video frame number to be queried and a current tracking target type label; The target recognition unit sends the target tracking request to the block retrieval module and acquires a decoded query result frame from the block retrieval module; The target tracking unit is configured to match objects and calculate trajectories based on the decoded query result frame, retain all target information of the previous frame internally, match the target of the current frame to determine the trajectory information of the tracked target.

Images