Frame-based video retrieval method, device, system and medium

Abstract

The present invention relates to a frame-based video retrieval method, device, system and medium, and belongs to the technical field of network operation and maintenance audit and video data retrieval. The method in the invention includes the steps of: setting an index table, using each record therein to store lens information; collecting the video stream displayed on the screen of the network terminal, and logically dividing the video stream into lenses, creating a corresponding record in the index table for each lens, and storing the starting frame of each lens, the storage size of the starting frame and other information in its corresponding record; obtaining the retrieval key frame input by the user and collecting the value of its storage size; between the starting frame of each lens and the user key frame, first compare whether their storage sizes are equal and then compare whether the two frames are consistent, and finally return the lens retrieval result. The present invention is suitable for video data retrieval in technical fields such as bastion machine operation and maintenance audit, especially in the application scenario of fast video clip retrieval in the case of massive video data.

Description

Technical Field

[0001] The present invention relates to video data processing, and in particular to a frame-based video retrieval method, device, system and medium, belonging to the technical field of network operation and maintenance auditing and video data retrieval. Background Art

[0002] Video retrieval is to find the video clips needed by users from a large amount of video data. It is widely used in event monitoring as evidence for the identification of a certain behavior of an actor in relevant social fields such as justice, security inspection, and auditing. It is obviously an inefficient practice to review the entire video recording of the event. Traditional video retrieval methods usually perform manual searches in sequence by fast forwarding or rewinding during the playback of the video. The operation is cumbersome, which not only consumes a lot of manpower, but also easily misses the video clips that are really needed, and usually cannot meet the retrieval requirements of large-capacity video databases. In practice, people are often very eager for a content-based video retrieval method, that is, to be able to search based on the content features of the video clips in the entire video file. However, as a retrieval condition, the human language expression of the retrieval keyword faced by this content feature often makes the problem solution more complicated because it involves the NLP (Natural Language Processing) process.

[0003] In content-based video retrieval, a technical solution that can be easily thought of by technicians in the relevant technical field is to divide the entire video into several video segments and assign certain text content features to each video segment, and then perform appropriate keyword retrieval based on these text content features. Compared with traditional manual video search, this content-based video retrieval method will inevitably improve efficiency, but there are still some problems: since video segments are relatively independent video retrieval units, after the video is divided into several video segments, it is usually necessary to extract features from each video segment in order to establish index data for each video segment, such as extracting features such as color, texture, motion, and even high-level semantics of the lens to form a feature space that describes the video segment. However, the difficulty of the problem lies in that the features extracted here are not something that users can simply describe and input as parameters of retrieval conditions for video retrieval. That is, due to the lack of semantic information in the video features, users feel inconvenient when expressing the retrieval conditions for video data.

[0004] In the patent document previously applied by the present applicant, with the authorization announcement number CN117633297B and the invention name "Annotation-based video retrieval method, device, system and medium", a video retrieval method is provided: when the input device of the network terminal moves each time, the video stream displayed on the screen of the network terminal and the character input and instruction operation of the input device are triggered to be collected, and the multiple continuous shots constituting the entire video are automatically segmented, and the related video clips are quickly retrieved based on the annotation content of the character input and instruction operation. Although the patent realizes the automatic segmentation and content annotation of the video clip, there are still some shortcomings: the instruction operation from the input device of the network terminal needs to be encoded accordingly, and it is difficult to realize the keyword retrieval of the instruction operation encoding without semantic definition or description. In addition, the character input of the input device of the network terminal, as the object of keyword retrieval, may be difficult for the user to predict and can only rely on subjective guessing. Due to the inaccuracy of the guessed retrieval keyword, it is easy to cause the retrieval clip that actually meets the user's retrieval requirements to be difficult to retrieve in the end. In addition, the patent cannot realize the retrieval of video clips involving specific frames. Summary of the invention

[0005] The technical problem that the present invention attempts to solve is, in a network system, when investigating the source of a certain network operation event, how to confirm the identity of the network terminal at the source of the operation and obtain necessary evidence by means of video retrieval.

[0006] When investigating the source of a certain network operation event, the prior art does not use a method of performing frame retrieval on the video stream displayed on the network terminal screen to confirm the network terminal that is the source of the operation. In fact, the retrieval of specific frames is an important clue and breakthrough in the investigation of network operation and maintenance audit events, but it is often overlooked by technical personnel in this field. Part of the reason is that the retrieval key frame cannot be determined in advance, and only vague subjective descriptions or assumptions can be made. Fortunately, in network systems such as bastion hosts, there are some convenient factors that can solve the above technical problems.

[0007] The inventor found that when searching for a video clip by comparing the user search key frame with the representative frame in the video clip, for a video clip with continuous changes between frames, it is difficult to achieve the same goal between the sampling of its representative frame and the sampling of the user search key frame, which easily leads to the error of retrieval failure when the video clip actually containing the user search key frame is searched due to frame mismatch. In this case, since the boundaries between the frames displayed by the software interfaces in the video stream are clear when the network terminal is running the network software, and the pictures in the video stream are mostly static, that is, the frames often do not change. The user search key frame and the representative frame in the video clip are not only convenient to sample, but they are both based on the static screen display, and there is no deviation in the sampling of the two when comparing, thereby avoiding the error of missed detection. Therefore, the above-mentioned technical problem can be easily solved by using this frame search method.

[0008] In video retrieval based on specific frames, after a video with a large time domain is divided into several video segments with relatively independent content, it is necessary to further consider how to quickly locate a specific video segment containing a specific frame, especially when collecting video streams on the display screen of a network terminal, how to implement the retrieval process of video segments by frame retrieval. The purpose of solving this technical problem is to provide a more convenient and easy-to-implement problem-solving solution in the technical field of network operation and maintenance auditing and video data retrieval, and to apply it to post-monitoring of network terminal operations in network systems such as bastion hosts.

[0009] In order to solve the above-mentioned technical problems, the present invention provides a frame-based video retrieval method, device, system and medium, and adopts the following technical solutions.

[0010] In a first aspect, the present invention provides a frame-based video retrieval method for finding a shot containing a given frame from the entire video, the method comprising:

[0011] Step S100: setting an index table, wherein each record in the index table stores data information corresponding to a shot constituting the video, and each record includes four data items, namely, first, second, third and fourth data items;

[0012] Step S200: collecting a video stream forwarded by a network server and originating from a network terminal screen display, and in the process of collecting the video stream, logically dividing the video stream into a plurality of shots according to shot changes and storing them in a memory, creating a corresponding record in the index table in step S100 for each shot generated by the logical segmentation, storing a shot number containing the identification information of the network terminal in a first data item of the record, storing a corresponding pointer value of the shot generated by the logical segmentation in a second data item of the record, storing a starting frame in the shot generated by the logical segmentation in a third data item of the record, and collecting a storage size value of the starting frame and storing the storage size value in a fourth data item of the record;

[0013] Step S300: obtaining a search key frame input by a user, and collecting a storage size value of the search key frame, wherein the search key frame originates from a static interface display in a software system running on the network terminal, and the static interface display is related to the network terminal operation that causes the network event to occur;

[0014] Step S400: first perform a numerical comparison between the storage size of the retrieval key frame described in step S300 and the fourth data item in each record created in step S200, and then perform a picture comparison between the third data item and the retrieval key frame in each record with equal numerical comparison. If the picture comparison is consistent, the video retrieval is successful and the first and second data items in the record with consistent picture comparison are returned to the user. If the numerical comparison is not equal or the picture comparison is inconsistent, the information that the video retrieval failed is returned to the user.

[0015] Preferably, the acquisition process of the storage size value of the starting frame described in step S200 and the acquisition process of the storage size value of the retrieval key frame described in step S300 are implemented by storing the starting frame or the retrieval key frame as a computer image file, and the storage size value of the starting frame or the storage size value of the retrieval key frame is taken as the storage size of the computer image file in the computer memory, wherein the file format of the computer image file stored in the starting frame is required to be the same as that of the computer image file stored in the retrieval key frame.

[0016] Preferably, the file format of the computer image file is PNG or GIF.

[0017] Preferably, the corresponding pointer value of the logical segmentation generated shot in step S200 is the storage start address of the logical segmentation generated shot in the computer memory.

[0018] Preferably, the corresponding pointer value of the logical segmentation generation shot in step S200 is the time code of the starting frame in the logical segmentation generation shot in the video.

[0019] Preferably, the step S400 further includes a sub-step of playing the video clip: after all the record entries meeting the search criteria are retrieved, the video clips indicated by the second data items in all the record entries are played.

[0020] In a second aspect, the present invention provides a frame-based video retrieval device for finding a shot containing a given frame from the entire video, the device comprising:

[0021] Module M100 is used to: set an index table, each record in the index table stores data information of a shot of the video, and each record includes four data items, namely, first, second, third and fourth;

[0022] The module M200 is used to: collect a video stream forwarded by a network server and originating from a network terminal screen display, and in the process of collecting the video stream, logically divide the video stream into a plurality of shots according to shot changes and store them in a memory, and for each shot generated by the logical segmentation, create a corresponding record in the index table in step S100, and store the shot number containing the identity information of the network terminal in the first data item of the record, store the corresponding pointer value of the shot generated by the logical segmentation in the second data item of the record, store the starting frame of the shot generated by the logical segmentation in the third data item of the record, and collect the storage size value of the starting frame and store the storage size value in the fourth data item of the record;

[0023] Module M300 is used to: obtain a search key frame input by a user and collect a storage size value of the search key frame, wherein the search key frame originates from a static interface display in a software system running on the network terminal, and the static interface display is related to the network terminal operation that causes the network event to occur;

[0024] Module M400 is used to: first perform a numerical comparison on the storage size of the retrieval key frame in step S300 with the fourth data item in each record created in step S200, and then perform a picture comparison on the third data item in each record with equal numerical comparison with the retrieval key frame; if the picture comparison is consistent, the video retrieval is successful and the first and second data items in the record with consistent picture comparison are returned to the user; if the numerical comparison is not equal or the picture comparison is inconsistent, information on video retrieval failure is returned to the user.

[0025] Preferably, the module M400 further includes a submodule for playing video clips, which is used to: after all record entries meeting the search condition are retrieved, play the video clips respectively indicated by the second data items in all the record entries.

[0026] In a third aspect, the present invention provides a computer system comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the processor executes the computer program, the video retrieval method as described in the first aspect of the present invention is implemented.

[0027] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the video retrieval method as described in the first aspect of the present invention is implemented.

[0028] The beneficial effects produced by the technical solution of the present invention are as follows.

[0029] The technical solution of the present invention provides a method for retrieving video clips. In a network system with a bastion host as a typical application scenario, the video stream displayed on the screen of a network terminal when running a software system is used as a data source, and the specific frame input by the user is used as a retrieval target. By retrieving and locating the source of the video clip containing the specific frame, the network terminal of the operation source is identified and necessary video clips are obtained as relevant evidence, so as to finally achieve the purpose of the present invention. Compared with the prior art, the technical solution of the present invention not only overcomes the disadvantages of the traditional video retrieval method based on feature matching, which is inconvenient to objectively define the content of the retrieval key frame and can only be subjectively vaguely recognized, resulting in inconvenient video retrieval or inaccurate retrieval, but also quickly eliminates video clips that do not meet the retrieval requirements through the first comparison of the storage size of the lens start frame and the user retrieval key frame, thereby improving the overall retrieval efficiency, and ensuring the accuracy of the video retrieval results through the secondary comparison of the lens start frame and the user retrieval key frame, that is, while providing a problem-solving method for network software operation accident investigation through network terminal video acquisition and retrieval, it also achieves the efficiency and accuracy of the video retrieval process. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 : A flowchart of the steps of the method of the present invention;

[0031] Figure 2 : A schematic diagram showing the composition of the four data items in a single record in the index table and the data they store;

[0032] Figure 3 : A schematic block diagram of an embodiment of the method of the present invention;

[0033] Figure 4: Schematic diagram of the storage structure of the index table in the embodiment of the method of the present invention;

[0034] Figure 5 : A schematic diagram of the screen display when the command is input;

[0035] Figure 6 : A schematic diagram of the screen display during form input;

[0036] Figure 7 : Flowchart of the two comparison processes and condition judgment in lens retrieval;

[0037] Figure 8 : Schematic diagram of the working mechanism of the bastion host in the prior art;

[0038] Fig. 9 : A schematic diagram of an embodiment of the present invention applied in the bastion host field. DETAILED DESCRIPTION

[0039] In order to more clearly illustrate the features of the technical solution of the present invention, the present invention is further elaborated in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0040] The video retrieval involved in the present invention refers to finding the video clips required by users from a large amount of video data. It can be applied to fields such as network event monitoring. As an auxiliary technical means for accurately obtaining evidence of certain operation behaviors performed by relevant operators in network terminals, the main task of the present invention is to find the video clip where the specified frame is located from the entire video; the video clip referred to in this specification is also called a shot in some technical documents. It is the basic unit of video data. Video processing usually first divides the entire video into several video clips as basic index units. This is the basis for realizing shot retrieval; the frame referred to in this specification refers to a still image, which is a component of the video. The smallest unit, a shot is a video composed of a series of frames; the network terminal referred to in this specification should at least have output devices such as a display and input devices such as a keyboard and a mouse, but it is not limited to whether it has independent CPU, memory, hard disk, motherboard and other devices, nor is it limited to the specific performance parameters of these devices. Although in the technical field to which the present invention belongs, the network terminal is sometimes regarded as a computer without a CPU, memory, hard disk and motherboard, in the bastion host audit system as a typical technical application field of the present invention, the distribution form of the network terminal can be specifically manifested as administrators, developers, operation and maintenance personnel, outsourced personnel, registered users, visitors and many other identities.

[0041] For those skilled in the art, the specific meanings of the relevant terms in the present invention can be understood according to specific circumstances.

[0042] As a first embodiment, the present invention provides a frame-based video retrieval method for finding a shot containing a given frame from the entire video, the method comprising:

[0043] Step S100: setting an index table, wherein each record in the index table stores data information corresponding to a shot constituting the video, and each record includes four data items, namely, first, second, third and fourth data items;

[0044] Step S200: collecting a video stream forwarded by a network server and originating from a network terminal screen display, and in the process of collecting the video stream, logically dividing the video stream into a plurality of shots according to shot changes and storing them in a memory, creating a corresponding record in the index table in step S100 for each shot generated by the logical segmentation, storing a shot number containing the identification information of the network terminal in a first data item of the record, storing a corresponding pointer value of the shot generated by the logical segmentation in a second data item of the record, storing a starting frame in the shot generated by the logical segmentation in a third data item of the record, and collecting a storage size value of the starting frame and storing the storage size value in a fourth data item of the record;

[0045] Step S300: obtaining a search key frame input by a user, and collecting a storage size value of the search key frame, wherein the search key frame originates from a static interface display in a software system running on the network terminal, and the static interface display is related to the network terminal operation that causes the network event to occur;

[0046] Step S400: first perform a numerical comparison between the storage size of the retrieval key frame described in step S300 and the fourth data item in each record created in step S200, and then perform a picture comparison between the third data item and the retrieval key frame in each record with equal numerical comparison. If the picture comparison is consistent, the video retrieval is successful and the first and second data items in the record with consistent picture comparison are returned to the user. If the numerical comparison is not equal or the picture comparison is inconsistent, the information that the video retrieval failed is returned to the user.

[0047] The execution steps in the above method embodiment are: Figure 1 To facilitate the overall understanding of the technical solution of the present invention, the four steps in the embodiment of the method can be summarized as follows: format definition of video data storage; video data collection, structured processing and storage; obtaining the search key frame input by the user and collecting its storage size value; returning the search result of the video clip to the user.

[0048] The order of the written records between and within the steps in the above method embodiment is only for the convenience of expression and understanding, but it does not limit the scope of protection required by the technical solution of the present invention. For example, in step S200, there is no restriction on the storage order of the four data items in a record; there is no necessary order limitation between step S200 and step S300; for the execution of other specific operation steps, there are still other transformation forms different from the current specific embodiment under the premise that the technical problem of the present invention can be solved and the technical effect of the present invention can be achieved.

[0049] In step S100 of the above method embodiment, the storage structure of each record stored in the index table is as follows: Figure 2 As shown, the data items in each record are composed of: the first data item is used to save the shot number, and the shot number carries the information of the network terminal identity; the second data item is used to save the shot pointer; the third data item is used to save the starting frame in the shot; the fourth data item is used to save the storage size value of the starting frame in the shot.

[0050] In the above method embodiment, the first to fourth data items in a record in the index table are named for the convenience of description only. They are not distinguished from each other by the difference in the serial number itself, but by the difference in their data storage. The change of the arrangement order of the four data items does not deviate from the idea of ​​the present invention.

[0051] The first data item in a record in the index table in the above method embodiment is used to store the shot number, and the shot number carries the identity information of the network terminal. The identity is used to trace relevant personnel in network operation events. Those skilled in the art should know that the storage of the identity can be implemented flexibly. For example, the identity information can be saved by defining a fifth data item in a record.

[0052] The execution steps in the above method embodiment can also be combined with Figure 3 To simplify the application scenario of the technical problem, it is assumed that there are only two network terminals A and B that need screen video acquisition, and the video acquired by each network terminal consists of three shots (video clips), so six shots are finally generated, which may be numbered as A1, A2, A3, B1, B2 and B3. The relevant information of these shots is stored in six records in the index table, such as Figure 4 shown.

[0053] In step S200 of the above method embodiment, the video stream displayed on the screen of the same network terminal needs to be divided into several shots, i.e., video clips. If the video retrieval function based on shots is to be realized, the video must first be divided into several video clips and used as basic retrieval units. In the prior art, the video is usually divided into several video clips by using the shot boundary detection method. The basis is often that when the shot is switched, a series of changes will occur in the video data, which are mainly manifested in the sudden increase of color difference, the distance between the new and old edges, the change of the object shape, and the discontinuity of motion. At present, the shot boundary detection is usually performed by calculating the frame difference. Generally speaking, the difference between the frames in the same shot (video clip) is small, while the difference between the frames of different shots is large. In the present invention, since the collected video stream originates from the screen display of a network terminal when a certain software system is running, and since the interface display of the software in different running states is completely different, with the operation of the personnel of the network terminal, the boundaries of the shots are clear when switching, and the display between the frames in the same shot is relatively stable, which is different from the traditional film and television; in addition, the starting frame in the shot originates from the moment after the software interface is switched, and does not involve the command operation or form input operation of the personnel, and the retrieval key frame originates from a static display state in the software running interface, so there is usually no error between the starting frame in the collected shot and the retrieval key frame used by the user.

[0054] It should also be noted that in step S200, the so-called logical segmentation does not necessarily require that the shots in the complete video stream displayed on the network terminal screen be physically stored separately. In fact, it is also allowed that they are stored consecutively in the memory.

[0055] The starting frame in each shot is predictable because for a network-operated software platform, its interface display is usually stable and limited in number, and can even be determined during the UI (User Interface) design stage of software development. Figure 5 A schematic diagram of inputting commands by clicking the mouse or touching the screen is given. Figure 6 A schematic diagram of inputting character text through a keyboard or other means is given. They are all common scenarios for network terminal screen operation and display, and they can be used to represent the starting frame in the shot as well as the retrieval key frame. On the other hand, the retrieval key frame specified and input by the customer for video retrieval comparison is also a specific picture, not an abstract expression of content. Before video retrieval, the user needs to provide a retrieval key frame as the target, which can be achieved by extracting a frame related to video retrieval as the retrieval key frame from all interface displays of the software.

[0056] In step S300 of the above method embodiment, the search keyframe given by the user is similar to the search keyword used in text search. Each frame in the video collected by the present invention originates from a fixed UI screen in software development, and the keyframe used by the user for search also comes from the UI screen, and the UI screen can be accurately obtained through the software operation process and combined with the problem reasoning of the occurrence of network events. In short, the specific application scenario provides sufficient convenience for the implementation of the technical solution of the present invention. Figure 5 or Figure 6 The frames shown are key frames for user video retrieval and can be easily obtained by users. For example, a relatively simple implementation method is to simulate the operation scene or traverse the route and status of the software operation, and capture the screen display in a specific scene.

[0057] When a network event occurs and an investigation into the source of the operation is required, the user presumes that the network event originated from Figure 5 or Figure 6 When a command operation or form input operation is performed on a network terminal under the software system interface display state, the Figure 5 or Figure 6 is collected as a key frame for user retrieval, and the Figure 5 or Figure 6 It can be collected through all network terminals.

[0058] In step S400 of the above method embodiment, the retrieval of related shots that meet the matching conditions is involved. Figure 7 A flowchart of the two comparison processes and condition judgment in shot retrieval is given. The storage size value of the user-retrieved key frame is first compared with the storage size value of the starting frame stored in the fourth data item of each record in the index table, and preliminary rapid screening is achieved for each shot that meets the retrieval conditions; the user-retrieved key frame is secondarily compared with the first frame in the shot stored in the third data item of each record in the index table, that is, the precise comparison of the two pictures is used to finally confirm whether the shot retrieval is successful.

[0059] From the root of the problem, the reason why manual retrieval of the original captured video is inefficient is not only that the original generated video data is usually an unstructured two-dimensional image stream sequence, and this unstructured image stream is difficult to retrieve and process before it is structured, but also that the user retrieval key frame is usually not used as an objectively existing data, so it is impossible to realize the software search of the frame, and it can only be searched by manual visual judgment. In order to overcome the above-mentioned disadvantages of low efficiency of manual video retrieval, in the problem-solving ideas of the present invention, firstly, the user retrieval key frame used for video retrieval is objectively described as a specific picture, which creates an objective basis for searching the frame in the entire video, rather than a subjective fuzzy content cognition; secondly, through the first comparison of the frame storage size, a large number of unmatched shots are quickly eliminated, and for the shots with matching frame storage sizes, the retrieval key frame is compared to realize the accurate search of the target shot.

[0060] According to the above analysis ideas, the technical means to solve the technical problem of the present invention and realize the technical solution of the present invention mainly take into account the following two points.

[0061] First of all, the process of comparing two pictures and judging whether they are consistent is much more complicated than the process of judging whether two values ​​are equal. In terms of the amount of data read, the comparison factors of the data amount, the hash operation of the picture data, the feature matching and many other aspects, the former not only consumes more computer computing resources, but also ultimately affects the efficiency of video retrieval. The present invention attempts to convert the complex task of judging whether two pictures are consistent into a simple task of judging whether certain parameters of the two pictures are equal, so as to achieve the early and rapid elimination of a large number of failed retrieval shots.

[0062] Secondly, the frame storage size is used as the feature description parameter of the starting frame of each shot in the video and the retrieval key frame used by the user. Any shot whose starting frame feature description parameters are not equal to the user's retrieval key frame will definitely not meet the user's retrieval conditions; and for the shots whose starting frame feature description parameters are equal to the user's retrieval key frame feature description parameters, they are only shots that may meet the retrieval conditions. At this time, accurate frame comparison is required to draw the final conclusion of the video retrieval.

[0063] In summary, the technical features of the technical solution of the present invention constitute the most critical technical means: objective definition of user retrieval key frames, associating the starting frame in the shot with the shot itself, converting the problem of whether the two frames are consistent into the problem of whether the storage size values ​​of the two frames are equal, etc.

[0064] The technical solution of the present invention is particularly suitable for the retrieval of massive video clips during post-supervision in scenarios such as bastion host operation and maintenance audit in network security management, branch teller operation record inspection in financial institution management, and network user operation monitoring. A common feature of these technical application fields is that the video acquisition object originates from the screen display of the remote terminal, and the user retrieval key frames are objective and accessible.

[0065] As a preferred embodiment, the acquisition process of the storage size value of the starting frame described in step S200 and the storage size value of the retrieval key frame described in step S300 is implemented by storing the starting frame or the retrieval key frame as a computer image file, and the storage size value of the starting frame or the storage size value of the retrieval key frame is taken as the storage size of the computer image file in the computer memory, wherein the file format of the computer image file stored in the starting frame is required to be the same as that of the computer image file stored in the retrieval key frame.

[0066] In this preferred embodiment, when the shot start frame or the retrieval key frame is stored as a computer file, in order to facilitate the numerical comparison between the two in the subsequent step S400, it is obvious that they must use the same storage format of computer image files, for example JPEG, PNG, GIF, BMP, TIFF, PSD, etc. Otherwise, different file formats are meaningless for judging whether two frames are the same by storage size.

[0067] As a preferred embodiment, the file format of the computer picture file is PNG or GIF.

[0068] In this preferred embodiment, when the starting frame in the shot or the user retrieval key frame is stored as a computer file respectively, there are many options for the unified file format, but considering the storage space and storage efficiency, the PNG format or the GIF format is obviously a better choice. For example, the PNG format supports transparent background of the picture, which will be particularly suitable for storing software interfaces whose background color is not very meaningful. The GIF format also supports transparent background of the picture and greatly reduces the storage size of the picture due to the small number of colors it supports. After all, the quality of the picture of the starting frame of the shot or the user retrieval key frame on which the storage size is based does not affect the result of the first comparison in the present invention. For this reason, although the formats such as JPEG, BMP, TIFF, PSD have the advantages of high picture preservation quality and support for layers, they will occupy a large storage space and affect the time required for storage.

[0069] As a preferred embodiment, the corresponding pointer value of the logical segmentation generated shot in step S200 is the storage start address of the logical segmentation generated shot in the computer memory.

[0070] In the preferred embodiment, the value of the lens pointer, i.e., the starting address of the lens when it is stored in the computer memory, is usually automatically assigned by the computer operating system in software programming implementation, and the storage location of the lens can be directly located through the starting address. As a part of the video master data, further operations such as playing and copying can be performed on the lens to realize the user output of the video retrieval result in the present invention.

[0071] As a preferred embodiment, the corresponding pointer value of the logical segmentation generation shot in step S200 is the time code of the starting frame in the logical segmentation generation shot in the video.

[0072] In this preferred embodiment, the value of the lens pointer, that is, the time code of the starting frame of the lens in the video, is used in the prior art to mark the time point of a specific frame in the video. Using this time point, the lens can also be directly positioned, and its playback, copying and other operations can be performed independently.

[0073] As a preferred embodiment, the step S400 further includes a sub-step of playing the video clip: after all the record entries meeting the search criteria are retrieved, the video clips indicated by the second data items in all the record entries are played.

[0074] In this preferred embodiment, by playing the video clips that are the search results that meet the search conditions, the user is given a more intuitive display of the search results, which will facilitate the user to further verify whether the shots that are the video search results really belong to the required video clips. The second data item in all the record entries that meet the search conditions mentioned here stores the shot pointers of the video clips that meet the search requirements.

[0075] The following is an example of a specific application of the method of the present invention in the field of bastion host technology.

[0076] In enterprises and organizations, in order to protect networks and data from intrusion and destruction from external and internal users, improve network security, and protect important information assets of organizations, during network deployment, technical personnel often use the bastion host as the entrance to monitor and record the operations of operation and maintenance personnel on servers, network devices, security devices, databases and other devices in the network, so as to centrally issue alarms, handle them in a timely manner, and audit and determine responsibilities. Figure 8The current working mechanism of the bastion host is schematically given: the operators of the operation and maintenance management area first connect to the bastion host during the operation process, and then submit an operation request to the bastion host. After the request passes the permission check of the bastion host, the application proxy module of the bastion host will connect to a target device in the operation and maintenance asset set on behalf of the user to complete the operation. After that, the target device returns the operation result to the bastion host, and finally the bastion host returns the operation result to the operator of the operation and maintenance management. In this way, the bastion host logically separates the operator of the operation and maintenance management from the target device to establish a management mode of operator → bastion host user account → authorization → target device account → target device, thereby solving the problem of operation permission control and behavior auditing, and also solving the problem that encryption protocols and graphic protocols cannot be audited through protocol restoration. As an operation and maintenance security audit system, the bastion host often needs to record the user's operation behavior throughout the process in order to realize the monitoring and auditing of the operator's operation process, and when necessary, record the operator's screen display for subsequent restoration of the operation and maintenance scene, so as to determine the responsibility of the relevant operators, but how to efficiently retrieve the massive amount of recorded videos constitutes a new technical task.

[0077] In the prior art, although the video stream of the network terminal display screen is collected as a remote evidence collection method, it can only be retrieved by manual visual inefficiency. Fig. 9 As shown in the figure, after the video database derived from the screen display of the network terminal is formed, the video retrieval method of first fast screening and then accurate comparison can be used to quickly retrieve the shots containing the user's search key frames, thereby improving the video retrieval efficiency as a whole. Especially when the total amount of video data is large, the improvement of the video clip retrieval efficiency will be more realistic.

[0078] As a second embodiment, the present invention provides a frame-based video retrieval device for finding a shot containing a given frame from the entire video, the device comprising:

[0079] Module M100 is used to: set an index table, each record in the index table stores data information of a shot of the video, and each record includes four data items, namely, first, second, third and fourth;

[0080] The module M200 is used to: collect a video stream forwarded by a network server and originating from a network terminal screen display, and in the process of collecting the video stream, logically divide the video stream into a plurality of shots according to shot changes and store them in a memory, and for each shot generated by the logical segmentation, create a corresponding record in the index table in step S100, and store the shot number containing the identity information of the network terminal in the first data item of the record, store the corresponding pointer value of the shot generated by the logical segmentation in the second data item of the record, store the starting frame of the shot generated by the logical segmentation in the third data item of the record, and collect the storage size value of the starting frame and store the storage size value in the fourth data item of the record;

[0081] Module M300 is used to: obtain a search key frame input by a user and collect a storage size value of the search key frame, wherein the search key frame originates from a static interface display in a software system running on the network terminal, and the static interface display is related to the network terminal operation that causes the network event to occur;

[0082] Module M400 is used to: first perform a numerical comparison on the storage size of the retrieval key frame in step S300 with the fourth data item in each record created in step S200, and then perform a picture comparison on the third data item in each record with equal numerical comparison with the retrieval key frame; if the picture comparison is consistent, the video retrieval is successful and the first and second data items in the record with consistent picture comparison are returned to the user; if the numerical comparison is not equal or the picture comparison is inconsistent, information on video retrieval failure is returned to the user.

[0083] As a preferred embodiment, the module M400 further includes a submodule for playing video clips, which is used to: after all record entries that meet the search conditions are retrieved, play the video clips indicated by the second data items in all the record entries.

[0084] As a third embodiment, the present invention provides a computer system, including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the processor executes the computer program, the video retrieval method as described in the first embodiment of the present invention is implemented.

[0085] As a fourth embodiment, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the video retrieval method as described in the first embodiment of the present invention is implemented.

[0086] Finally, it should be noted that although the present invention has been exemplarily described through specific implementation methods, it does not constitute a limitation on the scope of patent protection of the present invention. Technical personnel in the relevant technical field should understand that various equivalent substitutions and optimization improvements can still be made to the specific embodiments of the present invention, and any substitutions and improvements that do not deviate from the spirit of the present invention should be covered within the scope of patent protection of the present invention.

Claims

1. A frame-based video retrieval method, which finds shots containing a given frame from the entire video, characterized in that: The method comprises: Step S100: setting an index table, wherein each record in the index table stores data information corresponding to a shot constituting the video, and each record includes four data items, namely, first, second, third and fourth data items; Step S200: collecting a video stream forwarded by a network server and originating from a network terminal screen display, and in the process of collecting the video stream, logically dividing the video stream into a plurality of shots according to shot changes and storing them in a memory, creating a corresponding record in the index table in step S100 for each shot generated by the logical segmentation, storing a shot number containing the identification information of the network terminal in a first data item of the record, storing a corresponding pointer value of the shot generated by the logical segmentation in a second data item of the record, storing a starting frame in the shot generated by the logical segmentation in a third data item of the record, and collecting a storage size value of the starting frame and storing the storage size value in a fourth data item of the record; Step S300: obtaining a search key frame input by a user, and collecting a storage size value of the search key frame, wherein the search key frame originates from a static interface display in a software system running on the network terminal, and the static interface display is related to the network terminal operation that causes the network event to occur; Step S400: first perform a numerical comparison between the storage size of the retrieval key frame described in step S300 and the fourth data item in each record created in step S200, and then perform a picture comparison between the third data item and the retrieval key frame in each record with equal numerical comparison. If the picture comparison is consistent, the video retrieval is successful and the first and second data items in the record with consistent picture comparison are returned to the user. If the numerical comparison is not equal or the picture comparison is inconsistent, the information that the video retrieval failed is returned to the user.

2. The video retrieval method according to claim 1, characterized in that: The acquisition process of the storage size value of the starting frame described in step S200 and the storage size value of the retrieval key frame described in step S300 is implemented by storing the starting frame or the retrieval key frame as a computer image file, and the storage size value of the starting frame or the storage size value of the retrieval key frame is taken as the storage size of the computer image file in the computer memory, wherein the file format of the computer image file stored in the starting frame is required to be the same as that of the computer image file stored in the retrieval key frame.

3. The video retrieval method according to claim 2, characterized in that: The computer image file has a file format of PNG or GIF.

4. The video retrieval method according to claim 1, characterized in that: The corresponding pointer value of the logical segmentation generated shot in step S200 is the storage start address of the logical segmentation generated shot in the computer memory.

5. The video retrieval method according to claim 1, characterized in that: The corresponding pointer value of the logical segmentation generation shot described in step S200 is the time code of the starting frame in the logical segmentation generation shot in the video.

6. The video retrieval method according to claim 1, characterized in that: In the step S400, a sub-step of playing the video clip is also included: after all the record entries that meet the search condition are retrieved, the video clips indicated by the second data items in all the record entries are played.

7. A frame-based video retrieval device for finding shots containing a given frame from the entire video, characterized in that: The device comprises: Module M100 is used to: set an index table, each record in the index table stores data information of a shot of the video, and each record includes four data items, namely, first, second, third and fourth; The module M200 is used to: collect a video stream forwarded by a network server and originating from a network terminal screen display, and in the process of collecting the video stream, logically divide the video stream into a plurality of shots according to the shot change and store them in a memory, and for each shot generated by the logical segmentation, create a corresponding record in the index table of the module M100, and store the shot number containing the identity information of the network terminal in the first data item of the record, store the corresponding pointer value of the shot generated by the logical segmentation in the second data item of the record, store the starting frame of the shot generated by the logical segmentation in the third data item of the record, and collect the storage size value of the starting frame and store the storage size value in the fourth data item of the record; Module M300 is used to: obtain a search key frame input by a user and collect a storage size value of the search key frame, wherein the search key frame originates from a static interface display in a software system running on the network terminal, and the static interface display is related to the network terminal operation that causes the network event to occur; Module M400 is used to: first perform a numerical comparison between the storage size of the retrieval key frame in module M300 and the fourth data item in each record created in module M200, and then perform a picture comparison between the third data item and the retrieval key frame in each record with equal numerical comparison; if the picture comparison is consistent, the video retrieval is successful and the first and second data items in the record with consistent picture comparison are returned to the user; if the numerical comparison is not equal or the picture comparison is inconsistent, information that the video retrieval failed is returned to the user.

8. The video retrieval device according to claim 7, characterized in that: The module M400 also includes a submodule for playing video clips, which is used to: after all record entries that meet the search conditions are retrieved, play the video clips indicated by the second data items in all the record entries.

9. A computer system comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, characterized in that: When the processor runs the computer program, it implements the video retrieval method as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video retrieval method according to any one of claims 1 to 6 is implemented.

Images