Video query method, device and equipment, computer storage medium and program product
By establishing a preset semantic library in the video data and filtering the target video clip data based on the query conditions entered by the user, the problem that refined video query cannot be realized in the prior art is solved, and the effect of high-fine query is achieved.
Patent Information
- Application Number
- CN202410754385.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-12
- Publication Date
- 2024-11-08
AI Technical Summary
Existing video query technology cannot achieve more refined queries and cannot meet users' high-quality needs for video query.
By establishing a preset semantic library in the video data, storing the feature semantics in the video data, and filtering and positioning the target video clip data in the semantic library according to the query conditions entered by the user, fine-grained query of the video data is achieved.
It realizes more refined query of video data, meets users' high-refine query needs, and improves the accuracy and efficiency of query.
Smart Images

Figure CN118916514A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of video processing, and in particular, relates to a video query method, device, equipment, computer storage medium and program product. Background Art
[0002] At present, with the development of science and technology, people have higher and higher requirements for the refinement of video query. In related technologies, when querying a video, the target features to be queried are usually input into a trained special model, and then the special model is used to query the video library to obtain the desired video.
[0003] In the above-mentioned related technologies, when a dedicated model is used for video query, only a specific query target can be input, and the dedicated model extracts and analyzes the features of the video, and then uses the extracted features for analysis to realize video query. In this process, more refined query cannot be achieved. Summary of the invention
[0004] The embodiments of the present application provide a video query method, apparatus, device, computer storage medium and program product, which can realize more refined video query.
[0005] On the one hand, an embodiment of the present application provides a video query method, the method comprising:
[0006] In response to the user's operation on the video query condition configuration interface, determining a target video query condition, wherein the target video query condition at least includes a query object and its corresponding spatial information query condition;
[0007] Searching for a target semantics matching the target video query condition from a preset semantic library in the original video data, wherein the feature semantics in the preset semantic library include at least one of an object identifier, object attribute information, and object space information;
[0008] Determine the corresponding target video segment data according to the target semantics;
[0009] The target video segment data is output.
[0010] Optionally, before searching a preset semantic library in the original video data for a target semantic that matches the target video query condition, the method further includes:
[0011] Get the original video data;
[0012] Using a first preset image extraction model to perform feature extraction on the original video data to obtain at least one feature object and its corresponding semantic frame;
[0013] Determine the object type of each feature object according to a preset feature library;
[0014] Extracting information from each of the feature objects and the corresponding semantic frame according to the object type to obtain object attribute information and / or object space information of the feature object;
[0015] Constructing feature semantics of the feature object according to the object attribute information and / or object space information;
[0016] A preset semantic library is built according to the feature semantics of at least one of the feature objects.
[0017] Optionally, information is extracted from each of the feature objects and the corresponding semantic frame according to the object type to obtain object attribute information of the feature object, including:
[0018] Determine the attribute information type of the current feature object according to the object type;
[0019] Using a second preset semantic extraction model to extract keywords from each of the feature objects to obtain keyword information of each feature object;
[0020] The keyword information is filled in according to the attribute information type to obtain the object attribute information of the feature object.
[0021] Optionally, after extracting keywords from each of the feature objects using the second preset semantic extraction model to obtain keyword information of each feature object, the method further includes:
[0022] In the preset feature library, searching for an object identifier corresponding to the feature object according to the keyword information;
[0023] The object identifier is added to the preset semantic library.
[0024] Optionally, information is extracted from each of the feature objects and the corresponding semantic frame according to the object type to obtain object space information of the feature object, including:
[0025] Performing image extraction on the original video data to obtain at least one video frame image;
[0026] In the video frame image, establishing a position coordinate system;
[0027] Determine first position information of each of the semantic boxes based on the position coordinate system;
[0028] Recursively dividing the video frame image into regions according to a preset threshold value to obtain a plurality of rectangular regions;
[0029] Determine second position information of each of the rectangular areas based on the position coordinate system;
[0030] Allocating the feature object into the rectangular area based on the first position information and the second position information;
[0031] In each rectangular area, a spatial index is established for each semantic box according to a preset indexing method to obtain object spatial information of the feature object, wherein the object spatial information includes position information of the feature object in the rectangular area.
[0032] Optionally, the preset indexing method includes quadtree indexing or R-tree indexing.
[0033] Optionally, before building a preset semantic library according to the feature semantics of at least one of the feature formations, the method further includes:
[0034] Get the adjacent frame images of the feature semantics in the current video frame image
[0035] Calculating an overlap value of the semantic frame, where the overlap value is a ratio of an intersection area to a union area of a semantic frame in a current video frame image and a semantic frame in an adjacent frame image;
[0036] When the overlap value is not greater than a preset threshold, merging the current video frame image and the adjacent video frame image to obtain a video frame image set;
[0037] Using the current video frame image as a key frame image of the video frame image set;
[0038] The feature semantics of at least one of the feature images in the key frame image is used as the feature semantics of at least one of the feature objects in the video frame image set.
[0039] Optionally, the step of building a preset semantic library according to the feature semantics of at least one of the feature objects includes:
[0040] Converting the feature semantics into text to obtain text semantic feature data corresponding to the feature semantics;
[0041] A preset semantic library is built according to the text semantic feature data.
[0042] Optionally, the step of building a preset semantic library according to the feature semantics of at least one of the feature objects includes:
[0043] Establishing a video link with a corresponding video frame image according to the feature semantics of at least one feature object;
[0044] A preset semantic library is built according to the feature semantics and the video link.
[0045] Optionally, the target video query condition also includes keyword information of the query object and a target query time period.
[0046] On the other hand, an embodiment of the present application provides a video query device, the device comprising:
[0047] A response determination module, used to determine a target video query condition in response to a user's operation on the video query condition configuration interface, wherein the target video query condition at least includes a query object and its corresponding spatial information query condition;
[0048] A search module, used to search for target semantics matching the target video query condition from a preset semantic library in the original video data, wherein the feature semantics in the preset semantic library include at least one of object identification, object attribute information and object space information;
[0049] A determination module, used to determine corresponding target video segment data according to target semantics;
[0050] An output module is used to output the target video segment data.
[0051] In another aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions;
[0052] When the processor executes the computer program instructions, the video query method as described in the first aspect is implemented.
[0053] On the other hand, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the video query method described in the first aspect is implemented.
[0054] On the other hand, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the video query method as described in the first aspect.
[0055] The video query method, apparatus, device, computer storage medium and program product of the embodiments of the present application can, in the embodiments of the present application, when searching for target video data from original video data, pre-store the feature semantics of all query objects in the original video data by using a preset semantic library established based on the original video data, and then filter the preset semantic library based on the target video query conditions input by the user to find the target semantics that match the target video query conditions, locate the target video clip data through the target semantics, and realize the query of the target video clip data. In the process of searching for the target video data, the target semantics can be queried according to the target video query conditions, so as to realize a more refined query on the original video data, thereby meeting the demand for refined query video. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0057] Figure 1 It is a flowchart of a video query method provided by an embodiment of the present application;
[0058] Figure 2 It is a flowchart of a method for building a preset semantic library provided by an embodiment of the present application;
[0059] Figure 3 It is a flowchart of a method for extracting object attribute information provided by an embodiment of the present application;
[0060] Figure 4 is a structural diagram of a video query device provided by another embodiment of the present application;
[0061] Figure 5 It is a structural diagram of an electronic device provided by yet another embodiment of the present application. DETAILED DESCRIPTION
[0062] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.
[0063] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0064] In order to solve the problems of the prior art, embodiments of the present application provide a video query method, apparatus, device, storage medium and program product.
[0065] In an embodiment of the present application, when searching for target video data from original video data, the characteristic semantics of all query objects in the original video data are pre-stored through a preset semantic library established based on the original video data, and then the target video query conditions input by the user are screened in the preset semantic library to find the target semantics that match the target video query conditions, and the target video fragment data is located through the target semantics to achieve the query of the target video fragment data. In the process of searching for the target video data, the target semantics are queried according to the target video query conditions, which can achieve a more refined query on the original video data, thereby meeting the needs of refined query videos.
[0066] The following first introduces the video query method provided in the embodiment of the present application.
[0067] Figure 1 FIG. 1 is a flow chart of a video method provided by an embodiment of the present application. Figure 1 As shown, the video query method includes the following steps S101-S104:
[0068] S101, in response to a user's operation on a video query condition configuration interface, determining a target video query condition.
[0069] In one embodiment, when a user wants to search for target video segment data in the original video data, the user needs to operate on the video query condition configuration interface of the electronic device and input the target video query condition, and the electronic device will perform a query based on the target video query condition.
[0070] As an example, the target video query condition may at least include a query object and its corresponding spatial information query condition.
[0071] In one embodiment, the video query condition configuration interface may include a search box, which is used by the user to input the query object and its corresponding spatial information query condition, wherein the query object may be a specific query name such as a car, a person or an object, and the spatial information query condition may include the location information corresponding to the query object or a specific location in the video image customized by the user in the original video data.
[0072] In this embodiment, the video query condition configuration interface may also display a video frame image in the original video data, and the user draws a search box in the video frame image while operating the electronic device, and then uses the search box as the target video query condition for querying.
[0073] In one embodiment, the target video query condition may further include keyword information of the query object and a query time period.
[0074] In the case where the target query condition is keyword information of the query object, the user can input keyword information of the query object, such as a white car, through the search box, and then the electronic device uses the white car as the target video query condition.
[0075] In the case where the target query condition is the target query time period of the query object, the user can also enter the target query time period through the search box to search for target video segment data.
[0076] For example, the original video data is the surveillance video recording of intersection A from 00:00 on May 5, 2024 to 24:00 on May 6, 2024. The user can enter 13:00 on May 5, 2024 to 13:30 on May 5, 2024 in the search box and use this target query time period as the target video query condition.
[0077] In another embodiment, taking the original video data as the surveillance video of intersection B from 0:00 on May 5, 2024 to 24:00 on May 6, 2024 as an example, the user enters the target query condition of a white car passing through intersection B from 12:01 on May 5, 2024 to 13:01 on May 5, 2024 through the search box, and the electronic device will perform a refined query based on the input target video query condition, that is, a combined query is performed through keyword, time and space information query conditions.
[0078] The combined query can more finely query the original video data, so that the user can obtain the target video query segment that better meets the target video query conditions.
[0079] S102, searching for target semantics matching the target video query condition from a preset semantic library in the original video data.
[0080] As an example, the feature semantics in the preset semantic library includes at least one of object identification, object attribute information, and object space information.
[0081] In one embodiment, when querying the target video segment data, semantic extraction can be performed on the original database in advance, and the target video segment can be finely located according to the semantics of the original video data and the target video query conditions, thereby achieving a fine query on the original video data.
[0082] In this embodiment, in order to satisfy the user's various query methods and obtain more accurate target semantics, it is necessary to pre-process the original video data to find a more accurate target semantics that matches the target video query condition. Figure 2 Before S102, the following steps S1021-S1026 may also be included:
[0083] S1021, obtaining original video data.
[0084] S1022: Perform feature extraction on the original video data using a first preset image extraction model to obtain at least one feature object and its corresponding semantic frame.
[0085] S1023: Determine the object type of each feature object according to a preset feature library.
[0086] S1024: extract information from each feature object and the corresponding semantic frame according to the object type to obtain object attribute information and / or object space information of the feature object.
[0087] S1025, constructing feature semantics of the feature object according to the object attribute information and / or the object space information.
[0088] S1026: Build a preset semantic library according to the feature semantics of at least one feature object.
[0089] In this embodiment, by performing feature extraction on the original video data, all feature objects appearing in the original video data can be obtained. At the same time, the extracted feature objects can be framed through the first preset image extraction model to obtain a semantic frame corresponding to the feature object. After obtaining the feature object, since a preset feature library related to the feature object is pre-stored in the electronic device, the object type of the feature object can be identified through the preset feature library to facilitate subsequent information extraction of the feature object. Then, information extraction is performed on the feature object and the semantic frame respectively to obtain object attribute information and / or object space information of the feature object. Each object attribute information and / or object space information is used as a feature semantic to facilitate the electronic device to determine the feature semantics according to the target video query condition, thereby realizing the search of the target semantics and further realizing a more refined video query.
[0090] As an example, the semantic box may be a rectangular box that selects the feature object.
[0091] In one embodiment, in S1021, when the electronic device obtains the original video data, it may be the original video data directly transmitted by the video shooting device, or the original video data input into the electronic device by the user, which is not limited here.
[0092] In one embodiment, in S1022, when performing feature extraction on original video data, it is first necessary to extract video frame images from the original video data to obtain multiple video frame images, and then input each video frame image into a first preset image extraction model. The first preset image extraction model performs image recognition on each video frame image to extract at least one feature object contained in each video frame image. After extracting at least one feature image, the first preset image extraction model can also select the at least one extracted feature image to obtain a semantic box corresponding to the feature image.
[0093] As an example, the semantic box may be a rectangular box containing a feature image.
[0094] As an example, the first preset image extraction model may be an image segmentation model such as transformer or segment-anything.
[0095] In one embodiment, in S1023, taking the surveillance video of intersection B as an example of the original video data, the electronic device can pre-store the object types of characteristic objects that may pass through the surveillance video. For example, the object types of characteristic objects passing through intersection B can be large categories such as cars, people, and objects. The large categories can be further refined in the preset feature library, such as: a white car, a person wearing white clothes, or white goods, etc.
[0096] That is, the electronic device can store the object type of each feature object according to a preset feature library, and then obtain the object type of the feature object according to the identified feature object.
[0097] In one embodiment, see Figure 3 S1024 may specifically include the following steps S301-S303:
[0098] S301, determining the attribute information type of the current feature object according to the object type.
[0099] S302: extract keywords from each feature object using a second preset semantic extraction model to obtain keyword information of each feature object.
[0100] S303, filling the keyword information according to the attribute information type to obtain the object attribute information of the feature object.
[0101] In this embodiment, when extracting object attribute information, it is necessary to extract different object attribute information according to the object type determined in S1023, for example:
[0102] When the object type of the feature object is car, the attribute information types of the feature object may include car model, size and color.
[0103] When the object type of the feature object is a person, the attribute information type of the feature object may include height, clothing color, and gender.
[0104] When the object type of the feature object is object, the attribute information categories of the feature object may include size, packaging color, and type of goods.
[0105] As a specific example, taking the object type of the feature object as a car, when performing keyword extraction, the second preset semantic extraction type is used to extract the keyword information of the car. For example, in a video frame image, the keyword information of the feature object may include the model, size, and color. Then, these keyword information are filled in according to the attribute information type to form a table corresponding to the attribute information type and the keyword information, and then this table is used as the object attribute information of the feature object.
[0106] As an example, the second preset semantic model may be a semantic segmentation model such as transformer or segment-anything.
[0107] In this embodiment, keyword information is extracted to achieve refined labeling of feature images, thereby providing refined support for keyword query when users query target video segment data.
[0108] In one embodiment, in order to make the electronic device perform the target semantic query more quickly, before S302, the process further includes:
[0109] In the preset feature library, searching for the object identifier corresponding to the feature object according to the keyword information;
[0110] Supplement object identification into the preset semantic library.
[0111] In this embodiment, after obtaining the keyword information, the object identifier corresponding to the feature object can also be searched in the preset feature library through the keyword information, wherein the preset feature library can store object identifiers corresponding to the feature objects, such as the object identifier corresponding to a white car can be 21, and the object identifier corresponding to a black car can be 22.
[0112] Storing the object identifier in the preset semantic library can more conveniently search for feature objects corresponding to the target video query conditions in the preset semantic library, thereby improving the search efficiency of the target video frequency band.
[0113] In one embodiment, the extraction of the keyword information described above can satisfy the user's query conditions using the keyword information, providing a query basis for the keyword query, that is, the user can search for the corresponding target semantics by inputting the keyword information to be queried.
[0114] In another embodiment, for the semantic box corresponding to each feature object, a spatial index may be established for the semantic box to meet the spatial index requirements of the user. S1024 may further specifically include the following steps:
[0115] Performing image extraction on the original video data to obtain at least one video frame image;
[0116] In the video frame image, a position coordinate system is established;
[0117] Determine first position information of each semantic box based on the position coordinate system;
[0118] Recursively divide the video frame image into regions according to a preset threshold to obtain multiple rectangular regions;
[0119] Determine second position information of each rectangular area based on the position coordinate system;
[0120] Allocating the feature object into the rectangular area based on the first position information and the second position information;
[0121] In each rectangular area, a spatial index is established for each semantic box according to a preset indexing method to obtain the object spatial information of the feature object.
[0122] In this embodiment, in the process of establishing a spatial index, it is necessary to establish a position coordinate system in the video frame image, and then determine the first position information of each semantic box in the current video frame image based on the position coordinate system, and at the same time perform recursive region division or grid division on the video frame image to obtain multiple rectangular areas, and perform region division for establishing a spatial index to provide a position basis, and then determine the second position information of each rectangular area based on the position coordinate system, build a spatial index structure based on the second position information, and then assign the corresponding feature objects to the corresponding rectangular areas based on the first position information and the second position information, provide the features of the feature objects for establishing a spatial index, and then encode the feature objects and convert them into a data structure suitable for indexing, such as a vector or a matrix, and then establish a spatial index for the semantic box according to a preset indexing method to obtain the object space information of the feature object, that is, the object space information may include the position information of the feature object in the rectangular area.
[0123] In this embodiment, by establishing a spatial index for the semantic frame, the user can find the corresponding object spatial information in the preset semantic library by means of spatial query, so that the user can quickly query the target video segment data according to the spatial index.
[0124] As an example, the preset indexing method includes a quadtree index or an R-tree index.
[0125] Among them, the quadtree index can be used for multi-level queries, and the query efficiency can be improved by dividing the video frame image into regions. At the same time, by modifying the quadtree index parameters, the redundancy can be reduced and the query efficiency can be optimized. The R-tree index can be used to realize dynamic indexing, ensure the efficiency of the query, continuously optimize the R-tree query, and improve the performance of the spatial index.
[0126] In one embodiment, in S1025, after extracting the object attribute information and / or object space information of the feature object in the above manner, the feature semantics of the feature object is constructed, where, as an example, the feature semantics may include object attribute information and object space information, and in another example, the feature semantics may also include video frame images, video links, object types, and object identifiers.
[0127] In this embodiment, when constructing feature semantics, in order to meet the needs of time query, the start and end time can be added to the feature semantics according to the video frame image when constructing the feature semantics, so that the user can find the corresponding target semantics through the start and end time when querying time.
[0128] In one embodiment, in the original video data, for some video frame images, there may be a situation where the same feature object does not change for a period of time, resulting in redundancy in the feature semantics stored when building the preset semantic library. Therefore, before S1026, the following steps may be further included:
[0129] Get the adjacent frame images of the feature semantics in the current video frame image
[0130] Calculate the overlap value of semantic boxes;
[0131] When the overlap value is not greater than a preset threshold, the current video frame image and the adjacent video frame image are merged to obtain a video frame image set;
[0132] Using the current video frame image as a key frame image of the video frame image set;
[0133] The feature semantics of at least one feature image in the key frame image is used as the feature semantics of at least one feature object in the video frame image set.
[0134] In this embodiment, by determining the overlap value between the current frame image and the adjacent frame image, it is determined whether the current frame image and the adjacent frame image can be merged, thereby reducing storage redundancy and reducing the storage load of the electronic device.
[0135] Specifically, since the semantic frame of the feature object has been obtained before building the preset semantic library, the overlap value can be calculated based on the intersection area and union area of the semantic frames between the current video frame image and the adjacent frame image, and the overlap value is compared with the preset threshold. When the overlap value is not greater than the preset threshold, it means that the semantic frame of the current video frame image has a high degree of overlap with the semantic frame of the adjacent frame image. At this time, the current frame image and the adjacent frame image can be merged to reduce storage redundancy.
[0136] As an example, the overlap value is a ratio of an intersection area to a union area between a semantic box in a current video frame image and a semantic box in an adjacent frame image.
[0137] After merging the current video frame image and the adjacent frame image, the start and end times can be updated according to the obtained video frame image set, and the updated start and end times can be used as the start and end times of the key frame image, and the feature semantics in the key frame image can be used as the feature semantics of the video frame image set to reduce storage redundancy when building a preset semantic library.
[0138] In one embodiment, in order to better fit the target query video condition input by the user, S1026 may include:
[0139] Convert the feature semantics into text to obtain text semantic feature data corresponding to the feature semantics;
[0140] Build a preset semantic library based on text semantic feature data.
[0141] In this embodiment, storing in the form of text can facilitate the electronic device to search in the preset semantic library according to the target video query condition, and storing in the form of text can reduce storage space. On the other hand, it can enable the electronic device to determine the target semantics more quickly and improve the query speed.
[0142] In another embodiment, S1026 may further include:
[0143] Establishing a video link with a corresponding video frame image according to the feature semantics of at least one feature object;
[0144] Build a preset semantic library based on feature semantics and video links.
[0145] In this embodiment, in order to reduce storage space, there is no need to store video data corresponding to feature semantics in a preset semantic library. A video link can be established between the feature semantics and the video frame image, and then the video link can be associated with the original video data, thereby reducing the occupied storage space.
[0146] S103: Determine corresponding target video segment data according to the target semantics.
[0147] In this embodiment, when the electronic device finds the target semantics corresponding to the target video query condition, the target video segment data can be determined through the video link of the target semantics, and the video link corresponding to the target video segment data is returned in the electronic device.
[0148] S104: Output the target video segment data.
[0149] In this embodiment, after the electronic device searches for the corresponding video link according to the target video query conditions, the video link can be directly displayed in the electronic device. The user clicks the video link to watch the target video segment data, and can also directly open the target video segment data, thereby completing the query of the target video segment.
[0150] In an embodiment of the present application, when searching for target video data from original video data, the characteristic semantics of all query objects in the original video data are pre-stored through a preset semantic library established based on the original video data, and then the target video query conditions input by the user are screened in the preset semantic library to find the target semantics that match the target video query conditions, and the target video fragment data is located through the target semantics to achieve the query of the target video fragment data. In the process of searching for the target video data, the target semantics are queried according to the target video query conditions, which can achieve a more refined query on the original video data, thereby meeting the needs of refined query videos.
[0151] See also Figure 4 The embodiment of the present application further provides a video query device 400, which may include:
[0152] A response determination module 401 is used to determine a target video query condition in response to a user's operation on the video query condition configuration interface, where the target video query condition at least includes a query object and its corresponding spatial information query condition;
[0153] A search module 402 is used to search for a target semantic matching the target video query condition from a preset semantic library in the original video data, wherein the feature semantics in the preset semantic library include at least one of an object identifier, object attribute information, and object space information;
[0154] A determination module 403, configured to determine corresponding target video segment data according to target semantics;
[0155] The output module 404 is used to output the target video segment data.
[0156] As an optional implementation, the search module 402 may include:
[0157] An acquisition unit, used for acquiring original video data;
[0158] A first extraction unit, configured to perform feature extraction on the original video data using a first preset image extraction model to obtain at least one feature object and its corresponding semantic frame;
[0159] A determination unit, used to determine the object type of each feature object according to a preset feature library;
[0160] A second extraction unit is used to extract information from each feature object and the corresponding semantic frame according to the object type to obtain object attribute information and / or object space information of the feature object;
[0161] A construction unit, used for constructing feature semantics of feature objects according to object attribute information and / or object space information;
[0162] A building unit is used to build a preset semantic library according to the feature semantics of at least one feature object.
[0163] As an optional implementation, the second extraction unit may also be used for:
[0164] Determine the attribute information type of the current feature object according to the object type;
[0165] Using the second preset semantic extraction model to extract keywords from each feature object, to obtain keyword information of each feature object;
[0166] The keyword information is filled in according to the attribute information type to obtain the object attribute information of the feature object.
[0167] As an optional implementation, the second extraction unit may also be used for:
[0168] In the preset feature library, searching for the object identifier corresponding to the feature object according to the keyword information;
[0169] Supplement object identification into the preset semantic library.
[0170] As an optional implementation, the second extraction unit may also be used for:
[0171] Performing image extraction on the original video data to obtain at least one video frame image;
[0172] In the video frame image, a position coordinate system is established;
[0173] Determine first position information of each semantic box based on the position coordinate system;
[0174] Recursively divide the video frame image into regions according to a preset threshold to obtain multiple rectangular regions;
[0175] Determine second position information of each rectangular area based on the position coordinate system;
[0176] Allocating the feature object into the rectangular area based on the first position information and the second position information;
[0177] In each rectangular area, a spatial index is established for each semantic box according to a preset indexing method to obtain object spatial information of the feature object, and the object spatial information includes position information of the feature object in the rectangular area.
[0178] As an optional implementation, the preset indexing method includes quadtree indexing or R-tree indexing.
[0179] As an optional embodiment, the building unit can also be used for:
[0180] Get the adjacent frame images of the feature semantics in the current video frame image
[0181] Calculate the overlap value of the semantic box, where the overlap value is the ratio of the intersection area and the union area of the semantic box in the current video frame image and the semantic box in the adjacent frame image;
[0182] When the overlap value is not greater than a preset threshold, the current video frame image and the adjacent video frame image are merged to obtain a video frame image set;
[0183] Using the current video frame image as a key frame image of the video frame image set;
[0184] The feature semantics of at least one feature image in the key frame image is used as the feature semantics of at least one feature object in the video frame image set.
[0185] As an optional embodiment, the building unit can also be used for:
[0186] Convert the feature semantics into text to obtain text semantic feature data corresponding to the feature semantics;
[0187] Build a preset semantic library based on text semantic feature data.
[0188] As an optional embodiment, the building unit can also be used for:
[0189] Establishing a video link with a corresponding video frame image according to the feature semantics of at least one feature object;
[0190] Build a preset semantic library based on feature semantics and video links.
[0191] As an optional implementation, the target video query condition also includes keyword information of the query object and a target query time period.
[0192] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.
[0193] The electronic device may include a processor 501 and a memory 502 storing computer program instructions.
[0194] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0195] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In one example, the memory 602 may include a removable or non-removable (or fixed) medium, or the memory 602 is a non-volatile solid-state memory. The memory 502 may be inside or outside the integrated gateway disaster recovery device.
[0196] In one example, the memory 502 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.
[0197] The memory 502 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the video query method according to the first aspect of the present disclosure.
[0198] The processor 501 reads and executes the computer program instructions stored in the memory 502 to implement Figure 1 A video query method in the illustrated embodiment.
[0199] In one example, the electronic device may further include a communication interface 503 and a bus 504. Figure 5 As shown, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 504 and communicate with each other.
[0200] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.
[0201] Bus 504 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (Accelerated Graphics Port, AGP) or other graphics bus, enhanced industry standard architecture (Extended Industry Standard Architecture, EISA) bus, front side bus (Front Side Bus, FSB), hypertransmission (Hyper Transport, HT) interconnection, industry standard architecture (IndustryStandard Architecture, ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 504 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.
[0202] The electronic device can execute the video query method in the embodiment of the present application, thereby realizing the combination Figure 1-Figure 4The video query method and device described.
[0203] In addition, in combination with the video query method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the video query methods in the above embodiment is implemented.
[0204] In an optional embodiment, in combination with the video query method in the above embodiment, the embodiment of the present application can provide a computer program product for implementation, and the instructions in the computer program product are executed by a processor of an electronic device, so that the electronic device can implement any one of the video query methods in the above embodiment.
[0205] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.
[0206] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.
[0207] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.
[0208] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.
[0209] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.
Claims
1. A video query method, characterized in that: include: In response to the user's operation on the video query condition configuration interface, determining a target video query condition, wherein the target video query condition at least includes a query object and its corresponding spatial information query condition; Searching for a target semantics matching the target video query condition from a preset semantic library in the original video data, wherein the feature semantics in the preset semantic library include at least one of an object identifier, object attribute information, and object space information; Determine the corresponding target video segment data according to the target semantics; The target video segment data is output.
2. The method according to claim 1, characterized in that Before searching the preset semantic library in the original video data for the target semantics matching the target video query condition, the method further includes: Get the original video data; Using a first preset image extraction model to perform feature extraction on the original video data to obtain at least one feature object and its corresponding semantic frame; Determine the object type of each feature object according to a preset feature library; Extracting information from each of the feature objects and the corresponding semantic frame according to the object type to obtain object attribute information and / or object space information of the feature object; Constructing feature semantics of the feature object according to the object attribute information and / or object space information; A preset semantic library is built according to the feature semantics of at least one of the feature objects.
3. The method according to claim 2, characterized in that Information is extracted from each of the feature objects and the corresponding semantic frame according to the object type to obtain object attribute information of the feature object, including: Determine the attribute information type of the current feature object according to the object type; Using a second preset semantic extraction model to extract keywords from each of the feature objects to obtain keyword information of each feature object; The keyword information is filled in according to the attribute information type to obtain the object attribute information of the feature object.
4. The method according to claim 3, characterized in that After extracting keywords from each of the feature objects using the second preset semantic extraction model to obtain keyword information of each feature object, the method further includes: In the preset feature library, searching for an object identifier corresponding to the feature object according to the keyword information; The object identifier is added to the preset semantic library.
5. The method according to claim 2, characterized in that: Extracting information from each of the feature objects and the corresponding semantic frame according to the object type to obtain object space information of the feature object includes: Performing image extraction on the original video data to obtain at least one video frame image; In the video frame image, establishing a position coordinate system; Determine first position information of each of the semantic boxes based on the position coordinate system; Recursively dividing the video frame image into regions according to a preset threshold value to obtain a plurality of rectangular regions; Determine second position information of each of the rectangular areas based on the position coordinate system; Allocating the feature object into the rectangular area based on the first position information and the second position information; In each rectangular area, a spatial index is established for each semantic box according to a preset indexing method to obtain object spatial information of the feature object, wherein the object spatial information includes position information of the feature object in the rectangular area.
6. The method according to claim 5, characterized in that The preset indexing method includes quadtree indexing or R-tree indexing.
7. The method according to claim 2, characterized in that Before building a preset semantic library according to the feature semantics of at least one of the feature formations, the method further includes: Get the adjacent frame images of the feature semantics in the current video frame image Calculating an overlap value of the semantic frame, where the overlap value is a ratio of an intersection area to a union area of a semantic frame in a current video frame image and a semantic frame in an adjacent frame image; When the overlap value is not greater than a preset threshold, merging the current video frame image and the adjacent video frame image to obtain a video frame image set; Using the current video frame image as a key frame image of the video frame image set; The feature semantics of at least one of the feature images in the key frame image is used as the feature semantics of at least one of the feature objects in the video frame image set.
8. The method according to any one of claims 2 to 7, characterized in that: The step of building a preset semantic library according to the feature semantics of at least one of the feature objects comprises: Converting the feature semantics into text to obtain text semantic feature data corresponding to the feature semantics; A preset semantic library is built according to the text semantic feature data.
9. The method according to any one of claims 2 to 7, characterized in that: The step of building a preset semantic library according to the feature semantics of at least one of the feature objects comprises: Establishing a video link with a corresponding video frame image according to the feature semantics of at least one feature object; A preset semantic library is built according to the feature semantics and the video link.
10. The method according to any one of claims 1 to 7, characterized in that: The target video query condition also includes keyword information of the query object and a target query time period.
11. A video query device, characterized in that: The device comprises: A response determination module, used to determine a target video query condition in response to a user's operation on the video query condition configuration interface, wherein the target video query condition at least includes a query object and its corresponding spatial information query condition; A search module, used to search for target semantics matching the target video query condition from a preset semantic library in the original video data, wherein the feature semantics in the preset semantic library include at least one of object identification, object attribute information and object space information; A determination module, used to determine corresponding target video segment data according to target semantics; An output module is used to output the target video segment data.
12. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the video query method according to any one of claims 1 to 10 is implemented.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the video query method according to any one of claims 1 to 10 is implemented.
14. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the video query method as described in any one of claims 1-10.
