Video information processing method and device, equipment and storage medium
By extracting frames on the video and extracting entity information using the entity information detection model, the problem of time-consuming and labor-consuming manual labeling is solved, and efficient video information processing and entity identity marking are achieved.
Patent Information
- Application Number
- CN202510095531.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-21
- Publication Date
- 2025-04-25
AI Technical Summary
In the prior art, video content needs to be labeled manually, resulting in low processing efficiency, long time-consuming and inability to meet actual business needs.
By obtaining the video to be processed, the frame extraction process obtains image frame and video attribute information, and inputs the image frame to the trained entity information detection model, extracts entity association information, determines entity information vector and identity identification vector, performs stitching processing, and finally stores the relevant information in the database.
It realizes the extraction of entity information in the video without manual identification or labeling, improves processing efficiency, and uniquely marks the entity, which facilitates subsequent video analysis, retrieval and recommendation.
Smart Images
Figure CN119893048A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of video technology, and in particular, to a video information processing method, apparatus, device, and storage medium. Background Art
[0002] With the rapid development of video technology, video has been widely used in various fields. In the field of video surveillance, it is necessary to use cameras and related equipment to monitor and record activities in specific areas or places such as public safety areas and transportation hubs in real time, and extract information from the recorded videos to provide reliable reference data for security management, risk warning and business optimization.
[0003] However, the relevant technology requires labeling personnel to manually identify video content and mark various entity objects in the video, such as pedestrians, vehicles, animals, etc., as well as the entity relationships between entity objects, such as peer relationships, following relationships, driver-passenger relationships, etc. For application scenarios with large video data capacity, this will consume a lot of time and manpower costs, and the processing efficiency is low, which cannot meet actual business needs. Summary of the invention
[0004] The embodiments of the present application provide a video information processing method, apparatus, device and storage medium, which solve the problem in related technologies that labeling personnel need to manually identify and label video content, which consumes a lot of time and labor costs, has low processing efficiency and cannot meet actual business needs. It realizes the effective use of entity information detection models to extract entity information without manual identification or labeling, improves processing efficiency, and uniquely labels different entities, effectively stores entity information corresponding to the video to be processed, and provides strong data support for subsequent video analysis, retrieval and recommendation applications.
[0005] In a first aspect, an embodiment of the present application provides a video information processing method, the method comprising:
[0006] Acquire a video to be processed, and perform frame extraction processing on the video to be processed to obtain at least one image frame to be processed and video attribute information corresponding to each image frame to be processed;
[0007] Inputting the image frame to be processed into a trained entity information detection model to obtain entity association information, wherein the entity association information includes entity image, entity feature description information and entity relationship description information;
[0008] Determine a first entity information vector and an identity vector according to the entity image, entity feature description information and entity relationship description information, and concatenate the first entity information vector and the identity vector to obtain a second entity information vector;
[0009] Each of the to-be-processed image frames, the corresponding video attribute information, the entity association information and the second entity information vector are associated and stored in a set database.
[0010] Optionally, determining the first entity information vector and the identity vector according to the entity image, the entity feature description information and the entity relationship description information includes:
[0011] Respectively performing vector conversion processing on the entity image, entity feature description information and entity relationship description information to obtain an entity image vector, an entity feature vector and an entity relationship vector;
[0012] The entity image vector, the entity feature vector and the entity relationship vector are concatenated to obtain a first entity information vector;
[0013] The entity image vector and the entity feature vector are concatenated to obtain a model input vector, and the model input vector is input into a trained entity identity confidence model to obtain an identity identification vector.
[0014] Optionally, after associating and storing each of the to-be-processed image frames, the corresponding video attribute information, the entity association information and the second entity information vector in a set database, the method further includes:
[0015] When receiving a video query request sent by a client, extracting target entity information and target relationship information from the video query request, and performing vector conversion processing on the target entity information and the target relationship information to obtain a first target entity information vector and a target relationship vector;
[0016] The first target entity information vector is input into the entity identity confidence model to obtain a target identity identification vector, and the first target entity information vector, the target relationship vector and the target identity identification vector are concatenated to obtain a second target entity information vector
[0017] Searching the database based on the second target entity information vector to obtain at least one target image frame, corresponding target video attribute information and target entity association information;
[0018] The at least one target image frame, corresponding target video attribute information and target entity association information are sent to the client.
[0019] Optionally, searching the database based on the second target entity information vector to obtain at least one target image frame, corresponding target video attribute information and target entity association information includes:
[0020] Calculating similarity between the second target entity information vector and existing entity information vectors stored in the database, and selecting a preset number of candidate entity information vectors that meet the highest similarity from the existing entity information vectors;
[0021] The database is queried to obtain target image frames associated with the plurality of candidate entity information vectors, corresponding target video attribute information and target entity association information.
[0022] Optionally, before inputting the model input vector into the trained entity identity confidence model, the method further includes:
[0023] Obtain multiple sample feature information vectors;
[0024] The pre-constructed entity identity confidence model is pre-trained based on the multiple sample feature information vectors.
[0025] Optionally, before obtaining the plurality of sample feature information vectors, the method further includes:
[0026] Acquire a plurality of sample image frames;
[0027] Inputting the sample image frame into a trained entity information detection model to obtain a sample entity image and sample entity feature information;
[0028] The sample entity image and the sample entity feature information are respectively subjected to vector conversion processing to obtain a sample entity image vector and a sample entity feature vector, and the sample entity image vector and the sample entity feature vector are spliced to obtain a sample feature information vector.
[0029] Optionally, before acquiring a plurality of sample image frames, the method further includes:
[0030] Acquire multiple first historical videos, and pre-process the first historical videos to obtain second historical videos;
[0031] The second historical video is subjected to frame extraction processing to obtain a plurality of sample image frames.
[0032] In a second aspect, an embodiment of the present application further provides a video information processing device, including:
[0033] A video frame extraction processing unit is configured to obtain a video to be processed, and perform frame extraction processing on the video to be processed to obtain at least one image frame to be processed and video attribute information corresponding to each image frame to be processed;
[0034] An entity information determination unit is configured to input the image frame to be processed into a trained entity information detection model to obtain entity association information, wherein the entity association information includes an entity image, entity feature description information, and entity relationship description information;
[0035] an entity information vector determining unit, configured to determine a first entity information vector and an identity vector according to the entity image, the entity feature description information and the entity relationship description information, and concatenate the first entity information vector and the identity vector to obtain a second entity information vector;
[0036] The video information association storage unit is configured to associate and store each of the to-be-processed image frames, the corresponding video attribute information, the entity association information and the second entity information vector in a set database.
[0037] Wherein, the entity information vector determination unit includes:
[0038] A vector conversion module is configured to perform vector conversion processing on the entity image, entity feature description information and entity relationship description information to obtain an entity image vector, an entity feature vector and an entity relationship vector;
[0039] A vector concatenation module is configured to concatenate the entity image vector, the entity feature vector and the entity relationship vector to obtain a first entity information vector;
[0040] The identification vector generation module is configured to concatenate the entity image vector and the entity feature vector to obtain a model input vector, and input the model input vector into the trained entity identity confidence model to obtain an identity identification vector.
[0041] The device further comprises:
[0042] The query request processing unit is configured as follows:
[0043] When receiving a video query request sent by a client, extracting target entity information and target relationship information from the video query request, and performing vector conversion processing on the target entity information and the target relationship information to obtain a first target entity information vector and a target relationship vector;
[0044] The first target entity information vector is input into the entity identity confidence model to obtain the target identity identification vector, and the first target entity information vector, the target relationship vector and the target identity identification vector are concatenated to obtain the second target entity information vector
[0045] Searching a database based on the second target entity information vector to obtain at least one target image frame, corresponding target video attribute information and target entity association information;
[0046] At least one target image frame, corresponding target video attribute information and target entity association information are sent to the client.
[0047] The query request processing unit includes:
[0048] An information vector screening module, configured to calculate similarity between the second target entity information vector and existing entity information vectors stored in the database, and screen out a preset number of candidate entity information vectors that meet the highest similarity from the existing entity information vectors;
[0049] The information query module is configured to query the database to obtain target image frames associated with multiple candidate entity information vectors, corresponding target video attribute information and target entity association information.
[0050] The device further comprises:
[0051] Model training unit, configured as:
[0052] Obtain multiple sample feature information vectors;
[0053] The pre-built entity identity confidence model is pre-trained based on multiple sample feature information vectors.
[0054] The device further comprises:
[0055] The sample vector generation unit is configured as follows:
[0056] Acquire a plurality of sample image frames;
[0057] Input the sample image frame into the trained entity information detection model to obtain the sample entity image and sample entity feature information;
[0058] The sample entity image and the sample entity feature information are respectively subjected to vector conversion processing to obtain a sample entity image vector and a sample entity feature vector, and the sample entity image vector and the sample entity feature vector are concatenated to obtain a sample feature information vector.
[0059] The device further comprises:
[0060] Sample image frame generation unit, configured as:
[0061] Acquire multiple first historical videos, and pre-process the first historical videos to obtain second historical videos;
[0062] The second historical video is subjected to frame extraction processing to obtain a plurality of sample image frames.
[0063] In a third aspect, an embodiment of the present application further provides an electronic device, the device comprising:
[0064] one or more processors;
[0065] a storage device configured to store one or more programs,
[0066] When the one or more programs are executed by the one or more processors, the one or more processors implement the video information processing method described in the embodiment of the present application.
[0067] In a fourth aspect, an embodiment of the present application further provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are configured to execute the video information processing method described in the embodiment of the present application.
[0068] In an embodiment of the present application, by acquiring a video to be processed, frame extraction processing is performed on the video to be processed to obtain at least one image frame to be processed and video attribute information corresponding to each image frame to be processed; the image frame to be processed is input into a trained entity information detection model to obtain entity association information, and the entity association information includes an entity image, entity feature description information and entity relationship description information; according to the entity image, the entity feature description information and the entity relationship description information, a first entity information vector and an identity identification vector are determined, and the first entity information vector and the identity identification vector are spliced to obtain a second entity information vector; each image frame to be processed, the corresponding video attribute information, the entity association information and the second entity information vector are associated and stored in a set database. In the above scheme, by extracting frames from the video to be processed, the image frames to be processed that need to be processed for entity information extraction and the video attribute information corresponding to each image frame to be processed can be determined, and the association reference information between the image frames to be processed and the video to be processed can be provided; by inputting the image frames to be processed into the trained entity information detection model, the entity information detection model can be effectively used to extract entity information without manual recognition or labeling, thereby improving processing efficiency; by determining the first entity information vector and the identity identification vector, unique identity labeling is performed on different entities, which can help distinguish different entities and improve the distinction of entities; by associating and storing each image frame to be processed, the corresponding video attribute information, entity association information, and the second entity information vector in a set database, the entity information corresponding to the video to be processed can be effectively stored, providing strong data support for subsequent video analysis, retrieval, and recommendation applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] Figure 1 A flowchart of a video information processing method provided in an embodiment of the present application;
[0070] Figure 2 A flowchart of a specific implementation process of determining a first entity information vector and an identity identification vector provided in an embodiment of the present application;
[0071] Figure 3 A flowchart of a specific implementation process of training an entity identity confidence model provided in an embodiment of the present application;
[0072] Figure 4 A flowchart of a specific implementation process of generating multiple sample feature information vectors provided in an embodiment of the present application;
[0073] Figure 5 A flowchart of a specific implementation process of generating multiple sample image frames provided in an embodiment of the present application;
[0074] Figure 6 A flowchart of a specific implementation process of processing a video query request provided in an embodiment of the present application;
[0075] Figure 7 A flowchart of a specific implementation process of retrieving a target image frame, corresponding target video attribute information and target entity association information provided by an embodiment of the present application;
[0076] Figure 8 A structural block diagram of a video information processing device provided in an embodiment of the present application;
[0077] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0078] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only parts related to the embodiments of the present application are shown in the accompanying drawings, rather than all structures.
[0079] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0080] The video information processing method provided in the embodiment of the present application can extract the image frames to be processed from the video to be processed, use the entity information detection model to extract the entity association information of the image frames to be processed, and determine the corresponding identity identification vector to complete the entity identity labeling, and finally store each image frame to be processed, the corresponding video attribute information, the entity association information and the entity information vector in an associated manner to a set database, which can effectively store the entity association information of the video to be processed and provide entity reference information for subsequent video processing. The relevant application scenarios can be video analysis, video retrieval, video recommendation, etc. The several application scenarios listed above are only exemplary and explanatory. In actual applications, the video information processing method can also be used in video services in other scenarios, and the embodiment of the present application does not limit this.
[0081] Figure 1 The flowchart of a video information processing method provided in the embodiment of the present application can be implemented by using a background server as the execution subject. Figure 1 As shown, the video information processing method specifically includes the following steps:
[0082] Step S110: Acquire a video to be processed, perform frame extraction on the video to be processed to obtain at least one image frame to be processed and video attribute information corresponding to each image frame to be processed.
[0083] Among them, the video to be processed can be a real-time video or offline video obtained by recording through a camera device and related equipment in a specific application scenario. The frame extraction process can be to evenly extract image frames from the video to be processed at a preset time interval, and use the extracted image frames as the image frames to be processed, or it can be to perform still frame detection on the extracted image frames after uniformly extracting the image frames, and remove the image frames determined to be still frames, and use the remaining image frames as the image frames to be processed. The specific still frame detection algorithm can adopt the inter-frame difference method, optical flow method, background modeling method, etc., which is not limited in this application. In addition, while extracting the image frames to be processed, it is also necessary to record the video attribute information corresponding to the image frames to be processed. The video attribute information may include the video identifier of the video to be processed and the video position information of the image frames to be processed. The video position information is used to indicate the specific time position of the image frame to be processed in the video timeline of the video to be processed, for example, the Xth minute and Xth second.
[0084] Step S120: input the image frame to be processed into the trained entity information detection model to obtain entity association information, where the entity association information includes entity image, entity feature description information and entity relationship description information.
[0085] It should be noted that the entity information detection model can be an open source multimodal large model, such as the Yi-VL (YiVision Language) model, the LLaVA (Large Language and Vision Assistant) model, etc. Specifically, the aforementioned multimodal large model can be fine-tuned and trained based on the pre-set sample image frames and the corresponding standard entity association information to obtain an entity information detection model that can detect entities on images and obtain entity association information. The entity association information includes entity images, entity feature description information, and entity relationship description information. The entity image is a partial screenshot of an entity in the image frame to be processed, such as a captured person, vehicle, animal, etc. The entity feature description information is text information that describes the structured features of the entity, for example, for a person, it includes the person's body shape, appearance, and clothing. The entity relationship description information is text information that describes the spatial relationship or action relationship between entities in the image frame to be processed, for example, "entity A is sitting in the taxi of entity B", "entity A and entity B are holding hands", etc.
[0086] Step S130: determine a first entity information vector and an identity identification vector according to the entity image, entity feature description information and entity relationship description information, and concatenate the first entity information vector and the identity identification vector to obtain a second entity information vector.
[0087] Among them, the first entity information vector is a vector representation of the associated information corresponding to each entity in the image frame to be processed, and the identity vector is a unique identity corresponding to each entity in the image frame to be processed. Optionally, the first entity information vector can be obtained by converting the entity image, entity feature description information and entity relationship description information corresponding to the same entity into corresponding vectors and then splicing them. Specifically, the vector representation corresponding to the entity image can be obtained by directly converting all pixel values into a vector, or extracting local features through the SIFT (Scale-Invariant Feature Transform) algorithm and obtaining the target vector, or by inputting pre-trained VGG16, ResNet and other models for processing to obtain the target vector, which is not limited in this application; the vector representation corresponding to the entity feature description information and the entity relationship description information can be generated by word embedding methods such as Word2Vec, GloVe, FastText, etc., or by extracting text features through pre-trained deep learning models such as BERT and RoBERT to obtain the target vector, which is not limited in this application. Optionally, the identity identification vector can be obtained by splicing and normalizing the vectors obtained by converting the entity image and entity feature description information corresponding to the same entity, or by splicing the vectors obtained by converting the entity image and entity feature description information corresponding to the same entity and inputting the spliced vector into a trained deep learning model, which is not limited in this application.
[0088] In a specific implementation, a process for determining a first entity information vector and an identity vector is described. Figure 2 , which is a flowchart of a specific implementation process of determining a first entity information vector and an identity identification vector provided by an embodiment of the present application, such as Figure 2 As shown, the specific implementation steps of determining the first entity information vector and the identity identification vector are as follows:
[0089] Step S131 , respectively perform vector conversion processing on the entity image, entity feature description information and entity relationship description information to obtain an entity image vector, an entity feature vector and an entity relationship vector.
[0090] Step S132: Concatenate the entity image vector, the entity feature vector and the entity relationship vector to obtain a first entity information vector.
[0091] Step S133: concatenate the entity image vector and the entity feature vector to obtain a model input vector, and input the model input vector into the trained entity identity confidence model to obtain an identity identification vector.
[0092] It should be noted that the entity identity confidence model can be a deep learning model based on the Transformer architecture, which is used to identify different entities according to the model input vector and give each entity a unique virtual identity.
[0093] In another specific embodiment, a process for training an entity identity confidence model is described. Figure 3 , which is a flowchart of a specific implementation process of training an entity identity confidence model provided in an embodiment of the present application. Before inputting the model input vector into the trained entity identity confidence model, as shown in FIG. Figure 3 As shown, the specific implementation steps for training the entity identity confidence model are as follows:
[0094] Step S1331, obtain multiple sample feature information vectors.
[0095] Step S1332: pre-train the pre-constructed entity identity confidence model based on multiple sample feature information vectors.
[0096] It is worth noting that the entity identity confidence model can be a deep learning model based on the Transformer architecture. The pre-training of the entity identity confidence model is an unsupervised training process. The sample feature information vector is input into the entity identity confidence model, and a result vector of a fixed length output after feature analysis and learning can be obtained. By inputting a large number of sample feature information vectors into the entity identity confidence model, the entity identity confidence model can be trained to have the ability to generate corresponding unique result vectors according to different sample feature information vectors. Therefore, for the model input vectors generated for the same entity in different image frames to be processed, the entity identity confidence model can output the same result vector as the identity identification vector.
[0097] In a specific embodiment, a process of generating multiple sample feature information vectors is described, please refer to Figure 4 , which is a flowchart of a specific implementation process of generating multiple sample feature information vectors provided by an embodiment of the present application. Before obtaining multiple sample feature information vectors, such as Figure 4 As shown, the specific implementation steps for generating multiple sample feature information vectors are as follows:
[0098] Step S1333: Acquire multiple sample image frames.
[0099] Step S1334: input the sample image frames into the trained entity information detection model respectively to obtain the sample entity image and sample entity feature information.
[0100] Step S1335: perform vector conversion processing on the sample entity image and the sample entity feature information respectively to obtain a sample entity image vector and a sample entity feature vector, and perform splicing processing on the sample entity image vector and the sample entity feature vector to obtain a sample feature information vector.
[0101] Among them, by inputting each sample image frame into the trained entity information detection model, the relevant information of the sample entity can be effectively extracted from the sample image frame to obtain the sample entity image and sample entity feature information. By splicing the sample entity image vector and the sample entity feature vector to obtain the sample feature information vector, the entity-related information of different dimensions can be effectively integrated to obtain the sample feature information vector that comprehensively characterizes the sample entity.
[0102] In another specific embodiment, a process for generating multiple sample image frames is described. Figure 5 , which is a flowchart of a specific implementation process of generating multiple sample image frames provided by an embodiment of the present application. Before obtaining multiple sample image frames, such as Figure 5 As shown, the specific implementation steps of generating multiple sample image frames are as follows:
[0103] Step S1336: Acquire multiple first historical videos, and pre-process the first historical videos to obtain second historical videos.
[0104] Step S1337: perform frame extraction processing on the second historical video to obtain multiple sample image frames.
[0105] It should be noted that the first historical video can be collected by developers from multiple public data sources based on the principle of data diversity and richness. The preprocessing can be noise elimination, clarity enhancement, frame rate adjustment, etc. for the first historical video to improve the accuracy of related data processing of subsequent sample image frames. By extracting frames from the second historical video, multiple sample image frames can be obtained to provide basic image data for subsequent training of the entity identity confidence model.
[0106] Step S140: associate and store each to-be-processed image frame, the corresponding video attribute information, the entity association information and the second entity information vector in a set database.
[0107] Among them, by associating and storing the video attribute information, entity association information and the second entity information vector corresponding to the image frame to be processed, it is beneficial to subsequently query the target image frame based on the entity-related information provided by the user, and facilitates the related query and analysis of the target video.
[0108] In the above, by acquiring a video to be processed, the video to be processed is subjected to frame extraction processing to obtain at least one image frame to be processed and video attribute information corresponding to each image frame to be processed; the image frame to be processed is input into a trained entity information detection model to obtain entity association information, and the entity association information includes an entity image, entity feature description information and entity relationship description information; according to the entity image, the entity feature description information and the entity relationship description information, a first entity information vector and an identity identification vector are determined, and the first entity information vector and the identity identification vector are spliced to obtain a second entity information vector; each image frame to be processed, the corresponding video attribute information, the entity association information and the second entity information vector are associated and stored in a set database. In the above scheme, by extracting frames from the video to be processed, the image frames to be processed that need to be processed for entity information extraction and the video attribute information corresponding to each image frame to be processed can be determined, and the association reference information between the image frames to be processed and the video to be processed can be provided; by inputting the image frames to be processed into the trained entity information detection model, the entity information detection model can be effectively used to extract entity information without manual recognition or labeling, thereby improving processing efficiency; by determining the first entity information vector and the identity identification vector, unique identity labeling is performed on different entities, which can help distinguish different entities and improve the distinction of entities; by associating and storing each image frame to be processed, the corresponding video attribute information, entity association information, and the second entity information vector in a set database, the entity information corresponding to the video to be processed can be effectively stored, providing strong data support for subsequent video analysis, retrieval, and recommendation applications.
[0109] In a specific implementation, a process for processing a video query request is described. Figure 6 , which is a flowchart of a specific implementation process of processing a video query request provided by an embodiment of the present application, after each image frame to be processed, the corresponding video attribute information, the entity association information and the second entity information vector are associated and stored in a set database, such as Figure 6 As shown, the specific implementation steps for processing a video query request are as follows:
[0110] Step S151: upon receiving a video query request sent by a client, extract target entity information and target relationship information from the video query request, and perform vector conversion processing on the target entity information and the target relationship information to obtain a first target entity information vector and a target relationship vector.
[0111] The target entity information may be entity images or structured description information provided by the user, for example, entity A: male, red clothes, medium build, or entity B: male, blue clothes, fat build. The target relationship information may be spatial relationships or action relationships between target entities provided by the user, for example, entity A is sitting in the taxi of entity B.
[0112] Step S152: input the first target entity information vector into the entity identity confidence model to obtain a target identity identification vector, and concatenate the first target entity information vector, the target relationship vector and the target identity identification vector to obtain a second target entity information vector.
[0113] Step S153: searching the database based on the second target entity information vector to obtain at least one target image frame, corresponding target video attribute information and target entity association information.
[0114] Among them, by searching the data block based on the second target entity information vector, one or more target image frames with the highest correlation with the entity that the user expects to query can be determined. The specific retrieval method can be based on vector similarity calculations such as cosine similarity and Euclidean distance to determine the degree of matching.
[0115] Step S154: sending at least one target image frame, corresponding target video attribute information and target entity association information to the client.
[0116] Therefore, by sending at least one target image frame, corresponding target video attribute information and target entity association information to the client, the user can be effectively fed back in which target image frame the expected queried entity appears, and based on the target video attribute information, the target image frame can be specifically located from which video and at which specific time position in the video.
[0117] In another specific embodiment, a process of searching a database based on a second target entity information vector to obtain at least one target image frame, corresponding target video attribute information and target entity association information is described. Please refer to Figure 7 , which is a flowchart of a specific implementation process of retrieving a target image frame, corresponding target video attribute information and target entity association information provided by an embodiment of the present application, such as Figure 7 As shown, the specific implementation steps of searching the database based on the second target entity information vector to obtain at least one target image frame, corresponding target video attribute information and target entity association information are as follows:
[0118] Step S1531: Calculate the similarity between the second target entity information vector and the existing entity information vectors stored in the database, and select a preset number of candidate entity information vectors that meet the highest similarity from the existing entity information vectors.
[0119] Step S1532: query the database to obtain target image frames associated with multiple candidate entity information vectors, corresponding target video attribute information and target entity association information.
[0120] It is worth noting that by calculating the similarity between the second target entity information vector and the existing entity information vector, the matching degree between the second target entity information vector and the existing entity information vector can be effectively measured. By selecting a preset number of candidate entity information vectors that meet the highest similarity from the existing entity information vectors, it is equivalent to selecting the first few entity information vectors that have the highest matching degree with the second target entity information vector, providing the user with multiple target image frames that are closest to the entity description content, the corresponding target video attribute information and the target entity association information, and meeting the user's video query needs based on the entity dimension.
[0121] Figure 8 This is a structural block diagram of a video information processing device provided in an embodiment of the present application. The device is configured to execute the video information processing method provided in the above embodiment and has the corresponding functional modules and beneficial effects of the execution method. Figure 8 As shown, the device specifically includes:
[0122] The video frame extraction processing unit 110 is configured to obtain a video to be processed, and perform frame extraction processing on the video to be processed to obtain at least one image frame to be processed and video attribute information corresponding to each image frame to be processed;
[0123] The entity information determination unit 120 is configured to input the image frame to be processed into the trained entity information detection model to obtain entity association information, where the entity association information includes entity image, entity feature description information and entity relationship description information;
[0124] The entity information vector determining unit 130 is configured to determine a first entity information vector and an identity vector according to the entity image, the entity feature description information and the entity relationship description information, and concatenate the first entity information vector and the identity vector to obtain a second entity information vector;
[0125] The video information association storage unit 140 is configured to associate and store each to-be-processed image frame, the corresponding video attribute information, the entity association information and the second entity information vector in a set database.
[0126] In the above scheme, by extracting frames from the video to be processed, the image frames to be processed that need to be processed for entity information extraction and the video attribute information corresponding to each image frame to be processed can be determined, and the association reference information between the image frames to be processed and the video to be processed can be provided; by inputting the image frames to be processed into the trained entity information detection model, the entity information detection model can be effectively used to extract entity information without manual recognition or labeling, thereby improving processing efficiency; by determining the first entity information vector and the identity identification vector, unique identity labeling is performed on different entities, which can help distinguish different entities and improve the distinction of entities; by associating and storing each image frame to be processed, the corresponding video attribute information, entity association information, and the second entity information vector in a set database, the entity information corresponding to the video to be processed can be effectively stored, providing strong data support for subsequent video analysis, retrieval, and recommendation applications.
[0127] In a possible embodiment, the entity information vector determining unit 130 includes:
[0128] A vector conversion module is configured to perform vector conversion processing on the entity image, entity feature description information and entity relationship description information to obtain an entity image vector, an entity feature vector and an entity relationship vector;
[0129] A vector concatenation module is configured to concatenate the entity image vector, the entity feature vector and the entity relationship vector to obtain a first entity information vector;
[0130] The identification vector generation module is configured to concatenate the entity image vector and the entity feature vector to obtain a model input vector, and input the model input vector into the trained entity identity confidence model to obtain an identity identification vector.
[0131] In a possible embodiment, the device further includes:
[0132] The query request processing unit is configured as follows:
[0133] When receiving a video query request sent by a client, extracting target entity information and target relationship information from the video query request, and performing vector conversion processing on the target entity information and the target relationship information to obtain a first target entity information vector and a target relationship vector;
[0134] The first target entity information vector is input into the entity identity confidence model to obtain the target identity identification vector, and the first target entity information vector, the target relationship vector and the target identity identification vector are concatenated to obtain the second target entity information vector
[0135] Searching a database based on the second target entity information vector to obtain at least one target image frame, corresponding target video attribute information and target entity association information;
[0136] At least one target image frame, corresponding target video attribute information and target entity association information are sent to the client.
[0137] In a possible embodiment, the query request processing unit includes:
[0138] An information vector screening module, configured to calculate similarity between the second target entity information vector and existing entity information vectors stored in the database, and screen out a preset number of candidate entity information vectors that meet the highest similarity from the existing entity information vectors;
[0139] The information query module is configured to query the database to obtain target image frames associated with multiple candidate entity information vectors, corresponding target video attribute information and target entity association information.
[0140] In a possible embodiment, the device further includes:
[0141] Model training unit, configured as:
[0142] Obtain multiple sample feature information vectors;
[0143] The pre-built entity identity confidence model is pre-trained based on multiple sample feature information vectors.
[0144] In a possible embodiment, the device further includes:
[0145] The sample vector generation unit is configured as follows:
[0146] Acquire a plurality of sample image frames;
[0147] Input the sample image frame into the trained entity information detection model to obtain the sample entity image and sample entity feature information;
[0148] The sample entity image and the sample entity feature information are respectively subjected to vector conversion processing to obtain a sample entity image vector and a sample entity feature vector, and the sample entity image vector and the sample entity feature vector are concatenated to obtain a sample feature information vector.
[0149] In a possible embodiment, the device further includes:
[0150] Sample image frame generation unit, configured as:
[0151] Acquire multiple first historical videos, and pre-process the first historical videos to obtain second historical videos;
[0152] The second historical video is subjected to frame extraction processing to obtain a plurality of sample image frames.
[0153] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the device includes a processor 210, a memory 220, an input device 230, and an output device 240; the number of processors 210 in the device can be one or more. Fig. 9 A processor 210 is taken as an example; the processor 210, the memory 220, the input device 230 and the output device 240 in the device can be connected via a bus or other means. Fig. 9 The example of connecting via a bus is taken. The memory 220, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the video information processing method in the embodiment of the present application. The processor 210 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 220, that is, implements the above-mentioned video information processing method. The input device 230 can be configured to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device 240 may include display devices such as display screens.
[0154] The electronic device provided above can be used to execute the video information processing method provided in any of the above embodiments, and has corresponding functions and beneficial effects.
[0155] An embodiment of the present application also provides a non-volatile storage medium containing computer executable instructions, which, when executed by a computer processor, is configured to execute a video information processing method described in the above embodiment, which includes: obtaining a video to be processed, extracting frames of the video to be processed to obtain at least one image frame to be processed and video attribute information corresponding to each image frame to be processed; inputting the image frame to be processed into a trained entity information detection model to obtain entity association information, wherein the entity association information includes an entity image, entity feature description information and entity relationship description information; determining a first entity information vector and an identity identification vector based on the entity image, the entity feature description information and the entity relationship description information, and splicing the first entity information vector and the identity identification vector to obtain a second entity information vector; and associating and storing each image frame to be processed, the corresponding video attribute information, the entity association information and the second entity information vector in a set database.
[0156] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media or optical storage; registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., embodied as a computer program) that can be executed by one or more processors.
[0157] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application, whose computer executable instructions are not limited to the above video information processing method, can also execute related operations in the video information processing method provided in any embodiment of the present application.
[0158] It is worth noting that in the embodiments of the above-mentioned electronic device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not configured to limit the protection scope of the embodiments of the present application.
[0159] It should be noted that the numbering of each step in this scheme is only used to describe the overall design framework of this scheme, and does not represent the necessary order relationship between the steps. On the basis that the overall implementation process conforms to the overall design framework of this scheme, it belongs to the protection scope of this scheme, and the order of precedence in the form of text when describing is not an exclusive limitation on the specific implementation process of this scheme. It should be understood by those skilled in the art that the embodiments of the present application can be provided as methods, systems, or computer program products. In a typical configuration, a computing device includes one or more processors (CPU), an input / output interface, a network interface, and a memory. The memory may include non-permanent memory in a computer-readable medium, a random access memory (RAM) and / or a non-volatile memory in the form of a read-only memory (ROM) or a flash memory (fl ash RAM). The memory is an example of a computer-readable medium.
[0160] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0161] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A video information processing method, characterized in that: include: Acquire a video to be processed, and perform frame extraction processing on the video to be processed to obtain at least one image frame to be processed and video attribute information corresponding to each image frame to be processed; Inputting the image frame to be processed into a trained entity information detection model to obtain entity association information, wherein the entity association information includes entity image, entity feature description information and entity relationship description information; Determine a first entity information vector and an identity vector according to the entity image, entity feature description information and entity relationship description information, and concatenate the first entity information vector and the identity vector to obtain a second entity information vector; Each of the to-be-processed image frames, the corresponding video attribute information, the entity association information and the second entity information vector are associated and stored in a set database.
2. The video information processing method according to claim 1, characterized in that: The determining of the first entity information vector and the identity vector according to the entity image, the entity feature description information and the entity relationship description information comprises: Respectively performing vector conversion processing on the entity image, entity feature description information and entity relationship description information to obtain an entity image vector, an entity feature vector and an entity relationship vector; The entity image vector, the entity feature vector and the entity relationship vector are concatenated to obtain a first entity information vector; The entity image vector and the entity feature vector are concatenated to obtain a model input vector, and the model input vector is input into a trained entity identity confidence model to obtain an identity identification vector.
3. The video information processing method according to claim 1, characterized in that: After storing each of the to-be-processed image frames, the corresponding video attribute information, the entity association information, and the second entity information vector in association with each other in a set database, the method further includes: When receiving a video query request sent by a client, extracting target entity information and target relationship information from the video query request, and performing vector conversion processing on the target entity information and the target relationship information to obtain a first target entity information vector and a target relationship vector; The first target entity information vector is input into the entity identity confidence model to obtain a target identity identification vector, and the first target entity information vector, the target relationship vector and the target identity identification vector are concatenated to obtain a second target entity information vector Searching the database based on the second target entity information vector to obtain at least one target image frame, corresponding target video attribute information and target entity association information; The at least one target image frame, corresponding target video attribute information and target entity association information are sent to the client.
4. The video information processing method according to claim 3, characterized in that: The step of searching the database based on the second target entity information vector to obtain at least one target image frame, corresponding target video attribute information and target entity association information includes: Calculating similarity between the second target entity information vector and existing entity information vectors stored in the database, and selecting a preset number of candidate entity information vectors that meet the highest similarity from the existing entity information vectors; The database is queried to obtain target image frames associated with the plurality of candidate entity information vectors, corresponding target video attribute information and target entity association information.
5. The video information processing method according to claim 2, characterized in that: Before inputting the model input vector into the trained entity identity confidence model, the method further includes: Obtain multiple sample feature information vectors; The pre-constructed entity identity confidence model is pre-trained based on the multiple sample feature information vectors.
6. The video information processing method according to claim 5, characterized in that: Before obtaining a plurality of sample feature information vectors, the method further includes: Acquire multiple sample image frames; Inputting the sample image frame into a trained entity information detection model to obtain a sample entity image and sample entity feature information; The sample entity image and the sample entity feature information are respectively subjected to vector conversion processing to obtain a sample entity image vector and a sample entity feature vector, and the sample entity image vector and the sample entity feature vector are spliced to obtain a sample feature information vector.
7. The video information processing method according to claim 6, characterized in that: Before acquiring a plurality of sample image frames, the method further includes: Acquire multiple first historical videos, and pre-process the first historical videos to obtain second historical videos; The second historical video is subjected to frame extraction processing to obtain a plurality of sample image frames.
8. A video information processing device, characterized in that: include: A video frame extraction processing unit is configured to obtain a video to be processed, and perform frame extraction processing on the video to be processed to obtain at least one image frame to be processed and video attribute information corresponding to each image frame to be processed; An entity information determination unit is configured to input the image frame to be processed into a trained entity information detection model to obtain entity association information, wherein the entity association information includes an entity image, entity feature description information, and entity relationship description information; an entity information vector determining unit, configured to determine a first entity information vector and an identity vector according to the entity image, the entity feature description information and the entity relationship description information, and concatenate the first entity information vector and the identity vector to obtain a second entity information vector; The video information association storage unit is configured to associate and store each of the to-be-processed image frames, the corresponding video attribute information, the entity association information and the second entity information vector in a set database.
9. An electronic device, comprising: one or more processors; A storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the video information processing method described in any one of claims 1-7.
10. A non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the video information processing method according to any one of claims 1 to 7 when executed by a computer processor.