Knowledge extraction method and device, electronic equipment and computer program product

Abstract

The invention provides a knowledge extraction method and device, electronic equipment and a computer program product, and the method comprises the steps: carrying out the retrieval in a first data set and a second data set of a target video according to a query statement, and obtaining a first retrieval result and a second retrieval result; the first data set comprises multiple pieces of data obtained after text content of the target video is segmented according to different contents, the second data set comprises description statements of video images in the target video, image content understanding is conducted on the video images corresponding to the first retrieval result and the second retrieval result, and image content understanding data is obtained; and determining knowledge extracted from the target video according to the first retrieval result and the image content understanding data. According to the arrangement, fine-grained content understanding can be only carried out on the part associated with the query statement, and fine-grained content understanding does not need to be carried out on the whole video, so that the resource consumption is less, and the knowledge extraction speed is high.

Description

Technical Field

[0001] The present application relates to the field of information retrieval technology, and in particular to a knowledge extraction method, device, electronic device and computer program product. Background Art

[0002] By retrieving information related to the question from external knowledge and inputting it as a prompt into the large language model, the reliability of the large language model's answer can be improved. However, when the external knowledge includes long videos, since long videos involve a lot of content and a large amount of information, in order to ensure the output effect of the large language model, it is necessary to perform fine-grained content understanding of the long videos and input the obtained knowledge as a prompt into the large language model, which consumes a lot of resources and takes longer to retrieve. Summary of the invention

[0003] In view of this, the present application proposes a knowledge extraction method, device, electronic device and computer program product to solve the problem in the prior art that when the external knowledge includes long videos, the resource consumption is high and the retrieval time is longer.

[0004] The technical solutions proposed in this application are as follows:

[0005] In a first aspect, an embodiment of the present application provides a knowledge extraction method, comprising:

[0006] According to the query statement, a first data set of the target video is searched to obtain a first search result, and a second data set of the target video is searched to obtain a second search result; the first data set includes a plurality of text data obtained by segmenting the text content of the target video according to different contents, and the second data set includes a description statement of the video image in the target video;

[0007] Performing image content understanding on the video images corresponding to the first search result and the second search result to obtain image content understanding data;

[0008] The knowledge extracted from the target video is determined based on the first retrieval result, the second retrieval result and the image content understanding data.

[0009] Furthermore, in the above method, the process of generating the first data set includes:

[0010] In the text content of the target video, a region where the blank duration is greater than the duration threshold is used as a demarcation region;

[0011] The text content of the target video is segmented according to the boundary area, and the first data set is determined according to the content obtained after segmentation; the text content of the target video includes the subtitles of the target video, or the text obtained by converting the audio of the target video into text.

[0012] Furthermore, in the above method, the process of generating the second data set includes:

[0013] Detecting image content of a video image in the target video; the image content of the video image includes an object in the video image and / or a scene in the video image;

[0014] The image content is expanded with associated words to obtain a description sentence of the video image, and the second data set is determined according to the description sentence.

[0015] Further, in the above method, determining the second data set according to the description statement includes:

[0016] The video image in the target video is segmented according to the boundary area to obtain a plurality of video slices; the boundary area includes an area in the text content of the target video where the blank time is greater than the time threshold, and the video image is obtained by extracting frames from the target video;

[0017] The description sentences of the video images in each video slice are combined to obtain a description word list of each video slice, and the description word lists are combined into the second data set.

[0018] Furthermore, in the method described above, the video image corresponding to the first search result includes a video slice having the same boundary area as the first search result; the video image corresponding to the second search result includes a video slice corresponding to the description word list where the second search result is located.

[0019] Furthermore, in the above method, searching in the first data set of the target video to obtain a first search result includes:

[0020] Convert the query statement into a vector statement;

[0021] According to the vector sentence, a search is performed in the first data set to obtain a first search result.

[0022] Furthermore, in the above method, searching in the second data set of the target video to obtain a second search result includes:

[0023] Generate a query word list of the query statement;

[0024] According to the query word list, a search is performed in the second data set to obtain a second search result.

[0025] Furthermore, in the above method, the query word list for generating the query statement includes:

[0026] Extracting keywords from the query statement and performing associated word expansion on the keywords to obtain a text word list;

[0027] According to the text vocabulary, searching is performed in the entity relationship of the target video to obtain a third search result; the third search result includes entities whose text similarity with the text vocabulary is greater than a similarity threshold, and the parent entity of the entity; the entity relationship of the target video is constructed based on the entities extracted from the text content of the target video;

[0028] The text vocabulary and the third search result are combined into the query vocabulary.

[0029] Furthermore, in the above method, performing image content understanding on the video images corresponding to the first search result and the second search result to obtain image content understanding data includes:

[0030] Removing duplicate images from the video images corresponding to the first search result and the second search result to obtain a target image; the duplicate images include video images with the same sequence number and / or video images with image similarity greater than a similarity threshold;

[0031] Perform image content understanding on the target image to obtain image content understanding data.

[0032] Furthermore, in the above method, determining the knowledge extracted from the target video according to the first search result, the second search result and the image content understanding data includes:

[0033] Determining text data corresponding to the second search result according to the video slice corresponding to the second search result;

[0034] Deduplication processing is performed on the text data corresponding to the second search result and the first search result to obtain a first target text;

[0035] Calculating similarities between multiple texts in the image content understanding data and the query statement, and extracting at least one text with the highest similarity as a second target text;

[0036] The first target text and the second target text are determined to be knowledge extracted from the target video.

[0037] In a second aspect, an embodiment of the present application provides a knowledge extraction device, including:

[0038] A retrieval unit, configured to search a first data set of a target video according to a query statement to obtain a first retrieval result, and to search a second data set of the target video to obtain a second retrieval result; the first data set includes a plurality of text data obtained by segmenting the text content of the target video according to different contents, and the second data set includes a description statement of a video image in the target video;

[0039] an understanding unit, configured to understand the image content of the video images corresponding to the first search result and the second search result, and obtain image content understanding data;

[0040] A determination unit is used to determine the knowledge extracted from the target video according to the first retrieval result, the second retrieval result and the image content understanding data.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0042] A memory and a processor; wherein the memory is used to store programs; and the processor is used to implement any of the methods described above by running the programs in the memory.

[0043] In a fourth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the computer program implements any one of the above methods. Optionally, the computer program can be stored in a readable storage medium of a computer device or in the cloud; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0044] The knowledge extraction method proposed in the present application can search in the first data set of the target video according to the query statement to obtain the first search result, and search in the second data set of the target video to obtain the second search result; the first data set includes multiple data obtained by segmenting the text content of the target video according to different contents, and the second data set includes description statements of the video image in the target video. The video images corresponding to the first search result and the second search result are subjected to image content understanding to obtain image content understanding data, and the knowledge extracted from the target video is determined based on the first search result and the image content understanding data. Such a setting can retrieve the part associated with the query statement from the target video based on the two dimensions of video image and video text content, and then only perform fine-grained content understanding on the part associated with the query statement, without the need to perform fine-grained content understanding on the entire video. Not only does it consume less resources, but it also has a fast knowledge extraction speed, thereby improving the retrieval speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0046] Figure 1 It is a flow chart of a knowledge extraction method provided in an embodiment of the present application.

[0047] Figure 2 It is a schematic diagram of the first data set and the second data set provided in an embodiment of the present application.

[0048] Figure 3 It is a structural diagram of an entity relationship provided in an embodiment of the present application.

[0049] Figure 4 It is a structural diagram of a knowledge extraction device provided in an embodiment of the present application.

[0050] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] With the rise of big model applications, major artificial intelligence companies have launched their own big model platforms, and users are becoming more and more accustomed to using big model technology to help them acquire knowledge, assist in office work, generate content, and improve work efficiency. Retrieval-augmented Generation (RAG) is one of the most widely used scenarios for big models. It greatly alleviates problems such as big model illusions, untimely information updates, and lack of knowledge in professional or private domains, and significantly improves the reliability of big models in knowledge base question and answer, web search, customer service, and other fields.

[0053] Most of the current RAG solutions are aimed at retrieval enhancement of text content and have begun to slowly shift to the multimodal field. Many companies have launched multimodal RAG systems that can simultaneously retrieve content in the form of sound, images, videos, etc., and send the retrieved relevant sound, image, video and other content together with the text content into the big model, which will understand and respond.

[0054] The existing common multimodal RAG solutions are mainly to optimize multimodal conversion and multimodal alignment. However, when the external knowledge includes long videos, since long videos involve a lot of content and a large amount of information, in order to ensure the output effect of the large language model, it is necessary to perform fine-grained content understanding of the long videos, which consumes a lot of resources and takes longer to retrieve. If only coarse-grained multimodal content understanding is performed, it will be difficult to obtain the details in the user's questions through retrieval, and there will be a certain sacrifice in the effect. Current technical solutions generally make a trade-off between effect and performance, sacrificing one indicator to achieve the other effect, and it is difficult to achieve dual-indicator optimization of efficiency and effect.

[0055] Based on this, the present application proposes a knowledge extraction method, device, electronic device and computer program product. This technical solution retrieves the part associated with the query statement from the target video based on the two dimensions of video image and video text content, and only performs fine-grained content understanding on the part associated with the query statement, thereby achieving the effect of improving the retrieval speed.

[0056] The present application embodiment provides a knowledge extraction method, which can be executed by an electronic device, which can be any device with data and instruction processing functions, such as a notebook computer, a tablet computer, a desktop computer, a mobile device (e.g., a mobile phone, a personal digital assistant, a dedicated messaging device) and other types of user terminals, or a combination of any two or more of these electronic devices, or a server. Figure 1 As shown, the method includes:

[0057] S101. According to a query statement, a first data set of a target video is searched to obtain a first search result, and a second data set of the target video is searched to obtain a second search result.

[0058] The query statement mentioned above refers to the question statement that the user needs to query. For example, when the user asks a related question using a large language model (LLM), the question content input by the user is the query statement of this embodiment.

[0059] The target video is a video that needs to be used for knowledge extraction based on the query statement. In some embodiments, any type of video can be used as the target video of this embodiment, such as a tutorial video, an entertainment video, a documentary, etc., which is not limited in this embodiment. The content of a tutorial video is generally long and involves a large amount of content information. Users often expect to perform more fine-grained retrieval and inquiry for a certain step. Therefore, in some embodiments, only the tutorial video is set as the target video.

[0060] After the target video is uploaded, the target video may be processed to obtain the first data set and the second data set of the target video. In some embodiments, the process of processing the target video to obtain the first data set and the second data set of the target video is:

[0061] For the first data set, the text content of the target video can be segmented according to different contents to obtain multiple data after segmentation, and the multiple data after segmentation are combined into the first data set. More specifically, the text content of the target video can be segmented according to different contents to obtain multiple text data after segmentation, and the multiple text data after segmentation are combined into the first data set. Among them, the subtitles in the target video can be used as the text content of the target video, and the voice content in the target video can also be converted into text, and the text obtained after text conversion is used as the text content of the target video.

[0062] In some embodiments, the text content of the target video is segmented according to the content using a content segmentation model. A large amount of text content can be obtained as training samples, and then each training sample is manually segmented according to the content to obtain training labels. During training, the training samples are input into the content segmentation model to obtain the prediction results output by the content segmentation model. By comparing the prediction results of the content segmentation model with the training labels, the parameters of the content segmentation model are adjusted, and then the above training process is repeated until the various indicators of the content segmentation model meet the requirements. Among them, the content segmentation model can be obtained based on the training of the large language model, which is not limited in this embodiment.

[0063] The text content of the target video is input into the trained content segmentation model to obtain multiple text data output by the content segmentation model.

[0064] Further, when the text content of the target video is segmented, the target video can be segmented according to the same segmentation point to obtain multiple video segments, and the text data obtained by segmentation corresponds to the video segments one by one. It should be noted that since the target video contains a large number of video images, wherein the video image refers to the video frame in the target video, and adjacent video images often have the same content, in order to reduce the workload, the present embodiment extracts the frame of the target video at a certain time interval, and segmentation is performed on the basis of the video image obtained after the frame extraction to obtain multiple video segments. Wherein, each video segment does not contain a continuous video, but a plurality of video images obtained after the frame extraction. The above time interval can be set according to the actual situation, for example, it is set to 5 seconds, and the present embodiment does not make a limitation.

[0065] For the second data set, video images in the target video can be extracted, and each video image can be described to obtain a description sentence of the video image, and the description sentences of the video image can be combined into the second data set. Since the target video contains a large number of video images, and adjacent video images often have the same content, if each video image is described, not only the workload is large, but also a large amount of repeated content may be obtained. Therefore, this embodiment describes the video images obtained after the frame extraction process in the above embodiment, obtains the description sentence of the video image, and combines the description sentence of the video image into the second data set.

[0066] In some embodiments, when describing a video image, a traditional target detection algorithm is used to perform object detection on different video images to obtain the objects in the video image, and the object names are used as description sentences of the video image.

[0067] According to the query statement, a search is performed in the first data set of the target video to obtain a first search result. Specifically, the similarity between the query statement and each text data in the first data set can be calculated, and the text data with a similarity threshold greater than a first set threshold is selected as the first search result. The first set threshold can be set according to actual conditions, and this embodiment does not limit it. In some embodiments, the similarity between the query statement and each text data in the first data set can be calculated by converting the query statement and each text data in the first data set into vectors and calculating the vector similarity.

[0068] According to the query statement, a search is performed in the second data set of the target video to obtain a second search result. Specifically, the similarity between the query statement and each description statement in the second data set can be calculated, and the description statement whose similarity threshold is greater than the second set threshold is selected as the second search result. The second set threshold can be set according to actual conditions and is not limited in this embodiment. In some embodiments, the similarity between the query statement and each description statement in the second data set can be calculated by converting the query statement and each description statement in the second data set into vectors and calculating the vector similarity.

[0069] Among them, the content in the first data set is obtained based on the text content of the target video, so it can be determined that the first data set describes the target video in the video text dimension. The content in the second data set is obtained by describing the video image, so it can be determined that the second data set describes the target video in the video image dimension. In this embodiment, the target video can be retrieved in multiple dimensions based on the two dimensions of video image and video text content, thereby more accurately retrieving content related to the query statement from the target video.

[0070] S102: Perform image content understanding on the video images corresponding to the first search result and the second search result to obtain image content understanding data.

[0071] In the above embodiment, when segmenting the text content of the target video, the target video is segmented at the same segmentation point to obtain multiple video segments, and the video segments correspond to the text data one by one. Moreover, through the above steps, it can be determined that the first search result refers to at least one text data among the multiple data obtained by segmenting the text content of the target video according to different contents, and it can be determined that the video image corresponding to the first search result is the video image in the video segment corresponding to the first search result.

[0072] Through the above steps, it can be determined that the second search result refers to the description sentence of the video image. In some embodiments, the video image corresponding to the second search result includes the video image described by the description sentence in the second search result.

[0073] Through the multimodal image-to-text model, detailed text is generated for the video images corresponding to the first search result and the second search result, and the multiple generated texts are used as image content understanding data. With this setting, detailed text can be generated only for the part of the video images in the target video that is associated with the query statement, which not only consumes less resources but also accelerates knowledge extraction.

[0074] S103: Determine knowledge extracted from the target video according to the first search result, the second search result and the image content understanding data.

[0075] The similarity between each generated text in the image content understanding data and the query statement can be calculated. Specifically, each generated text in the image content understanding data and the query statement can be converted into a vector, and the similarity between the vector corresponding to each generated text in the image content understanding data and the vector corresponding to the query statement is calculated, and the similarity is determined as the similarity between each generated text in the image content understanding data and the query statement, and one or more generated texts with the highest similarity are taken as the target generated text.

[0076] In the above embodiment, when segmenting the text content of the target video, the target video is segmented at the same segmentation point to obtain multiple video segments, and the video segments correspond to the text data one by one. Then, according to the video image corresponding to the second search result, the text data corresponding to the video image can be determined. The text data is merged with the first search result to obtain the target text.

[0077] The target generated text and the target text are determined as the knowledge extracted from the target video. In this way, the one or more generated texts and the target text with the highest correlation with the query statement are used as the knowledge extracted from the target video. The obtained knowledge content is more accurate and more correlated with the query statement. The reliability of the response content generated by the large language model based on the knowledge is higher. In addition, this method removes redundant data, and the processing speed of the large language model will be faster.

[0078] The above-mentioned knowledge extracted from the target video can be sent back to the large language model, so that the large language model can generate response content for the query statement based on the RAG technology and using the knowledge extracted from the target video.

[0079] In the above embodiment, according to the query statement, a search can be performed in the first data set of the target video to obtain a first search result, and a search can be performed in the second data set of the target video to obtain a second search result; the first data set includes multiple data obtained by segmenting the text content of the target video according to different contents, and the second data set includes a description statement of the video image in the target video, and the video image corresponding to the first search result and the second search result is subjected to image content understanding to obtain image content understanding data, and the knowledge extracted from the target video is determined according to the first search result and the image content understanding data. Such a setting can retrieve the part associated with the query statement from the target video based on the two dimensions of video image and video text content, and then only perform fine-grained content understanding on the part associated with the query statement, without the need to perform fine-grained content understanding on the entire video, which not only consumes less resources, but also has a fast knowledge extraction speed, thereby improving the search speed.

[0080] As an optional implementation, another embodiment of the present application discloses that the generation process of the first data set in the above embodiment may specifically include the following steps:

[0081] The area in the text content of the target video where the blank time is greater than the time threshold is used as a demarcation area; the text content of the target video is segmented according to the demarcation area, and a first data set is determined according to the content obtained after the segmentation; the text content of the target video includes the subtitles of the target video, or the text obtained by converting the audio of the target video into text.

[0082] In addition to processing the target video to obtain the first data set according to the above embodiment, the target video may also be processed to obtain the first data set in the following manner as described in this embodiment:

[0083] First, the demarcation area in the text content of the target video is determined. It should be noted that the text content of the target video includes the subtitles of the target video, or the text obtained by converting the audio of the target video into text. According to the distribution of the text content of the target video on the time axis of the target video, the area with a blank duration greater than the duration threshold is considered to be the demarcation area in the text content. The duration threshold can be set according to actual conditions, and this embodiment does not limit it.

[0084] Then, the text content of the target video is segmented according to the demarcation area, and the first data set is determined based on the content obtained after segmentation. In some embodiments, the text content of the target video is segmented according to the demarcation area to obtain multiple text data, and the text data are combined into the first data set. Since the retrieval is generally performed using a vector similarity method, in an optional embodiment, the text content of the target video is segmented according to the demarcation area to obtain multiple text data, the multiple text data are converted into vector data respectively, and the multiple text data and the vector data corresponding to the multiple text data are combined into the first data set.

[0085] In the above embodiment, the first data set can be constructed by cutting the text content of the target video, so as to retrieve the content associated with the query sentence from the target video in the video text dimension.

[0086] As an optional implementation, another embodiment of the present application discloses that the generation process of the second data set in the above embodiment may specifically include the following steps:

[0087] Detect the image content of the video image in the target video; the image content of the video image includes objects in the video image and / or scenes of the video image; expand the image content with associated words to obtain a description sentence of the video image, and determine the second data set based on the description sentence.

[0088] In addition to processing the target video to obtain the second data set according to the above embodiment, the target video may also be processed to obtain the second data set in the following manner as described in this embodiment:

[0089] Detect the image content of the video image in the target video. The image content of the video image includes objects in the video image and / or scenes in the video image. The objects in the video image can be detected by a traditional target detection algorithm, and the scenes in the video image can be classified by an image scene classification model.

[0090] More specifically, the target detection model can be trained in advance to identify objects in the image. Specifically, a large number of images can be obtained as training samples, and the objects in the images can be manually marked as training labels. During training, the training samples are input into the target detection model to obtain the detection results of the target detection model. By comparing the detection results of the target detection model with the training labels, the loss of the target detection model is determined. With the goal of reducing the loss of the target detection model, the parameters of the target detection model are adjusted. Then the above training process is repeated until the target detection model meets the training requirements. The above target detection model can be obtained by training based on any neural network model, or can be obtained by training based on a pre-trained model, which is not limited in this embodiment.

[0091] The image scene classification model can be trained in advance to classify the scenes of the video images. Specifically, a large number of images can be obtained as training samples, and the scene type of each image can be manually marked as a training label. During training, the training samples are input into the scene classification model to obtain the classification results of the scene classification model. By comparing the classification results of the scene classification model and the training labels, the loss of the scene classification model is determined. With the goal of reducing the loss of the scene classification model, the parameters of the scene classification model are adjusted. Then repeat the above training process until the scene classification model meets the training requirements. The above scene classification model can be obtained by training based on any neural network model, or can be obtained by training based on a pre-trained model, which is not limited in this embodiment.

[0092] The video image is input into the trained target detection model to obtain the object in the video image output by the target detection model; the video image is input into the trained image scene classification model to obtain the scene of the video image output by the image scene classification model.

[0093] The object name text in the video image and / or the scene name text in the video image may be determined as the image content of the video image. In some embodiments, the object name text in the video image and the scene name text in the video image are determined as the image content of the video image.

[0094] Then, the associated words of the image content of the video image are expanded, and other words that have semantic association with the image content of the video image are introduced to enrich the description sentence of the video image and improve the accuracy of the search.

[0095] In the above embodiment, the second data set can be obtained according to the image content of the video image, so as to retrieve the content associated with the query statement from the target video in the video image dimension.

[0096] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment determine the second data set according to the description statement, which may specifically include the following steps:

[0097] The video image in the target video is segmented according to the boundary area to obtain multiple video slices; the boundary area includes an area in the text content of the target video where the blank duration is greater than the duration threshold, and the video image is obtained after frame extraction from the target video; the description sentences of the video image in each video slice are combined to obtain a description word list for each video slice, and the description word lists are combined into a second data set.

[0098] Specifically, since the target video contains a large number of video images, and adjacent video images often have the same content, if each video image is described, not only the workload is large, but also a large amount of repeated content may be obtained. Therefore, this embodiment performs frame extraction processing on the target video at a certain time interval to obtain the video image after frame extraction. More specifically, in the above embodiment, detecting the image content of the video image in the target video refers to detecting the image content of the video image obtained after frame extraction processing in the target video, and the description sentence of the video image refers to the description sentence of the video image obtained after frame extraction processing.

[0099] The video image obtained after the frame extraction is segmented according to the boundary area to obtain a plurality of video slices, wherein each video slice does not contain a continuous video but a video image after the frame extraction process.

[0100] The demarcation area in this embodiment refers to the area in the text content of the target video recorded in the above embodiment, where the blank duration is greater than the duration threshold. It should be noted that for the video images located in the demarcation area, common picture vector models such as ImageNet, Vgg, etc. can be used to extract vectors, and the cosine similarity of any adjacent video image vectors can be calculated. If there is a pair of adjacent video image vectors whose chord similarity is greater than the set value, it is considered that the scene is switched, and the area between the pair of adjacent video images is used as the video demarcation point to split the video. If there are multiple pairs or 0 pairs of adjacent video image vectors whose chord similarity is greater than the set value, the midpoint of the text timeline in the demarcation area is used as the video demarcation point to split the video. The set value can be set according to the actual situation, and this embodiment does not limit it.

[0101] Then, the description sentences of the video images in each video slice are combined, and repeated description sentences are deleted to obtain a description word list of each video slice, and the description word lists are combined into a second data set.

[0102] In the above embodiment, the description sentences can be grouped according to the video slices to obtain a description word list for each video slice, and then the description word list is combined into a second data set, and the search can be performed according to the description word list. When the second search result includes some description sentences in a description word list, the video slice corresponding to the entire description word list can be used as the video image corresponding to the second search result, so that all related video images can be extracted to avoid omissions.

[0103] As an optional implementation, another embodiment of the present application discloses that the method of the above embodiment may further include the following steps:

[0104] The video image corresponding to the first search result includes a video slice having the same boundary area as the first search result; the video image corresponding to the second search result includes a video slice corresponding to the description word list where the second search result is located.

[0105] Specifically, the video image corresponding to the first search result in the above embodiment refers to a video slice having the same boundary area as the first search result; the video image corresponding to the second search result in the above embodiment includes the video slice corresponding to the description word list where the second search result is located.

[0106] In a specific embodiment, after the target video is uploaded, the target video may be processed as follows:

[0107] The target video is subjected to frame extraction processing at a certain time interval to obtain a video image of the target video.

[0108] Through the traditional target detection algorithm, the video image is subjected to object detection to obtain the object name text in the video image; through the image scene classification model, the video image is subjected to scene classification to obtain the scene name text in the video image.

[0109] The object name text and the scene name text in the video image are expanded with associated words to obtain a description sentence of the video image, and the corresponding relationship between each video image and its description sentence is stored.

[0110] The area in the text content of the target video where the blank time is greater than the time threshold is taken as the boundary area, and the text content and the video image of the target video are segmented according to the boundary area. After the text content of the target video is segmented, multiple text data are obtained, and the multiple text data constitute the first data set of the above embodiment; after the video image of the target video is segmented, multiple video slices are obtained, wherein the text data corresponds to the video slices one by one, and this embodiment preserves the correspondence between the text data and the video slices.

[0111] Each text data obtained after segmentation is vectorized and a vector index is established. The description sentences in the same video slice are de-duplicated and merged to obtain a description word list for each video slice, which is stored as an inverted index. The description word list for each video slice constitutes the second data set of the above embodiment.

[0112] In this way, the video content and text content in the same time range are merged into the same time slice range, such as Figure 2 As shown in the figure, by aligning the time axis in this way, the relevance between text content and video content can be strengthened in a small range, and the retrieval efficiency can be further improved in the subsequent retrieval process.

[0113] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment are to search in the first data set of the target video to obtain the first search result, which may specifically include the following steps:

[0114] The query statement is converted into a vector statement; and a search is performed in the first data set according to the vector statement to obtain a first search result.

[0115] When searching in the first data set using a query statement, the search can be performed by calculating vector similarity. Specifically, the query statement can be converted into a vector statement. If the first data set contains text data, the text data is converted into a vector statement, and the similarity between the vector statement corresponding to each text data in the first data set and the vector statement of the query statement is calculated as the similarity between each text data in the first data set and the query statement, and one or more text data with a similarity to the query statement greater than a set threshold is taken as the first search result. More specifically, the vector statement of the query statement can be used to search in the vector index of the above embodiment. The above-mentioned set threshold can be set according to actual conditions, and this embodiment does not limit it.

[0116] In the above embodiment, it is possible to search from the text dimension in the first data set and retrieve the content associated with the query statement from the target video.

[0117] As an optional implementation method, another embodiment of the present application discloses that the steps of the above embodiment perform a search in a second data set of the target video to obtain a second search result, which may specifically include the following steps: generating a query word list of the query statement; and performing a search in the second data set according to the query word list to obtain a second search result.

[0118] The query word list of the query statement may be determined first. In some embodiments, keywords of the query statement may be extracted as the query word list of the query statement. In other embodiments, in order to enrich the content of the query word list and improve the reliability of the search results, the query word list of the query statement may be generated according to the following steps:

[0119] Extract keywords from the query statement and expand the keywords with associated words to obtain a text vocabulary; based on the text vocabulary, search in the entity relationship of the target video to obtain a third search result; the third search result includes entities whose text similarity with the text vocabulary is greater than a similarity threshold, and the parent entity of the entity; the entity relationship of the target video is constructed based on the entities extracted from the text content of the target video; the text vocabulary and the third search result are combined into a query vocabulary.

[0120] In the embodiment of the present application, entity extraction is performed in advance based on the text content in the target video, and entity relationships are constructed. For example, Figure 3 Shown is a schematic diagram of entity relationships provided by an embodiment.

[0121] Specifically, extract the keywords in the query statement and expand the keywords with associated words and synonyms to obtain a text word list, which can enrich the search content to a certain extent. Then, based on the text word list, search in the entity relationship of the target video, use the retrieved entity and the parent entity of the entity as the third search result, and combine the text word list and the third search result into a query word list. In this way, in addition to expanding the query statement with associated words, it can also be expanded in combination with the entities in the target video, further improving the richness of the search content and the accuracy of the search results.

[0122] Based on the query word list, a search is performed in the second data set to obtain a second search result. According to the query word list, when searching in the second data set, the search can be performed by calculating the vector similarity. Specifically, the words in the query word list can be converted into vector sentences, and the description sentences in the second data set can also be converted into vector sentences. The similarity between the vector sentences corresponding to the words in the query word list and the vector sentences of the description sentences is calculated as the similarity between the words in the query word list and the description sentences, and the description word list containing one or more description sentences whose similarity with the words in the query word list is greater than the set threshold is taken as the second search result. More specifically, the vector sentences of the query word list can be used to search in the inverted index of the above embodiment. The above-mentioned set threshold can be set according to the actual situation, and this embodiment does not limit it.

[0123] In the above embodiment, it is possible to search from the image dimension in the second data set and retrieve the content associated with the query statement from the target video.

[0124] As an optional implementation, another embodiment of the present application discloses that performing image content understanding on the video images corresponding to the first search result and the second search result in the above embodiment to obtain image content understanding data may specifically include the following steps:

[0125] Remove duplicate images from the video images corresponding to the first retrieval result and the second retrieval result to obtain a target image; the duplicate images include video images with the same serial number, and / or video images whose image similarity is greater than a similarity threshold; perform image content understanding on the target image to obtain image content understanding data.

[0126] There may be some duplicate images in the video images corresponding to the first search result and the second search result. In order to improve the processing speed, the duplicate images in the video images corresponding to the first search result and the second search result may be removed first. Video images with the same sequence number are duplicate images, and / or video images with a similarity between the images greater than a similarity threshold are duplicate images.

[0127] In some embodiments, each video image may be first vectorized by a picture-to-quantity model to obtain a video image vector, and the similarity between corresponding video images may be determined by calculating the vector similarity between any two video image vectors. If the similarity between two video images is greater than a set similarity threshold, it means that the two video images contain the same or similar content, and the two video images are similar images. The similar image content is deduplicated, and only one of the similar images is retained.

[0128] After removing the image content, the image content can be understood for the target image to obtain image content understanding data.

[0129] Deduplicating video images before understanding the image content can reduce the workload of the image content understanding process and improve retrieval efficiency.

[0130] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment determine the knowledge extracted from the target video according to the first search result, the second search result and the image content understanding data, specifically including:

[0131] According to the video slice corresponding to the second search result, determine the text data corresponding to the second search result; deduplicate the text data corresponding to the second search result and the first search result to obtain the first target text; calculate the similarity between multiple texts and the query statement in the image content understanding data, and extract at least one text with the highest similarity as the second target text; determine the first target text and the second target text as knowledge extracted from the target video.

[0132] Specifically, in this embodiment, the text content and video image of the target video are segmented according to the boundary area.

[0133] After the text content of the target video is segmented, multiple text data are obtained, and after the video image of the target video is segmented, multiple video slices are obtained, wherein the text data corresponds to the video slices one by one. Therefore, after obtaining the second search result, the text data corresponding to the second search result can be determined based on the video slice corresponding to the second search result.

[0134] The text data corresponding to the first search result and the second search result are deduplicated to obtain a first target text, thereby avoiding knowledge omissions.

[0135] For the image content understanding data, the similarity between the multiple generated texts in the image content understanding data and the query statement can be calculated, for example, the vector similarity calculation is performed according to the description of the above embodiment, which is not repeated in this embodiment. One or more generated texts with the highest similarity to the query statement are taken as the second target text, and the specific number can be set according to the actual situation, which is not limited in this embodiment.

[0136] With this setting, one or more second target texts and the first target text that are most associated with the query statement are used as knowledge extracted from the target video. The obtained knowledge content is more precise and more strongly associated with the query statement. The reliability of the response content generated by the large language model based on this knowledge is higher. In addition, this method removes redundant data, and the processing speed of the large language model will be faster.

[0137] In a specific embodiment, the query statement may be processed as follows to obtain the knowledge in the target video:

[0138] The query statement is converted into a vector, and a search is performed in the first data set to obtain a first search result, and the video slice corresponding to the first search result is determined; the query statement is expanded to obtain a query word list, and the query word list is used to search in the second data set to obtain a second search result, and the video slice corresponding to the second search result is determined.

[0139] The video slice corresponding to the first search result and the video slice corresponding to the second search result are merged and deduplicated to obtain a target image, and the image content of the target image is understood to obtain image content understanding data. Figure 2 The corresponding relationship is used to determine the text content corresponding to the video slice corresponding to the second search result, and the text content is combined with the first search result to remove duplicates to obtain the target text. The knowledge extracted from the target video is determined based on the image content understanding data and the target text.

[0140] Corresponding to the above-mentioned knowledge extraction method, the present application embodiment also discloses a knowledge extraction device, see Figure 4 As shown, the device comprises:

[0141] The retrieval unit 100 is used to search a first data set of the target video according to a query statement to obtain a first retrieval result, and to search a second data set of the target video to obtain a second retrieval result; the first data set includes a plurality of text data obtained by segmenting the text content of the target video according to different contents, and the second data set includes a description statement of a video image in the target video;

[0142] An understanding unit 110 is used to understand the image content of the video images corresponding to the first search result and the second search result to obtain image content understanding data;

[0143] The determination unit 120 is used to determine the knowledge extracted from the target video according to the first search result, the second search result and the image content understanding data.

[0144] As an optional implementation, another embodiment of the present application discloses that the device of the above embodiment includes:

[0145] A generating unit is used to use the area in the text content of the target video where the blank duration is greater than the duration threshold as a demarcation area; segment the text content of the target video according to the demarcation area, and determine a first data set according to the content obtained after segmentation; the text content of the target video includes the subtitles of the target video, or the text obtained by converting the audio of the target video into text.

[0146] As an optional implementation method, another embodiment of the present application discloses that the generation unit of the above embodiment is also used to detect the image content of the video image in the target video; the image content of the video image includes objects in the video image, and / or the scene of the video image; the image content is expanded with associated words to obtain a description sentence of the video image, and the second data set is determined based on the description sentence.

[0147] As an optional implementation method, another embodiment of the present application discloses that the generation unit of the above embodiment is specifically used to segment the video image in the target video according to the demarcation area to obtain multiple video slices; the demarcation area includes an area in the text content of the target video where the blank duration is greater than the duration threshold, and the video image is obtained after frame extraction of the target video; the description sentences of the video image in each video slice are combined to obtain a description word list of each video slice, and the description word list is combined into a second data set.

[0148] As an optional implementation method, another embodiment of the present application discloses that the video image corresponding to the first search result includes a video slice having the same boundary area as the first search result; the video image corresponding to the second search result includes a video slice corresponding to the description word list where the second search result is located.

[0149] As an optional implementation, another embodiment of the present application discloses that the retrieval unit 100 of the above embodiment is specifically used to convert the query statement into a vector statement; based on the vector statement, a search is performed in the first data set to obtain a first search result.

[0150] As an optional implementation, another embodiment of the present application discloses that the retrieval unit 100 of the above embodiment is specifically used to generate a query word list of the query statement; based on the query word list, a search is performed in the second data set to obtain a second search result.

[0151] As an optional implementation method, another embodiment of the present application discloses that the retrieval unit 100 of the above embodiment is specifically used to extract keywords in the query statement and expand the keywords with associated words to obtain a text vocabulary; based on the text vocabulary, a search is performed in the entity relationship of the target video to obtain a third search result; the third search result includes entities whose text similarity with the text vocabulary is greater than a similarity threshold, and the parent entity of the entity; the entity relationship of the target video is constructed based on the entities extracted from the text content of the target video; the text vocabulary and the third search result are combined into a query vocabulary.

[0152] As an optional implementation method, another embodiment of the present application discloses that the understanding unit 110 of the above embodiment is specifically used to remove duplicate images in the video images corresponding to the first retrieval result and the second retrieval result to obtain a target image; the duplicate images include video images with the same serial number, and / or video images whose image similarity is greater than a similarity threshold; the target image is subjected to image content understanding to obtain image content understanding data.

[0153] As an optional implementation method, another embodiment of the present application discloses that the determination unit 120 of the above embodiment is specifically used to determine the text data corresponding to the second retrieval result based on the video slice corresponding to the second retrieval result; deduplicate the text data corresponding to the second retrieval result and the first retrieval result to obtain a first target text; calculate the similarity between multiple texts and the query statement in the image content understanding data, and extract at least one text with the highest similarity as the second target text; determine that the first target text and the second target text are knowledge extracted from the target video.

[0154] Specifically, the unit provided in this embodiment belongs to the same application concept as the method provided in the above embodiment of this application, can execute the method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For the technical details not described in detail in this embodiment, please refer to the specific processing content of the method provided in the above embodiment of this application, which will not be repeated here.

[0155] The functions implemented by the above units can be implemented by the same or different processors respectively, and the embodiments of the present application are not limited thereto.

[0156] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0157] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0158] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0159] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0160] An embodiment of the present application further provides a control device, which includes a processor and an interface circuit. The processor in the control device is connected to an input-output component through the interface circuit of the control device.

[0161] The input-output component specifically refers to a hardware component that enables a user to input information and output information to the user, such as a microphone, keyboard, handwriting tablet, touch screen, display, speaker, printer, etc.

[0162] The above-mentioned interface circuit can be any interface circuit that can realize the data communication function, for example, it can be a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0163] The processor in the control device is a circuit with signal processing capability, which improves the speed of knowledge extraction by executing any one of the knowledge extraction methods introduced in the above embodiments. The specific implementation of the processor can refer to the above processor implementation, and the embodiments of this application are not strictly limited.

[0164] When the control device is applied to a device with a human-computer interaction function, the input and output components of the control device may be input components and output components on the device, such as a microphone, a keyboard, a handwriting tablet, a touch screen, a display, an audio player, etc. At the same time, the processor of the control device may be a CPU or GPU, etc. provided by the device, and the interface circuit of the control device may be an interface circuit between the information input component of the device and a processor such as a CPU or GPU.

[0165] Corresponding to the above-mentioned knowledge extraction method, the present application embodiment also discloses an electronic device, see Figure 5 As shown, the electronic device includes:

[0166] Memory 200 and processor 210;

[0167] The memory 200 is connected to the processor 210 and is used to store programs;

[0168] The processor 210 is used to implement the knowledge extraction method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0169] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0170] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0171] A bus may include a pathway that transfers information between components of a computer system.

[0172] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0173] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0174] The memory 200 stores a program for executing the technical solution of the present application, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes computer operation instructions. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0175] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0176] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0177] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0178] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of the knowledge extraction method provided in the above embodiments of the present application.

[0179] In addition to the above methods and devices, the embodiments of the present application may also be a computer program product, which includes a computer program. When the computer program is run by a processor, the knowledge extraction method provided by any of the above embodiments of the present application may be executed. Optionally, the computer program may be stored in a readable storage medium or in the cloud of a computer device; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0180] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0181] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK).

[0182] In addition, the embodiments of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes each step of the knowledge extraction method provided in the above embodiments.

[0183] The computer readable storage medium may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0184] Specifically, the specific working contents of each part of the above-mentioned electronic device, computer program product and storage medium, as well as the specific processing contents when the computer program product or the computer program on the above-mentioned storage medium is executed by the processor, can be found in the contents of the various embodiments of the above-mentioned knowledge extraction method, and will not be repeated here.

[0185] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0186] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0187] The steps in the methods of each embodiment of the present application can be adjusted in sequence, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0188] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0189] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0190] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0191] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0192] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0193] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0194] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0195] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A knowledge extraction method, characterized in that: include: According to the query statement, a first data set of the target video is searched to obtain a first search result, and a second data set of the target video is searched to obtain a second search result; the first data set includes a plurality of text data obtained by segmenting the text content of the target video according to different contents, and the second data set includes a description statement of the video image in the target video; Performing image content understanding on the video images corresponding to the first search result and the second search result to obtain image content understanding data; The knowledge extracted from the target video is determined based on the first retrieval result, the second retrieval result and the image content understanding data.

2. The method according to claim 1, characterized in that The process of generating the first data set includes: In the text content of the target video, a region where the blank duration is greater than the duration threshold is used as a demarcation region; The text content of the target video is segmented according to the boundary area, and the first data set is determined according to the content obtained after segmentation; the text content of the target video includes the subtitles of the target video, or the text obtained by converting the audio of the target video into text.

3. The method according to claim 1, characterized in that The process of generating the second data set includes: Detecting image content of a video image in the target video; the image content of the video image includes an object in the video image and / or a scene in the video image; The image content is expanded with associated words to obtain a description sentence of the video image, and the second data set is determined according to the description sentence.

4. The method according to claim 3, characterized in that The determining the second data set according to the description statement includes: The video image in the target video is segmented according to the boundary area to obtain a plurality of video slices; the boundary area includes an area in the text content of the target video where the blank time is greater than the time threshold, and the video image is obtained by extracting frames from the target video; The description sentences of the video images in each video slice are combined to obtain a description word list of each video slice, and the description word lists are combined into the second data set.

5. The method according to claim 4, characterized in that The video image corresponding to the first search result includes a video slice having the same boundary area as the first search result; the video image corresponding to the second search result includes a video slice corresponding to the description word list where the second search result is located.

6. The method according to claim 1, characterized in that The step of searching in the first data set of the target video to obtain a first search result includes: Convert the query statement into a vector statement; According to the vector sentence, a search is performed in the first data set to obtain a first search result.

7. The method according to claim 1, characterized in that The searching in the second data set of the target video to obtain a second search result includes: Generate a query word list of the query statement; According to the query word list, a search is performed in the second data set to obtain a second search result.

8. The method according to claim 7, characterized in that The query word list for generating the query statement includes: Extracting keywords from the query statement and performing associated word expansion on the keywords to obtain a text word list; According to the text vocabulary, searching is performed in the entity relationship of the target video to obtain a third search result; the third search result includes entities whose text similarity with the text vocabulary is greater than a similarity threshold, and the parent entity of the entity; the entity relationship of the target video is constructed based on the entities extracted from the text content of the target video; The text vocabulary and the third search result are combined into the query vocabulary.

9. The method according to any one of claim 1, characterized in that: The performing image content understanding on the video images corresponding to the first search result and the second search result to obtain image content understanding data includes: Removing duplicate images from the video images corresponding to the first search result and the second search result to obtain a target image; the duplicate images include video images with the same sequence number and / or video images with image similarity greater than a similarity threshold; Perform image content understanding on the target image to obtain image content understanding data.

10. The method according to claim 1, characterized in that The step of determining the knowledge extracted from the target video according to the first search result, the second search result and the image content understanding data includes: Determining text data corresponding to the second search result according to the video slice corresponding to the second search result; Deduplication processing is performed on the text data corresponding to the second search result and the first search result to obtain a first target text; Calculating similarities between multiple texts in the image content understanding data and the query statement, and extracting at least one text with the highest similarity as a second target text; The first target text and the second target text are determined to be knowledge extracted from the target video.

11. A knowledge extraction device, characterized in that: include: A retrieval unit, configured to search a first data set of a target video according to a query statement to obtain a first retrieval result, and to search a second data set of the target video to obtain a second retrieval result; the first data set includes a plurality of text data obtained by segmenting the text content of the target video according to different contents, and the second data set includes a description statement of a video image in the target video; A comprehension unit, configured to perform image content comprehension on the video images corresponding to the first search result and the second search result to obtain image content comprehension data; A determination unit is used to determine the knowledge extracted from the target video according to the first retrieval result, the second retrieval result and the image content understanding data.

12. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 10 by running the program in the memory.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images