Knowledge extraction method, device, electronic device and storage medium

Abstract

The present disclosure provides a knowledge extraction method, device, electronic device, and storage medium. The method includes: obtaining a video to be processed and text information of the video to be processed; extracting multiple candidate knowledge texts from the video to be processed and the text information; obtaining the video features of the video to be processed and the text features of each candidate knowledge text; determining the similarity between the video features and each text feature through a first model, and determining multiple target candidate knowledge texts from the multiple candidate knowledge texts based on the similarity; determining the degree of matching between each target candidate knowledge text and the video content of the video to be processed through a second model, and determining the target knowledge text from the multiple target candidate knowledge texts based on the matching degree. This method improves the matching efficiency and recall rate of video and knowledge text, while also improving the accuracy of video and knowledge text matching.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a knowledge extraction method, device, electronic device, and computer-readable storage medium. Background Art

[0002] In the field of general knowledge short videos, identifying the content of these videos is extremely valuable. By mining the knowledge content of general knowledge videos, it can directly influence various fields such as video analysis, user behavior analysis, and recommendation.

[0003] In related technologies, information extraction is usually performed on text information such as titles and descriptions of short videos, and selection is performed only based on the similarity between texts without considering the semantic information of the entire video. As a result, the accuracy of the matching between the extracted knowledge text and the video is poor.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] The embodiments of the present disclosure provide a knowledge extraction method, device, electronic device and computer-readable storage medium. On the one hand, the method improves the matching efficiency and recall rate of video and knowledge text, and on the other hand, it takes into account the matching degree of the semantic information of the video and the knowledge text, thereby improving the accuracy of the matching between the video and the knowledge text.

[0006] An embodiment of the present disclosure provides a knowledge extraction method, which includes: obtaining a video to be processed and text information of the video to be processed; extracting multiple candidate knowledge texts from the video to be processed and the text information; obtaining video features of the video to be processed and text features of each candidate knowledge text; determining the similarity between the video features and each text feature through a first model, and determining multiple target candidate knowledge texts from the multiple candidate knowledge texts based on the similarity; determining the degree of matching between each target candidate knowledge text and the video content of the video to be processed through a second model, and determining the target knowledge text from the multiple target candidate knowledge texts based on the matching degree.

[0007] In some exemplary embodiments of the present disclosure, obtaining the video features of the video to be processed includes: encoding each video frame in the video to be processed to obtain video frame features of each video frame of the video to be processed; encoding text information of the video to be processed to obtain video text features of the video to be processed; and determining the video frame features of each video frame and the video text features as the video features of the video to be processed.

[0008] In some exemplary embodiments of the present disclosure, the candidate knowledge text includes candidate terms and candidate knowledge points of the candidate terms; obtaining the text features of each candidate knowledge text includes: for each candidate knowledge text, splicing the candidate terms and knowledge points of the candidate terms of the candidate knowledge text to obtain a spliced ​​knowledge text; encoding the spliced ​​knowledge text to obtain the text features of the candidate knowledge text.

[0009] In some exemplary embodiments of the present disclosure, the first model is trained in the following manner: obtaining a first training sample, wherein the first training sample includes multiple first training video text groups, and the training videos and training knowledge texts in the first training video text groups correspond to each other; training the first initial model through the first training video text group, and outputting the first similarity between the training videos and training knowledge texts in the first training video text group; if the first similarity is less than or equal to a first preset value, adjusting the model parameters of the first initial model, and training the adjusted first initial model again through the first training video text group until the first similarity output by the adjusted first initial model is greater than the first preset value, and determining the first model based on the model parameters of the adjusted first initial model.

[0010] In some exemplary embodiments of the present disclosure, the first model is trained in the following manner: obtaining a second training sample, wherein the second training sample includes multiple second training video text groups, and the training videos and training knowledge texts in the second training video text groups do not correspond; training the second initial model through the second training video text group, and outputting the second similarity between the training videos and training knowledge texts in the second training video text group; if the second similarity is greater than or equal to a second preset value, adjusting the model parameters of the second initial model, and training the adjusted second initial model again through the second training video text group until the second similarity output by the adjusted second initial model is less than the second preset value, and determining the first model based on the model parameters of the adjusted second initial model.

[0011] In some exemplary embodiments of the present disclosure, the target candidate knowledge text includes target candidate terms and target candidate knowledge points of the target candidate terms, and the target knowledge text includes target terms and target knowledge points of the target knowledge text; determining the degree of matching between each target candidate knowledge text and the video content of the video to be processed includes: determining a first degree of matching between each target candidate term and the video content of the video to be processed; determining a second degree of matching between each target candidate knowledge point and the video content of the video to be processed; determining the target knowledge text from the multiple target candidate knowledge texts based on the degree of matching includes: determining the target term and the target knowledge point of the target term from the multiple target candidate terms and the target candidate knowledge points of the target candidate terms based on the first degree of matching and the second degree of matching.

[0012] In some exemplary embodiments of the present disclosure, the second model is trained in the following manner: obtaining a third training sample, wherein the third training sample includes multiple third training video text groups and matching degree labels between the training videos and training knowledge texts in the third training video text group; training the third initial model through the third training video text group, and outputting the predicted matching degree between the training videos and training knowledge texts in the third training video text group; if the predicted matching degree is inconsistent with the matching degree label, adjusting the model parameters of the third initial model, and training the adjusted third initial model again through the third training video text group until the predicted matching degree output by the adjusted third initial model is consistent with the matching degree label, and determining the second model based on the model parameters of the adjusted third initial model.

[0013] In some exemplary embodiments of the present disclosure, the training knowledge text includes training terms and training knowledge points of the training terms, the matching degree labels include a first matching degree label between the training video and the training terms, and a second matching degree label between the training video and the training knowledge points, and the predicted matching degree includes a first predicted matching degree between the training video and the training terms, and a second predicted matching degree between the training video and the training knowledge points.

[0014] In some exemplary embodiments of the present disclosure, the candidate knowledge text includes candidate terms and candidate knowledge points of the candidate terms; extracting multiple candidate knowledge texts from the video to be processed and the text information includes: extracting multiple candidate terms from the text information of the video to be processed; and obtaining candidate knowledge points of each candidate term based on the position of each candidate term in the text information.

[0015] An embodiment of the present disclosure provides a knowledge extraction device, comprising: an acquisition module, configured to execute acquisition of a video to be processed and text information of the video to be processed; an extraction module, configured to execute extraction of multiple candidate knowledge texts from the video to be processed and the text information; the acquisition module is further configured to execute acquisition of video features of the video to be processed and text features of each candidate knowledge text; a determination module, configured to execute determination of the similarity between the video features and each text feature through a first model, and determine multiple target candidate knowledge texts from the multiple candidate knowledge texts based on the similarity; the determination module is further configured to execute determination of the degree of matching between each target candidate knowledge text and the video content of the video to be processed through a second model, and determine the target knowledge text from the multiple target candidate knowledge texts based on the matching degree.

[0016] An embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement any of the above-mentioned knowledge extraction methods.

[0017] An embodiment of the present disclosure provides a computer-readable storage medium. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can perform any of the above-mentioned knowledge extraction methods.

[0018] An embodiment of the present disclosure provides a computer program product, including a computer program, which implements any of the above-mentioned knowledge extraction methods when executed by a processor.

[0019] The knowledge extraction method provided by the embodiment of the present disclosure, on the one hand, extracts multiple candidate knowledge texts from the video to be processed and the text information of the video to be processed, determines the similarity between the video features of the video to be processed and the text features of each candidate knowledge text through a first model, determines multiple target candidate knowledge texts from the multiple candidate knowledge texts based on the similarity, comprehensively considers the multimodal information of the video to be processed, and improves the matching efficiency and recall rate of the video and the knowledge text; on the other hand, determines the matching degree between each target candidate knowledge text and the content of the video to be processed through a second model, determines the target knowledge text from the multiple target candidate knowledge texts based on the matching degree, considers the matching degree of the semantic information of the video and the knowledge text, and improves the accuracy of the matching between the video and the knowledge text. In addition, the method can extract the text information of the video to be processed and the video to be processed to obtain candidate knowledge texts without pre-defining the knowledge text (also called label), so that the method can be applied not only to closed domain knowledge extraction scenarios, but also to open domain knowledge extraction scenarios.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0022] Figure 1 A schematic diagram showing an exemplary system architecture to which the knowledge extraction method according to an embodiment of the present disclosure can be applied.

[0023] Figure 2 The figure is a flowchart of a knowledge extraction method according to an exemplary embodiment.

[0024] Figure 3 The figure is a flowchart of another knowledge extraction method according to an exemplary embodiment.

[0025] Figure 4 The figure is a flowchart of another knowledge extraction method according to an exemplary embodiment.

[0026] Figure 5 The figure is a block diagram of a knowledge extraction apparatus according to an exemplary embodiment.

[0027] Figure 6 It is a schematic structural diagram showing an electronic device suitable for implementing the exemplary embodiments of the present disclosure according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repetitive description thereof will be omitted.

[0029] The features, structures or characteristics described in the present disclosure may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0030] The accompanying drawings are merely schematic illustrations of the present disclosure. Identical reference numerals in the drawings denote identical or similar components, and thus their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in at least one hardware module or integrated circuit, or in different networks and / or processor devices and / or microcontroller devices.

[0031] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all content and steps, nor must they be executed in the order described. For example, some steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0032] In this specification, the terms "a", "an", "the", "said" and "at least one" are used to indicate the presence of at least one element / component / etc.; the terms "comprising", "including" and "having" are used to express open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first", "second" and "third" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0033] Figure 1 A schematic diagram showing an exemplary system architecture to which the knowledge extraction method according to an embodiment of the present disclosure can be applied.

[0034] like Figure 1 As shown, the system architecture may include a server 101, a network 102, a terminal device 103, a terminal device 104, and a terminal device 105. The network 102 is used as a medium for providing a communication link between the terminal device 103, the terminal device 104, or the terminal device 105 and the server 101. The network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0035] Server 101 may be a server that provides various services, such as a background management server that supports devices operated by users using terminal device 103, terminal device 104, or terminal device 105. The background management server may analyze and process received data such as requests, and feed back the processing results to terminal device 103, terminal device 104, or terminal device 105.

[0036] Terminal device 103, terminal device 104 and terminal device 105 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a wearable smart device, a virtual reality device, an augmented reality device, etc., but are not limited thereto.

[0037] In the disclosed embodiment, for example, a video producer can upload a video to be processed and text information of the video to be processed through the terminal device 103; the server 101 can obtain the video to be processed and text information of the video to be processed; extract multiple candidate knowledge texts from the video to be processed and text information; obtain the video features of the video to be processed and text features of each candidate knowledge text; determine the similarity between the video features and each text feature through a first model; determine multiple target candidate knowledge texts from multiple candidate knowledge texts based on the similarity; determine the degree of match between each target candidate knowledge text and the video content of the video to be processed through a second model; determine the target knowledge text from multiple target candidate knowledge texts based on the degree of match. The server 101 can send the video to be processed and its target knowledge text to the terminal device 104 and the terminal device 105 of the video viewer, and the video viewer can understand the video to be processed in advance through the target knowledge text of the video to be processed and choose to watch the video to be processed.

[0038] It should be understood that Figure 1 The number of terminal devices 103, terminal devices 104, terminal devices 105, networks 102 and servers 101 is merely illustrative. Server 101 may be a physical server, a server cluster consisting of multiple servers, or a cloud server. Depending on actual needs, it may have any number of terminal devices, networks and servers.

[0039] Below, the steps of the knowledge extraction method in the exemplary embodiment of the present disclosure will be described in more detail with reference to the accompanying drawings and embodiments. The method provided in the exemplary embodiment of the present disclosure can be executed by any electronic device, such as the above-mentioned Figure 1 The server and / or terminal device in the present disclosure is not limited to this.

[0040] Figure 2 The figure is a flowchart of a knowledge extraction method according to an exemplary embodiment.

[0041] like Figure 2 As shown, the method provided by the embodiment of the present disclosure may include the following steps.

[0042] In step S210, the video to be processed and text information of the video to be processed are obtained.

[0043] Among them, the video to be processed can refer to any video uploaded by the user. The video to be processed can be a short video recorded by the user, or a video produced by the user through an application such as editing software, or a movie video, TV series video, variety show video, etc. on the Internet. This disclosure does not make any specific restrictions on this.

[0044] For example, the videos to be processed could be general knowledge short videos on short video platforms. General knowledge short videos refer to short videos that explain knowledge. Unlike entertainment short videos, general knowledge short videos focus on life skills, scientific knowledge, learning techniques, and sharing experiences. Users can learn knowledge by watching general knowledge short videos.

[0045] The text information of the video to be processed may be the video title, video subtitles, video description text and other information edited by the user for the video to be processed, or the video title, video subtitles, video description text and other information automatically generated by the server based on the video frame of the video to be processed.

[0046] In the embodiment of the present disclosure, the server can automatically obtain a batch of multiple videos to be processed and text information of each video to be processed. The processing process of the video to be processed is described below using one video to be processed as an example, but the present disclosure is not limited to this.

[0047] In step S220, a plurality of candidate knowledge texts are extracted from the video and text information to be processed.

[0048] In the disclosed embodiment, a plurality of candidate knowledge texts may be extracted from the video to be processed, and a plurality of candidate knowledge texts may be extracted from the text information of the video to be processed.

[0049] Among them, knowledge text refers to words that can represent the knowledge content of the video to be processed, and candidate knowledge text refers to words that are preliminarily determined and are candidates to become the target knowledge text of the video to be processed.

[0050] In the disclosed embodiment, the knowledge text may include entries and their knowledge points (i.e., "entry-knowledge point" pairs). An entry may be a concept (e.g., plant) and / or entity (e.g., rose) representing the knowledge content of the video to be processed. A knowledge point may refer to a certain aspect of knowledge points corresponding to a concept or entity. For example, the knowledge point corresponding to the entry "plant" is "history" (i.e., "plant-history"). For example, the knowledge point corresponding to the entry "rose" is "cutting" (i.e., rose-cutting). Entries and knowledge points may be in a one-to-one correspondence, and an entry and its corresponding knowledge point may constitute an entry-knowledge point pair.

[0051] In the embodiment of the present disclosure, the candidate knowledge text may include candidate entries and candidate knowledge points of the candidate entries (i.e., "candidate entry-candidate knowledge point" pairs). The candidate entry refers to the entry that is preliminarily determined to be a candidate to become the target entry of the video to be processed, and the candidate knowledge point refers to the knowledge point that is preliminarily determined to be a candidate to become the target knowledge point of the video to be processed.

[0052] In the embodiment of the present disclosure, multiple candidate entries and candidate knowledge points corresponding to each candidate entry are extracted from the video to be processed and the text information of the video to be processed. For example, multiple candidate knowledge texts can be extracted from the video to be processed and the text information about the "nutritional value of apples", where one of the candidate entries is "apple" and the candidate knowledge point corresponding to the candidate entry is "nutritional value".

[0053] In an exemplary embodiment, multiple candidate knowledge texts are extracted from the text information of the video to be processed, including: extracting multiple candidate terms from the text information of the video to be processed; and obtaining candidate knowledge points of each candidate term based on the position of each candidate term in the text information.

[0054] Specifically, a keyword recall method can be used to extract multiple candidate terms from the text information of the video to be processed, for example, by using TF-IDF (Term Frequency-Inverse Document Frequency) and TextRank (text ranking algorithm) to obtain the keywords mentioned in the text information as candidate terms; or, a hashtag (topic tag) recall method can be used to extract multiple candidate terms from the text information of the video to be processed, for example, entities are obtained from the topic tags (Hashtags) of the video as candidate terms.

[0055] Specifically, multiple candidate terms can be extracted from the text information of the video to be processed, and then the candidate knowledge points corresponding to each candidate term can be obtained by using keyword recall. For example, the sentence where the candidate term is located, the previous sentence and the next sentence of the sentence are selected from the text information as candidate texts, and the keywords of these candidate texts are selected as candidate knowledge points corresponding to the candidate terms; or, the candidate knowledge points corresponding to each candidate term can be obtained by using the longest common subsequence method, for example, the sentences where the candidate term is located are selected, and the longest common subsequence of the word segmentation results of these sentences are calculated as the candidate knowledge points corresponding to the candidate terms.

[0056] In the embodiment of the present disclosure, a sequence annotation model may also be pre-trained so as to use the annotation model to automatically extract multiple candidate knowledge texts from the video to be processed and the text information of the video to be processed.

[0057] Specifically, batches of training videos (e.g., general knowledge short videos), along with modal information such as training text corresponding to the training videos, and training knowledge text corresponding to the training videos (e.g., training terms and training knowledge points) can be obtained through short video platforms. The training knowledge text can be obtained by manually annotating the training videos. The sequence labeling model is trained using the training text and training knowledge text corresponding to the training videos to obtain a trained sequence labeling model.

[0058] Specifically, the text information of the video to be processed can include the video caption, OCR (Optical Character Recognition) text, ASR (Automatic Speech Recognition) text, etc. The trained sequence labeling model can automatically identify the terms in these texts as candidate terms. Similarly, the trained sequence labeling model can automatically identify the knowledge points corresponding to the terms as candidate knowledge points.

[0059] In the disclosed embodiment, after extracting a plurality of candidate knowledge texts, the candidate knowledge texts may be recalled through keyword mining and entity naming recognition, thereby enriching the obtained candidate knowledge texts.

[0060] In step S230, the video features of the video to be processed and the text features of each candidate knowledge text are obtained.

[0061] In the embodiment of the present disclosure, feature extraction can be performed on the video to be processed to obtain the video features of the video to be processed; feature extraction can be performed on each candidate knowledge text to obtain the text features of each candidate knowledge text.

[0062] Here, step S230 can be performed using the first model or other neural network models.

[0063] In an exemplary embodiment, obtaining video features of a video to be processed includes: encoding each video frame in the video to be processed to obtain video frame features of each video frame of the video to be processed; encoding text information of the video to be processed to obtain video text features of the video to be processed; and determining the video frame features and video text features of each video frame as the video features of the video to be processed.

[0064] Specifically, the i-th video to be processed can be represented by V (i) =[v (i) ,t (i) ] means, where v (i) Represents the video frame in the video to be processed, t (i) Represents the video text information of the video to be processed; the candidate entry corresponding to the i-th video to be processed can be represented by T (i) Indicates that the candidate knowledge points corresponding to the i-th video to be processed can be represented by C (i) express.

[0065] Specifically, the video frame v (i) Represented as a picture frame Use ResNet to encode each picture frame and obtain K1 picture frame vectors Take the average of K1 picture frame vectors to obtain the video frame feature V of the video to be processed v (i) .

[0066] Specifically, the video text information t (i) Represented as a sequence of characters Encode the character vector of length K2 using BERT (Bidirectional Encoder Representation from Transformers) Take the average of K2 character vectors to get the video text feature V of the video to be processed t (i) ; The video frame feature V v (i) With video text features V t (i) Combined to get the video feature V (i) =V v (i) V t (i) .

[0067] In an exemplary embodiment, the candidate knowledge texts include candidate entries and candidate knowledge points of the candidate entries; obtaining the text features of each candidate knowledge text includes: for each candidate knowledge text, splicing the candidate entries and knowledge points of the candidate entries to obtain a spliced ​​knowledge text; encoding the spliced ​​knowledge text to obtain the text features of the candidate knowledge text.

[0068] Specifically, each candidate term and each candidate term's knowledge point can be spliced ​​together, for example, into a BERT format: "[CLS] T (i) [SEP]C (i) ", input it into the pre-trained BERT model, and obtain the representation vector of the [CLS] position as the text feature of the candidate term and the knowledge point of the candidate term (i.e., candidate term-candidate knowledge point representation)

[0069] In step S240 , the similarity between the video features and each text feature is determined by the first model, and based on the similarity, a plurality of target candidate knowledge texts are determined from the plurality of candidate knowledge texts.

[0070] Generally speaking, after extracting a video to be processed and its text information (i.e., after step S230), a lot of candidate knowledge texts can be obtained. At this time, the candidate knowledge texts can be filtered through the first model to select the target candidate text from the candidate knowledge texts.

[0071] In an embodiment of the present disclosure, the first model can be a fast rough sorting model, which is used to screen out multiple target candidate texts from multiple candidate knowledge texts, where the number of candidate knowledge texts is greater than the target candidate texts, for example, the number of candidate knowledge texts is 200 and the number of target candidate knowledge texts is 20.

[0072] In the embodiment of the present disclosure, the first model can be a multimodal feature similarity matching model based on metric learning. It only needs to obtain video features and various text features (such as features of several terms-knowledge points) to perform rough sorting based on the similarity between the video features and the various text features.

[0073] For example, the text features of 200 videos to be processed are obtained through the above steps. The video feature V of the video to be processed can be determined by the first model (i) The similarity between these 200 text features

[0074] In the embodiment of the present disclosure, a similarity threshold can be set to determine the candidate knowledge texts with a similarity greater than the similarity threshold as the target candidate knowledge texts; a preset number of target candidate knowledge texts can also be set to determine a preset number of target candidate knowledge texts from multiple candidate knowledge texts; a preset ratio can also be set to determine a preset ratio of target candidate knowledge texts from multiple candidate knowledge texts.

[0075] Among them, the target candidate knowledge text may include target candidate terms and target candidate knowledge points of the target candidate terms; based on the similarity between the video features and the text features of each candidate term and the knowledge points of each target candidate term, multiple target candidate terms and the target candidate knowledge text of each target candidate term can be determined from multiple candidate terms and the candidate knowledge points of each candidate term.

[0076] In step S250, the matching degree between each target candidate knowledge text and the video content of the video to be processed is determined by the second model, and the target knowledge text is determined from the multiple target candidate knowledge texts based on the matching degree.

[0077] In the disclosed embodiment, the first model is used to filter target candidate knowledge texts from candidate knowledge texts according to similarity, thereby improving the matching efficiency and recall rate of videos and knowledge texts.

[0078] When the semantic information of the video is taken into consideration, for a video to be processed, a candidate knowledge text with a high degree of similarity to the video is not necessarily a knowledge text that can better express the content of the video to be processed.

[0079] For example, the video to be processed is a video about the development history of automobiles, and the candidate knowledge text includes "car model". Although the similarity between the video features of the video about the development history of automobiles and the text features of "car model" may be very high, the degree of match between the video content of the video and "car model" is not high. If "car model" is used as the target knowledge text of the video, it cannot accurately express the content of the video.

[0080] Therefore, the embodiment of the present disclosure uses the second model (fine sorting model) to re-screen the target candidate texts filtered by the first model, and selects target knowledge texts that are more compatible with the video content of the video to be processed from the target candidate texts.

[0081] In the disclosed embodiment, the degree of matching between each target candidate knowledge text and the video content of the video to be processed can be determined through the trained second model.

[0082] Specifically, the training video and the training knowledge text corresponding to the training video can be used as labels to input the training video and the training knowledge text corresponding to the training video into the second model to be trained. The matching of the training video and the training knowledge text corresponding to the training video can be determined manually by determining whether the video content of the training video and the training knowledge text match.

[0083] In the embodiment of the present disclosure, the second model can be a BERT model based on a prompt template, or it can be other neural network models, which is not limited in the present disclosure.

[0084] In an exemplary embodiment, the target candidate knowledge text includes a target candidate term and a target candidate knowledge point of the target candidate term, and the target knowledge text includes a target term and a target knowledge point of the target knowledge text; determining the degree of matching between each target candidate knowledge text and the video content of the video to be processed includes: determining a first degree of matching between each target candidate term and the video content of the video to be processed; determining a second degree of matching between each target candidate knowledge point and the video content of the video to be processed; determining the target knowledge text from multiple target candidate knowledge texts based on the degree of matching includes: determining the target term and the target knowledge point of the target term from multiple target candidate terms and the target candidate knowledge points of the target candidate terms based on the first degree of matching and the second degree of matching.

[0085] Specifically, the first matching degree between each target candidate term and the video content of the video to be processed can be determined respectively; and the second matching degree between each target candidate knowledge point and the video content of the video to be processed can be determined, so that the target term and the target knowledge point of the target term can be determined from multiple target candidate terms and their target candidate knowledge points based on the first matching degree between each target candidate term and the video content of the video to be processed, and the second matching degree between each target candidate knowledge point and the video content of the video to be processed.

[0086] The target knowledge text is a knowledge text determined from multiple target knowledge texts and used to describe the content of the video to be processed. There can be one or more target knowledge texts. The target knowledge text can include a target term and a target knowledge point for the target term. For example, the term "apple" and the knowledge point "nutritional value" can be used as the target term and target knowledge point for a video about the nutritional value of apples.

[0087] In the embodiment of the present disclosure, a matching degree threshold can be set to determine the target candidate knowledge text with a matching degree greater than the matching degree threshold as the target knowledge text; a preset number of target knowledge texts can also be set (the preset number of target knowledge texts is less than the preset number of the above-mentioned target candidate knowledge texts) to determine a preset number of target knowledge texts from multiple target candidate knowledge texts; a preset ratio can also be set to determine a preset ratio of target knowledge texts from multiple target candidate knowledge texts.

[0088] The knowledge extraction method provided by the embodiment of the present disclosure, on the one hand, extracts multiple candidate knowledge texts from the video to be processed and the text information of the video to be processed, determines the similarity between the video features of the video to be processed and the text features of each candidate knowledge text through a first model, determines multiple target candidate knowledge texts from the multiple candidate knowledge texts based on the similarity, comprehensively considers the multimodal information of the video to be processed, and improves the matching efficiency and recall rate of the video and the knowledge text; on the other hand, determines the matching degree between each target candidate knowledge text and the content of the video to be processed through a second model, determines the target knowledge text from the multiple target candidate knowledge texts based on the matching degree, considers the matching degree of the semantic information of the video and the knowledge text, and improves the accuracy of the matching between the video and the knowledge text. In addition, the method can extract the text information of the video to be processed and the video to be processed to obtain candidate knowledge texts without pre-defining the knowledge text (also called label), so that the method can be applied not only to closed domain knowledge extraction scenarios, but also to open domain knowledge extraction scenarios.

[0089] In addition, the knowledge extraction method provided by the embodiments of the present disclosure can generate more conceptual tags related to the videos produced by video creators, thereby attracting more traffic; for video consumers, it can have a more conceptual representation of the videos they browse, making the portrayal of their interests more accurate, so that more relevant videos can be recommended to them.

[0090] Figure 3 The figure is a flowchart of another knowledge extraction method according to an exemplary embodiment.

[0091] like Figure 3 As shown, Figure 2 The first model in the knowledge extraction method shown can be trained through the following steps.

[0092] In step S310, a first training sample is obtained, wherein the first training sample includes a plurality of first training video text groups, and the training videos and training knowledge texts in the first training video text groups correspond to each other.

[0093] In the disclosed embodiment, multiple training videos and training knowledge texts corresponding to each training video can be obtained, and each training video and its corresponding training knowledge text are used as positive examples (i.e., the first training sample), and each training video and its non-corresponding training knowledge text are used as negative examples (i.e., the second training sample described below). The first training sample and / or the second training sample are used to train the first initial model so that when the input data is a positive example, the output similarity value is maximized, and when the input data is a negative example, the output similarity value is minimized.

[0094] In a batch of training data, for example, there are N training videos and training knowledge texts corresponding to each training video. Then, these N training videos and the training knowledge texts corresponding to each training video can be combined into N groups of positive examples, and the N training videos and the other (N-1) training knowledge texts that do not correspond to them can be combined into N (N-1) groups of negative examples.

[0095] It should be noted that in the embodiment of the present disclosure, the first training sample can be used alone to train the first initial model to obtain the first model (i.e., steps S310 to S330), or the second training sample can be used alone to train the second initial model to obtain the first model (i.e., steps S340 to S360); the first training sample and the second training sample can be used successively (or alternately) to train the first initial model; when the first training sample and the second training sample are used successively (or alternately) to train the first initial model, the first training sample can be used to train the first initial model first, and then the second training sample can be used to train the first initial model; the second training sample can be used to train the first initial model first, and then the first training sample can be used to train the first initial model, and the present disclosure does not limit this.

[0096] The training knowledge text may include training terms and training knowledge points of the training terms.

[0097] Specifically, you can get the format training data pairs (wherein the training terms and the training knowledge points of the training terms can be manually annotated); in the training process of the first initial model, the i-th training video can be (i) =[v (i) ,t (i) ] means, where v (i) represents the video frame in the training video, t (i) Represents the video text information of the training video; the training term corresponding to the i-th training video can be represented by T (i) Indicates that the training knowledge point corresponding to the i-th video to be processed can be represented by C (i) Indicates that it can be used for training term T (i)Expand and obtain the subtitle, description information, category information, etc. of the entry and add it to T (i) ; Training knowledge points C for training terms (i) Perform data enhancement, select aliases and synonyms of knowledge points and add them to C (i) middle.

[0098] Specifically, the video frame v (i) Represented as a picture frame Use ResNet to encode each picture frame and obtain K1 picture frame vectors Take the average of K1 picture frame vectors to obtain the video frame feature V of the training video v (i) .

[0099] Specifically, the video text information t of the training video can be (i) Represented as a sequence of characters Encode with BERT to get a character vector of length K2 Take the average of K2 character vectors to get the video text feature V of the training video t (i) ; The video frame feature V v (i) With video text features V t (i) Combined to get the video feature V (i) =V v (i) V t (i) .

[0100] Specifically, the training terms and their training knowledge points can be concatenated, for example, into a BERT format: “[CLS] T (i) [SEP]C (i) ", input it into the pre-trained BERT model, and obtain the representation vector of the [CLS] position as the text feature of the training term and its training knowledge point (i.e. training term-training knowledge point)

[0101] In step S320, the first initial model is trained using the first training video text group, and a first similarity between the training video and the training knowledge text in the first training video text group is output.

[0102] In an embodiment of the present disclosure, the first initial model can be a neural network model, the training videos and training knowledge texts in the first training video text group correspond to each other, the training videos and training knowledge texts in the first training video text group are input as positive examples into the first initial model for training, and the first similarity between the training videos and training knowledge texts in the first training video text group is output.

[0103] Specifically, the similarity between the i-th training video and the i-th training knowledge text can be calculated by the first initial model

[0104] In step S330, if the first similarity is less than or equal to the first preset value, the model parameters of the first initial model are adjusted, and the adjusted first initial model is trained again through the first training video text group until the first similarity output by the adjusted first initial model is greater than the first preset value, and the first model is determined based on the model parameters of the adjusted first initial model.

[0105] When the first training video text group is input into the first initial model, the similarity between the training video and the training knowledge text in the first training video text group is calculated by the first initial model as the first similarity; by adjusting the model parameters of the first initial model, the first similarity output by the adjusted first initial model is maximized, and the model of the adjusted first initial model is used as the first model.

[0106] For example, the first preset value can be used as a label. If the first similarity is less than or equal to the first preset value, the model parameters of the first initial model are adjusted so that the first similarity output by the adjusted first initial model is greater than the first preset value.

[0107] The first preset value may be a value close to the right end point of the similarity interval. For example, if the similarity interval is [0, 1], the first preset value may be set to 0.9, for example.

[0108] In an exemplary embodiment, Figure 2 The first model in the knowledge extraction method shown can also be trained through the following steps.

[0109] In step S340, a second training sample is obtained, wherein the second training sample includes a plurality of second training video text groups, and the training videos and the training knowledge texts in the second training video text groups do not correspond to each other.

[0110] In the disclosed embodiment, multiple training videos and training knowledge texts corresponding to each training video can be obtained, and each training video and its non-corresponding training knowledge text can be used as a negative example (i.e., a second training sample). The second initial model is trained using the second training sample so that when the input data is a negative example, the output similarity value is minimized.

[0111] In a batch of training data, for example, there are N training videos and training knowledge texts corresponding to each training video. Then, the N training videos and the other (N-1) non-corresponding training knowledge texts can be combined into N(N-1) groups of negative examples.

[0112] In step S350, the second initial model is trained using the second training video text group, and a second similarity between the training videos and the training knowledge texts in the second training video text group is output.

[0113] In an embodiment of the present disclosure, the second initial model can be a neural network model. The training videos and training knowledge texts in the second training video text group do not correspond to each other. The training videos and training knowledge texts in the second training video text group are input as positive examples into the second initial model for training, and the second similarity between the training videos and training knowledge texts in the second training video text group is output.

[0114] Specifically, the similarity between the i-th training video and the j-th training knowledge text can be calculated by the second initial model: Where i≠j.

[0115] It should be noted that in the embodiment of the present disclosure, the second initial model can be trained using the second training sample alone to obtain the first model (i.e., steps S340 to S360); or the first initial model can be trained using the first training sample and the second training sample successively (or alternately); when the first initial model is trained using the first training sample and the second training sample successively (or alternately), the second initial model is the first initial model in steps S340 to S360.

[0116] In step S360, if the second similarity is greater than or equal to the second preset value, the model parameters of the second initial model are adjusted, and the adjusted second initial model is trained again through the second training video text group until the second similarity output by the adjusted second initial model is less than the second preset value, and the first model is determined based on the model parameters of the adjusted second initial model.

[0117] In an embodiment of the present disclosure, when the second training video text group is input into the second initial model, the similarity between the training video and the training knowledge text in the second training video text group is calculated by the second initial model as the second similarity; by adjusting the model parameters of the second initial model, the second similarity output by the adjusted second initial model is minimized, and the model of the adjusted second initial model is used as the first model.

[0118] For example, the second preset value can be used as a label. If the second similarity is greater than or equal to the second preset value, the model parameters of the second initial model are adjusted so that the second similarity output by the adjusted second initial model is less than the second preset value.

[0119] The second preset value may be a value close to the left end point of the similarity interval. For example, if the similarity interval is [0, 1], the second preset value may be set to 0.1, for example.

[0120] It should be noted that the present disclosure does not limit the order of the steps of inputting positive examples into the first initial model for training (i.e., steps S310 to S330) and the steps of inputting negative examples into the second initial model for training (i.e., steps S340 to S360). That is, the positive examples can be first input into the first initial model for training and then the negative examples can be input into the second initial model for training, or the negative examples can be first input into the second initial model for training and then the positive examples can be input into the first initial model for training, or the training can be performed alternately.

[0121] The knowledge extraction method provided by the embodiment of the present disclosure maximizes the first similarity determined by the first initial model by inputting corresponding training videos and training knowledge texts into a first initial model for training; minimizes the second similarity determined by the second initial model by inputting non-corresponding training videos and training knowledge texts into a second initial model for training; the first model obtained through the above training process can more accurately determine the similarity between the video to be processed and the candidate knowledge text, while improving the matching efficiency of the video to be processed and the candidate knowledge text.

[0122] Figure 4 The figure is a flowchart of another knowledge extraction method according to an exemplary embodiment.

[0123] like Figure 4 As shown, Figure 2 The second model in the knowledge extraction method shown can be trained through the following steps.

[0124] In step S410, a third training sample is obtained, wherein the third training sample includes a plurality of third training video text groups and matching degree labels between the training videos and the training knowledge texts in the third training video text groups.

[0125] The training knowledge text includes training terms and training knowledge points of the training terms.

[0126] In the embodiment of the present disclosure, the training videos and training knowledge texts in the third training sample and the training videos and training knowledge texts in the first training sample and the second training sample can be the same batch of training videos and training knowledge texts, or they can be different training videos and training knowledge texts. The present disclosure does not limit this.

[0127] In the embodiment of the present disclosure, the format can be obtained The training data pair (i.e. the third training video text group) and Figure 3 The difference between the first training video text combination and the second training video text combination in the embodiment is that the training videos, training terms and training knowledge points in the training data pairs here may be matched or may not be matched.

[0128] In the embodiment of the present disclosure, each third training video text group may include a training video and a training knowledge text, and the matching degree labels of the training video and the training knowledge text in each third training video text group may be obtained through manual annotation.

[0129] Specifically, the input data of the third initial model can be expressed as: where Y (i) A label indicating the degree of matching between the training video and the training knowledge text (i.e., the training term-training knowledge point pair).

[0130] In an exemplary embodiment, the matching degree labels include a first matching degree label between the training video and the training term, and a second matching degree label between the training video and the training knowledge point.

[0131] For example, Y (i) =[ya (i) ,yb (i) ], ya (i) Indicates the first matching degree label between the training video and the training term, yb (i) Indicates the second matching degree label between the training video and the training knowledge point.

[0132] In step S420, the third initial model is trained using the third training video text group, and the predicted matching degree between the training video and the training knowledge text in the third training video text group is output.

[0133] In an exemplary embodiment, the predicted matching degree includes a first predicted matching degree between the training video and the training terms, and a second predicted matching degree between the training video and the training knowledge points.

[0134] Specifically, the training video and training terms are input into the third initial model, and the first predicted matching degree between the training video and the training terms is output; the training video and training knowledge points are input into the third initial model, and the second predicted matching degree between the training video and the training knowledge points is output.

[0135] In step S430, if the predicted matching degree and the matching degree label are inconsistent, the model parameters of the third initial model are adjusted, and the adjusted third initial model is trained again through the third training video text group until the predicted matching degree and the matching degree label output by the adjusted third initial model are consistent, and the second model is determined based on the model parameters of the adjusted third initial model.

[0136] In the embodiment of the present disclosure, the training video and the training knowledge text can be input into the third initial model to obtain the predicted matching degree between the training video and the training knowledge text; the matching degree label between the training video and the training knowledge text is used as a label, and the parameters of the third initial model are adjusted according to the predicted matching degree and the corresponding matching degree label, so that the predicted matching degree output by the adjusted third initial model is consistent with the corresponding matching degree label (or similar), and the model of the adjusted third initial model is used as the second model.

[0137] In an exemplary embodiment, a third initial model is trained using a third training video text group, and a predicted degree of matching between the training video and the training knowledge text in the third training video text group is output, including: obtaining a prompt template, the prompt template including a text portion, a first portion to be filled in, and a second portion to be filled in; encoding the training video in the third training video text group, and filling the encoded training video into the first portion to be filled in, filling the training knowledge text in the third training video text group into the text portion, and filling the matching degree label into the second portion to be filled in to generate prompt data; processing the prompt data using the third initial model to obtain a predicted degree of matching between the training video and the training knowledge text in the third training video text group.

[0138] For example, construct a prompt input template for input data:

[0139] [CLS][MASK]t (i) [SEP]T (i) [SEP]C (i) [SEP]=>[MASK][MASK]

[0140] Among them, the first [MASK] on the left is the first part to be filled in, and the two [MASK] on the right are the second part to be filled in. The first part to be filled in and the second part to be filled in can be filled with the matching degree label of the training video and the training term, and the matching degree label of the training video and the training knowledge point, respectively. (i) 、T (i) 、C (i) is the text part, where t (i) represents the video text of the training video, T (i) Represents the text of the training term, C (i) The text representing the training knowledge point.

[0141] Similar to the multimodal representation in the first model training stage, the video frame v can be trained by ResNet (i) Represent it, then replace the representation of the first [MASK] with the result, and then input the entire prompt (i.e. prompt data) into the BERT encoder to predict the values ​​of the last two [MASK], which respectively indicate whether the training video-training term matches, and whether the training video-training knowledge point matches.

[0142] The knowledge extraction method provided by the embodiment of the present disclosure inputs the training video and the training knowledge text into the third initial model for training, so that the predicted matching degree between the training video and the training knowledge text determined by the third initial model is consistent with (or similar to) the corresponding matching degree label; the second model obtained through the above training process can more accurately determine the target knowledge text with a higher degree of matching with the video content of the video to be processed, thereby improving the accuracy of the matching between the video to be processed and the target knowledge text.

[0143] It should also be understood that the above is merely intended to help those skilled in the art better understand the embodiments of the present disclosure, and is not intended to limit the scope of the embodiments of the present disclosure. Based on the above examples, those skilled in the art can obviously make various equivalent modifications or variations. For example, certain steps in the above method may be unnecessary, or certain new steps may be added. Or any combination of any two or more of the above embodiments. Such modifications, variations, or combinations also fall within the scope of the embodiments of the present disclosure.

[0144] It should also be understood that the above description of the embodiments of the present disclosure focuses on emphasizing the differences between the various embodiments. The same or similar points that are not mentioned can be referenced to each other. For the sake of brevity, they will not be repeated here.

[0145] It should also be understood that the size of the sequence numbers of the above processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present disclosure.

[0146] It should also be understood that in the various embodiments of the present disclosure, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments based on their internal logical relationships.

[0147] The above describes in detail an example of the knowledge extraction method provided by the present disclosure. It is understandable that, in order to implement the above functions, the computer device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present disclosure.

[0148] The following are embodiments of the apparatus disclosed herein, which can be used to implement the method embodiments disclosed herein. For details not disclosed in the apparatus embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0149] Figure 5 FIG. 1 is a block diagram of a knowledge extraction device according to an exemplary embodiment. Figure 5 The knowledge extraction device 500 may include an acquisition module 510 , an extraction module 520 and a determination module 530 .

[0150] Among them, the acquisition module 510 is configured to execute acquisition of the video to be processed and the text information of the video to be processed; the extraction module 520 is configured to execute extraction of multiple candidate knowledge texts from the video to be processed and its text information; the acquisition module 510 is also configured to execute acquisition of the video features of the video to be processed and the text features of each candidate knowledge text; the determination module 530 is configured to execute determination of the similarity between the video features and each text feature through a first model, and determine multiple target candidate knowledge texts from the multiple candidate knowledge texts based on the similarity; the determination module 530 is also configured to execute determination of the degree of matching between each target candidate knowledge text and the video content of the video to be processed through a second model, and determine the target knowledge text from the multiple target candidate knowledge texts based on the matching degree.

[0151] In some exemplary embodiments of the present disclosure, the acquisition module 510 is further configured to perform encoding processing on each video frame in the video to be processed to obtain video frame features of each video frame of the video to be processed; perform encoding processing on the text information of the video to be processed to obtain video text features of the video to be processed; and determine the video frame features of each video frame and the video text features as the video features of the video to be processed.

[0152] In some exemplary embodiments of the present disclosure, the candidate knowledge text includes candidate terms and candidate knowledge points of the candidate terms; the acquisition module 510 is also configured to perform splicing processing on the candidate terms and knowledge points of the candidate terms for each candidate knowledge text to obtain a spliced ​​knowledge text; and perform encoding processing on the spliced ​​knowledge text to obtain text features of the candidate knowledge text.

[0153] In some exemplary embodiments of the present disclosure, the first model is trained in the following manner: obtaining a first training sample, wherein the first training sample includes multiple first training video text groups, and the training videos and training knowledge texts in the first training video text groups correspond to each other; training the first initial model through the first training video text group, and outputting the first similarity between the training videos and training knowledge texts in the first training video text group; if the first similarity is less than or equal to a first preset value, adjusting the model parameters of the first initial model, and training the adjusted first initial model again through the first training video text group until the first similarity output by the adjusted first initial model is greater than the first preset value, and determining the first model based on the model parameters of the adjusted first initial model.

[0154] In some exemplary embodiments of the present disclosure, the first model is trained in the following manner: obtaining a second training sample, wherein the second training sample includes multiple second training video text groups, and the training videos and training knowledge texts in the second training video text groups do not correspond; training the second initial model through the second training video text group, and outputting the second similarity between the training videos and training knowledge texts in the second training video text group; if the second similarity is greater than or equal to a second preset value, adjusting the model parameters of the second initial model, and training the adjusted second initial model again through the second training video text group until the second similarity output by the adjusted second initial model is less than the second preset value, and determining the first model based on the model parameters of the adjusted second initial model.

[0155] In some exemplary embodiments of the present disclosure, the target candidate knowledge text includes target candidate terms and target candidate knowledge points of the target candidate terms, and the target knowledge text includes target terms and target knowledge points of the target knowledge text; the determination module 530 is also configured to determine a first degree of matching between each target candidate term and the video content of the video to be processed; determine a second degree of matching between each target candidate knowledge point and the video content of the video to be processed; the determination module 530 is also configured to determine the target term and the target knowledge point of the target term from the multiple target candidate terms and the target candidate knowledge points of the target candidate terms based on the first degree of matching and the second degree of matching.

[0156] In some exemplary embodiments of the present disclosure, the second model is trained in the following manner: obtaining a third training sample, wherein the third training sample includes multiple third training video text groups and matching degree labels between the training videos and training knowledge texts in the third training video text group; training the third initial model through the third training video text group, and outputting the predicted matching degree between the training videos and training knowledge texts in the third training video text group; if the predicted matching degree is inconsistent with the matching degree label, adjusting the model parameters of the third initial model, and training the adjusted third initial model again through the third training video text group until the predicted matching degree output by the adjusted third initial model is consistent with the matching degree label, and determining the second model based on the model parameters of the adjusted third initial model.

[0157] In some exemplary embodiments of the present disclosure, the training knowledge text includes training terms and training knowledge points of the training terms, the matching degree labels include a first matching degree label between the training video and the training terms, and a second matching degree label between the training video and the training knowledge points, and the predicted matching degree includes a first predicted matching degree between the training video and the training terms, and a second predicted matching degree between the training video and the training knowledge points.

[0158] In some exemplary embodiments of the present disclosure, the candidate knowledge text includes candidate terms and candidate knowledge points of the candidate terms; the extraction module 520 is also configured to extract multiple candidate terms from the text information of the video to be processed; and obtain the candidate knowledge points of each candidate term based on the position of each candidate term in the text information.

[0159] It should be noted that the block diagrams shown in the above figures are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor terminal devices and / or microcontroller terminal devices.

[0160] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0161] Refer to the following Figure 6 6 to describe the electronic device 600 according to such an embodiment of the present disclosure. Figure 6 The electronic device 600 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0162] like Figure 6 As shown, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, the aforementioned at least one processing unit 610, the aforementioned at least one storage unit 620, a bus 630 connecting various system components (including storage unit 620 and processing unit 610), and a display unit 640.

[0163] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 610 can perform the following steps: Figure 2 The steps shown in .

[0164] For example, electronic devices can achieve Figure 2 The steps shown.

[0165] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 621 and / or a cache memory unit 622 , and may further include a read-only memory unit (ROM) 623 .

[0166] The storage unit 620 may also include a program / utility 624 having a set (at least one) of program modules 625, such program modules 625 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0167] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0168] The electronic device 600 can also communicate with one or more external devices 670 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0169] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0170] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, wherein the instructions are executable by a processor of the device to perform the above method. Alternatively, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0171] In an exemplary embodiment, a computer program product is further provided, including a computer program / instruction, which implements the knowledge extraction method in the above embodiment when the computer program / instruction is executed by a processor.

[0172] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0173] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A knowledge extraction method, characterized in that: include: Obtaining a video to be processed and text information of the video to be processed; Extracting a plurality of candidate knowledge texts from the video to be processed and the text information; Obtaining video features of the video to be processed and text features of each candidate knowledge text; Determining similarities between the video features and each text feature using a first model, and determining a plurality of target candidate knowledge texts from the plurality of candidate knowledge texts based on the similarities; The matching degree between each target candidate knowledge text and the video content of the video to be processed is determined by the second model, and the target knowledge text is determined from the multiple target candidate knowledge texts based on the matching degree.

2. The method according to claim 1, characterized in that Obtaining video features of the video to be processed includes: Performing encoding processing on each video frame in the video to be processed to obtain video frame features of each video frame in the video to be processed; Encoding the text information of the video to be processed to obtain video text features of the video to be processed; The video frame features of the respective video frames and the video text features are determined as the video features of the video to be processed.

3. The method according to claim 1, characterized in that The candidate knowledge text includes candidate entries and candidate knowledge points of the candidate entries; Obtain text features of each candidate knowledge text, including: For each candidate knowledge text, performing splicing processing on the candidate terms and the knowledge points of the candidate terms of the candidate knowledge text to obtain a spliced ​​knowledge text; The concatenated knowledge texts are coded to obtain text features of the candidate knowledge texts.

4. The method according to claim 1, wherein The first model is trained in the following way: Acquire a first training sample, wherein the first training sample includes a plurality of first training video text groups, and the training videos and training knowledge texts in the first training video text groups correspond to each other; Training a first initial model using the first training video text group, and outputting a first similarity between the training videos and the training knowledge texts in the first training video text group; If the first similarity is less than or equal to a first preset value, the model parameters of the first initial model are adjusted, and the adjusted first initial model is trained again through the first training video text group until the first similarity output by the adjusted first initial model is greater than the first preset value, and the first model is determined based on the model parameters of the adjusted first initial model.

5. The method according to claim 1, wherein The first model is trained in the following way: Acquire a second training sample, wherein the second training sample includes a plurality of second training video text groups, and the training videos and training knowledge texts in the second training video text groups do not correspond to each other; Training the second initial model using the second training video text group, and outputting a second similarity between the training video and the training knowledge text in the second training video text group; If the second similarity is greater than or equal to the second preset value, the model parameters of the second initial model are adjusted, and the adjusted second initial model is trained again through the second training video text group until the second similarity output by the adjusted second initial model is less than the second preset value, and the first model is determined based on the model parameters of the adjusted second initial model.

6. The method according to claim 1, characterized in that The target candidate knowledge text includes a target candidate entry and a target candidate knowledge point of the target candidate entry, and the target knowledge text includes a target entry and a target knowledge point of the target knowledge text; Determining the degree of match between each target candidate knowledge text and the video content of the video to be processed includes: Determining a first degree of matching between each target candidate term and the video content of the video to be processed; Determining a second degree of matching between each target candidate knowledge point and the video content of the video to be processed; Determining a target knowledge text from the plurality of target candidate knowledge texts according to the matching degree includes: A target term and a target knowledge point of the target term are determined from the plurality of target candidate terms and the target candidate knowledge points of the target candidate terms according to the first matching degree and the second matching degree.

7. The method according to claim 1 or 6, characterized in that The second model is trained in the following way: Acquire a third training sample, wherein the third training sample includes a plurality of third training video text groups and matching degree labels between training videos and training knowledge texts in the third training video text groups; Training the third initial model using the third training video text group, and outputting a predicted matching degree between the training video and the training knowledge text in the third training video text group; If the predicted matching degree and the matching degree label are inconsistent, the model parameters of the third initial model are adjusted, and the adjusted third initial model is trained again through the third training video text group until the predicted matching degree output by the adjusted third initial model is consistent with the matching degree label, and the second model is determined based on the model parameters of the adjusted third initial model.

8. The method according to claim 7, characterized in that The training knowledge text includes training terms and training knowledge points of the training terms, the matching degree labels include a first matching degree label between the training video and the training terms, and a second matching degree label between the training video and the training knowledge points, and the predicted matching degree includes a first predicted matching degree between the training video and the training terms, and a second predicted matching degree between the training video and the training knowledge points.

9. The method according to claim 1, characterized in that The candidate knowledge text includes candidate entries and candidate knowledge points of the candidate entries; Extracting multiple candidate knowledge texts from the video to be processed and the text information, including: Extracting multiple candidate terms from the text information of the video to be processed; According to the position of each candidate term in the text information, the candidate knowledge point of each candidate term is obtained.

10. A knowledge extraction device, characterized in that: include: An acquisition module is configured to acquire a video to be processed and text information of the video to be processed; An extraction module is configured to extract a plurality of candidate knowledge texts from the video to be processed and the text information; The acquisition module is further configured to acquire video features of the video to be processed and text features of each candidate knowledge text; a determination module configured to determine similarities between the video features and the respective text features using a first model, and determine a plurality of target candidate knowledge texts from the plurality of candidate knowledge texts based on the similarities; The determination module is further configured to determine the matching degree between each target candidate knowledge text and the video content of the video to be processed through a second model, and determine the target knowledge text from the multiple target candidate knowledge texts based on the matching degree.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the executable instructions to implement the method according to any one of claims 1 to 9. 12 . A computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method according to claim 1 .

Images