A method for generating abstracts and related devices

Abstract

The present application relates to the field of computer technology, and provides a summary generation method and related devices for improving the accuracy of summary generation, wherein the method includes: based on the acquired multiple modal information, after obtaining at least one candidate modal node corresponding to each modal information, respectively, constructing a corresponding target node association relationship, and then, based on the target node association relationship, using a bidirectional iterative update method to obtain the target feature representation corresponding to each candidate modal node, in each round of iteration, based on the candidate feature representation of the adjacent candidate modal node, the candidate feature representation is updated, and then based on the obtained target feature representations, the target modal node is determined, and based on the target modal node, the target summary information is generated. In this way, through the bidirectional iterative update method, the interaction between multiple modal information can be enhanced, the accuracy of the feature representation can be improved, and the accurate recommendation of the target summary information can be achieved.

Description

Background Art

[0002] With the continuous development of computer technology, there is a large amount of multimodal data on the Internet. Multimodal information usually includes text, audio, image, video and other information. Therefore, it is very necessary to summarize the multimodal information. Multimodal summary generation is one of the important means. Multimodal summary generation refers to inputting multimodal information and outputting a core summary after comprehensively considering the multimodal information.

[0003] In the related technology, considering the existence of semantic correlation between various modal information contained in multimodal information, a one-to-one semantic correlation is usually learned for every two modal information, and then based on the learned semantic correlation, the corresponding feature representations of various modal information are generated, and then based on the corresponding feature representations of various modal information, the summary corresponding to the multimodal information is output.

[0004] However, due to the large semantic gap between information in different modalities, the above method is difficult to bridge the semantic gap between information in different modalities, resulting in the inability of the generated feature representation to simultaneously include the semantic correlation between information in various modalities, resulting in low accuracy of feature representation, which in turn affects the accuracy of summary generation. Taking text and images as an example, since images are visual features, it is difficult to directly obtain the semantic correlation between images and text, resulting in low accuracy of feature expression. Summary of the invention

[0005] The embodiments of the present application provide a summary generation method and related devices to improve the accuracy of feature representation, thereby improving the accuracy of summary generation.

[0006] In a first aspect, an embodiment of the present application provides a method for generating a summary, including:

[0007] Based on the obtained multiple modal information, at least one candidate modal node corresponding to each of the modal information is obtained; each candidate modal node is used to represent a key data contained in the corresponding modal information;

[0008] Based on each obtained candidate modal node, a corresponding target node association relationship is constructed;

[0009] Based on the target node association relationship, an iterative update method is used to obtain the target feature representation corresponding to each candidate modal node, wherein in each round of iteration, the following operations are performed for each candidate modal node in turn: based on the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round is updated, and the node update directions of two adjacent iterations are opposite;

[0010] Based on the obtained target feature representations, at least one target modality node is determined from the candidate modality nodes, and based on the at least one target modality node, target summary information is generated.

[0011] In a second aspect, an embodiment of the present application provides a summary generation device, including:

[0012] A node acquisition unit, configured to obtain, based on the acquired multiple modal information, at least one candidate modal node corresponding to each of the modal information; each candidate modal node is used to represent a key data contained in the corresponding modal information;

[0013] A relationship building unit, used to build a corresponding target node association relationship based on each obtained candidate modal node;

[0014] A feature representation unit is used to obtain the target feature representation corresponding to each of the candidate modal nodes respectively by an iterative update method based on the target node association relationship, wherein in each round of iteration, the following operations are performed for each candidate modal node in turn: based on the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round is updated, and the node update directions of two adjacent iterations are opposite;

[0015] A summary generation unit is used to determine at least one target modality node from the candidate modality nodes based on the obtained target feature representations, and generate target summary information based on the at least one target modality node.

[0016] As a possible implementation manner, when updating the candidate feature representation of the next candidate modal node in the previous round based on the node relevance, the feature representation unit is specifically used to:

[0017] Based on the target node association relationship, at least one other candidate modal node associated with the next candidate modal node is selected from the candidate modal nodes except the one candidate modal node;

[0018] Based on the candidate feature representations corresponding to each of the at least one other candidate modality node and the candidate feature representations of the next candidate modality node in the previous round, obtaining the node correlation between the next candidate modality and the at least one other candidate modality node;

[0019] Based on the obtained relevance of each node, the candidate feature representation of the next candidate modal node in the previous round is updated.

[0020] As a possible implementation manner, when obtaining at least one candidate modal node corresponding to each modal information based on the obtained multiple modal information, the node acquisition unit is specifically used to:

[0021] If the modality type of one modality information among the multiple modality information is video, extracting audio and at least one candidate video frame from the one modality information, and determining at least one key data corresponding to the one modality information based on the audio and the at least one candidate video frame, and obtaining at least one candidate modality node corresponding to the one modality information based on the at least one key data;

[0022] If the modality type of one modal information among the multiple modal information is text, at least one text key information is extracted from the one modal information, and the at least one text key information is used as at least one key data corresponding to the one modal information, and based on the at least one key data, at least one candidate modal node corresponding to the one modal information is obtained.

[0023] As a possible implementation manner, when determining at least one key data corresponding to the one modality information based on the audio and the at least one candidate video frame, the node acquisition unit is specifically used to perform at least one of the following operations:

[0024] Extracting at least one target video frame from the at least one candidate video frame according to a preset extraction interval, and using each obtained target video frame as corresponding various key data;

[0025] Based on the audio, at least one audio text information is obtained, and each piece of audio text information obtained is used as corresponding various key data.

[0026] As a possible implementation, when constructing the corresponding target node association relationship based on each obtained candidate modal node, the relationship construction unit is specifically used to:

[0027] Based on the modality types corresponding to the multiple modal information, respectively, determining a node hierarchical relationship between the candidate modal nodes, wherein in the node hierarchical relationship, at least one candidate modal node corresponding to each modal information is located at the same level;

[0028] Determine the node connection relationship between the candidate modal nodes based on the key data represented by each of the candidate modal nodes;

[0029] Based on the node hierarchical relationship and the node connection relationship, the target node association relationship is obtained.

[0030] As a possible implementation manner, the relationship building unit is further used for:

[0031] Based on the node hierarchical relationship contained in the target node association relationship, the node update order of each candidate modal node is determined.

[0032] As a possible implementation manner, when determining the node connection relationship between the candidate modal nodes based on the key data represented by each of the candidate modal nodes, the relationship construction unit is specifically used to:

[0033] Based on the key data represented by each of the candidate modal nodes, at least one target keyword is extracted, and the at least one target keyword is used as each connection node;

[0034] Based on the respective connection nodes, a node connection relationship between the respective candidate modal nodes is obtained, in which the candidate modal nodes at different levels are connected via the respective connection nodes.

[0035] As a possible implementation, the target node association relationship further includes the edge weights between each connection node and each candidate modality node;

[0036] Then, based on the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round is updated. The feature representation unit is specifically used for:

[0037] Based on the edge weights and the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round is updated.

[0038] As a possible implementation manner, when determining at least one target modality node from the candidate modality nodes based on the obtained target feature representations, the summary generation unit is specifically configured to:

[0039] Based on each obtained target feature representation, determining at least one target modal node corresponding to each of the multiple modal information from at least one candidate modal node corresponding to each of the multiple modal information;

[0040] When generating target summary information based on the at least one target modal node, the summary generating unit is specifically used for:

[0041] Based on at least one target modality node corresponding to each of the multiple modality information, obtaining summary sub-information corresponding to each of the multiple modality information;

[0042] Based on the obtained individual summary sub-information, target summary information is generated.

[0043] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the above-mentioned summary generation method.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is executed on an electronic device, the computer program is used to enable the electronic device to execute the steps of the above-mentioned summary generation method.

[0045] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of an electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device performs the steps of the above-mentioned summary generation method.

[0046] In an embodiment of the present application, after obtaining at least one candidate modal node corresponding to each modal information based on the acquired multiple modal information, a corresponding target node association relationship is constructed, and then, based on the target node association relationship, a bidirectional iterative update method is adopted to obtain the target feature representation corresponding to each candidate modal node. In each round of iteration, the candidate feature representation is updated based on the candidate feature representation of the adjacent candidate modal node, and then the target modal node is determined based on the obtained target feature representations, and the target summary information is generated based on the target modal node.

[0047] In this way, by updating the candidate feature representation through the candidate feature representation of adjacent candidate modal nodes, the candidate modal nodes corresponding to each modal information can learn the information contained in the candidate modal nodes corresponding to other modal information, thereby improving the accuracy of summary generation. In addition, through the bidirectional iterative update method, the interaction between multiple modal information can be enhanced, and the information flow can be enhanced, so that the candidate modal nodes corresponding to each modal information can better learn the information contained in the candidate modal nodes corresponding to other modal information, and fill the semantic gap between different modal information, thereby learning better feature representation, improving the accuracy of feature representation, and further improving the accuracy of summary generation.

[0048] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0050] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of a process for generating a summary provided in an embodiment of the present application;

[0052] Figure 3 A logical schematic diagram of obtaining candidate modal nodes corresponding to a video provided in an embodiment of the present application;

[0053] Figure 4 A logical schematic diagram of extracting a target video frame provided in an embodiment of the present application;

[0054] Figure 5 A schematic diagram of a process for building a target node association relationship provided in an embodiment of the present application;

[0055] Figure 6 A schematic diagram of a target node association relationship provided in an embodiment of the present application;

[0056] Figure 7 A schematic diagram of determining a node connection relationship provided in an embodiment of the present application;

[0057] Fig. 8A A schematic diagram of the first information flow direction provided in an embodiment of the present application;

[0058] Figure 8B A schematic diagram of the second information flow direction provided in an embodiment of the present application;

[0059] Fig. 9 A logical schematic diagram of a summary generation method provided in an embodiment of the present application;

[0060] Fig.10 A schematic diagram of the structure of a summary generation device provided in an embodiment of the present application;

[0061] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0063] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.

[0064] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

[0065] Big data refers to a collection of data that cannot be captured, managed, and processed by conventional software tools within a certain time frame. It is a massive, high-growth, and diverse information asset that requires new processing models to have stronger decision-making power, insight discovery, and process optimization capabilities. With the advent of the cloud era, big data has also attracted more and more attention. Big data requires special technologies to effectively process large amounts of data within a tolerable time frame. Technologies applicable to big data include large-scale parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.

[0066] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0067] With the continuous development of computer technology, there is a large amount of multimodal data on the Internet. Multimodal information usually includes text, audio, image, video and other information. Therefore, it is very necessary to summarize the multimodal information. Multimodal summary generation is one of the important means. Multimodal summary generation refers to inputting multimodal information and outputting a core summary after comprehensively considering the multimodal information.

[0068] In the related technology, considering the existence of semantic correlation between various modal information contained in multimodal information, a one-to-one semantic correlation is usually learned for every two modal information, and then based on the learned semantic correlation, the corresponding feature representations of various modal information are generated, and then based on the corresponding feature representations of various modal information, the summary corresponding to the multimodal information is output.

[0069] However, due to the large semantic gap between information in different modalities, the above method is difficult to bridge the semantic gap between information in different modalities, resulting in the inability to simultaneously include the semantic correlation between information in various modalities in the generated feature representation, resulting in low accuracy of feature expression, which in turn affects the accuracy of summary generation. Taking text and images as an example, since images are visual features, it is difficult to directly obtain the semantic correlation between images and text, resulting in low accuracy of feature expression.

[0070] In an embodiment of the present application, after obtaining at least one candidate modal node corresponding to each modal information based on the acquired multiple modal information, a corresponding target node association relationship is constructed, and then, based on the target node association relationship, a bidirectional iterative update method is adopted to obtain the target feature representation corresponding to each candidate modal node. In each round of iteration, the candidate feature representation is updated based on the candidate feature representation of the adjacent candidate modal node, and then the target modal node is determined based on the obtained target feature representations, and the target summary information is generated based on the target modal node.

[0071] In this way, by updating the candidate feature representation through the candidate feature representation of adjacent candidate modal nodes, the candidate modal nodes corresponding to each modal information can learn the information contained in the candidate modal nodes corresponding to other modal information, thereby improving the accuracy of summary generation. In addition, through the bidirectional iterative update method, the interaction between multiple modal information can be enhanced, and the information flow can be enhanced, so that the candidate modal nodes corresponding to each modal information can better learn the information contained in the candidate modal nodes corresponding to other modal information, and fill the semantic gap between different modal information, thereby learning better feature representation, improving the accuracy of feature representation, and further improving the accuracy of summary generation.

[0072] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0073] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0074] The solution provided in the embodiment of the present application involves artificial intelligence machine learning technology. For example, when a bidirectional iterative update method is adopted based on the target node association relationship to obtain the target feature representation corresponding to each candidate modal node, a bidirectional iterative update can be performed based on a graph neural network and a feedforward neural network.

[0075] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0076] See also Figure 1 As shown, it is a schematic diagram of an application scenario provided in an embodiment of the present application. The application scenario includes at least a terminal device 110 and a server 120. The number of terminal devices 110 can be one or more, and the number of servers 120 can also be one or more. The present application does not specifically limit the number of terminal devices 110 and servers 120.

[0077] In the embodiment of the present application, the terminal device 110 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an Internet of Things device, a smart home appliance, a car terminal, etc., but is not limited thereto.

[0078] The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device 110 and the server 120 may be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0079] Exemplarily, a client corresponding to the summary generation software is installed in the terminal device 110, and each terminal device 110 can obtain multiple modal information input by the target object through the client, and send the multiple modal information to the server 120. Correspondingly, after receiving the multiple modal information from the terminal device 110, the server 120 sends the target summary information to the terminal device 110, so that the terminal device 110 obtains and presents the target summary information.

[0080] See also Figure 2 As shown, it is a flow chart of a method for generating a summary provided in an embodiment of the present application. The flow of the method can be executed by an electronic device, and the electronic device can be a terminal device or a server. The specific flow is as follows:

[0081] S201. Based on the acquired multiple modal information, at least one candidate modal node corresponding to each modal information is obtained respectively, and each candidate modal node is used to represent a key data contained in the corresponding modal information.

[0082] Among them, the modal types of modal information include but are not limited to video, text, audio, image and other types.

[0083] For the convenience of description, modal information i is taken as an example for explanation, and modal information i is any one of multiple modal information.

[0084] Specifically, according to the type of modal information i, there are but are not limited to the following possible situations:

[0085] Case 1: See Figure 3 As shown, if the type of modal information i is video, audio and at least one candidate video frame are extracted from the modal information i, and based on the audio and at least one candidate video frame, at least one key data corresponding to the modal information i is determined, and based on the at least one key data, at least one candidate modal node corresponding to the modal information i is obtained.

[0086] It should be noted that in the embodiment of the present application, the video frame may also be referred to as an image, and the candidate video frame may be extracted from the modal information i using a natural picture recognition network, and the natural picture recognition network may use but is not limited to a convolutional neural network, which may be trained using a structure having a residual grid. Exemplarily, the natural picture recognition network may be trained using the ImageNet database, which is a large visualization database used for the study of visual object recognition software. Since the ImageNet database has 1,000 categories of pictures, in actual application, pictures of the required categories may be selected for training.

[0087] Through the above implementation method, when multiple modal information includes video, audio and several images can be extracted from the video, and then the key data corresponding to the video can be determined based on the audio and several images contained in the video, so that the generated summary can accurately express the content of the video and improve the accuracy of the summary.

[0088] Specifically, when determining at least one key data corresponding to the modality information i based on the audio and at least one candidate video frame, the following operations may be used but are not limited to:

[0089] Operation 1-a: extract at least one target video frame from at least one candidate video frame according to a preset extraction interval, and use each obtained target video frame as corresponding various key data.

[0090] For example, see Figure 4 As shown, corresponding audio and candidate video frames are extracted from video 1, where video 1 is a food recommendation video. Assuming that the preset extraction interval is to extract one frame every 24 frames, the candidate video frames include video frame 1, video frame 2, video frame 3, etc. First, video frame 1 is extracted from the candidate video frames as a target video frame, and then, according to the extraction interval of extracting one frame every 24 frames, video frame 26 is extracted as a target video frame. Similarly, according to the extraction interval, video frame 51, video frame 76, etc. are extracted in sequence as target video frames. Afterwards, the extracted target video frames are used as various key data corresponding to video 1, and then based on the various key data obtained, the candidate modal nodes corresponding to video 1 are obtained.

[0091] Through the above operation, the number of candidate modal nodes can be effectively reduced, thereby reducing the complexity of the target node association relationship, thereby reducing the amount of calculation in the subsequent summary generation process, and further improving the summary generation efficiency.

[0092] Operation 1-b: Based on the audio, obtain at least one audio text information, and use each obtained audio text information as corresponding various key data.

[0093] It should be noted that, in the embodiment of the present application, the audio text information may also be referred to as lines, and the lines are presented in text form. The audio text information may be obtained using speech recognition technology.

[0094] Still taking Video 1 as an example, after extracting the audio and each candidate video frame from Video 1, Line 1 and Line 2 are obtained based on the audio, where Line 1 is "better than hot pot" and Line 2 is "rich in taste and simple, fast and low in fat". The obtained Line 1 and Line 2 are used as the two key data corresponding to Video 1. Then, based on the obtained two key data, the two candidate modal nodes corresponding to Video 1 are obtained.

[0095] Through the above operations, the corresponding text information can be obtained from the video, that is, the lines corresponding to the video can be obtained. In this way, sentences and images can be connected according to the lines, thereby improving the accuracy of summary generation.

[0096] Operation 1-c: According to a preset extraction interval, extract at least one target video frame from at least one candidate video frame, and based on the audio, obtain at least one audio text information, and use at least one target video frame and at least one audio text information as corresponding various key data.

[0097] Still taking video 1 as an example, first, audio and candidate video frames are extracted from video 1, and the candidate video frames include video frame 1, video frame 2, video frame 3, etc. Assume that the preset extraction interval is to extract one frame every 24 frames. Then, according to the preset extraction interval, video frame 1, video frame 26, video frame 51, video frame 76, etc. are extracted in turn as target video frames, and based on the audio, lines 1 and lines 2 are obtained. Then, the extracted target video frames such as video frame 1, video frame 26, video frame 51, video frame 76, as well as lines 1 and lines 2 are respectively used as various key data corresponding to video 1. Then, based on the various key data obtained, the candidate modal nodes corresponding to video 1 are obtained.

[0098] Case 2: If the type of modal information i is text, at least one text key information is extracted from the modal information i, and the at least one text key information is used as at least one key data corresponding to the modal information i, and based on the at least one key data, at least one candidate modal node corresponding to the modal information i is obtained.

[0099] It should be noted that, in the embodiment of the present application, the key text information can also be referred to as a sentence.

[0100] Taking the modal information i as text 1 as an example, text 1 is a food recommendation article. Sentence 1, sentence 2 and sentence 3 are extracted from text 1, among which sentence 1 is "the preparation method is simple", sentence 2 is "in line with the concept of healthy diet", and sentence 3 is "it tastes rich". Sentence 1, sentence 2 and sentence 3 are used as the three key data corresponding to text 1. Then, based on the three key data, three candidate modal nodes corresponding to text 1 are obtained.

[0101] Through the above operations, some key data can be extracted from the text, thereby reducing the amount of calculation and improving the efficiency of summary generation. In the embodiment of the present application, audio can also be extracted from the video, and the audio can be converted into lines through speech recognition technology. At the same time, multiple images are extracted from the video. In this way, when the multiple modal information obtained is video and text, a summary with relatively consistent semantics with the text and video can be output through the text, as well as the sentences and pictures extracted from the video, thereby improving the accuracy of the generated summary.

[0102] S202: Based on each obtained candidate modal node, a corresponding target node association relationship is constructed.

[0103] Considering that as the number of candidate modal nodes increases, the constructed target node association relationship becomes more complex, in the embodiment of the present application, for each candidate modal node, by constructing a node hierarchy relationship and a node connection relationship, the target node association relationship is simplified and the updating efficiency of the feature representation is improved. For details, see Figure 5 As shown, when executing S202, the following steps may be adopted:

[0104] S2021. Based on the modal types corresponding to the multiple modal information, determine the node hierarchy relationship between the candidate modal nodes, in which at least one candidate modal node corresponding to each modal information is located at the same level.

[0105] Taking into account that a certain modal information may contain multiple types of information, in an embodiment of the present application, at least one candidate modal node corresponding to the modal information i can be subdivided into multiple levels.

[0106] See also Figure 6As shown, it is a schematic diagram of a target node association relationship provided in an embodiment of the present application, in which the candidate modal nodes corresponding to video 1 include: node 1, node 2, node 3, node 4 and node 5, wherein node 1, node 2 and node 3 are obtained based on the image extracted from video 1, and node 4 and node 5 are obtained based on the sentences contained in the audio extracted from video 1. Node 1, node 2 and node 3 are located at the first layer, and node 4 and node 5 are located at the second layer. The target node association relationship also includes nodes 6 to 11, wherein node 11 is obtained based on the sentence contained in text 1, node 6, node 7, node 8 and node 9 are located at the third layer, and node 10 and node 11 are located at the fourth layer.

[0107] S2022. Determine the node connection relationship between each candidate modal node based on the key data represented by each candidate modal node.

[0108] In order to better learn other modal information and improve the accuracy of feature representation, in the embodiment of the present application, the fine-grainedness of the connection nodes can be determined according to the fine-grainedness of various key information. For example, since both the lines and the sentences in the text can be composed of words, the words can be used as connection nodes to connect the selected modal nodes corresponding to the lines and the selected modal nodes corresponding to the sentences. In the embodiment of the present application, only the type of the target keyword is a word as an example for explanation.

[0109] For details, see Figure 7 As shown, when executing S2022, the following steps may be performed:

[0110] S20221. Based on the key data represented by each candidate modal node, extract at least one target keyword, and use the at least one target keyword as each connection node.

[0111] For example, see Figure 6As shown, it is assumed that, among the candidate modal nodes, the key data represented by node 4 is line 1, the key data represented by node 5 is line 2, line 1 contains target keywords "A" and target keywords "C", and line 2 contains target keywords "B" and target keywords "D", the key data represented by node 10 is sentence 1, and the key data represented by node 11 is sentence 2, sentence 1 contains target keywords "A" and target keywords "B", and sentence 2 contains target keywords "B", target keywords "C" and target keywords "D". Based on each candidate modal node, the target keywords "A", "B", "C" and "D" are extracted, and the target keywords "A", "B", "C" and "D" are used as 4 connection nodes, among which nodes 6, 7, 8 and 9 are all connection nodes, node 6 is used to represent the target keyword "A", node 7 is used to represent the target keyword "B", node 8 is used to represent the target keyword "C", and node 9 is used to represent the target keyword "D".

[0112] S20222. Based on each connection node, obtain a node connection relationship between each candidate modal node. In the node connection relationship, candidate modal nodes at different levels are connected through each connection node.

[0113] In the embodiment of the present application, considering that each frame of the image may be related to each line of dialogue, a dense connection may be used to connect the image and the sentence. In addition, the relationship between the lines and the sentence may be established by using words as intermediate nodes.

[0114] Specifically, when connecting lines and words, full connection is not used, that is, each line is connected to each word, but each line is connected to the words contained in the line. Similarly, when connecting sentences and words, each sentence is connected to the words contained in the sentence.

[0115] For example, see Figure 6 As shown, based on each connection node, the node connection relationship between each candidate modal node is obtained, wherein node 4 connects node 6 and node 8, node 5 connects node 7 and node 9, node 10 connects node 6 and node 7, and node 11 connects node 7, node 8 and node 9.

[0116] S2023. Obtain target node association relationships based on node hierarchical relationships and node connection relationships.

[0117] S203. Based on the target node association relationship, an iterative update method is used to obtain the target feature representation corresponding to each candidate modal node. In each round of iteration, the following operations are performed for each candidate modal node in turn: based on the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round is updated, and the node update directions of two adjacent iterations are opposite.

[0118] In the embodiment of the present application, before executing S203, key information representing each candidate modal node contained in the target node association relationship is encoded.

[0119] Since the data types of key data include images, lines, words and sentences, etc., in the embodiments of the present application, only the encoding objects including images, lines, words and sentences are used as examples for explanation.

[0120] Adopt X w , X s , X t , X v Represent the feature representations of words, sentences, lines, and images respectively, among which, Among them, |*| represents the number of candidate modal nodes of the corresponding modal information, d |*| Indicates the dimension of the feature representation of the corresponding candidate modality node.

[0121] For the word encoding X w , we can use but not limited to Glove embedding to get the feature representation of words.

[0122] For the code of the lines X t and the encoding of the sentence X s , since lines and sentences are composed of words, the encoding of the sentence is X s For example, we can use the bidirectional long short-term memory network (BiLTSM) at the word level and sentence level to get the encoding of the lines X t and the encoding of the sentence X s , where the word-level bidirectional long short-term memory network is used to extract fine-grained information within the sentence, and the sentence-level bidirectional long short-term memory network is used to extract contextual information between sentences. Specifically, the encoding of the sentence X s It can be expressed as:

[0123] W i =BiLTSM(w i1 ,w i2 ,…,w in ) Formula (1)

[0124] h i =∑ j W ijFormula (2)

[0125] X s =BiLTSM(h1,h2,…,h |S| ) Formula (3)

[0126] Among them, w ij represents the embedding representation of the jth word in the i-th sentence, n represents the number of words contained in the i-th sentence, and W ij is the output representation of the word-level bidirectional long short-term memory network for the jth word in the i-th sentence, h i is the latent variable representation of the i-th sentence, X s The bidirectional long short-term memory network at the sentence level models the contextual information between sentences. t The encoding process of X is similar to the encoding of the sentence s The encoding process is similar and will not be repeated here.

[0127] For the image encoding X v , a pre-trained ResNet-101 convolutional neural network can be used to extract features from the image. For example, the dimension of feature extraction for the image can be 2048, that is, d v The value is 2048.

[0128] It should be noted that in the embodiment of the present application, each candidate modal node can be encoded separately after obtaining each candidate modal node and before constructing the target node association relationship, or each candidate modal node can be encoded after constructing the target node association relationship, and the comparison is not limited. In the embodiment of the present application, only the encoding of each candidate modal node before constructing the target node association relationship is used as an example for explanation, therefore, each candidate modal node included in the target node association relationship has been encoded.

[0129] In order to further enhance the interaction between various modal information and thus bridge the semantic gap between different modal information, in an embodiment of the present application, a two-way message propagation mechanism is adopted to enhance the information flow between different modal information so that each modal information can learn the characteristics of other modal information.

[0130] Specifically, in an embodiment of the present application, the node update order of each candidate modal node can be determined based on the node hierarchy relationship contained in the target node association relationship. That is, in each round of iteration, each candidate modal node can be updated in turn according to the node hierarchy relationship contained in the target node association relationship.

[0131] See also Fig. 8A As shown, Fig. 8AThis is a schematic diagram of the first information flow direction provided in an embodiment of the present application. Fig. 8A The direction of information flow in is: image → lines → words → sentences, among which the next candidate modal nodes associated with node 1 are node 4 and node 5, the next candidate modal nodes associated with node 2 are node 4 and node 5, the next candidate modal nodes associated with node 3 are node 4 and node 5, the next candidate modal nodes associated with node 4 are node 6 and node 8, the next candidate modal nodes associated with node 5 are node 7 and node 9, the next candidate modal node associated with node 6 is node 10, the next candidate modal nodes associated with node 7 are node 10 and node 11, the next candidate modal node associated with node 8 is node 11, and the next candidate modal node associated with node 9 is node 11.

[0132] See also Figure 8B As shown, Figure 8B This is a schematic diagram of the second information flow direction provided in an embodiment of the present application. Figure 8B The direction of information flow in is: sentence → word → line → image, among which, the next candidate modal node associated with node 10 is node 6 and node 7, the next candidate modal node associated with node 11 is node 7, node 8 and node 9, the next candidate modal node associated with node 6 is node 4, the next candidate modal node associated with node 7 is node 5, the next candidate modal node associated with node 8 is node 4, the next candidate modal node associated with node 9 is node 5, the next candidate modal node associated with node 4 is node 1, node 2 and node 3, and the next candidate modal node associated with node 5 is node 1, node 2 and node 3.

[0133] The information flow direction of the first round of iteration is taken as an example to illustrate.

[0134] In the first round of iteration, the process of updating the candidate feature representation of the candidate modal node corresponding to the line based on the candidate feature representation of the candidate modal node corresponding to the image can be described as:

[0135]

[0136]

[0137] Based on the candidate feature representations of the candidate modal nodes corresponding to the lines, the process of updating the candidate feature representations of the candidate modal nodes corresponding to the words can be described as:

[0138]

[0139]

[0140] Based on the candidate feature representations of the candidate modal nodes corresponding to the words, the process of updating the candidate feature representations of the candidate modal nodes corresponding to the sentences can be described as:

[0141]

[0142]

[0143] In GAT(Q, K, V), Q, K, and V represent query, key, and value, respectively. The value of X w , The value of X s , The value of X t , The value of X v .

[0144] The superscript 1 indicates the first round of iteration. Represents the feature representation of the candidate modal nodes corresponding to the lines output by the graph attention layer in the first iteration, Represents the new candidate feature representation of the candidate modality node corresponding to the lines output by the feedforward network layer in the first round of iteration.

[0145] akin, Represents the feature representation of the candidate modal nodes corresponding to the words output by the graph attention layer in the first iteration, Represents the new candidate feature representation of the candidate modal node corresponding to the word output by the feedforward network layer in the first round, Represents the feature representation of the candidate modal node corresponding to the sentence output by the graph attention layer in the first iteration, Represents the new candidate feature representation of the candidate modal node corresponding to the sentence output by the feedforward network layer in the first round.

[0146] Correspondingly, the information flow direction of the second round of iteration is the second information flow direction.

[0147] In the second round of iteration, based on the candidate feature representations of the candidate modal nodes corresponding to the sentences, the process of updating the candidate feature representations of the candidate modal nodes corresponding to the words can be described as follows:

[0148]

[0149]

[0150] Based on the candidate feature representations of the candidate modal nodes corresponding to the words, the process of updating the candidate feature representations of the candidate modal nodes corresponding to the lines can be described as:

[0151]

[0152]

[0153] Based on the candidate feature representations of the candidate modal nodes corresponding to the lines, the process of updating the candidate feature representations of the candidate modal nodes corresponding to the images can be described as:

[0154]

[0155]

[0156] Specifically, when using the graph attention layer for feature learning, the node V of the graph can be represented by the union of the candidate modal nodes corresponding to the image, line, word, and sentence, that is, V = V I ∪V T ∪V W ∪V S , where V I Represents the set of candidate modal nodes corresponding to the image, V T Represents the set of candidate modal nodes corresponding to the lines, V W Represents the set of candidate modal nodes corresponding to the word, V S Represents the set of candidate modal nodes corresponding to the sentence.

[0157] use Indicates the candidate feature representation corresponding to each candidate modality node. For the sake of description, the candidate modality node is referred to as a node in the following text.

[0158] As a possible implementation method, in the graph attention layer, first, formula (16) can be used to calculate the importance of node j to node i for a node i. In the embodiment of the present application, the importance of node j to node i can also be referred to as the node correlation between node i and node j:

[0159] z ij =f(W a [W q h i |W k h j ]) Formula (16)

[0160] Among them, W a , W q , W kThere are three conversion matrices, and their parameters can be obtained through training. “|” represents a series operation, and f() represents an activation function, which can be but is not limited to LeakyReLU.

[0161] In fact, for a node, it is not necessary to calculate the influence of all other nodes on it, but only the influence of its adjacent nodes on it. In order to improve the updating efficiency of feature representation, in the embodiment of the present application, based on the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round is updated, which can be adopted but not limited to the following methods:

[0162] Based on the candidate feature representation of a candidate modal node and based on the candidate feature representation of the next candidate modal node in the previous round, a node correlation between a candidate modal node and a next candidate modal node is obtained; based on the node correlation, the candidate feature representation of the next candidate modal node in the previous round is updated.

[0163] For example, see Figure 6 As shown, the next candidate modal node associated with node 1 is node 4. Based on the candidate feature representation of node 1 and the candidate feature representation of node 4 in the previous round, the node correlation between node 1 and node 4 can be obtained using the above formula (16).

[0164] Taking into account the influence of all adjacent nodes of a node on it, in the embodiment of the present application, in order to improve the accuracy of the generated feature representation, based on the node relevance, the candidate feature representation of the next candidate modal node in the previous round is updated, and the following method can be adopted:

[0165] Based on the target node association relationship, at least one other candidate modal node associated with the next candidate modal node is selected from each candidate modal node except for one candidate modal node;

[0166] Based on the candidate feature representation corresponding to each of at least one other candidate modality node and the candidate feature representation of the next candidate modality node in the previous round, obtaining the node correlation between the next candidate modality and at least one other candidate modality node;

[0167] Based on the obtained relevance of each node, the candidate feature representation of the next candidate modal node in the previous round is updated.

[0168] Specifically, the following formula (17) can be used to calculate the influence of each adjacent node included in the adjacent node set N of node i on node i, and normalize it:

[0169]

[0170] Calculate α ij Then, the new feature representation u of node i can be calculated using formula (18): i :

[0171] u i =σ(∑ j∈N α ij W V h j ) Formula (18)

[0172] Among them, W V is a transformation matrix, whose parameters can be obtained through training, and σ represents a nonlinear change function.

[0173] For example, for node 1 and node 4, based on the target node association relationship, from each candidate modal node, other candidate modal nodes associated with node 4 are screened out except node 1, and other candidate modal nodes include node 2 and node 3. Then, based on the candidate feature representations corresponding to node 2 and node 3 respectively, and the candidate feature representation of node 4 in the previous round, formula (16) is used to obtain the node correlation between node 4 and node 2 and node 3 respectively. Then, based on the node correlation between node 4 and node 1, node 2 and node 3 respectively, the candidate feature representation of node 4 in the previous round is updated.

[0174] Based on the obtained relevance of each node, when updating the candidate feature representation of the next candidate modal node in the previous round, the above formula (18) can be used to obtain u i , after getting u i After that, through the feed-forward network layer, u i Perform feature transformation to obtain new candidate feature representations, so as to update the candidate feature representations of the candidate modal nodes and obtain the corresponding target feature representations.

[0175] S204. Based on the obtained target feature representations, determine at least one target modality node from the candidate modality nodes.

[0176] Specifically, when determining at least one target modal node from each candidate modal node, the following method can be adopted: based on the obtained target feature representations, at least one target modal node corresponding to each of the multiple modal information is determined from at least one candidate modal node corresponding to each of the multiple modal information.

[0177] For example, based on the obtained target feature representations, from the candidate modal nodes corresponding to video 1, the target modal node corresponding to video 1 is determined to be node 1, and the key data represented by node 1 is video frame 1, which contains a food display screen. At the same time, from the candidate modal nodes corresponding to text 1, the target modal node corresponding to text 1 is determined to be node 10, and the key data represented by node 10 is sentence 1, which is "in line with the tastes of most people."

[0178] As a possible implementation, in an embodiment of the present application, each target feature representation obtained can be input into a classification layer corresponding to each of the multiple modal information, and at least one target modal node can be directly output. The classification layer is used to determine whether a candidate modal node is a target modal node based on each target feature representation.

[0179] Among them, the classification layer corresponding to each modal information can be supervised by but not limited to the cross entropy loss of binary classification, which will not be repeated here.

[0180] It should be noted that before inputting the obtained target feature representations into the corresponding classification layer, the following formula can be used to project different modal information into a common space. Taking sentences and images as examples, formulas (19) and (20) are used respectively to project sentences and images into a common space:

[0181] p=W2*f(W1*H s +b S ) Formula (19)

[0182] q=V2*f(V1*H i +b i ) Formula (20)

[0183] Among them, W1, W2, V1, V2 are projection matrices, which can be obtained through training, b S and b i is a bias term, and f() is any activation function.

[0184] S205. Generate target summary information based on at least one target modal node.

[0185] Specifically, when executing S205, the following methods may be used:

[0186] Based on at least one target modal node corresponding to each of the multiple modal information, summary sub-information corresponding to each of the multiple modal information is obtained; based on each of the obtained summary sub-information, target summary information is generated.

[0187] For example, the target modal node corresponding to video 1 is node 1, and the key data represented by node 1 is video frame 1, which contains a food display screen. The target modal node corresponding to text 1 is node 10, and the key data represented by node 10 is sentence 1, which is "in line with the tastes of most people". Based on the target modal node corresponding to video 1, the summary sub-information corresponding to video 1 is obtained, and the summary sub-information corresponding to video 1 is video frame 1. Based on the target modal node corresponding to text 1, the summary sub-information corresponding to text 1 is obtained, and the summary sub-information corresponding to text 1 is sentence 1. Based on the summary sub-information corresponding to video 1 and the summary sub-information corresponding to text 1, the target summary information including video frame 1 and sentence 1 is generated.

[0188] Through the above implementation method, when the multiple modal information obtained is video and text respectively, a multimodal summary consisting of several sentences contained in the text and several images contained in the video can be obtained, which enriches the presentation form of the summary and enables the target object to quickly and conveniently understand the meaning to be expressed by the multiple modal information, thereby improving the target object's usage experience.

[0189] In order to avoid the gradient vanishing problem, in the embodiment of the present application, the graph attention layer can be trained by the residual mechanism. Specifically, the residual h′ is calculated using formula (21) i :

[0190] h′ i =u i +h i Formula (21)

[0191] The model convergence conditions based on the residual mechanism can be adopted but are not limited to: the residuals of all nodes are not greater than the preset residual threshold, the residuals of all nodes for M consecutive times are not greater than the preset residual threshold, the difference between the residuals of two consecutive rounds for M consecutive times is not greater than the preset residual threshold, etc., but are not limited to these.

[0192] In some embodiments, in order to further enhance the intrinsic connection between words and sentences, and enhance the intrinsic connection between words and lines, the target node association relationship also includes each edge weight between each connection node and each candidate modality node.

[0193] Taking words and sentences as an example, we can introduce term frequency-inverse document frequency (TF-IDF) as the edge weight between the candidate modal node corresponding to the word and the candidate modal node corresponding to the sentence. Among them, TF represents the number of times a word appears in a sentence, and IDF represents the inverse of the number of sentences and words in which the word appears.

[0194] For example, the target keyword "A" represented by node 6 appears 1 times in sentence 1 represented by node 10. Since both line 1 represented by node 4 and sentence 1 represented by node 10 contain the target keyword "A", the number of sentences and words in which the target keyword "A" appears is 2. Therefore, the edge weight between node 6 and node 10 is 0.5.

[0195] Correspondingly, when the candidate feature representation of the associated next candidate modal node in the previous round is updated based on the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round can be updated based on each edge weight and based on the candidate feature representation of a candidate modal node.

[0196] Specifically, the node correlation between node i and node j is calculated using the following formula (22):

[0197] z ij =f(W a [W q h i |W k h j |e ij ]) Formula (22)

[0198] Among them, W a , W q , W k are three conversion matrices whose parameters can be obtained through training. “|” indicates a series operation, and f() indicates an activation function, which can be but is not limited to LeakyReLU, e ij represents the edge weight between node i and node j, e ij It can also be called a weight factor.

[0199] It should be noted that in the embodiments of the present application, corresponding edge weights may also be set for each candidate modal node. For example, in order to enhance the correlation between images and lines, corresponding edge weights may be set for the candidate modal nodes corresponding to the images and lines.

[0200] The following is a description of the invention in conjunction with a specific embodiment.

[0201] Application scenario: video playback scenario.

[0202] See also Fig. 9 As shown, video 2 and text 2 are obtained, wherein video 2 is a TV series, and text 2 is a film review corresponding to the TV series.

[0203] Based on the acquired video 2 and text 2, for video 2, multiple images and corresponding audios are extracted from video 2, and based on the audios, multiple lines are obtained. For text 2, multiple sentences are obtained from text 2, and words are used to connect sentences and lines. The images, lines, words, and sentences are encoded respectively, and the nodes corresponding to the images, lines, words, and sentences are obtained.

[0204] Based on the obtained nodes, the corresponding target node association relationship is constructed. The target node association relationship can also be called a heterogeneous graph neural network. Then, based on the target node association relationship, a bidirectional iterative update method is adopted to obtain the target feature representation corresponding to each node.

[0205] Then, based on the obtained target feature representations, multiple target modal nodes corresponding to the image are determined from the nodes corresponding to the image, multiple target modal nodes corresponding to the sentence are determined from the nodes corresponding to the sentence, and target summary information is generated based on the target modal nodes.

[0206] Based on the same inventive concept, the present application embodiment provides a summary generation device. Fig.10 As shown, it is a schematic diagram of the structure of the summary generation device 1000, which may include:

[0207] The node acquisition unit 1001 is used to obtain at least one candidate modal node corresponding to each modal information based on the acquired multiple modal information; each candidate modal node is used to represent a key data included in the corresponding modal information;

[0208] A relationship building unit 1002 is used to build a corresponding target node association relationship based on each obtained candidate modal node;

[0209] The feature representation unit 1003 is used to obtain the target feature representation corresponding to each candidate modal node respectively by adopting an iterative update method based on the target node association relationship, wherein in each round of iteration, the following operations are performed for each candidate modal node in turn: based on the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round is updated, and the node update directions of two adjacent iterations are opposite;

[0210] The summary generating unit 1004 is used to determine at least one target modality node from the candidate modality nodes based on the obtained target feature representations, and generate target summary information based on the at least one target modality node.

[0211] As a possible implementation, when the candidate feature representation of a candidate modal node is updated in the previous round based on the candidate feature representation of the associated next candidate modal node, the feature representation unit 1003 is specifically used to:

[0212] Based on the candidate feature representation of the one candidate modal node and based on the candidate feature representation of the next candidate modal node in the previous round, obtaining a node correlation between the one candidate modal node and the next candidate modal node;

[0213] Based on the node relevance, the candidate feature representation of the next candidate modal node in the previous round is updated.

[0214] As a possible implementation manner, when the candidate feature representation of the next candidate modal node in the previous round is updated based on the node relevance, the feature representation unit 1003 is specifically used to:

[0215] Based on the target node association relationship, at least one other candidate modal node associated with the next candidate modal node is selected from the candidate modal nodes except the one candidate modal node;

[0216] Based on the candidate feature representations corresponding to each of the at least one other candidate modality node and the candidate feature representations of the next candidate modality node in the previous round, obtaining the node correlation between the next candidate modality and the at least one other candidate modality node;

[0217] Based on the obtained relevance of each node, the candidate feature representation of the next candidate modal node in the previous round is updated.

[0218] As a possible implementation manner, when obtaining at least one candidate modal node corresponding to each modal information based on the obtained multiple modal information, the node acquisition unit 1001 is specifically used to:

[0219] If the modality type of one modality information among the multiple modality information is video, extracting audio and at least one candidate video frame from the one modality information, and determining at least one key data corresponding to the one modality information based on the audio and the at least one candidate video frame, and obtaining at least one candidate modality node corresponding to the one modality information based on the at least one key data;

[0220] If the modality type of one modal information among the multiple modal information is text, at least one text key information is extracted from the one modal information, and the at least one text key information is used as at least one key data corresponding to the one modal information, and based on the at least one key data, at least one candidate modal node corresponding to the one modal information is obtained.

[0221] As a possible implementation manner, when determining at least one key data corresponding to the one modality information based on the audio and the at least one candidate video frame, the node acquisition unit 1001 is specifically used to perform at least one of the following operations:

[0222] Extracting at least one target video frame from the at least one candidate video frame according to a preset extraction interval, and using each obtained target video frame as corresponding various key data;

[0223] Based on the audio, at least one audio text information is obtained, and each piece of audio text information obtained is used as corresponding various key data.

[0224] As a possible implementation, when constructing the corresponding target node association relationship based on each obtained candidate modal node, the relationship construction unit 1002 is specifically used to:

[0225] Based on the modality types corresponding to the multiple modal information, respectively, determining a node hierarchical relationship between the candidate modal nodes, wherein in the node hierarchical relationship, at least one candidate modal node corresponding to each modal information is located at the same level;

[0226] Determine the node connection relationship between the candidate modal nodes based on the key data represented by each of the candidate modal nodes;

[0227] Based on the node hierarchical relationship and the node connection relationship, the target node association relationship is obtained.

[0228] As a possible implementation manner, the relationship building unit 1002 is further configured to:

[0229] Based on the node hierarchical relationship contained in the target node association relationship, the node update order of each candidate modal node is determined.

[0230] As a possible implementation, when determining the node connection relationship between the candidate modal nodes based on the key data represented by each of the candidate modal nodes, the relationship construction unit 1002 is specifically used to:

[0231] Based on the key data represented by each of the candidate modal nodes, at least one target keyword is extracted, and the at least one target keyword is used as each connection node;

[0232] Based on the respective connection nodes, a node connection relationship between the respective candidate modal nodes is obtained, in which the candidate modal nodes at different levels are connected via the respective connection nodes.

[0233] As a possible implementation, the target node association relationship further includes each edge weight between each connection node and each candidate modality node;

[0234] Then, based on the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round is updated. The feature representation unit 1003 is specifically used for:

[0235] Based on the edge weights and the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round is updated.

[0236] As a possible implementation manner, when determining at least one target modality node from the candidate modality nodes based on the obtained target feature representations, the summary generating unit 1004 is specifically configured to:

[0237] Based on each obtained target feature representation, determining at least one target modal node corresponding to each of the multiple modal information from at least one candidate modal node corresponding to each of the multiple modal information;

[0238] When generating target summary information based on the at least one target modal node, the summary generating unit 1004 is specifically used for:

[0239] Based on at least one target modality node corresponding to each of the multiple modality information, obtaining summary sub-information corresponding to each of the multiple modality information;

[0240] Based on the obtained individual summary sub-information, target summary information is generated.

[0241] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0242] Regarding the device in the above embodiment, the specific manner in which each unit executes the request has been described in detail in the embodiment of the method, and will not be elaborated here.

[0243] In an embodiment of the present application, by updating the candidate feature representation through the candidate feature representation of adjacent candidate modal nodes, the candidate modal node corresponding to each modal information can learn the information contained in the candidate modal nodes corresponding to other modal information, thereby improving the accuracy of summary generation. In addition, through a two-way iterative update method, the interaction between multiple modal information can be enhanced, and the information flow can be enhanced, so that the candidate modal node corresponding to each modal information can better learn the information contained in the candidate modal nodes corresponding to other modal information, and fill the semantic gap between different modal information, thereby learning better feature representation, improving the accuracy of feature representation, and further improving the accuracy of summary generation.

[0244] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0245] Based on the same inventive concept, the embodiment of the present application also provides an electronic device. In one embodiment, the electronic device can be a server or a terminal device. Fig.11 As shown, it is a schematic diagram of the structure of a possible electronic device provided in an embodiment of the present application, Fig.11 In the embodiment, the electronic device 1100 includes: a processor 1110 and a memory 1120.

[0246] The memory 1120 stores a computer program executable by the processor 1110 , and the processor 1110 can perform the steps of the above summary generation method by executing the instructions stored in the memory 1120 .

[0247] The memory 1120 may be a volatile memory, such as a random-access memory (RAM); the memory 1120 may also be a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1120 may be any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1120 may also be a combination of the above memories.

[0248] The processor 1110 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1110 is configured to implement the above-mentioned summary generation method when executing the computer program stored in the memory 1120 .

[0249] In some embodiments, the processor 1110 and the memory 1120 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0250] The specific connection medium between the processor 1110 and the memory 1120 is not limited in the embodiment of the present application. In the embodiment of the present application, the processor 1110 and the memory 1120 are connected via a bus as an example. Fig.11 The connections between the other components are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.11 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or one type of bus.

[0251] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is executed on an electronic device, the computer program is used to enable the electronic device to execute the steps of the above-mentioned summary generation method. In some possible implementations, various aspects of the summary generation method provided by the present application can also be implemented in the form of a program product, which includes a computer program. When the program product is executed on an electronic device, the computer program is used to enable the electronic device to execute the steps in the above-mentioned summary generation method. For example, the electronic device can execute the following steps: Figure 2 Follow the steps shown in .

[0252] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (Compact Disk Read Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0253] The program product of the embodiment of the present application may adopt a CD-ROM and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a computer program, which may be used by or in combination with a command execution system, apparatus or device.

[0254] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a computer program for use by or in conjunction with a command execution system, apparatus, or device.

[0255] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0256] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for generating a summary, characterized in that: The method comprises: Based on the obtained multiple modal information, at least one candidate modal node corresponding to each of the modal information is obtained; each candidate modal node is used to represent a key data contained in the corresponding modal information; Based on each obtained candidate modal node, a corresponding target node association relationship is constructed; Based on the target node association relationship, an iterative update method is used to obtain the target feature representation corresponding to each candidate modal node, wherein in each round of iteration, the following operations are performed for each candidate modal node in turn: based on the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round is updated, and the node update directions of two adjacent iterations are opposite; Based on the obtained target feature representations, determining at least one target modality node from the candidate modality nodes, and generating target summary information based on the at least one target modality node; The step of constructing a corresponding target node association relationship based on each obtained candidate modal node includes: Based on the modality types corresponding to the multiple modal information, respectively, determining a node hierarchical relationship between the candidate modal nodes, wherein in the node hierarchical relationship, at least one candidate modal node corresponding to each modal information is located at the same level; Determine the node connection relationship between the candidate modal nodes based on the key data represented by each of the candidate modal nodes; Based on the node hierarchical relationship and the node connection relationship, the target node association relationship is obtained.

2. The method according to claim 1, characterized in that The updating of the candidate feature representation of the associated next candidate modal node in the previous round based on the candidate feature representation of a candidate modal node includes: Based on the candidate feature representation of the one candidate modal node and based on the candidate feature representation of the next candidate modal node in the previous round, obtaining a node correlation between the one candidate modal node and the next candidate modal node; Based on the node relevance, the candidate feature representation of the next candidate modal node in the previous round is updated.

3. The method according to claim 2, characterized in that The updating of the candidate feature representation of the next candidate modal node in the previous round based on the node relevance includes: Based on the target node association relationship, at least one other candidate modal node associated with the next candidate modal node is selected from the candidate modal nodes except the one candidate modal node; Based on the candidate feature representations corresponding to each of the at least one other candidate modality node and the candidate feature representations of the next candidate modality node in the previous round, obtaining the node correlation between the next candidate modality and the at least one other candidate modality node; Based on the obtained relevance of each node, the candidate feature representation of the next candidate modal node in the previous round is updated.

4. The method according to claim 1, characterized in that The obtaining of at least one candidate modal node corresponding to each type of modal information based on the obtained multiple modal information includes: If the modality type of one modality information among the multiple modality information is video, extracting audio and at least one candidate video frame from the one modality information, and determining at least one key data corresponding to the one modality information based on the audio and the at least one candidate video frame, and obtaining at least one candidate modality node corresponding to the one modality information based on the at least one key data; If the modality type of one modal information among the multiple modal information is text, at least one text key information is extracted from the one modal information, and the at least one text key information is used as at least one key data corresponding to the one modal information, and based on the at least one key data, at least one candidate modal node corresponding to the one modal information is obtained.

5. The method according to claim 4, characterized in that When determining at least one key data corresponding to the one modality information based on the audio and the at least one candidate video frame, at least one of the following operations is performed: Extracting at least one target video frame from the at least one candidate video frame according to a preset extraction interval, and using each obtained target video frame as corresponding various key data; Based on the audio, at least one audio text information is obtained, and each piece of audio text information obtained is used as corresponding various key data.

6. The method according to any one of claims 1 to 5, characterized in that After obtaining the target node association relationship based on the node hierarchical relationship and the node connection relationship, and before obtaining the target feature representation corresponding to each of the candidate modal nodes by adopting an iterative update method based on the target node association relationship, the method further includes: Based on the node hierarchical relationship contained in the target node association relationship, the node update order of each candidate modal node is determined.

7. The method according to any one of claims 1 to 5, characterized in that The determining the node connection relationship between the candidate modal nodes based on the key data represented by each of the candidate modal nodes includes: Based on the key data represented by each of the candidate modal nodes, at least one target keyword is extracted, and the at least one target keyword is used as each connection node; Based on the respective connection nodes, a node connection relationship between the respective candidate modal nodes is obtained, in which the candidate modal nodes at different levels are connected via the respective connection nodes.

8. The method according to claim 7, characterized in that The target node association relationship also includes the edge weights between each connection node and each candidate modality node; Then, based on the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round is updated, including: Based on the edge weights and the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round is updated.

9. The method according to any one of claims 1 to 5, characterized in that The determining at least one target modality node from the candidate modality nodes based on the obtained target feature representations includes: Based on each obtained target feature representation, determining at least one target modal node corresponding to each of the multiple modal information from at least one candidate modal node corresponding to each of the multiple modal information; The generating target summary information based on the at least one target modal node includes: Based on at least one target modality node corresponding to each of the multiple modality information, obtaining summary sub-information corresponding to each of the multiple modality information; Based on the obtained individual summary sub-information, target summary information is generated.

10. A summary generation device, characterized in that: include: A node acquisition unit, configured to obtain, based on the acquired multiple modal information, at least one candidate modal node corresponding to each of the modal information; each candidate modal node is used to represent a key data contained in the corresponding modal information; A relationship building unit, used to build a corresponding target node association relationship based on each obtained candidate modal node; A feature representation unit is used to obtain the target feature representation corresponding to each candidate modal node in an iterative update manner based on the target node association relationship, wherein in each round of iteration, the following operations are performed for each candidate modal node in turn: based on the candidate feature representation of a candidate modal node, the candidate feature representation of the associated next candidate modal node in the previous round is updated, and the node update directions of two adjacent iterations are opposite; A summary generating unit, configured to determine at least one target modality node from the candidate modality nodes based on the obtained target feature representations, and generate target summary information based on the at least one target modality node; Wherein, when constructing the corresponding target node association relationship based on each obtained candidate modal node, the relationship construction unit is specifically used for: Based on the modality types corresponding to the multiple modal information, respectively, determining a node hierarchical relationship between the candidate modal nodes, wherein in the node hierarchical relationship, at least one candidate modal node corresponding to each modal information is located at the same level; Determine the node connection relationship between the candidate modal nodes based on the key data represented by each of the candidate modal nodes; Based on the node hierarchical relationship and the node connection relationship, the target node association relationship is obtained.

11. The device according to claim 10, characterized in that When the candidate feature representation of a candidate modal node is updated in the previous round based on the candidate feature representation of the associated next candidate modal node, the feature representation unit is specifically used to: Based on the candidate feature representation of the one candidate modal node and based on the candidate feature representation of the next candidate modal node in the previous round, obtaining a node correlation between the one candidate modal node and the next candidate modal node; Based on the node relevance, the candidate feature representation of the next candidate modal node in the previous round is updated.

12. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 9.

13. A computer-readable storage medium, characterized in that: It comprises a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any method described in claims 1 to 9.

14. A computer program product, characterized in that It includes a computer program, which is stored in a computer-readable storage medium. The processor of the electronic device reads and executes the computer program from the computer-readable storage medium, so that the electronic device executes the steps of any method described in claims 1 to 9.

Images