Topic-based data synthesis method and device, terminal equipment and medium

Abstract

The embodiment of the invention provides a data synthesis method and device based on topics, terminal equipment and a medium, and belongs to the technical field of video processing. The method comprises the following steps: acquiring initial video content and initial voice content corresponding to an online multi-person dialogue; speaker separation processing is carried out on the initial voice content, and target voice content corresponding to the target user and voice position information corresponding to the target voice content are obtained; performing voice recognition on the target voice content to obtain initial text information; performing topic clustering analysis on the initial text information to obtain a target topic and related text information corresponding to the target topic; obtaining target position information corresponding to the related text information from the voice position information; obtaining target video content corresponding to the target position information from the initial video content, and obtaining target voice content corresponding to the target position information from the initial voice content; and performing data merging on the target video content and the target voice content to obtain a data synthesis result corresponding to the target topic.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a topic-based data synthesis method, device, terminal equipment and medium. Background Art

[0002] In some online sales and service scenarios, it is necessary to record online conversations between insurance agents and customers. This not only makes it easier for agents to record and trace after the meeting, but also allows customers to view conversation records to better understand the services provided by insurance agents and make purchasing decisions. However, when customers or insurance agents view video content, they cannot quickly locate the target location corresponding to the target topic, which greatly reduces the efficiency of video viewing for customers or insurance agents. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to provide a topic-based data synthesis method, apparatus, terminal device and medium, aiming to solve the problem in related technologies that the target position corresponding to the target topic cannot be quickly located, thereby greatly reducing the efficiency of video viewing by customers or insurance agents.

[0004] In a first aspect, an embodiment of the present invention provides a topic-based data synthesis method, comprising:

[0005] Obtaining initial video content and initial voice content corresponding to an online multi-person conversation;

[0006] Performing speaker separation processing on the initial voice content to obtain target voice content corresponding to a target user and voice position information corresponding to the target voice content;

[0007] Performing speech recognition on the target speech content to obtain initial text information;

[0008] Performing topic cluster analysis on the initial text information to obtain a target topic and related text information corresponding to the target topic;

[0009] Obtaining target location information corresponding to the relevant text information from the voice location information;

[0010] Obtaining target video content corresponding to the target position information from the initial video content, and obtaining target voice content corresponding to the target position information from the initial voice content;

[0011] The target video content and the target voice content are data-merged to obtain a data synthesis result corresponding to the target topic.

[0012] In a second aspect, an embodiment of the present invention provides a topic-based data synthesis device, comprising:

[0013] A data acquisition module, used to obtain initial video content and initial voice content corresponding to an online multi-person conversation;

[0014] A separation processing module, used for performing speaker separation processing on the initial voice content to obtain target voice content corresponding to a target user and voice position information corresponding to the target voice content;

[0015] A speech recognition module, used for performing speech recognition on the target speech content to obtain initial text information;

[0016] A cluster analysis module, used to perform topic cluster analysis on the initial text information to obtain a target topic and related text information corresponding to the target topic;

[0017] A position identification module, used to obtain target position information corresponding to the relevant text information from the voice position information;

[0018] A data determination module, configured to obtain target video content corresponding to the target position information from the initial video content, and to obtain target voice content corresponding to the target position information from the initial voice content;

[0019] The data synthesis module is used to merge the target video content and the target voice content to obtain a data synthesis result corresponding to the target topic.

[0020] In a third aspect, an embodiment of the present invention further provides a terminal device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of any topic-based data synthesis method provided in the specification of the present invention are implemented.

[0021] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any topic-based data synthesis method provided in the specification of the present invention.

[0022] The embodiment of the present invention provides a topic-based data synthesis method, device, terminal device and medium. The method obtains the initial video content and initial voice content of an online multi-person conversation, and then uses the speaker separation technology to extract the target voice content of each target user and the corresponding voice position information. This technology effectively separates the speeches of different speakers and avoids confusion of speech content. Next, the extracted target voice content is subjected to voice recognition to generate initial text information, and then the initial text information is clustered by topic clustering analysis, so as to identify the target topic and the relevant text information corresponding to the target topic. Then, the target position information related to the target topic is extracted from the voice position information, and the corresponding target video content is extracted from the initial video content using the target position information. At the same time, the corresponding target voice content is extracted from the initial voice content. Finally, the target video content and the target voice content are merged to generate a data synthesis result related to the target topic. Therefore, the complete data synthesis result related to the target topic can be obtained and displayed in real time, which greatly improves the interactive experience. Participants can view and understand the relevant information of the target topic more clearly. This not only improves the efficiency of viewing the target topic content, but also effectively solves the problem that the technology cannot quickly locate the corresponding position of the target topic, thereby greatly improving the efficiency of customers or insurance agents in the video viewing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 A flowchart of a topic-based data synthesis method provided by an embodiment of the present invention;

[0025] Figure 2 for Figure 1 A schematic flow chart of sub-step S104 of the topic-based data synthesis method;

[0026] Figure 3 A schematic diagram of a scenario for implementing the topic-based data synthesis method provided in this embodiment;

[0027] Figure 4 A schematic diagram of the module structure of a topic-based data synthesis device provided by an embodiment of the present invention;

[0028] Figure 5 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0031] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0032] The embodiment of the present invention provides a topic-based data synthesis method, apparatus, terminal device and medium. The topic-based data synthesis method can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant and a wearable device. The terminal device can be a server or a server cluster.

[0033] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0034] Please refer to Figure 1 , Figure 1 A flowchart of a topic-based data synthesis method provided in an embodiment of the present invention.

[0035] like Figure 1 As shown, the topic-based data synthesis method includes steps S101 to S107.

[0036] Step S101: obtaining initial video content and initial voice content corresponding to an online multi-person conversation.

[0037] For example, install a high-resolution video camera or use a video conferencing system with an integrated camera that can clearly capture the facial expressions and movements of each participant. Make sure the camera configuration covers all areas of the meeting room so that the entire conversation is fully recorded. Use a high-quality microphone array or voice capture device that can clearly capture the voice of each speaker.

[0038] For example, in the insurance field or the financial field, when an insurance agent or a financial agent is explaining a product to a customer, a video camera and a voice acquisition device are used to collect information to obtain initial video content and initial voice content corresponding to an online multi-person conversation.

[0039] Step S102: performing speaker separation processing on the initial voice content to obtain target voice content corresponding to the target user and voice position information corresponding to the target voice content.

[0040] Exemplarily, the initial speech content is subjected to noise reduction and sound quality optimization to improve the accuracy of speaker separation. Then, audio features (such as energy, spectrum, etc.) are used to detect speech activity in the initial speech content, thereby helping to identify and separate the speech of different speakers, and then decompose the mixed initial speech content into multiple independent speaker speech signals, that is, to obtain the target speech content corresponding to the target user. Then, by analyzing the timestamps or audio tags in the separation process, the time period when the target user speaks is recorded, so as to accurately track and determine the target user's speaking position, thereby obtaining the speech position information of the target speech content.

[0041] In some embodiments, the speaker separation processing is performed on the initial voice content to obtain the target voice content corresponding to the target user and the voice position information corresponding to the target voice content, including: obtaining the target voiceprint data corresponding to the target user; performing data segmentation on the initial voice content to obtain multiple sub-voice contents and the initial position information corresponding to the sub-voice contents; performing voiceprint similarity calculation on the target voiceprint data and the sub-voice contents to obtain a voiceprint similarity value; performing data classification on the multiple sub-voice contents according to the voiceprint similarity value to obtain the target voice content corresponding to the target user; and determining the voice position information corresponding to the target voice content according to the initial position information.

[0042] Exemplarily, a speech sample of the target user is collected, and then the target voiceprint data of the target user is extracted from the speech sample using voiceprint extraction technology. The initial speech content is cut into multiple sub-speech contents using an appropriate window size (such as a few seconds to tens of seconds), and the start and end time of each sub-speech content, i.e., the initial position information, is recorded.

[0043] For example, if the target users include insurance agents and customers, the voice samples corresponding to the insurance agents and the voice samples corresponding to the customers are collected respectively, and then the voiceprint data corresponding to the insurance agents and the voiceprint data corresponding to the customers are extracted.

[0044] Exemplarily, for each sub-voice content, its voiceprint features are extracted, and a voiceprint comparison algorithm, such as dynamic time warping (DTW), cosine similarity or other machine learning algorithms, is used to calculate the similarity between the target voiceprint data and the voiceprint features of each sub-voice content to obtain the voiceprint similarity value between each sub-voice content and the target voiceprint data.

[0045] Exemplarily, according to the voiceprint similarity value, a threshold is set to determine which sub-voice contents belong to the target user. The sub-voice contents with voiceprint similarity values ​​higher than the set threshold are classified as the target voice contents of the target user. And according to the previously recorded initial position information (i.e., the start and end time of each sub-voice content), the voice position information of the sub-voice contents classified as the target user in the initial voice content is determined.

[0046] Specifically, by calculating the voiceprint similarity between the target voiceprint data and the sub-voice content corresponding to the target user, each sub-voice content can be carefully analyzed and compared with the voiceprint data of the target user, thereby identifying the target voice content that best matches the characteristics of the target user. This precise matching not only improves the accuracy of recognition, but also ensures the reliability of the data synthesis results.

[0047] Step S103: Perform voice recognition on the target voice content to obtain initial text information.

[0048] Exemplarily, the target speech content is subjected to speech recognition using a speech recognition model to obtain initial text information. The speech recognition model may be an open source model, such as Kaldi, Mozilla DeepSpeech, or a commercial service, such as Google Cloud Speech-to-Text, Microsoft Azure Speech Service, etc.

[0049] Step S104: performing topic cluster analysis on the initial text information to obtain a target topic and related text information corresponding to the target topic.

[0050] Exemplarily, first, the initial text information is segmented according to punctuation marks to obtain multiple sub-text information, and then the Jieba word segmentation tool is used to process the sub-text information and split it into individual words or phrases. Next, stop word processing is performed to remove irrelevant common words, such as "de" (的), "le" (了), etc., to extract the initial keywords corresponding to each sub-text information. Next, advanced pre-trained language models such as BERT and GPT are used to generate in-depth word embedding representations for these initial keywords. Finally, the K-Means clustering algorithm is used to perform data clustering on the word embedding representations to obtain the clustering results. The K-Means algorithm can divide the vocabulary into different clusters according to the similarity between the word embeddings, and each cluster represents a potential target topic. Finally, each cluster in the clustering results is further analyzed and processed to obtain the target topic. Specifically, the keywords in each cluster can be viewed to understand the theme or concept they represent, thereby determining the target topic. Thus, the sub-text information corresponding to each initial keyword in the cluster corresponding to the target topic is determined as the relevant text information corresponding to the target topic.

[0051] In one embodiment, the topic clustering analysis of the initial text information to obtain the target topic and the relevant text information corresponding to the target topic is specifically as follows with reference to Figure 2 , step S104 includes: sub-steps S1041 to sub-step S1045.

[0052] Sub-step S1041, perform word segmentation processing on the initial text information to obtain the initial keywords corresponding to the initial text information.

[0053] Exemplarily, use TF-IDF or TextRank to perform word segmentation processing on the initial text information to obtain the initial keywords corresponding to the initial text information.

[0054] Sub-step S1042, obtain the left keywords and right keywords corresponding to the initial keywords from the initial text information.

[0055] Exemplarily, determine a window size for capturing the context of the initial keyword. The window size can be a fixed number of characters or words. For example, select a window size of 5, which means extracting two words on each side of the keyword. Traverse each initial keyword in the initial text information to obtain the vocabulary on the left side of the initial keyword. The vocabulary within the window can be extracted by extending a certain window size to the left from the position of the initial keyword to obtain the left keywords. Similarly, extend a certain window size to the right from the position of the initial keyword to extract the vocabulary within the window to obtain the right keywords.

[0056] In addition, if the initial keyword is at the beginning of the initial text information, the left window may exceed the scope of the initial text information. In this case, only the right words in the window are extracted. If the initial keyword is at the end of the initial text information, the right window may exceed the scope of the initial text information. In this case, only the left words in the window are extracted.

[0057] Sub-step S1043: calculating a first entropy value corresponding to the left keyword and a second entropy value corresponding to the right keyword from the initial text information.

[0058] Exemplarily, the number of occurrences of each left keyword in the initial text information is counted. The number of occurrences of each right keyword in the initial text information is counted. The number of occurrences of each left keyword is divided by the total number of occurrences of all left keywords to obtain the probability of each left keyword. And the number of occurrences of each right keyword is divided by the total number of occurrences of all right keywords to obtain the probability of each right keyword.

[0059] Exemplarily, the entropy value of the left keyword is calculated according to the probability of the left keyword to obtain the first entropy value of the left keyword. And the entropy value of the right keyword is calculated according to the probability of the right keyword to obtain the second entropy value of the right keyword. The first entropy value of the left keyword reflects the diversity and information content of the left keyword. A higher entropy value indicates that the distribution of the left keyword is more uniform and the amount of information is larger. The second entropy value of the right keyword reflects the diversity and information content of the right keyword. A higher entropy value indicates that the distribution of the right keyword is more uniform and the amount of information is larger.

[0060] Sub-step S1044: Calculate the semantic similarity of the initial keywords to obtain a target similarity value.

[0061] Exemplarily, a pre-trained word embedding model, such as Word2Vec, GloVe, or BERT, is selected to convert each initial keyword into a word vector. These models are able to map words into a high-dimensional vector space and capture the semantic information of the words. The target similarity value is obtained by calculating the cosine angle between the two word vectors. The target similarity value ranges from -1 (completely opposite) to 1 (completely similar).

[0062] In some embodiments, the initial text information includes at least first text information and second text information, the initial keywords include at least first keywords corresponding to the first text information and second keywords corresponding to the second text information, and the semantic similarity calculation of the initial keywords to obtain a target similarity value includes: calculating the first similarity corresponding to each first word in the first keyword in the second keyword; performing semantic similarity calculation on each of the first words and the second word in the second keyword to obtain a second similarity; obtaining the maximum value of the second similarity as a third similarity; and performing data fusion on the first similarity and the third similarity to obtain the target similarity value corresponding to the first text information and the second text information.

[0063] Exemplarily, the initial text information includes at least first text information and second text information, and the initial keyword includes at least a first keyword corresponding to the first text information and a second keyword corresponding to the second text information, that is, the initial text information contains multiple sub-text information, each sub-text information corresponds to a corresponding keyword.

[0064] Exemplarily, the second keyword includes multiple second words, and then the entity similarity of the first word in the second keyword is calculated, so that the entity similarity is determined as the first similarity. The total number of words corresponding to the second keyword is obtained, and the number of times the first word appears in the second text information corresponding to the second keyword is obtained, and then the first similarity is obtained by dividing the number of times and the total number of words.

[0065] Exemplarily, a pre-trained word embedding model, such as Word2Vec, GloVe or BERT, is selected, and each first word is converted into a first vector using the word embedding model, and each second word is converted into a second vector using the word embedding model, and then the cosine similarity is used to calculate the semantic similarity between the first vector and the second vector to obtain the second similarity, so that the maximum value of the second similarity is determined as the third similarity between the first word and the second keyword.

[0066] Exemplarily, the first similarity and the third similarity are summed to obtain the fourth similarity between the first word and the second keyword, and then the fourth similarity corresponding to each first word is solved, and then all the fourth similarities are summed to determine the corresponding target similarity value between the first text information and the second text information.

[0067] Specifically, data fusion of the first similarity and the third similarity can comprehensively consider information at different levels and obtain a more comprehensive target similarity value corresponding to the first text information and the second text information, thereby providing good support for the accuracy of subsequent topic clustering.

[0068] Sub-step S1045: performing topic cluster analysis on the initial text information according to the target similarity value, the first entropy value, and the second entropy value to obtain the target topic and the related text information corresponding to the target topic.

[0069] Exemplarily, the calculated target similarity is used to perform a preliminary cluster analysis on the initial text information. The target similarity provides the correlation information between the initial text information and the target topic, which is used to divide the initial text information into different categories or clusters. The preliminary clustering results are screened and adjusted. The target entropy value is then used as a screening criterion to filter out clusters whose entropy values ​​do not meet expectations. The target entropy value helps to evaluate the diversity and information content of each cluster, thereby optimizing the clustering results. According to the screened results, the final target clustering results are determined. These target clustering results represent the text groups screened based on the target entropy value, which can more accurately reflect the theme and structure of the text. The most representative keywords are extracted from each cluster in the target clustering results. By analyzing the text content in each cluster, the keywords that best represent the theme of the cluster are determined. Thereby, the extracted target keywords are defined as target topics, and the text information corresponding to each cluster in the target clustering results is determined as the relevant text information corresponding to the target topic.

[0070] Specifically, extracting the left and right keywords from the initial text information can deeply capture the meaning and relationship of words in their context. This context analysis method can more accurately reflect the distribution of information in the text. In this way, the clustering quality of the text can be optimized, thereby providing reliable support for subsequent video synthesis.

[0071] Step S105: Obtain target location information corresponding to the relevant text information from the voice location information.

[0072] Exemplarily, initial text information corresponding to the target voice content is obtained, and then text position information of the relevant text information in the initial text information is calculated, and then the text position information is quantified into target position information corresponding to the relevant text information in the voice position information.

[0073] Step S106: obtaining target video content corresponding to the target position information from the initial video content, and obtaining target voice content corresponding to the target position information from the initial voice content.

[0074] Exemplarily, the target position information is used to perform video segmentation on the initial video content to obtain the target video content corresponding to the target position information.

[0075] Exemplarily, the target position information is used to perform voice segmentation on the initial voice content to obtain the target voice content corresponding to the target position information.

[0076] Step S107: Merge the target video content and the target voice content to obtain a data synthesis result corresponding to the target topic.

[0077] Exemplarily, suitable tools or software are selected according to the needs. These tools can be video editing software (such as Adobe Premiere Pro, Final Cut Pro), or multimedia synthesis tools (such as Camtasia, Filmora). Thereby, the target video content and the target voice content are imported into the selected tool, and the target video content and the target voice content are synchronized to ensure that the relevant parts of the voice and the video are consistent. For example, it is ensured that the corresponding relationship between the commentary and the video screen is correct. Thereby, the video and voice content are edited, the volume is adjusted, transition effects are added, etc., as needed to optimize the synthesis effect and obtain the data synthesis result corresponding to the target topic.

[0078] In some embodiments, the data merging of the target video content and the target voice content to obtain a data synthesis result corresponding to the target topic includes: matching information with the target video content according to the relevant text information to obtain a matching value; filtering the target video content and the relevant text information according to the matching value to obtain intermediate video content and intermediate text information; and data merging the intermediate video content and the intermediate text information to obtain the data synthesis result corresponding to the target topic.

[0079] Exemplarily, key information or keywords are extracted from the relevant text information. The target video content is analyzed to extract the corresponding key information. This may include subtitles, video descriptions, or speech-to-text results. The key information of the relevant text information is matched with the key information corresponding to the target video content through matching scores, similarity calculations, keyword frequencies, etc. to obtain a matching value.

[0080] Exemplarily, the matching value is compared with a preset value. If the matching value is less than the preset value, the target video content and related text information corresponding to the matching value are eliminated; if the matching value is greater than or equal to the preset value, the corresponding target video content and related text information are retained to obtain the intermediate video content and intermediate text information. Next, the intermediate video content and intermediate text information are sorted to ensure that they can be effectively combined together. This step may include adjusting the format or synchronizing the time code. Then, use appropriate tools or software to merge the data, such as video editing software, audio and video synthesis tools, or text editing tools. Combining the intermediate video content with the intermediate text information may include adding text annotations to the video, or generating a video commentary based on the text information. Finally, the synthesized data synthesis results are exported to the required format, such as a video file, report document, or other applicable format.

[0081] Specifically, by matching the relevant text information with the target video content, the synthesis results can be ensured to be highly relevant, improving the accuracy and relevance of the information. The intermediate video content and intermediate text information obtained through screening can be better integrated together to ensure the coherence and consistency of the final synthesized content. Therefore, the combined data synthesis results can more clearly convey the core information of the target topic and enhance the expression effect of the information.

[0082] In some embodiments, the data merging based on the intermediate video content and the intermediate text information to obtain the data synthesis result corresponding to the target topic includes: using the intermediate text information using a speech synthesis algorithm to obtain corresponding initial speech information; adjusting speech parameters of the initial speech information to obtain target speech information; and data merging the target speech information and the intermediate video content to obtain the data synthesis result corresponding to the target topic.

[0083] Exemplarily, the intermediate text information is converted into initial voice information using a speech synthesis algorithm. The generated initial voice information is imported into the voice adjustment tool for processing. The volume of the voice is adjusted to ensure that the voice is clear and audible; the speech speed is adjusted as needed to match the intermediate video content; and the tone of voice is modified to suit the desired emotion or style. After completing these adjustments, the processed voice information is saved as the target voice information.

[0084] Exemplarily, the target voice information and the intermediate video content are both imported into the same synthesis tool to ensure that the target voice information and the intermediate video content are accurately synchronized on the time axis to ensure the consistency of the voice and video content, thereby obtaining the data synthesis result corresponding to the target topic.

[0085] Specifically, by synchronizing the target voice information with the intermediate video content, the target voice information and the intermediate video content can be highly consistent, thereby significantly improving the coherence of the overall information transmission. This synchronization process enables users to naturally accept consistent feedback of voice and visual content when viewing the data synthesis results, reducing the sense of discontinuity in information transmission, enhancing the integrity and fluency of the content, and ensuring the effect and accuracy of information transmission, so that the final content is more in line with expected needs.

[0086] In some implementations, the method further includes: determining corresponding topic association relationships between the target topics based on the relevant text information; and performing video splicing on the data synthesis results based on the topic association relationships to obtain a data splicing result.

[0087] Exemplarily, a relationship network between target topics can be constructed based on keywords extracted from relevant text information. First, by analyzing the co-occurrence frequency, contextual relevance, and correlation of the target topics, the direct and indirect connections between them are identified. Next, the identified association relationships are classified, and the relationship types between each pair of target topics are defined, such as causal relationships, contrast relationships, or parallel relationships, and the strength and nature of these relationships are annotated. Finally, based on these classification results, the specific association relationships between each pair of target topics are clarified, so that these relationships are integrated to form topic association relationships corresponding to all target topics.

[0088] For example, the data synthesis results are sequentially spliced ​​according to the topic association relationship to obtain the data splicing result. In addition, transition interval videos are inserted at the target topic transition. These videos can be transition animations, illustrations or short introduction clips to help users better understand the relationship between the target topics. By adding these interval videos, the separation between different topics is strengthened, making the switching of each topic clearer and improving the overall viewing experience of the video.

[0089] Specifically, by determining the topic association relationship between target topics, it can be ensured that the video content has a clear structure and logical order, so that users can better understand and follow the development of the video content.

[0090] Please refer to Figure 3 , Figure 3 A schematic diagram of a scenario for implementing the topic-based data synthesis method provided in this embodiment, such as Figure 3As shown, the method includes obtaining the initial video content and initial voice content of the online multi-person conversation, first extracting the target voiceprint data of the target user, and then cutting the initial voice content to obtain multiple sub-voice contents and their corresponding initial position information. The target voiceprint data and these sub-voice contents are subjected to voiceprint similarity calculation to obtain the voiceprint similarity value of each sub-voice content. According to the voiceprint similarity value, the sub-voice content is classified, the target voice content corresponding to the target user is identified, and its voice position information is determined. Next, voice recognition is performed on the target voice content to obtain initial text information. The initial text information is segmented to extract initial keywords. Further, the left keywords and right keywords related to the initial keywords are extracted from the initial text information, and the first entropy value of the left keywords and the second entropy value of the right keywords are calculated. Then, the semantic similarity of the initial keywords is calculated to obtain the target similarity value. Based on the target similarity value, the first entropy value and the second entropy value, the initial text information is subjected to topic clustering analysis to obtain the target topic and the related text information corresponding to the target topic. The target position information corresponding to the related text information is extracted from the voice position information. Furthermore, the target video content related to the target location information is extracted from the initial video content, and the target voice content related to the target location information is extracted from the initial voice content. Finally, the target video content and the target voice content are merged to generate the data synthesis result corresponding to the target topic. In this way, the complete data synthesis result related to the target topic can be obtained and displayed in real time, which greatly improves the interactive experience. Participants can view and understand the relevant information of the target topic more clearly. This not only improves the viewing efficiency of the target topic content, but also effectively solves the problem that the technology cannot quickly locate the corresponding position of the target topic, thereby greatly improving the efficiency of customers or insurance agents in the video viewing process.

[0091] See also Figure 4 , Figure 4A topic-based data synthesis device 200 is provided in an embodiment of the present application. The topic-based data synthesis device 200 includes a data acquisition module 201, a separation processing module 202, a speech recognition module 203, a cluster analysis module 204, a position recognition module 205, a data determination module 206, and a data synthesis module 207. The data acquisition module 201 is used to obtain initial video content and initial speech content corresponding to an online multi-person conversation; the separation processing module 202 is used to perform speaker separation processing on the initial speech content to obtain target speech content corresponding to a target user and speech position information corresponding to the target speech content; the speech recognition module 203 is used to perform speaker separation processing on the target speech content to obtain target speech content corresponding to a target user and speech position information corresponding to the target speech content; the speech recognition module 203 is used to perform speaker separation processing on the target speech content to obtain target speech content corresponding to a target user and speech position information corresponding to the target speech content; the speech recognition module 203 is used to perform speaker separation processing on the target speech content to obtain target speech content corresponding to a target user and speech position information corresponding to the target speech content; the speech recognition module 204 is used to perform speaker separation processing on the target speech content to obtain target speech content corresponding to a target user and speech position information corresponding to the target speech content; the speech recognition module 205 ... A clustering analysis module 204 is used to perform topic clustering analysis on the initial text information to obtain a target topic and related text information corresponding to the target topic; a position identification module 205 is used to obtain the target position information corresponding to the related text information from the voice position information; a data determination module 206 is used to obtain the target video content corresponding to the target position information from the initial video content, and to obtain the target voice content corresponding to the target position information from the initial voice content; a data synthesis module 207 is used to merge the target video content and the target voice content to obtain the data synthesis result corresponding to the target topic.

[0092] In some implementations, the separation processing module 202 performs, during the process of performing speaker separation processing on the initial voice content to obtain the target voice content corresponding to the target user and the voice position information corresponding to the target voice content:

[0093] Obtain target voiceprint data corresponding to the target user;

[0094] Performing data segmentation on the initial voice content to obtain a plurality of sub-voice contents and initial position information corresponding to the sub-voice contents;

[0095] Calculating the voiceprint similarity between the target voiceprint data and the sub-voice content to obtain a voiceprint similarity value;

[0096] Classifying the plurality of sub-voice contents according to the voiceprint similarity values ​​to obtain the target voice content corresponding to the target user;

[0097] The voice position information corresponding to the target voice content is determined according to the initial position information.

[0098] In some implementations, the cluster analysis module 204 performs, during the process of performing topic cluster analysis on the initial text information to obtain a target topic and related text information corresponding to the target topic:

[0099] Performing word segmentation processing on the initial text information to obtain initial keywords corresponding to the initial text information;

[0100] Obtaining a left keyword and a right keyword corresponding to the initial keyword from the initial text information;

[0101] Calculate a first entropy value corresponding to the left keyword and a second entropy value corresponding to the right keyword from the initial text information;

[0102] Calculating semantic similarity of the initial keywords to obtain a target similarity value;

[0103] The initial text information is subjected to topic clustering analysis according to the target similarity value, the first entropy value, and the second entropy value to obtain the target topic and the related text information corresponding to the target topic.

[0104] In some implementations, the initial text information includes at least first text information and second text information, the initial keywords include at least first keywords corresponding to the first text information and second keywords corresponding to the second text information, and the clustering analysis module 204, in the process of calculating the semantic similarity of the initial keywords to obtain the target similarity value, executes:

[0105] Calculating a first similarity between each first word in the first keyword and the second keyword;

[0106] Calculating the similarity of the meaning of each of the first words and the second word in the second keyword to obtain a second similarity;

[0107] Obtaining a maximum value of the second similarities as a third similarity;

[0108] The first similarity and the third similarity are data-fused to obtain the target similarity value corresponding to the first text information and the second text information.

[0109] In some implementations, the data synthesis module 207 performs, during the process of merging the target video content and the target voice content to obtain the data synthesis result corresponding to the target topic:

[0110] Performing information matching between the relevant text information and the target video content to obtain a matching value;

[0111] Performing data screening on the target video content and the related text information according to the matching value to obtain intermediate video content and intermediate text information;

[0112] Data is merged according to the intermediate video content and the intermediate text information to obtain the data synthesis result corresponding to the target topic.

[0113] In some implementations, the data synthesis module 207 performs, during the process of performing data merging according to the intermediate video content and the intermediate text information to obtain the data synthesis result corresponding to the target topic:

[0114] Using a speech synthesis algorithm to obtain corresponding initial speech information from the intermediate text information;

[0115] Adjusting the initial voice information by voice parameters to obtain target voice information;

[0116] The target voice information and the intermediate video content are data-merged to obtain the data synthesis result corresponding to the target topic.

[0117] In some implementations, the data synthesis module 207 further performs:

[0118] Determine the corresponding topic association relationship between the target topics according to the relevant text information;

[0119] The data synthesis results are subjected to video splicing according to the topic association relationship to obtain a data splicing result.

[0120] In some implementations, the topic-based data synthesis apparatus 200 may be applied to a terminal device.

[0121] It should be noted that those skilled in the art can clearly understand that, for the sake of convenience and brevity of description, the specific working process of the topic-based data synthesis device 200 described above can refer to the corresponding process in the aforementioned topic-based data synthesis method embodiment, and will not be repeated here.

[0122] See also Figure 5 , Figure 5 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention.

[0123] like Figure 5 As shown, the terminal device 300 includes a processor 301 and a memory 302, and the processor 301 and the memory 302 are connected via a bus 303, such as an I2C (Inter-integrated Circuit) bus.

[0124] Specifically, the processor 301 is used to provide computing and control capabilities to support the operation of the entire terminal device. The processor 301 can be a central processing unit (CPU), and the processor 301 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0125] Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0126] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the terminal device to which the embodiment of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0127] The processor is used to run a computer program stored in the memory, and implement any one of the topic-based data synthesis methods provided by the embodiments of the present invention when executing the computer program.

[0128] In one embodiment, the processor is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:

[0129] Obtaining initial video content and initial voice content corresponding to an online multi-person conversation;

[0130] Performing speaker separation processing on the initial voice content to obtain target voice content corresponding to a target user and voice position information corresponding to the target voice content;

[0131] Performing speech recognition on the target speech content to obtain initial text information;

[0132] Performing topic cluster analysis on the initial text information to obtain a target topic and related text information corresponding to the target topic;

[0133] Obtaining target location information corresponding to the relevant text information from the voice location information;

[0134] Obtaining target video content corresponding to the target position information from the initial video content, and obtaining target voice content corresponding to the target position information from the initial voice content;

[0135] The target video content and the target voice content are data-merged to obtain a data synthesis result corresponding to the target topic.

[0136] In some implementations, the processor 301, during the process of performing speaker separation processing on the initial voice content to obtain the target voice content corresponding to the target user and the voice position information corresponding to the target voice content, executes:

[0137] Obtain target voiceprint data corresponding to the target user;

[0138] Performing data segmentation on the initial voice content to obtain a plurality of sub-voice contents and initial position information corresponding to the sub-voice contents;

[0139] Calculating the voiceprint similarity between the target voiceprint data and the sub-voice content to obtain a voiceprint similarity value;

[0140] Classifying the plurality of sub-voice contents according to the voiceprint similarity values ​​to obtain the target voice content corresponding to the target user;

[0141] The voice position information corresponding to the target voice content is determined according to the initial position information.

[0142] In some implementations, the processor 301, in the process of performing topic cluster analysis on the initial text information to obtain a target topic and related text information corresponding to the target topic, executes:

[0143] Performing word segmentation processing on the initial text information to obtain initial keywords corresponding to the initial text information;

[0144] Obtaining a left keyword and a right keyword corresponding to the initial keyword from the initial text information;

[0145] Calculate a first entropy value corresponding to the left keyword and a second entropy value corresponding to the right keyword from the initial text information;

[0146] Calculating semantic similarity of the initial keywords to obtain a target similarity value;

[0147] The initial text information is subjected to topic clustering analysis according to the target similarity value, the first entropy value, and the second entropy value to obtain the target topic and the related text information corresponding to the target topic.

[0148] In some implementations, the initial text information includes at least first text information and second text information, the initial keyword includes at least a first keyword corresponding to the first text information and a second keyword corresponding to the second text information, and the processor 301 performs, in the process of calculating the semantic similarity of the initial keyword to obtain the target similarity value:

[0149] Calculating a first similarity between each first word in the first keyword and the second keyword;

[0150] Calculating the similarity of the meanings of each of the first words and the second words in the second keywords to obtain a second similarity;

[0151] Obtaining a maximum value of the second similarities as a third similarity;

[0152] The first similarity and the third similarity are data-fused to obtain the target similarity value corresponding to the first text information and the second text information.

[0153] In some implementations, the processor 301 performs, during the process of merging the target video content and the target voice content to obtain the data synthesis result corresponding to the target topic:

[0154] Performing information matching between the relevant text information and the target video content to obtain a matching value;

[0155] Performing data screening on the target video content and the related text information according to the matching value to obtain intermediate video content and intermediate text information;

[0156] Data is merged according to the intermediate video content and the intermediate text information to obtain the data synthesis result corresponding to the target topic.

[0157] In some implementations, the processor 301, during the process of merging data according to the intermediate video content and the intermediate text information to obtain the data synthesis result corresponding to the target topic, executes:

[0158] Using a speech synthesis algorithm to obtain corresponding initial speech information from the intermediate text information;

[0159] Adjusting the initial voice information by voice parameters to obtain target voice information;

[0160] The target voice information and the intermediate video content are data-merged to obtain the data synthesis result corresponding to the target topic.

[0161] In some implementations, the processor 301 further executes:

[0162] Determine the corresponding topic association relationship between the target topics according to the relevant text information;

[0163] The data synthesis results are subjected to video splicing according to the topic association relationship to obtain a data splicing result.

[0164] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned topic-based data synthesis method embodiment, and will not be repeated here.

[0165] An embodiment of the present invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any topic-based data synthesis method provided in the description of the embodiment of the present invention.

[0166] The storage medium may be an internal storage unit of the terminal device in the aforementioned embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device.

[0167] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0168] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0169] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A topic-based data synthesis method, characterized in that: The method comprises: Obtaining initial video content and initial voice content corresponding to an online multi-person conversation; Performing speaker separation processing on the initial voice content to obtain target voice content corresponding to a target user and voice position information corresponding to the target voice content; Performing speech recognition on the target speech content to obtain initial text information; Performing topic cluster analysis on the initial text information to obtain a target topic and related text information corresponding to the target topic; Obtaining target location information corresponding to the relevant text information from the voice location information; Obtaining target video content corresponding to the target position information from the initial video content, and obtaining target voice content corresponding to the target position information from the initial voice content; The target video content and the target voice content are data-merged to obtain a data synthesis result corresponding to the target topic.

2. The method according to claim 1, characterized in that The speaker separation processing is performed on the initial voice content to obtain the target voice content corresponding to the target user and the voice position information corresponding to the target voice content, including: Obtain target voiceprint data corresponding to the target user; Performing data segmentation on the initial voice content to obtain a plurality of sub-voice contents and initial position information corresponding to the sub-voice contents; Calculate the voiceprint similarity between the target voiceprint data and the sub-voice content to obtain a voiceprint similarity value; Classifying the plurality of sub-voice contents according to the voiceprint similarity values ​​to obtain the target voice content corresponding to the target user; The voice position information corresponding to the target voice content is determined according to the initial position information.

3. The method according to claim 1, characterized in that The performing a topic cluster analysis on the initial text information to obtain a target topic and related text information corresponding to the target topic includes: Performing word segmentation processing on the initial text information to obtain initial keywords corresponding to the initial text information; Obtaining a left keyword and a right keyword corresponding to the initial keyword from the initial text information; Calculate a first entropy value corresponding to the left keyword and a second entropy value corresponding to the right keyword from the initial text information; Calculating semantic similarity of the initial keywords to obtain a target similarity value; The initial text information is subjected to topic clustering analysis according to the target similarity value, the first entropy value, and the second entropy value to obtain the target topic and the related text information corresponding to the target topic.

4. The method according to claim 3, characterized in that The initial text information includes at least first text information and second text information, the initial keywords include at least first keywords corresponding to the first text information and second keywords corresponding to the second text information, and the performing semantic similarity calculation on the initial keywords to obtain a target similarity value includes: Calculating a first similarity between each first word in the first keyword and the second keyword; Calculating the similarity of the meanings of each of the first words and the second words in the second keywords to obtain a second similarity; Obtaining a maximum value of the second similarities as a third similarity; The first similarity and the third similarity are data-fused to obtain the target similarity value corresponding to the first text information and the second text information.

5. The method according to claim 1, characterized in that The step of merging the target video content and the target voice content to obtain a data synthesis result corresponding to the target topic includes: Performing information matching between the relevant text information and the target video content to obtain a matching value; Performing data screening on the target video content and the related text information according to the matching value to obtain intermediate video content and intermediate text information; Data is merged according to the intermediate video content and the intermediate text information to obtain the data synthesis result corresponding to the target topic.

6. The method according to claim 5, characterized in that The step of merging data according to the intermediate video content and the intermediate text information to obtain the data synthesis result corresponding to the target topic includes: Using a speech synthesis algorithm to obtain corresponding initial speech information from the intermediate text information; Adjusting the initial voice information by voice parameters to obtain target voice information; The target voice information and the intermediate video content are data-merged to obtain the data synthesis result corresponding to the target topic.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Determine the corresponding topic association relationship between the target topics according to the relevant text information; The data synthesis results are subjected to video splicing according to the topic association relationship to obtain a data splicing result.

8. A topic-based data synthesis device, characterized in that: include: A data acquisition module, used to obtain initial video content and initial voice content corresponding to an online multi-person conversation; A separation processing module, used for performing speaker separation processing on the initial voice content to obtain target voice content corresponding to a target user and voice position information corresponding to the target voice content; A speech recognition module, used for performing speech recognition on the target speech content to obtain initial text information; A cluster analysis module, used to perform topic cluster analysis on the initial text information to obtain a target topic and related text information corresponding to the target topic; A position identification module, used to obtain target position information corresponding to the relevant text information from the voice position information; A data determination module, configured to obtain target video content corresponding to the target position information from the initial video content, and to obtain target voice content corresponding to the target position information from the initial voice content; The data synthesis module is used to merge the target video content and the target voice content to obtain a data synthesis result corresponding to the target topic.

9. A terminal device, characterized in that: The terminal device includes a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the topic-based data synthesis method according to any one of claims 1 to 7 when executing the computer program.

10. A computer storage medium for computer storage, characterized in that: The computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the topic-based data synthesis method according to any one of claims 1 to 7.

Images