Message processing method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of Internet, in particular to a message processing method and device, electronic equipment and a storage medium, and aims to improve the message reading efficiency. The method comprises the steps that a message interface is presented, and the message interface comprises at least one chat message to be simplified; in response to a selection operation triggered for the at least one chat message, obtaining at least one selected target chat message; in response to a message simplification operation triggered for the at least one target chat message, presenting the generated simplified message and a text abstract corresponding to the simplified message; wherein the simplified message is obtained by integrating the key information in the at least one target chat message. The target chat message selected by the object can be automatically integrated and simplified, and the generated simplified message and the corresponding text abstract are directly presented to the object, so that the object can directly know the details of the target chat message by reading the simplified message, and the message reading efficiency is improved.

Description

Background Art

[0002] The development of the Internet has brought great convenience to people. For example, people can browse information, shop, socialize, etc. through the Internet. In social scenarios, objects can send chat messages such as text messages, voice messages, video messages, etc.

[0003] Take voice messages as an example. If one party sends multiple voice messages to another party, the current social scenario only supports the object to listen to each voice message one by one; or supports the object to convert the voice into text, but in this case, the object needs to actively convert one by one. Therefore, when encountering multiple long voice messages, the object's reading efficiency will be very low.

[0004] In summary, how to improve message reading efficiency is an urgent issue to be solved. Summary of the invention

[0005] Embodiments of the present application provide a message processing method, device, electronic device, and storage medium to improve message reading efficiency.

[0006] A message processing method provided in an embodiment of the present application includes:

[0007] Presenting a message interface, wherein the message interface includes at least one chat message to be condensed;

[0008] In response to a selection operation triggered on the at least one chat message, obtaining at least one selected target chat message;

[0009] In response to a message simplification operation triggered for the at least one target chat message, a generated simplification message and a text summary corresponding to the simplification message are presented; wherein the simplification message is obtained by integrating key information in the at least one target chat message.

[0010] Another message processing method provided in an embodiment of the present application includes:

[0011] Acquire at least one target chat message, where the at least one target chat message is selected by the client in response to a selection operation triggered on the at least one chat message, and the at least one chat message is a chat message to be condensed contained in a message interface presented by the client;

[0012] Extracting key information from the at least one target chat message and integrating the key information to obtain a simplified message;

[0013] Convert the simplified message into text form to obtain a text summary corresponding to the simplified message;

[0014] The condensed message and the text summary are sent to the client, so that the client presents the condensed message and the text summary in response to a message condensation operation triggered for the at least one target chat message.

[0015] An embodiment of the present application provides a message processing device, including:

[0016] A presenting unit, configured to present a message interface, wherein the message interface includes at least one chat message to be condensed;

[0017] A selection unit, configured to obtain at least one selected target chat message in response to a selection operation triggered on the at least one chat message;

[0018] A display unit is used to present a generated condensed message and a text summary corresponding to the condensed message in response to a message condensation operation triggered for the at least one target chat message; wherein the condensed message is obtained by integrating key information in the at least one target chat message.

[0019] Optionally, the message length of the at least one chat message reaches a length threshold;

[0020] Then the presenting unit is specifically used for:

[0021] In the message interface, the at least one chat message is presented, as well as operation prompt information; the operation prompt information is used to prompt the object how to perform message simplification processing.

[0022] Optionally, the display unit is further configured to perform at least one of the following operations:

[0023] After responding to the message simplification operation triggered for the at least one target chat message, canceling the display of the operation prompt information;

[0024] After the display time of the operation prompt information reaches a preset time threshold, the display of the operation prompt information is canceled.

[0025] Optionally, the selection unit is used to perform at least one of the following operations:

[0026] In response to a selection operation triggered by a pinch gesture of the object, taking a chat message to be simplified in a pinch area determined based on the pinch gesture as the at least one target chat message, and highlighting the at least one target chat message;

[0027] In response to a selection operation triggered by long pressing a relevant position of a single chat message and dragging the object, the chat messages to be streamlined in the dragged area are taken as the at least one target chat message, and the at least one target chat message is highlighted.

[0028] Optionally, the display unit is also used to present processing prompt information in the message interface after responding to a message simplification operation triggered for at least one target chat message and before presenting the generated simplification message and the text summary corresponding to the simplification message, wherein the processing prompt information is used to prompt the object that the message simplification processing is being performed.

[0029] Optionally, the message type of the target chat message is at least one of voice and video; the display unit is further used to present the current playback progress of the simplified message in response to a playback progress adjustment operation triggered for the simplified message after presenting the generated simplified message and the text summary corresponding to the simplified message;

[0030] The summary segment corresponding to the current playing progress is highlighted in the text summary.

[0031] Optionally, the display unit is also used for:

[0032] Backtracking prompt information is presented in the message interface, and the backtracking prompt information is used to prompt the object how to backtrack the original target chat message based on the summary fragment.

[0033] Optionally, the display unit is also used for:

[0034] In response to a message backtracking operation triggered based on the summary segment, presenting a target chat message associated with the summary segment and text content corresponding to the target chat message;

[0035] The associated segment of the summary segment is highlighted in the text content corresponding to the target chat message, and the current playback progress of the target chat message corresponds to the associated segment, wherein the associated segment is a segment whose semantic association with the summary segment reaches a preset association threshold.

[0036] Optionally, the simplified message includes multiple topics; then the display unit is specifically used to:

[0037] The text abstracts corresponding to the plurality of topics are presented in groups according to the topics, and the topic names corresponding to each group of text abstracts are marked.

[0038] Optionally, the display unit is also used for:

[0039] In response to a message backtracking operation triggered based on the topic name, a target chat message associated with the topic name and text content corresponding to the target chat message are presented.

[0040] Optionally, the display unit is also used for:

[0041] In response to a message backtracking operation triggered by inputting key information to be searched in the message interface, a target chat message associated with the key information and text content corresponding to the target chat message are presented.

[0042] Optionally, there are multiple associated target chat messages determined based on the message backtracking operation; then the display unit is specifically used to:

[0043] According to the release time of the message, the related target chat messages are presented in sequence, and the release information of the target chat message is marked before each target chat message.

[0044] Optionally, the display unit is specifically used for:

[0045] In the message interface, the associated target chat message is presented and highlighted; or

[0046] A message sub-interface is presented in the message interface, and an associated target chat message is presented through the message sub-interface.

[0047] Optionally, when the target message type is voice and the at least one target chat message corresponds to multiple sound-generating objects, the voice feature corresponding to the simplified message is any one of the following:

[0048] The speech feature of any one of the multiple sound-making objects;

[0049] A speech feature obtained by integrating the speech features of the plurality of sound-generating objects;

[0050] Preset template voice features.

[0051] Optionally, the message type of the target chat message is at least one of voice and video; the device further includes:

[0052] A play control unit, configured to present a play pause control at a relevant position of the simplified message when playing the simplified message;

[0053] In response to a pause operation triggered by the play pause control, pausing the playing of the simplified message;

[0054] In response to a play operation triggered by the play pause control, continue playing the condensed message.

[0055] Optionally, the display style of the simplified message is different from the display style of the at least one target chat message.

[0056] Optionally, the device further comprises:

[0057] A restoring unit, used for presenting restoration prompt information in the message interface, wherein the restoration prompt information is used for prompting the object how to restore the simplified message;

[0058] In response to the message restoration operation triggered based on the restoration prompt information, the simplified message is restored to the at least one target chat message.

[0059] Another message processing device provided in an embodiment of the present application includes:

[0060] a message acquisition unit, configured to acquire at least one target chat message, wherein the at least one target chat message is selected by the client in response to a selection operation triggered on the at least one chat message, and the at least one chat message is a chat message to be condensed contained in a message interface presented by the client;

[0061] A processing unit, configured to extract key information from the at least one target chat message and integrate the key information to obtain a simplified message; convert the simplified message into a text form to obtain a text summary corresponding to the simplified message;

[0062] A feedback unit is used to send the simplified message and the text summary to the client, so that the client presents the simplified message and the text summary in response to the message simplification operation triggered for the at least one target chat message.

[0063] Optionally, the message type of the target chat message is voice; then the processing unit is specifically configured to:

[0064] Merging the text contents corresponding to the at least one target chat message, and extracting key information from the merged result;

[0065] Convert the extracted key information into speech unit elements;

[0066] Based on the speech synthesis model, generating a speech waveform through the speech unit elements;

[0067] Determining the speech features corresponding to the simplified message by adjusting the model parameters of the speech synthesis model;

[0068] Based on the voice feature, the voice waveform is converted into an audio signal to obtain a simplified message corresponding to the at least one target chat message.

[0069] Optionally, the device further comprises:

[0070] A query unit, configured to receive an original message query request sent by the client; the original message query request is triggered by the client in response to a message backtracking operation or a message recovery operation;

[0071] Obtaining a target chat message associated with the information to be queried in the original message query request;

[0072] Returning the target chat message to the client, so that the client presents the target chat message;

[0073] Among them, if the original message query request is triggered by the client in response to a message backtracking operation, the information to be queried in the original message query request includes at least one of the following: a summary fragment in the text summary, a topic name of any one of the multiple topics contained in the streamlined message, and the key information to be searched entered in the message interface.

[0074] Optionally, the query unit is specifically used for:

[0075] Obtaining text content corresponding to each of the at least one target chat messages;

[0076] By using natural language processing technology, text matching is performed on each of the obtained text contents and the information to be queried;

[0077] The target chat message corresponding to the successfully matched text content is used as the target chat message associated with the original message query request; wherein, successful matching includes: the text content contains a fragment whose semantic relevance to the information to be queried reaches a preset relevance threshold.

[0078] An electronic device provided by an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-mentioned message processing methods.

[0079] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is executed on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the above-mentioned message processing methods.

[0080] An embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the above-mentioned message processing methods.

[0081] The beneficial effects of this application are as follows:

[0082] The embodiments of the present application provide a message processing method, device, electronic device and storage medium. Since the present application supports the subject to select the chat messages to be simplified by himself, and then automatically integrates and simplifies the target chat messages selected by the subject, and directly presents the generated simplified messages and corresponding text summaries to the subject, in this way, the subject no longer needs to view the target chat messages before simplification one by one, and can directly read the simplified messages and text summaries, which facilitates the subject to directly view the summaries of multiple target chat messages based on the simplified messages, understand the key information of these target chat messages, and improve the subject's message reading efficiency.

[0083] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0085] Figure 1 An optional schematic diagram of an application scenario in an embodiment of the present application;

[0086] Figure 2 This is a flowchart of a message processing method in an embodiment of the present application;

[0087] Figure 3 A schematic diagram of a message interface in an embodiment of the present application;

[0088] Figure 4 A schematic diagram of an operation prompt information in an embodiment of the present application;

[0089] Figure 5 A schematic diagram of another operation prompt information in an embodiment of the present application;

[0090] Figure 6 A schematic diagram of a message highlighting method in an embodiment of the present application;

[0091] Fig. 7A A schematic diagram of processing prompt information and simplifying messages in an embodiment of the present application;

[0092] Figure 7B A schematic diagram of a simplified message in an embodiment of the present application;

[0093] Figure 8A schematic diagram of a simplified process of voice and text messages in an embodiment of the present application;

[0094] Fig. 9 A schematic diagram of a message recovery process in an embodiment of the present application;

[0095] Fig.10 A schematic diagram of a point-slide-drag voice in an embodiment of the present application;

[0096] Fig.11 A schematic diagram of backtracking prompt information in an embodiment of the present application;

[0097] Fig.12 A schematic diagram of a presentation method of a backtracking target chat message in an embodiment of the present application;

[0098] Fig.13 A schematic diagram of another presentation method of a backtracking target chat message in an embodiment of the present application;

[0099] Fig.14 A schematic diagram of a text summary containing multiple topics in an embodiment of the present application;

[0100] Fig.15 This is a schematic diagram of a process of tracing back a target chat message based on a topic name in an embodiment of the present application;

[0101] Fig.16 A schematic diagram of a text summary including a search box in an embodiment of the present application;

[0102] Fig.17 This is a schematic diagram of a process of tracing back a target chat message based on a search term in an embodiment of the present application;

[0103] Fig.18 This is a flowchart of another message processing method in an embodiment of the present application;

[0104] Fig.19 This is a logic diagram of interaction between a client and a server in an embodiment of the present application;

[0105] Fig. 20 Another logic diagram of interaction between a client and a server in an embodiment of the present application;

[0106] Fig.21A A schematic diagram of a video simplification process in an embodiment of the present application;

[0107] Fig. 21B A schematic diagram of a video backtracking process in an embodiment of the present application;

[0108] Fig. 21C A schematic diagram of a video restoration process in an embodiment of the present application;

[0109] Fig.21D A schematic diagram of a simplified process of voice and video messages in an embodiment of the present application;

[0110] Fig. 22 A schematic diagram of the structure of a message processing device in an embodiment of the present application;

[0111] Fig.23 A schematic diagram of the structure of another message processing device in an embodiment of the present application;

[0112] Fig.24 A schematic diagram of a hardware structure of an electronic device to which an embodiment of the present application is applied;

[0113] Fig.25 The present invention is a schematic diagram of the hardware structure of another electronic device to which the embodiments of the present application are applied. DETAILED DESCRIPTION

[0114] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0115] The following is an introduction to some concepts involved in the embodiments of the present application.

[0116] Simplification: refers to removing redundant and unnecessary content while retaining the necessary content. In the embodiment of the present application, the chat message to be simplified refers to the chat message that can be simplified in the message interface. Specifically, the chat message to be simplified can be any chat message, or it can refer to a chat message whose length reaches a preset length threshold, such as a voice message with a duration of more than 15 seconds, a short video message with a duration of more than 2 minutes, a text message with more than 500 words, and so on.

[0117] Simplified message: refers to a message obtained by integrating the key information in one or more original chat messages, and contains the message content of one or more original chat messages. The simplified message is more concise than the original chat message, removing the redundant content in the original chat message and retaining the necessary content.

[0118] Key information: refers to the core content that a chat message wants to express. Specifically, it can refer to the message fragments, original words and sentences (such as names of people and project names) in the original target chat message. It can also be a summary of the original content in the chat message. Although it is not the original words and sentences, it is a new word or short sentence that appears after simplification, but it is semantically related.

[0119] Message backtracking operation: refers to the operation of searching for target chat messages that match the original target chat messages to be streamlined based on some information.

[0120] Message restoration operation: refers to the operation of restoring the simplified chat message (i.e., simplified message) obtained through message simplification processing to at least one target chat message before simplification.

[0121] The embodiments of the present application relate to artificial intelligence (AI) technology, and are designed based on natural language processing (NLP) technology and machine learning (ML) / deep learning technology in artificial intelligence.

[0122] Artificial intelligence technology mainly includes computer vision technology, speech processing technology, NLP technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions. With the research and progress of artificial intelligence technology, artificial intelligence has been studied and applied in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, robots, smart medical care, etc. It is believed that with the development of technology, artificial intelligence will be applied in more fields and play an increasingly important role. The message processing method in the embodiment of the present application can combine artificial intelligence with tasks such as speech synthesis and speech-to-text in these fields to intelligently generate a simplified version of speech and the corresponding text summary.

[0123] The key technologies of speech technology include automatic speech recognition (ASR), text to speech (TTS) and voiceprint recognition. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.

[0124] NLP is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0125] In the embodiments of the present application, the processing of voice chat messages, such as speech synthesis, speech-to-text, etc., mainly needs to be implemented in combination with the above-mentioned technologies.

[0126] ML is a multi-disciplinary cross-discipline involving probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. For example, the acoustic model, speech model, etc. in the embodiments of this application are all models learned based on ML technology.

[0127] In an embodiment of the present application, in combination with the above-mentioned technology, it is possible to simplify at least one target chat message so as to improve the message reading efficiency of the target.

[0128] The following is a brief introduction to the design concept of the embodiment of the present application:

[0129] The development of the Internet has brought great convenience to people. For example, people can browse information, shop, socialize, etc. through the Internet. In social scenarios, objects can send chat messages such as text messages, voice messages, video messages, etc.

[0130] In addition to the above-mentioned problems, when encountering multiple long voice messages, the reading efficiency of the target will be very low, and other types of chat messages will also have the same problem.

[0131] For example, in the case of video messages, if one party sends multiple video messages to another party, the current social scenario only supports the parties to browse each video message one by one; if the party wants to increase the speed of reading messages, they can also browse the video at double the speed during the browsing process, but this method may cause the party to ignore some details or key information in the video. Therefore, when encountering multiple video messages, the reading efficiency of the party will also be very low; and so on.

[0132] In summary, how to improve message reading efficiency is an urgent issue to be solved.

[0133] In view of this, the embodiments of the present application propose a message processing method, device, electronic device and storage medium. Since the present application supports the object to select the chat message to be simplified by itself, and then automatically integrates and simplifies the target chat message selected by the object, and directly presents the generated simplified message and the corresponding text summary to the object, in this way, the object no longer needs to view the target chat message before simplification one by one, and can directly read the simplified message and text summary, which is convenient for the object to directly view the summary of multiple target chat messages based on the simplified message, understand the key information of these target chat messages, and improve the message reading efficiency of the object.

[0134] In addition, this application also supports the object to quickly trace back the original target chat message based on some information, so that the object can view key messages in time.

[0135] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0136] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes two terminal devices 110 and a server 120.

[0137] In the embodiment of the present application, the terminal device 110 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals and other devices; a client associated with message processing may be installed on the terminal device, which may be software (such as a browser, instant messaging software, short video software, etc.), or a web page, a small program, etc. The server 120 is a background server corresponding to the software or web page, small program, etc., or a server specifically used for message processing, which is not specifically limited in the present application. The server 120 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0138] It should be noted that the message processing method in each embodiment of the present application can be executed by an electronic device, which may be a terminal device 110 or a server 120, that is, the method can be executed by the terminal device 110 or the server 120 alone, or can be executed jointly by the terminal device 110 and the server 120. For example, when the process is executed jointly by the terminal device 110 and the server 120, a client associated with message processing is installed on the terminal device 110, and the object can view the message interface through the client, and the message interface includes at least one chat message to be simplified. On this basis, the object can select the chat message for message simplification. Specifically, the client responds to the selection operation triggered for at least one chat message, and after obtaining the selected at least one target chat message, responds to the message simplification operation triggered for at least one target chat message, and sends a message simplification request to the server 120 through the terminal device 110. After receiving the request, the server 120 extracts key information from the at least one target chat message and integrates the information to obtain a simplified message; then, the simplified message is converted into text form to obtain a text summary corresponding to the simplified message; the simplified message and the text summary are sent to the client through the terminal device 110, and the simplified message and the text summary are presented on the client.

[0139] In an optional implementation, the terminal device 110 and the server 120 may communicate with each other via a communication network.

[0140] In an optional implementation, the communication network is a wired network or a wireless network.

[0141] It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0142] In an embodiment of the present application, when there are multiple servers, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as in the message processing method disclosed in the embodiment of the present application, the data related to the message processing involved can be saved on the blockchain, for example, chat messages, target chat messages, text summaries, etc.

[0143] In addition, the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving and other scenarios.

[0144] It is understandable that in the specific implementation of this application, related data such as chat messages are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0145] The message processing method provided by the exemplary embodiment of the present application is described below in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation methods of the present application are not limited in this regard.

[0146] See also Figure 2 As shown, it is an implementation flow chart of a message processing method provided in an embodiment of the present application. Taking the client as the execution subject as an example, the specific implementation process of the method is as follows:

[0147] S21: The client presents a message interface, which includes at least one chat message to be condensed.

[0148] In the embodiment of the present application, the message interface can be used to present various types of chat messages, such as text messages, voice messages, video messages, picture messages, external link sharing messages, etc. The message type of the chat message is not specifically limited herein.

[0149] In the embodiment of the present application, the chat message to be simplified refers to the chat message that can be used for simplification in the message interface. Specifically, the chat message to be simplified can be any chat message, or it can refer to a chat message whose message length reaches a preset length threshold, such as a voice message with a duration of more than 15 seconds, a short video message with a duration of more than 2 minutes, a text message with more than 500 words, etc., which is not specifically limited in this article.

[0150] See also Figure 3 As shown, it is a schematic diagram of a message interface in an embodiment of the present application. Figure 3 The message interface shown presents 7 chat messages. The object can select the target chat message that needs to be streamlined from these chat messages, or can view more chat messages by sliding up and down in the message interface, and select the target chat message that needs to be streamlined from more chat messages, and so on.

[0151] This application takes into account that the object may not understand the message simplification function, and can also provide operation prompts to the object. An optional implementation method is as follows:

[0152] When the message length of at least one chat message reaches a preset length threshold, at least one chat message is presented in the message interface, as well as operation prompt information; the operation prompt information is used to prompt the object how to perform message simplification processing.

[0153] That is, when there is at least one chat message in the message interface whose length reaches the preset length threshold, the object can be prompted in the message interface how to simplify the message, so that the object can refer to the prompt and perform subsequent message simplification operations to improve message reading efficiency.

[0154] It should be noted that in addition to the above conditions, the operation prompt can also be triggered in other ways, for example, when the message lengths of at least two chat messages are detected to reach a preset length threshold, the operation prompt information is presented in the message interface. Or, when the message lengths of at least three chat messages are detected to reach a preset length threshold, the operation prompt information is presented in the message interface, and so on.

[0155] In summary, the present application supports presenting operation prompt information in the message interface when it is detected that the message length of at least N chat messages reaches a preset length threshold, where N is a positive integer.

[0156] Taking voice messages as an example, if it is detected that the length of at least two voice messages reaches a preset length threshold (voices that meet this condition can be recorded as long voices), the operation prompt information will be presented in the message interface. The client can detect whether there are two or more long voices through a message counter, or detect whether there are two or more long voices by detecting the arrival of new messages, and then determine whether multi-voice synthesis is needed.

[0157] For example, see Voice Messages. Figure 4 As shown, it is a schematic diagram of an operation prompt information in an embodiment of the present application. Figure 4 As shown in S41, it is an operation prompt information in an embodiment of the present application, such as the prompt object "Pinch multiple voice messages with your fingers to quickly synthesize a simplified version of the voice."

[0158] See also Figure 5 As shown, it is a schematic diagram of another operation prompt information in an embodiment of the present application. Figure 5 As shown in S51, it is another operation prompt information in the embodiment of the present application, such as the prompt object "click and drag multiple voice messages to quickly synthesize a simplified version of the voice."

[0159] It should be noted that in the embodiment of the present application, the object can trigger subsequent message streamlining processing in a variety of ways, such as gesture triggering, convenient entry triggering, etc. Correspondingly, the operation prompt information listed above is just a simple example. Any operation prompt information is applicable to the embodiment of the present application and will not be described one by one here.

[0160] In the above implementation, the object can quickly select at least one target chat message through gesture sliding or a convenient entrance, thereby improving the efficiency of message streamlining.

[0161] Optionally, you can cancel the display of the operation prompt information by performing at least one of the following operations:

[0162] Operation 1: After responding to a message simplification operation triggered for at least one target chat message, cancel the display of the operation prompt information.

[0163] That is, after the object subsequently triggers a message simplification operation for the target chat message, the operation prompt information will no longer be displayed.

[0164] When the subsequent object opens the message interface again, the operation prompt information may be presented again, or no prompt may be given (the operation prompt is only given when the object opens the message interface for the first time and the above conditions are met). This article does not make specific restrictions.

[0165] Operation 2: After the display time of the operation prompt information reaches a preset time threshold, the display of the operation prompt information is canceled.

[0166] For example, the preset time threshold is set to 1 minute, and the operation prompt information will no longer be displayed after the display time reaches 1 minute.

[0167] Similarly, when a subsequent object opens the message interface again, the operation prompt information may be presented again, or no prompt may be presented. This document does not make any specific limitations.

[0168] In the above implementation manner, the operation prompt information can be used to guide the subject to quickly understand the triggering method of the message simplification operation, thereby improving the efficiency of message simplification.

[0169] S22: The client obtains at least one selected target chat message in response to the selection operation triggered on at least one chat message.

[0170] The target chat message selected by the object may be one or more.

[0171] Among them, if the target chat message selected by the object is one, generally speaking, the target chat message can be a chat message whose message length reaches a preset length threshold. In this way, by simplifying the target chat message, a simplified message corresponding to the key information in the target chat message can be obtained, thereby ensuring that the object can understand the main content of the target chat message more quickly.

[0172] If the object selects multiple target chat messages, there is no limit on the message length of these target chat messages; the message length of some target chat messages may reach the preset length threshold; or all of them may reach the preset length threshold, and so on.

[0173] In this way, by simplifying the multiple target chat messages, a simplified message can be obtained by extracting and integrating the key information in the multiple target chat messages, ensuring that the target can understand the main content of these target chat messages more quickly.

[0174] In addition, in an embodiment of the present application, if the object selects multiple target chat messages, these target chat messages can be the same message type, such as selecting multiple voice messages; or they can be different message types, such as selecting two voice messages and three video messages, and so on.

[0175] Among them, if the target chat message selected by the object is of the same message type, the generated simplified message is the same as the message type of the target chat message; if the target chat message selected by the object is of a different message type, the generated simplified message can be consistent with the message type that appears most frequently in the selected target chat message. For example, if two voice messages and three video messages are selected, the simplified message is a video message. If three voice messages and one text message are selected, the simplified message is a voice message. Alternatively, it can be consistent with any message type in the selected target chat message. Alternatively, the object is asked before the simplified message is generated (such as asking the object through a pop-up window about the message type of the simplified message to be generated), and the object chooses the message type of the simplified message to be generated, etc. This article does not make specific restrictions.

[0176] In an embodiment of the present application, there are many ways for an object to select a target chat message, such as the object can be selected through gestures, or through a convenient entry (such as presenting a selection control before each chat message in the message interface, the selection control can be presented directly, or it can be presented after the object triggers a certain operation, etc.).

[0177] Here are two ways to select via gestures:

[0178] Method 1: Selection based on pinch gesture.

[0179] In response to a selection operation triggered by a pinch gesture of the object, the chat messages to be condensed in the pinch area determined based on the pinch gesture are taken as at least one target chat message, and the at least one target chat message is highlighted.

[0180] The pinch gesture may be a two-finger pinch, a multi-finger pinch, or other pinch gestures, which are not specifically limited herein. Taking a two-finger pinch as an example, the pinch area refers to the area of ​​the object pinched by two fingers.

[0181] Take the target chat message as multiple voice messages as an example, Figure 4 As shown, the object can select the target chat message through the pinch gesture shown in S42. If the chat message area covered by the thumb and index finger of the object is used as the pinch area, the voice message in the area can be used as the target chat message.

[0182] Method 2: Selection based on tap-and-swipe gestures.

[0183] In response to a selection operation triggered by long pressing a relevant position of a single chat message and dragging the object, the chat messages to be streamlined in the dragged area are treated as at least one target chat message, and the at least one target chat message is highlighted.

[0184] The dragged area refers to the chat message area involved in the process of long pressing a single chat message and dragging it to a certain position.

[0185] The relevant position of the chat message may be the blank space after the chat message, or the display box of the chat message, or below the chat message, etc., which is not specifically limited in this document. Taking a voice message as an example, it may be the blank space after the voice message, or the voice bar of the voice message, or below the voice message, etc.

[0186] Take the target chat message as multiple voice messages as an example, Figure 5 As shown in S52, the object can click and long press the blank space after a voice message, and then drag it in any direction. If the object is dragged downward to the position shown in S53, the chat message area covered by S52 to S53 can be used as the dragged area, and the voice message in this area can be used as the target chat message.

[0187] After selecting the target chat message, the message simplification operation can be triggered to simplify the selected target chat message.

[0188] Based on the above implementation methods, the present application sets up a variety of message selection methods, which can adapt to various message processing-related clients, avoid accidental touches, and ensure that the object triggers instructions more accurately.

[0189] Regarding the method of selecting target chat messages based on gestures, in an embodiment of the present application, the client side can add a gesture recognizer on the message interface. When a sliding gesture is detected by the object, the client will select multiple chat messages, cache and store the received voice messages, and trigger the multi-chat message synthesis function.

[0190] Among them, the gesture recognizer can recognize the gesture operation of the object through the following technical implementation principles:

[0191] (1) Touching the screen: When an object touches the screen on the message interface, the touch event is captured and passed to the gesture recognizer.

[0192] (2) Coordinate recording: The gesture recognizer records the coordinates of the object touched for subsequent analysis.

[0193] (3) Path detection: The gesture recognizer detects the path based on the trajectory of the object's finger movement in combination with the above-recorded coordinates.

[0194] (4) Classification algorithm: The gesture recognizer uses a machine learning algorithm or other classification algorithm to classify the object's gesture operation into a specific gesture type based on the detected path.

[0195] (5) Action response: Once a gesture is recognized, the gesture recognizer can trigger a corresponding action or event, such as calling an instant messaging software function or performing a specific operation, etc.

[0196] In the embodiment of the present application, if it is identified that Figure 4 , Figure 5 (It may also be other selection gestures), then the target chat message selected by the object can be determined.

[0197] Taking voice messages as an example, it should be noted that the relevant technology does not support tapping or quickly selecting multiple voice messages. Long pressing the voice message will only display basic functions such as multiple selection, withdrawal, text conversion, favorites, and reminders. To select multiple voice messages, you can only click one by one, which is inefficient. In the embodiment of the present application, detecting gesture recognition can improve the operating efficiency of the object and facilitate the interactive mode of multiple operations of the object.

[0198] It should be noted that the above-listed target chat message selection methods are just simple examples, and any selection method is applicable to the embodiments of the present application, and will not be described one by one here.

[0199] In an embodiment of the present application, the selected target chat message can also be highlighted so that the object can clearly understand whether the chat message selected by him / her is accurate.

[0200] Specifically, there are many ways to highlight, including but not limited to: special processing of message background color, special processing of message border, special processing of message content, etc. The special processing here can be the transformation of color, line, content font, etc. The following takes the highlighting method of graying out the message background color as an example.

[0201] See also Figure 6 As shown, it is a schematic diagram of a message highlighting method in an embodiment of the present application. Figure 6 The example shown is the graying out of the message background color, wherein the durations of the four voice messages in the area covered by the gray background are 30 seconds, 59 seconds, 59 seconds and 29 seconds respectively, and these four voice messages are the target chat messages.

[0202] It should be noted that the above-mentioned methods of highlighting the target chat message are just simple examples. In fact, any method of highlighting the target chat message is applicable to the embodiments of the present application, and this document does not make any specific limitations.

[0203] S23: The client presents a generated condensed message and a text summary corresponding to the condensed message in response to a message condensation operation triggered for at least one target chat message; wherein the condensed message is obtained by integrating key information in at least one target chat message.

[0204] In an embodiment of the present application, the object can trigger the message simplification operation in a variety of ways. For example, after the object selects at least one target chat message through a gesture, the end gesture can trigger the message simplification operation for the selected target chat message.

[0205] For another example, after the object selects a target chat message, a pop-up window appears asking the object whether to perform message simplification processing on the selected target chat message. If the object confirms, the message simplification operation on the selected target chat message can be triggered.

[0206] For another example, after the object selects the target chat message, clicking the "Message Simplification" button triggers the message simplification operation for the selected target chat message, and so on.

[0207] It should be noted that any triggering method of the message simplification operation is applicable to the embodiments of the present application and is not specifically limited herein.

[0208] In an embodiment of the present application, after the object triggers a message simplification operation, at least one target chat message selected by the object can be simplification processed. After the simplification is completed, the generated simplification message and the corresponding text summary are presented.

[0209] Furthermore, while presenting the simplified message, the presentation of the target chat message may be cancelled, that is, the display of the target chat message is replaced by the simplified message.

[0210] In this way, the display position of the simplified message can be determined according to the position of the selected target chat message. If there is only one target chat message, the simplified message can be directly presented at the display position corresponding to the target chat message.

[0211] For another example, when there are multiple target chat messages, the multiple target chat messages can be sorted in the order of message publishing time, and the simplified message can be directly presented in the display position corresponding to the last target chat message among the multiple chat messages determined in the order of message publishing time. That is, sorting by time, the target chat message with the most recent time is displayed by default.

[0212] For example, there are three target chat messages that are published consecutively, and there are no other non-target chat messages between these target chat messages. If they are recorded as voice message 1, voice message 2, and voice message 3 in the order of publication time, and these three voice messages are published consecutively within one or two minutes, then the newly synthesized streamlined message can be directly presented in the display position corresponding to voice message 3 (such as the display position corresponding to voice message 3, or below this position, etc.).

[0213] For another example, there are three target chat messages that are not published consecutively, and there are other non-target chat messages between these target chat messages, such as voice message 1, voice message 2, and voice message 3 in the order of publication time. For example, voice message 1 and voice message 2 are published at 10:15, and voice message 3 is published at 11:30. Then the newly synthesized streamlined message can be directly presented in the display position corresponding to voice message 3.

[0214] In addition, the simplified message is presented while the target chat message is still presented. For example, the simplified message is presented in a pop-up window or floating layer, and the target can choose to close the pop-up window or floating layer after reading the simplified message.

[0215] It should be noted that the display position of the above-mentioned simplified message can be flexibly set according to actual needs, and any display position located or not located in the current message interface is applicable to the embodiments of the present application, and this document will not further limit it.

[0216] Since the present application supports the object to select the chat messages to be simplified by himself, and then automatically integrates and simplifies the target chat messages selected by the object, and directly presents the generated simplified messages and corresponding text summaries to the object, in this way, the object no longer needs to view the target chat messages before simplification one by one, but can directly read the simplified messages and text summaries, which makes it convenient for the object to directly view the summaries of multiple target chat messages based on the simplified messages, understand the key information of these target chat messages, and improve the object's message reading efficiency.

[0217] Optionally, after the client responds to a message simplification operation triggered for at least one target chat message, before presenting the generated simplification message and the text summary corresponding to the simplification message, processing prompt information may also be presented in the message interface. The processing prompt information is used to prompt the object that the message simplification processing is being performed.

[0218] See also Fig. 7A As shown, it is a schematic diagram of processing prompt information and simplified messages in an embodiment of the present application. Fig. 7A S71 in the embodiment of the present application is an example of processing prompt information. Taking voice message as an example, a prompt "speech intelligent synthesis..." may be displayed. After synthesizing the simplified message, a simplified message may be presented, such as Fig. 7AS72 in , and the text summary corresponding to the simplified message, such as Fig. 7A S73 in the above method cancels the presentation of the target chat message at the same time.

[0219] In the above implementation, the object can quickly generate a condensed message and a corresponding text summary based on at least one target chat message through a gesture slide or a convenient entrance, and can directly view the condensed message and the corresponding text summary, effectively improving the object's message reading efficiency.

[0220] Optionally, in order to facilitate the object to quickly distinguish which are the simplified chat messages (i.e., simplified messages) and which are the chat messages before simplification, the present application may also set the display style of the simplified messages to be different from the display style of at least one target chat message.

[0221] Specifically, the display style of the message display box and the display style of the message content may be set differently, and this document does not make any specific limitations.

[0222] For example, the border and background color of the display box of a simplified message can be set to be colorful to distinguish it from a regular chat message (which can be referred to as a regular message). For another example, the border of the display box of a simplified message can be set to a dotted line frame to distinguish it from a solid line frame of a regular message.

[0223] like Figure 7B As shown, it is a schematic diagram of a simplified message in an embodiment of the present application. Figure 7B Take the dotted line frame and solid line frame as an example. Figure 7B Concise messages are displayed in a dotted frame, and regular messages are displayed in a solid frame.

[0224] In the above implementation, regular messages and simplified messages are distinguished by different display styles, so that the object can quickly understand the simplified message for reading.

[0225] It should be noted that the above is an example of a target chat message being a voice message. In addition, the target chat message may also be a text message, a video message, etc. Alternatively, the target chat message may also be a combination of the above multiple message types. In the case where the target chat message is a plurality of message types, the following is a simple example in which the simplified message generated by default is consistent with the message type that appears most frequently in the selected target chat message.

[0226] For example, if the target chat message is a voice message + a text message, refer to Figure 8 As shown, it is a schematic diagram of the simplification process of voice and text messages in an embodiment of the present application.

[0227] like Figure 8As shown, the object selects a text message and two voice messages as target chat messages in the current message interface. In this case, the voice messages appear more frequently, and a simplified voice message can be generated, such as Figure 8 As shown in the right interface, a simplified voice is synthesized.

[0228] Specifically, after the object selects at least one target chat message through the client and triggers the message simplification operation, the client can send a message simplification request to the server. After receiving the request, the server can simplify the message based on these target chat messages and generate a simplification message. When the server generates a simplification message, it can directly extract the key information in the text message, and for the voice message, it can perform voice recognition, convert it into text content, and then extract the key information. Then, the extracted key information is integrated and converted into voice to obtain a simplification voice.

[0229] The original message corresponding to some of the voice content in the simplified voice may be a text message. In this case, a speech synthesis engine can be used to convert the text to be synthesized into a voice stream. The specific implementation method can be found in the following embodiments, such as the relevant description of the technical implementation principle of the speech synthesis engine listed below, and the repeated parts will not be repeated.

[0230] It should be noted that Figure 8 It is only taken as an example that the target chat message includes a voice message and a text message. In addition, the target chat message can also be one or more other message types. The specific streamlining process is similar and will not be repeated here.

[0231] On the basis of obtaining the above-mentioned simplified message, the object can also perform message backtracking, message recovery, etc. The following still takes the target chat message as a voice message as an example to continue to explain the message processing method in the embodiment of the present application:

[0232] Optionally, the present application also supports one-key restoration of objects, restoring simplified messages to regular messages before simplification. An optional implementation is as follows:

[0233] The client presents restoration prompt information in the message interface, and the restoration prompt information is used to prompt the object how to restore the simplified message; the client responds to the message restoration operation triggered by the restoration prompt information, and restores the simplified message to at least one target chat message.

[0234] In the embodiment of the present application, while presenting the simplified message, restoration prompt information can also be presented to prompt the object how to restore the simplified message to restore the regular message before the simplification. In addition, the restoration prompt information can also play another role, that is, to prompt the object that the simplified message is a simplified version of the regular message obtained by intelligent synthesis.

[0235] See also Fig. 9 As shown, it is a schematic diagram of a message recovery process in an embodiment of the present application. Fig. 9 Below the simplified message shown, there is a recovery prompt message "The above is a simplified version of the intelligently synthesized voice, restore". The "Restore" in the recovery prompt message is a text link. After the object clicks the text control, the message recovery operation can be directly triggered, and the simplified message can be restored to a regular message, such as Fig. 9 as shown in .

[0236] In the above implementation, when the object is not satisfied with the streamlined content or wants to view the content before the streamlined content, the message recovery message is directly triggered to return to the initial state before the message is streamlined.

[0237] Optionally, when the message type of the target chat message is at least one of voice and video, after presenting the generated simplified message and the text summary corresponding to the simplified message, you can also adjust the current playback progress of the simplified message and perform message backtracking based on the text summary.

[0238] An optional implementation is as follows:

[0239] In response to the playback progress adjustment operation triggered for the condensed message, the client presents the current playback progress of the condensed message and highlights the summary segment corresponding to the current playback progress in the text summary.

[0240] In the embodiment of the present application, the object can directly adjust the playback progress of the simplified message by dragging the progress bar of the voice message or video message.

[0241] In addition, it should be noted that the object may not actively trigger the playback progress adjustment operation for the simplified message. During the playback of the simplified message, the summary segment corresponding to the current playback progress is automatically highlighted in the text summary.

[0242] The abstract segment includes at least one character or word. In the embodiments of the present application, there are many ways to highlight the abstract segment in the text abstract, such as highlighting, bolding, italicizing, underlining, changing the font, changing the font size, changing the text color, etc., which are not specifically limited herein.

[0243] See also Fig.10 As shown, it is a schematic diagram of a point-slide drag voice in an embodiment of the present application. Fig.10 Taking the voice message shown as an example, the object can click and drag the voice. At this time, the voice will become a progress bar and show the current playing time position. When the object is dragged, the text summary below will also display the corresponding part of the currently playing voice in bold.

[0244] like Fig.10In the example, the subject drags the speech progress bar by clicking and sliding, and when the simplified version of the speech starts playing from the 6th second, the summary segment "Be healthier" corresponding to the currently playing speech is highlighted in the text summary.

[0245] In an embodiment of the present application, the client can extract highlighted keywords (that is, the summary segment corresponding to the current playback progress) according to the position of the condensed message playback, specifically including the steps of text segmentation, keyword extraction, audio to text conversion, alignment, and highlighting. For details, please refer to the following S1961 to S1965 sections, and the repeated parts will not be repeated.

[0246] In the above implementation, the object is supported to adjust the playback progress of the simplified message by itself, and in combination with the adjustment of the playback progress, the content corresponding to the currently playing part can be marked in real time in the text summary through different colors or underlines, etc., so that the object can know the text corresponding to the simplified messages at different times, so that the object can quickly trace back to the original target chat message.

[0247] On the basis of the above, the object can also perform message backtracking based on the summary fragment, that is, search for content corresponding to the summary fragment in the original target chat message.

[0248] Optionally, backtracking prompt information is presented in the message interface, and the backtracking prompt information is used to prompt the object how to backtrack the original target chat message based on the summary fragment.

[0249] It should be noted that the purpose of presenting the backtracking prompt information in this application is to facilitate the subject to understand how to backtrack the message. For some subjects, if the subject has already understood the function, the backtracking prompt may not be performed. For some subjects, such as new users, who are not familiar with the function, the backtracking prompt information may be presented to the subject on the client when the subject first uses the text summary of the simplified synthesized voice, so that the subject can quickly backtrack to the original voice.

[0250] The backtracking prompt information may also be displayed for a certain period of time and then no longer displayed, such as when the backtracking prompt information is displayed for 1 minute.

[0251] In the embodiment of the present application, message backtracking can also be performed through the backtracking prompt information prompt object. Fig.11 As shown, it is a schematic diagram of a backtracking prompt information in an embodiment of the present application. Fig.11 In S111, the backtracking prompt information is "Click the blue text to locate the original voice", where, subject to the requirements of the attached drawings, Fig.11 The bold “Last Day” is the “blue text”, which is the currently highlighted excerpt. The subject can click “Last Day” to go back to the corresponding part of the original speech.

[0252] In the above implementation, by presenting the backtracking prompt information to the object, it is convenient for the object to quickly backtrack to the original target chat message.

[0253] An optional implementation of message backtracking is as follows:

[0254] The client responds to a message backtracking operation triggered based on a summary segment, presents a target chat message associated with the summary segment, and text content corresponding to the target chat message; highlights an associated segment of the summary segment in the text content corresponding to the target chat message, and the current playback progress of the target chat message corresponds to the associated segment, wherein the associated segment is a segment whose semantic association with the summary segment reaches a preset association threshold.

[0255] In an embodiment of the present application, when an object performs message backtracking based on a summary fragment, the original target chat message located based on the summary fragment can be presented, and the text content corresponding to the target chat message can also be presented at the same time, and the text part corresponding to the above-mentioned summary fragment, i.e., the associated fragment in this article, can be highlighted in the text content.

[0256] The target chat message associated with the summary segment refers to a target chat message that includes the above-mentioned associated segment in at least one target chat message.

[0257] Specifically, the associated segment may be completely identical to the summary segment, that is, the summary segment is a message segment or an original word or sentence (such as a person's name or a project name) in the original target chat message.

[0258] In addition, the associated segment can also be a content with basically the same semantics as the summary segment, that is, the summary segment is obtained by further summarizing and generalizing the original message segment in the original target chat message, and is not the original words or sentences, but new words / short sentences that appear after simplification (semantically related to projects, names, etc.). In either case, the associated segment is a segment whose semantic relevance to the summary segment reaches a preset relevance threshold.

[0259] Among them, the semantic relevance represents the semantic similarity between two texts, which can be obtained by cosine similarity calculation. For example, the text content in the original target chat message is processed by word segmentation and the like, and then represented in the form of vectors. The cosine similarity between each vector obtained and the vector corresponding to the summary fragment is calculated respectively, and the calculation result is used as the corresponding semantic relevance; or other methods are used for calculation, such as the semantic analysis technology listed later, which is not specifically limited in this article.

[0260] For example, assuming that the value of the semantic relevance is 0 to 1, and setting the preset relevance threshold to 0.8, the part of the text content of the target chat message whose semantic relevance to the summary fragment reaches 0.8 is used as the associated fragment of the summary fragment, and the target chat message containing these associated fragments is used as the target chat message associated with the summary fragment.

[0261] In an embodiment of the present application, there are many ways to present a target chat message associated with a summary fragment, such as directly presenting the target chat message obtained by tracing back based on the summary fragment in a partial area of ​​the message interface; or presenting a message sub-interface in the message interface, and presenting the target chat message obtained by tracing back based on the summary fragment in the message sub-interface.

[0262] The following are the descriptions of the above two methods:

[0263] Method 1: In the message interface, the related target chat message is presented and highlighted.

[0264] There are many ways to highlight the target chat message in the message interface, including but not limited to: special processing of the message background color, special processing of the message border, special processing of the message content, etc. The special processing here can be a change in color, line, content font, etc. The following takes the highlighting method of graying out the message background color as an example.

[0265] See also Fig.12 As shown, it is a schematic diagram of a presentation method of a backtracking target chat message in an embodiment of the present application. Fig.12 In the figure, the target chat message presented in the area shown in S121 is the target chat message associated with the summary segment "the last day", and the associated segment in the target chat message is also "the last day". The target chat messages associated with the summary segments are highlighted by graying out the message background color in the area shown in S121. In addition, a backtracking structure prompt message "The above are the original voice keywords, click to continue playing" is also displayed at the bottom of the S121 area, and a play pause control S122 is presented after the voice bar of the target chat message. The subject can click on the control to continue playing from the position of "the last day", that is, to continue playing the original target chat message from the 36th second.

[0266] in, Fig.12 This example takes the case where only one target chat message associated with the summary segment is found. Of course, if multiple target chat messages associated with the summary segment are found, these target chat messages can be sorted according to the release time and / or the semantic relevance corresponding to the associated segment, etc., and the associated target chat messages are presented and highlighted in the message interface according to the sorting result.

[0267] For example, the target chat messages associated with the summary segments may be presented in sequence according to the release time of the messages.

[0268] For example, if two target chat messages containing "last day" are found, the two target chat messages can be presented in the message interface in sequence according to the release time.

[0269] For another example, the target chat messages associated with the summary segments may be presented in sequence according to the semantic association between the associated segments and the summary segments in the message.

[0270] For example, if a target chat message containing "the last day" is found, and the associated segment is the last day, and a target chat message containing "only one day left" is found, and the associated segment is only one day left, then according to the semantic relevance, the target chat message with a higher semantic relevance to the summary segment "the last day" can be presented first in the message interface, and so on.

[0271] For another example, when comprehensively considering the release time and semantic relevance, the release time may be prioritized for sorting, and for those with the same release time, the semantic relevance may be used for sorting, and so on. This article does not make specific limitations.

[0272] It should be noted that when there are multiple target chat messages, the method for determining the order of these target chat messages is just a simple example, and this article does not make specific restrictions. Similarly, the relevant content below can also adopt the same ordering method, and the repeated parts will not be repeated.

[0273] Additionally, when presenting the target chat message associated with the summary snippet, you can also refer to Fig.15 The listed method is to mark the release information of the target chat message before each target chat message. For details, please refer to the following embodiment, which will not be repeated here.

[0274] Method 2: Present a message sub-interface in the message interface, and present the associated target chat message through the message sub-interface.

[0275] The message sub-interface can be in the form of a pop-up window, floating layer, etc., which is not specifically limited in this article. This method does not affect the original chat content structure.

[0276] The following takes the floating layer as an example. Fig.13 As shown, it is a schematic diagram of another presentation method of the backtracking target chat message in the embodiment of the present application. Fig.13 This example is based on the case where only two target chat messages associated with the summary segment are found. Fig.13In the figure, the floating layer shown in S131 is a message sub-interface for presenting the traceback results in an embodiment of the present application. Two target chat messages are presented in S131, both of which are target chat messages associated with the summary segment "the last day". The associated segment in these two target chat messages is also "the last day". The presentation order of these two target chat messages can be determined according to the release time, etc. For details, please refer to the above embodiments, and the repeated parts will not be repeated.

[0277] In addition, at the bottom of the S132 area, a backtracking structure prompt message "The above are the original voice keywords, click to continue playing" is displayed, and a play pause control is presented after the voice bar of each target chat message in S132. As shown in S132, the subject can click the control to continue playing from the position of "the last day", that is, to continue playing the original target chat message from the 36th second. Similarly, for another target chat message, the subject can click the play pause control after the voice bar of the message, that is, to continue playing from the 48th second.

[0278] Additionally, when presenting the target chat message associated with the summary snippet, you can also refer to Fig.15 The listed method is to mark the release information of the target chat message before each target chat message. For details, please refer to the following embodiment, which will not be repeated here.

[0279] In summary, it should be noted that in the embodiment of the present application, when the summary segment is the content that exists in the original target chat message (such as the message segment, original words and sentences in the original target chat message), the starting and ending points of the corresponding associated segment are determined according to the voice played above. Fig.12 , Fig.13 When the object plays the voice, the voice content also starts from the "last day". Fig.12 , Fig.13 shown.

[0280] When the summary segment is not the content of the original target chat message, but a new word / short sentence that appears after simplification, the server will analyze the text content of the target chat message according to the implementation principle of semantic analysis technology, and record the sentence with the highest semantic correlation with the summary segment (it can also be a sentence exceeding the preset correlation threshold) as the associated segment, and present the target chat message containing the associated segment to the object. At this time, the associated segment will also be highlighted. When the object plays the voice, the voice content will also start playing from the associated segment.

[0281] In the above implementation, when the subject is viewing the simplified version of the voice, the subject can quickly trace back to the original voice message according to the converted highlighted keywords, which can not only improve the subject's reading efficiency, but also facilitate the subject to view the summary of multiple voice messages by listening to the voice, and also facilitate the subject to trace back to the key messages in time.

[0282] In addition, the client side can automatically locate the voice position corresponding to the currently highlighted keyword. When the object clicks to play, the original voice clip content corresponding to this keyword can be quickly heard, saving the time of listening to the voice from beginning to end.

[0283] Optionally, a condensed message may also contain multiple topics. In this case, when presenting the text summary corresponding to the condensed message, the text summaries corresponding to the multiple topics may be presented in groups according to the topics, and the topic names corresponding to each group of text summaries may be marked.

[0284] In an embodiment of the present application, natural language processing technology can be used to determine whether the original voice contains multiple topics. When it contains multiple topics, the simplified message will also contain multiple topics, and the text summary corresponding to the simplified message will also contain multiple topics. The client-side display logic can be grouped and displayed based on the topic name, and the object can trace back the message and view the summary of multiple voice messages.

[0285] The subject name may be a topic, a person's name, a project name, etc., and this article does not make any specific restrictions.

[0286] In an embodiment of the present application, if the message content of at least one target chat message contains multiple topics, before presenting the text summary of the condensed message, the text summary can be classified through natural language processing technology, and texts of different topics can be classified separately. When presenting the text summary of the condensed message, different texts can be grouped and displayed according to the topic name, and the corresponding topic name is marked in front of each group of texts. The object can select the corresponding topic for backtracking as needed.

[0287] See also Fig.14 As shown, it is a schematic diagram of a text summary containing multiple topics in an embodiment of the present application. For example, the message content of the original multiple target chat messages contains multiple topics, such as Fig.14 As shown in section S141 of the figure, it includes Topic 1, Topic 2 and Topic 3, and the text summaries corresponding to each topic are presented in groups.

[0288] Optionally, the topic name may be highlighted, such as in bold; in addition, the text summary corresponding to each topic may also be displayed differently, such as using text of different colors or using fonts of different colors, etc., which is not specifically limited in this article.

[0289] In the above implementation, the text summary is presented in groups and categories based on themes, so that the subject can quickly trace back the target chat message corresponding to the subject related to the subject.

[0290] In the embodiment of the present application, in addition to the message backtracking methods listed above, the object can also perform message backtracking based on the topic name. An optional implementation method is as follows:

[0291] In response to the message backtracking operation triggered based on the topic name, the client presents the target chat message associated with the topic name and the text content corresponding to the target chat message.

[0292] Specifically, the object can trigger the message backtracking operation by clicking or long pressing, select at least one topic name, and then present the target chat message associated with the topic name selected by the object. There are many specific presentation methods, such as directly presenting the target chat message obtained by backtracking based on the topic name in a partial area of ​​the message interface; or presenting a message sub-interface in the message interface, and presenting the target chat message obtained by backtracking based on the topic name in the message sub-interface, and so on.

[0293] Optionally, when there are multiple target chat messages associated with a selected topic name, the target chat messages associated with the topic name can be presented in sequence according to the release time of the messages, and the release information of the target chat message can be marked before each target chat message.

[0294] Among them, the earlier the message is released, the earlier it will be presented in the order.

[0295] In the embodiment of the present application, the published information includes but is not limited to part or all of the following: publishing time, publishing object, message length, etc.

[0296] Among them, taking voice message as an example, the message length specifically refers to the voice duration, the publishing time specifically refers to the sending time of the original voice message, and the publishing object specifically refers to the name, nickname, etc. of the sender of the original voice message.

[0297] For example, when an object needs to review the voice related to itself in the work group, the server can first convert the voice into text, and then use natural language processing technology to match the text to match keywords related to itself (such as its own name or the name of the project it is interested in). After the match is successful, the server can extract the matched text to form a list of voice clips related to itself, and feed it back to the client, which will present it for the object to review.

[0298] Specifically, the client can arrange the list of voice clips related to itself in chronological order on the page display, and mark each voice clip with the sender's name, sending time, voice duration and other information to facilitate quick browsing and positioning of the object.

[0299] The following example shows the target chat message obtained by tracing back based on the topic name directly in a part of the message interface:

[0300] like Fig.15 As shown, it is a schematic diagram of a process of tracing back a target chat message based on a topic name in an embodiment of the present application. Fig.15 In the text summary of the simplified message shown on the left, the summary content corresponding to each topic name is grouped and presented in units of topics. For example, if Topic 1 is about oneself, when the subject wants to view the original voice related to oneself, he can click the topic name of Topic 1, and then the following summary content will be presented: Fig.15 In the backtracking results shown on the right side, suppose two original voices related to topic 1 are found and can be sorted in order of release time, such as Fig.15 As shown in , the original voice with earlier release time is presented first.

[0301] Among them, part S151 is an example of publishing information in the embodiment of the present application, and part S152 is the text content corresponding to the original voice, and the description related to the topic name can also be highlighted in the text content. For example, when topic 1 is a person's name, the person's name can be highlighted in the text content, and so on.

[0302] It should be noted that the above-listed information release content and display location are simple examples, and in fact, there may be other ways, which are not specifically limited in this article.

[0303] In the above implementation, the information dimension of the chat message is further enriched by presenting the published information, so that the object can quickly understand the details of the target chat message.

[0304] Optionally, the present application may also provide a search function on the page, allowing the subject to directly enter the key information to be searched to quickly locate the relevant target chat message. An optional implementation is as follows:

[0305] In response to a message backtracking operation triggered by inputting key information to be searched in a message interface, the client presents a target chat message associated with the key information and text content corresponding to the target chat message.

[0306] In the embodiment of the present application, the key information may be a keyword, a word, a sentence, etc. For example, the subject may use keywords such as his own name or the name of a project he is concerned about as key information, and directly search for target chat messages that match these key information.

[0307] Specifically, a search box can be set in the text summary part of the simplified message, and the object can enter the key information to be searched in the search box to search. Fig.16 As shown, it is a schematic diagram of a text summary including a search box in an embodiment of the present application. Fig.16 As shown in S161, it is an example of a search box in an embodiment of the present application. For example, the object can search in the search box using his own name as a keyword.

[0308] In the embodiment of the present application, the target chat message associated with the key information to be searched input by the object is presented in many specific presentation ways, such as directly presenting the target chat message obtained by tracing back based on the key information in a partial area of ​​the message interface; or presenting a message sub-interface in the message interface, presenting the target chat message obtained by tracing back based on the key information in the message sub-interface, etc. For details, please refer to the above embodiment, and the repeated parts will not be repeated.

[0309] Take the example of directly presenting the target chat message obtained based on key information backtracking in a partial area of ​​the message interface:

[0310] See also Fig.17 As shown, it is a schematic diagram of a process of tracing back a target chat message based on a search term in an embodiment of the present application. Fig.17 The text summary of the simplified message shown on the left shows a search box, in which the subject can enter the key information to be searched, such as the name "Xiao Li", and then click search to display the following Fig.17 The backtracking results shown on the right side of the figure, assuming that two original voices related to Xiao Li are found, can be sorted in order of release time, such as Fig.17 As shown in , the original voice with earlier release time is presented first.

[0311] In addition, you can also refer to Fig.15 The listed method marks the release information of the target chat message before each target chat message, which will not be repeated here.

[0312] In the above implementation, by providing a search function, the object is supported to flexibly search for different target chat messages according to its own needs, the operation is simpler and more convenient, and the object's operating experience is enhanced to improve the object's stickiness.

[0313] Optionally, the target chat message type is at least one of voice and video. In this case, the simplified message generated after message simplification is also in the form of voice or video and is playable. When playing the simplified message, a play and pause control can be presented at the relevant position of the simplified message. The object can control the playback of the simplified message based on the control. For example:

[0314] The client pauses playing the simplified message in response to a pause operation triggered by the play pause control;

[0315] The client continues to play the condensed message in response to the play operation triggered by the play pause control.

[0316] Still Fig.11 As shown in the example, Fig.11 S112 in the figure is an example of a play pause control in an embodiment of the present application. When playing the simplified version of the voice, the play pause control after the voice is in a pauseable state, which can be understood as a pauseable button. When the object clicks the pause button, the voice will stop playing, and the play pause control will automatically become a playable state. At this time, it can be understood that the pause button will automatically become a play button. If the object clicks the play button at this time, the voice will continue to play based on the point where the object was previously paused.

[0317] In the above implementation, by setting the play pause control, the object can flexibly view the simplified message according to its own needs, the operation is simpler and more convenient, and the operation experience of the object is enhanced to improve the object stickiness.

[0318] It should be noted that in the above embodiments, voice messages are mainly used as examples. When the target chat message is other message types, the message processing method is the same. Please refer to the above embodiments for details, and this article will not go into details one by one.

[0319] In addition, in the embodiment of the present application, if one or more target chat messages selected by the object are all voices, and these target chat messages correspond to the same sound object, such as voices released by the same speaker, the sound object corresponding to the synthesized simplified voice remains unchanged. If the sound object of the voice to be simplified is Xiao Wang, the sound object of the synthesized simplified voice is also Xiao Wang, and the voice features such as timbre, speech speed, volume, pitch, intonation, voice quality, and pronunciation method corresponding to the synthesized simplified voice can be determined based on voice 1, and remain consistent or close to the voice features of Xiao Wang.

[0320] If these target chat messages correspond to multiple sound-producing objects, the speech features corresponding to the simplified messages are any of the following:

[0321] Voice feature 1: Voice feature of any one of multiple voice-generating objects.

[0322] That is, if multiple voices are voices of different sounding objects, when synthesizing a simplified message, one of the sounding objects can be used as a reference to generate a simplified message. For example, if there are three voices to be simplified, the sounding object of voice 1 is Xiao Li, the sounding object of voice 2 is Xiao Zhang, and the sounding object of voice 3 is Xiao Bai, then the sounding object of the synthesized simplified version of the voice can be any one of Xiao Li, Xiao Zhang and Xiao Bai.

[0323] Taking Xiao Li as an example, the voice features such as timbre, speaking speed, volume, pitch, intonation, voice quality, and pronunciation method corresponding to the synthesized simplified voice can be determined based on voice 1, and remain consistent with or close to Xiao Li's voice features.

[0324] Speech feature 2: Speech feature obtained by integrating the speech features of multiple sound-making objects.

[0325] Still taking the above example, the voice features corresponding to the simplified version of the voice, such as timbre, speaking speed, volume, pitch, intonation, voice quality, and pronunciation method, can be determined based on voice 1, voice 2, and voice 3.

[0326] Voice feature three: preset template voice feature.

[0327] This means that some template voices can be preset. When the target chat message corresponds to multiple sound-producing objects, the template voice is selected, and a simplified version of the voice is generated based on the voice features of the voice.

[0328] The above parameters may also be adjusted according to the object's preferences. Specifically, the preferences may include the object's preferences for sound effects, such as liking a gentler, lower, or higher pitched tone, etc., which is not specifically limited herein.

[0329] Based on the above implementation, it supports the generation of personalized voice customization, and automatically adjusts the sound effects of the audio according to the voice characteristics and preferences of the object to achieve the best auditory effect.

[0330] It should be noted that the above mainly describes the message processing method in the embodiment of the present application from the client side. The message processing method in the embodiment of the present application is further described from the server side below:

[0331] See also Fig.18 As shown, it is an implementation flow chart of another message processing method provided in an embodiment of the present application. Taking the server as the execution subject as an example, the specific implementation process of the method is as follows:

[0332] S181: The server obtains at least one target chat message.

[0333] The at least one target chat message is selected by the client in response to a selection operation triggered on the at least one chat message, and the at least one chat message is a chat message to be condensed contained in a message interface presented by the client.

[0334] In the embodiment of the present application, after the object selects at least one target chat message through the client and triggers the message simplification operation, the client can send a message simplification request to the server. After receiving the request, the server can perform message simplification based on these target chat messages to generate a simplification message. The specific operations are as follows S182-S184:

[0335] S182: The server extracts key information from at least one target chat message and integrates the key information to obtain a simplified message.

[0336] S183: The server converts the simplified message into text form and obtains a text summary corresponding to the simplified message.

[0337] Among them, when extracting key information in the target chat message, the key information fragments can be directly intercepted, or further summarized and concluded on the basis of intercepting the key information fragments, etc., which is not specifically limited in this article.

[0338] In the embodiment of the present application, if the target chat message itself is not a text message, when extracting key information, the target chat message can be converted into a text form before subsequent processing. If the target chat message itself is a text message, the generated simplified message is also a text message, in which case step S183 can be omitted.

[0339] S184: The server sends the condensed message and the text summary to the client, so that the client presents the condensed message and the text summary in response to the message condensation operation triggered for at least one target chat message.

[0340] Since the present application supports the object to select the chat messages to be simplified by himself, and then automatically integrates and simplifies the target chat messages selected by the object, and directly presents the generated simplified messages and corresponding text summaries to the object, in this way, the object no longer needs to view the target chat messages before simplification one by one, but can directly read the simplified messages and text summaries, which makes it convenient for the object to directly view the summaries of multiple target chat messages based on the simplified messages, understand the key information of these target chat messages, and improve the object's message reading efficiency.

[0341] It should be noted that the target chat message in the embodiment of the present application can be any message type. For example, when the message type of the target chat message is a video, the key segments in the video can be directly intercepted and integrated to generate a simplified version of the video. For another example, when the message type of the target chat message is a text, the key segments in the original text content can be further summarized and integrated to generate a simplified version of the article, and so on.

[0342] For another example, when the target chat message type is voice, an optional implementation method is to implement S182 according to the following process, including the following steps S1821 to S1825 ( Fig.18 Not shown):

[0343] S1821: Merge the text contents corresponding to at least one target chat message, and extract key information from the merged result.

[0344] S1822: Convert the extracted key information into speech unit elements.

[0345] S1823: Generate a speech waveform using speech unit elements based on a speech synthesis model.

[0346] S1824: Determine speech features corresponding to the simplified message by adjusting model parameters of the speech synthesis model.

[0347] S1825: Based on the voice features, convert the voice waveform into an audio signal to obtain a simplified message corresponding to at least one target chat message.

[0348] The following takes the condensing process of multiple voice messages as an example to illustrate the implementation process of the above S1821 to S1825:

[0349] In the embodiment of the present application, the server can use speech synthesis technology to synthesize multiple voices into a simplified version of the voice (i.e., step S182), and then convert the synthesized voice into a text summary (i.e., step S183). This process needs to be implemented using technologies such as text processing, synthesized voice stream, synthesized timbre, and synthesized voice adjustment. In this process, the server can use tools such as speech recognition engine and speech synthesis engine to implement the entire speech synthesis process.

[0350] Specifically, when the server uses speech synthesis technology to synthesize multiple voices into a simplified version of the voice, it first needs to convert the multiple voices into text, which can be achieved through speech recognition technology. The server converts each voice segment into text, then merges these texts and extracts key information from the merged results to generate the text to be synthesized. This process can be specifically implemented based on the speech recognition engine. The technical implementation principle of the speech recognition engine is as follows:

[0351] (1) Audio input: The speech to be simplified is input into the speech recognition engine, which receives the audio input and converts the audio signal into a digital signal.

[0352] (2) Feature extraction: The speech recognition engine uses signal processing technology and digital signal processing algorithms to extract the features of the digital signal obtained in the above steps, such as the frequency, time domain features, speech energy and other features of the sound, in order to better understand the speech signal.

[0353] (3) Acoustic model: The speech recognition engine uses the acoustic model to convert speech features into text.

[0354] Among them, the acoustic model can be a hidden Markov model (HMM), a deep neural network (DNN) or a recurrent neural network (RNN), etc., which is not specifically limited in this article.

[0355] In the embodiment of the present application, based on the above process, each voice can be converted into text to obtain the text content corresponding to each voice.

[0356] (4) Language model: The speech recognition engine uses a language model in combination with an acoustic model to convert the text output by the acoustic model into the final text result.

[0357] In an embodiment of the present application, after the texts output by the acoustic model are merged, a language model can be used to extract key information from the merged result.

[0358] The language model may be an n-gram model (a language model commonly used in large vocabulary continuous speech recognition), a neural network language model (NNLM), etc., which is not specifically limited in this article.

[0359] On the basis of the above, S1821 can be completed, and then the extracted key information and texts need to be converted into voice messages. Specifically, the server can use a speech synthesis engine to achieve this, and the process involves technologies such as synthesized voice stream, synthesized timbre and synthesized voice adjustment.

[0360] First, the server converts the text to be synthesized into a speech stream, which can be achieved through a speech synthesis engine. The speech synthesis engine can use a series of algorithms and models to convert text into a speech stream. During this process, the speech synthesis engine can use phonemes, syllables, words or phrases as the basic units of the speech stream. Then, based on the synthesized speech stream, the synthesized timbre and synthesized speech are adjusted, and finally the synthesized simplified version of the speech can be output.

[0361] In the embodiment of the present application, the technical implementation principle of the speech synthesis engine is as follows:

[0362] (1) Text input: The speech synthesis engine first receives the text finally output by the speech recognition engine (i.e., the key information mentioned in S1821), and converts the text into speech unit elements such as phonemes, syllables, words or phrases, corresponding to the above S1822.

[0363] (2) Speech synthesis model: The speech synthesis engine uses the speech synthesis model to generate a speech waveform through the speech unit elements obtained through the above conversion, corresponding to the above S1823.

[0364] The speech synthesis model may be rule-based, statistical, or machine-learning based, etc., which is not specifically limited in this document.

[0365] (3) Speech synthesis model parameters: The speech synthesis engine uses the speech synthesis model parameters to control the voice characteristics of the speech output, such as timbre, speech speed, volume, pitch, intonation, voice quality, and pronunciation method, corresponding to the above S1824. The model parameters can be the fundamental frequency, spectrum envelope, speech speed, etc. of speech synthesis, which are not specifically limited in this article.

[0366] Among them, this process mainly corresponds to the synthetic timbre and synthetic voice adjustment process. In the embodiment of the present application, the server can use different timbres to generate the synthesized voice. In addition, the server can also adjust the synthesized voice to optimize its sound quality and fluency, which can be achieved by adjusting parameters such as timbre, speech speed, volume, pitch, intonation and voice quality, pronunciation method, etc.

[0367] Among them, timbre refers to different speakers or sound samples. The synthesized timbre can be extracted based on the original voice or from the template voice.

[0368] (4) Speech synthesis output: The speech synthesis engine outputs the speech waveform generated by the speech synthesis model as an audio signal. The output speech is a simplified version of the speech, which can be natural speech, synthesized speech or template speech, corresponding to the above S1825.

[0369] Among them, natural speech means that the speaker of the simplified speech is consistent with the speaker in the original target chat message, both of which are natural speakers, and the speech features of the simplified speech are the speech features of any one of the multiple speaker objects corresponding to the original target chat message. Synthesized speech means that the speech features of the simplified speech are obtained by fusing the speech features of different speaker objects in the original target chat message. Template speech means that the speech features of the simplified speech are the speech features of the preset template speech.

[0370] Based on the above implementation, a simplified version of speech synthesized from multiple original speech can be obtained. On the basis of the above, the server can use speech recognition technology to convert the synthesized speech into a text summary, corresponding to the above S183. This can be achieved by converting the speech signal into a digital signal and then converting it into text using a speech recognition engine.

[0371] Specifically, speech-to-text is a natural language processing technology, and its technical implementation principle mainly includes the following steps:

[0372] (1) Recording and digital signal processing: convert the voice input into digital signals and perform preprocessing.

[0373] Specifically, in this step, the previously generated simplified version of the speech is converted into a digital signal and preprocessed, such as noise reduction, filtering, and silence removal, etc. The steps in this article are specifically defined.

[0374] (2) Feature extraction: The features of the digital signal obtained in the above steps are extracted through signal processing technology and digital signal processing algorithms, such as Mel-Frequency Cepstral Coefficients (MFCC) and linear predictive coding (LPC), etc., which are not specifically limited in this article.

[0375] (3) Acoustic model: Use the acoustic model to convert speech features into text.

[0376] As above, the acoustic model usually adopts a hidden Markov model (HMM), a deep neural network (DNN) or a recurrent neural network (RNN), etc., which is not specifically limited in this article.

[0377] (4) Language model: Use the language model in combination with the acoustic model to convert the text output by the acoustic model into the final text result.

[0378] The language model may be an n-gram model, a neural network language model (NNLM), or the like.

[0379] In the embodiment of the present application, the text result obtained based on this step can be directly used as a text summary of the simplified version of the speech, and the post-processing described in the following step (5) can also be performed on this basis to improve the recognition accuracy and the correctness of speech-to-text conversion.

[0380] (5) Post-processing: The text results obtained in the above steps are post-processed, such as grammatical analysis, semantic analysis, error correction, etc., to improve the recognition accuracy and the correctness of speech-to-text conversion.

[0381] Compared with related technologies, related technologies only support multi-speech to text conversion and generate text summaries, but do not support the generation of a simplified version of the original speech. The present application can not only generate a simplified text summary, but also generate a simplified original speech of the object, or generate a simplified speech that conforms to a preset template.

[0382] On this basis, this application also supports the rapid positioning of text corresponding to different speech stages.

[0383] An optional implementation is as follows:

[0384] The server receives the original message query request sent by the client; then, the server obtains the target chat message associated with the information to be queried in the original message query request; finally, the server returns the target chat message to the client, so that the client presents the target chat message.

[0385] In the embodiment of the present application, the original message query request is triggered by the client in response to a message backtracking operation or a message recovery operation. If the original message query request is triggered by the client in response to a message backtracking operation, the target chat message that the server needs to return is matched with the message backtracking operation, which is part or all of the original target chat message to be streamlined. For example Fig.12 , Fig.13 , Fig.15 or Fig.17 As shown, the example is to return a part of the original target chat message to be simplified, and the part is the target chat message associated with the information to be queried corresponding to the message backtracking operation.

[0386] If the original message query request is triggered by the client in response to a message recovery operation, the chat messages that the server needs to return are all the target chat messages to be streamlined selected by the object this time. Fig. 9 As shown, the server needs to return the 4 long voices selected by the object this time, which are: 30 seconds, 59 seconds, 59 seconds and 29 seconds.

[0387] It should be noted that, in the embodiment of the present application, there are many ways to trigger the object message backtracking operation, and different triggering methods correspond to different information to be queried.

[0388] In an optional implementation manner, if the original message query request is triggered by the client in response to a message backtracking operation, the information to be queried in the original message query request includes at least one of the following:

[0389] The summary fragment in the text summary, the topic name of any one of the multiple topics contained in the simplified message, and the key information to be searched entered in the message interface.

[0390] Specifically, if the message backtracking operation is a message backtracking operation triggered by the client in response to the object based on the summary fragment, the information to be queried is the summary fragment in the text summary, such as Fig.11 The server can query the associated target chat message based on the summary fragment for the listed "last day". If the message backtracking operation is a message backtracking operation triggered by the client in response to the object topic name, the information to be queried is the topic name, such as Fig.15The server can query the associated target chat message based on the topic name for the listed "Topic 1"; if the message tracing back operation is triggered by the client in response to the object inputting the key information to be searched in the message interface, the information to be queried is the key information input by the object, such as Fig.17 For the listed “Student Li”, the server can query the associated target chat message based on the key information.

[0391] In the embodiment of the present application, when the server obtains the target chat message associated with the information to be queried in the original message query request, an optional implementation method is as follows:

[0392] First, the server obtains the text content corresponding to at least one target chat message. Then, the server uses natural language processing technology to perform text matching between each obtained text content and the information to be queried. Finally, the server uses the target chat message corresponding to the successfully matched text content as the target chat message associated with the original message query request.

[0393] A successful match includes: the text content contains a segment whose semantic relevance to the information to be queried reaches a preset relevance threshold, and the segment can be recorded as a related segment.

[0394] In the embodiment of the present application, if the target chat message itself is a text message, the server can directly obtain the text content corresponding to the target chat message. If the target chat message is voice or video (including audio), the server also needs to obtain the text content of the target chat message based on speech-to-text technology. The speech-to-text process can be specifically referred to in the above embodiment, and will not be repeated here.

[0395] On the basis of the above, the server can perform word segmentation and other processing on the text content of the target chat message, and then perform text matching on the text content with the information to be queried (such as summary fragments, topic names, key information, etc.). Finally, the successfully matched target chat message is returned to the client for display.

[0396] In the above implementation, the object can quickly trace back to the original target chat message in a variety of ways, which can not only improve the object's message reading efficiency, but also facilitate the object to trace back key messages in time.

[0397] Still taking the multi-speech synthesis (simplified) scenario as an example, the message processing method in the embodiment of the present application is briefly described from the perspective of interaction between the client and the server.

[0398] See also Fig.19 As shown, it is a logic diagram of interaction between a client and a server in an embodiment of the present application, which specifically includes the following steps S191 to S199:

[0399] S191: The subject receives multiple long voice messages.

[0400] S192: The client detects two or more long voice messages.

[0401] S193: The object triggers the multi-speech synthesis function through a sliding gesture.

[0402] Specifically, the subject can log in to an instant messaging client and receive multiple long voice messages through the client. When the subject receives multiple long voice messages, the client can detect whether there are two or more long voice messages through a message quantity counter, or detect whether there are two or more long voice messages by detecting the arrival of new messages, and then determine whether multi-voice synthesis is required.

[0403] In an embodiment of the present application, the process in which the client determines whether there are two or more long voices through any of the above methods is mainly used to present operation prompt information to the object, such as being used in the first object novice prompt, to facilitate the object's quick perception, and also to facilitate the client to pass parameters to the server so that the server can perform multi-voice synthesis.

[0404] Of course, the present application is also applicable to condensing a single voice message. For example, if a single voice message exceeds 30 seconds, the message content of the single voice message can also be condensed, which will not be repeated here.

[0405] S194: The server synthesizes the multiple voice messages into a simplified voice message.

[0406] S195: The server converts the synthesized speech into a text summary.

[0407] In the embodiment of the present application, the server can implement the process described in S194 to S195 using tools such as a speech recognition engine and a speech synthesis engine. For details, please refer to the above embodiment, and the repeated parts will not be repeated.

[0408] S196: The client extracts highlighted keywords according to the audio playback position.

[0409] In an embodiment of the present application, the highlighted keywords shown in S196, that is, the summary fragment in the text summary described in this article, correspond to the current playback progress of the simplified version of the voice, and the object can quickly trace back to the original voice based on the summary fragment in the text summary.

[0410] Specifically, the technical implementation principle of the client extracting the highlighted keywords according to the position of the audio playback mainly includes the following steps S1961~S1965 ( Fig.19 Not shown):

[0411] S1961: Text segmentation: The client performs word segmentation on the text summary of the simplified voice, divides the text summary into individual words or phrases, and annotates its part of speech, named entity and other information.

[0412] S1962: Keyword extraction: The client extracts keywords from the above text summary according to predefined rules or algorithms, and may use algorithms such as word frequency statistics, term frequency-inverse document frequency (TF-IDF), and text ranking (TextRank, used to generate keywords and summaries for text).

[0413] S1963: Audio to text: The client uses speech recognition technology to convert the audio of the simplified speech into text to obtain the text information contained in the audio.

[0414] S1964: Alignment: The client aligns the text information in the audio with the original text summary (i.e., the text summary that needs to be segmented in S1961) to determine the time point when each word or phrase in the audio appears and the corresponding text information.

[0415] S1965: Highlighting: The client determines the keywords that need to be highlighted based on the audio playback position information and alignment results, and highlights them in the client.

[0416] It should be noted that the above-mentioned technology of extracting highlighted keywords according to the position of audio playback by the client is just a simple example, and any related processing method is applicable to the embodiments of the present application, which will not be described one by one here.

[0417] S197: The subject can quickly trace back to the original speech based on the highlighted keywords.

[0418] The specific implementation of S197 can refer to the above embodiment, such as object click Fig.11 The highlighted keyword "last day" will not be repeated here.

[0419] S198: The server returns the voice module content to the client.

[0420] S199: Subject can review key messages and view summaries of multiple voice messages via condensed voice or text.

[0421] In an embodiment of the present application, when the client detects that the object clicks on a highlighted keyword in the text summary of the simplified voice, the server will return the voice module content corresponding to the highlighted keyword to the client. When the object clicks on the highlighted restored text link, the server will return the original voice content to the object.

[0422] It should be noted that the related technology only supports sliding the progress bar to play the voice of the corresponding time, while the present application can not only adjust the progress bar by sliding gestures, but also feedback the time progress corresponding to the current playback on the progress bar in real time, and highlight the text currently playing, and can quickly trace back to the original voice according to the highlighted keywords. The object can be paused and resumed at any time during playback, and the current playback time progress is clearly visible. In addition, it also supports the generation of personalized voice customization, and automatically adjusts the audio sound effects according to the sound characteristics and preferences of the object to achieve the best auditory effect.

[0423] The following is an explanation of the specific implementation process related to S197 to S199:

[0424] See also Fig. 20 FIG. 2 is another logic diagram of interaction between a client and a server in an embodiment of the present application, which specifically includes the following steps S201 to S206:

[0425] S201: The client detects that an object clicks a highlighted keyword.

[0426] In the embodiment of the present application, the object in S197 can click on the highlighted keyword. If the client detects that the object clicks on the highlighted keyword, the highlighted keyword can be sent to the server as information to be queried.

[0427] S202: The server extracts keywords based on speech recognition technology.

[0428] S203: The server determines the original speech that matches the highlighted keyword.

[0429] In the embodiment of the present application, the process of determining the original speech matching the highlighted keywords can be implemented through speech recognition and text matching. Specifically, the server can convert multiple long speech into text, and then generate a simplified version of the speech and text summary according to the gesture sliding or convenient entry of the object, and the highlighted keywords in the text summary correspond one-to-one with the keywords in the text content corresponding to the original speech.

[0430] Specifically, the highlighted keywords in the text summary can be the content of the summary segment that is already in the original target chat message, or they can be new words / short sentences that appear after simplification. If the highlighted keywords are new words / short sentences that appear after simplification, the server will analyze the text content of the target chat message according to the implementation principle of the semantic analysis technology, and record the sentences with the highest semantic relevance to the summary segment (or sentences that exceed the preset relevance threshold) as the related segments. The implementation principle of the semantic analysis technology includes the following steps:

[0431] (1) Word segmentation: The server divides the sentence into individual words, usually using a Chinese word segmenter or an English word segmenter (tokenizer).

[0432] (2) Part-of-speech tagging: Each word is tagged with its part of speech, such as noun, verb, adjective, etc. This is usually achieved using a part-of-speech tagger.

[0433] (3) Syntactic analysis: Analyze the grammatical structure of a sentence, such as the subject-verb-object structure, usually implemented using a syntactic analyzer.

[0434] (4) Semantic role labeling: Analyze the semantic role of each word in a sentence, such as subject, object, predicate, etc., usually using a semantic role labeler.

[0435] (5) Dependency analysis: Analyze the dependency relationships between words in a sentence, such as subject-predicate relationship, verb-object relationship, etc., which is usually implemented using a dependency analyzer.

[0436] (6) Semantic analysis: Based on the above information, the sentence is semantically analyzed, that is, the meaning of the sentence is understood and the information is extracted. This is usually achieved using natural language processing technology, such as text classification, sentiment analysis, and entity recognition.

[0437] Specifically, the server can perform the above-mentioned semantic analysis on the text content corresponding to the original voice and the sentences in the text summary corresponding to the simplified version of the voice based on the semantic analysis technology listed above, so as to analyze the text content of the target chat message, and record the sentences with the highest semantic correlation with the summary fragment (which may also be sentences exceeding a preset correlation threshold) as associated fragments, and feed back the target chat message containing the associated fragments to the client for presentation to the object.

[0438] In addition, if the original voice in this application contains multiple topics, the client can also display them in groups based on the topic name. The specific implementation logic is as follows:

[0439] S204: The server determines whether the original speech contains multiple topics based on natural language processing technology.

[0440] S205: When there are multiple topics, the client display logic can display them in groups based on the topic names.

[0441] Specifically, the client can display the text summaries of the simplified version of the speech in groups based on the subject name (such as topic, person name, project name, etc.).

[0442] S206: The subject can trace back the message and view the text content of multiple original voice messages.

[0443] The following is the technical implementation principle of quickly tracing back to the original voice based on the highlighted keywords:

[0444] (1) Speech recognition technology: Speech needs to be converted into text for key message extraction and summary generation. This can be achieved through speech recognition technology, which converts speech into text.

[0445] (2) Key message extraction: For speech converted into text, key information needs to be extracted so that the subject can trace back the key message. This can be achieved through keyword extraction technology, such as extracting keywords from the text based on algorithms such as word frequency, TF-IDF, and TextRank.

[0446] (3) Summary generation: For multiple voices, summaries need to be generated so that the subjects can view the summaries of multiple voices through simplified voice or text. This can be achieved through text summarization technology, such as generating text summaries based on information such as text length, sentence position, and keywords.

[0447] (4) Client presentation: Finally, the client needs to present the extracted key information and generated summary to the subject so that the subject can browse and view it.

[0448] In summary, this application provides a multi-speech synthesis method, and the subject can quickly synthesize multiple voices into a simplified voice and text summary through gesture sliding or a convenient entrance. When the subject is listening to the simplified voice, he can quickly trace back to the original voice according to the converted highlighted keywords, which can not only improve the reading efficiency of the subject, but also facilitate the subject to view the summary of multiple voices by listening to the voice, and also facilitate the subject to trace back to key messages in time.

[0449] In addition, it should be noted that the above is mainly illustrated by taking one or more voice condensation as an example. In addition to the above voice condensation, it can also be applied to video condensation, article condensation, etc. Fig.21A As shown, it is a schematic diagram of a video simplification process in an embodiment of the present application. In the same scenario, the object shares multiple video clips, and generates a condensed video clip and summary by aggregating the videos. Specifically, in the above scenario, operation prompt information (such as S211) and other contents may also be presented. The object may select multiple videos by any of the methods listed above, and then present Fig.21A The interface shown on the right side presents a condensed video (such as S212) and a corresponding text summary (such as S213, which can be obtained by integrating key information of the video text or the audio in the video), and the object can perform message backtracking and restoration on this basis.

[0450] like Fig. 21B As shown in the figure, a schematic diagram of a video backtracking process in an embodiment of the present application. If the object is based on a summary clip or other methods, the message backtracking can be presented as follows Fig. 21BThe original video message obtained by backtracking is highlighted with a gray background, and the position corresponding to the associated segment can be located in the progress bar of the original video, and the associated segment "last day" can be highlighted in the text content of the original video, etc. The text content corresponding to the original video can specifically be the video copy corresponding to the original video when it was released, or the text information obtained by recognizing or recognizing and summarizing the audio in the video, etc.

[0451] like Fig. 21C As shown, a schematic diagram of a video restoration process in an embodiment of the present application is shown. For example, the object can click on the restoration text link to trigger the message restoration operation and present the video content before condensation.

[0452] Optionally, the above example takes the target chat message as a video message. In addition, the target chat message can also be a combination of a video message and other messages. For example, the target chat message is a voice message + a video message. Fig.21D As shown, it is a schematic diagram of the simplified process of voice and video messages in an embodiment of the present application.

[0453] like Fig.21D As shown in , the subject selects a voice message and two video messages as target chat messages in the current message interface. In this case, the video message appears more frequently, and a simplified message of the video type can be generated, such as Fig.21D As shown in the right interface, a streamlined video is synthesized.

[0454] Specifically, after the object selects at least one target chat message through the client and triggers the message simplification operation, the client can send a message simplification request to the server. After receiving the request, the server can simplify the message based on these target chat messages and generate a simplification message. When the server generates a simplification message, for voice messages, voice recognition can be performed, converted into text content, and then key information is extracted. For video messages, key video clips can be directly extracted as key information. After the extracted key information is integrated, a simplification video can be obtained.

[0455] Specifically, the original message corresponding to some video content in the streamlined video may be a voice message. In this case, the video screen of this part of the video can be a default video screen, or a video screen intelligently generated based on the voice message corresponding to this part, etc. This article does not make specific limitations.

[0456] Similarly, the target chat message can also be one or more other message types, such as video message + text message, video message + text message + voice message, etc. The specific streamlining process is similar and will not be repeated here. On the basis of obtaining the above-mentioned streamlining message, the object can also perform message backtracking, message recovery, etc. The specific implementation method is the same as the above embodiment, and the repeated parts will not be repeated; etc., will not be repeated here.

[0457] Based on the same inventive concept, the embodiment of the present application also provides a message processing device. Fig. 22 As shown, it is a schematic diagram of the structure of the message processing device 2200, which may include:

[0458] The presenting unit 2201 is used to present a message interface, where the message interface includes at least one chat message to be simplified;

[0459] A selection unit 2202, configured to obtain at least one selected target chat message in response to a selection operation triggered on at least one chat message;

[0460] The display unit 2203 is used to present the generated simplified message and the text summary corresponding to the simplified message in response to the message simplification operation triggered for at least one target chat message; wherein the simplified message is obtained by integrating the key information in at least one target chat message.

[0461] Optionally, the message length of at least one chat message reaches a length threshold;

[0462] The presenting unit 2201 is specifically used for:

[0463] In the message interface, at least one chat message is presented, as well as operation prompt information; the operation prompt information is used to prompt the object how to perform message simplification processing.

[0464] Optionally, the display unit 2203 is further configured to perform at least one of the following operations:

[0465] After responding to a message simplification operation triggered for at least one target chat message, canceling display of the operation prompt information;

[0466] After the display time of the operation prompt information reaches a preset time threshold, the display of the operation prompt information is canceled.

[0467] Optionally, the selection unit 2202 is configured to perform at least one of the following operations:

[0468] In response to a selection operation triggered by a pinch gesture of the object, the chat messages to be simplified in the pinch area determined based on the pinch gesture are used as at least one target chat message, and the at least one target chat message is highlighted;

[0469] In response to a selection operation triggered by long pressing a relevant position of a single chat message and dragging the object, the chat messages to be streamlined in the dragged area are treated as at least one target chat message, and the at least one target chat message is highlighted.

[0470] Optionally, the display unit 2203 is also used to present processing prompt information in the message interface after responding to a message simplification operation triggered for at least one target chat message and before presenting the generated simplification message and the text summary corresponding to the simplification message, and the processing prompt information is used to prompt the object that the message simplification processing is being performed.

[0471] Optionally, the message type of the target chat message is at least one of voice and video; then the display unit 2203 is further configured to present the current playback progress of the simplified message in response to a playback progress adjustment operation triggered for the simplified message after presenting the generated simplified message and the text summary corresponding to the simplified message;

[0472] The summary segment corresponding to the current playback progress is highlighted in the text summary.

[0473] Optionally, the display unit 2203 is also used for:

[0474] Backtracking prompt information is presented in the message interface, and the backtracking prompt information is used to prompt the object how to backtrack the original target chat message based on the summary fragment.

[0475] Optionally, the display unit 2203 is also used for:

[0476] In response to a message backtracking operation triggered based on the summary segment, presenting a target chat message associated with the summary segment and text content corresponding to the target chat message;

[0477] The associated segment of the summary segment is highlighted in the text content corresponding to the target chat message, and the current playback progress of the target chat message corresponds to the associated segment, wherein the associated segment is a segment whose semantic association with the summary segment reaches a preset association threshold.

[0478] Optionally, the simplified message includes multiple topics; then the display unit 2203 is specifically used to:

[0479] The text summaries corresponding to multiple topics are presented in groups according to the topics, and the topic names corresponding to each group of text summaries are marked.

[0480] Optionally, the display unit 2203 is also used for:

[0481] In response to a message backtracking operation triggered based on a topic name, a target chat message associated with the topic name and text content corresponding to the target chat message are presented.

[0482] Optionally, the display unit 2203 is also used for:

[0483] In response to a message backtracking operation triggered by inputting key information to be searched in a message interface, a target chat message associated with the key information and text content corresponding to the target chat message are presented.

[0484] Optionally, there are multiple associated target chat messages determined based on the message backtracking operation; then the display unit 2203 is specifically used to:

[0485] According to the release time of the message, the related target chat messages are presented in sequence, and the release information of the target chat message is marked before each target chat message.

[0486] Optionally, the display unit 2203 is specifically used for:

[0487] In the message interface, the associated target chat message is presented and highlighted; or

[0488] A message sub-interface is presented in the message interface, and the associated target chat message is presented through the message sub-interface.

[0489] Optionally, when the target message type is voice and at least one target chat message corresponds to multiple sound-generating objects, the voice feature corresponding to the simplified message is any one of the following:

[0490] The speech feature of any one of the multiple sound-making objects;

[0491] A speech feature obtained by integrating the speech features of multiple sound-generating objects;

[0492] Preset template voice features.

[0493] Optionally, the message type of the target chat message is at least one of voice and video; the device further includes:

[0494] The play control unit 2204 is used to present a play pause control at a relevant position of the simplified message when playing the simplified message;

[0495] In response to a pause operation triggered by a play pause control, pausing the playback of the simplified message;

[0496] In response to a play operation triggered based on a play pause control, the condensed message continues to be played.

[0497] Optionally, the display style of the condensed message is different from the display style of at least one target chat message.

[0498] Optionally, the device further comprises:

[0499] The restoring unit 2205 is used to present restoration prompt information in the message interface, and the restoration prompt information is used to prompt the object how to restore the simplified message;

[0500] In response to a message restoration operation triggered based on the restoration prompt information, the simplified message is restored to at least one target chat message.

[0501] Based on the same inventive concept, the embodiment of the present application also provides a message processing device. Fig.23 As shown, it is a schematic diagram of the structure of the message processing device 2300, which may include:

[0502] The message acquisition unit 2301 is used to acquire at least one target chat message, where the at least one target chat message is selected by the client in response to a selection operation triggered on the at least one chat message, and the at least one chat message is a chat message to be condensed contained in a message interface presented by the client;

[0503] The processing unit 2302 is used to extract key information from at least one target chat message and integrate it to obtain a simplified message; convert the simplified message into a text form to obtain a text summary corresponding to the simplified message;

[0504] The feedback unit 2303 is used to send the simplified message and the text summary to the client, so that the client presents the simplified message and the text summary in response to the message simplification operation triggered for at least one target chat message.

[0505] Optionally, the message type of the target chat message is voice; then the processing unit 2302 is specifically configured to:

[0506] Merging text contents corresponding to at least one target chat message, and extracting key information from the merged result;

[0507] Convert the extracted key information into speech unit elements;

[0508] Generate speech waveforms through speech unit elements based on the speech synthesis model;

[0509] By adjusting the model parameters of the speech synthesis model, the speech features corresponding to the simplified message are determined;

[0510] Based on the voice features, the voice waveform is converted into an audio signal to obtain a simplified message corresponding to at least one target chat message.

[0511] Optionally, the device further comprises:

[0512] The query unit 2304 is used to receive an original message query request sent by the client; the original message query request is triggered by the client in response to a message backtracking operation or a message recovery operation;

[0513] Obtaining a target chat message associated with the information to be queried in the original message query request;

[0514] Returning the target chat message to the client, so that the client presents the target chat message;

[0515] Among them, if the original message query request is triggered by the client in response to a message backtracking operation, the information to be queried in the original message query request includes at least one of the following: a summary fragment in the text summary, a topic name of any one of the multiple topics contained in the simplified message, and the key information to be searched entered in the message interface.

[0516] Optionally, the query unit 2304 is specifically configured to:

[0517] Obtaining text content corresponding to at least one target chat message;

[0518] Through natural language processing technology, each text content obtained is matched with the information to be queried;

[0519] The target chat message corresponding to the successfully matched text content is used as the target chat message associated with the original message query request; wherein, a successful match includes: the text content contains a fragment whose semantic relevance to the information to be queried reaches a preset relevance threshold.

[0520] Since the present application supports the object to select the chat messages to be simplified by himself, and then automatically integrates and simplifies the target chat messages selected by the object, and directly presents the generated simplified messages and corresponding text summaries to the object, in this way, the object no longer needs to view the target chat messages before simplification one by one, but can directly read the simplified messages and text summaries, which makes it convenient for the object to directly view the summaries of multiple target chat messages based on the simplified messages, understand the key information of these target chat messages, and improve the object's message reading efficiency.

[0521] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0522] After introducing the message processing method and device according to the exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0523] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0524] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device may be a server, such as Figure 1 In this embodiment, the structure of the electronic device can be as follows: Fig.24 As shown, it includes a memory 2401, a communication module 2403 and one or more processors 2402.

[0525] The memory 2401 is used to store computer programs executed by the processor 2402. The memory 2401 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0526] The memory 2401 may be a volatile memory, such as a random-access memory (RAM); the memory 2401 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 2401 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2401 may be a combination of the above memories.

[0527] The processor 2402 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 2402 is configured to implement the above message processing method when calling the computer program stored in the memory 2401 .

[0528] The communication module 2403 is used to communicate with terminal devices and other servers.

[0529] The specific connection medium between the memory 2401, the communication module 2403 and the processor 2402 is not limited in the embodiment of the present application. Fig.24 In the embodiment, the memory 2401 and the processor 2402 are connected via a bus 2404. The bus 2404 is connected to the processor 2402 via a bus 2404. Fig.24 The connections between the other components are only for illustration and are not intended to be limiting. The bus 2404 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.24 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or one type of bus.

[0530] The memory 2401 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the message processing method of the embodiment of the present application. The processor 2402 is used to execute the above-mentioned message processing method, such as Fig.18 shown.

[0531] In another embodiment, the electronic device may also be other electronic devices, such as Figure 1 The terminal device 110 shown in FIG. 1 is a terminal device 110 shown in FIG. 1 . In this embodiment, the structure of the electronic device can be as follows: Fig.25 As shown, it includes: a communication component 2510, a memory 2520, a display unit 2530, a camera 2540, a sensor 2550, an audio circuit 2560, a Bluetooth module 2570, a processor 2580 and other components.

[0532] The communication component 2510 is used to communicate with the server. In some embodiments, a circuit wireless fidelity (Wireless Fidelity, WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The electronic device can help the user to send and receive information through the WiFi module.

[0533] The memory 2520 can be used to store software programs and data. The processor 2580 executes various functions and data processing of the terminal device 110 by running the software programs or data stored in the memory 2520. The memory 2520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The memory 2520 stores an operating system that enables the terminal device 110 to run. In the present application, the memory 2520 can store an operating system and various application programs, and can also store a computer program for executing the message processing method of the embodiment of the present application.

[0534] The display unit 2530 may also be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the terminal device 110. Specifically, the display unit 2530 may include a display screen 2532 disposed on the front of the terminal device 110. The display screen 2532 may be configured in the form of a liquid crystal display, a light emitting diode, etc. The display unit 2530 may be used to display a message interface, a message sub-interface, etc. in the embodiment of the present application.

[0535] The display unit 2530 can also be used to receive input digital or character information, and generate signal input related to user settings and function control of the terminal device 110. Specifically, the display unit 2530 may include a touch screen 2531 arranged on the front of the terminal device 110, which can collect user touch operations on or near it, such as clicking a button, dragging a scroll box, etc.

[0536] The touch screen 2531 can be covered on the display screen 2532, or the touch screen 2531 and the display screen 2532 can be integrated to realize the input and output functions of the terminal device 110. The integrated touch screen can be referred to as a touch display screen. In this application, the display unit 2530 can display the application and the corresponding operation steps.

[0537] Camera 2540 can be used to capture static images, and users can publish images taken by camera 2540 through the application. Camera 2540 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the processor 2580 to convert it into a digital image signal.

[0538] The terminal device may further include at least one sensor 2550, such as an acceleration sensor 2551, a distance sensor 2552, a fingerprint sensor 2553, and a temperature sensor 2554. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0539] The audio circuit 2560, the speaker 2561, and the microphone 2562 can provide an audio interface between the user and the terminal device 110. The audio circuit 2560 can transmit the electrical signal converted from the received audio data to the speaker 2561, which is converted into a sound signal for output. The terminal device 110 can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 2562 converts the collected sound signal into an electrical signal, which is received by the audio circuit 2560 and converted into audio data, and then the audio data is output to the communication component 2510 to be sent to, for example, another terminal device 110, or the audio data is output to the memory 2520 for further processing.

[0540] The Bluetooth module 2570 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 2570 to exchange data.

[0541] The processor 2580 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 2520 and calling data stored in the memory 2520. In some embodiments, the processor 2580 may include one or more processing units; the processor 2580 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 2580. In the present application, the processor 2580 can run the operating system, application programs, user interface display and touch response, as well as the message processing method of the embodiment of the present application. In addition, the processor 2580 is coupled to the display unit 2530.

[0542] In some possible implementations, various aspects of the message processing method provided by the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the message processing method according to various exemplary embodiments of the present application described above in this specification. For example, the electronic device may execute the following steps: Figure 2 or Fig.18 Follow the steps shown in .

[0543] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0544] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0545] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0546] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0547] The computer program for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed entirely on the user electronic device, partially on the user electronic device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0548] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0549] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0550] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0551] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0552] These computer program commands may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer readable memory produce an article of manufacture comprising a command device, the command device implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0553] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0554] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A message processing method, characterized in that: The method comprises: Presenting a message interface, wherein the message interface includes at least one chat message to be condensed; In response to a selection operation triggered on the at least one chat message, obtaining at least one selected target chat message; In response to a message simplification operation triggered for the at least one target chat message, a generated simplification message and a text summary corresponding to the simplification message are presented; wherein the simplification message is obtained by integrating key information in the at least one target chat message.

2. The method according to claim 1, characterized in that The message length of the at least one chat message reaches a preset length threshold; The message presentation interface includes: In the message interface, the at least one chat message is presented, as well as operation prompt information; the operation prompt information is used to prompt the object how to perform message simplification processing.

3. The method according to claim 2, characterized in that The method further comprises at least one of the following operations: After responding to the message simplification operation triggered for the at least one target chat message, canceling the display of the operation prompt information; After the display time of the operation prompt information reaches a preset time threshold, the display of the operation prompt information is canceled.

4. The method according to claim 1, characterized in that The obtaining, in response to the selection operation triggered on the at least one chat message, of the at least one selected target chat message comprises at least one of the following operations: In response to a selection operation triggered by a pinch gesture of the object, taking a chat message to be simplified in a pinch area determined based on the pinch gesture as the at least one target chat message, and highlighting the at least one target chat message; In response to a selection operation triggered by long pressing a relevant position of a single chat message and dragging the object, the chat messages to be streamlined in the dragged area are taken as the at least one target chat message, and the at least one target chat message is highlighted.

5. The method according to claim 1, characterized in that After responding to the message simplification operation triggered for the at least one target chat message, and before presenting the generated simplification message and the text summary corresponding to the simplification message, the method further includes: Processing prompt information is presented in the message interface, and the processing prompt information is used to prompt the object that the message simplification process is being performed.

6. The method according to any one of claims 1 to 5, characterized in that: The message type of the target chat message is at least one of voice and video; then after presenting the generated simplified message and the text summary corresponding to the simplified message, the method further includes: In response to a playback progress adjustment operation triggered for the simplified message, presenting a current playback progress of the simplified message; The summary segment corresponding to the current playing progress is highlighted in the text summary.

7. The method according to claim 6, characterized in that The method further comprises: Backtracking prompt information is presented in the message interface, and the backtracking prompt information is used to prompt the object how to backtrack the original target chat message based on the summary fragment.

8. The method according to claim 6, characterized in that The method further comprises: In response to a message backtracking operation triggered based on the summary segment, presenting a target chat message associated with the summary segment and text content corresponding to the target chat message; The associated segment of the summary segment is highlighted in the text content corresponding to the target chat message, and the current playback progress of the target chat message corresponds to the associated segment, wherein the associated segment is a segment whose semantic association with the summary segment reaches a preset association threshold.

9. The method according to any one of claims 1 to 5, characterized in that: The simplified message contains multiple topics; then presenting a text summary corresponding to the simplified message, including: The text abstracts corresponding to the plurality of topics are presented in groups according to the topics, and the topic names corresponding to each group of text abstracts are marked.

10. The method according to claim 9, characterized in that The method further comprises: In response to a message backtracking operation triggered based on the topic name, a target chat message associated with the topic name and text content corresponding to the target chat message are presented.

11. The method according to claim 1, characterized in that The method further comprises: In response to a message backtracking operation triggered by inputting key information to be searched in the message interface, a target chat message associated with the key information and text content corresponding to the target chat message are presented.

12. The method according to claim 8, 10 or 11, characterized in that: There are multiple associated target chat messages determined based on the message backtracking operation; then presenting the associated target chat messages includes: According to the release time of the message, the related target chat messages are presented in sequence, and the release information of the target chat message is marked before each target chat message.

13. The method according to claim 8, 10 or 11, characterized in that: Presents the associated target chat message, including: In the message interface, the associated target chat message is presented and highlighted; or A message sub-interface is presented in the message interface, and an associated target chat message is presented through the message sub-interface.

14. The method according to any one of claims 1 to 5, characterized in that: When the target message type is voice, and the at least one target chat message corresponds to multiple sound-generating objects, the voice feature corresponding to the simplified message is any one of the following: The speech feature of any one of the multiple sound-making objects; A speech feature obtained by integrating the speech features of the plurality of sound-generating objects; Preset template voice features.

15. The method according to any one of claims 1 to 5, characterized in that: The message type of the target chat message is at least one of voice and video; the method further includes: When playing the simplified message, presenting a play and pause control at a relevant position of the simplified message; In response to a pause operation triggered by the play pause control, pausing the playing of the simplified message; In response to a play operation triggered by the play pause control, continue playing the condensed message.

16. The method according to any one of claims 1 to 5, characterized in that: The display style of the simplified message is different from the display style of the at least one target chat message.

17. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Presenting restoration prompt information in the message interface, wherein the restoration prompt information is used to prompt the object how to restore the simplified message; In response to the message restoration operation triggered based on the restoration prompt information, the simplified message is restored to the at least one target chat message.

18. A message processing method, characterized in that: The method comprises: Acquire at least one target chat message, where the at least one target chat message is selected by the client in response to a selection operation triggered on the at least one chat message, and the at least one chat message is a chat message to be condensed contained in a message interface presented by the client; Extracting key information from the at least one target chat message and integrating the key information to obtain a simplified message; Convert the simplified message into text form to obtain a text summary corresponding to the simplified message; The condensed message and the text summary are sent to the client, so that the client presents the condensed message and the text summary in response to a message condensation operation triggered for the at least one target chat message.

19. The method according to claim 18, characterized in that The message type of the target chat message is voice; then extracting key information from the at least one target chat message and integrating it to obtain a simplified message includes: Merging the text contents corresponding to the at least one target chat message, and extracting key information from the merged result; Convert the extracted key information into speech unit elements; Based on the speech synthesis model, generating a speech waveform through the speech unit elements; Determining the speech features corresponding to the simplified message by adjusting the model parameters of the speech synthesis model; Based on the voice feature, the voice waveform is converted into an audio signal to obtain a simplified message corresponding to the at least one target chat message.

20. The method according to claim 18 or 19, characterized in that The method further comprises: Receiving an original message query request sent by the client; the original message query request is triggered by the client in response to a message backtracking operation or a message recovery operation; Obtaining a target chat message associated with the information to be queried in the original message query request; Returning the target chat message to the client, so that the client presents the target chat message; Among them, if the original message query request is triggered by the client in response to a message backtracking operation, the information to be queried in the original message query request includes at least one of the following: a summary fragment in the text summary, a topic name of any one of the multiple topics contained in the streamlined message, and the key information to be searched entered in the message interface.

21. The method of claim 20, wherein: The step of obtaining a target chat message associated with the information to be queried in the original message query request includes: Obtaining text content corresponding to each of the at least one target chat messages; By using natural language processing technology, text matching is performed on each of the obtained text contents and the information to be queried; The target chat message corresponding to the successfully matched text content is used as the target chat message associated with the original message query request; wherein, successful matching includes: the text content contains a fragment whose semantic relevance to the information to be queried reaches a preset relevance threshold.

22. A message processing device, characterized in that: include: A presenting unit, configured to present a message interface, wherein the message interface includes at least one chat message to be condensed; A selection unit, configured to obtain at least one selected target chat message in response to a selection operation triggered on the at least one chat message; A display unit is used to present a generated condensed message and a text summary corresponding to the condensed message in response to a message condensation operation triggered for the at least one target chat message; wherein the condensed message is obtained by integrating key information in the at least one target chat message.

23. A message processing device, characterized in that: include: a message acquisition unit, configured to acquire at least one target chat message, wherein the at least one target chat message is selected by the client in response to a selection operation triggered on the at least one chat message, and the at least one chat message is a chat message to be condensed contained in a message interface presented by the client; A processing unit, configured to extract key information from the at least one target chat message and integrate the key information to obtain a simplified message; convert the simplified message into a text form to obtain a text summary corresponding to the simplified message; A feedback unit is used to send the simplified message and the text summary to the client, so that the client presents the simplified message and the text summary in response to the message simplification operation triggered for the at least one target chat message.

24. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any method described in claims 1 to 21.

25. A computer-readable storage medium, characterized in that: It includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any method described in claims 1 to 21.

Images