A text processing method and related apparatus

Abstract

The application relates to the technical field of natural language understanding, and particularly discloses a text processing method and related devices. The method comprises the following steps: obtaining a target comment of a target video; the target comment is a comment published by a user on the target video; a word to be corrected in the target comment is detected; the word to be corrected is a word with a low likelihood in the target comment; the likelihood of the word to be corrected is obtained by inputting the word to be corrected into a language model; the language model is trained based on information of a candidate video; and the candidate video comprises a video in a video library that is highly similar to the target video. The method can automatically identify and correct errors in the comment of the current short video by comprehensively utilizing the current short video content and the overall comment, the other multi-source video content and the comment data that are similar to the current short video, thereby improving the identification rate of errors in the comment and the accuracy of correction, and improving the identification and correction efficiency.

Description

Technical Field

[0001] This application relates to the field of natural language understanding (NLU) technology, and more particularly to a text processing method and related apparatus. Background Technology

[0002] With the widespread use of computers and the rapid development of the internet, short videos are becoming increasingly abundant online. Short videos have become a primary medium for information dissemination. Simultaneously, based on the increasing user base and activity levels of major short video platforms, the number of comments on short videos is also gradually growing. High-quality comments can provide a positive social environment for user interaction and increase the number of views for short videos. Therefore, when faced with a massive amount of comments of varying quality on short videos, the identification and correction of errors in comment text is particularly important.

[0003] Currently, methods for identifying and correcting erroneous phrases in short video comment text have low error recognition rates and low accuracy in correction results, resulting in low efficiency in identification and correction. Summary of the Invention

[0004] This application provides a text processing method and related apparatus. By comprehensively utilizing the current short video content and overall comments, as well as other multi-source video content and comment data similar to the current short video, the method can automatically identify and correct errors in the comments of the current short video, thereby improving the error identification rate and correction accuracy in the comments and increasing the efficiency of identification and correction.

[0005] In a first aspect, embodiments of this application provide a text processing method, the method comprising:

[0006] Obtain the target comments for the target video; the target comments are user comments posted on the target video.

[0007] The target comment is detected as a word to be corrected; the word to be corrected is a word with low likelihood in the target comment, and the likelihood of the word to be corrected is obtained by inputting the word to be corrected into a language model, which is trained based on information from candidate videos, and the candidate videos include videos in the video library that have high similarity to the target video.

[0008] This application provides a method for error recognition in video comment text. Specifically, it involves acquiring a user's comment on a target video, inputting the comment into a language model to obtain the likelihood of each word in the comment, and identifying words with lower likelihood as the words to be corrected. It can be understood that a lower likelihood in the comment is considered any likelihood lower than the highest likelihood. Alternatively, it can be understood that a likelihood below a certain threshold, such as below the average, is considered low. This threshold is not fixed and can vary depending on the application scenario. It can be seen that the likelihood of the words to be corrected is obtained by inputting them into a language model, which is trained based on information from candidate videos. These candidate videos include those in a video library that have a high similarity to the target video.

[0009] This application embodiment improves the error recognition rate and efficiency of comments by comprehensively utilizing the current short video content, overall comments, other multi-source video content and comment data similar to the current short video, and automatically identifying errors in the comments of the current short video.

[0010] In one possible implementation, the method further includes:

[0011] Obtain the target word corresponding to the word to be corrected; the likelihood of the target word is greater than the likelihood of the word to be corrected, and the likelihood of the target word is obtained by inputting the target word into the language model;

[0012] Replace the word to be corrected with the target word.

[0013] This application provides a method for correcting words in video comment text that need to be corrected. Specifically, after detecting the word to be corrected in the target comment, a target word corresponding to the word to be corrected is obtained, and then the word to be corrected is replaced with the target word. The likelihood of the target word is greater than the likelihood of the word to be corrected, thus achieving the purpose of identifying and correcting errors in the video comment text. The likelihood of the target word is obtained by inputting the target word into a language model, which is trained based on information from candidate videos. These candidate videos include videos in a video library that have a high similarity to the target video.

[0014] This application embodiment utilizes the current short video content and overall comments, as well as other multi-source video content and comment data similar to the current short video, to automatically correct errors in the comments of the current short video. This can improve the accuracy and efficiency of error correction in comments.

[0015] In one possible implementation, the detection of the words to be corrected in the target comment includes:

[0016] The target comment is input into the language model to obtain the likelihood of the first word in the target comment;

[0017] If the likelihood of the first word is less than the first threshold, the first word is taken as the word to be corrected.

[0018] This application provides a specific implementation method for detecting words to be corrected in a target comment. By inputting the target comment into the aforementioned language model, the likelihood of each word in the target comment can be obtained, such as the likelihood of the first word in the target comment. If the likelihood of the first word is less than a first threshold, the first word is identified as a word to be corrected in the target comment. The first threshold is not a fixed value and can vary depending on different application scenarios. This application embodiment can improve the accuracy and efficiency of identifying words to be corrected in comments.

[0019] In one possible implementation, the language model includes a first language model and a second language model, and the information of the candidate video includes the video content text of the candidate video and the comment content text of the candidate video; the first language model is trained based on the video content text of the candidate video, and the second language model is trained based on the comment content text of the candidate video.

[0020] The step of inputting the target comment into the language model to obtain the likelihood of the first word in the target comment includes:

[0021] The target comment is input into the first language model to obtain the first probability of the first word;

[0022] The target comment is input into the second language model to obtain the second probability of the first word;

[0023] The weighted sum of the first probability and the second probability is used as the likelihood of the first word.

[0024] This application provides a specific implementation method for detecting phrase likelihood. Specifically, the language model includes a first language model and a second language model. The candidate video information includes video content text of candidate videos with high similarity to the target video and comment content text of candidate videos. The first language model can be trained based on the video content text of the candidate videos, and the second language model can be trained based on the comment content text of the candidate videos. Inputting the target comment into the first and second language models yields the likelihood of each word in the target comment. For example, inputting the target comment into the first language model yields a first probability of the first word in the target comment, which represents the probability that the first language model identifies the first word as a word to be corrected in the target comment. Inputting the target comment into the second language model yields a second probability of the first word in the target comment, which represents the probability that the second language model identifies the first word as a word to be corrected in the target comment. The weighted sum of the first and second probabilities is then used as the likelihood of the first word. This application embodiment improves the detection accuracy of phrase likelihood in the target comment, thereby improving the accuracy and efficiency of identifying words to be corrected in the comment.

[0025] In one possible implementation, obtaining the target word corresponding to the word to be corrected includes:

[0026] Obtain a second word from the dictionary that has a high similarity to the word to be corrected; the dictionary is obtained based on information from the candidate video.

[0027] The target comment after the word to be corrected is replaced by the second word is input into the language model to obtain the likelihood of the second word;

[0028] If the likelihood of the second word is greater than the second threshold, the second word is used as the target word.

[0029] This application provides a specific implementation method for obtaining the target word corresponding to the word to be corrected. Specifically, a lexicon can be obtained based on the information from the candidate videos. A second word with a high similarity to the word to be corrected is obtained from this lexicon. It can be understood that any similarity in the lexicon that is higher than the lowest similarity can be considered a high similarity. Alternatively, it can be understood that any similarity greater than a certain threshold, such as greater than the average, can be considered a high similarity. This threshold is not a fixed value and can vary depending on the application scenario. After obtaining the second word from the lexicon, the target comment after the word to be corrected has been replaced by the second word is input into a language model to obtain the likelihood of the second word. If the likelihood of the second word is greater than the second threshold, the second word is used as the target word to replace the word to be corrected in the target comment. The second threshold is not a fixed value and can vary depending on the application scenario. This application embodiment can improve the accuracy and efficiency of obtaining the target word corresponding to the word to be corrected, thereby improving the accuracy and efficiency of error correction in comments.

[0030] In one possible implementation, obtaining a second word from the lexicon that has a high similarity to the word to be corrected includes:

[0031] The word in the dictionary whose weighted similarity to the word to be corrected in terms of pinyin and stroke count is greater than the third threshold is selected as the second word.

[0032] This application provides a specific implementation method for obtaining the similarity between a second word in a dictionary and a word to be corrected. Specifically, words in the dictionary whose weighted value of pinyin similarity and stroke similarity with the word to be corrected is greater than a third threshold are selected as the second word. The weights can be adjusted to select only words with high pinyin similarity, or only words with high stroke similarity, or a combination of both. The third threshold is not a fixed value and can vary depending on the application scenario. This application improves the accuracy of similarity detection between the second word and the word to be corrected in the dictionary, thereby increasing the accuracy and efficiency of obtaining the target word corresponding to the word to be corrected, and improving the accuracy and efficiency of error correction in comments.

[0033] In one possible implementation, obtaining a second word from the lexicon that has a high similarity to the word to be corrected includes:

[0034] Words in the dictionary whose similarity to the word to be corrected in pinyin or strokes is greater than a third threshold are used as the second word.

[0035] In one possible implementation, the candidate video is a video in the video library that has a similarity to the target video greater than a fourth threshold.

[0036] In this embodiment, the candidate video information includes the text content of candidate videos and the text comment content of candidate videos that have a high similarity to the target video. It can be understood that candidate videos in the video library whose text or image content has a high similarity to the target video are considered to have a high similarity, as long as it is higher than the minimum similarity. This can also be understood as a similarity exceeding a certain threshold, such as above the average. Specifically, videos in the video library whose text or image content has a similarity to the target video greater than a fourth threshold are used as candidate videos to train the language model required for text recognition and error correction. The fourth threshold is not a fixed value and can vary depending on the application scenario. Through this embodiment, the obtained candidate videos can be made more similar to the target video, thereby improving the accuracy and efficiency of the language model trained based on the candidate video information in recognizing and correcting words in the target video.

[0037] In one possible implementation, the language model includes a first language model and a second language model, and the information of the candidate video includes the video content text of the candidate video and the comment content text of the candidate video; the first language model is trained based on the video content text of the candidate video, and the second language model is trained based on the comment content text of the candidate video.

[0038] The step of inputting the target comment after the word to be corrected is replaced by the second word into the language model to obtain the likelihood of the second word includes:

[0039] The target comment after the word to be corrected is replaced by the second word is input into the first language model to obtain the third probability of the second word;

[0040] The target comment after the word to be corrected is replaced by the second word is input into the second language model to obtain the fourth probability of the second word;

[0041] The likelihood of the second word is obtained by weighted summation of the third probability and the fourth probability, or by weighted summation of the third probability, the fourth probability, and the similarity.

[0042] This application provides a specific implementation method for detecting phrase likelihood. Specifically, the language model includes a first language model and a second language model. The candidate video information includes video content text of candidate videos with high similarity to the target video and comment content text of candidate videos. The first language model can be trained based on the video content text of the candidate videos, and the second language model can be trained based on the comment content text of the candidate videos. The target comment after the word to be corrected is replaced by the second word is input into the first and second language models to obtain the likelihood of the second word in the target comment. For example, inputting the target comment after the word to be corrected is replaced by the second word into the first language model yields a third probability of the second word in the target comment, which represents the probability that the first language model recognizes the second word may appear in the target comment. Inputting the target comment after the word to be corrected is replaced by the second word into the second language model yields a fourth probability of the second word in the target comment, which represents the probability that the second language model recognizes the second word may appear in the target comment. The weighted sum of the third and fourth probabilities is then used as the likelihood of the second word. Alternatively, the likelihood of the second word can be obtained by weighted summing the third probability, the fourth probability, and the similarity between the second word and the word to be corrected. Through the embodiments of this application, the detection accuracy of the likelihood of the second word in the lexicon can be improved, thereby increasing the accuracy and efficiency of obtaining the target word corresponding to the word to be corrected, and improving the accuracy and efficiency of error correction in comments.

[0043] Secondly, embodiments of this application provide a text processing apparatus, the apparatus comprising:

[0044] An acquisition unit is used to acquire target comments for a target video; the target comments are comments posted by users on the target video.

[0045] A detection unit is used to detect words to be corrected in the target comment; the words to be corrected are words with low likelihood in the target comment, and the likelihood of the words to be corrected is obtained by inputting the words to be corrected into a language model, which is trained based on information from candidate videos, including videos in a video library that have high similarity to the target video.

[0046] This application provides a method for error recognition in video comment text. Specifically, it involves acquiring a user's comment on a target video, inputting the comment into a language model to obtain the likelihood of each word in the comment, and identifying words with lower likelihood as the words to be corrected. It can be understood that a lower likelihood in the comment is considered any likelihood lower than the highest likelihood. Alternatively, it can be understood that a likelihood below a certain threshold, such as below the average, is considered low. This threshold is not fixed and can vary depending on the application scenario. It can be seen that the likelihood of the words to be corrected is obtained by inputting them into a language model, which is trained based on information from candidate videos. These candidate videos include those in a video library that have a high similarity to the target video.

[0047] This application embodiment improves the error recognition rate and efficiency of comments by comprehensively utilizing the current short video content, overall comments, other multi-source video content and comment data similar to the current short video, and automatically identifying errors in the comments of the current short video.

[0048] In one possible implementation, the device further includes:

[0049] The acquisition unit is further configured to acquire the target word corresponding to the word to be corrected; the likelihood of the target word is greater than the likelihood of the word to be corrected, and the likelihood of the target word is obtained by inputting the target word into the language model;

[0050] A correction unit is used to replace the word to be corrected with the target word.

[0051] This application provides a method for correcting words in video comment text that need to be corrected. Specifically, after detecting the word to be corrected in the target comment, a target word corresponding to the word to be corrected is obtained, and then the word to be corrected is replaced with the target word. The likelihood of the target word is greater than the likelihood of the word to be corrected, thus achieving the purpose of identifying and correcting errors in the video comment text. The likelihood of the target word is obtained by inputting the target word into a language model, which is trained based on information from candidate videos. These candidate videos include videos in a video library that have a high similarity to the target video.

[0052] This application embodiment utilizes the current short video content and overall comments, as well as other multi-source video content and comment data similar to the current short video, to automatically correct errors in the comments of the current short video. This can improve the accuracy and efficiency of error correction in comments.

[0053] In one possible implementation, the detection unit is specifically used to input the target comment into the language model to obtain the likelihood of the first word in the target comment;

[0054] The detection unit is further configured to use the first word as the word to be corrected when the likelihood of the first word is less than a first threshold.

[0055] This application provides a specific implementation method for detecting words to be corrected in a target comment. By inputting the target comment into the aforementioned language model, the likelihood of each word in the target comment can be obtained, such as the likelihood of the first word in the target comment. If the likelihood of the first word is less than a first threshold, the first word is identified as a word to be corrected in the target comment. The first threshold is not a fixed value and can vary depending on different application scenarios. This application embodiment can improve the accuracy and efficiency of identifying words to be corrected in comments.

[0056] In one possible implementation, the language model includes a first language model and a second language model, and the information of the candidate video includes the video content text of the candidate video and the comment content text of the candidate video; the first language model is trained based on the video content text of the candidate video, and the second language model is trained based on the comment content text of the candidate video.

[0057] The detection unit is specifically used to input the target comment into the first language model to obtain the first probability of the first word;

[0058] The detection unit is further configured to input the target comment into the second language model to obtain the second probability of the first word;

[0059] The detection unit is further configured to use the weighted sum of the first probability and the second probability as the likelihood of the first word.

[0060] This application provides a specific implementation method for detecting phrase likelihood. Specifically, the language model includes a first language model and a second language model. The candidate video information includes video content text of candidate videos with high similarity to the target video and comment content text of candidate videos. The first language model can be trained based on the video content text of the candidate videos, and the second language model can be trained based on the comment content text of the candidate videos. Inputting the target comment into the first and second language models yields the likelihood of each word in the target comment. For example, inputting the target comment into the first language model yields a first probability of the first word in the target comment, which represents the probability that the first language model identifies the first word as a word to be corrected in the target comment. Inputting the target comment into the second language model yields a second probability of the first word in the target comment, which represents the probability that the second language model identifies the first word as a word to be corrected in the target comment. The weighted sum of the first and second probabilities is then used as the likelihood of the first word. This application embodiment improves the detection accuracy of phrase likelihood in the target comment, thereby improving the accuracy and efficiency of identifying words to be corrected in the comment.

[0061] In one possible implementation, the acquisition unit is specifically used to acquire a second word in the lexicon that has a high similarity to the word to be corrected; the lexicon is obtained based on information from the candidate video;

[0062] The detection unit is specifically used to input the target comment after the word to be corrected is replaced by the second word into the language model to obtain the likelihood of the second word;

[0063] The detection unit is further configured to use the second word as the target word when the likelihood of the second word is greater than the second threshold.

[0064] This application provides a specific implementation method for obtaining the target word corresponding to the word to be corrected. Specifically, a lexicon can be obtained based on the information from the candidate videos. A second word with a high similarity to the word to be corrected is obtained from this lexicon. It can be understood that any similarity in the lexicon that is higher than the lowest similarity can be considered a high similarity. Alternatively, it can be understood that any similarity greater than a certain threshold, such as greater than the average, can be considered a high similarity. This threshold is not a fixed value and can vary depending on the application scenario. After obtaining the second word from the lexicon, the target comment after the word to be corrected has been replaced by the second word is input into a language model to obtain the likelihood of the second word. If the likelihood of the second word is greater than the second threshold, the second word is used as the target word to replace the word to be corrected in the target comment. The second threshold is not a fixed value and can vary depending on the application scenario. This application embodiment can improve the accuracy and efficiency of obtaining the target word corresponding to the word to be corrected, thereby improving the accuracy and efficiency of error correction in comments.

[0065] In one possible implementation, the detection unit is specifically used to identify words in the lexicon whose weighted similarity to the word to be corrected, based on both pinyin and stroke similarity, is greater than a third threshold, as the second word.

[0066] This application provides a specific implementation method for obtaining the similarity between a second word in a dictionary and a word to be corrected. Specifically, words in the dictionary whose weighted value of pinyin similarity and stroke similarity with the word to be corrected is greater than a third threshold are selected as the second word. The weights can be adjusted to select only words with high pinyin similarity, or only words with high stroke similarity, or a combination of both. The third threshold is not a fixed value and can vary depending on the application scenario. This application improves the accuracy of similarity detection between the second word and the word to be corrected in the dictionary, thereby increasing the accuracy and efficiency of obtaining the target word corresponding to the word to be corrected, and improving the accuracy and efficiency of error correction in comments.

[0067] In one possible implementation, the detection unit is further configured to use words in the lexicon whose similarity to the word to be corrected in pinyin or strokes is greater than a third threshold as the second word.

[0068] In one possible implementation, the candidate video is a video in the video library that has a similarity to the target video greater than a fourth threshold.

[0069] In this embodiment, the candidate video information includes the text content of candidate videos and the text comment content of candidate videos that have a high similarity to the target video. It can be understood that candidate videos in the video library whose text or image content has a high similarity to the target video are considered to have a high similarity, as long as it is higher than the minimum similarity. This can also be understood as a similarity exceeding a certain threshold, such as above the average. Specifically, videos in the video library whose text or image content has a similarity to the target video greater than a fourth threshold are used as candidate videos to train the language model required for text recognition and error correction. The fourth threshold is not a fixed value and can vary depending on the application scenario. Through this embodiment, the obtained candidate videos can be made more similar to the target video, thereby improving the accuracy and efficiency of the language model trained based on the candidate video information in recognizing and correcting words in the target video.

[0070] In one possible implementation, the language model includes a first language model and a second language model, and the information of the candidate video includes the video content text of the candidate video and the comment content text of the candidate video; the first language model is trained based on the video content text of the candidate video, and the second language model is trained based on the comment content text of the candidate video.

[0071] The detection unit is specifically used to input the target comment after the word to be corrected is replaced by the second word into the first language model to obtain the third probability of the second word;

[0072] The detection unit is further configured to input the target comment after the word to be corrected is replaced by the second word into the second language model to obtain the fourth probability of the second word;

[0073] The detection unit is further configured to use the weighted sum of the third probability and the fourth probability, or the weighted sum of the third probability, the fourth probability, and the similarity as the likelihood of the second word.

[0074] This application provides a specific implementation method for detecting phrase likelihood. Specifically, the language model includes a first language model and a second language model. The candidate video information includes video content text of candidate videos with high similarity to the target video and comment content text of candidate videos. The first language model can be trained based on the video content text of the candidate videos, and the second language model can be trained based on the comment content text of the candidate videos. The target comment after the word to be corrected is replaced by the second word is input into the first and second language models to obtain the likelihood of the second word in the target comment. For example, inputting the target comment after the word to be corrected is replaced by the second word into the first language model yields a third probability of the second word in the target comment, which represents the probability that the first language model recognizes the second word may appear in the target comment. Inputting the target comment after the word to be corrected is replaced by the second word into the second language model yields a fourth probability of the second word in the target comment, which represents the probability that the second language model recognizes the second word may appear in the target comment. The weighted sum of the third and fourth probabilities is then used as the likelihood of the second word. Alternatively, the likelihood of the second word can be obtained by weighted summing the third probability, the fourth probability, and the similarity between the second word and the word to be corrected. Through the embodiments of this application, the detection accuracy of the likelihood of the second word in the lexicon can be improved, thereby increasing the accuracy and efficiency of obtaining the target word corresponding to the word to be corrected, and improving the accuracy and efficiency of error correction in comments.

[0075] Thirdly, embodiments of this application provide an electronic device, which includes a processor and a memory; the memory is used to store computer execution instructions; the processor is used to execute the computer execution instructions stored in the memory to cause the electronic device to perform the method as described in the first aspect and any possible implementation thereof. Optionally, the electronic device further includes a transceiver for receiving or transmitting signals.

[0076] Fourthly, embodiments of this application provide a computer-readable storage medium for storing instructions or computer programs; when the instructions or computer programs are executed, the method described in the first aspect and any possible implementation is implemented.

[0077] Fifthly, embodiments of this application provide a computer program product, the computer program product including instructions or a computer program; when the instructions or the computer program are executed, the method described in the first aspect and any possible implementation is implemented.

[0078] Sixthly, embodiments of this application provide a chip including a processor configured to execute instructions that, when executed, cause the chip to perform the method described in the first aspect and any possible implementation. Optionally, the chip further includes a communication interface configured to receive or transmit signals.

[0079] In a seventh aspect, embodiments of this application provide a system comprising at least one text processing device as described in the second or third aspect or a chip as described in the sixth aspect.

[0080] Furthermore, in the process of performing the method described in the first aspect and any possible implementation above, the processes related to sending and / or receiving information in the above methods can be understood as the process of the processor outputting information, and / or the process of the processor receiving input information. When outputting information, the processor can output the information to a transceiver (or communication interface, or transmitting module) so that the transceiver can transmit it. After the information is output by the processor, it may need to undergo other processing before reaching the transceiver. Similarly, when the processor receives input information, the transceiver (or communication interface, or transmitting module) receives the information and inputs it to the processor. Furthermore, after the transceiver receives the information, the information may need to undergo other processing before being input to the processor.

[0081] Based on the above principles, for example, the information sent mentioned in the aforementioned method can be understood as information output by the processor. Similarly, the information received can be understood as information received by the processor from input.

[0082] Optionally, unless otherwise specified, or unless they contradict their actual function or internal logic in the relevant description, the operations of the processor, such as transmitting, sending, and receiving, can be more generally understood as processor output and receiving, input, and other operations.

[0083] Optionally, in performing the methods described in the first aspect and any possible implementation above, the processor may be a processor specifically designed to perform these methods, or it may be a processor that performs these methods by executing computer instructions stored in memory, such as a general-purpose processor. The memory may be a non-transitory memory, such as read-only memory (ROM), which may be integrated with the processor on the same chip or disposed on different chips. This application does not limit the type of memory or the arrangement of the memory and processor.

[0084] In one possible implementation, at least one of the aforementioned memories is located outside the device.

[0085] In yet another possible implementation, at least one of the aforementioned memories is located within the device.

[0086] In another possible implementation, a portion of the memory of the at least one memory is located inside the device, while another portion is located outside the device.

[0087] In this application, the processor and memory may also be integrated into a single device, that is, the processor and memory can be integrated together.

[0088] In this embodiment of the application, by comprehensively utilizing the current short video content and overall comments, as well as other multi-source video content and comment data similar to the current short video, the comments of the current short video can be automatically identified and corrected, thereby improving the error identification rate and correction accuracy in the comments and increasing the efficiency of identification and correction. Attached Figure Description

[0089] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0090] Figure 1 A schematic diagram of a text processing architecture provided in an embodiment of this application;

[0091] Figure 2 A flowchart illustrating a text processing method provided in an embodiment of this application;

[0092] Figure 3 A schematic diagram of the structure of a language model provided in an embodiment of this application;

[0093] Figure 4 A flowchart illustrating another text processing method provided in an embodiment of this application;

[0094] Figure 5 This is a schematic diagram of the structure of another language model provided in an embodiment of this application;

[0095] Figure 6 This is a schematic diagram of the structure of a text processing device provided in an embodiment of this application;

[0096] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0097] Figure 8a A data sharing system provided in this application embodiment;

[0098] Figure 8b A schematic diagram of a blockchain provided for an embodiment of this application;

[0099] Figure 8c This is a schematic diagram illustrating a new block generation process provided in an embodiment of this application. Detailed Implementation

[0100] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described below with reference to the accompanying drawings.

[0101] The terms "first" and "second," etc., used in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0102] The term "embodiment" as used herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0103] It should be understood that in this application, "at least one (item)" means one or more, "more than one" means two or more, "at least two (items)" means two or three or more, and "and / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0104] This application provides a text processing method involving technologies such as artificial intelligence natural language understanding. To more clearly describe the solution of this application, some knowledge related to text processing will be introduced below.

[0105] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0106] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0107] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0108] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0109] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0110] Likelihood: A metric that measures "how similar a text is to a normal text" or "the probability of a word appearing in a text," and can be calculated using a language model. The lower the likelihood, the less likely the word is to appear in the text, and the less similar the text is to a normal text.

[0111] Short videos originate from long videos: Many short videos are constructed by extracting a segment from a long video. The long video from which the short video was extracted is called the source long video. Long videos are generally professionally produced videos of longer duration, such as TV dramas, movies, variety shows, sports, animation, games, and children's videos.

[0112] Automatic speech recognition (ASR) text: The text content obtained by converting video speech using ASR.

[0113] Video optical character recognition (OCR) text: The text content converted from video images using OCR.

[0114] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0115] With the widespread use of computers and the rapid development of the internet, short videos are becoming increasingly abundant online. Short videos have become a primary medium for information dissemination. Simultaneously, based on the increasing user base and activity levels of major short video platforms, the number of comments on short videos is also gradually growing. High-quality comments can provide a positive social environment for user interaction and increase the number of views for short videos. Therefore, when faced with a massive amount of comments of varying quality on short videos, the identification and correction of errors in comment text is particularly important.

[0116] Currently, methods for identifying and correcting potential errors in short video comment text all rely on ordinary text correction techniques. They use the current comment content to identify and correct errors, resulting in low error recognition rates and low accuracy in correction, leading to low efficiency in identification and correction.

[0117] To address the issues of low accuracy and efficiency in the aforementioned identification and correction methods, this application provides a novel text processing architecture and, based on this architecture, a novel text processing method. By implementing the text processing architecture and method provided in this application, the current short video content and overall comments, as well as other multi-source video content and comment data similar to the current short video, can be comprehensively utilized to automatically identify and correct errors in the comments of the current short video. This improves the error identification rate and correction accuracy in the comments, thereby enhancing the efficiency of identification and correction.

[0118] The embodiments of this application are described below with reference to the accompanying drawings.

[0119] Please see Figure 1 , Figure 1 This is a schematic diagram of a text processing architecture provided in an embodiment of this application.

[0120] like Figure 1 As shown, the text processing architecture in this application embodiment mainly includes a multi-source video acquisition module, a multi-source vocabulary / language model construction module based on multi-source videos, a current video comment error identification module, and a current video comment error correction module.

[0121] The multi-source video acquisition module is primarily used to identify the source long video of the acquired short video, as well as other similar short videos. Specifically, it employs text content retrieval and image content retrieval. It constructs a text (title, OCR / ASR text) and image (image frames extracted from the video) similarity retrieval index for the videos in the video library. Based on the title, OCR / ASR text, and image frames of the current short video, it performs a similarity search on the text and image indexes in the video library. Videos whose text and image similarity meets a certain threshold are selected as a multi-source similar video candidate list for the current short video.

[0122] The module for building a multi-source lexicon / language model based on multi-source videos is mainly used to construct corresponding lexicons and language models based on video information from multiple sources, including current short videos, source long videos, and similar short videos. Specifically, it acquires the content text of the current short video, source long video, and similar short videos, including titles, OCR / ASR text, long video descriptions, and actor information. It then segments this multi-source video content text to construct a video content lexicon, which includes several video content words and their frequencies appearing in the video content text. These video content words are then used as training corpus to train a video content language model. This model can calculate the contextual language model probability of each word within the knowledge context of the multi-source video content text corpus, given the input video text content. Similarly, it acquires the comment text of the current short video, source long video, and similar short videos, segments this multi-source comment text, and constructs a comment lexicon, which includes several comment words and their frequencies appearing in the comment text. The comment words in the comment lexicon are used as training corpus to train a video comment language model. This model takes input text content and calculates the contextual language model probability of each word within a multi-source video comment text corpus. The aforementioned multi-source language model is first pre-trained on a general large-scale text corpus to fully learn the model's representation. Then, the general corpus language model is fine-tuned on video content text and video comment text to construct a multi-source video content language model and a multi-source video comment language model.

[0123] The current video comment error detection module primarily identifies errors in the comment to be corrected by combining the general language model, multi-source video content language model, and multi-source video comment language model obtained from the aforementioned module that constructs a multi-source lexicon / language model based on multi-source videos. Specifically, the current comment is segmented into words, and then processed through the aforementioned general language model, multi-source video content language model, and multi-source video comment language model to calculate the contextual language model probability of each word in the current comment. When the overall language model probability of a word is less than a certain threshold, the word is determined to be erroneous, and subsequent error correction will be performed.

[0124] The current video comment error correction module is primarily used to correct errors in words identified by the aforementioned current video comment error identification module, replacing them with target words. Specifically, the current video comment error correction module constructs correction candidate indexes for the aforementioned multi-source video content lexicon and multi-source video comment lexicon, which are used to construct correction candidates at the positions of words to be corrected. Correction candidate indexes can be constructed using pinyin indexes and stroke indexes for the aforementioned multi-source lexicon, and based on the constructed pinyin and stroke indexes, pinyin and stroke similarity correction candidates are constructed for the positions of words to be corrected. After constructing the correction candidate queue, a language model probability score is calculated for each correction candidate. By replacing the position of the word to be corrected with a correction candidate, the result is input into the aforementioned general language model, multi-source video content language model, and multi-source video comment language model to calculate the comprehensive language model probability of each correction candidate. When the comprehensive language model probability of a correction candidate is greater than a certain threshold, that correction candidate is determined to be the target word corresponding to the word to be corrected, and is used to replace the word to be corrected, thus achieving error correction in the comment text.

[0125] Based on the above Figure 1 In addition to the text processing architecture in the previous application, this application also provides a new text processing method, which will be discussed below. Figure 2 Explain it.

[0126] Please see Figure 2 , Figure 2 This application provides a flowchart illustrating a text processing method, which includes, but is not limited to, the following steps:

[0127] Step 201: Obtain the target comments for the target video.

[0128] The electronic device acquires the target comments for the target video, which are comments posted by users on the target video.

[0129] In this embodiment, the electronic device is a device equipped with a processor capable of executing computer instructions. This electronic device can be a mobile phone, computer, server, etc. A target comment is a sentence or a combination of sentences with complete and systematic meaning. A target comment can be a sentence, a short paragraph, or a short passage, etc.

[0130] Step 202: Detect the words to be corrected in the target comment.

[0131] The electronic device detects words to be corrected in the target comment. These words are those with low likelihood in the target comment. The likelihood of these words is obtained by inputting them into a language model. This language model is trained based on information from candidate videos, which include videos in a video library that have high similarity to the target video.

[0132] Understandably, any lower likelihood in the target comment than the highest likelihood can be considered low. Alternatively, it can be understood as any likelihood below a certain threshold, such as below the average, which is not a fixed value and can vary depending on the application scenario. It can be seen that the likelihood of the word to be corrected is obtained by inputting the word into a language model, which is trained based on information from candidate videos. These candidate videos include those in the video library that have a high similarity to the target video.

[0133] The specific implementation process is as follows: first, obtain videos with a high degree of similarity to the target video.

[0134] This study utilizes text content retrieval and image content retrieval to index videos in a video library. Text content retrieval involves constructing an inverted index from the video's subtitle text (OCR / ASR), allowing the search for similar videos by inputting the text content of the target video. Image content retrieval extracts image frames from the video, extracting one frame at intervals, and constructing a representation vector for each frame using a model, thus achieving vector similarity retrieval. When identifying the source long video of the current target video, text is recognized based on OCR / ASR, and image frames are extracted. The long video index is then searched using both text and image similarity, and the highest similarity long video meeting a certain threshold is identified as the source long video of the current target video. Similarly, when identifying similar short videos of the current target video, text is recognized based on OCR / ASR, and image frames are extracted. The long video index is then searched using both text and image similarity, and the highest similarity short video meeting a certain threshold is identified as the similar short video of the current target video. This method enables the acquisition of multi-source videos with high similarity to the target video. The threshold mentioned above is not a fixed value and can vary depending on different application scenarios. Through the embodiments of this application, the obtained candidate videos can be made more similar to the target video, thereby enabling the language model trained based on the information from the candidate videos to achieve higher accuracy and efficiency in recognizing and correcting the words to be corrected in the target video.

[0135] Secondly, based on the current target video and the multi-source candidate videos with high similarity to the target video obtained from the target video, a multi-source lexicon and language model are constructed.

[0136] As described above, multi-source candidate videos include, but are not limited to, long videos from the source of the target video and short videos with high similarity to the target video. The information in these multi-source candidate videos mainly includes the video content text of candidate videos with high similarity to the target video and the comment content text of candidate videos. It can be understood that candidate videos in the video library whose text or image content has high similarity to the target video, and whose text or image content has high similarity to the target video, can be considered to have high similarity as long as it is higher than the minimum similarity score. Alternatively, it can be understood that any similarity score above a certain threshold, such as above the average, can be considered high similarity.

[0137] Based on word segmentation of the video content text of the aforementioned candidate videos, a video content lexicon can be obtained, which includes several video content words and their frequencies appearing in the video content text. Similarly, based on word segmentation of the comment content text of the aforementioned candidate videos, a comment lexicon can be obtained, which includes several comment words and their frequencies appearing in the comment text. Furthermore, using the aforementioned video content lexicon as training corpus, a first language model can be trained. This first language model takes input video text content and calculates the contextual language model probability of each word within a multi-source video content text corpus. Using the aforementioned comment lexicon as training corpus, a second language model can be trained. This second language model takes input text content and calculates the contextual language model probability of each word within a multi-source video comment text corpus.

[0138] Finally, based on the constructed multi-source language model, errors in the target comments of the target video are identified.

[0139] By inputting the target comment into a language model trained based on the aforementioned candidate video information, the likelihood of each word in the target comment can be obtained. For example, the likelihood of the first word in the target comment can be obtained. If the likelihood of the first word is less than a first threshold, then the first word is considered the word to be corrected in the target comment. The first threshold is not a fixed value and can vary depending on the application scenario.

[0140] Alternatively, by inputting the target comment into the aforementioned first and second language models, the likelihood of each word in the target comment can be obtained. For example, inputting the target comment into the first language model yields a first probability of the first word in the target comment, which represents the probability that the first language model identifies the first word as a word to be corrected in the target comment; inputting the target comment into the second language model yields a second probability of the first word in the target comment, which represents the probability that the second language model identifies the first word as a word to be corrected in the target comment. The weighted sum of the first and second probabilities is then used as the likelihood of the first word. Through the embodiments of this application, the detection accuracy of phrase likelihood in the target comment can be improved, thereby improving the accuracy and efficiency of identifying words to be corrected in the comment.

[0141] Specifically, the language model built based on multi-source candidate videos mentioned above can be found in [reference needed]. Figure 3 , Figure 3 This is a schematic diagram of the structure of a language model provided in an embodiment of this application.

[0142] like Figure 3 As shown, the model is trained using various types of video content text as training corpora. A Transformer-Encoder network model is employed to calculate the contextual bidirectional language model probability of each word. The Transformer is a novel feature extractor based on a self-attention mechanism, which can replace Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) to extract features from text sequences. The Transformer can be used in encoder-decoder architectures, or independently in encoder or decoder architectures. When the video content text is input into the model, it is segmented into words such as word 0 at position 0, word 1 at position 1, word 2 at position 2, ..., word w at position w. Each word group is then passed through a Transformer-Encoder layer to obtain the language model probability of each word group. It can be seen that the video content language model trained using various types of video content text as training corpora is capable of calculating the contextual language model probability of each word within the context of multi-source video content text corpora.

[0143] Similarly, by training a network model with a Transformer-Encoder structure on the various types of video comment texts mentioned above, the contextual bidirectional language model probability of each word is calculated. It can be concluded that the video comment language model trained on various types of video comment texts as training corpus is capable of calculating the contextual language model probability of each word within the knowledge context of multi-source video comment text corpus.

[0144] For example, under a target video showcasing an actor's acting skills, a user posts the comment, "This actor's eye condition is amazing!" At this point, the language model, based on the video content text and video comment text of multiple source videos similar to the target video, identifies the word "eye condition" in the target comment and performs subsequent corrective actions.

[0145] Please see Figure 4 , Figure 4 This is a flowchart illustrating another text processing method provided in the embodiments of this application, and can also be understood as the above. Figure 2 This is a supplement to the flowchart of text processing methods in [the document / platform].

[0146] Step 401: Obtain the target comments for the target video.

[0147] The electronic device acquires the target comments for the target video, which are comments posted by users on the target video.

[0148] In this embodiment, the electronic device is a device equipped with a processor capable of executing computer instructions. This electronic device can be a mobile phone, computer, server, etc. A target comment is a sentence or a combination of sentences with complete and systematic meaning. A target comment can be a sentence, a short paragraph, or a short passage, etc.

[0149] Step 402: Construct a multi-source lexicon and language model based on multi-source video content and text similar to the target video.

[0150] Electronic devices construct multi-source lexicons and language models based on multi-source video content and text similar to the target video.

[0151] Among them, based on word segmentation of the video content text of multi-source candidate videos, a video content lexicon can be obtained. This video content lexicon includes several video content words and their word frequencies in the video content text, as shown in Table 1 below:

[0152] Table 1

[0153]

[0154]

[0155] Furthermore, based on the aforementioned video content lexicon as training data, a video content language model can be trained. This model is capable of calculating the contextual language model probability of each word within a multi-source video content text corpus. For details, please refer to [link to relevant documentation]. Figure 3 This will not be elaborated upon here.

[0156] Furthermore, based on word segmentation of the comment content text of multi-source candidate videos, a comment lexicon can be obtained. This comment lexicon includes several comment words and their word frequencies in the comment text, as shown in Table 2 below:

[0157] Table 2

[0158] Comment 1 Word 1 frequency Comment 2 Word 2 frequency …… …… Comment word w Word frequency

[0159] Furthermore, based on the aforementioned comment lexicon as training corpus, a comment language model can be trained. This model takes input text content and calculates the contextual language model probability of each word within a multi-source video comment text corpus. For details, please refer to [link / reference]. Figure 3 This will not be elaborated upon here.

[0160] Step 403: Detect the words to be corrected in the target comment.

[0161] The electronic device detects words to be corrected in the target comment. These words are those with low likelihood in the target comment. The likelihood of these words is obtained by inputting them into a language model. This language model is trained based on information from candidate videos, which include videos in a video library that have high similarity to the target video.

[0162] This step is the same as step 202 above, and will not be repeated here.

[0163] Step 404: Obtain the target word corresponding to the word to be corrected.

[0164] The electronic device acquires the target word corresponding to the word to be corrected, wherein the likelihood of the target word is greater than the likelihood of the word to be corrected. This achieves the purpose of identifying and correcting errors in video comment text. The likelihood of the target word is obtained by inputting the target word into a language model, which is trained based on information from candidate videos. The candidate videos here include videos in the video library that have a high similarity to the target video mentioned above.

[0165] Specifically, firstly, based on the multi-source lexicon constructed in step 402 above, a second word with a high similarity to the word to be corrected is obtained from this lexicon. It can be understood that any similarity in the lexicon that is higher than the lowest similarity can be considered a high similarity. Alternatively, it can be understood that any similarity greater than a certain threshold, such as greater than the average, can be considered a high similarity. This threshold is not a fixed value and can vary depending on the application scenario. Specifically, words in the lexicon whose weighted sum of pinyin and stroke similarity with the word to be corrected is greater than a third threshold can be used as the second word. This can be achieved by adjusting the weights of both, using only words with high pinyin similarity as the second word; or only words with high stroke similarity as the second word; or a combination of both. The third threshold is not a fixed value and can vary depending on the application scenario. Through the embodiments of this application, the detection accuracy of similarity between the second word and the word to be corrected in the dictionary can be improved, thereby improving the accuracy and efficiency of obtaining the target word corresponding to the word to be corrected, and improving the accuracy and efficiency of error correction in comments.

[0166] Then, after obtaining the second word from the dictionary, the target comment, after the word to be corrected has been replaced by the second word, is input into the above... Figure 3 In the language model shown, the likelihood of the second word is obtained. If the likelihood of the second word is greater than a second threshold, the second word is used as the target word to replace the word to be corrected in the target comment. The second threshold is not a fixed value and can vary depending on the application scenario.

[0167] Furthermore, the aforementioned language model includes a first language model and a second language model. The first language model is trained based on the video content text of the candidate video, and the second language model is trained based on the comment content text of the candidate video. Inputting the target comment after the word to be corrected is replaced by the second word into the first and second language models respectively yields the likelihood of the second word in the target comment. For example, inputting the target comment after the word to be corrected is replaced by the second word into the first language model yields a third probability of the second word in the target comment, which represents the probability that the first language model recognizes the second word may appear in the target comment; inputting the target comment after the word to be corrected is replaced by the second word into the second language model yields a fourth probability of the second word in the target comment, which represents the probability that the second language model recognizes the second word may appear in the target comment. The weighted sum of the third and fourth probabilities is then used as the likelihood of the second word. Alternatively, the weighted sum of the third and fourth probabilities, along with the similarity between the second word and the word to be corrected, is used as the likelihood of the second word. Through the embodiments of this application, the detection accuracy of the second word likelihood in the lexicon can be improved, thereby improving the accuracy and efficiency of obtaining the target word corresponding to the word to be corrected, and improving the accuracy and efficiency of error correction in comments.

[0168] Specifically, the model used in this step to obtain the target word corresponding to the word to be corrected can also be found in [reference needed]. Figure 5 , Figure 5 This is a schematic diagram of another language model provided in an embodiment of this application.

[0169] like Figure 5 As shown, this language model is used for candidate correction. After constructing a candidate correction queue by obtaining a second word with high similarity to the word to be corrected from the lexicon, a language model probability score is calculated for each candidate correction. By replacing the word to be corrected with the candidate correction and inputting it into the constructed general language model, multi-source video content language model, and multi-source video commentary language model, the comprehensive language model probability of each candidate correction is calculated. The comprehensive language model probability of the candidate correction must be greater than the comprehensive language model probability of the original word to be corrected, thus achieving the purpose of identification and correction. The weighted sum of the comprehensive language model probability of the candidate correction and the similarity probability between the candidate correction and the word to be corrected is used as the likelihood of the candidate correction. When the likelihood of a candidate correction exceeds a certain threshold, it is determined to be the target word corresponding to the word to be corrected, and is used to replace the word to be corrected, thereby correcting errors in the commentary text.

[0170] For example, under a target video showcasing an actor's acting skills, a user posts the target comment, "This actor's eye condition is amazing!". At this point, the language model, based on the video content text and video comment text of multiple source videos similar to the target video, identifies the word "eye condition" in the target comment that needs correction. It then obtains several correction candidates from a multi-source lexicon built from the video content text and video comment text of similar source videos. These correction candidates are input into the aforementioned general language model, multi-source video content language model, and multi-source video comment language model, calculating the comprehensive language model probability of each correction candidate. The comprehensive language model probability of the correction candidate "acting skills" is the highest, and it is used as the target word to replace the word "eye condition." Therefore, the original target comment "This actor's eye condition is amazing!" is replaced with "This actor's acting skills are amazing!" after text recognition and correction.

[0171] Step 405: Replace the word to be corrected with the target word.

[0172] After detecting the word to be corrected in the target comment and obtaining the target word corresponding to the word to be corrected, the electronic device replaces the word to be corrected with the target word.

[0173] This application embodiment utilizes the current short video content and overall comments, as well as other multi-source video content and comment data similar to the current short video, to automatically correct errors in the comments of the current short video. This can improve the accuracy and efficiency of error correction in comments.

[0174] The methods of the embodiments of this application have been described in detail above. The apparatus of the embodiments of this application is provided below.

[0175] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a text processing device provided in an embodiment of this application. The text processing device 60 may include an acquisition unit 601 and a detection unit 602, wherein the descriptions of each unit are as follows:

[0176] The acquisition unit 601 is used to acquire target comments of the target video; the target comments are comments posted by users on the target video.

[0177] The detection unit 602 is used to detect words to be corrected in the target comment; the words to be corrected are words with low likelihood in the target comment, and the likelihood of the words to be corrected is obtained by inputting the words to be corrected into a language model, the language model is trained based on information from candidate videos, and the candidate videos include videos in the video library that have high similarity to the target video.

[0178] This application provides a method for error recognition in video comment text. Specifically, it involves acquiring a user's comment on a target video, inputting the comment into a language model to obtain the likelihood of each word in the comment, and identifying words with lower likelihood as the words to be corrected. It can be understood that a lower likelihood in the comment is considered any likelihood lower than the highest likelihood. Alternatively, it can be understood that a likelihood below a certain threshold, such as below the average, is considered low. This threshold is not fixed and can vary depending on the application scenario. It can be seen that the likelihood of the words to be corrected is obtained by inputting them into a language model, which is trained based on information from candidate videos. These candidate videos include those in a video library that have a high similarity to the target video.

[0179] This application embodiment improves the error recognition rate and efficiency of comments by comprehensively utilizing the current short video content, overall comments, other multi-source video content and comment data similar to the current short video, and automatically identifying errors in the comments of the current short video.

[0180] In one possible implementation, the device further includes:

[0181] The acquisition unit 601 is further configured to acquire the target word corresponding to the word to be corrected; the likelihood of the target word is greater than the likelihood of the word to be corrected, and the likelihood of the target word is obtained by inputting the target word into the language model;

[0182] The correction unit 603 is used to replace the word to be corrected with the target word.

[0183] This application provides a method for correcting words in video comment text that need to be corrected. Specifically, after detecting the word to be corrected in the target comment, a target word corresponding to the word to be corrected is obtained, and then the word to be corrected is replaced with the target word. The likelihood of the target word is greater than the likelihood of the word to be corrected, thus achieving the purpose of identifying and correcting errors in the video comment text. The likelihood of the target word is obtained by inputting the target word into a language model, which is trained based on information from candidate videos. These candidate videos include videos in a video library that have a high similarity to the target video.

[0184] This application embodiment utilizes the current short video content and overall comments, as well as other multi-source video content and comment data similar to the current short video, to automatically correct errors in the comments of the current short video. This can improve the accuracy and efficiency of error correction in comments.

[0185] In one possible implementation, the detection unit 602 is specifically used to input the target comment into the language model to obtain the likelihood of the first word in the target comment;

[0186] The detection unit 602 is further configured to use the first word as the word to be corrected when the likelihood of the first word is less than the first threshold.

[0187] This application provides a specific implementation method for detecting words to be corrected in a target comment. By inputting the target comment into the aforementioned language model, the likelihood of each word in the target comment can be obtained, such as the likelihood of the first word in the target comment. If the likelihood of the first word is less than a first threshold, the first word is identified as a word to be corrected in the target comment. The first threshold is not a fixed value and can vary depending on different application scenarios. This application embodiment can improve the accuracy and efficiency of identifying words to be corrected in comments.

[0188] In one possible implementation, the language model includes a first language model and a second language model, and the information of the candidate video includes the video content text of the candidate video and the comment content text of the candidate video; the first language model is trained based on the video content text of the candidate video, and the second language model is trained based on the comment content text of the candidate video.

[0189] The detection unit 602 is specifically used to input the target comment into the first language model to obtain the first probability of the first word;

[0190] The detection unit 602 is further configured to input the target comment into the second language model to obtain the second probability of the first word;

[0191] The detection unit 602 is further configured to use the weighted sum of the first probability and the second probability as the likelihood of the first word.

[0192] This application provides a specific implementation method for detecting phrase likelihood. Specifically, the language model includes a first language model and a second language model. The candidate video information includes video content text of candidate videos with high similarity to the target video and comment content text of candidate videos. The first language model can be trained based on the video content text of the candidate videos, and the second language model can be trained based on the comment content text of the candidate videos. Inputting the target comment into the first and second language models yields the likelihood of each word in the target comment. For example, inputting the target comment into the first language model yields a first probability of the first word in the target comment, which represents the probability that the first language model identifies the first word as a word to be corrected in the target comment. Inputting the target comment into the second language model yields a second probability of the first word in the target comment, which represents the probability that the second language model identifies the first word as a word to be corrected in the target comment. The weighted sum of the first and second probabilities is then used as the likelihood of the first word. This application embodiment improves the detection accuracy of phrase likelihood in the target comment, thereby improving the accuracy and efficiency of identifying words to be corrected in the comment.

[0193] In one possible implementation, the acquisition unit 601 is specifically used to acquire a second word in the lexicon that has a high similarity to the word to be corrected; the lexicon is obtained based on the information of the candidate video;

[0194] The detection unit 602 is specifically used to input the target comment after the word to be corrected is replaced by the second word into the language model to obtain the likelihood of the second word;

[0195] The detection unit 602 is further configured to use the second word as the target word when the likelihood of the second word is greater than the second threshold.

[0196] This application provides a specific implementation method for obtaining the target word corresponding to the word to be corrected. Specifically, a lexicon can be obtained based on the information from the candidate videos. A second word with a high similarity to the word to be corrected is obtained from this lexicon. It can be understood that any similarity in the lexicon that is higher than the lowest similarity can be considered a high similarity. Alternatively, it can be understood that any similarity greater than a certain threshold, such as greater than the average, can be considered a high similarity. This threshold is not a fixed value and can vary depending on the application scenario. After obtaining the second word from the lexicon, the target comment after the word to be corrected has been replaced by the second word is input into a language model to obtain the likelihood of the second word. If the likelihood of the second word is greater than the second threshold, the second word is used as the target word to replace the word to be corrected in the target comment. The second threshold is not a fixed value and can vary depending on the application scenario. This application embodiment can improve the accuracy and efficiency of obtaining the target word corresponding to the word to be corrected, thereby improving the accuracy and efficiency of error correction in comments.

[0197] In one possible implementation, the detection unit 602 is specifically used to identify words in the dictionary whose weighted similarity to the word to be corrected in terms of pinyin and stroke similarity is greater than a third threshold, as the second word.

[0198] This application provides a specific implementation method for obtaining the similarity between a second word in a dictionary and a word to be corrected. Specifically, words in the dictionary whose weighted value of pinyin similarity and stroke similarity with the word to be corrected is greater than a third threshold are selected as the second word. The weights can be adjusted to select only words with high pinyin similarity, or only words with high stroke similarity, or a combination of both. The third threshold is not a fixed value and can vary depending on the application scenario. This application improves the accuracy of similarity detection between the second word and the word to be corrected in the dictionary, thereby increasing the accuracy and efficiency of obtaining the target word corresponding to the word to be corrected, and improving the accuracy and efficiency of error correction in comments.

[0199] In one possible implementation, the detection unit 602 is further configured to use words in the lexicon whose similarity to the word to be corrected in pinyin or strokes is greater than a third threshold as the second word.

[0200] In one possible implementation, the candidate video is a video in the video library that has a similarity to the target video greater than a fourth threshold.

[0201] In this embodiment, the candidate video information includes the text content of candidate videos and the text comment content of candidate videos that have a high similarity to the target video. It can be understood that candidate videos in the video library whose text or image content has a high similarity to the target video are considered to have a high similarity, as long as it is higher than the minimum similarity. This can also be understood as a similarity exceeding a certain threshold, such as above the average. Specifically, videos in the video library whose text or image content has a similarity to the target video greater than a fourth threshold are used as candidate videos to train the language model required for text recognition and error correction. The fourth threshold is not a fixed value and can vary depending on the application scenario. Through this embodiment, the obtained candidate videos can be made more similar to the target video, thereby improving the accuracy and efficiency of the language model trained based on the candidate video information in recognizing and correcting words in the target video.

[0202] In one possible implementation, the language model includes a first language model and a second language model, and the information of the candidate video includes the video content text of the candidate video and the comment content text of the candidate video; the first language model is trained based on the video content text of the candidate video, and the second language model is trained based on the comment content text of the candidate video.

[0203] The detection unit 602 is specifically used to input the target comment after the word to be corrected is replaced by the second word into the first language model to obtain the third probability of the second word;

[0204] The detection unit 602 is further configured to input the target comment after the word to be corrected is replaced by the second word into the second language model to obtain the fourth probability of the second word;

[0205] The detection unit 602 is further configured to use the weighted sum of the third probability and the fourth probability, or the weighted sum of the third probability, the fourth probability and the similarity as the likelihood of the second word.

[0206] This application provides a specific implementation method for detecting phrase likelihood. Specifically, the language model includes a first language model and a second language model. The candidate video information includes video content text of candidate videos with high similarity to the target video and comment content text of candidate videos. The first language model can be trained based on the video content text of the candidate videos, and the second language model can be trained based on the comment content text of the candidate videos. The target comment after the word to be corrected is replaced by the second word is input into the first and second language models to obtain the likelihood of the second word in the target comment. For example, inputting the target comment after the word to be corrected is replaced by the second word into the first language model yields a third probability of the second word in the target comment, which represents the probability that the first language model recognizes the second word may appear in the target comment. Inputting the target comment after the word to be corrected is replaced by the second word into the second language model yields a fourth probability of the second word in the target comment, which represents the probability that the second language model recognizes the second word may appear in the target comment. The weighted sum of the third and fourth probabilities is then used as the likelihood of the second word. Alternatively, the likelihood of the second word can be obtained by weighted summing the third probability, the fourth probability, and the similarity between the second word and the word to be corrected. Through the embodiments of this application, the detection accuracy of the likelihood of the second word in the lexicon can be improved, thereby increasing the accuracy and efficiency of obtaining the target word corresponding to the word to be corrected, and improving the accuracy and efficiency of error correction in comments.

[0207] According to the embodiments of this application, Figure 6 The various units in the illustrated device can be individually or entirely combined into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the network device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0208] It should be noted that the implementation of each unit can also refer to the above. Figure 2 , Figure 4 The corresponding description of the method embodiments shown.

[0209] exist Figure 6The text processing device 60 described herein can automatically identify and correct errors in the comments of the current short video by comprehensively utilizing the current short video content and overall comments, as well as other multi-source video content and comment data similar to the current short video. This can improve the error identification rate and correction accuracy in the comments, and increase the efficiency of identification and correction.

[0210] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device 70 provided in an embodiment of this application. The electronic device 70 may include a memory 701 and a processor 702. Optionally, it may also include a communication interface 703 and a bus 704, wherein the memory 701, processor 702, and communication interface 703 are interconnected via the bus 704. The communication interface 703 is used for data interaction with the aforementioned text processing device 60.

[0211] The memory 701 provides storage space, which can store data such as the operating system and computer programs. The memory 701 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM).

[0212] Processor 702 is a module that performs arithmetic and logical operations. It can be one or a combination of processing modules such as a central processing unit (CPU), a graphics processing unit (GPU), or a microprocessor unit (MPU).

[0213] The memory 701 stores a computer program, and the processor 702 calls the computer program stored in the memory 701 to execute the above-mentioned... Figure 2 , Figure 4 The text processing method shown:

[0214] Obtain the target comments for the target video; the target comments are user comments posted on the target video.

[0215] The target comment is detected as a word to be corrected; the word to be corrected is a word with low likelihood in the target comment, and the likelihood of the word to be corrected is obtained by inputting the word to be corrected into a language model, which is trained based on information from candidate videos, and the candidate videos include videos in the video library that have high similarity to the target video.

[0216] For details regarding the execution method of the processor 702 mentioned above, please refer to the above. Figure 2 , Figure 4 This will not be elaborated upon here.

[0217] Correspondingly, the processor 702 can call the computer program stored in the memory 701, and can also be used to execute the above-mentioned... Figure 6 The specific details of the method steps performed by each unit in the text processing device 60 shown can be found in the above description. Figure 6 This will not be elaborated upon here.

[0218] exist Figure 7 The described electronic device 70 can automatically identify and correct errors in the comments of the current short video by comprehensively utilizing the current short video content and overall comments, as well as other multi-source video content and comment data similar to the current short video. This can improve the error identification rate and correction accuracy in the comments, and increase the efficiency of identification and correction.

[0219] This application also provides a computer-readable storage medium storing a computer program. When the computer program is run on one or more processors, it can perform the above-mentioned tasks. Figure 2 , Figure 4 The method shown.

[0220] This application also provides a computer program product, which includes a computer program. When the computer program product runs on a processor, it can achieve the above-mentioned... Figure 2 , Figure 4 The method shown.

[0221] This application also provides a chip, which includes a processor for executing instructions. When the processor executes the instructions, it can achieve the above-mentioned... Figure 2 , Figure 4 The method shown. Optionally, the chip also includes a communication interface for inputting or outputting signals.

[0222] This application also provides a system that includes at least one text processing device 60, electronic device 70, or chip as described above.

[0223] In summary, by comprehensively utilizing the current short video content and overall comments, as well as other multi-source video content and comment data similar to the current short video, automatic error identification and correction of the comments in the current short video can improve the error identification rate and correction accuracy, thereby increasing the efficiency of identification and correction.

[0224] Furthermore, the embodiments of this application can also be applied to the blockchain field. The following will introduce some blockchain knowledge related to the text processing method in the embodiments of this application.

[0225] See Figure 8a The data sharing system shown, data sharing system 100, refers to a system for data sharing between nodes. This data sharing system may include multiple nodes 101, which can refer to various clients within the data sharing system. Each node 101, during normal operation, can receive input information and maintain shared data within the data sharing system based on the received input information. To ensure information interoperability within the data sharing system, information connections can exist between each node, allowing information transmission between nodes. For example, when any node in the data sharing system receives input information, other nodes in the system obtain this input information according to a consensus algorithm and store it as data in the shared data, ensuring consistency of data stored on all nodes in the data sharing system.

[0226] In this embodiment of the application, the data sharing system is a text processing system.

[0227] Each node in the data sharing system has a corresponding node identifier, and each node can also store the node identifiers of other nodes in the data sharing system. This allows for the subsequent broadcasting of generated blocks to other nodes in the data sharing system based on their node identifiers. Each node can maintain a node identifier list as shown in the table below, storing the node name and node identifier in this list. The node identifier can be an IP (Internet Protocol) address or any other information that can be used to identify the node; the table below only uses IP addresses as an example.

[0228] Node name Node identifier

[0229] Node 1 117.114.151.174

[0230] Node 2 117.116.189.145

[0231] ...

[0232] Node N 119.123.789.258

[0233] The functions of each node in the data sharing system include:

[0234] 1) Routing: A basic function of nodes used to support communication between nodes.

[0235] In addition to routing capabilities, nodes can also have the following functions:

[0236] 2) Applications are deployed in the blockchain to implement specific business needs. They record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system. When other nodes successfully verify the source and integrity of the record data, they add the record data to a temporary block.

[0237] For example, the business logic implemented by the application includes:

[0238] 2.1) A wallet is used to provide the function of conducting electronic currency transactions, including initiating transactions (i.e., sending the transaction record of the current transaction to other nodes in the blockchain system; after other nodes successfully verify the transaction, they store the transaction record data in the temporary block of the blockchain as a response to acknowledge the validity of the transaction; of course, the wallet also supports querying the remaining electronic currency in the electronic currency address;

[0239] 2.2) Shared ledger, used to provide functions such as storage, query and modification of ledger data. It sends the record data of the operation on the ledger data to other nodes in the blockchain system. After the other nodes verify the validity, as a response to acknowledge the validity of the ledger data, they store the record data in a temporary block. They can also send confirmation to the node that initiated the operation.

[0240] 2.3) Smart contracts are computerized protocols that can execute the terms of a contract. They are implemented through code deployed on a shared ledger that executes when certain conditions are met. Based on actual business needs, the code is used to complete automated transactions, such as querying the logistics status of goods purchased by a buyer and transferring the buyer's electronic money to the merchant's address after the buyer signs for the goods. Of course, smart contracts are not limited to executing contracts for transactions; they can also execute contracts for processing received information.

[0241] 3) A blockchain consists of a series of blocks that are sequentially generated. Once a new block is added to the blockchain, it will not be removed. The blocks contain the data submitted by the nodes in the blockchain system.

[0242] Each node in the data-sharing system stores the same blockchain. A blockchain consists of multiple blocks; see [link to blockchain documentation]. Figure 8b A blockchain consists of multiple blocks. The genesis block includes a block header and a block body. The block header stores input information feature values, version number, timestamp, and difficulty value, while the block body stores the input information. The next block after the genesis block takes the genesis block as its parent block. The next block also includes a block header and a block body. The block header stores the input information feature values ​​of the current block, the block header feature values ​​of the parent block, version number, timestamp, and difficulty value, and so on. This ensures that the block data stored in each block is related to the block data stored in the parent block, guaranteeing the security of the input information in the blocks.

[0243] When generating the individual blocks in the blockchain, see Figure 8c When a node in the blockchain receives input information, it verifies the input information. After verification, it stores the input information in a memory pool and updates its hash tree used to record the input information. Then, it updates the timestamp to the time the input information was received and tries different random numbers multiple times to calculate the feature value, ensuring that the calculated feature value satisfies the following formula:

[0244] SHA256(SHA256(version+prev_hash+merkle_root+ntime+nbits+x)) <TARGET

[0245] Wherein, SHA256 is the feature value algorithm used to calculate the feature value; version number (version) is the version information of the relevant block protocol in the blockchain; prev_hash is the block header feature value of the parent block of the current block; merkle_root is the feature value of the input information; ntime is the update time of the update timestamp; nbits is the current difficulty, which is a fixed value for a period of time and is determined again after exceeding the fixed time period; x is a random number; TARGET is the feature value threshold, which can be determined based on nbits.

[0246] Thus, when a random number satisfying the above formula is calculated, the information can be stored accordingly, generating a block header and a block body to obtain the current block. Subsequently, the node where the blockchain resides sends the newly generated block to other nodes in its data sharing system based on the node identifiers of other nodes in the data sharing system. The other nodes then verify the newly generated block and add it to their stored blockchain after verification.

[0247] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by hardware related to a computer program. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing computer program code, such as read-only memory (ROM) or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A text processing method, characterized in that, include: Get the target comments for the target video; The target comments are comments posted by users on the target video. The target comment is input into a language model to obtain the likelihood of each word in the target comment; based on the likelihood of each word, words to be corrected in the target comment are detected; the language model is trained based on information from similar videos, which include videos in a video library that have a high similarity to the target video; Based on the language model, the target word corresponding to the word to be corrected is obtained from the vocabulary generated from the information of the similar videos; Replace the word to be corrected in the target comment with the target word.

2. The method according to claim 1, characterized in that, The detection of words to be corrected in the target comment based on the likelihood of each word includes: If the likelihood of the first word among all the words is less than a first threshold, the first word is taken as the word to be corrected.

3. The method according to claim 2, characterized in that, The language model includes a first language model and a second language model, and the information of the similar videos includes the video content text of the similar videos and the comment content text of the similar videos. The first language model is trained based on the video content text of the similar videos, and the second language model is trained based on the comment content text of the similar videos. The step of inputting the target comment into a language model to obtain the likelihood of each word in the target comment includes: The target comment is input into the first language model to obtain the first probability of the first word; The target comment is input into the second language model to obtain the second probability of the first word; The weighted sum of the first probability and the second probability is used as the likelihood of the first word.

4. The method according to claim 1, characterized in that, The step of obtaining the target word corresponding to the word to be corrected from the vocabulary generated from the information of the similar videos based on the language model includes: Obtain a second word from the dictionary that is similar to the word to be corrected; The target comment after the word to be corrected is replaced by the second word is input into the language model to obtain the likelihood of the second word; If the likelihood of the second word is greater than the second threshold, the second word is taken as the target word, and the likelihood of the target word is greater than the likelihood of the word to be corrected.

5. The method according to claim 4, characterized in that, The step of obtaining a second word in the lexicon that is similar to the word to be corrected includes: The word in the dictionary whose weighted similarity to the word to be corrected in terms of pinyin and strokes is greater than the third threshold is selected as the second word.

6. The method according to claim 4, characterized in that, The step of obtaining a second word in the dictionary that is similar to the word to be corrected includes: Words in the dictionary whose similarity to the word to be corrected in pinyin or strokes is greater than a third threshold are used as the second word.

7. The method according to any one of claims 1 to 6, characterized in that, The similar video is a video in the video library whose similarity to the target video is greater than a fourth threshold.

8. A text processing device, characterized in that, include: The acquisition unit is used to acquire the target comments of the target video; The target comments are comments posted by users on the target video. The detection unit is used to input the target comment into a language model to obtain the likelihood of each word in the target comment; based on the likelihood of each word, it detects the words to be corrected in the target comment; the language model is trained based on information from similar videos, and the similar videos include videos in the video library that have a high similarity to the target video; Based on the language model, the target word corresponding to the word to be corrected is obtained from the vocabulary generated from the information of the similar videos; A correction unit is used to replace the word to be corrected in the target comment with the target word.

9. The apparatus according to claim 8, characterized in that, The detection unit is also used for: If the likelihood of the first word among all the words is less than a first threshold, the first word is taken as the word to be corrected.

10. The apparatus according to claim 9, characterized in that, The language model includes a first language model and a second language model. The information of the similar videos includes the video content text of the similar videos and the comment content text of the similar videos. The first language model is trained based on the video content text of the similar videos, and the second language model is trained based on the comment content text of the similar videos. The detection unit is further configured to input the target comment into the first language model to obtain a first probability of the first word; input the target comment into the second language model to obtain a second probability of the first word; and use the weighted sum of the first probability and the second probability as the likelihood of the first word.

11. The apparatus according to claim 8, characterized in that, The acquisition unit is further configured to acquire a second word in the lexicon that is similar to the word to be corrected; The detection unit is further configured to input the target comment after the word to be corrected is replaced by the second word into the language model to obtain the likelihood of the second word; If the likelihood of the second word is greater than the second threshold, the second word is taken as the target word, and the likelihood of the target word is greater than the likelihood of the word to be corrected.

12. The apparatus according to claim 11, characterized in that, The detection unit is also used for: The word in the dictionary whose weighted similarity to the word to be corrected in terms of pinyin and strokes is greater than the third threshold is selected as the second word.

13. The apparatus according to claim 11, characterized in that, The detection unit is also used for: Words in the dictionary whose similarity to the word to be corrected in pinyin or strokes is greater than a third threshold are used as the second word.

14. The apparatus according to any one of claims 8 to 13, characterized in that, The similar video is a video in the video library whose similarity to the target video is greater than a fourth threshold.

15. An electronic device, characterized in that, include: Processor and memory; The memory is used to store computer-executed instructions; The processor is configured to execute computer execution instructions stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, include: The computer-readable storage medium is used to store instructions or computer programs that, when executed, cause the method as described in any one of claims 1 to 7 to be implemented.

17. A computer program product, characterized in that, include: Instructions or computer programs; When the instructions or the computer program are executed, the method as described in any one of claims 1 to 7 is implemented.

Images