Content quality identification method, device, equipment, storage medium and program product

Abstract

The present disclosure provides a content quality identification method, device and equipment, storage medium and program product, and relates to the technical field of artificial intelligence. The content quality identification method in the present disclosure comprises: generating a predetermined number of simulated comments and comment positions corresponding to the simulated comments based on the to-be-identified content through a pre-trained content quality identification model; each simulated comment is a different comment associated with the to-be-identified content, the comment position is used to indicate whether the corresponding simulated comment supports the to-be-identified content, and the comment position comprises a support position and a non-support position; determining a quality score of the to-be-identified content based on a first comment number of the support position and a second comment number of the non-support position; determining a quality label of the to-be-identified content based on the quality score, and the quality label comprises high quality and low quality.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a content quality identification method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the development of computer technology, ordinary users can create content (such as video shooting and editing) and publish it on internet platforms. This content is usually called UGC (User-Generated Content). UGC is characterized by low barriers to entry, large quantity, and diverse styles. For the large amount of UGC uploaded to the platform, the platform can achieve efficient content management through quality identification. Taking video UGC as an example, identification is usually based on the quality of the video signal (such as clarity, color, pixel integrity, noise, artifacts, etc.), while ignoring factors such as the knowledge guidance or emotional resonance provided by the video content. This makes the quality identification results relatively one-sided and unable to accurately reflect the true value of the content.

[0003] In view of this, embodiments of this specification provide a content quality identification method, apparatus, device, storage medium, and program product, in order to identify content quality from the perspective of content quality and improve the accuracy of content quality identification. Summary of the Invention

[0004] This specification provides one or more embodiments of a content quality identification method, the method comprising: generating a predetermined number of simulated comments and corresponding comment stances based on the content to be identified using a pre-trained content quality identification model; each simulated comment being a different comment associated with the content to be identified, the comment stance indicating whether the corresponding simulated comment supports the content to be identified, and the comment stance including a supportive stance and a non-supportive stance; determining a quality score for the content to be identified based on a first number of supportive comments and a second number of non-supportive comments; and determining a quality label for the content to be identified based on the quality score, the quality label including high quality and low quality.

[0005] According to one or more embodiments of this specification, the method further includes: using unlabeled first content and first real comments associated with the first content as a first prompt, instructing a pre-trained teacher model to generate each first simulated comment associated with the first content, a first comment stance corresponding to each first simulated comment, a first quality score and a first quality label for the first content; the first quality score is determined based on the first comment stance, and the first quality label is determined based on the first quality score; constructing a first dataset based on the first content, first simulated comments, first comment stance, first quality score and first quality label; using labeled second content, second real comments associated with the second content, and human-generated quality labels for the second content as a second prompt, instructing the pre-trained teacher model to generate each second simulated comment associated with the second content, a second comment stance corresponding to each second simulated comment, a second quality score and a second quality label for the second content, and instructing the second quality label to be consistent with the human-generated quality label; the human-generated quality label is manually labeled, the second quality score is determined based on the second comment stance, and the second quality label is determined based on the second quality score; constructing a second dataset based on the second content, second simulated comments, second comment stance, second quality score and second quality label; and merging the first dataset and the second dataset to obtain machine-labeled data for training a content quality recognition model.

[0006] According to one or more embodiments of this specification, a method is provided that uses unlabeled first content and first real comments associated with the first content as a first prompt, instructing a pre-trained teacher model to generate each first simulated comment associated with the first content, a first comment stance corresponding to each first simulated comment, a first quality score and a first quality tag for the first content, including: using unlabeled first content and first real comments associated with the first content as a first prompt, instructing the pre-trained teacher model to select a portion of the first real comments from the initial first real comments according to preset screening conditions; based on the first content and the portion of the first real comments, instructing the pre-trained teacher model to generate each first simulated comment associated with the first content, a first comment stance corresponding to each first simulated comment, a first quality score and a first quality tag for the first content; and using labeled second content and second real comments associated with the second content as a first prompt. The second content's artificial quality label serves as a second prompt, instructing the pre-trained teacher model to generate each second simulated comment associated with the second content, a second comment stance corresponding to each second simulated comment, a second quality score for the second content, and a second quality label, and instructing the second quality label to be consistent with the artificial quality label. This includes: using the labeled second content, the second real comments associated with the second content, and the artificial quality label of the second content as a second prompt, instructing the pre-trained teacher model to select a portion of the second real comments from the initial second real comments according to preset screening conditions; based on the second content, the portion of the second real comments, and the artificial quality label, instructing the pre-trained teacher model to generate each second simulated comment associated with the second content, a second comment stance corresponding to each second simulated comment, a second quality score for the second content, and a second quality label, and instructing the second quality label to be consistent with the artificial quality label.

[0007] According to one or more embodiments of this specification, the preset filtering conditions include sorting the number of likes corresponding to real comments associated with each target content from high to low, and / or, the priority of real comments on content elements within the target content is higher than the priority of real comments on the overall theme of the target content; wherein, the target content includes first content and second content, and the real comments include first real comments and second real comments.

[0008] According to the method provided in one or more embodiments of this specification, the target content includes video content, and the content elements include at least one of cover image, keyframe, title, content tag, voice text, category, resolution, and playback format; the human quality tag includes high quality and low quality; the human quality tag is determined based on at least one of the creativity, emotional engagement, information value, narrative coherence, and originality of the second content when it is labeled.

[0009] According to one or more embodiments of this specification, the training process of the content quality recognition model includes: obtaining a first content quality recognition model through supervised fine-tuning training based on machine-annotated data, such that the first content quality recognition model can output simulated comments, comment stances, quality scores, and quality tags for the target content; the target content includes the first content and the second content; training a second content quality recognition model by optimizing the first content quality recognition model through reinforcement learning based on manually annotated data, wherein the second content quality recognition model outputs simulated comments with better diversity, similarity between simulated comments and real comments, and accuracy of quality tags than the first content quality recognition model; the manually annotated data includes at least the second content and manually annotated quality tags; the real comments include the first real comments and the second real comments; and using the trained second content quality recognition model as a pre-trained content quality recognition model.

[0010] According to one or more embodiments of this specification, supervised fine-tuning constructs a first loss function based on the difference between the output of a first content quality recognition model and machine-labeled data. The training process of supervised fine-tuning causes the loss value of the first loss function to gradually decrease until convergence.

[0011] According to one or more embodiments of this specification, reinforcement learning determines a reward score based on the output of a second content quality recognition model, and constructs a second loss function based on the reward score. The training process of reinforcement learning causes the reward score to gradually increase until it reaches a stable state where it no longer increases. The reward score includes at least one of format reward score, label reward score, diversity reward score, and social reward score. The format reward score is determined based on whether the output of the second content quality recognition model meets the required format. The label reward score is determined based on whether the quality label output by the second content quality recognition model is consistent with the human quality label. The diversity reward score is determined based on the degree of repetition of the simulated comments output by the second content quality recognition model, and the diversity reward score increases as the degree of repetition decreases. The social reward score is determined based on the degree of similarity between the simulated comments output by the second content quality recognition model and real comments, and the social reward score increases as the degree of similarity increases.

[0012] According to one or more embodiments of this specification, a method for determining a quality score of content to be identified based on a first number of comments supporting a position and a second number of comments not supporting a position includes: determining the difference between the first number of comments and the second number of comments; obtaining a quality score of the content to be identified by dividing the difference by the square root of the sum of the first number of comments and the second number of comments; and determining a quality label of the content to be identified based on the quality score, including: if the quality score is greater than or equal to a quality score threshold, then determining that the content to be identified is of high quality; otherwise, determining that the content to be identified is of low quality.

[0013] One or more embodiments of this specification also provide a content quality identification device, the device comprising: a comment generation module, configured to generate a predetermined number of simulated comments and corresponding comment stances for the simulated comments based on the content to be identified using a pre-trained content quality identification model; each simulated comment is a different comment associated with the content to be identified, and the comment stance is used to indicate whether the corresponding simulated comment supports the content to be identified, the comment stance including a supportive stance and a non-supportive stance; a rating determination module, configured to determine a quality rating for the content to be identified based on a first number of comments supporting the stance and a second number of comments not supporting the stance; and a quality identification module, configured to determine a quality label for the content to be identified based on the quality rating, the quality label including high quality and low quality.

[0014] One or more embodiments of this specification also provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement the content quality identification method described in some embodiments of this specification.

[0015] One or more embodiments of this specification also provide a computer-readable storage medium storing computer instructions that, when executed by a processor, can implement the content quality identification method described in some embodiments of this specification.

[0016] One or more embodiments of this specification also provide a computer program product, including a computer program that, when at least a portion of the computer program is executed by a processor, can implement the content quality identification method described in some embodiments of this specification.

[0017] The effective effects that the embodiments of this specification may bring include, but are not limited to: generating simulated comments based on the content to be identified, the comment stance corresponding to the simulated comments, and determining the quality score of the content to be identified based on the number of first comments supporting the stance and the number of second comments not supporting the stance, and determining the quality label based on the quality score, by simulating the feelings of different viewers when viewing the content to be identified through the model, inferring the comments that specific people might make on the overall or specific content elements of the content to be identified, thereby avoiding the identification of content quality solely from the perspective of video signal, and determining the quality score based on the number of comments supporting and not supporting the stance of the simulated comments, and determining the quality label based on the quality score, thus providing a standardized and quantifiable quality identification method, which not only considers the quality of the content during the identification process, but also uses a unified identification method for each piece of content to be identified, so that the final quality label can objectively and accurately reflect the true value of the content.

[0018] By using a large amount of unlabeled first content and first real comments as the first cue, the teacher model is instructed to generate a first simulated comment, a first comment stance, a first quality score, and a first quality label. Conversely, by using labeled second content, second real comments, and human-generated quality labels as the second cue, the teacher model is instructed to generate a second simulated comment, a second comment stance, a second quality score, and a second quality label. Furthermore, the generated second quality label is required to be consistent with the human-generated quality label. On the one hand, labeling the large amount of unlabeled first content by the model improves the efficiency of data labeling. On the other hand, guiding the model to generate process data (second simulated comment, second comment stance, second quality score) consistent with the human-generated quality label through a small amount of human-generated labeled data improves the accuracy of data labeling. For the final machine-labeled data, this approach balances the advantages of efficiency and accuracy, controls the cost of data labeling, and yields a large amount of training data generated by the model for training the content quality recognition model.

[0019] By using a teacher model to select a portion of the first set of real comments from the first set of real comments and a portion of the second set of real comments from the second set of real comments according to preset screening criteria, the reliability of the real comments used as prompts can be improved, some low-value comments can be filtered out, and interference with the generation of machine-labeled data can be avoided.

[0020] Authentic comments are filtered by sorting them from highest to lowest based on the number of likes they receive. Since likes reflect the degree of resonance evoked by authentic comments, filtering authentic comments based on likes is efficient. Furthermore, the filtered authentic comments reflect the majority of viewers' evaluations of the target content, making them more valuable for reference. By prioritizing authentic comments that target specific elements of the target content over those that target the overall theme, more detailed and diverse comments from multiple perspectives can be selected, further enhancing their relevance.

[0021] The first content quality recognition model is obtained by supervised fine-tuning based on machine-labeled data. Since the machine-labeled data contains the model-generated first simulated comment, first comment stance, first quality score, and first quality label corresponding to the first content, as well as the second simulated comment, second comment stance, second quality score, and second quality label corresponding to the second content, it can provide a large amount of training data for the content quality recognition model. Therefore, based on a large amount of machine-labeled data, the trained first content quality recognition model can be made capable of outputting simulated comments, comment stance, quality scores, and quality labels corresponding to the target content through supervised training. Furthermore, since the machine-labeled data contains some data consistent with human quality labels, the accuracy of the training data provided is higher, which can also ensure the accuracy of the trained first content quality recognition model.

[0022] By optimizing the first content quality identification model based on manually labeled data using reinforcement learning, a second content quality identification model is obtained. Since the first content quality identification model is trained on a large amount of machine-labeled data generated by the teacher model, the accuracy of the first dataset generated by the teacher model may be lower than that of the second dataset for unlabeled first content. Therefore, by further optimizing the first content quality identification model based on accurately labeled manually labeled data using reinforcement learning, the second content quality identification model can improve performance in terms of diversity, accuracy, and similarity to real comments.

[0023] A first loss function is constructed based on the difference between the output of the first content quality recognition model and the machine-annotated data. If the loss value decreases to convergence, it indicates that the difference between the output of the first content quality recognition model and the machine-annotated data is very small. This indicates that the first content quality recognition model can output output results similar to the machine-annotated data. Therefore, when the loss value of the first loss function decreases to convergence, the first content quality recognition model in the supervised fine-tuning stage can be regarded as having completed training.

[0024] The reward score is determined based on the output of the second content quality recognition model, and a second loss function is constructed based on the reward score. Since the reward score is determined based on the format reward score, tag reward score, diversity reward score, and social reward score, if the reward score increases to a stable state where it no longer increases, it indicates that the second content quality recognition model has reached a very good state in terms of output format, accuracy of quality tags, diversity of simulated comments, and similarity between simulated comments and real comments. Therefore, when the reward score increases to a stable state where it no longer increases, the second content quality recognition model in the reinforcement learning phase can be regarded as having completed training.

[0025] By calculating the quality score based on the number of first comments supporting a position and the number of second comments not supporting a position, the number of different comment positions for different content can be normalized, thereby obtaining a standardized quality score and achieving the division of quality labels under the same standard.

[0026] It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects may be any one or a combination of the above, or any other possible beneficial effects. Attached Figure Description

[0027] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. The same numbers in the drawings denote the same structures or steps.

[0028] Figure 1 This is a schematic diagram illustrating an application scenario for content quality identification based on some embodiments of this specification.

[0029] Figure 2 This is an exemplary flowchart of a content quality identification method according to some embodiments of this specification.

[0030] Figure 3 This is an exemplary flowchart illustrating a method for determining a quality score according to some embodiments of this specification.

[0031] Figure 4 This is an exemplary flowchart illustrating a method for constructing machine-labeled data according to some embodiments of this specification.

[0032] Figure 5 This is an exemplary flowchart illustrating one method of constructing a first dataset according to some embodiments of this specification.

[0033] Figure 6 This is an exemplary flowchart illustrating one method of constructing a second dataset according to some embodiments of this specification.

[0034] Figure 7 This is an exemplary flowchart illustrating a training content quality recognition model according to some embodiments of this specification.

[0035] Figure 8 This is an exemplary block diagram of a content quality recognition model training framework according to some embodiments of this specification.

[0036] Figure 9 This is an exemplary block diagram of a content quality identification device according to some embodiments of this specification. Detailed Implementation

[0037] To more clearly illustrate the technical solutions of the embodiments in this specification, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the content described below are some examples or embodiments of this specification. For those skilled in the art, without creative effort, the technical solutions or means disclosed in this specification can be applied to other scenarios based on this technical content.

[0038] It should be understood that the terms "system," "device," "unit," and / or "module" used in this specification are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0039] Unless otherwise specified, the technical terms used to describe components, elements, etc. in this specification are not singular but may include plural. Generally speaking, terms such as "comprising" or "including" only indicate that explicitly identified steps, elements, or components are included, and these steps, elements, and components do not constitute an exclusive list, as the described method or apparatus may also include other steps or components.

[0040] This specification uses flowcharts to illustrate the operational steps performed by the apparatus or system of related embodiments. However, unless otherwise specified, the order in which these steps are described should not be construed as a limitation on the order of execution. Those skilled in the art can adjust the order of these steps based on the knowledge and information conveyed by the embodiments in this specification. Such adjustments include, but are not limited to, reversing the order of steps, merging multiple steps, and splitting a step.

[0041] With the development of computer technology, ordinary users can create content (such as video shooting and editing) and publish it on internet platforms. This content is usually called UGC (User-Generated Content). UGC is characterized by low barriers to entry, large quantity, and diverse styles. For the large amount of UGC uploaded to the platform, the platform can achieve efficient content management through quality identification. Taking video UGC as an example, identification is usually based on the quality of the video signal (such as clarity, color, pixel integrity, noise, artifacts, etc.), while ignoring factors such as the knowledge guidance or emotional resonance provided by the video content. This makes the quality identification results relatively one-sided and unable to accurately reflect the true value of the content.

[0042] Therefore, some embodiments of this specification propose a content quality identification method, apparatus, device, storage medium, and program product to improve the accuracy of content quality identification from a content quality perspective. It is understood that the specific implementations of this specification involve the collection, use, or processing of data (e.g., content to be identified, first content, second content, first genuine comment, second genuine comment, manual quality labels, etc.). When one or more embodiments of this specification are applied to specific products or technical implementations, permission or consent from the data subject is required. Furthermore, the collection, use, or processing of related data must strictly comply with the relevant laws, regulations, and standards of the data source country, implementation country, and other relevant countries and regions. De-identification technology ensures that the final data used is securely processed de-identified data, protecting the rights and interests of the data subject and data security.

[0043] Figure 1 This is a schematic diagram illustrating an application scenario for content quality identification based on some embodiments of this specification. In some embodiments, such as Figure 1 As shown, application scenario 100 may include client 110, server 120, and network 130. Client 110 and server 120 can transmit data through network 130. Client 110 can provide an upload interface and upload portal for training sample data (e.g., machine-annotated data, manually annotated data, etc.) and / or content to be identified, and send the user-uploaded training sample data and / or content to be identified to server 120. Server 120 can train a content quality recognition model based on the training sample data, and can also use the trained content quality recognition model to perform quality recognition on the content to be identified.

[0044] In some embodiments, client 110 may provide an upload interface and upload entry for training sample data and / or content to be identified. In some embodiments, client 110 may include, but is not limited to, terminal devices such as desktop computers, smartphones, laptops, VR devices, tablets, smart TVs, and in-vehicle terminals. Client 110 may include a display screen and a processor. The display screen may be used to present a graphical user interface (GUI). For example, client 110 may present an upload interface and upload entry for training sample data and / or content to be identified through the GUI. In some embodiments, the display screen may be separate from the human-machine interface device. Users can operate on the GUI through the GUI, and the processor of client 110 may receive operation instructions generated by operations on the GUI through the GUI. For example, the processor may generate a content quality identification request based on the content to be identified uploaded by the user and receive the content quality identification result fed back by server 120. In some embodiments, the display screen may be a touch screen, through which user operation instructions input based on the GUI can be received. In some embodiments, client 110 may include a memory for storing uploaded training sample data, content to be identified, and received content quality identification results for the content to be identified.

[0045] Server 120 may be a high-performance computer device used to receive content quality identification requests sent by client 110. The content quality identification request may include content to be identified. Server 120 can generate simulated comments and corresponding comment stances (supportive or non-supportive) based on the content to be identified. A quality score is determined based on the number of simulated comments with different stances, and a quality label (high quality or low quality) is determined based on the quality score. In some embodiments, server 120 may include a local server or a cloud server. Depending on different service requirements, a local server or cloud server corresponding to that region may be deployed in one or more regions. In some embodiments, server 120 may include a backend processing server. The backend processing server can train a content quality identification model based on training sample data sent by client 110, and can also perform content quality identification on the content to be identified using the trained content quality identification model based on the content to be identified sent by client 110. In some embodiments, server 120 may be a single computer device or a computing cluster composed of multiple computer devices, thereby providing more powerful computing power and more efficient data processing, such as training a content quality identification model based on provided training sample data and quickly performing content quality identification on the content to be identified based on the trained content quality identification model.

[0046] Network 130 can be any form of wired or wireless network, or any combination thereof. As an example, network 130 can be one or more combinations of wired networks, fiber optic networks, telecommunications networks, internal networks, the Internet, local area networks (LANs), wide area networks (WANs), wireless local area networks (WLANs), metropolitan area networks (MANs), public switched telephone networks (PSTNs), and Bluetooth networks. Network 130 can have multiple access points, and clients 110 and servers 120 can access network 130 through these access points.

[0047] It should be noted that, Figure 1 The illustrated application scenario diagram for content quality recognition is merely an example. The application scenarios for content quality recognition described in the embodiments of this specification are intended to more clearly illustrate the technical solutions of the embodiments of this specification and do not constitute a limitation on the technical solutions provided in the embodiments of this specification. For example, Figure 1 The number of client 110 and server 120 is merely illustrative and not intended to limit the scope of patent protection of this application. Depending on the actual situation, there can be any number of client 110 and server 120. For example, client 110 and server 120 can be implemented as a single device, and the uploading of training sample data and / or the content to be identified, as well as the content quality identification process, can be completed on the same device. As will be apparent to those skilled in the art, with the development of data processing technology and the emergence of new business scenarios, the technical solutions provided in the embodiments of this specification are also applicable to similar technical problems.

[0048] In order to improve the accuracy of content quality identification from the perspective of content quality, this specification provides a content quality identification method. Figure 2 This is an exemplary flowchart of a content quality identification method according to some embodiments of this specification. Figure 2 The process 200 shown can be executed by a processing device, for example, by... Figure 1 The server 120 shown executes the process. In some embodiments, process 200 may be implemented by a content quality recognition device 900 deployed on a processing device. The content quality recognition device 900 may include a comment generation module 910, a rating determination module 920, and a quality recognition module 930. In some embodiments, such as Figure 2 As shown, process 200 may include the following steps.

[0049] Step 210: Using a pre-trained content quality recognition model, a predetermined number of simulated comments and corresponding comment stances are generated based on the content to be identified. Comment stances include supportive and non-supportive stances. Each simulated comment is a different comment associated with the content to be identified, and the comment stance indicates whether the corresponding simulated comment supports the content to be identified. Non-supportive stances can include opposing and neutral stances. In some embodiments, step 210 can be implemented by the comment generation module 910.

[0050] In some embodiments, the content to be identified refers to content that requires quality identification. In some embodiments, the content to be identified may refer to UGC uploaded by users to the platform that requires quality identification by the platform. In some embodiments, the content to be identified may include video content, image content, text content, etc. In some embodiments, the content to be identified may include multimodal content. In some embodiments, taking video content as an example, the content quality identification model may be based on one or more of the following: cover image, keyframes, title, content tags (topics associated with the video content, such as food reviews, travel guides, life skills, etc.), speech and text (e.g., Automatic Speech Recognition (ASR)), category (the video platform's classification of video content, such as fitness, animals, animation, cars, games, music, travel, parenting, crafts, etc.), resolution, and playback format (portrait or landscape).

[0051] In some embodiments, the pre-trained content quality recognition model is a trained model capable of generating simulated comments, comment stances, quality scores, and quality labels based on content. In some embodiments, the content quality recognition model can be trained based on labeled training sample data (i.e., a dataset consisting of sample content and its corresponding simulated comments, comment stances, quality scores, and quality labels), enabling it to output simulated comments, comment stances, quality scores, and quality labels based on content.

[0052] In some embodiments, simulated comments can refer to natural language statements automatically generated by the model that mimic the tone and stance of real users, used to represent the hypothetical audience's reaction to the content to be identified. In some embodiments, simulated comments can adopt a specific and uniform output sentence structure, such as "When a user sees *** of the content, they might post *** comments." In some embodiments, each simulated comment is a different comment associated with the content to be identified, and each simulated comment can express diverse comments on different content elements from different perspectives (simulating differentiated audiences with different personalities, professions, life experiences, etc.). In some embodiments, for a single piece of content to be identified, the content quality identification model can generate 15 to 20 simulated comments to balance the diversity of simulated comments with the computational resources required to generate them.

[0053] In some embodiments, the comment stance is used to indicate whether the corresponding simulated comment supports the content to be identified, and may include a supportive stance and a non-supportive stance. For example, for the same travel video, if one simulated comment is "The scenery in the video is so beautiful, I want to go too!", then this simulated comment corresponds to a supportive stance; if another simulated comment is "The editing is messy, I don't understand it.", then this simulated comment corresponds to a non-supportive stance.

[0054] In some embodiments, the quality score can refer to a normalized score obtained based on the statistical analysis of comment positions. This score can transform the number of simulated comments corresponding to discrete comment positions into a continuous, threshold-comparable value. It can be used to represent the degree of recognition of the content to be identified by the simulated comments, and thus reflect the quality of the content to be identified from the degree of recognition.

[0055] In some embodiments, quality labels are the result of quality ratings. In some embodiments, quality labels can be a binary classification result (high quality or low quality) for the content to be identified. A high-quality label indicates that the content to be identified can resonate with the community, while a low-quality label indicates that the content to be identified has failed to gain community approval. In some embodiments, the community can refer to the set of users who view, interact with, and comment on the same content.

[0056] Step 220: Determine a quality score for the content to be identified based on the number of first comments supporting the position and the number of second comments not supporting the position. In some embodiments, step 220 may be implemented by a score determination module 920.

[0057] In some embodiments, the comment stance can refer to the attitude of the simulated comments toward the quality of the identified content. A supportive stance can reflect the simulated comments’ appreciation, recognition, recommendation, emotional resonance, positive evaluation, etc., toward the identified content, while a non-supportive stance can reflect the simulated comments’ criticism, questioning, denial, negative feedback, etc., toward the identified content.

[0058] In some embodiments, counting the number of first comments supporting a position and the number of second comments not supporting a position can provide a discrete source of comment-position-related data for determining a quality score.

[0059] In some embodiments, the quality score may employ a Skellam score, i.e., the quality score follows a Skellam distribution, normalizing the difference between the number of first comments supporting the position and the number of second comments opposing the position, to measure the significance of the supporting position relative to the non-supporting position within the community.

[0060] Figure 3 This is an exemplary flowchart illustrating a method for determining a quality score according to some embodiments of this specification. Figure 3 The process 300 shown can be executed by a processing device, for example, by... Figure 1 The server 120 shown is executed. In some embodiments, process 300 may be a further description of step 220. In some embodiments, process 300 may be implemented by a scoring determination module 920 in a content quality recognition device 900 deployed on a processing device. In some embodiments, such as Figure 3 As shown, process 300 may include the following steps.

[0061] Step 310: Determine the difference between the number of first comments and the number of second comments.

[0062] Step 320: Divide the difference by the square root of the sum of the first and second comment counts to obtain the quality score of the content to be identified.

[0063] In some embodiments, assuming the quality score z follows a Skellam distribution, and assuming x is the number of first comments supporting the position and y is the number of second comments opposing the position, then the quality score... When the number of supporting positions significantly exceeds the number of non-supporting positions, the quality score z is a large positive number; when the number of supporting and non-supporting positions is roughly the same, the quality score z is close to 0; when the number of supporting positions is less than the number of non-supporting positions, the quality score z is a negative number.

[0064] Step 230: Determine the quality label of the content to be identified based on the quality score. The quality label includes high quality and low quality. In some embodiments, step 230 can be implemented by the quality identification module 930.

[0065] In some embodiments, taking a binary quality label as an example, the quality score can be compared with a quality score threshold to determine different types of quality labels.

[0066] In some embodiments, step 230 may further include: if the quality score is greater than or equal to the quality score threshold, then the content to be identified is determined to be of high quality; otherwise, the content to be identified is determined to be of low quality.

[0067] In some embodiments, combining the preceding quality score z, if the quality score z ≥ the quality score threshold, the quality label is high quality; if the quality score z < the quality score threshold, the quality label is low quality. In some embodiments, the quality score threshold can be set to 1.5 based on experience.

[0068] In some embodiments, in order to train a content quality identification model capable of identifying the content quality of newly uploaded or content to be identified, a training dataset needs to be constructed. The dataset can be constructed based on a large amount of unlabeled first content with a first real comment and a small amount of second content labeled with human quality tags and with a second real comment. Here, the first and second content constitute the target content in the training data. Figure 4 This is an exemplary flowchart illustrating a method for constructing machine-labeled data according to some embodiments of this specification. Figure 4 The process 400 shown can be executed by a processing device, for example, by... Figure 1 The server 120 shown is executed. In some embodiments, such as Figure 4 As shown, process 400 may include the following steps.

[0069] Step 410: Using the unlabeled first content and the first real comment associated with the first content as the first prompt, instruct the pre-trained teacher model to generate each first simulated comment associated with the first content, the first comment stance corresponding to each first simulated comment, the first quality score of the first content, and the first quality label. The first quality score is determined based on the first comment stance, and the first quality label is determined based on the first quality score.

[0070] In some embodiments, the pre-trained teacher model can refer to a pre-trained large neural network model with multimodal understanding and generation capabilities, used to provide supervisory signals to the student model to be trained (i.e., the first content quality recognition model trained in the supervised fine-tuning stage mentioned later). In some embodiments, the teacher model can output content that meets the prompt requirements based on the input prompts. Taking video content as an example, the video content and its corresponding real comments are used as prompts. The prompt requirements can be: "Based on the provided video content, determine the following elements: cover, keyframe, title, category, ASR, resolution, playback format. Combine the analyzed content elements and the provided real comments, imitate the comments that different audience roles may make on the video content as a whole or specific content elements, and finally output 15-20 simulated comments, indicating whether each simulated comment supports or opposes the video content. Combine the number of comments with different stances to calculate the quality score according to the Skellam scoring method. Compare the quality score with a threshold of 1.5. If it is greater than 1.5, output the quality label as high quality; otherwise, output the quality label as low quality."

[0071] In some embodiments, the first quality score may be determined by referring to the method for determining the quality score of the previously identified content, and the first quality label may be determined by referring to the method for determining the quality label of the previously identified content.

[0072] In some embodiments, a large amount of unlabeled first content can obtain supervisory signals (first simulated comment, first comment stance, first quality score, first quality label) corresponding to the first content through the teacher model. Since the first comment stance is determined based on the generated first simulated comment, the first quality score is determined based on the number of comments with different first comment stances, and the first quality label is determined based on the first quality score, the first quality label can be regarded as a "pseudo-label" of the first content. The "pseudo-label" may not be completely consistent with the real label of the first content, but it can greatly improve the labeling efficiency of the first content.

[0073] Step 420: Construct the first dataset based on the first content, the first simulated comment, the first comment stance, the first quality score, and the first quality label.

[0074] In some embodiments, the first content serves as a training sample, and the first simulated comment, the first comment stance, the first quality score, and the first quality label can serve as supervisory signals corresponding to the training samples. The first dataset constructed together can serve as part of the training data for the content quality recognition model. The first dataset contains a large number of training samples and "pseudo-supervisory signals" generated by the teacher model, which can meet the sample data requirements during the training of the content quality recognition model.

[0075] Step 430: Using the labeled second content, the second real comments associated with the second content, and the human-generated quality label of the second content as second prompts, the pre-trained teacher model is instructed to generate each second simulated comment associated with the second content, a second comment stance corresponding to each second simulated comment, a second quality score and a second quality label for the second content, and to ensure that the second quality label is consistent with the human-generated quality label. The human-generated quality label is manually labeled, the second quality score is determined based on the second comment stance, and the second quality label is determined based on the second quality score.

[0076] In some embodiments, to improve the accuracy of the supervision signal, a small amount of labeled second content, second real comments associated with the second content, and artificial quality labels of the second content can be used as second prompts input into the teacher model. The prompt requirements for the second prompt are similar to those for the first prompt. In particular, since the second content has been labeled with accurate artificial quality labels, the prompt requirements for the second prompt include "the quality labels derived from the generated simulated comments, comment stances, and quality scores must be consistent with the input artificial quality labels," thereby constraining the process data (second simulated comments, second comment stances, and second quality scores) generated by the teacher model based on the second content.

[0077] In some embodiments, the second content and the artificial quality label constitute artificially labeled data. The artificially labeled data can be identified by professional content recognition experts based on specific dimensions to determine the artificial quality label. When labeling the artificially labeled data, only the quality label can be labeled. The process data such as the second simulated comment, the second comment stance, and the second quality score corresponding to the second content can be supplemented by the teacher model after generation.

[0078] In some embodiments, human-generated quality labels may include high quality and low quality. In some embodiments, human-generated quality labels may be determined based on at least one dimension of the second content: creativity, emotional engagement, informational value, narrative coherence, and originality.

[0079] Step 440: Construct a second dataset based on the second content, the second simulated comments, the second comment stance, the second quality score, and the second quality label.

[0080] In some embodiments, the second content serves as a training sample, while the second simulated comment, second comment stance, second quality score, and second quality label can serve as supervisory signals corresponding to the training samples. The resulting second dataset can be used as another part of the training data for the content quality recognition model. Considering the cost of manual annotation, although the second dataset contains fewer training samples, the second quality label is consistent with the manually labeled quality, which can meet the sample accuracy requirements during the training of the content quality recognition model.

[0081] Step 450: Based on the merging of the first and second datasets, machine-labeled data is obtained for training the content quality recognition model.

[0082] In some embodiments, machine-annotated data, consisting of a first dataset obtained by the teacher model based on unannotated content and a second dataset obtained based on a small amount of manually annotated content, can not only meet the sample data volume requirements during the training of the content quality recognition model, but also take into account the requirements for sample data accuracy, thus balancing annotation efficiency and annotation cost.

[0083] In some embodiments, in order to further optimize the first dataset, the first real comments may be initially filtered. Figure 5 This is an exemplary flowchart illustrating one method of constructing a first dataset according to some embodiments of this specification. Figure 5 The process 500 shown can be executed by a processing device, for example, by... Figure 1 The server 120 shown is executed. In some embodiments, process 500 may be a further description of step 410. In some embodiments, such as Figure 5 As shown, process 500 may include the following steps.

[0084] Step 510: The unlabeled first content and the first real comments related to the first content are used as the first prompts to instruct the pre-trained teacher model to select some first real comments from the initial first real comments according to the preset screening conditions.

[0085] In some embodiments, the teacher model can also filter out a portion of valuable real comments from the provided initial real comments according to preset filtering criteria, thereby filtering out a large number of irrelevant or low-value real comments. In some embodiments, the preset filtering criteria can also be input into the teacher model as prompts.

[0086] In some embodiments, the preset filtering criteria include sorting the real comments associated with each target content by the number of likes, from highest to lowest. In some embodiments, when users view comments, they typically support those that they agree with, appreciate, or that resonate with them by liking them. Therefore, the number of likes can reflect the degree of resonance evoked by the real comments. Sorting the real comments by the number of likes, prioritizing the comments at the top of the list, can efficiently filter real comments, and the filtered real comments can reflect the evaluation of most users on the target content, making them more valuable for reference. In some embodiments, for example, the top 15 to 20 real comments can be filtered from 50 real comments. The number of real comments to be filtered can be determined by considering both computing resources and the initial number of real comments.

[0087] In some embodiments, the target content includes first content and second content. In some embodiments, genuine comments include first genuine comments (associated with the first content) and second genuine comments (associated with the second content). In some embodiments, the first genuine comments and the second genuine comments may use the exact same preset filtering conditions, or they may use different preset filtering conditions.

[0088] In some embodiments, preset filtering criteria include prioritizing genuine comments on content elements within the target content over genuine comments on the overall theme of the target content.

[0089] In some embodiments, genuine comments on content elements within the target content are typically more detailed and specific. Genuine comments on different content elements are usually based on different perspectives, resulting in significant differences in comment content. In contrast, genuine comments on the overall theme of the target content can be more generalized and contain more generalized information, making similar comments more likely. Therefore, genuine comments on content elements have higher reference value. Prioritizing the screening of genuine comments on content elements can not only filter out more detailed comments but also more diverse and multi-perspective comments.

[0090] In some embodiments, two preset filtering conditions, likes and comment subject, can be combined to filter real comments. For example, real comments on content elements and real comments on the overall theme can be sorted based on likes. According to the preset filtering quantity, the real comments on content elements are filtered first. If the quantity is insufficient, the remaining quantity is supplemented by sorting the real comments on the overall theme.

[0091] In some embodiments, the target content includes video content, and content elements include at least one of cover image, keyframe, title, content tags, audio / text, category, resolution, and playback format. In some embodiments, content elements may also include content elements reflecting video signal quality, such as clarity, color, and duration. If the simulated commentary comments on the content elements of the video signal, the impact of the video signal on the viewing experience can be appropriately considered in the subsequent determination of commentary stance, quality score, and quality tags, thereby comprehensively considering both content quality and signal quality.

[0092] Step 520: Based on the first content and some first real comments, instruct the pre-trained teacher model to generate each first simulated comment associated with the first content, a first comment stance corresponding to each first simulated comment, a first quality score for the first content, and a first quality label.

[0093] In some embodiments, after filtering a portion of the first real comments, the generation of the first simulated comment, the first comment stance, the first quality score, and the first quality tag is similar to the specific implementation in step 410 above, and will not be repeated here.

[0094] In some embodiments, the second set of real comments may be initially screened to further optimize the second dataset. Figure 6 This is an exemplary flowchart illustrating one method of constructing a second dataset according to some embodiments of this specification. Figure 6 The process 600 shown can be executed by a processing device, for example, by... Figure 1 The server 120 shown is executed. In some embodiments, process 600 may be a further description of step 430. In some embodiments, such as Figure 6 As shown, process 600 may include the following steps.

[0095] Step 610: The labeled second content, the second real comments associated with the second content, and the artificial quality labels of the second content are used as second prompts to instruct the pre-trained teacher model to select a portion of the second real comments from the initial second real comments according to preset screening conditions.

[0096] In some embodiments, the filtering of the second genuine comments is similar to that of the first genuine comments. For details regarding the preset filtering conditions, please refer to the description in step 510, which will not be repeated here. In some embodiments, the second genuine comments and the first genuine comments may use the same preset filtering conditions, or they may use different preset filtering conditions.

[0097] Step 620: Based on the second content, some second real comments, and human quality labels, instruct the pre-trained teacher model to generate each second simulated comment associated with the second content, a second comment stance corresponding to each second simulated comment, a second quality score and a second quality label for the second content, and instruct the second quality label to be consistent with the human quality label.

[0098] In some embodiments, after filtering a portion of the second real comments, the generation of the second simulated comments, the second comment stance, the second quality score, and the second quality label is similar to the specific implementation in step 430 above, and will not be repeated here.

[0099] In some embodiments, a content quality recognition model can be trained based on constructed machine-annotated data and manually annotated data. Figure 7 This is an exemplary flowchart illustrating a training content quality recognition model according to some embodiments of this specification. Figure 7 The process 700 shown can be executed by a processing device, for example, by... Figure 1 The server 120 shown is executed. In some embodiments, such as Figure 7 As shown, process 700 may include the following steps.

[0100] Step 710: Based on machine-labeled data, a first content quality recognition model is obtained through supervised fine-tuning training, enabling the first content quality recognition model to output simulated comments, comment stances, quality scores, and quality tags for the target content.

[0101] In some embodiments, the target content includes a first content and a second content.

[0102] In some embodiments, since the machine-annotated data includes a first dataset entirely generated by the teacher model (the first quality label is equivalent to a "pseudo-label," with lower reliability than purely manually annotated data) and a second dataset generated by the teacher model based on manually annotated data (the second quality label = manually annotated quality label, with high accuracy), the supervisory signal used in supervised training during the supervised fine-tuning phase combines both sample quantity and accuracy. This allows for the training of a first content quality recognition model that initially outputs simulated reviews, review stances, quality scores, and quality labels. In some embodiments, the second dataset included in the machine-annotated data contains a portion of the manually annotated data; therefore, during the supervised fine-tuning phase, manually annotated data can be used as training input along with machine-annotated data, or machine-annotated data can be used as training input alone.

[0103] In some embodiments, Supervised Fine-tuning (SFT) Tuning can use sample data with supervised signals to train model parameters in a supervised learning manner, enabling the model to complete the process of generating simulated comments, comment stances, quality scores, and quality labels based on content.

[0104] In some embodiments, supervised fine-tuning constructs a first loss function based on the difference between the output of the first content quality recognition model and the machine-annotated data. The training process of supervised fine-tuning causes the loss value of the first loss function to gradually decrease until it converges.

[0105] In some embodiments, for the same target content, the target content serves as sample data and has corresponding supervision signals (simulated comments, comment stances, quality scores, and quality labels generated by the teacher model). The first content quality recognition model, after training, can also output corresponding simulated comments, comment stances, quality scores, and quality labels based on the target content. The similarity between the output of the first content quality recognition model and the supervision signals is compared. The higher the similarity, the smaller the difference between the two. A first loss function is constructed based on the difference. Based on the loss value of the first loss function, the parameters of the first content quality recognition model can be adjusted through the error backpropagation algorithm, so that the difference between the output of the adjusted first content quality recognition model and the supervision signals gradually decreases. If the loss value decreases to convergence (a stable state where the loss value no longer decreases regardless of how the model parameters are adjusted), it indicates that the difference between the output of the first content quality recognition model and the machine-labeled data is very small. At this point, the first content quality recognition model can be considered to have completed training.

[0106] Step 720: Based on manually labeled data, the first content quality recognition model is optimized through reinforcement learning to train the second content quality recognition model. The second content quality recognition model outputs simulated comments with better diversity, similarity between simulated comments and real comments, and accuracy of quality labels than the first content quality recognition model.

[0107] In some embodiments, the training of the first content quality recognition model, having learned from a large dataset entirely labeled by a teacher model, still has room for improvement in accuracy and can be further optimized based on manually labeled data. In some embodiments, the manually labeled data includes at least the second content and manually labeled quality tags.

[0108] In some embodiments, reinforcement learning (RL) is a branch of machine learning that learns optimal policies through trial and error to maximize cumulative rewards. Unlike supervised learning, which has readily available correct answers (supervision signals), reinforcement learning does not have ready-made answers. It judges its own performance based on reward or penalty signals and continuously adjusts accordingly. In some embodiments, the reinforcement learning phase can be trained using a group-based relative policy optimization approach. For example, the first content quality recognition model obtained during the supervised fine-tuning phase can output multiple sets of results based on the second content, or the second content quality recognition model obtained during the reinforcement learning optimization phase can output multiple sets of results based on the second content. The relative advantage of each set of results is calculated (positive reward for results better than the group average, negative reward for results worse than the group average). The model is updated based on this relative advantage, allowing it to learn more good results and fewer bad results, thereby continuously improving its output performance.

[0109] In some embodiments, using more accurate manually labeled data during the reinforcement learning phase allows for comparison with accurate answers (manually labeled data) when verifying the output of the second content quality recognition model during the optimization process, thereby improving the reliability of reward score calculation.

[0110] In some embodiments, reinforcement learning determines a reward score based on the output of a second content quality recognition model during the optimization process, and constructs a second loss function based on the reward score. The training process of reinforcement learning causes the reward score to gradually increase until it reaches a stable state where it no longer increases.

[0111] In some embodiments, the reward score is an evaluation value used to assess the quality of the output of the second content quality recognition model during the reinforcement learning optimization process. The training objective of the reinforcement learning phase is to maximize the reward score, thereby enabling the second content quality recognition model to generate outputs that are more in line with expectations.

[0112] In some embodiments, the reward score may include at least one of format reward score, tag reward score, diversity reward score, and social reward score. In some embodiments, the reward score may be obtained by directly summing the format reward score, tag reward score, diversity reward score, and social reward score. In some embodiments, the reward score may also be obtained by weighted calculation based on the format reward score, tag reward score, diversity reward score, and social reward score. In some embodiments, exemplarily, the reward score... ,in, The format indicates the reward score. This indicates the tag reward score. Indicates the diversity reward score, This represents the social reward score.

[0113] In some embodiments, the format reward score can be determined based on whether the output of the second content quality recognition model conforms to the required format. The format reward score reflects whether the output format is standard, and the format of the output can be optimized based on the format reward score, so that the output of the second content quality recognition model has a uniform format. In some embodiments, the output of the second content quality recognition model can follow a structured format, such as... <think> …< / think> The format output reasoning process first analyzes the content elements, then simulates comments… The simulated comments can follow the format "When viewers see ***, they will post *** comments (comment stance analysis results)". Then, the number of different comment stances is counted, and the quality score is calculated according to a predetermined formula. Finally, the quality score and quality label are displayed. In some embodiments, if the format output by the second content quality recognition model meets the requirements, then… ,otherwise .

[0114] In some embodiments, the label reward score can be determined based on whether the quality label output by the second content quality recognition model is consistent with the human quality label. The label reward score can reflect the accuracy of the output result, and the accuracy of the quality label can be optimized based on the label reward score. In some embodiments, if the quality label output by the second content quality recognition model is consistent with the human quality label, then... ,otherwise .

[0115] In some embodiments, the diversity reward score can be determined based on the degree of repetition of the simulated comments output by the second content quality recognition model, and the diversity reward score increases as the degree of repetition decreases. The diversity reward score reflects the degree of repetition of the simulated comments, and based on the diversity reward score, the diversity of the simulated comments in the output can be optimized, enabling the simulated comments output by the second content quality recognition model to output different simulated comments from different perspectives, avoiding the output of duplicate simulated comments. In some embodiments, for example, Where C represents the set of generated simulated comments, and c represents one of the simulated comments. This indicates the frequency of identical or similar simulated comments. This represents the penalty coefficient (positive value). The higher the frequency of identical or similar simulated comments (i.e., the more repetitive the simulated comments), the higher the penalty coefficient becomes. The larger the calculation result, the more it is multiplied by Diversity reward score deducted later The more.

[0116] In some embodiments, the social reward score can be determined based on the similarity between simulated comments and real comments output by the second content quality recognition model, and the social reward score increases with the increase of similarity. The social reward score can reflect the similarity between simulated and real comments, and based on the social reward score, the authenticity of simulated comments in the output can be optimized, making the simulated comments output by the second content quality recognition model closer to real comments. In some embodiments, it is assumed that... Denotes the generated set of simulated comments, where This represents the i-th simulated comment in the set of simulated comments. This represents a collection of real comments, in which Let j be the j-th real comment in the set of real comments. Calculate the cosine similarity between the embeddings of the simulated comment and the real comment: ,in, , x This represents the comment text. f ( x This represents the result of mapping the comment text to a high-dimensional vector space. Represents the length (modulus) of a vector. This represents a normalized unit vector. The purpose is to convert each comment into a fixed-length, normalized numerical vector, so that semantically similar comment texts have a small angle in the vector space (cosine similarity value close to 1), while semantically unrelated comment texts have a large angle in the vector space (cosine similarity value close to 0). Represents the dot product of vectors. Let the cosine similarity between the i-th simulated comment and the j-th real comment be denoted as the cosine similarity between two normalized vectors. sum vector The dot product of vectors is equal to the cosine similarity, therefore This value ranges from -1 to 1, with values ​​closer to 1 indicating a greater semantic similarity between the two comments. A greedy matching method is used to find the closest real comment for each simulated comment (assuming there are 10 simulated comments and 10 real comments, the first simulated comment is selected from the 10 real comments, and its cosine similarity is calculated; the second simulated comment is selected from the remaining 9 real comments, and so on). The corresponding cosine similarity is calculated, and the final social reward score is the average of these cosine similarities. Where S represents the set of cosine similarities calculated earlier, and s represents the value of one of the cosine similarities. This represents the total number of simulated comments generated, and the social reward score. This is equal to the average cosine similarity of the best-matched generated simulated comments. The closer the generated simulated comments are to real comments overall, the higher the social reward score.

[0117] In some embodiments, since the reward score can be obtained based on one or more of the four aspects of format reward score, tag reward score, diversity reward score, and social reward score, the reward score can be a composite reward corresponding to multiple optimization directions. The second content quality recognition model optimized based on the reward score is superior to the first content quality recognition model in terms of output format standardization, accuracy of quality tags, diversity of simulated comments, and similarity between simulated comments and real comments.

[0118] In some embodiments, the manually labeled data may further include second real comments associated with the second content. In this case, the simulated comments output by the second content quality identification model can be compared with the corresponding second real comments of the second content, making the comparison more targeted and relevant. In some embodiments, real comments include first real comments and second real comments. In some embodiments, the real comments used for comparison with simulated comments can be either second real comments associated with the second content or first real comments associated with the first content. If first real comments are used, the comparison primarily examines whether the expression style of the simulated comments closely resembles the expression habits of real comments.

[0119] Step 730: Use the trained second content quality recognition model as a pre-trained content quality recognition model.

[0120] In some embodiments, the optimized second content quality recognition model serves as a pre-trained content quality recognition model, which can be used for subsequent content quality recognition of the content to be recognized. It can efficiently and quickly output corresponding simulated comments, comment stances, quality scores, and quality tags based on the input content to be recognized, and can objectively perform quality recognition based on the content.

[0121] Figure 8 This is an exemplary block diagram of a content quality recognition model training framework according to some embodiments of this specification. Figure 8As shown, Framework 800 includes a data construction section and a model training section. In the data construction section, a large amount of unlabeled first content and corresponding first real comments are input into the teacher model. The teacher model generates a first simulated comment, a first comment stance, a first quality score, and a first quality label. A small amount of labeled, manually annotated data (containing second content and human quality labels) and second real comments are also input into the teacher model. The teacher model generates a second simulated comment, a second comment stance, a second quality score, and a second quality label (the output second quality label is required to be consistent with the human quality label). Finally, the machine-labeled data is constructed based on the data results from both outputs. The model training section includes a supervised fine-tuning stage and a reinforcement learning stage. First, in the supervised fine-tuning phase, the target content, including the first and second content, along with the machine-annotated data obtained from the data construction phase, are used as training data to train the first content quality recognition model. Then, based on the first content quality recognition model, the reinforcement learning phase begins. In the reinforcement learning phase, manually annotated data is used for optimization. The first content quality recognition model is used to obtain the model output based on the first content. Then, the model output is validated from four aspects: format, label, diversity, and social, to obtain format reward scores, label reward scores, diversity reward scores, and social reward scores, respectively. Based on these four or several reward scores, the final reward score is obtained. Based on the final reward score, the first content quality recognition model is continuously optimized to obtain the second content quality recognition model. When the reward score corresponding to the model output of the second content quality recognition model increases to a stable state where it no longer increases, the second content quality recognition model is considered to have completed training and is used as a pre-trained content quality recognition model for subsequent content quality recognition.

[0122] This specification also provides a content quality identification device. Figure 9 This is an exemplary block diagram of a content quality identification device according to some embodiments of this specification. In some embodiments, the content quality identification device 900 may be deployed on server 120. Figure 8 As shown, in some embodiments, the content quality identification model 900 may include a comment generation module 910, a rating determination module 920, and a quality identification module 930.

[0123] In some embodiments, the comment generation module 910 can be used to generate a predetermined number of simulated comments and corresponding comment stances based on the content to be identified using a pre-trained content quality recognition model; each simulated comment is a different comment associated with the content to be identified, and the comment stance is used to indicate whether the corresponding simulated comment supports the content to be identified, and the comment stance includes a supportive stance and a non-supportive stance.

[0124] In some embodiments, the rating determination module 920 can be used to determine a quality rating for the content to be identified based on the number of first comments supporting a position and the number of second comments not supporting a position.

[0125] In some embodiments, the rating determination module 920 can also be used to determine the difference between the number of first comments and the number of second comments; and to obtain a quality score for the content to be identified by dividing the difference by the square root of the sum of the number of first comments and the number of second comments.

[0126] In some embodiments, the quality identification module 930 can be used to determine the quality label of the content to be identified based on the quality score, and the quality label includes high quality and low quality.

[0127] In some embodiments, the quality identification module 930 can also be used to determine that the content to be identified is of high quality if the quality score is greater than or equal to the quality score threshold, otherwise determine that the content to be identified is of low quality.

[0128] For more information on each module, please refer to [link / reference]. Figures 2 to 8 The relevant explanations will not be repeated here. It should be understood that... Figure 9 The modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated hardware. Those skilled in the art will understand that the methods and systems described above can be implemented using computer-executable instructions and / or included in the control code of a processor, such as on a media such as a disk, CD, or DVD-ROM, or in the memory of a programmable device. The systems and modules of this specification can be implemented not only with hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., but also with software, for example, executed by various types of processors, or with a combination of the aforementioned hardware circuits and software (e.g., firmware).

[0129] It should be noted that the above description of the system and its modules is for convenience only and should not be construed as limiting this specification to the embodiments described. It is understood that those skilled in the art, after understanding the principles of this system, may arbitrarily combine the various modules without departing from these principles to form subsystems connected to other modules. Alternatively, some modules may be split to obtain more modules or multiple units under a single module. Such modifications are all within the scope of this specification.

[0130] Some embodiments of this specification also provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement this specification. Figures 2 to 8 The content quality identification method shown.

[0131] Some embodiments of this specification also provide a computer-readable storage medium storing computer instructions that, when executed by a processor, can implement this specification. Figures 2 to 8 The content quality identification method shown.

[0132] Some embodiments of this specification also provide a computer program product, including a computer program that, when at least a portion of the computer program is executed by a processor, can implement this specification. Figures 2 to 8 The content quality identification method is illustrated. In some embodiments, the computer program product may refer only to a computer program, which may be carried on a storage medium or a processing device. In other embodiments, the computer program product may also be a storage medium or a processing device containing the aforementioned computer program. The processing device may include one or more processors, and the storage medium.

[0133] In some embodiments, the processor may be a combination of one or more of the following processors: central processing unit (CPU), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), graphics processing unit (GPU), physical processing unit (PPU), digital signal processor (DSP), field-programmable gate array (FPGA), programmable logic device (PLD), programmable logic controller (PLC), reduced instruction set computer (RISC), and microprocessor.

[0134] In some embodiments, the storage medium may include one or more combinations of the following: mass storage, removable storage, volatile read-write memory, and read-only memory (ROM). Exemplary mass storage may include disks, optical disks, solid-state drives, etc. Exemplary removable storage may include flash drives, floppy disks, optical disks, memory cards, compressed hard disks, magnetic tapes, etc. Exemplary volatile read-write memory may include random access memory (RAM). Exemplary RAM may include dynamic random access memory (DRAM), dual data rate synchronous dynamic random access memory (DDRSDRAM), static random access memory (SRAM), silicon controlled retrieval memory (T-RAM), and zero-capacitance memory (Z-RAM), etc. Exemplary read-only memory may include masked read-only memory (MROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compressed hard disk read-only memory (CD-ROM), and digital multifunction hard disk read-only memory, etc.

[0135] The basic concepts have been described above. It is obvious that the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, various modifications, improvements, and corrections may be made to this specification by those skilled in the art. Such modifications, improvements, and corrections are taught in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

Claims

1. A content quality identification method, characterized in that, The method includes: Using a pre-trained content quality recognition model, a predetermined number of simulated comments and corresponding comment stances are generated based on the content to be identified. Each simulated comment is a different comment associated with the content to be identified, and the comment stance is used to indicate whether the corresponding simulated comment supports the content to be identified. The comment stance includes a supportive stance and a non-supportive stance. A quality score for the content to be identified is determined based on the number of first comments supporting the stated position and the number of second comments opposing the stated position. The quality label of the content to be identified is determined based on the quality score, and the quality label includes high quality and low quality.

2. The method according to claim 1, characterized in that, The method further includes: The first unlabeled content and the first real comments associated with the first content are used as the first prompts to instruct the pre-trained teacher model to generate each first simulated comment associated with the first content, a first comment stance corresponding to each first simulated comment, a first quality score and a first quality label for the first content; the first quality score is determined based on the first comment stance, and the first quality label is determined based on the first quality score. A first dataset is constructed based on the first content, the first simulated comment, the first comment stance, the first quality score, and the first quality label; The pre-trained teacher model is instructed to generate each second simulated comment associated with the second content, a second comment stance corresponding to each second simulated comment, a second quality score and a second quality label for the second content, and to ensure that the second quality label is consistent with the artificial quality label. The artificial quality label is manually annotated, the second quality score is determined based on the second comment stance, and the second quality label is determined based on the second quality score. A second dataset is constructed based on the second content, the second simulated comments, the second comment stance, the second quality score, and the second quality label; Machine-labeled data for training the content quality recognition model is obtained by merging the first dataset and the second dataset.

3. The method according to claim 2, characterized in that, The step of using unlabeled first content and first real comments associated with the first content as first prompts to instruct the pre-trained teacher model to generate each first simulated comment associated with the first content, a first comment stance corresponding to each first simulated comment, a first quality score and a first quality label for the first content includes: The first content without annotation and the first real comments associated with the first content are used as the first prompt, instructing the pre-trained teacher model to filter out a portion of the first real comments from the initial first real comments according to preset filtering conditions. Based on the first content and some of the first real comments, the pre-trained teacher model is instructed to generate each of the first simulated comments associated with the first content, the first comment stance corresponding to each of the first simulated comments, the first quality score of the first content, and the first quality label; The step of using the labeled second content, the second real comments associated with the second content, and the artificial quality label of the second content as a second prompt to instruct the pre-trained teacher model to generate each second simulated comment associated with the second content, a second comment stance corresponding to each second simulated comment, a second quality score and a second quality label for the second content, and instructing the second quality label to be consistent with the artificial quality label, includes: The second content that has been labeled, the second real comments associated with the second content, and the artificial quality label of the second content are used as the second prompt to instruct the pre-trained teacher model to filter a portion of the second real comments from the initial second real comments according to the preset filtering conditions. Based on the second content, some of the second real comments, and the artificial quality label, the pre-trained teacher model is instructed to generate each of the second simulated comments associated with the second content, the second comment stance corresponding to each of the second simulated comments, the second quality score of the second content, and the second quality label, and to indicate that the second quality label is consistent with the artificial quality label.

4. The method according to claim 3, characterized in that, The preset filtering conditions include sorting the number of likes corresponding to the real comments of each target content from high to low, and / or, the priority of real comments on content elements in the target content is higher than the priority of real comments on the overall theme of the target content; The target content includes the first content and the second content, and the real comments include the first real comments and the second real comments.

5. The method according to claim 4, characterized in that, The target content includes video content, and the content elements include at least one of the following: cover image, keyframe, title, content tags, audio / text, category, resolution, and playback format; The artificial quality labels include high quality and low quality; the artificial quality labels are determined based on at least one of the following when labeling the second content: creativity, emotional engagement, information value, narrative coherence, and originality.

6. The method according to claim 2 or 3, characterized in that, The training process of the content quality recognition model includes: Based on the machine-annotated data, a first content quality recognition model is obtained through supervised fine-tuning training, enabling the first content quality recognition model to output simulated comments, comment stances, quality scores, and quality tags for the target content; the target content includes the first content and the second content; Based on manually labeled data, a second content quality identification model is trained by optimizing the first content quality identification model through reinforcement learning. The second content quality identification model outputs simulated comments with better diversity, similarity between the simulated comments and real comments, and accuracy of quality labels than the first content quality identification model. The manually labeled data includes at least the second content and the manually labeled quality tags. The real comments include the first real comments and the second real comments. The trained second content quality recognition model is used as the pre-trained content quality recognition model.

7. The method according to claim 6, characterized in that, The supervised fine-tuning constructs a first loss function based on the difference between the output of the first content quality recognition model and the machine-annotated data. The training process of the supervised fine-tuning causes the loss value of the first loss function to gradually decrease until it converges.

8. The method according to claim 6, characterized in that, The reinforcement learning determines a reward score based on the output of the second content quality recognition model, and constructs a second loss function based on the reward score. The training process of the reinforcement learning causes the reward score to gradually increase until it reaches a stable state where it no longer increases. The reward score includes at least one of format reward score, tag reward score, diversity reward score, and social reward score; the format reward score is determined based on whether the output of the second content quality recognition model meets the required format; the tag reward score is determined based on whether the quality tag output by the second content quality recognition model is consistent with the human quality tag; the diversity reward score is determined based on the degree of repetition of the simulated comments output by the second content quality recognition model, and the diversity reward score increases as the degree of repetition decreases; the social reward score is determined based on the degree of similarity between the simulated comments output by the second content quality recognition model and the real comments, and the social reward score increases as the degree of similarity increases.

9. The method according to any one of claims 1 to 8, characterized in that, The quality score of the content to be identified is determined based on the number of first comments supporting the stance and the number of second comments not supporting the stance, including: Determine the difference between the first number of comments and the second number of comments; The quality score of the content to be identified is obtained by dividing the difference by the square root of the sum of the first number of comments and the second number of comments. The process of determining the quality label of the content to be identified based on the quality score includes: If the quality score is greater than or equal to the quality score threshold, the content to be identified is determined to be of high quality; otherwise, the content to be identified is determined to be of low quality.

10. A content quality identification device, characterized in that, The device includes: The comment generation module is used to generate a predetermined number of simulated comments and corresponding comment stances based on the content to be identified using a pre-trained content quality recognition model; each simulated comment is a different comment associated with the content to be identified, and the comment stance is used to indicate whether the corresponding simulated comment supports the content to be identified, and the comment stance includes a supportive stance and a non-supportive stance. The rating determination module is used to determine the quality rating of the content to be identified based on the number of first comments supporting the position and the number of second comments not supporting the position. A quality identification module is used to determine the quality label of the content to be identified based on the quality score, wherein the quality label includes high quality and low quality.

11. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it is able to implement the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, enable the implementation of the method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, It includes a computer program that, when at least a portion of the computer program is executed by a processor, enables the implementation of the method as described in any one of claims 1 to 9.

Images