Content auditing method and device, electronic equipment and storage medium

Abstract

The invention discloses a content auditing method and device, electronic equipment and a storage medium, belongs to the field of generative artificial intelligence, and is used for improving the content auditing efficiency. The method comprises the following steps: receiving streaming content through WebSocket, wherein the streaming content is generated by packaging content generated by a large model through a message queue; auditing the streaming content to generate an auditing instruction; and sending an event SSE through a server to display the streaming content to a user side, and adjusting the display state of the streaming content according to the auditing instruction.

Description

Technical Field

[0001] The present application belongs to the field of generative artificial intelligence, and specifically relates to a content review method, device, electronic device and storage medium. Background Art

[0002] Generative artificial intelligence shows great potential in improving work efficiency and innovation capabilities. However, the technology of automatically generated content (AIGC) is also accompanied by risks such as illegal information, false information, low-quality content, and AI performance degradation. These problems are usually dealt with by combining machine review and manual review to ensure the quality and compliance of the content. However, in the streaming dialogue system, due to the continuous and constantly updated output, its risks and limitations are also revealed. Risk of illegal content: the generated content may contain sensitive information, inappropriate content or false information, which may easily have a negative impact on users or platforms; inefficient review process: traditional review relies on a combination of machine review and manual review, and the process is lagging and difficult to meet real-time requirements; insufficient compliance: the generated content may cause legal disputes or ethical issues without strict review; high-cost manual reliance: the workload of manual review is large, and it is difficult to adapt to high-concurrency scenarios, which puts pressure on the economy of the system.

[0003] Currently, there are some methods that attempt to optimize content review through static review rules or machine review technology, but these methods face major challenges in streaming content generation scenarios. Review lag: Content generation is real-time output word by word or paragraph by paragraph, and traditional static review processes are difficult to adapt to the real-time requirements of streaming generation. Synchronization difficulties: The content seen by reviewers is not synchronized with the content displayed on the user side, resulting in failure or lag in manual intervention. High-concurrency processing bottleneck: In high-traffic scenarios, the system's review and display capabilities are easily restricted by bottlenecks, resulting in a decline in user experience. Review policy update lag: Existing methods lack effective use of manual review experience, and the update cycle of machine review models is long, making it difficult to cope with emerging content risks. It can be seen that current review technologies cannot efficiently and reliably perform security reviews on review content. Summary of the invention

[0004] The embodiments of the present application provide a content review method, device, electronic device and storage medium, which can solve the problem that current review technology cannot efficiently and reliably perform security review on the content to be reviewed.

[0005] In the first aspect, an embodiment of the present application provides a content review method, which includes: receiving streaming content through WebSocket, wherein the streaming content is generated by encapsulating content generated by a large model through a message queue; reviewing the streaming content and generating review instructions; sending an event SSE through a server to display the streaming content to a user terminal and adjusting the display status of the streaming content according to the review instructions.

[0006] In the second aspect, an embodiment of the present application provides a content review device, which includes: a transmission module, which is used to receive streaming content through WebSocket, and the streaming content is generated by encapsulating the content generated by a large model through a message queue; an audit module, which is used to review the streaming content and generate review instructions; a display module, which is used to send an event SSE through a server to display the streaming content to a user end and adjust the display status of the streaming content according to the review instructions.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In an embodiment of the present application, streaming content is received through WebSocket, and the streaming content is generated by encapsulating the content generated by the large model through a message queue; the streaming content is audited and an audit instruction is generated; the streaming content is displayed to the user end by sending an event SSE through the server and the display status of the streaming content is adjusted according to the audit instruction, so that the content generated by the large model can be audited efficiently and reliably. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of a content review method provided in an embodiment of the present application; Figure 2 This is a schematic diagram of a content review system architecture provided by an embodiment of the present application; Figure 3 It is a flowchart of another content review method provided in an embodiment of the present application; Figure 4 It is a structural diagram of a content review device provided in an embodiment of the present application; Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0012] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0013] The content review method, device, electronic device and storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0014] Figure 1 A content review method provided by an embodiment of the present application is shown, and the method can be executed by an electronic device, and the electronic device may include: a server, etc. In other words, the method can be executed by software or hardware installed in the electronic device, and the method includes the following steps: Step 102: Receive streaming content via WebSocket.

[0015] Specifically, the above-mentioned streaming content is generated by encapsulating the content generated by the large model through a message queue, that is, the large model can generate content word by word or paragraph by paragraph based on the user's questions and answers, and then encapsulate and generate streaming content through a message queue and transmit it. The review platform of the electronic device can receive the streaming content through WebSocket, which is a protocol for full-duplex communication on a single TCP connection. WebSocket makes data exchange between the client and the server easier, allowing the server to actively push data to the user end.

[0016] Step 104: review the streaming content and generate review instructions.

[0017] Specifically, the audit platform receives streaming content through WebSocket technology for real-time viewing and operation by manual auditors. Auditors view streaming content in real time through WebSocket, and if risky content is found, they can immediately send audit instructions, such as blocking instructions.

[0018] Step 106: The server sends an event SSE to display the streaming content to the user end and adjusts the display status of the streaming content according to the review instruction.

[0019] In an embodiment of the present application, the above-mentioned streaming content can be displayed to the user end through the server sending event SSE. The user end uses SSE technology to realize non-refresh display of streaming content and dynamically adjusts the display status according to the review instruction, such as normal display, delay or blocking.

[0020] In one implementation, the sending of the event SSE via the server displays the streaming content to the user end and adjusts the display status of the streaming content according to the review instruction, including: when the review instruction is a normal instruction, displaying the streaming content normally to the user end via SSE; when the review instruction is a delay instruction, displaying the streaming content to the user end via SSE after a preset delay time; and when the review instruction is a blocking instruction, blocking the display of the streaming content to the user end via SSE.

[0021] Specifically, the audit platform receives streaming content through WebSocket technology, and the auditors view the streaming content in real time through WebSocket. If no risky content is found, normal instructions can be sent immediately, and the streaming content can be displayed normally to the user end through SSE. If risky content is found, a delay instruction can be issued immediately, and the streaming content can be displayed to the user end through SSE after a preset delay time. The preset time can be set according to actual needs, or if risky content is found, a blocking instruction can be issued immediately to block the streaming content from being displayed to the user end through SSE.

[0022] The content review method provided in the embodiment of the present application receives streaming content through WebSocket, and the streaming content is generated by encapsulating the content generated by the large model through the message queue; the streaming content is reviewed and an review instruction is generated; the server sends an event SSE to display the streaming content to the user end and adjusts the display status of the streaming content according to the review instruction. A dual-channel communication architecture for review and display, WebSocket technology is used for interaction between the review platform and the background, and SSE technology is used for user-side streaming content display, which improves the interaction efficiency. The method supports real-time manual monitoring and real-time intervention, supports configurable user-side delay, ensures that reviewers have sufficient time to intervene in risky content, ensures content security, and at the same time ensures the user experience. It significantly improves the efficiency, accuracy and reliability of manual review, ensures the security and quality control of content generation, realizes real-time synchronization of information and immediate response of manual review, and solves the problem of not being able to efficiently and reliably perform security review on the content to be reviewed.

[0023] In one implementation, after reviewing the streaming content and generating review instructions, the method further includes: generating annotation data based on the streaming content and the review instructions; and training and optimizing a machine review model through the annotation data.

[0024] In the embodiments of the present application, real-time auxiliary tools for manual review, dynamic feedback and strategy self-learning are provided. The review background records the streaming content and review instructions determined by manual review, and generates high-quality annotated data. The machine review model is regularly trained and optimized through the annotated data, and a machine learning mechanism is introduced to dynamically adjust the review strategy and enhance the model's ability to identify complex content. The closed-loop feedback of the manual review results is used for strategy optimization and model iteration of machine review. By optimizing the machine review strategy through a dynamic feedback mechanism, the workload of manual review can be gradually reduced.

[0025] In an implementation, before reviewing the streaming content and generating a review instruction, the method further includes: highlighting keywords in the streaming content and marking the streaming content with predicted sensitive content.

[0026] In the embodiment of the present application, auxiliary functions such as keyword highlighting and sensitive content prediction and annotation are also provided. When reviewing streaming content, keywords in streaming content can be highlighted and sensitive content can be predicted for streaming content. This can help reviewers quickly locate and handle potential risk content in streaming content.

[0027] In one implementation, the streaming content is encrypted using an end-to-end encryption method based on TLS.

[0028] In the embodiment of the present application, TLS-based end-to-end encryption technology is used in the entire link from model generation to background review and user-side display to ensure the security of streaming content data transmission.

[0029] In one implementation, after receiving the streaming content through WebSocket, the method further includes: triggering a security alarm when high-frequency blocking instructions or abnormal marking operations are detected.

[0030] In an embodiment of the present application, access rights control is added to the audit background operation log and the user-side display record to prevent data leakage and abuse. The audit background can monitor high-frequency blocking instructions, abnormal marking operations and other behaviors, and can trigger a security alarm when high-frequency blocking instructions or abnormal marking operations are detected. The user side can record access behaviors to sensitive content, generate risk reports and issue warnings to improve the operational security of the system. Permission control can ensure the security of content generation, review, transmission and display. The abnormal behavior monitoring and alarm mechanism can effectively prevent security risks caused by misoperation or malicious operation.

[0031] Figure 2 A schematic diagram of a content review system architecture provided by an embodiment of the present application is shown as follows: Figure 2 As shown in the figure, the content audit system architecture includes: content generation module, message queue module, content audit module and content display module. Among them, the generation module: generates content in real time and pushes it to the message queue. The message queue module: realizes content transmission and module decoupling, and supports distributed deployment to cope with high concurrency scenarios. Audit module: realizes content flow synchronization through WebSocket, and provides normal instructions, delay instructions, blocking instructions and other instructions. Display module: receives SSE content flow and dynamically adjusts content display logic according to audit instructions.

[0032] Figure 3 A flow chart showing another content review method provided by an embodiment of the present application is shown as follows: Figure 3 As shown in the figure, the process of the content audit method includes: the user initiates a question-answering server request to the model, the large model generates content and encapsulates it into messages word by word or paragraph by paragraph, and transmits it to the audit background through the message queue in priority. Auditors view the content stream in real time through WebSocket to conduct security audit judgments. If risky content is found, blocking or delaying instructions can be sent immediately. The user end receives the content through SSE and displays it normally, delays it, or blocks it according to the audit instructions.

[0033] Among them, real-time auxiliary tools for manual review are provided. The review background provides auxiliary functions such as keyword highlighting and sensitive content prediction and annotation to help reviewers quickly locate and handle potential risk content. Real-time content classification function is implemented to group content by topic or risk level to improve the efficiency of manual review. Dynamic feedback and strategy self-learning are implemented. The review background records manual review judgments and operation results to generate high-quality annotated data. Regularly train and optimize the machine review model, introduce machine learning mechanisms, dynamically adjust the review strategy, and improve the model's ability to identify complex content. End-to-end data encryption transmission, TLS-based end-to-end encryption technology is used in the entire link from content generation to review and user display to ensure the security of content data transmission. The review background operation log and user-side display record increase access control to prevent data leakage and abuse. Abnormal behavior monitoring and alarm, the review background monitors high-frequency blocking instructions, abnormal annotation operations and other behaviors, and triggers security alarms. The user end records the access behavior of sensitive content, generates risk reports and issues warnings to improve the operational security of the system.

[0034] The content review method provided in the embodiment of the present application can be manually reviewed in real time and intervened in real time, supports configurable user-side delay, and ensures content security while ensuring user experience. Dual-channel communication architecture for review and display: WebSocket technology is used for interaction between the review platform and the background, and SSE technology is used for streaming content display on the user side. Real-time content transmission mechanism based on message queue: used for streaming transmission and priority control of generated content between the generation module, review module and user display module. Dynamic feedback and policy self-learning mechanism: closed-loop feedback of manual review results is used for policy optimization and model iteration of machine review. End-to-end encryption and anomaly monitoring mechanism: encrypted data transmission of the entire review process and abnormal behavior detection and alarm in the background and user end. Distributed review and load balancing mechanism for high-concurrency scenarios: real-time review and display of large-scale content generation are supported through distributed architecture and load prediction algorithms.

[0035] It should be noted that the content review method provided in the embodiment of the present application can be executed by a content review device or a control module in the content review device for executing the content review method. In the embodiment of the present application, the content review device provided in the embodiment of the present application is described by taking the content review device executing the content review method as an example.

[0036] Figure 4 Schematic diagram of the structure of the content review device according to the embodiment of the present application. Figure 4 As shown, the content review device 400 includes: a transmission module 410, a review module 420 and a display module 430.

[0037] The transmission module 410 is used to receive streaming content through WebSocket, and the streaming content is generated by encapsulating the content generated by the large model through the message queue; the review module 420 is used to review the streaming content and generate review instructions; the display module 430 is used to display the streaming content to the user end by sending an event SSE through the server and adjust the display status of the streaming content according to the review instructions.

[0038] In one implementation, the display module 430 is used to display the streaming content normally to the user end through SSE when the review instruction is a normal instruction; to display the streaming content to the user end through SSE with a preset delay time when the review instruction is a delay instruction; and to block the display of the streaming content to the user end through SSE when the review instruction is a blocking instruction.

[0039] In one implementation, the audit module 420 is further used to generate annotation data based on the streaming content and the audit instruction; and train and optimize the machine audit model through the annotation data.

[0040] In one implementation, the review module 420 is further configured to highlight keywords in the streaming content and perform a predicted sensitive content tagging on the streaming content.

[0041] In one implementation, the streaming content is encrypted using an end-to-end encryption method based on TLS.

[0042] In one implementation, the audit module 420 is also used to trigger a security alarm when high-frequency blocking instructions or abnormal marking operations are detected.

[0043] In one implementation, the review module 420 is further configured to group the streaming content according to themes or risk levels.

[0044] The content review device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0045] The content review device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0046] The content review device provided in the embodiment of the present application can achieve Figures 1 to 3 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0047] like Figure 5 As shown, an embodiment of the present application further provides an electronic device 500, including a processor 501 and a memory 502, wherein the memory 502 stores a program or instruction that can be executed on the processor 501, and the program or instruction, when executed by the processor 501, implements: receiving streaming content through WebSocket, wherein the streaming content is generated by encapsulating the content generated by the large model through a message queue; reviewing the streaming content and generating review instructions; and sending an event SSE through the server to display the streaming content to the user end and adjust the display status of the streaming content according to the review instructions.

[0048] In one implementation, when the review instruction is a normal instruction, the streaming content is normally displayed to the user end through SSE; when the review instruction is a delay instruction, the streaming content is delayed for a preset time and displayed to the user end through SSE; when the review instruction is a blocking instruction, the streaming content is blocked from being displayed to the user end through SSE.

[0049] In one implementation, after the streaming content is audited and the audit instruction is generated, annotation data is generated according to the streaming content and the audit instruction; and the machine audit model is trained and optimized using the annotation data.

[0050] In one implementation, before reviewing the streaming content and generating a review instruction, the keywords in the streaming content are highlighted and the streaming content is marked with predicted sensitive content.

[0051] In one implementation, the streaming content is encrypted using an end-to-end encryption method based on TLS.

[0052] In one implementation, after receiving the streaming content through WebSocket, if high-frequency blocking instructions or abnormal marking operations are detected, a security alarm is triggered.

[0053] Before reviewing the streaming content and generating a review instruction, the streaming content is grouped according to themes or risk levels.

[0054] The specific execution steps can refer to the various steps of the above-mentioned content review method implementation example, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0055] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals or other devices except terminals.

[0056] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0057] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0058] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0059] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned content review method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0060] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as ROM, RAM, magnetic disk or optical disk.

[0061] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0062] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0063] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A content review method, characterized in that: include: Receiving streaming content through WebSocket, where the streaming content is generated by encapsulating the content generated by the large model through a message queue; Reviewing the streaming content and generating review instructions; The streaming content is displayed to the user end through a server sending event SSE and the display state of the streaming content is adjusted according to the review instruction.

2. The method according to claim 1, characterized in that: The step of displaying the streaming content to the user terminal through the server sending event SSE and adjusting the display state of the streaming content according to the review instruction includes: In the case where the audit instruction is a normal instruction, the streaming content is normally displayed to the user end through SSE; In the case where the review instruction is a delay instruction, the streaming content is displayed to the user terminal by delaying the preset time through SSE; In the case where the audit instruction is a blocking instruction, the display of the streaming content to the user end through SSE is blocked.

3. The method according to claim 1, characterized in that After the streaming content is audited and the audit instruction is generated, the method further includes: generating annotation data according to the streaming content and the review instruction; The machine audit model is trained and optimized through the labeled data.

4. The method according to claim 1, characterized in that: Before reviewing the streaming content and generating a review instruction, the method further includes: The keywords in the streaming content are highlighted, and the streaming content is marked with predicted sensitive content.

5. The method according to claim 1, characterized in that The streaming content is encrypted using TLS-based end-to-end encryption.

6. The method according to claim 1, characterized in that After receiving the streaming content through WebSocket, the method further includes: When high-frequency blocking instructions or abnormal marking operations are detected, a safety alarm is triggered.

7. The method according to claim 1, characterized in that Before reviewing the streaming content and generating a review instruction, the method further includes: The streaming content is grouped according to subject matter or risk level.

8. A content review device, characterized in that: include: A transmission module, used for receiving streaming content through WebSocket, wherein the streaming content is generated by encapsulating the content generated by the large model through a message queue; An audit module, used to audit the streaming content and generate an audit instruction; The display module is used to display the streaming content to the user end through the server sending event SSE and adjust the display state of the streaming content according to the review instruction.

9. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the content review method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the content review method as described in any one of claims 1-7 are implemented.

Images