Audio violation detection method, apparatus and electronic device

Abstract

The application relates to an audio violation detection method and device and an electronic device, and the method comprises the following steps: receiving, by a server, mixed audio translation text obtained by performing audio translation on mixed audio data of a current channel by a client, and identifying whether sensitive information exists in the mixed audio translation text; when the mixed audio translation text contains sensitive information, obtaining, by the server, original audio of each user speaking in a time period corresponding to the mixed audio translation text in the current channel; performing audio translation on the original audio of each user in the current channel by the server respectively, and locating a user containing sensitive information according to original audio translation text obtained by identification. The application combines client mixed audio identification and server original audio tracing, and significantly reduces the consumption of server audit resources.

Description

Technical Field

[0001] This application relates to the field of network voice interaction technology, and in particular to an audio violation detection method, apparatus and electronic device thereof. Background Technology

[0002] In the field of network voice interaction technology, group voice communication is widely used in scenarios such as online meetings, voice chat rooms, and online gaming due to its immediacy and interactivity. To ensure a healthy network environment, compliance review of user voice content is an essential step.

[0003] Traditional audio violation detection solutions typically employ a centralized server-side processing architecture. The principle is that the server pulls the raw audio stream from each user within a channel in real time, uses a high-performance transliteration model deployed on the server to perform independent speech-to-text recognition on each audio stream, and then matches the recognized text against a prohibited keyword database to determine if any violations exist. For matches that are found to be in violation, manual review is required before penalties are imposed.

[0004] With the explosive growth of group voice services, the aforementioned traditional solutions face severe resource consumption bottlenecks in practical applications. First, the server needs to process massive amounts of audio data. Since multiple users exist simultaneously in a group setting, the server must maintain real-time connections with each user and retrieve their audio streams separately. This leads to enormous bandwidth consumption, especially in high-concurrency scenarios where network transmission costs rise exponentially. Second, transcribing and recognizing each audio stream requires significant GPU computing power. To meet real-time requirements, enterprises must invest heavily in hardware to expand their server clusters. Third, group voice scenarios have unique user behavior characteristics: most users in the channel are in a silent listening state, with very few actually speaking simultaneously. Traditional solutions indiscriminately retrieve and recognize the audio streams of all users in real time, ignoring the processing of invalid data from non-speaking or silent states in group voice communication, resulting in a serious waste of computing power and bandwidth resources.

[0005] Therefore, how to effectively reduce the bandwidth and computing power consumption of the server in the massive group audio review while ensuring the coverage of audio violation detection is a technical problem that needs to be solved by existing technologies. Summary of the Invention

[0006] The purpose of this application is to solve at least one of the above-mentioned problems by providing an audio violation detection method and corresponding apparatus, device, non-volatile readable storage medium, and computer program product.

[0007] According to one aspect of this application, an audio violation detection method is provided, comprising: The server receives the transliterated audio data of the current channel from the client and identifies whether there is sensitive information in the transliterated audio data. When the mixed transliteration contains sensitive information, the server obtains the original audio of each user who spoke within the current channel during the time period corresponding to the mixed transliteration; The server performs transliteration recognition on the original audio of each user and locates the offending user containing sensitive information based on the transliterated original audio.

[0008] According to another aspect of this application, an audio violation detection device is provided, comprising: The edge recognition module is configured to receive the transliterated audio data of the current channel from the client and identify whether there is sensitive information in the transliterated audio data. The original audio acquisition module is configured to obtain the original audio of each user who spoke within the current channel during the time period corresponding to the mixed transliteration when the mixed transliteration contains sensitive information; The violation location module is configured to have the server perform transliteration recognition on the original audio of each user, and locate the violating user containing sensitive information based on the transliterated original audio.

[0009] According to another aspect of this application, an electronic device is provided, including a central processing unit and a memory, wherein the central processing unit is configured to invoke and run a computer program stored in the memory to perform the steps of the method described in this application.

[0010] According to another aspect of this application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the audio violation detection method in the form of computer-readable instructions, wherein the computer program, when invoked by a computer, performs the steps included in the method.

[0011] According to another aspect of this application, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method.

[0012] Compared to traditional technologies, this application effectively solves the problem of excessive server bandwidth and computing power consumption in massive group audio review by integrating server-side coordination and client-side distributed computing architecture. It utilizes the client to perform local transliteration recognition on the current channel's mixed audio data for initial screening. Only when sensitive information is detected does the server backtrack and download the original audio of the users who spoke within the corresponding time period for precise source tracing. This mechanism reuses the client's inherent audio receiving path, avoiding bandwidth waste caused by indiscriminate audio retrieval. Simultaneously, it offloads the high-load transliteration task to the client for edge computing, significantly reducing dependence on server-side GPU resources. By combining initial mixed audio screening with original audio verification, this application achieves low-cost, high-efficiency group audio compliance review while ensuring high accuracy in violation detection. Attached Figure Description

[0013] Figure 1 This is an exemplary network architecture for this application; Figure 2 This is a flowchart illustrating one embodiment of the audio violation detection method of this application; Figure 3 This is a schematic block diagram of the audio violation detection device of this application; Figure 4 This is a schematic diagram of the structure of an electronic device used in this application. Detailed Implementation

[0014] The following section provides a general description of the network architecture, typical application scenarios, and basic concepts that can be deployed with the technical solutions of this application, in order to facilitate a more efficient understanding of the subsequent specific embodiments of this application.

[0015] The technical solution of this application can be deployed in a distributed network environment, which typically includes multiple computing devices interconnected via a network. For example... Figure 1 As shown, the network architecture may include a client 80, a service server 81, and a media server 82.

[0016] Client 80 can be a terminal device used by a user to send and receive network voice data, such as a smartphone, tablet, or personal computer. Its network requests can be requests to join or create a voice channel, and it can interact with the business server 81 and media server 82 via the network.

[0017] The business server 81, as the server that executes the audio violation detection method of this application, runs a computer program implemented according to the audio violation detection method of this application. It is responsible for performing operations involving calculation and decision-making, including receiving device information reported by the client, maintaining the device activity table, coordinating and scheduling transliteration tasks, receiving mixed transliterations for initial screening, and backtracking and downloading the original audio for accurate responsibility determination.

[0018] Media server 82 can be deployed with streaming media services that support real-time voice communication, used to maintain voice channels and handle audio stream forwarding between clients. For example, in scenarios such as voice rooms, social groups, live broadcast rooms or multi-person conference rooms, media server 82 is responsible for mixing or distributing the audio streams of each user in the channel.

[0019] In one implementation, the business server 81 and the media server 82 can be integrated into the same physical or logical server. Specifically, the mixing logic in the media server can be invoked by the business server 81, or different server clusters can respectively handle business logic processing and real-time media transmission functions. Furthermore, the network environment can also include a database server for storing a prohibited keyword database and a device activity table, providing rule data and status data support for the business server 81.

[0020] In terms of application scenarios, the technical solution of this application is mainly applied to real-time group audio interaction scenarios of various Internet products. For example, in scenarios such as voice chat rooms, in-game voice chat, online meetings, or multi-party calls, when a large number of users are simultaneously in a voice channel, the method of this application can be used to obtain the device capability information reported by each client by querying the device activity table through the business server 81, and to obtain the audio translation resource package based on the bandwidth load of the server. Subsequently, the selected client is scheduled to perform local audio translation recognition on the mixed audio data in the channel and report the recognition results to the business server 81. After recognizing sensitive information, the business server 81 will instruct the media server 82 to backtrack and download the original audio of the users who spoke within a specific time period for secondary recognition to accurately locate the violating users. In this way, while ensuring the compliance of voice content, the computing power and bandwidth consumption of the server in processing massive amounts of audio data can be effectively reduced.

[0021] The term "channel" in this application refers to a logical room or virtual space in an internet application where multiple users can engage in real-time audio interaction. Its forms include, but are not limited to, voice chat rooms, social groups, live streaming rooms, and multi-user conference rooms. Users can listen to or send voice data after joining a channel. The term "mixed audio data" in this application refers to a single audio stream created by the media server 82 or client 80 by mixing the audio streams of multiple users within the channel in chronological order. This data is a superposition of all spoken sounds within the channel, and the client is responsible for playing this audio stream to present the audio content within the channel. The term "transliteration recognition" in this application refers to the automatic speech recognition process that converts audio data into text content. This can be achieved using an Automatic Speech Recognition (ASR) model. In this application, this process is primarily executed locally on client 80, while being executed on the server as needed. The result is either a mixed transliteration or a transliteration of the original audio. The term "time period corresponding to the mixed transliteration" in this application refers to the specific time interval in the audio stream corresponding to the text containing sensitive information in the mixed transliteration. The business server 81 uses this time period to backtrack and download the original audio, which is a key element for achieving accurate responsibility determination.

[0022] It should be noted that although the description of the technical solution of this application is mainly based on user activities within a single channel, since it is implemented on the server side and the server side serves a massive number of channels, such as a small number of live broadcast rooms, voice chat rooms, conference accounts, social groups, etc., the technical solution of this application is also applicable to multiple channels that concurrently serve the same platform.

[0023] Having explained the above architecture, principles, and related concepts, the following section will elaborate on various implementation methods of this application in conjunction with the accompanying drawings.

[0024] Please see Figure 2 The audio violation detection method provided in this application can be implemented as a computer program product and run on a server, including the following steps: Step S5100: The server receives the transliterated text of the mixed audio data of the current channel obtained by the client through transliteration recognition, and identifies whether there is sensitive information in the transliterated text. Within the same channel, multiple users participate in voice chat activities through their respective clients. The service server, acting as the server, can pre-schedule one or more clients within a given channel to perform transliteration recognition tasks. These selected clients need to pre-configure a transliteration function resource package. The transliteration function resource package refers to a software collection containing an automatic speech recognition model, decoder, and related dependency libraries, used to support the client in converting audio data into text content locally. In one embodiment, during the client's pre-configuration of the resource package, the service server can, according to a specific scheduling strategy, issue installation or loading instructions to the client under certain conditions, thereby enabling the client to perform transliteration recognition.

[0025] When the client performs the transliteration recognition task, the mixed data it processes can be either audio data from the media server (processed individually) or composite, meaning it includes not only audio data from other users in the current channel forwarded by the media server, but also local user audio data collected by the client's own audio pickup device. In one embodiment, the client's local audio acquisition module converts the analog signal captured by the microphone into a digital audio stream, while simultaneously receiving audio streams from other users on the media server. The client's internal mixer unit mixes these two or more audio streams in chronological order to generate a single mixed data stream. Subsequently, the client calls a pre-built transliteration function resource package and uses an automatic speech recognition model to perform transliteration recognition on the mixed data, converting the audio signal into text content, i.e., the transliterated audio, which is then reported to the service server. In another embodiment, the timestamp of the transliterated audio in the recognized audio data can be predetermined, forming a time period. This time period, associated with the transliterated audio, is then used to construct a network request submitted to the service server.

[0026] After receiving the mixed transliteration reported by the client, the business server performs a sensitive information identification operation on the mixed transliteration. This identification operation can be applied in various specific embodiments. In one embodiment, the business server matches the mixed transliteration with a preset prohibited keyword database. The prohibited keyword database stores a set of words used to determine violations. This set includes, but is not limited to, sensitive words containing word roots and derivatives. The matching method can use exact string matching or a fuzzy matching algorithm based on edit distance. When any keyword in the prohibited keyword database is detected in the mixed transliteration, it is determined that sensitive information exists. In another embodiment, the business server inputs the mixed transliteration into a pre-trained deep learning neural network classification model. This model is trained based on a large-scale dataset of violating texts and can output the probability or confidence level of the mixed transliteration belonging to a violation category. When the probability or confidence level exceeds a preset threshold, it is determined that the mixed transliteration contains sensitive information.

[0027] Step S5200: When the mixed transliteration contains sensitive information, the server obtains the original audio of each user who spoke within the time period corresponding to the mixed transliteration in the current channel; The audio data continuously generated by the current channel is typically stored in a database by the media server. For the server-side, i.e., the business server, when it needs to reuse this audio data, it can retrieve it from the media server or directly from the database. Therefore, after determining that the mixed audio transcript contains sensitive information, the business server can retrieve the original audio of each user speaking within the corresponding time period of the mixed audio transcript from the database.

[0028] Because the audio data packet containing the transliterated audio contains audio data pushed by the media server, the two have a temporal correspondence. Therefore, accurate data backtracking can be performed based on this temporal correlation. When the transliterated audio is recognized and reported by the client, it is associated with the timestamp information of the transliteration in the original audio data. This timestamp defines the specific time interval in which the sensitive information occurred, i.e., the time period. Based on this time period, the business server sends a data retrieval command to the media server or database, thereby requesting to obtain the audio data corresponding to the original audio of all active users speaking within the channel during that specific time period.

[0029] There are several specific implementation methods for obtaining raw audio. In a first embodiment, the service server directly sends a raw audio download request carrying a time period parameter to the media server. Based on the time period index, the media server extracts the independent audio streams of all users speaking within that time period from a real-time streaming audio stream cache or a persistently stored database, and then encapsulates and transmits these audio streams to the service server. In a second embodiment, the service server can also directly pull data from a database that stores audio files segmented by channel and time. The service server uses a query interface to locate the current channel identifier and the corresponding time period, and then downloads the raw audio files of each user within that time period in batches.

[0030] The raw audio obtained here refers to the unprocessed, pure audio data from a single user. Unlike the mixed audio data processed in step S5100, this raw audio data retains the unique voiceprint characteristics and speech content of the speaking user. After obtaining this raw audio, the business server can temporarily store it, waiting to proceed to step S5300 for unified processing. Through this on-demand, time-slot-based backtracking download mechanism, the business server avoids the huge bandwidth waste caused by indiscriminately pulling all-time, all-user audio streams in traditional solutions, triggering high-precision data download only when there is suspected violation.

[0031] Step S5300: The server performs transliteration recognition on the original audio of each user and locates the offending user containing sensitive information based on the transliterated original audio.

[0032] The server in this application does not perform comprehensive violation detection on all audio data generated within the channel. Instead, it triggers detection of audio data from specific users within a particular time period based on the violation detection results from the client. Therefore, after completing step S5200 and obtaining the original audio of each user speaking within the time period corresponding to the mixed transliteration in the current channel, the server immediately performs transliteration recognition on the original audio of each user and locates the violating user containing sensitive information based on the transliterated original audio.

[0033] First, on the business server side or a designated computing node, an independent transliteration recognition operation is performed on each acquired raw audio data stream. The raw audio is a clean audio stream from a single user, without any mixing processing, preserving the speaker's independent voiceprint characteristics and speech content. The business server invokes an automatic speech recognition model deployed on the server side. This model can adopt the same model architecture as the client side in step S5100, or it can adopt a higher-precision model optimized for the server environment to convert each raw audio signal into corresponding text content, i.e., transliteration of the original audio. To improve conversion efficiency, in one embodiment, the server can concurrently invoke the automatic speech recognition model to perform transliteration recognition on the raw audio of multiple users.

[0034] After completing the transliteration recognition of all original audio and generating corresponding transliterations, the business server performs a violation user location operation. Specific embodiments of this location operation include several. In the first embodiment, the business server matches each transliteration with a preset prohibited keyword database. The prohibited keyword database stores a set of words used to determine violations, including but not limited to sensitive words containing word roots and derivatives. The matching method can employ precise string matching or an edit distance-based fuzzy matching algorithm. When a transliteration contains any keyword from the prohibited keyword database, it is determined that the transliteration contains sensitive information. In the second embodiment, the business server inputs each transliteration into a pre-trained deep learning neural network classification model. This model is trained on a large-scale dataset of violating texts and can output the probability or confidence level of the text belonging to a violation category. When the probability or confidence level exceeds a preset threshold, the corresponding transliteration is determined to contain sensitive information.

[0035] When a transliteration of a given original audio track is determined to contain sensitive information using any of the above embodiments, the business server locates the offending user based on the principle of data correlation. Since each original audio track and its corresponding transliteration are uniquely bound to a specific user identifier during data generation and processing, the business server can uniquely identify the user identifier corresponding to the transliteration containing sensitive information as the offending user. This dual-location mechanism based on time correlation and user binding effectively solves the technical problem of not being able to directly determine responsibility in mixed audio identification, achieving a complete closed loop from channel-level initial screening to precise user-level punishment.

[0036] It should be noted that when the client and / or server use the prohibited keyword database to detect sensitive information, the determination of sensitive information in this application may be related to the application scenario, and its specific content may vary depending on the business model of the internet product. For example, in voice chat rooms or online gaming scenarios, sensitive information includes, but is not limited to, words and their derivatives involving violence, pornography, reactionary content, or gambling, such as illegal content spread by users through homophones or variant characters; in online meetings or multi-party calls, sensitive information may focus on words such as those related to the disclosure of trade secrets or abusive and defamatory remarks. By matching the mixed transliteration or original transliteration with a pre-set prohibited keyword database customized for different scenarios, the aforementioned violations generated in different application environments can be identified more accurately, thereby achieving differentiated compliance control.

[0037] Furthermore, there are various specific implementations regarding the number of target clients and disaster recovery mechanisms used for transliteration recognition within the same channel. In one embodiment, the service server schedules only a single client within the current channel as the target client to perform transliteration recognition and preliminary violation detection, and configures a backup takeover mechanism for it. That is, when the target client experiences communication anomalies such as network interruption, process crash, or heartbeat timeout, the service server detects the corresponding event and automatically triggers a failover process, rescheduling other clients within the channel to replace the abnormal client and perform the task, thus ensuring the continuity of detection. In another embodiment, the service server schedules two or more clients in parallel as target clients to simultaneously perform transliteration recognition and preliminary violation detection. In this case, the service server uses a decision arbitration mechanism to process the results reported from multiple sources, specifically including but not limited to selecting the first valid mixed transliteration as the decision basis, or performing confidence-weighted fusion, voting, and other data aggregation operations on the mixed transliterations reported by multiple clients, and then determining whether to trigger subsequent server-side transliteration recognition and violation detection for the original audio based on the fusion result.

[0038] As can be easily understood from the above embodiments, this application, by constructing a fusion architecture of server-side coordination and client-side distributed computing, has achieved significant beneficial effects in the field of group audio compliance review. It effectively solves the server-side resource bottleneck problem caused by real-time review of massive amounts of group audio, and significantly reduces the operating costs of enterprises. By performing preliminary identification of channel mixing data on the client side, and only triggering the server to backtrack and download the original audio of each user for accurate responsibility when suspected sensitive information is found, without increasing additional bandwidth costs, it successfully offloads most of the high-load transliteration recognition tasks from the server to the client for edge computing, achieving a perfect balance between review efficiency and resource consumption. Specifically, the beneficial effects achieved by this application include, but are not limited to, the following aspects: Firstly, regarding reducing server-side computing power and hardware costs, traditional solutions rely on the server to independently perform transliteration recognition on each user's audio stream, requiring the deployment of large-scale GPU clusters to support high-concurrency computing. This application, however, receives the transliterated audio data from the client's current channel mix data, allowing the server to process only text data instead of the raw audio stream. Since most users are silent in group conversations, the effective speech percentage in the mix data is low. This method of filtering invalid audio on the client side and then processing and simplifying the text on the server side significantly reduces the workload of deep learning required on the server. Simultaneously, utilizing the client's idle CPU resources to complete the main transliteration work avoids reliance on expensive GPU resources, thus significantly reducing the enterprise's capital expenditure on machine procurement and maintenance.

[0039] Secondly, regarding bandwidth resource consumption control, conventional server-side review models require real-time retrieval of the raw audio streams from all users within a channel. As the number of users in a channel increases, the uplink and downlink bandwidth pressure increases linearly. This application cleverly utilizes the inherent characteristics of group voice communication, namely, that the client needs to receive the mixed audio data within the channel in order to play sound. By directly reusing this existing data path, the transliteration of the mixed audio data is completed locally on the client, without the need to upload any additional audio data to the server. Only when the transliteration of the mixed audio hits sensitive information—a very low-probability event—does the server selectively download the raw audio of the users who spoke within the corresponding time period. This on-demand backtracking and precise download mechanism completely avoids the bandwidth waste caused by indiscriminately retrieving all users' audio in traditional solutions, effectively curbing the surge in bandwidth costs.

[0040] Furthermore, this application balances the accuracy of violation detection with the precision of liability determination. By identifying whether the transliterated audio mix contains sensitive information, the vast majority of normal voice channels can be quickly filtered out, achieving efficient initial screening. In the subsequent source tracing stage, by acquiring the original audio of each user speaking within the time period corresponding to the transliterated audio mix in the current channel, and performing transliteration recognition on the original audio separately, the voices of different users in the mix can be accurately distinguished, thereby precisely locating the specific user containing sensitive information. This two-level review mechanism based on time correspondence utilizes the efficiency of mix recognition while ensuring the accuracy of penalties through secondary recognition of the original audio, effectively solving the technical problem of being unable to determine liability solely based on mix recognition, and constructing an efficient, accurate, and low-cost audio compliance protection system.

[0041] Based on any embodiment of the method in this application, the server receives the transliterated audio data of the current channel obtained by the client through transliteration recognition, including: Step S5110: Call the device capability information of each client in the current channel, and select at least one client as the target client based on the device capability information; The server can select target clients for transliteration recognition and violation detection within the current channel based on the client's device capability information. This device capability information can be reported by the client individually or iteratively, and the server collects and temporarily updates it, for example, by storing it in a device activity table for subsequent selection decisions and scheduling. In this application, device capability information serves as a key basis for the server to assess whether a client is qualified to perform transliteration tasks and its execution efficiency. The specific content of this information can be divided and combined in different dimensions in various embodiments of this application.

[0042] Device capability information includes, but is not limited to, the following categories: The first category is device support information, which characterizes whether the client's hardware and system environments meet the minimum requirements for running the transliteration function resource package. Specifically, this may include the CPU architecture type, operating system version number, and whether it has dedicated instruction set acceleration support. The second category is resource package status information, which characterizes the client's current software readiness status. Specifically, this may include whether the transliteration function resource package is pre-installed, the resource package version number, and the integrity verification result. The third category is operational capability parameters, which characterize the client's real-time load and performance margin at the current moment. Specifically, this may include the current CPU utilization rate, memory usage rate, remaining storage space, and the number of idle threads. The fourth category is resource capability parameters, which characterize the inherent static performance limits of the client hardware. Specifically, this may include the number of CPU cores, total memory capacity, graphics card model, and floating-point arithmetic capability. The fifth category is network environment type, which characterizes the network scenario in which the client is located. Specifically, this may include whether it is an internet cafe device, an enterprise office device, a home broadband device, or a mobile data network device. This information can be used to assist in download scheduling decisions.

[0043] In specific embodiments of selecting target clients based on device capability information, there are several feasible selection mechanisms. In one embodiment, the server adopts a performance-based selection mechanism, that is, based on the resource capability parameters and operational capability parameters in the device capability information, a comprehensive performance score is calculated for each client. The scoring algorithm can assign a first weight to the number of CPU cores, a second weight to the total memory capacity, and a negative weight to the current CPU utilization rate. The higher the score, the more suitable the device is for executing the task. The server selects the top N clients with the highest scores as target clients.

[0044] In another embodiment, the server adopts a rule-based hierarchical filtering mechanism. First, it filters out clients with low system versions or those that do not support instruction sets based on device support information. Second, it filters out clients with pre-configured resource packages based on resource package status information to form an initial set. Third, it filters out high-load clients with CPU utilization exceeding a corresponding preset percentage or memory utilization exceeding a corresponding preset percentage based on runtime capability parameters. Finally, it selects the client with the most CPU cores and the largest memory from the remaining clients as the target client based on resource capability parameters.

[0045] In another embodiment, the server can adopt a load-balanced round-robin mechanism. In the set of clients with pre-set resource packages and current load below a preset threshold, a specified number of target clients are selected in round-robin order according to the hash value of the client identifier or the earliest registration time, so as to ensure that the frequency of each client undertaking the transliteration task is relatively balanced and to avoid a single device running at high load for a long time.

[0046] Step S5120: Send a transliteration scheduling instruction to the target client so that the target client switches to the transliteration function enabled state and starts transliteration recognition of the mixed data of the current channel; The transliteration scheduling command is a control signaling communication between the service server and the target client used to control the lifecycle of the transliteration task. Triggering this command signifies that the target client has been officially authorized to execute the edge computing task. In one embodiment, the transliteration scheduling command is a unicast command. The service server connects via User Datagram Protocol (UDP) or Transmission Control Protocol (TCP) and independently sends a command message containing a start identifier, task priority, and heartbeat interval parameters to each selected target client. In another embodiment, the transliteration scheduling command can also be multicast or broadcast. The service server broadcasts the command to all clients within the current channel. Only clients whose device identifiers match the target client list will respond and execute the enable operation. This method is suitable for scenarios with relatively fixed channel members and can effectively reduce signaling interaction overhead.

[0047] Upon receiving the transliteration scheduling instruction, the target client performs a state switching operation, transitioning from standby or normal voice reception to the transliteration function enabled state. This state switching process includes initializing the local transliteration engine, loading a pre-configured transliteration function resource package, and activating the audio stream capture pipeline. Specifically, the client's internal audio processing module intercepts the mixed audio data stream forwarded by the media server and redirects it to the local automatic speech recognition model for processing. Simultaneously, the client begins to encapsulate and report the generated mixed transliteration according to preset time intervals or data volume thresholds.

[0048] To ensure the reliability and timeliness of scheduling, the execution process of transliteration scheduling instructions includes several specific control implementations. In one embodiment, the service server employs a retransmission mechanism with acknowledgment. After sending the transliteration scheduling instruction, a timer is started. If no task initiation acknowledgment response is received from the target client within a preset time, it is determined that the instruction was lost or the client is abnormal. The service server will then reschedule other candidate clients to execute the task. In another embodiment, the service server includes a current server timestamp when sending the instruction. The target client must send back this timestamp for verification when responding to the instruction to prevent replay attacks or process expired instructions, ensuring the consistency between the transliteration recognition task and the server's timeline.

[0049] Furthermore, the transliteration scheduling instructions can include fine-grained control parameters to adapt to different business scenarios. For example, in a voice chat room scenario, the instructions can be set with a higher task priority and a shorter heartbeat interval to ensure the real-time nature of violation detection; while in an online meeting scenario, the instructions can be set with a lower resource utilization limit, allowing the client to automatically pause transliteration recognition when the system is busy, prioritizing the user's call quality. Through this dynamically adjusted scheduling strategy, the business server can achieve efficient audio review while minimizing the impact on the client's normal voice interaction experience.

[0050] Step S5130: Receive the mixed transliteration reported by the target client in response to the transliteration scheduling instruction.

[0051] The service server is responsible for receiving the mixed transliteration reported by the target client in response to the transliteration scheduling command. This mixed transliteration is the text result generated by the target client after transliterating the mixed data of the current channel when the transliteration function is enabled.

[0052] There are several specific implementation methods for the target client to report mixed audio translations. In the first embodiment, the client adopts a data volume-based triggering reporting mechanism. That is, whenever the locally cached mixed audio data reaches a preset byte threshold or time slice length, the client automatically calls the automatic speech recognition model to complete the recognition and sends the generated mixed audio translation to the business server after encapsulation via network protocol. In the second embodiment, the client adopts an event-based triggering reporting mechanism. That is, the mixed audio translation is only reported when the recognition result changes or when valid text is recognized for the first time, in order to reduce the transmission of redundant data.

[0053] The reported transliterated audio mix is ​​typically encapsulated into a specific data structure during network transmission. This data structure, in addition to containing the identified text content, also includes a timestamp of the text within the original audio mix. This timestamp defines the specific time interval, or time period, in which the sensitive information occurred. In one embodiment, the data structure may further include a client identifier, a channel identifier, and a confidence score for subsequent verification and arbitration by the service server.

[0054] To ensure the reliability and real-time performance of data transmission, a long-lived connection channel can be established between the business server and the target client. In one embodiment, when the business server receives the mixed transliteration, it returns an acknowledgment response to the target client. If the client does not receive the response within a specified time, a retransmission mechanism is triggered, thereby ensuring the integrity of the mixed transliteration data received by the business server.

[0055] The organic whole constructed through the above embodiments effectively overcomes the practical dilemma of server bandwidth congestion and GPU computing power exhaustion caused by the indiscriminate retrieval of full audio in traditional solutions in high-concurrency group scenarios such as voice chat rooms and online gaming. By utilizing device capability information, dynamic orchestration and load balancing of client computing resources are achieved, enabling the transliteration task to be accurately pushed down to edge nodes with sufficient performance. This ensures the real-time interception of illegal content while offloading the high-energy-consuming audio transliteration load from the central server to a large number of user terminals, significantly reducing the marginal expansion cost of enterprise infrastructure and improving the overall robustness of the system.

[0056] Based on any embodiment of the method in this application, selecting at least one client as the target client according to the device capability information includes: Step S5111: Determine the clients in the current channel that have pre-installed transliteration function resource packages to form the initial client set; The server in this application has pre-collected device capability information for each client device within the channel. Based on this, it can perform this step using the resource package status information to identify clients in the current channel that have a pre-installed transliteration function resource package, and then form an initial client set from these clients. The resource package status information is data used to characterize the current software readiness status of the client, specifically including whether a transliteration function resource package has been pre-installed, the resource package version number, and the integrity verification result. In one embodiment, the server queries the device activity table to filter out clients that are marked as pre-installed and have passed verification in the resource package status information. These clients possess the software foundation to perform local transliteration recognition and are thus included in the initial client set, providing a data source for subsequent refined filtering.

[0057] Step S5112: Based on the operating capability parameters in the device capability information, select clients whose operating capability parameters match the preset operating capability conditions from the initial client set to form an available client set; The initial client set contains clients of varying performance. To ensure service stability, the server can perform this step to remove clients that are currently under excessive load and unable to stably perform transliteration tasks, thus ensuring that the final selected target clients have the ability to respond in real time.

[0058] The operational capability parameters are data used to characterize the client's real-time load and performance margin at the current moment. In this embodiment, these parameters may include, but are not limited to, any one or more of the following: current CPU utilization, memory usage, remaining storage space, and number of idle threads. Correspondingly, the preset operational capability conditions are performance threshold standards pre-configured by the business server to determine whether the client is suitable for performing transliteration tasks.

[0059] The screening process can be implemented in various ways. In one embodiment, the business server employs a filtering mechanism based on hard thresholds, setting a maximum CPU utilization of, for example, 70%, and a maximum memory utilization of, of 80%. For each client in the initial client set, based on the corresponding operational capability conditions, if its operational capability parameters show a CPU utilization of less than 70% and a memory utilization of less than 80%, then the client is determined to meet the preset operational capability conditions and is thus included in the available client set. Conversely, if either indicator exceeds the limit, the client will be excluded to avoid delays or failures in transliteration recognition due to device overheating or lag.

[0060] In another embodiment, the business server employs a weighted filtering mechanism based on remaining resource availability. This mechanism considers not only the aforementioned percentage occupancy rate but also a comprehensive judgment based on remaining storage space and the number of idle threads. For example, the preset operating capacity conditions can be set as follows: remaining storage space greater than 500MB and the number of available threads greater than or equal to 2. The business server iterates through the initial client set, retaining only clients that simultaneously meet all the above resource availability conditions, thus forming the available client set. This ensures that clients will not crash due to resource exhaustion when running the transliteration function for extended periods.

[0061] Step S5113: Determine whether the total number of target clients in the available client set that are in the state of having the transliteration function enabled exceeds a preset number threshold. If the preset number threshold is not reached, according to the number of clients required to reach the preset number threshold, select a corresponding number of clients with relatively better resource capability parameters from the available client set as target clients based on the resource capability parameters in the client's device capability information.

[0062] After determining the set of available clients, the service server first checks whether the total number of target clients in the available client set that are already in the transliteration function enabled state exceeds a preset threshold. The preset threshold is an upper limit set by the service server to ensure the redundancy and resource utilization of transliteration tasks within a channel. In one embodiment, this threshold can be set to 2, meaning that a maximum of two clients can simultaneously execute transliteration recognition tasks within a channel, forming a stable architecture of one primary and one backup. The service server queries the device activity table to count the number of target clients currently scheduled and in working state within the channel. If this number has reached or exceeded the preset threshold, no new clients are started, and the existing calculation results are directly reused.

[0063] When the judgment result shows that the total number of target clients in the available client set that are in the transliteration function enabled state has not reached the preset threshold, the business server will calculate the shortage number, which is equal to the preset threshold minus the number of currently enabled target clients. Subsequently, based on the resource capability parameters in the client's device capability information, the business server will select a corresponding number of clients with relatively better resource capability parameters from the available client set to fill the shortage.

[0064] There are several specific implementation methods for selecting clients with relatively superior resource capability parameters. In the first embodiment, the business server adopts an absolute selection mechanism based on hardware performance, that is, it calculates based on the number of CPU cores and total memory capacity in the resource capability parameters. The business server traverses the set of available clients and selects the client with the most CPU cores; if multiple clients have the same number of CPU cores, it further selects the client with the largest total memory capacity. If the number of missing clients is 2, the above process is repeated, and the top two clients are selected as the target clients.

[0065] In the second embodiment, the service server employs a weighted scoring-based comprehensive ranking mechanism. This mechanism assigns preset weights to various indicators in the resource capability parameters; for example, the number of CPU cores is assigned a first weight value, the total memory capacity a second weight value, and the GPU's floating-point performance a third weight value. The service server calculates a weighted total score for each client in the available client set, using the formula: Total Score = Number of CPU Cores × First Weight + Total Memory Capacity × Second Weight + Floating-Point Performance × Third Weight. Subsequently, the service server sorts the clients according to their total scores from highest to lowest and selects the top N clients as target clients, where N equals the number of vacancies calculated above.

[0066] Through this on-demand and optimal selection mechanism, the business server can ensure that the optimal number of transliteration execution nodes are always maintained in each channel. This avoids detection interruptions caused by the failure of a single node, and also prevents the waste of terminal resources and vicious competition for computing resources within the channel caused by too many clients executing tasks at the same time.

[0067] The organic whole constructed through the above embodiments effectively solves the problems of terminal resource contention and server load fluctuation caused by indiscriminate scheduling in traditional review schemes in group voice scenarios. By leveraging the linkage of multi-dimensional device capability information, a three-level progressive screening is achieved from software readiness, real-time load margin to hardware performance limit. This ensures that transliteration tasks are accurately deployed only to edge nodes with sufficient performance and stable status. Thus, while ensuring the real-time detection capability of illegal content, it avoids the situation of low-performance devices crashing due to overload or high-performance devices being idle and wasted, significantly improving the overall utilization rate of client edge computing resources and the robustness of system operation.

[0068] Based on any embodiment of the method in this application, before the server receives the transliterated audio data of the current channel obtained by the client through transliteration recognition, the following steps are included: Step S3100: Receive device capability information and associated channel identifier reported by any client in the current channel, and store the association in the device activity table; To generate the device activity table, the business server, acting as the server-side component, is responsible for receiving device capability information and associated channel identifiers reported by any client in the current channel, and storing this information in association. As mentioned earlier, the device activity table is a logical database table maintained by the business server for centralized management of the status and performance data of each client within the channel. It records client identifiers, channel identifiers, and multi-dimensional device capability information, serving as the sole data source for subsequent scheduling decisions.

[0069] There are several specific implementations regarding the timing and method of client-side reporting of device capability information. In one implementation, an event-triggered reporting mechanism is used. When the client successfully joins the current channel, it proactively sends a registration message containing complete device capability information to the business server. This message is generated when the user clicks to join the channel or when the channel status changes to active, ensuring the business server has the latest device status before entering voice interaction. In another implementation, an incremental reporting mechanism based on status changes is used. The client continuously monitors its local status during operation and only reports updated specific field data to the business server when there are significant changes in device capability information, such as a CPU utilization rate exceeding a preset threshold for 5 seconds, a decrease in remaining storage space exceeding 1GB, or the completion of the transliteration function resource package download, thereby reducing the transmission of redundant data. In yet another implementation, a full reporting mechanism based on timed polling is used. The client collects and reports complete device capability information at preset fixed time intervals, such as every 60 seconds, to maintain the timeliness of data in the device activity table.

[0070] The reported device capability information includes multiple data types associated with the same channel identifier. In addition to the device support information, resource package status information, operational capability parameters, resource capability parameters, and network environment type mentioned earlier, the reported data also includes a unique session identifier for the client within the channel, used to bind device performance data to a specific voice session. Upon receiving this data, the service server performs a related storage operation, specifically creating or updating a record in the device activity table. This record uses the client identifier as the primary key and the channel identifier as the index, integrating the scattered performance parameters into structured data entries.

[0071] Step S3200: Based on the device support information and resource package status information in the device capability information of each client in the device activity table, determine whether the client supports the transliteration function and whether there is currently no pre-installed transliteration function resource package; Based on the device activity table, the business server can determine whether a client supports the transliteration function and whether a transliteration function resource package is currently pre-installed, by referring to the device support information and resource package status information in the device capability information of each client in the device activity table. This determination operation is the admission control of the download scheduling process, aiming to filter out clients that have the potential to run the software but have not yet obtained the necessary computing resources.

[0072] The specific decision-making logic involves joint verification of two types of information. First, the business server reads the device support information to verify whether the client's hardware architecture type is compatible, whether the operating system version number meets the minimum requirements, and whether it has dedicated instruction set acceleration support. If the device support information shows that the client does not meet any of the above conditions, such as an operating system version that is too low or a central processing unit architecture that is not supported, then it is determined that the client does not support the transliteration function, the process terminates, and no further download operations are performed.

[0073] Then, for clients that pass the device support verification, the business server further reads their resource package status information to check whether the transliteration function resource package is pre-configured, whether the resource package version number is up-to-date, and whether the integrity verification result is passed. If the resource package status information shows that the client has pre-configured it and the verification is passed, it is determined to be in a ready state, and the download process is terminated to avoid duplicate downloads. Only when the device support information is determined to be supported, and the resource package status information is determined to be not pre-configured or the verification failed, will the business server output a positive judgment result and trigger the subsequent step S3300.

[0074] Step S3300: When the judgment result is supported and not preset, according to the download bandwidth load capacity information corresponding to the current channel, the transliteration function resource package is sent to the corresponding client to drive the client to preset the transliteration function resource package and switch to the transliteration function enabled state.

[0075] In response to the aforementioned affirmative judgment result—that is, when the judgment result is supportive and not pre-configured—the business server sends the audio-to-text translation function resource package to the corresponding client based on the download bandwidth load capacity information corresponding to the current channel. Download bandwidth load capacity information reflects the current outbound bandwidth pressure of the server cluster to which the business server belongs, and is a key control parameter to prevent network congestion caused by large-scale concurrent downloads.

[0076] The specific implementation methods of the phonetic transcription function resource package include several types. In the first embodiment, the service server adopts a direct delivery mechanism based on real-time load, that is, it monitors the current download bandwidth load capacity information in real time. If the monitored value is lower than the preset safety threshold, it immediately delivers the complete phonetic transcription function resource package to clients that are determined to support it but have not pre-installed it, through a file distribution service or a peer-to-peer transmission protocol.

[0077] In the second embodiment, the service server employs a delayed delivery mechanism based on a backoff strategy, which is particularly suitable for scenarios with limited bandwidth resources. When the current download bandwidth load information triggers bandwidth limitations, the service server does not deliver the data directly but instead initiates a backoff timer. A specific application scenario involves implementing differentiated backoff duration strategies for internet cafe devices and non-internet cafe devices, as well as the number of clients with pre-installed resource packages within the channel. For example, if no client has pre-installed a resource package within the channel and the current client is a non-internet cafe device, a first backoff duration is applied if bandwidth limitations are triggered; if it is an internet cafe device, a longer second backoff duration is applied. After the backoff duration expires and bandwidth pressure eases, the service server then delivers the audio-visual translation function resource package to the client. In this way, the service server can systematically deploy client-side edge computing capabilities while ensuring the stability of core services.

[0078] In large-scale deployment scenarios of group voice services, the above embodiments effectively solve the problems of server bandwidth peak impact and terminal storage resource waste caused by the blind distribution of computing resources in traditional solutions. They utilize device activity tables to achieve accurate profiling of client hardware and software status, and ensure that download commands only reach edge nodes with potential through dual filtering of support verification and resource package status. Furthermore, they combine real-time bandwidth load and network environment type to execute differentiated fallback strategies, thereby ensuring the orderly distribution of transliteration capabilities while smoothing server outbound traffic, significantly improving resource distribution efficiency and system stability in high-concurrency environments.

[0079] Based on any embodiment of the method in this application, according to the download bandwidth load capacity information corresponding to the current channel, the transliteration function resource package is sent to the corresponding client, including: Step S3310: When the current download bandwidth load capacity information triggers bandwidth limitation, determine the download yield strategy to be executed on the client based on the network environment type of each client and the number of clients in the channel where the client is located that have the transliteration function resource package pre-set. Download bandwidth load capacity information refers to the current outbound bandwidth utilization or throughput data of the server cluster to which the business server belongs. When this data exceeds a preset security threshold, it is determined that bandwidth limitation has been triggered. In this case, directly sending resource packets will lead to network congestion risks. If bandwidth limitation has not been triggered, the business server can directly execute step S3320 to immediately send resource packets to the client.

[0080] When a bandwidth limit is triggered, the service server determines the download backoff strategy based on two key dimensions. The first dimension is the network environment type of each client, which is stored in the device activity table to distinguish the network scenario in which the client is located, specifically divided into internet cafe devices and non-internet cafe devices. The second dimension is the number of clients in the channel that have the pre-installed transliteration function resource package, which reflects the redundancy of edge computing capabilities in the current channel.

[0081] In one embodiment, the business server identifies whether a client belongs to the internet cafe type, mainly by parsing the network environment type field in the device capability information reported by the client. This field is determined by the client based on its public network exit IP address. The specific process usually involves querying a third-party IP geolocation and scenario database. If the IP address is marked as a public place or has high-concurrency device connection characteristics, it is classified as an internet cafe device; otherwise, it is classified as a non-internet cafe device. The business server executes a differentiated download yield strategy based on the classification result.

[0082] The specific implementation of the download backoff strategy includes various rule-based combination logics. In the first embodiment, the service server adopts a four-quadrant decision matrix. When the number of clients with pre-installed resource packages in the channel is zero and the client is not a network cafe device, if bandwidth limitation is triggered, a backoff strategy for a first backoff duration is executed; when the number of clients with pre-installed resource packages in the channel is zero and the client is a network cafe device, since network cafe environments typically have ample bandwidth but high concurrency, a backoff strategy for a second backoff duration is executed, where the second backoff duration is longer than the first backoff duration; when the number of clients with pre-installed resource packages in the channel is one and the client is not a network cafe device, a backoff strategy for a third backoff duration is executed; when the number of clients with pre-installed resource packages in the channel is one and the client is a network cafe device, a backoff strategy for a fourth backoff duration is executed.

[0083] In the second embodiment, the service server employs a dynamic weighted backoff duration calculation mechanism. This mechanism assigns a first environment weight to the network environment type and a second quantity weight to the number of clients with pre-configured resource packets. The service server calculates the specific backoff duration for each client using the formula: Backoff Duration = Base Backoff Value + First Environment Weight × Adjustment Coefficient + Second Quantity Weight × Adjustment Coefficient. For example, non-internet cafe devices have a negative weight, while devices with a pre-configured quantity of 1 have a positive weight; differentiated waiting times are obtained through weighted calculation.

[0084] In the third embodiment, the service server employs a priority-based fallback strategy. The service server adds all pending client requests to a priority queue, ordered based on a combination of network environment type and pre-set quantity. For example, clients without resource packets in the channel and not located in internet cafes have the lowest priority and are placed at the end of the queue, requiring a longer fallback period; while clients with one resource packet in the channel and located in internet cafes have higher priority and shorter fallback periods. The service server then sends out requests sequentially according to the queue order after the fallback period expires.

[0085] After determining the download backoff strategy through any of the above embodiments, the service server enters a waiting state until the backoff duration set in the strategy is met. During the backoff period, the service server continuously monitors the download bandwidth load capacity information. If the bandwidth pressure is relieved in advance, the backoff can be terminated early and the download operation can be executed, thereby achieving fine-grained management and protection of server-side bandwidth resources.

[0086] Step S3320: According to the download backoff strategy, after the backoff time set in the strategy is met, the transliteration function resource package is sent to the client.

[0087] When executing step S3320, the business server waits for the set backoff duration to be met according to the determined download backoff policy. The start point for the backoff duration is usually the moment when the business server determines that bandwidth limitation has been triggered and generates the backoff policy. During the waiting period, the business server continuously monitors download bandwidth load capacity information, i.e., the outbound bandwidth utilization or throughput of the server cluster. If, before the backoff duration expires, the bandwidth pressure is detected to have fallen below the preset safety threshold, the business server can terminate the waiting state early and immediately execute the distribution operation, thereby improving the distribution efficiency of resource packets.

[0088] Once the fallback period is met or bandwidth pressure is alleviated ahead of schedule, the service server distributes the audio-visual translation function resource package to the corresponding client. There are several implementation methods for this distribution. In the first embodiment, the service server uses a direct download mechanism based on the Hypertext Transfer Protocol (HTTP). This involves sending a command containing a download link for the resource package to the client via a deployed file distribution service. The client then directly downloads the complete compressed package through this link. This method is suitable for scenarios with relatively stable networks, such as those outside internet cafes. In the second embodiment, the service server uses a distributed distribution mechanism based on point-to-point transmission. This involves using other clients within the channel that have pre-installed resource packages as seed nodes to transmit fragments of the resource package to the clients awaiting distribution. This method effectively reduces the bandwidth pressure on the central server and is particularly suitable for high-concurrency environments such as internet cafes.

[0089] After receiving the transliteration function resource package, the client performs local preset operations. These operations include verifying the integrity of the resource package, decompressing the files, and loading the automatic speech recognition model and decoder dependency libraries. Upon successful preset, the client's state switches from standby to transliteration function enabled, and it returns a successful preset confirmation message to the business server. Upon receiving the confirmation message, the business server updates the client's resource package status information in the device activity table, marking it as preset and verified, thus completing the entire download scheduling loop.

[0090] The organic whole constructed by the above embodiments in this application effectively solves the problem of instantaneous peak bandwidth and network congestion caused by the indiscriminate and high-concurrency distribution of computing resources in the operation and maintenance scenario of massive group voice services. By dynamically generating differentiated yield strategies using the dual dimensions of network environment type and existing resource distribution within the channel, it realizes peak shaving and valley filling and orderly scheduling of bandwidth resources, significantly reducing the expansion pressure of enterprise network infrastructure, while ensuring that edge computing nodes obtain the necessary computing power support at the appropriate time.

[0091] Based on any embodiment of the method in this application, the download fallback strategy includes: when the number of clients with the pre-installed transliteration function resource package in the channel is zero and the network environment type of the client is a non-internet cafe device, if bandwidth limitation is triggered, a first fallback duration is triggered; when the number of clients with the pre-installed transliteration function resource package in the channel is zero and the network environment type of the client is an internet cafe device, if bandwidth limitation is triggered, a second fallback duration is triggered; when the number of clients with the pre-installed transliteration function resource package in the channel is one and the network environment type of the client is a non-internet cafe device, if bandwidth limitation is triggered, a third fallback duration is triggered; when the number of clients with the pre-installed transliteration function resource package in the channel is one and the network environment type of the client is an internet cafe device, if bandwidth limitation is triggered, a fourth fallback duration is triggered.

[0092] In this embodiment, when executing the download fallback strategy, the service server performs differentiated fallback duration control based on the combination of the number of clients with pre-installed transliteration function resource packages within the channel and the client's network environment type. The core logic of this strategy is to dynamically adjust the delivery delay based on the redundancy of edge computing capabilities within the current channel and the stability of the client's network, in order to balance bandwidth pressure and review coverage.

[0093] When the number of clients with pre-installed transliteration function resource packages in a channel is zero, it means that there are currently no edge nodes capable of performing local recognition in the channel. The business server needs to deploy the resource package as soon as possible to establish auditing capabilities. In this scenario, if bandwidth limitations are triggered, the business server sets different fallback durations based on the network environment type. For clients in non-internet cafe networks, since they typically represent home or mobile networks with relatively limited and unstable bandwidth resources, the business server implements a fallback strategy with a first fallback duration, which is relatively short to complete the distribution as quickly as possible while ensuring network stability. For clients in internet cafe networks, although their concurrency is high, considering that internet cafes typically have good bandwidth and high device density, to avoid backbone network congestion caused by a sudden surge in downloads, the business server implements a fallback strategy with a second fallback duration, which is longer than the first fallback duration, to achieve a smoother release of traffic.

[0094] When there is only one client with pre-installed transliteration function resource package in a channel, it means that the current channel already has basic unilateral review capabilities, and the urgency for the business server to add new nodes is reduced. In this scenario, if bandwidth limitations are triggered, the business server will also implement differentiated control based on the network environment type. For non-internet cafe devices, the business server will implement a fallback policy with a third fallback duration; for internet cafe devices, a fallback policy with a fourth fallback duration will be implemented. In the entire policy system, the size relationship of each fallback duration is set as follows: the second fallback duration is longer than the first fallback duration, the fourth fallback duration is longer than the third fallback duration, and the third fallback duration is longer than the first fallback duration. This hierarchical duration setting ensures that when bandwidth is tight, priority is given to ensuring the stability of channels with existing review capabilities, and resource packages are prioritized for distribution to nodes with better network environments.

[0095] In one embodiment, the aforementioned backoff duration can be implemented using specific numerical values, such as setting the first backoff duration to 1 second, the second backoff duration to 5 seconds, the third backoff duration to 7 seconds, and the fourth backoff duration to 9 seconds. After the backoff duration expires, the service server sends the pronunciation and translation function resource package to the corresponding client, thereby completing the scheduling.

[0096] Through the above embodiments, in the resource distribution scenario of large-scale group voice services, the present application effectively solves the problem of the peak impact on the server bandwidth caused by the traditional solution's blind distribution regardless of the network environment and node distribution. By using the retreat strategy matrix in the four quadrants, it realizes refined traffic control based on the computing power redundancy within the channel and the terminal network type, significantly smoothes the server egress bandwidth curve, reduces the risk of network congestion, and at the same time ensures that the audio and translation capabilities can sink to the edge nodes in an orderly manner according to the priority.

[0097] Based on any embodiment of the method of the present application, identifying whether there is sensitive information in the mixed audio and translation text includes: matching the mixed audio and translation text with a preset prohibited keyword library. When the mixed audio and translation text hits a keyword in the prohibited keyword library, it is determined that the mixed audio and translation text contains sensitive information.

[0098] In this embodiment, after receiving the mixed audio and translation text reported by the target client, the business server performs a sensitive information identification operation, which is specifically embodied as matching the mixed audio and translation text with a preset prohibited keyword library. The prohibited keyword library is a structured data set stored in the business server or an associated database, which contains a list of words used to determine violations. The design of this list is not limited to complete standard words, but especially includes roots and derivatives optimized for Internet slang to cope with variant avoidance behaviors.

[0099] The specific implementation methods of the matching process include multiple technical paths based on string comparison. In the first embodiment, the business server uses an exact string matching algorithm, that is, the mixed audio and translation text is disassembled by word segmentation or character granularity and compared one by one with the entries in the prohibited keyword library. When it is detected that the mixed audio and translation text completely contains any keyword, it is determined that the mixed audio and translation text contains sensitive information. In the second embodiment, the business server uses a fuzzy matching algorithm based on the edit distance. This algorithm calculates the similarity score between the string in the mixed audio and translation text and the entries in the prohibited keyword library. When the score exceeds the preset similarity threshold, it is determined to be a hit. This is particularly effective for identifying violations where users avoid censorship by using homophonic words, typos, or inserting special symbols. For example, when the sensitive word "gambling" is rewritten as "lottery" or "gambling*", the fuzzy matching can still capture it.

[0100] The determination result will directly drive the subsequent process. When it is determined that the mixed audio and translation text contains sensitive information, the business server will trigger step S5200 to start the original audio backtracking process for a specific time period; otherwise, if it does not hit, the current detection cycle ends, and the business server continues to receive the mixed audio and translation text in the next cycle for polling detection.

[0101] In the real-time risk control scenario of group voice content, this application effectively solves the problem of low recognition accuracy caused by direct processing of high-noise mixing in traditional solutions. By using a prohibited keyword library containing word roots and derivatives, combined with a dual matching mechanism of precise and fuzzy matching, it achieves efficient interception of variant illegal words. Thus, while ensuring the low bandwidth consumption advantage of edge computing, it significantly improves the recall and precision in the initial screening stage.

[0102] Based on any embodiment of the method in this application, the method for locating users who violate regulations by identifying transliterated original audio information includes: matching each transliterated original audio information with a preset prohibited keyword database; when the transliterated original audio information matches a keyword in the prohibited keyword database, it is determined that the transliterated original audio information contains sensitive information; and the user corresponding to the transliterated original audio information containing sensitive information is identified as a violating user.

[0103] In this embodiment, after completing step S5200 and obtaining the original audio of each user speaking within the time period corresponding to the mixed transliteration in the current channel, the service server immediately performs transliteration recognition on each original audio stream to generate the corresponding original audio transliteration. Subsequently, the service server matches each original audio transliteration with a preset prohibited keyword database to locate specific violating users. The application of the prohibited keyword database in this stage is consistent with the initial screening stage in step S5100 above. It contains a vocabulary set of word roots and derivatives, designed to address variant prohibited terms in the network environment.

[0104] There are several specific implementation methods for matching and location. In a first embodiment, the business server employs a precise matching mechanism based on user identifier binding. Since each original audio stream and its corresponding transliterated version are uniquely bound to a specific user identifier during data acquisition and transmission, the business server simultaneously verifies the user identifier corresponding to the transliterated version of the matched text while traversing the prohibited keyword database. When a transliterated version is detected to contain any keyword from the prohibited keyword database, the business server immediately determines that the transliterated version contains sensitive information and uniquely identifies the bound user identifier as the violating user.

[0105] In the second embodiment, the service server employs a confidence-weighted accountability mechanism. This mechanism not only performs keyword matching but also semantic analysis of the context of the matched keywords, calculating a confidence score indicating whether the text belongs to a violation category. Only when a keyword is matched and the confidence score exceeds a preset threshold does the service server determine that the transliterated original audio contains sensitive information, and based on the principle of data correlation, confirm the user who sent the original audio as a violator. This approach effectively filters false alarms caused by speech recognition errors or background noise.

[0106] In the third embodiment, the business server employs an arbitration mechanism based on timeline cross-validation. Considering that the mixed audio translations are associated with time periods containing sensitive information, the business server only performs keyword comparisons on the original audio translations within those time periods when performing original audio matching. If a user's original audio translation matches a keyword within that time period, while other users' audio is silent or background noise, that user is prioritized as a high-confidence violator, thus achieving a closed loop from channel-level initial screening to precise user-level accountability.

[0107] In the scenario of tracing the source of violations in group audio, this application effectively solves the technical problem that traditional audio mixing recognition cannot directly determine responsibility. By using the high signal-to-noise ratio of the original audio and combining it with a prohibited keyword library containing word roots and derivatives for secondary verification, it achieves accurate mapping from suspected violation segments to specific responsible persons, significantly improving the accuracy of the audio review system and the fairness of the punishment.

[0108] Based on any embodiment of the method in this application, after locating the offending user containing sensitive information according to the identified transliterated original audio, the method includes: Step S6100: The server sends the original audio of the located violating user, the corresponding transliteration of the original audio, and the violation information to the manual review terminal. After locating the offending user, the business server sends the original audio of the offending user, the corresponding transliterated audio, and the violation information to the human review terminal. As the link between the automated detection process and the human review mechanism, this step aims to utilize human judgment to handle ambiguous areas or controversial cases in machine identification, ensuring the fairness of the punishment.

[0109] The sent data packets contain three key types of information. The first type is the original audio of the violating user. This data is a pure, unprocessed audio stream from a single user, preserving complete voiceprint features and speech content, which can be used by auditors for verification. The second type is the corresponding transliteration of the original audio, i.e., the text result generated by the business server using an automatic speech recognition model, used to help auditors quickly locate suspected violations. The third type is violation information, which includes metadata such as prohibited keywords detected during the initial screening by the business server, the time period of the violation, the channel identifier, and the confidence score calculated by the system, providing decision-making context for manual review.

[0110] There are several specific implementation methods for data transmission. In the first embodiment, the business server adopts a proactive reporting mechanism based on real-time push, that is, through the established WebSocket or HTTP long connection, it encapsulates the above three types of information into a structured review task message and pushes it to the to-do list of the manual review terminal in real time. In the second embodiment, the business server adopts a task pool-based retrieval mechanism, that is, violation cases are stored in the database to be reviewed, and the manual review terminal obtains the list of tasks to be processed by polling or manually refreshing, and downloads the corresponding original audio and transliterated audio according to the index in the list.

[0111] The manual review terminal is a client application deployed on the reviewer's workstation, featuring audio playback, text highlighting, and a review operation interface. Reviewers listen to the original audio on the terminal, compare it with the transliterated original audio and violation information, and make a final compliance determination. If a violation is confirmed, an "approval" operation is performed on the terminal; if it is determined to be a false alarm, a "rejection" operation is performed. The review terminal encapsulates the reviewer's operation results into a review result and returns it to the business server via a network interface, thereby triggering subsequent steps S6200.

[0112] Step S6200: The server receives the review result returned by the manual review terminal, and performs corresponding penalty operations on the user permissions of the violating user in the current channel according to the review result.

[0113] The business server is responsible for receiving the review results returned by the manual review terminal. Upon receiving the review results, it executes corresponding penalty operations on the user permissions of the violating user in the current channel based on the review results. The review results are control instructions generated by the manual review terminal based on the reviewer's comprehensive judgment of the original audio, the original transliteration, and the violation information. These instructions are transmitted back to the business server through the network interface and serve as the sole basis for the final penalty decision.

[0114] There are several specific implementation methods for executing penalties based on the review results, each corresponding to different review conclusions. One or more methods can be flexibly selected and combined. In the first embodiment, if the review result shows that the reviewer has confirmed a violation, the business server will execute an account ban or muting operation. The specific granularity of the penalty can be graded according to the severity of the violation information. For example, if the violation information contains high-risk sensitive words, the violating user will be permanently banned from speaking in the current channel; if the violation is minor, a temporary muting will be executed, such as prohibiting the user from sending voice or text messages for the next 24 hours. The business server modifies the user's access control list in the current channel by calling the permission control interface of the media server or channel management interface, thereby adjusting the user's permissions.

[0115] In the second embodiment, if the review result indicates that the reviewer determined it to be a false alarm, meaning the machine recognition result is incorrect, the business server will perform a penalty revocation or exemption operation. The business server will clear the temporary mark or alarm status on the user's identifier and ensure that subsequent voice data of that user will not be repeatedly sent to the high-priority review process to avoid harassing normal users. Simultaneously, the business server can also feed this false alarm case back to the prohibited keyword database or the optimization module of the automatic speech recognition model to correct fuzzy matching rules or model parameters, thereby reducing the future false alarm rate.

[0116] In the third embodiment, the review result includes instructions requiring supplementary evidence or transfer to a higher-level review. The business server will suspend the execution of penalties for the violating user and forward the relevant data to a higher-level review queue or other related review systems for joint analysis. Upon receiving the final review instruction, the business server will then execute the corresponding penalty operation based on the final review result.

[0117] Regardless of the type of penalty action performed, the business server, after completing the permission adjustment, can return a confirmation message of successful operation to the human review terminal and update the review task status to "completed." Through this human-machine collaborative mechanism, this application ensures efficient initial review of large-scale group audio while utilizing artificial intelligence to handle edge cases, ensuring the accuracy and compliance of penalties.

[0118] Through the above embodiments, this application effectively solves the problem of false positives and false negatives caused by speech recognition noise or semantic ambiguity in the final execution stage of large-scale group audio risk control. By using the structured packaged transmission of original audio, original transliteration, and violation information, a highly efficient collaborative closed loop of automated initial screening and manual review is constructed, which significantly improves the accuracy and credibility of violation penalties. At the same time, the keyword library and recognition model are continuously optimized through the feedback mechanism of false positive cases, realizing the self-evolution of the system's risk control capabilities.

[0119] Please see Figure 3 According to one aspect of this application, an audio violation detection device includes an edge recognition module 5100, an original audio acquisition module 5200, and a violation location module 5300. The edge recognition module 5100 is configured to receive, from a client, a transliterated version of the mixed audio data of the current channel, obtained by transliteration, and identify whether sensitive information exists in the transliterated version. The original audio acquisition module 5200 is configured to, when sensitive information exists in the transliterated version, obtain the original audio of each user speaking within the corresponding time period of the transliterated version in the current channel. The violation location module 5300 is configured to, from a server, perform transliteration on the original audio of each user, and locate the violating user containing sensitive information based on the transliterated version.

[0120] Based on any embodiment of the device in this application, the edge recognition module 5100 includes: a target selection module, configured to call the device capability information of each client in the current channel, and select at least one client as a target client according to the device capability information; a scheduling control module, configured to send a transliteration scheduling instruction to the target client, so that the target client switches to the transliteration function enabled state and starts to perform transliteration recognition on the mixed data of the current channel; and a transliteration receiving module, configured to receive the mixed transliteration reported by the target client in response to the transliteration scheduling instruction.

[0121] Based on any embodiment of the device in this application, the target selection module includes: a resource determination module, configured to determine clients in the current channel that have a pre-installed transliteration function resource package, forming an initial client set; an operation determination module, configured to filter clients from the initial client set whose operation capability parameters match preset operation capability conditions according to the operation capability parameters in the device capability information, forming an available client set; and a device optimization module, configured to determine whether the total number of target clients in the available client set that are in the transliteration function enabled state exceeds a preset number threshold. When the preset number threshold is not reached, the module selects a corresponding number of clients with relatively superior resource capability parameters from the available client set as target clients, based on the resource capability parameters in the client's device capability information, according to the missing number required to reach the preset number threshold.

[0122] Based on any embodiment of the device in this application, prior to the edge recognition module 5100, it includes: an information receiving module, configured to receive device capability information and associated channel identifier reported by any client in the current channel, and store them in a device activity table; a device analysis module, configured to determine whether a client supports the transliteration function and has not currently pre-configured a transliteration function resource package based on the device support information and resource package status information in the device capability information of each client in the device activity table; and a configuration distribution module, configured to, when the determination result is that the client supports the transliteration function and has not pre-configured it, distribute the transliteration function resource package to the corresponding client based on the download bandwidth load capacity information corresponding to the current channel, so as to drive the client to pre-configure the transliteration function resource package and switch to the transliteration function enabled state.

[0123] Based on any embodiment of the device in this application, the configuration delivery module includes: a policy determination module, configured to determine a download backoff policy to be executed on the client when the current download bandwidth load capacity information triggers bandwidth limitation, based on the network environment type of each client and the number of clients in the channel where the client is located that have the transliterated text function resource package pre-configured; and a delivery scheduling module, configured to deliver the transliterated text function resource package to the client after the backoff duration set in the policy is satisfied, according to the download backoff policy.

[0124] Based on any embodiment of the device in this application, the download fallback strategy includes: when the number of clients with the pre-installed transliteration function resource package in the channel is zero and the network environment type of the client is a non-internet cafe device, if bandwidth limitation is triggered, a first fallback duration is triggered; when the number of clients with the pre-installed transliteration function resource package in the channel is zero and the network environment type of the client is an internet cafe device, if bandwidth limitation is triggered, a second fallback duration is triggered; when the number of clients with the pre-installed transliteration function resource package in the channel is one and the network environment type of the client is a non-internet cafe device, if bandwidth limitation is triggered, a third fallback duration is triggered; when the number of clients with the pre-installed transliteration function resource package in the channel is one and the network environment type of the client is an internet cafe device, if bandwidth limitation is triggered, a fourth fallback duration is triggered.

[0125] Based on any embodiment of the device in this application, identifying whether there is sensitive information in the mixed transliteration includes: matching the mixed transliteration with a preset prohibited keyword database; when the mixed transliteration matches a keyword in the prohibited keyword database, it is determined that the mixed transliteration contains sensitive information.

[0126] Based on any embodiment of the device in this application, the method of locating a violating user containing sensitive information according to the identified transliterated original audio includes: matching each transliterated original audio with a preset prohibited keyword database; when the transliterated original audio matches a keyword in the prohibited keyword database, it is determined that the transliterated original audio contains sensitive information; and the user corresponding to the transliterated original audio containing sensitive information is identified as a violating user.

[0127] Based on any embodiment of the device in this application, following the violation location module 5300, the device further includes: a review and sending module, configured to send the original audio of the located violation user, the corresponding transliteration of the original audio, and violation information to the manual review terminal; and a review and penalty module, configured to receive the review result returned by the manual review terminal and perform corresponding penalty operations on the user permissions of the violation user in the current channel according to the review result.

[0128] Another embodiment of this application also provides an electronic device. For example... Figure 4 The diagram shows the internal structure of an electronic device. This electronic device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database can store information sequences, and when executed by the processor, these computer-readable instructions enable the processor to implement an audio violation detection method.

[0129] The processor of this electronic device provides computing and control capabilities to support the operation of the entire device. The memory of this electronic device can store computer-readable instructions, which, when executed by the processor, cause the processor to perform the audio violation detection method of this application. The network interface of this electronic device is used for communication with a terminal.

[0130] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0131] In this embodiment, the processor is used to execute... Figure 3 The specific functions of each module are described, and the memory stores the program code and various data required to execute the aforementioned modules or sub-modules. A network interface is used to enable data transmission between user terminals or the server. In this embodiment, the non-volatile readable storage medium stores the program code and data required to execute all modules in the audio violation detection device of this application. The server can call the server's program code and data to execute the functions of all modules.

[0132] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the audio violation detection method of any embodiment of this application.

[0133] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0134] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

Claims

1. An audio violation detection method, characterized in that, include: The server receives the transliterated audio data of the current channel from the client and identifies whether there is sensitive information in the transliterated audio data. When the mixed transliteration contains sensitive information, the server obtains the original audio of each user who spoke within the current channel during the time period corresponding to the mixed transliteration; The server performs transliteration recognition on the original audio of each user and locates the offending user containing sensitive information based on the transliterated original audio.

2. The audio violation detection method according to claim 1, characterized in that, The server receives the mixed audio data from the client and performs transliteration recognition to obtain the transliterated text of the mixed audio, including: The device capability information of each client in the current channel is retrieved, and at least one client is selected as the target client based on the device capability information. Send a transliteration scheduling instruction to the target client so that the target client switches to the transliteration function enabled state and begins to perform transliteration recognition on the mixed data of the current channel; Receive the mixed transliteration reported by the target client in response to the transliteration scheduling instruction.

3. The audio violation detection method according to claim 2, characterized in that, Based on the device capability information, at least one client is selected as the target client, including: Identify the clients in the current channel that have pre-installed transliteration function resource packages, forming the initial client set; Based on the operational capability parameters in the device capability information, clients whose operational capability parameters match preset operational capability conditions are selected from the initial client set to form an available client set. Determine whether the total number of target clients in the available client set that are in the state of having the transliteration function enabled exceeds a preset number threshold. If the preset number threshold is not reached, select a corresponding number of clients with relatively better resource capability parameters from the available client set as target clients based on the resource capability parameters in the client's device capability information, according to the missing number required to reach the preset number threshold.

4. The audio violation detection method according to claim 1, characterized in that, Before the server receives the transliterated audio data of the current channel from the client and performs transliteration recognition, it includes: Receive device capability information and associated channel identifier reported by any client in the current channel, and store the association in the device activity table; Based on the device support information and resource package status information in the device capability information of each client in the device activity table, determine whether the client supports the transliteration function and whether there is currently no pre-installed transliteration function resource package; When the judgment result is "supported" and "not preset", the transliteration function resource package is sent to the corresponding client according to the download bandwidth load capacity information of the current channel, so as to drive the client to preset the transliteration function resource package and switch to the transliteration function enabled state.

5. The audio violation detection method according to claim 4, characterized in that, Based on the download bandwidth load information corresponding to the current channel, the transliteration function resource package is sent to the corresponding client, including: When the current download bandwidth load capacity information triggers bandwidth limitation, a download yielding strategy is determined for the client based on the network environment type of each client and the number of clients in the channel where the client is located that have the pre-installed transliteration function resource package; According to the download backoff strategy, after the backoff time set in the strategy is met, the transliteration function resource package is sent to the client.

6. The audio violation detection method according to claim 5, characterized in that, The download fallback strategy includes: When the number of clients with the pre-installed transliteration function resource package in the channel is zero and the network environment type of the client is a non-internet cafe device, if the bandwidth limit is triggered, the first fallback time will be given. When the number of clients with the pre-installed transliteration function resource package in the channel is zero and the network environment type of the client is an internet cafe device, if the bandwidth limit is triggered, the second fallback time will be given. When the number of clients with the pre-installed transliteration function resource package in the channel is one and the network environment type of the client is a non-internet cafe device, if the bandwidth limit is triggered, the third fallback time will be given. When the number of clients with the pre-installed transliteration function resource package in the channel is one and the network environment type of the client is an internet cafe device, if bandwidth limitation is triggered, the fourth fallback time will be given.

7. The audio violation detection method according to claim 1, characterized in that: Identifying whether there is sensitive information in the mixed transliteration includes: matching the mixed transliteration with a preset prohibited keyword database; when the mixed transliteration matches a keyword in the prohibited keyword database, it is determined that the mixed transliteration contains sensitive information. And / or: Locating users who violate regulations by identifying transliterated original audio messages includes: matching each transliterated original audio message with a preset prohibited keyword database; when the transliterated original audio message matches a keyword in the prohibited keyword database, it is determined that the transliterated original audio message contains sensitive information, and the user corresponding to the transliterated original audio message containing sensitive information is identified as a violating user.

8. The audio violation detection method according to claim 1, characterized in that, After locating the offending user containing sensitive information based on the transliterated original audio, the following steps are taken: The server sends the original audio of the violating user, the corresponding transliteration of the original audio, and the violation information to the manual review terminal. The server receives the review results returned by the manual review terminal and, based on the review results, performs corresponding penalty operations on the user permissions of the violating user in the current channel.

9. An audio violation detection device, characterized in that, include: The edge recognition module is configured to receive the transliterated audio data of the current channel from the client and identify whether there is sensitive information in the transliterated audio data. The original audio acquisition module is configured to obtain the original audio of each user who spoke within the current channel during the time period corresponding to the mixed transliteration when the mixed transliteration contains sensitive information; The violation location module is configured to have the server perform transliteration recognition on the original audio of each user, and locate the violating user containing sensitive information based on the transliterated original audio.

10. An electronic device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 7.

Images