Audio and video processing method, device, equipment, medium and product

Abstract

The present disclosure provides a method and device for processing audio and video, equipment, medium and product, relating to the technical field of audio and video processing, which comprises: obtaining a first timestamp of a server, taking the time corresponding to the first timestamp as a reference time, and the server is connected with at least one audio and video collection terminal; based on the reference time, a corresponding time calibration parameter is established for the audio and video stream of each audio and video collection terminal; according to the time calibration parameter, the local time of each audio and video collection terminal is mapped to the reference time to time-align each audio and video stream. The present disclosure takes the server time as the unified reference and establishes the time calibration parameter, maps the scattered local time to the unified reference time, thereby ensuring the alignment of multiple audio and video streams in the time dimension, and providing a reliable time reference for subsequent cross-validation and positioning cheating behavior.

Description

Technical Field

[0001] This disclosure relates to the field of audio and video processing technology, specifically to audio and video processing methods, apparatus, equipment, media, and products. Background Technology

[0002] In online education platforms, remote teaching systems, and large-scale online examination scenarios such as professional qualification certification and recruitment written tests, multi-view monitoring and audio collaborative verification are commonly used proctoring methods.

[0003] However, traditional remote online examination scenarios typically employ single-channel or multi-channel video monitoring. Single-channel monitoring lacks cross-verification between multiple signals and is prone to false alarms due to network jitter or limitations of a single perspective, leading to a significant increase in the workload of verification. In multi-channel video monitoring scenarios, it is often impossible to accurately align the audio and video timelines, resulting in broken chains of evidence and difficulty in accurately pinpointing the time of cheating. The ability to identify covert cheating behaviors such as whispered conversations is also limited, which severely restricts auditing efficiency and poses a potential threat to the fairness of the examination. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, medium, and product for processing audio and video data to solve the problem of how to achieve time synchronization and cross-verification of multi-channel audio and video monitoring data in order to effectively identify covert cheating behavior.

[0005] In a first aspect, this disclosure provides an audio and video processing method, the method comprising:

[0006] The server obtains the first timestamp and uses the time corresponding to the first timestamp as the base time. The server is connected to at least one audio and video acquisition terminal. Based on the reference time, corresponding time calibration parameters are established for the audio and video streams of each audio and video acquisition terminal; Based on the time calibration parameters, the local time of each audio and video acquisition terminal is mapped to the reference time to perform time alignment for each audio and video stream.

[0007] Secondly, this disclosure provides an audio / video processing apparatus, the apparatus comprising: The time acquisition module is used to acquire the first timestamp of the server and use the time corresponding to the first timestamp as the base time. The server is connected to at least one audio and video acquisition terminal. The time synchronization module is used to establish corresponding time calibration parameters for the audio and video streams of each audio and video acquisition terminal based on the reference time. The time alignment module is used to map the local time of each audio and video acquisition terminal to the reference time according to the time calibration parameters, so as to perform time alignment on each audio and video stream.

[0008] Thirdly, this disclosure provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the audio and video processing method of the first aspect or any corresponding embodiment described above.

[0009] Fourthly, this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to execute the audio and video processing method of the first aspect or any corresponding embodiment described above.

[0010] Fifthly, this disclosure provides a computer program product, including computer instructions for causing a computer to execute the audio and video processing method of the first aspect or any corresponding embodiment described above.

[0011] The audio and video processing method provided in this disclosure involves obtaining a first timestamp from a server, using the time corresponding to the first timestamp as a reference time, and connecting the server to at least one audio and video acquisition terminal. Based on the reference time, the method synchronizes the time of each audio and video acquisition terminal and establishes corresponding time calibration parameters for the audio and video streams of each terminal. According to the time calibration parameters, the local time of each audio and video acquisition terminal is mapped to the reference time to align the time of each audio and video stream. This disclosure uses the server time as a unified reference and establishes time calibration parameters to map dispersed local times to a unified reference time, thereby ensuring the alignment of multiple audio and video streams in the time dimension and providing a reliable time reference for subsequent cross-validation and detection of cheating behavior. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure; Figure 2 This is a schematic flowchart of an audio and video processing method according to an embodiment of the present disclosure; Figure 3This is a structural block diagram of an audio / video processing apparatus according to an embodiment of the present disclosure; Figure 4 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0015] It should be noted that the information (including but not limited to user input information, such as information entered by the user into input boxes), data (including but not limited to data used for analysis, stored data, and displayed data, such as context code, all code of the current project, the service pressure corresponding to operations performed on all code of the current project, and the code development status of the current project), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. For example, the context code, operations performed on all code of the current project, the corresponding service pressure, and the code development status involved in this application were all obtained with full authorization.

[0016] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise expressly specified.

[0017] As one optional application scenario in the embodiments of this application, such as Figure 1 As shown, the system may include at least one terminal device and at least one server. Figure 1 The system is illustrated in the example, which includes a computer 101, a mobile terminal 102, and a server 103, and the terminal devices such as the computer 101 and the mobile terminal 102 are connected to the server 103 through a network 110.

[0018] Specifically, the terminal device can be a smartphone, tablet, laptop, PDA, desktop computer, smart TV, smart wearable device, VR (Virtual Reality) device, AR (Augmented Reality) device, etc. Server 103 can be a standalone physical server, a server cluster, a distributed system, or a cloud server providing cloud services. Network 110 can be a wired or wireless network, examples of which include, but are not limited to, the Internet, corporate intranet, local area network, wide area network, mobile communication network, and combinations thereof.

[0019] In online education platforms, remote teaching systems, and large-scale online examination scenarios such as professional qualification certification and recruitment written tests, multi-view monitoring and audio collaborative verification are commonly used proctoring methods.

[0020] However, traditional remote online examination scenarios typically employ single-channel or multi-channel video monitoring. Single-channel monitoring lacks cross-verification between multiple signals and is prone to false alarms due to network jitter or limitations of a single perspective, leading to a significant increase in the workload of verification. In multi-channel video monitoring scenarios, it is often impossible to accurately align the audio and video timelines, resulting in broken chains of evidence and difficulty in accurately pinpointing the time of cheating. The ability to identify covert cheating behaviors such as whispered conversations is also limited, which severely restricts auditing efficiency and poses a potential threat to the fairness of the examination.

[0021] Therefore, this disclosure obtains the first timestamp of the server and uses the time corresponding to the first timestamp as the reference time. The server is connected to at least one audio / video acquisition terminal. Based on the reference time, corresponding time calibration parameters are established for the audio / video streams of each audio / video acquisition terminal. According to the time calibration parameters, the local time of each audio / video acquisition terminal is mapped to the reference time to perform time alignment for each audio / video stream. This disclosure uses the server time as a unified reference and establishes time calibration parameters to map the scattered local times to a unified reference time, thereby ensuring the alignment of multiple audio / video streams in the time dimension and providing a reliable time reference for subsequent cross-validation and locating cheating behavior.

[0022] According to an embodiment of this disclosure, an audio and video processing method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0023] This embodiment provides an audio and video processing method that can be used in the aforementioned mobile terminals, such as mobile phones and tablet computers. Figure 1This is a flowchart of an audio / video processing method according to an embodiment of the present disclosure, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain the first timestamp of the server, use the time corresponding to the first timestamp as the reference time, and connect the server to at least one audio and video acquisition terminal.

[0024] Step S202: Based on the reference time, establish corresponding time calibration parameters for the audio and video streams of each audio and video acquisition terminal.

[0025] Step S203: Based on the time calibration parameters, map the local time of each audio and video acquisition terminal to the reference time to perform time alignment for each audio and video stream.

[0026] The audio and video processing method provided in this embodiment obtains a first timestamp from the server, uses the time corresponding to the first timestamp as a reference time, and connects the server to at least one audio and video acquisition terminal. Based on the reference time, the time of each audio and video acquisition terminal is synchronized, and corresponding time calibration parameters are established for the audio and video streams of each audio and video acquisition terminal. According to the time calibration parameters, the local time of each audio and video acquisition terminal is mapped to the reference time to align the time of each audio and video stream. This disclosure uses the server time as a unified reference and establishes time calibration parameters to map the scattered local times to a unified reference time, thereby ensuring the alignment of multiple audio and video streams in the time dimension and providing a reliable time reference for subsequent cross-validation and detection of cheating behavior.

[0027] The following is a detailed explanation of the above steps. In step S201, the first timestamp of the server is obtained, and the time corresponding to the first timestamp is used as the reference time. The server is connected to at least one audio and video acquisition terminal.

[0028] In remote proctoring scenarios for large-scale online exams, the server is the exam server, and the audio and video acquisition terminal is a mobile terminal used by each candidate, such as a smartphone or tablet, which has a dedicated proctoring application installed.

[0029] In this embodiment, the examination server first obtains the first timestamp generated by its own system clock. The first timestamp is a digital time stamp generated by the server's clock source. Subsequently, the absolute time corresponding to the first timestamp is determined as the reference time for the entire invigilation process. The reference time refers to the starting reference point of all invigilation data streams in the time dimension, aiming to eliminate the disorder of event sequence that may be caused by the local clock deviation of different terminal devices.

[0030] In this embodiment, the examination server establishes and maintains a stable communication connection with the mobile terminals of all candidates participating in the examination via a network. Under this architecture, the examination server's system time is designated as a globally unique clock source. Therefore, when all audio and video acquisition terminals collect candidate audio, video, and possible screen-sharing data, the generated data packets all contain timestamps synchronized or corrected based on the global clock source. Specifically, when the official start command for the examination is triggered, all connected candidates' mobile terminals are synchronously triggered to begin acquisition based on the reference time. Thus, when invigilators view any candidate's audio and video streams or review suspicious events in the backend, all audio and video data are based on a unified server time, ensuring that events collected by different mobile terminals that occur simultaneously in physical time have a consistent time identifier in the invigilation data records, improving the time consistency of the invigilation data and the reliability of post-audit.

[0031] In this embodiment, multiple clock synchronization mechanisms are supported. The first clock synchronization mechanism is software protocol-based clock synchronization, such as using NTP, or Network Time Protocol. The examination server can act as an NTP server, periodically exchanging time messages with all online candidate mobile terminals, and calibrating the local clock of the terminal devices by calculating network latency to keep their time synchronized with the server time.

[0032] However, for invigilation scenarios with even stricter time consistency requirements, this embodiment further supports PTP (Precision Time Protocol) based on hardware clock stamping. In this mode, if the examination server, network switches, and terminal devices all support the PTP protocol, sub-millisecond or even higher clock synchronization accuracy can be achieved through a master-slave clock architecture and hardware timestamps, thereby minimizing timing errors caused by network fluctuations. Furthermore, the examination server can also be configured with an independent local high-precision clock source, such as a GPS-disciplined clock, a rubidium atomic clock module, or a high-stability, temperature-controlled crystal oscillator, ensuring a stable, continuous, and reliable internal time reference even when an external time server is unavailable.

[0033] In step S202, based on the reference time, corresponding time calibration parameters are established for the audio and video streams of each audio and video acquisition terminal.

[0034] In one embodiment, time calibration parameters are established for the audio and video streams of each audio and video acquisition terminal, including: Multiple synchronization tracking packets are sent to each audio / video acquisition terminal, and the time information corresponding to each synchronization tracking packet is recorded. The time information includes the server sending time when the server sends the synchronization tracking packet to the audio / video acquisition terminal, the terminal receiving time when the audio / video acquisition terminal receives the synchronization tracking packet, the terminal response time when the audio / video acquisition terminal sends the synchronization tracking packet back to the server, and the server response time when the server receives the synchronization tracking packet.

[0035] Based on the time information, the single clock deviation and single round-trip delay of each synchronization packet are calculated. The time information is then filtered based on the single clock deviation and single round-trip delay to obtain the time calibration parameters.

[0036] After establishing a unified clock source based on the aforementioned reference time, the time of the audio and video streams of each audio and video acquisition terminal (i.e., the candidate's mobile terminal) is calibrated. Specifically, corresponding time calibration parameters are established for the audio and video streams of each audio and video acquisition terminal. The time calibration parameters are used to convert the timestamps marked on the audio and video streams acquired by the terminal devices to a global timeline based on the server clock source.

[0037] Specifically, before the exam begins or during breaks, the server periodically sends synchronization tracking packets multiple times to each audio / video acquisition terminal. These synchronization tracking packets are data messages actively sent by the exam server to the audio / video acquisition terminals, specifically for time synchronization measurement. Key time information for each tracking interaction is recorded: server sending time (the time marked by the server's clock source when the synchronization tracking packet leaves the server); terminal receiving time (the time marked by the local clock of the audio / video acquisition terminal when the synchronization tracking packet arrives at the audio / video acquisition terminal and is received by the proctoring application); terminal response time (the time marked by the local clock of the audio / video acquisition terminal when it generates and sends a response packet to the server); and server response time (the time marked by the server's own clock source when it receives the response packet). Through each tracking interaction, the single network round-trip latency and the clock deviation of the terminal relative to the server can be estimated. This embodiment obtains a large number of statistical samples by sending multiple synchronization packets to eliminate the error caused by a single network jitter, thereby calculating a more stable median or average network latency and a more reliable clock deviation value.

[0038] For each audio / video acquisition terminal, based on the time information of a single complete synchronization packet, the single clock deviation and single round-trip delay are calculated. The single round-trip delay, used to assess network latency for that communication, is obtained using the following formula:

[0039] in, This refers to the round-trip time. Send the time to the server; This refers to the terminal's reception time. Terminal response time; This refers to the server response time.

[0040] Single clock skew is the instantaneous offset of the audio / video acquisition terminal's clock relative to the server clock, estimated under the assumption of a symmetrical network path. It is used to extract clock relationship and network status information from a single observation and is calculated using the following formula:

[0041] in, This refers to a single clock deviation; Send the time to the server; This refers to the terminal's reception time. Terminal response time; This refers to the server response time.

[0042] However, due to unpredictable network fluctuations, such as packet queuing and momentary congestion, not all observation samples are reliable. Therefore, it is necessary to filter the raw time information based on the calculated single clock skew and single round-trip delay to remove outliers and form high-quality time calibration parameters. Specifically, firstly, statistical analysis is performed on multiple single round-trip delays collected over a period of time, for example, calculating the median. Then, filtering thresholds are set for single round-trip delay and single clock skew; for example, the filtering threshold for single round-trip delay can be set to 1.5 times the median. For each measurement point, if its single round-trip delay exceeds this threshold, it is determined that the measurement has been severely affected by network jitter, and the calculated single clock skew is of low reliability. Therefore, all time information from that measurement point is removed as an outlier, and the remaining valid sample pairs constitute the time calibration parameters.

[0043] In step S203, the local time of each audio and video acquisition terminal is mapped to the reference time according to the time calibration parameters, so as to perform time alignment for each audio and video stream.

[0044] In one embodiment, the local time of each audio / video acquisition terminal is mapped to a reference time based on time calibration parameters to perform time alignment for each audio / video stream, including: The clock offset and clock drift of the audio and video acquisition terminal relative to the examination server are calculated based on the time calibration parameters.

[0045] By using clock offset and clock drift, the local timestamp of the audio and video acquisition terminal is mapped to the reference time of the examination server to perform time alignment for each audio and video stream in the audio and video acquisition terminal.

[0046] In this embodiment, the time calibration parameters of the synchronization dot packet are input into the time mapping model to obtain the clock offset and clock drift output by the time mapping model.

[0047] In this embodiment, a time mapping model is pre-established for the audio and video streams of each audio and video acquisition terminal based on time calibration parameters. This allows the time mapping of the synchronization packet sent by the server to the audio and video acquisition terminal to be predicted according to the time mapping formula. The time mapping model includes the following time mapping formula:

[0048] in, The mapping time for the synchronization timing packets sent by the server to arrive at the audio and video acquisition terminal; This is the drift factor, used to reflect the frequency deviation of the terminal crystal oscillator relative to the server; the ideal value is 1.0. The original timestamp of the audio / video capture terminal; The overall offset includes the initial time difference between the audio / video acquisition terminal and the server, as well as the fixed transmission delay.

[0049] Next, sample pairs are calculated based on time calibration parameters. :

[0050]

[0051] in, This refers to the round-trip time. Terminal response time; This refers to the server response time. These are sample pairs in the time calibration parameters. A sliding window is preset in the time calibration parameters, and valid sample pairs within the sliding window are collected to form a valid sample set.

[0052] By fitting each valid sample in the valid sample set using a linear fitting algorithm (such as the least squares method), the valid samples and their corresponding straight lines are obtained. The slope of the straight lines is then calculated as the drift factor. and the intercept of the straight line as the composite offset .

[0053] This embodiment can also periodically acquire new valid sample pairs from the audio and video streams acquired by the audio and video acquisition terminal during the examination process, so as to recalculate the new drift factor and new comprehensive offset of the new valid sample pairs, update the drift factor and comprehensive offset in the time mapping formula based on the new drift factor and new comprehensive offset, and update the time mapping model based on the updated new drift factor and new comprehensive offset.

[0054] In one embodiment, when multiple synchronization tracking packets are sent to each audio / video acquisition terminal, a preset time mapping model is used to predict the mapping time of the synchronization tracking packets arriving at the audio / video acquisition terminal; the prediction error between the mapping time and the terminal receiving time is calculated, and when the prediction error is greater than a preset out-of-sync threshold, a resynchronization process is triggered; the resynchronization process includes: resetting the time calibration parameters and increasing the frequency of sending synchronization tracking packets to the audio / video acquisition terminal until the prediction error is less than or equal to the preset out-of-sync threshold.

[0055] Each time a synchronization timing packet is sent, the prediction error between the mapping time predicted by the time mapping model and the actual time when the synchronization timing packet arrives at the audio / video acquisition terminal is calculated:

[0056] in, This represents the prediction error between the mapped time and the actual time. The mapping time for the synchronization timing packets sent by the server to arrive at the audio and video acquisition terminal; This refers to the actual time when the synchronization packet arrives at the audio / video acquisition terminal, i.e., the terminal reception time.

[0057] The prediction error is compared with a preset step-out threshold. At milliseconds, this is within the normal range, indicating that the time information recorded when sending the synchronization timing packet can be used as data to generate valid sample pairs; when In milliseconds, this is within a tolerable range and is considered as a network jitter or transient interference received during the transmission of the synchronization packet. This sample is temporarily ignored. If it is triggered n times consecutively, a fast calibration is triggered. The value of n can be dynamically selected. If a serious synchronization failure occurs, such as due to device sleep / wake-up or system time modification, a resynchronization process is triggered. Specifically, in the resynchronization process, the audio / video streams corresponding to the sample pairs experiencing serious synchronization failure are marked as abnormally aligned, and a penalty term is introduced to reduce the alarm weight of the audio / video acquisition terminals corresponding to these abnormally aligned streams. Further, the server re-sends synchronization packets at high frequency to the audio / video acquisition terminals corresponding to the abnormally aligned video streams, rapidly accumulating new sample pairs to refit the straight line of the new sample pairs, obtaining the first drift factor and the first comprehensive offset, thereby constructing a first time mapping formula. The prediction error between the predicted mapping time and the actual time the synchronization packets arrive at the audio / video acquisition terminals is recalculated using the first time mapping formula until the prediction error is less than a preset synchronization failure threshold multiple times consecutively. At this point, the abnormally aligned audio / video streams are restored to normal alignment, and the alarm weight of the corresponding audio / video acquisition terminals is restored.

[0058] In one embodiment, abnormal event detection is performed on each time-aligned audio and video stream, and the occurrence time of the detected abnormal event relative to the reference time is recorded; based on the occurrence time, comprehensive alarm information of the abnormal event is generated.

[0059] The audio and video streams include at least one first video stream. On each time-aligned audio and video stream, anomaly detection is performed, and the occurrence time of the detected anomaly events relative to a reference time is recorded, including: When an abnormal event is detected in the first video stream, the occurrence time of the abnormal event relative to the reference time is determined.

[0060] Among them, abnormal event detection includes video abnormality detection and audio abnormality detection; video abnormality detection includes at least one of the following: personnel leaving their seats, detection of the presence of uncertified personnel, detection of equipment use, and detection of line of sight deviation; audio abnormality detection includes at least one of the following: human voice detection, keyword detection, and detection of abnormal environmental sounds.

[0061] Specifically, keyframes are extracted from the continuous first video stream at a fixed frame rate. Each keyframe is an image frame that represents continuous action. Each keyframe is preprocessed, including size adjustment and pixel value normalization. A pre-trained lightweight deep learning model, such as YOLOv8-Nano, deployed on the server side is used to analyze each preprocessed keyframe to locate and select target objects in the scene that are related to the exam behavior, such as the examinee's face, potentially unauthorized electronic devices, or unauthorized reference materials. The bounding boxes of the target objects and their category confidence scores are then obtained from the model output.

[0062] Based on a sequence of N consecutive frames of the same target object and their corresponding bounding boxes, feature information containing the target object's temporal and spatial dimensions is extracted and fused. Based on this feature information, a probability distribution function is used to predict the probability values ​​of the target object for various preset behaviors, such as "sitting normally," "leaving the seat," "looking around," and "using a mobile phone." The behavior category corresponding to the highest probability is determined as the event type, and the highest probability is used as the confidence level of this abnormal event.

[0063] Next, the continuously acquired first audio stream is segmented into a series of short frames, and the spectrogram or audio features of each short frame are extracted. The first audio stream corresponds to the first video stream. Subsequently, a pre-trained convolutional recurrent neural network (CRNN) is used to label the audio feature spectrogram or audio features, obtaining the event labels of the current short frame output by the model, such as "quiet environment," "speaking voice," "keyboard typing sound," "paper turning sound," etc., and their label confidence scores. Furthermore, the first audio stream is identified and located in real time to determine specific sensitive words and phrases that are highly correlated with the risk of cheating, such as "answer," "choose C," and "help."

[0064] Based on the target object and category confidence of the first video stream, and the event label and label confidence of the first audio stream, it is determined whether an abnormal event has occurred in the first audio and video streams. For example, if the target object in the video stream is identified as exhibiting "leaving its seat" behavior and the category confidence is higher than a preset category confidence threshold, or if the audio stream detects sensitive keywords, it is determined that an abnormal event has occurred, and the occurrence time of the abnormal event is then determined.

[0065] In one embodiment, the audio and video stream further includes at least one first audio stream, and based on the occurrence time, generates comprehensive alarm information for the abnormal event, including: Retrieve the media segment corresponding to the occurrence time in at least one first audio stream, and extract the event information corresponding to the media segment; Abnormal events and event information are fused to obtain a fusion score, and comprehensive alarm information is generated based on the fusion score.

[0066] Retrieve media segments corresponding to their occurrence time in at least one first audio stream and extract event information corresponding to the media segments. When an abnormal event is detected in the first video stream (such as candidate A's terminal) and its occurrence time is determined, retrieve media segments within a corresponding time window in the first audio stream corresponding to the first video stream, with the occurrence time as the time center. The media segment refers to the original or processed audio data within the time window. Subsequently, perform events such as event classification and keyword detection on the media segment to extract the event information contained therein. This information may include the detected sound event type such as "clear dialogue" or "multiple people speaking simultaneously", keyword content, and the confidence level of each event.

[0067] Secondly, abnormal events and event information are fused to obtain a fusion score. Based on the fusion score and a preset alarm threshold, different levels of comprehensive alarm information are generated. For example, a high fusion score can trigger a high-risk alarm, accompanied by a chain of fused evidence, such as abnormal video screenshots, audio waveforms and transcribed text fragments, and precise timestamps; a medium fusion score may trigger a medium-risk alert. This method greatly improves the accuracy and information content of alarms, effectively reducing the risk of false alarms or missed alarms from a single source.

[0068] This embodiment generates comprehensive alarm information for abnormal events based on their occurrence time. It aggregates abnormal events from audio and video streams from the same examinee's terminal within the same time period, as well as potentially related abnormal events from other examinees, such as identifying multiple individuals simultaneously exhibiting suspicious voices or actions through time alignment. These events, with precise and uniform time tags, are spatiotemporally correlated, and information such as event type, confidence level, occurrence time, and terminal device is integrated to generate comprehensive alarm information, which is then presented to the invigilator for final assessment. This improves the accuracy of alarms and provides a multimodal, multi-terminal collaborative evidence chain accurate to the frame level for tracing and verifying cheating behavior.

[0069] In one embodiment, anomaly fusion is performed on the abnormal event and event information to obtain a fusion score, including: Calculate temporal consistency and semantic consistency measures between anomalous events and event information; Obtain the alignment status of the audio and video streams. If there is an abnormality in the alignment status, introduce a penalty term. Based on preset weights, pre-set reliability, and penalty terms, anomaly fusion is performed on time consistency measures and semantic consistency measures to obtain a fusion score of abnormal events and event information.

[0070] In one embodiment, the server connects to the monitoring terminal and generates comprehensive alarm information based on the fusion score, including: The alarm level corresponding to the abnormal event is determined based on the fusion score, and alarm information corresponding to the alarm level is generated and sent to the monitoring terminal; wherein, the alarm information includes media segments corresponding to the first video stream and at least one first audio stream.

[0071] Specifically, consistency checks are performed on anomaly clues from different data sources. Consistency checks include computation time consistency measures and semantic consistency measures.

[0072] Temporal consistency metrics are used to quantify the proximity of multiple anomalous events in terms of their occurrence time. Specifically, a set of events to be fused is selected, such as the "looking down" event detected in the first video stream from the candidate's front-facing camera, the "rapid hand movement" event detected in the second video stream from the same candidate's side-facing camera, and the "rapid paper flipping sound" event detected in the first audio stream from the ambient microphone. Their respective occurrence times are then extracted. A measure of time consistency can be calculated using a decay function based on the time difference dispersion.

[0073] in, As a measure of time consistency; This is the cutoff time for the abnormal event; This represents the starting point of the abnormal event; This refers to the point in time when the abnormal event occurred; This is the preset time alignment tolerance parameter.

[0074] If the peak times of all abnormal events are highly concentrated, then the time difference is small. A value close to 1 indicates high temporal consistency; if events are dispersed in time, the time difference is large, and the exponential function value decays. A value close to 0 indicates low time consistency.

[0075] Semantic consistency metrics are used to assess the logical relevance of different events in terms of behavior or content. For example, examining the logical relationship between event types, the "eyes continuously shifting to the left" event identified from a video stream and the "second face appears on the left" event extracted from another video stream are highly semantically matched, forming a reasonable chain of evidence for "voyeurism." Conversely, the "leaving one's seat" event and the "keyboard tapping sound" event in an audio stream have a weaker semantic relevance. Semantic consistency can be determined based on a predefined event association rule table or through reasoning using a knowledge graph.

[0076] After obtaining the temporal consistency measure and semantic consistency measure, anomaly fusion is performed on the two based on preset weights and preset reliability to obtain the final fusion score. Preset weights and This reflects the importance of different data sources (such as frontal video, side video, and ambient audio) in the overall judgment. For example, the weight of frontal video can be set to 0.5, side video to 0.3, and ambient audio to 0.2. The raw confidence level of the video stream. This represents the raw confidence level of the audio stream.

[0077] This embodiment adopts an architecture that defines abstract rules and concrete implementations in a layered manner. The fusion rules used to characterize the abstract layer are mathematical models that elucidate the general principles and constituent elements of the fusion computation in this embodiment. This represents a relative time window. The specific implementation version is to take the above abstract rules An instance of computation performed at a specific point in time and in a specific context.

[0078] Specifically, at a more abstract level, the fusion score of multiple audio and video events can be recorded as a fusion score. Its general form can be expressed as:

[0079] in, The combined score of temporal consistency and semantic consistency measures; Preset weights for the video stream; Preset weights for the audio stream; The raw confidence level of the video stream; The raw confidence level of the audio stream; As a measure of time consistency; This is a consistency reward coefficient, which provides bonus points when multiple paths trigger abnormal events simultaneously.

[0080] In practical implementation, the alarm relative time is used. Centered on a specific time window, multiple audio and video events are aggregated, and the overall confidence level of the aggregated events is calculated. At this point, the overall confidence level within that time window is... This can be considered as the above fusion score One implementation method, specifically calculated as follows:

[0081] in, For overall confidence level; For video stream Event confidence in the window; For audio stream The confidence level of the event; For video stream The weights; For spatiotemporal consistency items, there are temporal consistency measures and semantic consistency measures; , is the default parameter; , is the default parameter; This is the default parameter. In one specific embodiment, it can be... , and Set to the above fusion score The corresponding weights and consistency reward coefficients are set to values ​​that are consistent or similar, so as to maintain the consistency of the fusion strategy at different implementation levels.

[0082] Check the time alignment of the audio and video streams. If the time alignment of the audio and video streams is unstable, introduce a penalty term. Then calculate its new comprehensive confidence level:

[0083] in, The new overall confidence level after introducing the penalty term; For overall confidence level; This is a penalty item.

[0084] In another embodiment, the server connects to the monitoring terminal and determines the alarm level corresponding to the abnormal event based on the fusion score. For example, a tiered threshold is set: when the fusion score is greater than a preset first alarm threshold, such as 0.85, it is determined to be a strong alarm; when the fusion score is less than or equal to the first alarm threshold but greater than a preset second alarm threshold, such as 0.65, it is determined to be a moderate alarm; when the fusion score is less than or equal to the second alarm threshold but greater than a preset third alarm threshold, such as 0.5, it is determined to be a minor alarm; when the fusion score is less than or equal to the third alarm threshold, no alarm is triggered. After determining the alarm level, alarm information corresponding to the alarm level is generated. The alarm information includes aligned media segments, such as video clips and audio waveform segments, from the first video stream that triggered the alarm and at least one first audio stream, centered around the time of the abnormality. This alarm information is sent to the monitoring terminal, such as the invigilator's workstation or a large screen, via a communication link, thereby assisting the invigilator in making efficient and accurate decisions.

[0085] In one embodiment, the method further includes: Calculate the summary value of the comprehensive alarm information and generate the index information of the comprehensive alarm information based on the summary value.

[0086] Obtain the server's private key and use it to digitally sign the index information to obtain the evidence storage information corresponding to the comprehensive alarm information.

[0087] Specifically, the generated alarm information and its associated original evidence data are solidified. This alarm information is typically represented as a structured data object or file package, the contents of which include, but are not limited to: alarm level, fusion score, event type list, occurrence time of the abnormal event, and media segment index. Media segments refer to the original or encoded audio and video files of a specific time period in the first time-aligned video stream and first audio stream.

[0088] The digest value of the comprehensive alarm information is calculated. Specifically, the digest value refers to the hash value calculated using a cryptographic hash function (such as the SHA-256 algorithm). Specifically, for each original evidence media file referenced in the alarm information data packet, its complete binary content is read and input into the SHA-256 algorithm to generate a unique and fixed file hash value for each file. Alternatively, the total hash value of the alarm information can also be calculated.

[0089] Based on the summary value, an index of comprehensive alarm information is generated. The index is a standardized descriptive file, usually in JSON format. The JSON index information is canonical serialized to ensure that the field order is fixed. It systematically records key metadata, such as: the corresponding audio and video stream identifier, the start and end time of the evidence fragment, the calculated summary value of each file, the version of the time calibration parameter used when generating the alarm, the alarm level, and the generation timestamp, etc.

[0090] Furthermore, the server obtains its own asymmetric encryption private key, preferably using an ECDSA algorithm based on elliptic curve cryptography, such as a private key generated using the P-256 curve. The server uses this private key to perform a digital signature operation on the generated index information. The digital signature process uses the private key to encrypt the hash value of the index information, generating a signature value bound to the index information content and the private key. The signature value is written to a separate field of the index information. Simultaneously, the index information records the corresponding public key fingerprint used to verify the signature. The public key fingerprint is a short identifier obtained by hashing the public key itself, allowing for quick location of the correct public key during subsequent verification.

[0091] Finally, based on the index information, digital signature value, and public key fingerprint, evidence storage information corresponding to the comprehensive alarm information is constructed. This evidence storage information can be archived along with the original media file. In any subsequent audit, review, or judicial proceedings, the verifying party can independently recalculate the file hash and verify the signature using the server's public key obtained from public or trusted channels. If the hash comparison matches and the digital signature verification passes, it proves that the alarm information and its associated evidence fragments were generated by the specific server at the claimed time, based on specific time parameters, and have not been tampered with by anyone since generation, thus forming a complete and credible evidence loop.

[0092] This embodiment also provides an audio and video processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0093] This embodiment provides an audio and video processing device, such as... Figure 3 As shown, it includes: The time acquisition module 301 is used to acquire the first timestamp of the server and use the time corresponding to the first timestamp as the reference time. The server is connected to at least one audio and video acquisition terminal.

[0094] The time synchronization module 302 is used to establish corresponding time calibration parameters for the audio and video streams of each audio and video acquisition terminal based on the reference time.

[0095] The time alignment module 303 is used to map the local time of each audio and video acquisition terminal to the reference time according to the time calibration parameters, so as to perform time alignment for each audio and video stream.

[0096] In some alternative implementations, the time synchronization module 302 includes: The time recording unit is used to send synchronization tracking packets to each audio / video acquisition terminal multiple times and record the time information corresponding to each synchronization tracking packet. The time information includes the server sending time when the server sends the synchronization tracking packet to the audio / video acquisition terminal, the terminal receiving time when the audio / video acquisition terminal receives the synchronization tracking packet, the terminal response time when the audio / video acquisition terminal sends the synchronization tracking packet back to the server, and the server response time when the server receives the synchronization tracking packet.

[0097] The parameter establishment unit is used to calculate the single clock deviation and single round-trip delay of each synchronization point packet based on time information, and to filter the time information according to the single clock deviation and single round-trip delay to obtain time calibration parameters.

[0098] In some alternative implementations, the time alignment module 303 includes: The offset calculation unit is used to calculate the clock offset and clock drift of the audio and video acquisition terminal relative to the examination server based on the time calibration parameters.

[0099] The time alignment unit is used to map the local timestamp of the audio and video acquisition terminal to the reference time of the examination server using clock offset and clock drift, so as to perform time alignment on each audio and video stream in the audio and video acquisition terminal.

[0100] In some alternative implementations, the time synchronization module 302 further includes: The time prediction unit is used to predict the mapping time of the synchronization timing packets arriving at the audio and video acquisition terminals when multiple synchronization timing packets are sent to each audio and video acquisition terminal, using a preset time mapping model.

[0101] The resynchronization unit is used to calculate the prediction error between the mapping time and the terminal reception time. When the prediction error is greater than the preset out-of-sync threshold, the resynchronization process is triggered. The resynchronization process includes: resetting the time calibration parameters and increasing the frequency of sending synchronization packets to the audio and video acquisition terminal until the prediction error is less than or equal to the preset out-of-sync threshold.

[0102] In some alternative embodiments, the apparatus further includes: The anomaly detection module is used to perform anomaly event detection on each time-aligned audio and video stream, and record the occurrence time of the detected anomaly events relative to the reference time.

[0103] The anomaly alarm module is used to generate comprehensive alarm information for abnormal events based on the time of occurrence.

[0104] In some optional implementations, the audio and video streams include at least one first video stream, and the anomaly detection module includes: The time determination unit is used to determine the occurrence time of the abnormal event relative to a reference time when an abnormal event is detected in the first video stream.

[0105] In some optional implementations, the audio / video stream further includes at least one first audio stream, and the anomaly alarm module includes: The event extraction unit is used to retrieve media segments corresponding to the occurrence time in at least one first audio stream and extract event information corresponding to the media segments.

[0106] The abnormal alarm unit is used to perform abnormal fusion on abnormal events and event information to obtain a fusion score, and generate comprehensive alarm information based on the fusion score.

[0107] In some optional implementations, the anomaly alarm unit includes: The measurement calculation subunit is used to calculate the temporal consistency measure and semantic consistency measure between abnormal events and event information.

[0108] The parameter introduction sub-unit is used to obtain the alignment status of the audio and video streams. If there is an abnormality in the alignment status, a penalty term is introduced.

[0109] The score calculation subunit is used to perform anomaly fusion of time consistency measure and semantic consistency measure based on preset weights, preset reliability and penalty terms, to obtain the fusion score of abnormal events and event information.

[0110] In some optional implementations, the server is connected to a monitoring terminal, and the anomaly alarm unit includes: An abnormal alarm subunit is used to determine the alarm level corresponding to an abnormal event based on the fusion score, generate alarm information corresponding to the alarm level, and send the alarm information to the monitoring terminal; wherein, the alarm information includes media segments corresponding to the first video stream and at least one first audio stream.

[0111] In some optional implementations, the anomaly alarm unit further includes: The evidence storage subunit is used to calculate the digest value of the comprehensive alarm information and generate the index information of the comprehensive alarm information based on the digest value; obtain the private key of the server and use the private key to digitally sign the index information to obtain the evidence storage information corresponding to the comprehensive alarm information.

[0112] In some optional implementations, anomaly detection includes video anomaly detection and audio anomaly detection; video anomaly detection includes at least one of the following: personnel leaving their seats, detection of the presence of uncertified personnel, detection of equipment use, and detection of line of sight deviation; audio anomaly detection includes at least one of the following: human voice detection, keyword detection, and detection of abnormal environmental sounds.

[0113] The audio and video processing apparatus provided in this disclosure can execute the audio and video processing methods provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the methods. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0114] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0115] The following is a detailed reference. Figure 4 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present disclosure. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 401, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 402 or a program loaded from memory 408 into random access memory (RAM) 403. RAM 403 also stores various programs and data required for the operation of the electronic device. The processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0116] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 408 including, for example, magnetic tapes, hard disks, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0117] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 409, or installed from a memory 408, or installed from a ROM 402. When the computer program is executed by the processor 401, it performs the functions defined in the audio and video processing methods of embodiments of this disclosure.

[0118] Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0119] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the audio and video processing methods shown in the above embodiments are implemented.

[0120] A portion of this disclosure can be applied to computer program products, such as computer program instructions, which, when executed by a computer, can invoke or provide methods and / or technical solutions according to this disclosure through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, and installation package files. Accordingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions; the computer compiling the instructions and then executing the corresponding compiled program; the computer reading and executing the instructions; or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0121] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for processing audio and video, characterized in that, The method includes: The server obtains the first timestamp and uses the time corresponding to the first timestamp as the base time. The server is connected to at least one audio and video acquisition terminal. Based on the reference time, corresponding time calibration parameters are established for the audio and video streams of each audio and video acquisition terminal; Based on the time calibration parameters, the local time of each audio and video acquisition terminal is mapped to the reference time to perform time alignment for each audio and video stream.

2. The method according to claim 1, characterized in that, The step of establishing corresponding time calibration parameters for the audio and video streams of each audio and video acquisition terminal includes: The system sends multiple synchronization tracking packets to each audio / video acquisition terminal and records the time information corresponding to each synchronization tracking packet. The time information includes the server sending time when the server sends the synchronization tracking packet to the audio / video acquisition terminal, the terminal receiving time when the audio / video acquisition terminal receives the synchronization tracking packet, the terminal response time when the audio / video acquisition terminal sends the synchronization tracking packet back to the server, and the server response time when the server receives the synchronization tracking packet. Based on the time information, the single clock deviation and single round-trip delay of each synchronization packet are calculated. The time information is then filtered according to the single clock deviation and the single round-trip delay to obtain time calibration parameters.

3. The method according to claim 2, characterized in that, The step of mapping the local time of each audio / video acquisition terminal to the reference time based on the time calibration parameters, in order to perform time alignment for each audio / video stream, includes: Calculate the clock offset and clock drift of the audio / video acquisition terminal relative to the server based on the time calibration parameters; Using the clock offset and the clock drift, the local timestamp of the audio and video acquisition terminal is mapped to the server's base time to perform time alignment for each audio and video stream in the audio and video acquisition terminal.

4. The method according to claim 2, characterized in that, The method further includes: When sending synchronization tracking packets to each audio and video acquisition terminal multiple times, a preset time mapping model is used to predict the mapping time when the synchronization tracking packets arrive at the audio and video acquisition terminal. Calculate the prediction error between the mapping time and the terminal reception time. When the prediction error is greater than a preset out-of-synchronization threshold, trigger the resynchronization process. The resynchronization process includes: resetting the time calibration parameters and increasing the frequency of sending the synchronization packet to the audio and video acquisition terminal until the prediction error is less than or equal to the out-of-synchronization threshold.

5. The method according to claim 1, characterized in that, The audio and video streams include at least one first video stream, and the method further includes: When an abnormal event is detected in the first video stream, the occurrence time of the abnormal event relative to the reference time is determined.

6. The method according to claim 5, characterized in that, The audio and video stream further includes at least one first audio stream, and the comprehensive alarm information generated based on the occurrence time of the abnormal event includes: Retrieve the media segment corresponding to the occurrence time in at least one first audio stream, and extract the event information corresponding to the media segment; An abnormal event and the event information are fused to obtain a fusion score, and comprehensive alarm information is generated based on the fusion score.

7. The method according to claim 6, characterized in that, The process of fusing the abnormal event and the event information to obtain a fusion score includes: Calculate the temporal consistency measure and semantic consistency measure between the abnormal event and the event information; Obtain the alignment status of the audio and video streams; if the alignment status is abnormal, introduce a penalty term. Based on preset weights, preset reliability, and the penalty term, the temporal consistency measure and the semantic consistency measure are fused abnormally to obtain the fusion score of the abnormal event and the event information.

8. The method according to claim 6, characterized in that, The method further includes: Calculate the summary value of the comprehensive alarm information, and generate the index information of the comprehensive alarm information based on the summary value; Obtain the private key of the server, and use the private key to digitally sign the index information to obtain the evidence storage information corresponding to the comprehensive alarm information.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the audio and video processing method according to any one of claims 1 to 8.

10. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the audio and video processing method according to any one of claims 1 to 8.

Images