Method and system for synchronizing audio and video of a homologous video stream and AI detection result

Abstract

The application discloses a method and system for synchronizing audio and video and AI detection results of homologous video streams. Through Redis message notification technology, the detection results of each AI detection service and the pull stream frame number information of the stream processing service are uniformly pushed into a predetermined Redis message channel at every whole second on the premise of having a unified clock. Each service can read the pull stream frame number and detection results of other services in the channel in real time, store them in the memory, perform frame tracking algorithm correction with the real-time frame number of the pull stream of the service, regularly update the frame number difference DFN between the leading frame number HFN in the channel and the latest frame number of a service that successfully pulls the stream earliest, so as to match the detection results and audio and video data of each service through the equation CFN (real-time frame number of the service) + DFN = HFN. Through the message channel above each service as a bridge, the audio and video and detection results are synchronized, each AI service is flexibly deployed, and the hardware resources are maximally utilized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio-video synchronization, in particular to a method and system for synchronizing audio-video and AI detection results of homologous video streams. BACKGROUND

[0002] The dual-recording system refers to a system for recording the sales process of self-financing products and agent-sold products of banking and other financial institutions. The dual-recording system can monitor and record the entire sales process of the staff. Through the analysis of the recorded data, it can be known whether the words and actions of the sales staff in the process of selling the products are in compliance. However, the dual-recording system has the following shortcomings, which makes it difficult to find many irregular behaviors:

[0003] 1. The dual-recording system only records the words and actions in the dual-recording time period, and the communication between the sales staff and the customers outside the dual-recording time period is not recorded, so it is difficult to find irregular behaviors outside the dual-recording time period.

[0004] 2. When the dual-recording system conducts compliance inspection, it only checks whether the sales staff has completed the corresponding words or actions in compliance, and does not check other irregular words and actions in the sales process.

[0005] In order to solve the above problems of the dual-recording system, the audio-video data of the sales staff is usually collected by establishing a 24-hour audio-video monitoring system, and then intelligent analysis is performed to find out whether there are irregular words and actions inside and outside the dual-recording time period, which serves as a supplement to the dual-recording system.

[0006] When the audio-video monitoring data is used to detect whether the words and actions of the sales staff are in compliance, it is usually necessary to perform or simultaneously perform multiple AI detection services such as ASR automatic speech recognition, target tracking, and face recognition on the homologous video stream, and it is required to accurately render the AI detection results obtained after a short processing time to the corresponding video frames, and to synchronize and merge the delayed audio and video streams, and finally to push out the effect stream.

[0007] However, it is impossible to ensure that the starting time of the successful stream pulling of each AI detection service is the same, which means that the initial stream pulling frame number timestamp of each AI detection service is usually not the same, and finally the result queue containing the video stream frame number and detection information based on the respective stream pulling time axis of each AI detection service will be detected for the same video stream. Obviously, rendering the result queues corresponding to each AI detection service with different stream pulling time axes to the same video stream will result in a very messy effect stream, and the rendering effect is poor. Therefore, there is an urgent need for a solution to synchronize the result queues of each AI detection service to a third-party time axis to solve the above problems.

[0008] At present, most of the audio and video synchronization schemes on the market generally open up a shared memory area, set a third-party clock as a reference clock in this memory area, and the playing speed of video and audio is based on this clock. When the playing source is slower than the reference clock, its playing speed is accelerated, or the frame is discarded; if it is too fast, the playing is delayed. Obviously, the scheme of audio and video synchronization through the third-party clock is not suitable for the application scenario of synchronizing the rendering of the corresponding AI detection result on the basis of audio and video synchronization, because AI detection requires a large amount of computing resources. This way of obtaining the clock by reading the memory means that a large number of AI detection services need to be deployed in the same server, which will cause the server hardware resources to be tight, increase the configuration standard of a single server, and is not conducive to the long-term stable operation of the server. SUMMARY

[0009] The present application realizes the synchronization of AI detection results on the basis of audio and video synchronization of homologous video streams, and provides an audio and video and AI detection result synchronization method and system for homologous video streams.

[0010] To achieve this purpose, the present application adopts the following technical solutions:

[0011] An audio and video and AI detection result synchronization method for homologous video streams is provided, and the steps include:

[0012] A unified clock is set for each service, including AI detection services and stream processing services;

[0013] Each service pulls the source video stream and pushes the latest frame information of the pulled stream to the Redis message channel in real time, each AI detection service performs corresponding AI detection on the pulled audio stream and video stream, and pushes the detection result to the Redis message channel in real time;

[0014] Each service that successfully pulls the stream executes a timing task every specified interval, and the timing task takes the frame number of the first frame pushed by the service that pulls the stream earliest in the task as the leading frame number, and updates the frame number difference between the real-time frame number of the stream continuously pulled by itself and the leading frame number;

[0015] The system stores the stream pulled by the stream processing service in real time and each AI detection result obtained from the Redis message channel into the corresponding cache buffer, and the cache buffer includes an audio frame cache buffer and a video frame and AI detection result cache buffer;

[0016] The audio frame, video frame and AI detection result corresponding to the frame number are matched from the audio frame cache buffer and the video frame and AI detection result cache buffer, and are synthesized as an effect stream and pushed out.

[0017] As preferred, each of the services first performs a forwarding operation on the source video stream to remove the pull stream number limit by default existing in the monitoring device generating the source video stream before starting to pull the source video stream.

[0018] As preferred, the frame information pushed to the Redis message channel includes the service name of the AI detection service, the frame timestamp of the pulled frame, the frame number, and the frame number difference from the leading frame number.

[0019] As preferred, the method for each service to update the frame number difference of itself includes the steps of:

[0020] L1, obtaining the system timestamp cTime;

[0021] L2, determining whether cTime is an integer second,

[0022] If yes, go to step L3;

[0023] If no, wait until the integer second and then go to step L3;

[0024] L3, obtaining the frame information completed by each of the services in the n seconds after the integer second from the Redis message channel, and creating a set current_frame_array to store the pull stream frame number and pull stream frame timestamp generated by the current service in real time within the n seconds;

[0025] L4, dividing each of the frame information generated by the same service within the n seconds obtained into the frame information set corresponding to the service;

[0026] L5, extracting the frame number recorded in each of the frame information with the same frame timestamp in each of the frame information sets;

[0027] L6, comparing the size of each of the frame numbers extracted, comparing a group of frame information sets with the largest frame number, and recording as head_frame_array;

[0028] L7, traversing the set head_frame_array, and searching whether the frame timestamp frame_time obtained each time exists in the set current_frame_array with a timestamp difference less than ±m time,

[0029] If yes, calculating the frame number difference frame_bias of the frame number corresponding to the two timestamps with a timestamp difference less than ±m time obtained by this search, and storing frame_bias in the set frame_bias_array,

[0030] If no, continue the next traversal;

[0031] L8, after the traversal is completed, remove the maximum and minimum values in the set frame_bias_array, then take the frame_bias with the highest frequency of occurrence in the set frame_bias_array as the latest difference between the frame number of the current service and the leading service, and mark it as server_frame_bias.

[0032] Preferably, in step S3, the specified interval time is 5 minutes,

[0033] n = 20 seconds;

[0034] m = 50 milliseconds.

[0035] Preferably, after the system renders the AI detection result obtained by the stream processing service into the corresponding video stream, the rendered video frame is stored in the video frame and AI detection result cache buffer. The rendering method is:

[0036] The system calculates the sum of the current frame number current_frame_id of the video frame and server_frame_bias.

[0037] According to the sum, the AI detection result corresponding to the video frame to be stored in the cache is searched from each AI detection result obtained by the stream processing service. The search success condition is:

[0038] current_frame_id + server_frame_bias = frame_id + frame_bias formula (1)

[0039] In formula (1), current_frame_id represents the frame number of the video frame in the video frame buffer.

[0040] server_frame_bias represents the frame number difference from the leading service frame number obtained by the stream processing service after frame synchronization.

[0041] frame_id represents the corresponding video frame frame number of the detection result stored in the cache by each AI service.

[0042] frame_bias represents the frame number difference from the leading service frame number obtained by each AI service after frame synchronization.

[0043] Render the retrieved AI detection result into the video frame, and then store the video frame into the video frame and AI detection result cache buffer.

[0044] As preferred, the stream processing service pulls the video stream first and then the audio stream when pulling the stream in real time.

[0045] As preferred, the method for matching the audio frame, the video frame and the AI detection result corresponding to the frame number from the audio frame cache buffer and the video frame and AI detection result cache buffer and synthesizing the effect stream after pushing out includes the steps of:

[0046] A1, calculating the video frame pushing interval and the audio frame pushing interval;

[0047] A2, recording the pulling timestamps of the audio stream and the video stream pulled by the stream processing service in real time as t2 and t1 respectively, and then calculating the stream frame number difference stream_bias=(t2-t1) / (1000 / video stream frame rate);

[0048] A3, calculating the frame number of the video stream to be pushed out currently according to the frame number audio_frame of the latest pulled audio stream and the calculated frame number difference stream_bias, which is audio_frame+stream_bias;

[0049] A4, searching in the video frame and AI detection result cache buffer to obtain the video frame with the frame number audio_frame+stream_bias;

[0050] A5, calculating the difference between the pulling timestamp VideoDTS of the video frame with the frame number audio_frame+stream_bias obtained in step A4 and the pulling timestamp audioDTS of the latest pulled audio stream as the timestamp actual offset amount Bav, Bav=VideoDTS-audioDTS;

[0051] A6, calculating the audio-video related unit average offset amount as the standard offset amount σ av ;

[0052] A7, judging whether the absolute value of Bav is less than the standard offset amount,

[0053] If yes, normally push the frame;

[0054] If no, go to step A8;

[0055] A8, judging whether Bav is >0,

[0056] If yes, slow down the rate of pushing the video frame;

[0057] If no, speed up the rate of pushing the video frame;

[0058] A9, the video frame with slowed down or accelerated and rendered by AI detection result and the corresponding audio frame are synthesized as the effect stream.

[0059] As preferred, the calculation method of the video frame pushing interval is:

[0060] Video frame pushing interval = 1000 / video stream frame rate.

[0061] The calculation method of the audio frame pushing interval is:

[0062] Audio frame pushing interval = 1000 / (audio sampling rate / (1000*one frame sampling number)).

[0063] As preferred, in step A6, the standard deviation σ av It is calculated by the following formula:

[0064]

[0065] In formula (2), Gv(n), Pv(n) are the generation time and the playing time of the nth media unit (MU) in the video stream;

[0066] Ga(m), Pa(m) are the generation time and the playing time of the mth MU in the audio stream;

[0067] Ma is the total number of MUs in the audio stream.

[0068] The application also provides an audio and video and AI detection result synchronization system for homologous video streams, which can realize the audio and video and AI detection result synchronization method for homologous video streams, and the system comprises a stream forwarding service module, an AI detection service module, a Redis message channel and a stream processing service module,

[0069] The stream forwarding service module is used to perform a forwarding operation on the source video stream before synchronization processing to remove the stream pulling restriction of the monitoring device generating the source video stream.

[0070] The AI detection service module is used to provide various AI detection services.

[0071] Each AI detection service and the stream processing service module pull the forwarded source video and push the latest frame information of the pulled stream, the AI detection result of each AI detection service to the Redis message channel in real time.

[0072] The stream processing service module is used to perform frame synchronization processing, and match the corresponding AI detection result, audio frame and video frame according to the frame number after synchronization, and push out the effect stream after synthesis.

[0073] The application pushes the detection results of each AI detection service and the pull stream frame number information of the stream processing service to the agreed Redis message channel at each whole second under the premise of having a unified clock, each service can read the pull stream frame number and detection result information of other services in the channel in real time, store them in the memory, perform frame chasing algorithm correction with the real-time frame number of the pull stream of the service, update the frame number difference DFN of the leading frame number HFN in the channel and the latest frame number of a pull stream success service in the same second, so as to match the detection results and audio and video data of each service through the equation (real-time frame number of the service CFN) + DFN = HFN, through the message channel above each service as a bridge, the limitation that the frame synchronization method in the prior art is generally limited to unified memory is largely solved, the purpose of audio and video and detection result synchronization and flexible deployment of each AI service and maximum utilization of hardware resources is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0074] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the embodiments of the application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0075] Figure 1 is a structural schematic diagram of the audio and video and AI detection result synchronization system of the homologous video stream provided by an embodiment of the application;

[0076] Figure 2 is a timing diagram of synchronizing audio and video and AI detection results;

[0077] Figure 3 is a flowchart of synchronizing audio and video and AI detection results;

[0078] Figure 4 is a method schematic diagram of pushing out the effect stream. DETAILED DESCRIPTION

[0079] The technical solutions of the application will be further described below by combining the drawings and through specific embodiments.

[0080] Among them, the drawings are only used for illustrative explanation, and the representation is only a schematic diagram, not a physical diagram, and cannot be understood as a limitation of the patent; in order to better illustrate the embodiments of the application, some components of the drawings will be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it can be understood that some known structures and their descriptions in the drawings can be omitted.

[0081] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that, if the terms "upper", "lower", "left", "right", "inner", "outer" and the like indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and therefore the terms describing the positional relationship in the drawings should not be understood as indicating or implying that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore the terms describing the positional relationship in the drawings are used only for exemplary illustration and cannot be understood as limiting the present patent.

[0082] In the description of the present application, unless otherwise explicitly specified and limited, if the term "connection" and the like indicating the connection relationship between components appears, the term should be understood in a broad sense, for example, it can be a fixed connection, or a detachable connection, or integrated; it can be a mechanical connection, or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium; it can be the communication inside two components or the interaction relationship between two components. For those skilled in the art, the specific meaning of the above-mentioned terms in the present application can be understood according to the specific circumstances.

[0083] The homologous video stream audio and AI detection result synchronization system provided by the embodiments of the present application, as shown in Figure 1 The stream forwarding service module is used to perform a forwarding operation on the source video stream before synchronization processing, so as to release the stream pulling restriction of the monitoring device generating the source video stream, and subsequently, each AI detection service provided by the AI detection service module and the stream processing service module successively pull the forwarded source video stream and push the latest frame information (including the stream pulling frame number) and the AI detection result detected by each AI detection service to the Redis message channel in real time, and then the stream processing service module performs frame synchronization processing, and matches the corresponding AI detection result, audio and video frames (including audio frames and video frames) according to the frame number after synchronization, and pushes out the effect stream after synthesis.

[0084] Figure 2 The timing diagram of synchronizing audio and video with AI detection result is shown. The following describes how the homologous video stream audio and AI detection result synchronization method provided by the embodiments of the present application is implemented: Figure 2 The homologous video stream audio and AI detection result synchronization method provided by the embodiments of the present application, as shown in

[0085] 1. A forwarding operation is performed on the source video stream (original stream) before the service (including AI detection service and stream processing service) pulls the stream, and the purpose of this operation is to release the default stream pulling number restriction of most monitoring cameras, so that each AI detection service can freely pull the stream, reconnect, etc.

[0086] 2、In each service pull stream, each AI detection service and stream processing service needs to ensure a unified clock, that is, different servers apply the same system time. This application uses NTP (Network Time Protocol) to set up master and slave nodes to unify the clock. After each AI detection service is normally started, it will pull and forward the stream in turn. At this time, each service will have a continuously increasing frame number axis based on the time stamp of their successful pull stream.

[0087] 3、The messages pushed by different services to the Redis message channel are divided into the following two categories:

[0088] (1) AI detection service

[0089] On the one hand, each AI detection service pushes real-time frame information to the frame_sync (frame_sync refers to a channel name of the Redis message channel, which is used to distinguish which type of message is received, and the frame_sync channel is used to receive the latest frame number and time stamp of each service pull channel) in the Redis message channel at each whole second. The frame information content includes: server_name: service name; frame_time: frame time stamp (pull frame time, unit: millisecond); frame_id: frame number; frame_bias: frame number difference from the leading frame number (the frame number of the earliest pull frame in the Redis message channel at the current pull stream success time of the service at 8:00:01, such as the pull frame time point of the service is 8:00:00:50, and the pull frame time point of the service 2 is 8:00:00:32. The system records the pull frame time stamp of service 1 and service 2 as 8:00:01. Here it needs to be explained that the pull frame time point of each service is rarely exactly at the whole second, such as the pull frame time point of service 1 is 8:00:00:50, and the pull frame time point of service 2 is 8:00:00:32. The system records the pull frame time stamp of service 1 and service 2 as 8:00:01) frame number difference, frame_bias defaults to 0, and the difference between the internal frame number of the service and the leading frame number will be used as the value of the fram_bias field after each subsequent synchronization check.

[0090] It is emphasized here that although each service controls the sending of frame synchronization messages at each frame second moment, in fact, the time of pulling the frame is just a few cases on the whole second, and most of them are just close to the whole second. The standard for judging this frame information as a whole second frame is that the timestamp of pulling the frame is within 1000 / video stream frame rate / 2 of the upper and lower difference of the whole second.

[0091] For example: assuming that the video stream frame rate is 25, when the timestamp of the pulled frame is within 1000 / 25 / 2=20 milliseconds of the upper and lower difference of the whole second, it is considered as an integer frame.

[0092] For example:

[0093] 1668671985 and 1668672000 differ by 15 milliseconds

[0094] 1668672008 and 1668672000 differ by 8 milliseconds

[0095] Both of these frames are considered as integer frames

[0096] On the other hand, when the AI detection service detects key information in the video stream (such as detecting a face, detecting a human target, ASR speech recognition, financial manager violating customer mobile phone operation, staff not wearing work uniform and nameplate, and dialogue containing prohibited words), it will push the detection result and the frame number of the corresponding frame to the Redis message channel ai_result (the ai_result channel is used to receive the key information detection results of each AI service, such as detecting a face in a frame, face information, and the frame number of the frame). It is noted that the frame number here refers to the value of frame_id+frame_bias, which is used for subsequent synchronization rendering of the detection result. frame_id refers to the frame number of the frame in which the AI service detects the key information, and frame_bias refers to the frame number difference between the frame number of the synchronized frame obtained after frame synchronization of the frame number frame_id detected by this AI service and the frame number of the leading service.

[0097] (2) Stream processing service

[0098] The stream processing service needs to push the real-time frame information of the video stream and the audio stream pulled respectively to the Redis message channel at each whole second moment.

[0099] The real-time frame information pulled by the stream processing service includes frame number and timestamp of pulling the real-time frame.

[0100] 4、When each service successfully pulls the stream, each service will automatically start a timing task that is executed every five minutes. The responsibility of the timing task is to update the frame number difference (i.e., frame_bias) between the real-time frame number of the stream continuously pulled by each service and the frame number of the leading frame, which is the frame number of the first frame pushed by the service that pulls the stream earliest. The calculation of frame_bias and some details of avoiding synchronization errors are as follows:

[0101] First, the system timestamp cTime is obtained;

[0102] It is determined whether cTime is an integer second, i.e., whether cTime%1000 (the % sign represents integer division and remainder, which is used to determine whether the current time point is an integer second, so as to determine whether the frame synchronization can be started) is 0. If it is an integer second, the frame number difference is started to be calculated. If it is not an integer second, the calculation of the frame number difference is started after waiting until the integer second.

[0103] After starting to calculate the frame number difference, the frame information pushed by each AI detection service and stream processing service in the next 20 seconds from the current time is obtained from the Redis message channel. This means that the frame information of each AI detection service and stream processing service in the last 20 seconds is obtained, and a set current_frame_array is created to store the pull stream frame number and pull stream frame timestamp generated by the current service in the 20 seconds.

[0104] Subsequently, the frame information pushed by each service in the 20 seconds is divided into a corresponding frame information set according to the server_name. Each type of service corresponds to a frame information set, and each frame information set contains the frame information pushed by the corresponding type of service in the 20 seconds.

[0105] Then, the frame numbers recorded in the frame information with the same frame_time in each frame information set are extracted,

[0106] The extracted frame numbers are compared in size, and a set of frame information with the largest frame number is compared, which is recorded as head_frame_array.

[0107] Then, the frame_time obtained by traversing the set head_frame_array is searched in the set current_frame_array to determine whether there is a record with a timestamp difference of less than ±50 milliseconds.

[0108] If yes, calculate the frame number difference frame_bias of the two timestamps with a timestamp difference less than ±50 milliseconds, frame_bias = frame_id1-frame_id2, frame_id1 and frame_id2 represent the frame timestamp of this iteration and the retrieved frame timestamp respectively, and store frame_bias in the set frame_bias_array,

[0109] If no, continue the next iteration.

[0110] After the frame number comparison of the 20 seconds is completed, a set frame_bias_array storing the frame number difference between the frame number of the leading service and the real-time frame number of the current service at each whole second moment within the 20 seconds is obtained, the maximum and minimum values in frame_bias_array are removed, and the frame_bias with the highest frequency in frame_bias_array is taken as the latest difference between the frame number of the current service and the frame number of the leading service (i.e., the leading frame number), which is recorded as server_frame_bias.

[0111] 5. Synchronize audio and video with AI detection results

[0112] After the stream processing service successfully pulls the audio stream and the video stream respectively, the system stores the newly pulled video frames, audio frames, and their corresponding frame numbers in a fixed-size set cache in real time. The size of cache depends on the larger value of the buffer threshold value M and the product of the maximum detection time allowed by each AI detection and the video frame rate (i.e., the size of cache is related to M or the product). The threshold value M is calculated by the following formula:

[0113] M = 2 x (dmax-dmin) x r (1)

[0114] Where dmax is the maximum delay of the media unit through the network, dmin is the minimum delay of the media unit through the network, and r is the data transmission rate.

[0115] Taking the ASR detection of a long sentence of about ten seconds as an example, if M < 10*audio frame rate, then the cache size of the audio frame is 10*audio frame rate, otherwise the cache size is M. Similarly, the cache size of the video frame is 10*video frame rate. These two audio and video frame buffers are stored in the memory in the form of a queue. When the set is not full, the latest real-time frame is added. When the set is full, the earliest frame is discarded and the latest real-time frame is added.

[0116] 6. The stream processing service needs to constantly read the various AI detection result information in the ai_result of the Redis message channel, and store these AI detection results in the form of a set in the video frame and AI detection result cache buffer.

[0117] 7. Unlike the direct update of audio frames, before storing the video frame in the video frame and AI detection result cache buffer, the system matches the AI detection results obtained by the stream processing service according to the value of the current frame number current_frame_id+server_frame_bias of the video frame and the frame information of each AI detection result, and draws the matched AI detection results into the corresponding video frame in the form of a frame or subtitles. The matching method is as follows:

[0118] current_frame_id+server_frame_bias=frame_id+frame_bias Formula (2)

[0119] current_frame_id in formula (2) represents the frame number of the video frame in the video frame buffer;

[0120] server_frame_bias represents the frame number difference with the frame number of the leading service obtained by the stream processing service after frame synchronization;

[0121] frame_id represents the corresponding video frame number of the result detected by each AI service stored in the cache;

[0122] frame_bias represents the frame number difference with the frame number of the leading service obtained by each AI service after frame synchronization.

[0123] 8. After successfully pulling the audio and video stream, the stream processing service starts the push stream work of the effect stream, and opens two timing task threads for pushing audio frames and video frames, respectively. The timing interval of each execution of the push frame task depends on the respective audio frame rate or video frame rate and sampling rate, and the calculation method is as follows:

[0124] video frame push frame interval=1000 / video stream frame rate;

[0125] audio frame push frame interval=1000 / (audio sampling rate / (1000*frame sampling number)).

[0126] For the push stream of audio frames, the present application directly extracts the first frame in the audio cache buffer, and pushes the audio frame data into the push streamer every time interval of audio frame push frame interval.

[0127] When pulling streams for a streaming service, the video stream needs to be pulled first, followed by the audio stream. When the video and audio streams are successfully pulled, the corresponding timestamps are recorded as t1 and t2, respectively. Then, the frame difference between the two pulled streams is calculated as stream_bias = (t2-t1) / (1000 / video stream frame rate). Therefore, for pushing video frames, the audio stream frame number audio_frame of the latest audio frame cache data needs to be obtained in real time. At this time, the frame number of the video frame to be pushed is audio_frame + stream_bias. Then, the video frame with frame number audio_frame + stream_bias is retrieved from the video frame and AI detection result cache buffer.

[0128] Then, the difference between the fetch timestamp VideoDTS of the obtained video frame with frame number audio_frame+stream_bias and the fetch timestamp audioDTS of the latest fetched audio stream is calculated as the actual timestamp offset Bav, Bav = VideoDTS - audioDTS.

[0129] Calculate the standard offset σ av The calculation method is as follows:

[0130]

[0131] Where Gv(n) and Pv(n) are the nth media unit (MU) in the video stream. A media unit refers to the actual creative size or length used in a media category, which can be understood as a custom unit length, here equivalent to a video frame and...

[0132] The generation and playback times of audio frames;

[0133] Ga(m) and Pa(m) are the generation time and playback time of the m-th MU in the audio stream;

[0134] Formula (3) assumes that the generation time of the m-th MU in the audio stream is closest to the generation time of the n-th MU in the video stream. Ma is the total number of MUs in the audio stream, and this value is used to measure the average offset of the audio-video related units.

[0135] Media synchronization degree (refers to the value obtained by subtracting the offsets at two different time points to assess the degree of offset).

[0136] It can also be represented using SPD. SPD is defined as the change in time interval between two strongly correlated objects (i.e., two temporally nearest objects) and their original time interval, used to measure the deviation of the currently forward-push audio / video frame from the original time interval.

[0137] The relevant formula is as follows:

[0138] Dav = (P a (m) - P v (n) ) - (G a (m) - G v (n)

[0139] Then compare Bav with the calculated standard deviation σ av , if the actual offset Bav is less than the standard deviation σ av , then normally push the frame, otherwise, further determine whether Bav is ≥ 0,

[0140] If yes, slow down the rate of pushing video frames;

[0141] In the timing task of the video frame pushing interval of 1000 / video stream frame rate, after each frame is pushed, the timing task thread of pushing the video frame is actively blocked, and the blocking time can be distributed to the subsequent ten frames to be pushed, that is, (Bav-σ av ) / 10, until the deviation between the standard deviation and the actual offset is reduced to a small value, which is temporarily set to 1000 / video stream frame rate per video frame pushing time, unit: millisecond. If no, speed up the rate of pushing video frames;

[0142] In the timing task of the video frame pushing interval of 1000 / video stream frame rate, the frame to be pushed at present is discarded, and the next frame in the cache is pushed.

[0143] The pushing of the effect stream is divided into two stages. In the initial stage of pushing, since the first frame of the buffer is pushed each time, the buffer is not full in the filling time of the first few seconds before pushing, and the first frame remains unchanged. At this time, the effect stream will present a short period of output of the same frame and the same audio, which can also be considered as white noise. After the buffer is full and the frame_bias value is updated, the effect stream can present the audio and video synchronous video stream effect with real-time AI detection results. The specific frame synchronization process is shown in Figure 4 .

[0144] The process of frame synchronization will be specifically explained as follows: Figure 4

[0145] ​The stream processing service pulls the video stream and the audio stream in sequence, records the time difference t2-t1 between the two times when the pulling of the stream is successful, and is influenced by the ASR service with a longer detection time. The stream processing service stores the video frames and the audio frames in the first fifteen seconds into a buffer with a length of fifteen seconds, and simultaneously starts to push the effect stream. First, a timing task thread with an execution interval of 1000 / video stream frame rate is started, which constantly pushes the first frame of the picture frame buffer at the original stream video frame pushing rate. Then, a timing task thread with an execution interval of 1000 / (audio sampling rate / (1000*one frame sampling number)) is started after waiting for t2-t1 milliseconds, which constantly pushes the first frame of the audio frame buffer at the original stream audio frame pushing rate. After the stable pulling of the streams by the services, the frame synchronization of the services is started, and the frame number difference of each service with the leading frame is obtained. For the AI detection service, the detected results and the frame number (here, the frame number after synchronization, i.e., the original frame number + the frame number difference of the detection service after synchronization) of the corresponding frame are constantly obtained. At this time, the stream processing service constantly reads the ai detection results from the Redis ai_result message channel and stores them in the detection result cache. When each picture frame is rendered, the detection result cache is traversed, the frame number difference obtained by the current pushed frame and the frame number of each detection result cache are matched, if the frame numbers are the same, the detection result is rendered on the picture frame in the form of subtitles, frames, etc., and is pushed out. Figure 4 For example, the stream processing service pulls the video stream and the audio stream in sequence, obtains t2-t1=the time required for pushing two video frames, preferentially pushes the picture frame, and pushes the audio frame after t2-t1 milliseconds. The ai detection result cache is matched when each picture frame is pushed out. In the figure, the frame number of the picture frame currently pushed by the stream processing service is 15, the frame number difference of the stream processing service relative to the leading service is 4, the detection result with the frame number + the frame number difference of the ai service equal to 19 is retrieved, and is finally rendered on the picture frame and pushed out.

[0146] It should be noted that the above specific embodiments are only the preferred embodiments of the present application and the technical principles applied. Those skilled in the art should understand that various modifications, equivalent replacements, changes, etc. can be made to the present application. However, as long as these changes do not deviate from the spirit of the present application, they should be within the protection scope of the present application. In addition, some terms used in the specification and claims of the present application are not limited, but are only used for convenient description.

Claims

1. A method for synchronizing audio-video and AI detection results of homologous video streams, characterized by the steps of Comprise: Set a uniform clock for each service, including AI detection service and stream processing service; Each of the services pulls the source video stream and pushes the latest frame information of the pulled stream to the Redis message channel in real time, and each of the AI detection services performs corresponding AI detection on the pulled audio stream and video stream, and pushes the detection result to the Redis message channel in real time; Each of the services that successfully pull the stream executes a timing task every specified interval, and the timing task takes the frame number of the frame first pushed by the service that pulls the stream earliest in the task as the leading frame number, and updates the frame number difference between the real-time frame number of the stream it continuously pulls and the leading frame number; The system stores the stream pulled by the stream processing service in real time and each of the AI detection results obtained from the Redis message channel into the corresponding cache buffer, which includes an audio frame cache buffer and a video frame and AI detection result cache buffer; Match the audio frame, video frame and AI detection result corresponding to the frame number from the audio frame cache buffer and the video frame and AI detection result cache buffer and synthesize them as an effect stream; Before step S1, first perform a forwarding operation on the source video stream to remove the default pull stream number limit of the monitoring device that generates the source video stream; The frame information pushed to the Redis message channel includes the service name of the AI detection service, the frame timestamp, the frame number, and the frame number difference with the leading frame number of the pulled frame; Wherein, the method for each service to update its frame number difference comprises the steps of: L1, obtain system timestamp cTime; L2, judge whether cTime is an integer second, If yes, go to step L3; If not, wait until the next second and go to step L3; L3, obtain the frame information completed by each of the services in the n seconds after the integer second from the Redis message channel, and create a set current_frame_array to store the pull stream frame number and pull stream frame timestamp generated by the current service in real time within the n seconds; L4, divide each of the frame information generated by the same service within the n seconds into the frame information set corresponding to the service; L5, extract the frame number recorded in each of the frame information with the same frame timestamp in each of the frame information set; L6, compare the size of each of the frame numbers extracted, and compare the frame information set with the largest frame number, which is recorded as head_frame_array; L7, loop through the set head_frame_array, and search whether the frame timestamp frame_time obtained each time exists in the set current_frame_array with a timestamp difference less than ±m time, If yes, calculate the frame number difference frame_bias between the frame numbers corresponding to the two timestamps with a timestamp difference less than ±m time obtained by this search, and store frame_bias in the set frame_bias_array, If not, continue the next iteration; L8, after the traversal is completed, remove the maximum and minimum values in the set frame_bias_array, and then take the frame_bias with the highest frequency of occurrence in the set frame_bias_array as the latest difference between the frame number of the current service and the leading service, and mark it as server_frame_bias; In step S3, the specified interval time is 5 minutes, n=20 seconds; m=50 milliseconds.

2. The method of claim 1, wherein, After the system stream processing service renders the AI detection result obtained by the system into the corresponding video stream, the rendered video frame is stored in the video frame and AI detection result cache buffer, and the rendering method is: The system calculates the sum of the current frame number current_frame_id of the video frame and server_frame_bias. According to the sum value, the AI detection result corresponding to the video frame to be stored in the cache is searched from the AI detection result obtained by the stream processing service, and the successful search condition is: current_frame_id+server_frame_bias=frame_id+frame_bias In formula 1, current_frame_id represents the frame number of the video frame in the video frame buffer; server_frame_bias represents the frame number difference from the leading service frame number obtained by the stream processing service after frame synchronization; frame_id represents the corresponding video frame frame number of the detection result stored in the cache by each AI service; frame_bias represents the frame number difference from the leading service frame number obtained by each AI service after frame synchronization; Render the retrieved AI detection result into the corresponding video frame, and then store the video frame into the video frame and AI detection result cache buffer.

3. The method of claim 1, wherein, The stream processing service first pulls the video stream and then pulls the audio stream when pulling the stream in real time.

4. The method of claim 3, wherein, The method for matching the frame number corresponding audio frame, video frame and AI detection result from the audio frame cache buffer and the video frame and AI detection result cache buffer and synthesizing to push the effect stream includes the following steps: A1, calculate the video frame pushing interval and the audio frame pushing interval; A2, record the pulling time stamps of the audio stream and the video stream pulled by the stream processing service in real time as t2 and t1 respectively, and then calculate the stream frame number difference stream_bias=(t2-t1) / (1000 / video stream frame rate); A3, according to the frame number audio_frame of the latest pulled audio stream and the calculated frame number difference stream_bias, calculate the frame number of the video stream to be pushed, which is audio_frame+stream_bias; A4, retrieve in the video frame and AI detection result cache buffer to obtain the video frame with frame number audio_frame+stream_bias; A5, calculate the difference between the pull timestamp VideoDTS of the video frame with frame number audio_frame+stream_bias obtained in step A4 and the pull timestamp audioDTS of the latest pulled audio stream in step A3 as the timestamp actual offset amount Bav, Bav=VideoDTS-audioDTS; A6, the calculated average offset of the audio-video correlation unit as the standard offset ; A7, judge whether the absolute value of Bav is less than the standard offset amount, if yes, normally push the frame; if no, turn to step A8; A8, judge whether Bav is >0, if yes, slow down the rate of pushing the video frame; if no, speed up the rate of pushing the video frame; A9, synthesize the video frame after AI detection result rendering and the corresponding audio frame after slowing down or speeding up and push out as the effect stream.

5. The method of claim 4, wherein the method further comprises: The calculation method of the video frame pushing interval is: video frame pushing interval=1000 / video stream frame rate; The calculation method of the audio frame pushing interval is: audio frame pushing interval=1000 / (audio sampling rate / (1000*one frame sampling number)).

6. The method of claim 5, wherein, In step A6, the standard deviation By calculation of the following equation 2: In Equation 2, is the production time and play time of the th media unit (MU) in the video stream; is the production time and play time of the th MU in the audio stream; is the total number of MUs in the audio stream.

7. A system for synchronizing audio and video and AI detection results of homologous video streams, which can implement the method for synchronizing audio and video and AI detection results of homologous video streams according to any one of claims 1-6, characterized in that, The system comprises a stream forwarding service module, an AI detection service module, a Redis message channel and a stream processing service module, The stream forwarding service module is used to perform a forwarding operation on the source video stream before synchronization processing to remove the pull stream limitation of the monitoring device generating the source video stream; The AI detection service module is used to provide various AI detection services, Each AI detection service and the stream processing service module pull the forwarded source video and push the latest frame information, the AI detection result of each AI detection service to the Redis message channel in real time; The stream processing service module is used to perform frame synchronization processing and match the corresponding AI detection result, audio frame, video frame after synchronization according to the frame number and push out the effect stream after synthesis.

Images