Real-time audio-video processing method and apparatus

Abstract

The application discloses a real-time audio and video processing method and device, and belongs to the field of audio and video detection. The real-time audio and video processing method is applied to a first end, and the method comprises the following steps: pulling a target video frame and target audio from a third end; the target video frame is pushed to the third end by a second end based on a target code rate; the target code rate is determined by the first end based on the content of a video frame before the target video frame in a target audio and video stream; the target code rate is updated based on the content of the target video frame; and the updated target code rate is used to push a video frame after the target video frame. The real-time audio and video processing method can dynamically adjust the code rate of the next pushed video frame through the content of the target video frame, can realize dynamic adjustment of the code rate, can improve the video transmission efficiency while guaranteeing the quality of the video, can improve the real-time quality inspection efficiency, and thus can help improving the interactive experience of users.

Description

Technical Field

[0001] This application belongs to the field of audio and video detection, and in particular relates to a real-time audio and video processing method and apparatus. Background Technology

[0002] In bank dual-recording scenarios, real-time detection of user-uploaded audio and video is required. Related technologies primarily involve the device capturing and encoding the audio and video streams, then pushing the encoded streams to a streaming media server. The server then retrieves the streams, decodes them into video frames and audio segments for detection. However, changes in the quality and size of the audio and video captured by the device, as well as the recording environment, often affect the streaming rate, thus impacting transmission efficiency and real-time quality inspection efficiency. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a real-time audio and video processing method and apparatus, which can dynamically adjust the bitrate, improving video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby contributing to a better user experience.

[0004] In a first aspect, this application provides a real-time audio and video processing method, applied to a first end, the method comprising:

[0005] The target video frame and target audio are retrieved from the third end; the target video frame is pushed to the third end by the second end based on the target bitrate, and the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream;

[0006] Based on the content of the target video frame, the target bitrate is updated, and the updated target bitrate is used to push video frames after the target video frame.

[0007] According to the real-time audio and video processing method of this application, the bitrate of the next pushed video frame can be dynamically adjusted by the content of the target video frame. This enables dynamic bitrate adjustment, allowing the selection of the optimal encoding bitrate for the video frame based on its specific content under fixed bandwidth conditions. This improves video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby enhancing the user's interactive experience.

[0008] According to one embodiment of this application, updating the target bitrate based on the content of the target video frame includes:

[0009] If the content of the target video frame is associated with the target content, the target bitrate is updated to the first bitrate;

[0010] If the content of the target video frame is not associated with the target content, the target bitrate is updated to a second bitrate; wherein the second bitrate is lower than the first bitrate.

[0011] According to one embodiment of this application, it also includes:

[0012] If the content of the received current video frame is detected to be associated with the target content for the first time, the sub-region in the content of the current video frame that is associated with the target content is determined as the template image;

[0013] Based on the template image, the content of subsequently received video frames is detected;

[0014] If the content of a subsequently received video frame is similar to the template image, it is determined that the content of the subsequently received video frame is associated with the target content.

[0015] According to one embodiment of this application, detecting the content of subsequently received video frames based on the template image includes:

[0016] Multiple sub-image blocks of a target size are extracted from the subsequently received video frames, where the target size is the size of the template image;

[0017] The template image and each of the sub-image blocks are subjected to fast FFT calculation to detect sub-image blocks similar to the template image from each of the sub-image blocks.

[0018] According to one embodiment of this application, after retrieving the target video frame and target audio from the third end, the method further includes:

[0019] The target video frame and the target audio are decoded respectively to obtain the first video frame and the first audio;

[0020] The first video frame and the first audio frame are detected respectively, and the video detection result and the audio detection result are output.

[0021] Secondly, this application provides a real-time audio and video processing method, applied to a second end, the method comprising:

[0022] Acquire a target audio / video stream, wherein the target audio / video stream includes multiple video frames;

[0023] The target video frame from the plurality of video frames is pushed to the third end based on the target bitrate; the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream.

[0024] Receive the updated target bitrate;

[0025] Based on the updated target bitrate, video frames following the target video frame are pushed to the third end.

[0026] According to the real-time audio and video processing method of this application, the bitrate of the next pushed video frame can be dynamically adjusted by the content of the target video frame. This enables dynamic bitrate adjustment, allowing the selection of the optimal encoding bitrate for the video frame based on its specific content under fixed bandwidth conditions. This improves video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby enhancing the user's interactive experience.

[0027] Thirdly, this application provides a real-time audio and video processing method applied to a third end, the method comprising:

[0028] The first end receives the target video frame pushed by the second end based on the target bitrate; the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream; the target video frame is pulled by the first end to update the target bitrate based on the content of the target video frame, and the updated target bitrate is used to push the video frames following the target video frame.

[0029] According to the real-time audio and video processing method of this application, the bitrate of the next pushed video frame can be dynamically adjusted by the content of the target video frame. This enables dynamic bitrate adjustment, allowing the selection of the optimal encoding bitrate for the video frame based on its specific content under fixed bandwidth conditions. This improves video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby enhancing the user's interactive experience.

[0030] Fourthly, this application provides a real-time audio and video processing apparatus applied to a first end, the apparatus comprising:

[0031] The first processing module is used to pull target video frames and target audio from the third end; the target video frame is pushed to the third end by the second end based on the target bitrate, and the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream;

[0032] The second processing module is used to update the target bitrate based on the content of the target video frame, and the updated target bitrate is used to push video frames after the target video frame.

[0033] According to the real-time audio and video processing apparatus of this application, the bitrate of the next pushed video frame can be dynamically adjusted by the content of the target video frame. This enables dynamic bitrate adjustment, allowing the selection of the optimal encoding bitrate for the video frame based on its specific content under fixed bandwidth conditions. This improves video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby enhancing the user's interactive experience.

[0034] Fifthly, this application provides a real-time audio and video processing apparatus for use at a second end, the apparatus comprising:

[0035] The third processing module is used to acquire the target audio and video stream, which includes multiple video frames.

[0036] The fourth processing module is used to push the target video frame from the plurality of video frames to the third end based on the target bitrate; the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream;

[0037] The fifth processing module is used to receive the updated target bitrate;

[0038] The sixth processing module is used to push video frames following the target video frame to the third end based on the updated target bitrate.

[0039] According to the real-time audio and video processing apparatus of this application, the bitrate of the next pushed video frame can be dynamically adjusted by the content of the target video frame. This enables dynamic bitrate adjustment, allowing the selection of the optimal encoding bitrate for the video frame based on its specific content under fixed bandwidth conditions. This improves video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby enhancing the user's interactive experience.

[0040] In a sixth aspect, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the real-time audio and video processing method as described in the first, second, and third aspects above.

[0041] In a seventh aspect, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the real-time audio and video processing methods described in the first, second, and third aspects above.

[0042] Eighthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the real-time audio and video processing methods as described in the first, second, and third aspects.

[0043] Ninthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the real-time audio and video processing methods described in the first, second, and third aspects above.

[0044] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects:

[0045] By dynamically adjusting the bitrate of the next video frame pushed by adjusting the content of the target video frame, dynamic bitrate adjustment can be achieved. Under fixed bandwidth conditions, the optimal encoding bitrate for the video frame can be selected based on its specific content. This improves video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby enhancing the user's interactive experience.

[0046] Furthermore, by dynamically adjusting the bitrate of the next pushed video frame based on the content of the target video frame, key frames associated with the target content are transmitted at a higher bitrate, while non-key frames not associated with the target content are transmitted at a lower bitrate. Under fixed bandwidth conditions, the optimal encoding bitrate for the pushed video frame is selected based on its specific content. This ensures video quality and does not affect the quality inspection results while improving video transmission efficiency and real-time quality inspection efficiency, thereby helping to improve the user's interactive experience.

[0047] Furthermore, by first detecting the initial video frame associated with the target content, the subsequent detection process uses the image similarity with the initial keyframe to determine whether the subsequent frame is a keyframe. This can significantly save computational overhead, improve detection efficiency, and enhance the accuracy of subsequent detection results.

[0048] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0049] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0050] Figure 1 This is one of the flowcharts illustrating the real-time audio and video processing method provided in the embodiments of this application;

[0051] Figure 2 This is a second schematic flowchart of the real-time audio and video processing method provided in the embodiments of this application;

[0052] Figure 3 This is the third flowchart illustrating the real-time audio and video processing method provided in the embodiments of this application;

[0053] Figure 4 This is the fourth flowchart of the real-time audio and video processing method provided in the embodiments of this application;

[0054] Figure 5 This is one of the structural schematic diagrams of the real-time audio and video processing device provided in the embodiments of this application;

[0055] Figure 6 This is a second schematic diagram of the structure of the real-time audio and video processing device provided in the embodiments of this application;

[0056] Figure 7 This is the third schematic diagram of the structure of the real-time audio and video processing device provided in the embodiments of this application;

[0057] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0059] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0060] The real-time audio and video processing method, real-time audio and video processing device, electronic device and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0061] Among them, real-time audio and video processing methods can be applied to terminals, specifically executed by hardware or software within the terminal.

[0062] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets. It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer.

[0063] The real-time audio and video processing method provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the real-time audio and video processing method. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The real-time audio and video processing method provided in this application embodiment will be described below using an electronic device as the execution subject as an example.

[0064] It should be noted that the real-time audio and video processing method provided in this application embodiment is applied to bank dual recording scenarios.

[0065] In the dual recording scenario of banks, the device side, streaming media server side and server side are involved.

[0066] The device is used to capture audio and video data and display the final processing results.

[0067] The streaming media server is electrically connected to both the device and the server to receive audio and video pushed by the device for the server to retrieve.

[0068] The server is used to pull audio and video from the streaming media server, perform relevant processing on the audio and video to generate processing results, and return the processing results to the device.

[0069] In the following text, the first end will refer to the server end, the second end to the device end, and the third end to the streaming media server end.

[0070] Among them, the device side includes, but is not limited to: mobile phones, PCs, kiosks, and other smart devices.

[0071] like Figure 1 As shown, the real-time audio and video processing method is applied to the first end, and the method includes: step 110 and step 120.

[0072] Step 110: Pull the target video frame and target audio from the third end; the target video frame is pushed to the third end by the second end based on the target bitrate, and the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream;

[0073] In this step, the first end is the server, the second end is the device, and the third end is the streaming media server.

[0074] The target video frame and target audio are the video frame and audio in the target audio and video stream.

[0075] The target audio and video stream is an audio and video stream captured by the second end, which includes multiple video frames and multiple audio files.

[0076] The target audio and video stream is the audio and video that needs to be detected for video content and / or audio content.

[0077] In a bank's dual-recording scenario, the target audio and video stream can be the recorded video of a user during the transaction process. In this scenario, the amount of data in the target audio and video stream to be transmitted is relatively large, while the network bandwidth used for transmission is generally fixed.

[0078] In actual execution, the second end can acquire video frames through an image sensor and audio through a microphone.

[0079] After the second end acquires the target audio and video stream, it uses a communication protocol to push the target audio and video stream to the third end. During the pushing process, the video encoding bitrate is dynamically adjusted based on the content in the video frames.

[0080] The communication protocol can be the RTMP protocol.

[0081] In some embodiments, the target audio may also be audio encoded based on a target bitrate.

[0082] Step 120: Based on the content of the target video frame, update the target bitrate. The updated target bitrate is used to push video frames after the target video frame.

[0083] In this step, the target bitrate is the video encoding bitrate.

[0084] The target bitrate is used by the third end to push video frames after the target video frame to the second end based on the target bitrate.

[0085] It should be noted that the target bitrate is a variable. The target bitrate varies based on the content of the target video frames.

[0086] For example, when the target video frame contains a lot of content, the target bitrate can be increased to improve image quality; when the target video frame contains little content, the target bitrate can be decreased to increase the push rate.

[0087] For example, if the target video frame contains key information, the target bitrate can be increased to improve image quality; if the target video frame does not contain key information, the target bitrate can be decreased to increase the push rate.

[0088] In actual implementation, the correspondence between content and target bitrate can be customized according to actual needs, and this application does not impose any restrictions.

[0089] like Figure 2 As shown, in the actual execution process, the second end acquires the target audio and video stream and pushes the video frame data in the target audio and video stream to the third end based on the dynamic bitrate.

[0090] The first end pulls video frame data from the target audio and video stream from the third end.

[0091] During the push process, the first end updates the target bitrate of the next video frame pushed by the second end to the third end based on the content of the currently received video frame.

[0092] The second end can receive the updated target bitrate from the first end in real time, and use the updated target bitrate as the encoding bitrate of the video frame currently pushed by the second end to encode the current video frame, and then push the encoded video frame to the third end.

[0093] The first end pulls a new encoded video frame from the third end. After parsing the video frame, it updates the target bitrate based on the content of the parsed video frame. The updated target bitrate is used by the second end to push video frames to the third end next time, and so on, until the entire target audio and video stream has been sent.

[0094] In this application, by detecting the content of video frames and adjusting the transmission bitrate based on the content, dynamic bitrate adjustment is achieved. This allows for the transmission of important content at a higher bitrate under fixed bandwidth conditions, thereby improving image quality, while transmitting other content at a lower bitrate, thus increasing transmission speed. This improves video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby enhancing the interactive experience.

[0095] According to the real-time audio and video processing method provided in the embodiments of this application, the bitrate of the next pushed video frame can be dynamically adjusted by the content of the target video frame. This enables dynamic adjustment of the bitrate, allowing the selection of the optimal encoding bitrate for the video frame based on its specific content under fixed bandwidth conditions. This improves video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby enhancing the user's interactive experience.

[0096] In some embodiments, step 120 may include:

[0097] If the content of the target video frame is associated with the target content, the target bitrate is updated to the first bitrate;

[0098] If the content of the target video frame is not related to the target content, the target bitrate is updated to the second bitrate; where the second bitrate is lower than the first bitrate.

[0099] In this embodiment, the target content is used to characterize the core content related to the business in the current scenario.

[0100] For example, in a bank's dual-recording scenario, the target content may include: presenting identification documents, certificates, product forms, and product information. The target content must be in high-resolution image format for detection.

[0101] Other data can include face detection and signing actions. These can be detected using only low-resolution images.

[0102] The target content can be user-defined or automatically determined by the terminal based on the current application scenario; this application does not impose any restrictions.

[0103] In this embodiment, video frames associated with the content and the target content can be identified as key frames, which are video frames that are strongly related to the current business scenario.

[0104] The first bitrate is higher than the second bitrate, meaning that the quality of video frames encoded based on the first bitrate is higher than that of video frames encoded based on the second bitrate.

[0105] In actual execution, the first end can detect the content of the received video frames and determine whether the content is related to the target content.

[0106] For example, if the first end determines that the content of video frame A is associated with the target content, it can approximate the video frame data transmitted in the current period and in the subsequent period (such as video frame B and video frame C) as key frames. Then, it can appropriately increase the bit rate of the subsequently transmitted video frames (such as increasing it to the first bit rate) to improve the quality of video frame B and video frame C.

[0107] If the first end determines that the content of video frame C is not related to the target content, it can be approximated that the video frame data transmitted in the current period and for a period of time thereafter (such as video frame D) are non-key frames. Then, the bit rate of the subsequently transmitted video frames can be appropriately reduced (such as reduced to the second bit rate) to improve the transmission rate of video frame D.

[0108] Determining whether the content of a target video frame is related to the target content can be done by frame-by-frame image recognition, such as by calling a neural network model for recognition and judgment; or by performing image recognition on some frames by skipping frames; or by any other feasible algorithm, which is not limited in this application.

[0109] According to the real-time audio and video processing method provided in the embodiments of this application, the bitrate of the next pushed video frame is dynamically adjusted based on the content of the target video frame. This enables key frames associated with the target content to be transmitted at a higher bitrate, while non-key frames not associated with the target content are transmitted at a lower bitrate. Under fixed bandwidth conditions, the optimal encoding bitrate for the pushed video frame is selected based on its specific content. This ensures video quality and does not affect the quality inspection results while improving video transmission efficiency and real-time quality inspection efficiency, thereby helping to improve the user's interactive experience.

[0110] The following section explains the process of determining whether the content of a target video frame is related to the target content from one implementation perspective.

[0111] In some embodiments, the method may further include:

[0112] If the content of the received current video frame is detected to be associated with the target content for the first time, the sub-region in the content of the current video frame that is associated with the target content is determined as the template image;

[0113] Based on the template image, detect the content of subsequent received video frames;

[0114] If the content of a subsequently received video frame is similar to the template image, it is determined that the content of the subsequently received video frame is associated with the target content.

[0115] In this embodiment, if the content of the received current video frame is detected to be associated with the target content for the first time, the first detected video frame can be determined as the initial keyframe.

[0116] Understandably, the content of this initial keyframe may include sub-regions associated with the target content as well as sub-regions not associated with the target content (such as background environment regions).

[0117] In this embodiment, the sub-region associated with the target content in the initial keyframe can be determined as a template image. The size of the template image is the target size. It is understood that the target size should not be larger than the size of the initial keyframe.

[0118] In subsequent detection, the similarity between the image and the initial keyframe is used to determine whether the subsequent video frames are related to the target content.

[0119] If a video frame is determined to be similar to the initial keyframe, it can be identified as a keyframe; if it is determined to be dissimilar to the initial keyframe, it can be identified as a non-keyframe.

[0120] During the research and development process, the inventors discovered that the relevant technologies require calling multiple deep learning training models and performing calculations on each frame or skipped frames, resulting in a large amount of computation, low detection efficiency, and blindness. In addition, frame skipping detection can easily lead to the loss of video details, thereby affecting the quality of subsequent detection.

[0121] In this application, the algorithm first detects the first video frame (i.e. the initial keyframe) associated with the target content. In subsequent detection processes, the similarity between the image and the initial keyframe is used to determine whether subsequent video frames are keyframes. This eliminates the need to call multiple models, thus significantly saving computational overhead and improving detection efficiency. In addition, it avoids the loss of video details caused by frame skipping detection, thereby improving the accuracy of subsequent detection results.

[0122] According to the real-time audio and video processing method provided in the embodiments of this application, by first detecting the initial video frame associated with the target content, and then determining whether the subsequent video frame is a key frame by the image similarity with the initial key frame in the subsequent detection process, it can significantly save computational overhead, improve detection efficiency and the accuracy of subsequent detection results.

[0123] In some embodiments, detecting the content of subsequently received video frames based on a template image may include:

[0124] Extract multiple sub-image blocks of the target size from the subsequently received video frames; the target size is the size of the template image.

[0125] The template image and each sub-image block are subjected to fast FFT calculation, and sub-image blocks similar to the template image are detected from each sub-image block.

[0126] In this embodiment, the target size is the size of the sub-region associated with the target content in the content of the current video frame.

[0127] For example, when a video frame that meets the requirements (such as an image frame containing the target content) is detected for the first time, this video frame is marked as the first frame. The main body of the image containing the target content can be set as a template image T, and the image size (i.e., the target size) is m×n. Subsequent video frames are I, with a size of M×N, where m≤M and n≤N.

[0128] Let I be a sub-image block of size m×n with (x,y) as the top-left vertex in subsequent video frame I. x，y The sub-image block I x，y The correlation coefficient with the template image T can be defined as:

[0129]

[0130] Where ρ(x, y) is the correlation coefficient; m×n is the sub-image patch I x，y Target size; I x，y (k, l) is used to characterize image patch I x，y The pixel value at pixel (k, l); Used to characterize image block I x，y The average pixel value; T(k, l) is used to characterize the pixel value at pixel (k, l) in the template image T; Used to characterize the average pixel value of the template image T.

[0131] Simplifying the above formula yields:

[0132]

[0133] make:

[0134]

[0135]

[0136]

[0137] but:

[0138]

[0139] Where R1(x, y) is the sub-image patch I x，y The correlation with the template image T can be quickly calculated using the Fast Fourier Transform (FFT). Based on the properties of the FFT, we can obtain:

[0140]

[0141] Where I(u, v) is the Fourier transform of the reference image (i.e., subsequent video frames) I, and T * u and v are the conjugates of the Fourier transforms of the template image T. After the above transformation, the sub-image blocks for all search locations can be calculated in one step as I. x，y Related to template image T.

[0142] Through numerous experiments by the inventors, the aforementioned modifications allow for the simultaneous calculation of the correlation between all search location sub-image blocks I and the template image T. By rapidly comparing the correlation between subsequent frames and the template image, and avoiding continuous calls to the object detection model, computation time can be saved by 40-60%.

[0143] According to the real-time audio and video processing method provided in this application embodiment, keyframe detection is performed using the aforementioned algorithm. By combining the recording specifications for dual-recording scenarios, image and audio processing can be performed scientifically and effectively. By comprehensively utilizing algorithms such as target localization, face detection, action detection, OCR, and text similarity comparison, computation time can be significantly saved, computational efficiency can be improved, and real-time detection of audio and video can be achieved. Moreover, users will not notice any lag in the processing, resulting in a superior user experience.

[0144] Continue to refer to Figure 2 In some embodiments, after step 110, the method may further include:

[0145] Decode the target video frame and target audio respectively to obtain the first video frame and the first audio;

[0146] The first video frame and the first audio frame are detected separately, and the video detection results and audio detection results are output.

[0147] In this embodiment, the first audio can be represented as an audio segment.

[0148] The first end pulls the audio and video streams from the second end and decodes them into the first video frame and the first audio.

[0149] The first end can detect the image content of the video frame and determine whether the image content conforms to the bank's dual recording process specifications.

[0150] The testing includes, but is not limited to: verifying the authenticity of employee ID cards; verifying the authenticity of ID cards; checking whether the salesperson and customer's faces are clearly visible in the video; checking whether certificates, product forms, and product information are presented in the video; verifying whether the salesperson and customer are always in the shot during the recording process; and checking the customer's signing actions and handwriting.

[0151] In actual execution, an image detection module can be set at the first end to detect the first video frame.

[0152] The first end can also transcribe the first audio into a text segment, and analyze and detect the text segment to determine whether there are any prohibited words or negative words in the text.

[0153] In actual implementation, an audio detection module can be set at the first end to detect the first audio signal.

[0154] Based on the audio and video detection results, video detection results and audio detection results can be generated separately and fed back to the second end in real time, so that the second end can display the video detection results and audio detection results in real time.

[0155] In some embodiments, video and audio detection results may also include error correction guidance.

[0156] In this embodiment, the error correction guidance can be fed back to the second end for display, so as to instruct the user to make corrections.

[0157] Table 1 illustrates the results obtained using the method proposed in this application, which demonstrates high accuracy.

[0158] Table 1

[0159]

[0160] In this application, by combining the recording specifications for dual-recording scenarios, images and audio can be processed scientifically and effectively. By comprehensively utilizing algorithms such as target localization, face detection, action detection, OCR, and text similarity comparison, the system can provide real-time alerts to bank workers regarding any non-compliance in the current audio and video recording process, and issue warnings or prompts.

[0161] According to the real-time audio and video processing method provided in the embodiments of this application, the target video frame and target audio are automatically detected, and accurate detection results can be obtained without manual evaluation. This avoids the influence of the subjectivity of manual evaluation on the accuracy of the detection results, and has high detection efficiency and accuracy, which helps to improve the first-pass rate of the bank's dual recording process.

[0162] The real-time audio and video processing method provided in this application can be executed by a real-time audio and video processing device. This application uses an example of a real-time audio and video processing device executing the real-time audio and video processing method to illustrate the real-time audio and video processing device provided in this application.

[0163] This application also provides a real-time audio and video processing device.

[0164] like Figure 5 As shown, the real-time audio and video processing device is applied to the first end, and the device includes: a first processing module 510 and a second processing module 520.

[0165] The first processing module 510 is used to pull target video frames and target audio from the third end; the target video frame is pushed to the third end by the second end based on the target bitrate, and the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream;

[0166] The second processing module 520 is used to update the target bitrate based on the content of the target video frame. The updated target bitrate is used to push the video frames after the target video frame.

[0167] According to the real-time audio and video processing apparatus provided in the embodiments of this application, the bitrate of the next pushed video frame can be dynamically adjusted by the content of the target video frame. This enables dynamic adjustment of the bitrate, allowing the selection of the optimal encoding bitrate for the video frame based on its specific content under fixed bandwidth conditions. This improves video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby enhancing the user's interactive experience.

[0168] In some embodiments, the second processing module 520 may also be used for:

[0169] If the content of the target video frame is associated with the target content, the target bitrate is updated to the first bitrate;

[0170] If the content of the target video frame is not related to the target content, the target bitrate is updated to the second bitrate; where the second bitrate is lower than the first bitrate.

[0171] In some embodiments, the device may further include:

[0172] The seventh processing module is used to determine the sub-region in the content of the current video frame that is associated with the target content as a template image when the content of the received current video frame is first detected to be associated with the target content.

[0173] The eighth processing module is used to detect the content of subsequently received video frames based on the template image;

[0174] The ninth processing module is used to determine the association between the content of the subsequently received video frames and the target content when the content of the subsequently received video frames is similar to the template image.

[0175] In some embodiments, the eighth processing module can also be used for:

[0176] Extract multiple sub-image blocks of the target size from the subsequently received video frames; the target size is the size of the template image.

[0177] The template image and each sub-image block are subjected to fast FFT calculation, and sub-image blocks similar to the template image are detected from each sub-image block.

[0178] In some embodiments, the device may further include:

[0179] The tenth processing module is used to decode the target video frame and target audio respectively after pulling the target video frame and target audio from the third end to obtain the first video frame and the first audio.

[0180] The eleventh processing module is used to detect the first video frame and the first audio frame respectively, and output the video detection result and the audio detection result.

[0181] This application also provides another real-time audio and video processing method.

[0182] like Figure 3 As shown, the real-time audio and video processing method is applied to the second end, and the method includes steps 310, 320, 330 and 340.

[0183] Step 310: Obtain the target audio and video stream, which includes multiple video frames;

[0184] In this step, the second end is the device end.

[0185] The target audio and video stream is an audio and video stream captured by the second end, which includes multiple video frames and multiple audio files.

[0186] The target audio and video stream is the audio and video that needs to be detected for video content and / or audio content.

[0187] In some embodiments, step 310 may further include: performing noise reduction processing on the acquired initial audio and video stream to obtain the target audio and video stream.

[0188] In this embodiment, the initial audio and video stream is the raw, unprocessed audio and video captured by the device.

[0189] Noise reduction processing of the acquired initial audio and video streams may include: performing a convolution operation on multiple acquired video frames using a Gaussian filter to obtain the processed target video frame.

[0190] In some embodiments, the Gaussian filter is a 3*3 Gaussian filter with σ = 0.8.

[0191]

[0192] In this embodiment, noise reduction processing facilitates subsequent character region positioning and connected component search operations, thereby improving the accuracy of the processing results.

[0193] Step 320: Push the target video frame from multiple video frames to the third end based on the target bitrate; the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream.

[0194] In this step, the third end is the streaming media server.

[0195] The target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio / video stream.

[0196] The target bitrate is a variable.

[0197] The target bitrate varies based on the content of the target video frames.

[0198] For example, when the target video frame contains a lot of content, a higher target bitrate can be used to push the video frame to improve image quality; when the target video frame contains less content, a lower target bitrate can be used to push the video frame to improve the push rate.

[0199] For example, if the target video frame contains key information, a higher target bitrate can be used to push the video frame to improve image quality; if the target video frame does not contain key information, a lower target bitrate can be used to push the video frame to improve the push rate.

[0200] In actual implementation, the correspondence between content and target bitrate can be customized according to actual needs, and this application does not impose any restrictions.

[0201] In some embodiments, the first end may use the RTMP protocol to push target video frames encoded based on the target bitrate to the third end.

[0202] In this embodiment, by adopting the RTMP protocol, the latency of audio and video transmission can be controlled between 0.8 and 3 seconds, achieving near real-time audio and video processing.

[0203] In some embodiments, the target audio / video stream may further include multiple audio segments, and after step 310, the method may further include: sending the target audio to a third end based on the target bitrate.

[0204] In some embodiments, the first end may use the RTMP protocol to push target audio encoded based on the target bit rate to the third end.

[0205] Step 330: Receive the updated target bitrate;

[0206] In this step, the updated target bitrate is used to push video frames following the target video frame to the third end.

[0207] Step 340: Based on the updated target bitrate, push the video frames following the target video frame to the third end.

[0208] In this step, continue to refer to Figure 2 In actual execution, the second end acquires the target audio and video stream and pushes the video frame data in the target audio and video stream to the third end based on the dynamic bitrate. The first end pulls the encoded video frame data from the third end.

[0209] During the push process, the first end decodes the retrieved video frame data and updates the target bitrate of the next video frame pushed by the second end to the third end based on the content of the currently received video frame.

[0210] The second end can receive the updated target bitrate from the first end in real time, and use the updated target bitrate as the encoding bitrate of the video frame currently pushed by the second end to encode the current video frame, and then push the encoded video frame to the third end.

[0211] The first end pulls a new encoded video frame from the third end. After parsing the video frame, it updates the target bitrate based on the content of the parsed video frame. The updated target bitrate is used by the second end to push video frames to the third end next time, and so on, until the entire target audio and video stream has been sent.

[0212] According to the real-time audio and video processing method provided in the embodiments of this application, the bitrate of the next pushed video frame can be dynamically adjusted by the content of the target video frame. This enables dynamic adjustment of the bitrate, allowing the selection of the optimal encoding bitrate for the video frame based on its specific content under fixed bandwidth conditions. This improves video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby enhancing the user's interactive experience.

[0213] In some embodiments, step 320 may further include:

[0214] If the content of the video frame preceding the target video frame is associated with the target content, the target video frame is pushed to the third end at the first bitrate.

[0215] If the content of the video frames preceding the target video frame is not related to the target content, the target video frame is pushed to the third end at the second bitrate.

[0216] In this case, the second bitrate is lower than the first bitrate, and the target content is determined by the first end.

[0217] In this embodiment, the target content is used to characterize the core content related to the business in the current scenario.

[0218] For example, in a bank's dual recording scenario, the target content may include: presenting identification documents, presenting certificates, product forms, and product information.

[0219] The target content can be user-defined or automatically determined by the terminal based on the current application scenario; this application does not impose any restrictions.

[0220] The first bitrate is higher than the second bitrate, meaning that the quality of video frames encoded based on the first bitrate is higher than that of video frames encoded based on the second bitrate.

[0221] For example, if the first end determines that the content of video frame A is associated with the target content, it can appropriately increase the bitrate of the subsequently transmitted video frames (such as video frame B and video frame C) (such as increasing it to the first bitrate). The second end then pushes the subsequently transmitted video frames based on the first bitrate to improve the quality of video frame B and video frame C.

[0222] If the first end determines that the content of video frame C is not related to the target content, it can appropriately reduce the bit rate of the subsequent transmitted video frames (such as video frame D) (such as reducing it to the second bit rate). The second end then pushes the subsequent transmitted video frames based on the second bit rate to improve the transmission rate of video frame D.

[0223] According to the real-time audio and video processing method provided in the embodiments of this application, the bitrate of the next pushed video frame is dynamically adjusted based on the content of the target video frame. This enables key frames associated with the target content to be transmitted at a higher bitrate, while non-key frames not associated with the target content are transmitted at a lower bitrate. Under fixed bandwidth conditions, the optimal encoding bitrate for the pushed video frame is selected based on its specific content. This ensures video quality and does not affect the quality inspection results while improving video transmission efficiency and real-time quality inspection efficiency, thereby helping to improve the user's interactive experience.

[0224] This application also provides a real-time audio and video processing device.

[0225] like Figure 6 As shown, the real-time audio and video processing device is applied to the second end, and the device includes: a first processing module 510 and a second processing module 520.

[0226] The third processing module 610 is used to acquire the target audio and video stream, which includes multiple video frames.

[0227] The fourth processing module 620 is used to push the target video frame from multiple video frames to the third end based on the target bitrate; the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream.

[0228] The fifth processing module 630 is used to receive the updated target bitrate;

[0229] The sixth processing module 640 is used to push video frames after the target video frame to the third end based on the updated target bitrate.

[0230] According to the real-time audio and video processing apparatus provided in the embodiments of this application, the bitrate of the next pushed video frame can be dynamically adjusted by the content of the target video frame. This enables dynamic adjustment of the bitrate, allowing the selection of the optimal encoding bitrate for the video frame based on its specific content under fixed bandwidth conditions. This improves video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby enhancing the user's interactive experience.

[0231] In some embodiments, the fourth processing module 620 may also be used for:

[0232] If the content of the video frame preceding the target video frame is associated with the target content, the target video frame is pushed to the third end at the first bitrate.

[0233] If the content of the video frames preceding the target video frame is not related to the target content, the target video frame is pushed to the third end at the second bitrate.

[0234] In this case, the second bitrate is lower than the first bitrate, and the target content is determined by the first end.

[0235] This application also provides another real-time audio and video processing method.

[0236] like Figure 4 As shown, this real-time audio and video processing method is applied to a third end, and the method includes: step 410.

[0237] Step 410: Receive the target video frame pushed by the second end based on the target bitrate; the target bitrate is determined by the first end based on the content of the video frames before the target video frame in the target audio and video stream; the target video frame is pulled by the first end to update the target bitrate based on the content of the target video frame, and the updated target bitrate is used to push the video frames after the target video frame.

[0238] In this step, the third end is the streaming media server.

[0239] In actual execution, the third end is used to receive the target video frames and audio segments encoded by the second end based on the target bitrate, and then the first end pulls them.

[0240] According to the real-time audio and video processing method provided in the embodiments of this application, the bitrate of the next pushed video frame can be dynamically adjusted by the content of the target video frame. This enables dynamic adjustment of the bitrate, allowing the selection of the optimal encoding bitrate for the video frame based on its specific content under fixed bandwidth conditions. This improves video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby enhancing the user's interactive experience.

[0241] The real-time audio and video processing method provided in this application can be executed by a real-time audio and video processing device. This application uses an example of a real-time audio and video processing device executing the real-time audio and video processing method to illustrate the real-time audio and video processing device provided in this application.

[0242] This application also provides a real-time audio and video processing device.

[0243] like Figure 7 As shown, the real-time audio and video processing device is applied to a third end. The device includes a twelfth processing module 710, which is used to: receive a target video frame pushed by a second end based on a target bitrate; the target bitrate is determined by the first end based on the content of video frames before the target video frame in the target audio and video stream; the target video frame is pulled by the first end to update the target bitrate based on the content of the target video frame, and the updated target bitrate is used to push video frames after the target video frame.

[0244] According to the real-time audio and video processing apparatus provided in the embodiments of this application, the bitrate of the next pushed video frame can be dynamically adjusted by the content of the target video frame. This enables dynamic adjustment of the bitrate, allowing the selection of the optimal encoding bitrate for the video frame based on its specific content under fixed bandwidth conditions. This improves video transmission efficiency and real-time quality inspection efficiency while ensuring video quality, thereby enhancing the user's interactive experience.

[0245] The real-time audio and video processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0246] The real-time audio and video processing device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0247] The real-time audio and video processing device provided in this application embodiment can achieve… Figures 1 to 4 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0248] In some embodiments, such as Figure 8 As shown, this application embodiment also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, it implements the various processes of the above-described real-time audio and video processing method embodiments and can achieve the same technical effects. To avoid repetition, it will not be described again here.

[0249] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0250] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described real-time audio and video processing method.

[0251] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0252] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described real-time audio and video processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0253] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0254] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0255] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0256] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0257] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0258] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A real-time audio and video processing method, characterized in that, Applied to the first end, the method includes: The target video frame and target audio are retrieved from the third end; the target video frame is pushed to the third end by the second end based on the target bitrate, and the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream; Based on the content of the target video frame, the target bitrate is updated, and the updated target bitrate is used to push video frames after the target video frame. Updating the target bitrate based on the content of the target video frame includes: If the content of the target video frame is associated with the target content, the target bitrate is updated to the first bitrate; If the content of the target video frame is not associated with the target content, the target bitrate is updated to a second bitrate; wherein the second bitrate is lower than the first bitrate.

2. The real-time audio and video processing method according to claim 1, characterized in that, Also includes: If the content of the received current video frame is detected to be associated with the target content for the first time, the sub-region in the content of the current video frame that is associated with the target content is determined as the template image; Based on the template image, the content of subsequently received video frames is detected; If the content of a subsequently received video frame is similar to the template image, it is determined that the content of the subsequently received video frame is associated with the target content.

3. The real-time audio and video processing method according to claim 2, characterized in that, The step of detecting the content of subsequently received video frames based on the template image includes: Multiple sub-image blocks of a target size are extracted from the subsequently received video frames, where the target size is the size of the template image; The template image and each of the sub-image blocks are subjected to fast FFT calculation to detect sub-image blocks similar to the template image from each of the sub-image blocks.

4. The real-time audio and video processing method according to any one of claims 1-3, characterized in that, After retrieving the target video frame and target audio from the third end, the method further includes: The target video frame and the target audio are decoded respectively to obtain the first video frame and the first audio; The first video frame and the first audio frame are detected respectively, and the video detection result and the audio detection result are output.

5. A real-time audio and video processing method, characterized in that, Applied to the second end, the method includes: Acquire a target audio / video stream, wherein the target audio / video stream includes multiple video frames; The target video frame from the plurality of video frames is pushed to the third end based on the target bitrate; the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream. Receive the updated target bitrate; Based on the updated target bitrate, push the video frames following the target video frame to the third end; The target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio / video stream, including: If the content of the video frame preceding the target video frame is associated with the target content, the target bitrate is determined as the first bitrate; If the content of video frames preceding the target video frame is not related to the target content, the target bitrate is determined as a second bitrate; wherein the second bitrate is lower than the first bitrate.

6. A real-time audio and video processing method, characterized in that, Applied to a third end, the method includes: The first end receives the target video frame pushed by the second end based on the target bitrate; the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream; the target video frame is pulled by the first end to update the target bitrate based on the content of the target video frame, and the updated target bitrate is used to push the video frames following the target video frame; The updating of the target bitrate includes: If the content of the target video frame is associated with the target content, the target bitrate is updated to the first bitrate; If the content of the target video frame is not associated with the target content, the target bitrate is updated to a second bitrate; wherein the second bitrate is lower than the first bitrate.

7. A real-time audio and video processing device, characterized in that, Applied to the first end, the device includes: The first processing module is used to pull target video frames and target audio from the third end; the target video frame is pushed to the third end by the second end based on the target bitrate, and the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream; The second processing module is used to update the target bitrate based on the content of the target video frame, and the updated target bitrate is used to push video frames after the target video frame. The second processing module is configured to: update the target bitrate to a first bitrate when the content of the target video frame is associated with the target content; If the content of the target video frame is not associated with the target content, the target bitrate is updated to a second bitrate; wherein the second bitrate is lower than the first bitrate.

8. A real-time audio and video processing device, characterized in that, Applied to the second end, the device includes: The third processing module is used to acquire the target audio and video stream, which includes multiple video frames. The fourth processing module is used to push the target video frame from the plurality of video frames to the third end based on the target bitrate; the target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio and video stream; The fifth processing module is used to receive the updated target bitrate; The sixth processing module is used to push video frames after the target video frame to the third end based on the updated target bitrate. The target bitrate is determined by the first end based on the content of the video frames preceding the target video frame in the target audio / video stream, including: If the content of the video frame preceding the target video frame is associated with the target content, the target bitrate is determined as the first bitrate; If the content of video frames preceding the target video frame is not related to the target content, the target bitrate is determined as a second bitrate; wherein the second bitrate is lower than the first bitrate.

Images