Audio and video lag detection method and device, electronic equipment and storage medium

Abstract

The invention provides an audio and video lag detection method and device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, and is suitable for the field of financial science and technology. The method comprises the following steps: acquiring a target audio / video; performing audio and video separation on the target audio and video to obtain a target video and a target audio; performing frame extraction on the target video according to a preset period to obtain image frames; performing audio sampling on the target audio according to a preset period to obtain audio sampling points; performing spectrum conversion on the audio sampling points to obtain a spectrogram; when the extraction moment of the image frame is the same as the sampling moment of the audio sampling point, performing image synthesis on the image frame and the spectrogram to obtain a target test chart; determining a lag detection result of the target audio and video according to the difference between any two target test diagrams; wherein the lagging detection result is used for representing that the target audio / video has lagging or does not have lagging. According to the invention, the audio and video lag detection accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, is applicable to the field of financial technology, and particularly relates to a method and device for detecting audio and video freezes, an electronic device, and a storage medium. Background Art

[0002] Audio and video refers to the content formed by audio and video recording. In the field of financial technology, when conducting financial business (such as when banks or insurance companies introduce financial products), it is necessary to record and video the entire process to obtain audio and video, which can facilitate future retrospective management. However, errors may occur during the recording process, resulting in audio and video jams, so jam detection is required.

[0003] In existing freeze detection technologies, freeze detection is usually performed based on the difference between image frames in the video. However, this method ignores the impact of audio on audio and video freezes, resulting in low accuracy in audio and video freeze detection. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method and device for detecting audio and video jams, an electronic device, and a storage medium, which can improve the accuracy of audio and video jam detection.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for detecting audio and video freeze, the method comprising:

[0006] Get the target audio and video;

[0007] Separating the target audio and video to obtain the target video and the target audio;

[0008] Extract frames from the target video according to a preset period to obtain image frames;

[0009] Performing audio sampling on the target audio according to the preset period to obtain audio sampling points;

[0010] Performing spectrum conversion on the audio sampling points to obtain a spectrum graph;

[0011] When the extraction time of the image frame is the same as the sampling time of the audio sampling point, performing image synthesis on the image frame and the spectrum diagram to obtain a target test diagram;

[0012] According to the difference between any two of the target test images, a jam detection result of the target audio and video is determined; wherein the jam detection result is used to characterize whether there is jam or not in the target audio and video.

[0013] In some embodiments, determining the jam detection result of the target audio and video according to the difference between any two of the target test images includes:

[0014] Obtaining a target time of each target test chart; wherein the target time, extraction time and sampling time of the same target test chart are the same;

[0015] Extracting a first target test graph and a second target test graph from each of the target test graphs according to the target time; wherein the target time of the second target test graph is after the target time of the first target test graph;

[0016] Performing image difference calculation according to the first target test image and the second target test image to obtain a first target difference image;

[0017] Perform pixel summation according to each pixel point in the first target difference map to obtain a first test pixel sum;

[0018] The jamming detection result is determined based on the first test pixel sum.

[0019] In some embodiments, determining the jam detection result of the target audio and video according to the difference between any two of the target test images further includes:

[0020] Extracting a third target test pattern from each of the target test patterns according to the target time corresponding to the second target test pattern; wherein the target time of the third target test pattern is after the target time of the second target test pattern;

[0021] Performing image difference calculation according to the first target test image and the third target test image to obtain a second target difference image;

[0022] Perform pixel summation according to each pixel point in the second target difference map to obtain a second test pixel sum;

[0023] Performing image difference calculation according to the second target test image and the third target test image to obtain a third target difference image;

[0024] Perform pixel summation according to each pixel point in the third target difference map to obtain a third test pixel sum;

[0025] Pixel stuttering detection is performed according to the first test pixel sum, the second test pixel sum, and the third test pixel sum to obtain the stuttering detection result.

[0026] In some embodiments, performing pixel jamming detection according to the first test pixel sum, the second test pixel sum, and the third test pixel sum to obtain the jamming detection result includes:

[0027] Construct a jam graph quantity mark, wherein the jam graph quantity mark is 0;

[0028] Taking the first test pixel sum, the second test pixel sum and the third test pixel sum as target test pixel sum in turn;

[0029] If the target test pixel sum is less than or equal to the preset pixel sum threshold, the number mark of the jam image is increased by 1;

[0030] A quantity jam detection is performed according to the jam graph quantity mark to obtain the video jam detection result.

[0031] In some embodiments, when the extraction time of the image frame is the same as the sampling time of the audio sampling point, performing image synthesis on the image frame and the spectrum diagram to obtain a target test diagram includes:

[0032] Scaling the image frame according to a preset target scale, and scaling the spectrogram according to the target scale, so that the scale of the image frame is the same as the scale of the spectrogram;

[0033] When the extraction time of the image frame is the same as the sampling time of the audio sampling point, the spectrum diagram is spliced ​​on the left side of the image frame to obtain the target test diagram.

[0034] In some embodiments, after determining the jam detection result of the target audio and video according to the difference between any two of the target test images, the method further includes:

[0035] If the jam detection result of the target audio and video indicates that there is no jam in the target audio and video, performing face detection on the first target test image to obtain first face area position information;

[0036] Performing region extraction on the first target difference image according to the first face region position information to obtain a first difference image face region;

[0037] Perform pixel summation on each pixel point in the face area of ​​the first difference image to obtain a pixel sum of the first face area;

[0038] According to the first face area pixel sum, update the jam detection result of the target audio and video.

[0039] In some embodiments, after performing audio and video separation on the target audio and video to obtain the target video and the target audio, the method further includes:

[0040] Obtaining the duration of the target video to obtain the target video duration;

[0041] Obtaining the duration of the target audio to obtain the target audio duration;

[0042] Calculate the difference between the target video duration and the target audio duration to obtain an absolute duration difference;

[0043] If the absolute duration difference is greater than a preset duration difference threshold, target prompt information is generated; wherein the target prompt information is used to prompt the re-recording of the target audio and video.

[0044] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an audio and video freeze detection device, the device comprising:

[0045] A data acquisition module, used to acquire target audio and video;

[0046] A data separation module, used to separate the target audio and video to obtain the target video and the target audio;

[0047] An image extraction module, used to extract frames from the target video according to a preset period to obtain image frames;

[0048] An audio sampling module, used to perform audio sampling on the target audio according to the preset period to obtain audio sampling points;

[0049] A spectrum conversion module, used for performing spectrum conversion on the audio sampling points to obtain a spectrum graph;

[0050] An image synthesis module, used for performing image synthesis on the image frame and the spectrum diagram to obtain a target test diagram when the extraction time of the image frame is the same as the sampling time of the audio sampling point;

[0051] The result determination module is used to determine the jam detection result of the target audio and video according to the difference between any two of the target test images; wherein the jam detection result is used to characterize whether the target audio and video has jam or not.

[0052] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the audio and video freeze detection method described in the first aspect when executing the computer program.

[0053] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the audio and video freeze detection method described in the first aspect is implemented.

[0054] The audio and video freeze detection method and device, electronic device, and storage medium proposed in the present application are aimed at the problem that the accuracy rate is low due to the fact that the video freeze detection in the related art only relies on the image information in the video. In the present application, when the target audio and video are obtained, the target audio and video will be separated first to obtain the target video and the target audio, so that the video and audio can be processed independently. Further, the target video is frame extracted according to the preset period to obtain the image frame. In this way, an image frame will be extracted every other preset period, and the extraction time of any two image frames is different. Further, in addition to extracting the image frame according to the preset period, the target audio is sampled according to the preset period to obtain the audio sampling point, and then the spectrum diagram is obtained. In this way, an audio sampling point will be extracted every other preset period, and the sampling time of any two audio sampling points is different. Since the image frame and the audio sampling point are collected based on the preset period, each extraction moment has and only has one sampling moment that is the same as the extraction moment. Furthermore, when the extraction time of the image frame is the same as the sampling time of the audio sampling point, the image frame and the spectrum diagram are synthesized to obtain a target test diagram, so that the target test diagram can represent the audio and video information corresponding to the target audio and video at a certain moment. Finally, the jam detection result of the target audio and video is determined based on the difference between any two target test diagrams. The target audio and video are first separated by audio and video, and then the target test diagram is generated through a series of operations. The target test diagram can represent the audio and video information corresponding to the target audio and video at a certain moment. Therefore, determining the jam detection result based on the target test diagram is equivalent to determining the jam detection result based on the audio and video information of the target audio and video at different moments. The information is richer and the jam detection accuracy is improved.

[0055] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of the audio and video freeze detection method provided in an embodiment of the present application;

[0057] Figure 2 yes Figure 1 Flow chart of step 106 in;

[0058] Figure 3 yes Figure 1 A flow chart of step 107 in FIG.

[0059] Figure 4 yes Figure 1 Another flow chart of step 107 in;

[0060] Figure 5 yes Figure 4Flowchart of step 410 in;

[0061] Figure 6 It is a module structure block diagram of the audio and video freeze detection device provided in an embodiment of the present application;

[0062] Figure 7 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] First, some nouns involved in this application are analyzed:

[0067] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] Audio and video are also called dual-recording videos. Dual recording means that when banks or insurance companies introduce financial products, they need to record and video the entire process. Dual-recording video jam detection refers to manual or intelligent jam detection of audio and video. For example, the data information generated by recording and videotaping the business handling of on-site personnel is obtained by shooting audio and video through dual-recording equipment, and the audio and video are saved in the cloud storage system to monitor the degree of standardization and business level of on-site personnel, as well as to preserve evidence of business-related operations and behaviors. After the recorded audio and video are uploaded to the business quality inspection platform, quality inspection is required to check whether the quality of the recorded audio and video meets the specified requirements. If the requirements are met, the recorded audio and video will be saved, otherwise it needs to be re-recorded. Complete and clear videos are convenient for future retrospective management and for resolving future disputes; therefore, it is necessary to detect whether the video is complete and whether there is jamming to reduce the risk of missing key parts of the video.

[0069] The embodiment of the present application provides a method and device for detecting audio and video freeze, an electronic device, and a storage medium. The core of the present application is to simultaneously consider the impact of audio and video on freeze, thereby improving the accuracy of freeze detection.

[0070] The audio and video freeze detection method provided in the embodiment of the present application can be applied to the terminal and the server side, and can also be software running on the server side. The server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the audio and video freeze detection method, but is not limited to the above forms.

[0071] The present application can be used in many general or special computer system environments or configurations. For example: server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0072] The embodiments of the present application provide an audio and video freeze detection method and an audio and video freeze detection device, an electronic device, and a computer-readable storage medium, which are specifically illustrated by the following embodiments. First, the audio and video freeze detection method in the embodiments of the present application is described.

[0073] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0074] Reference Figure 1 , Figure 1 It is an optional flowchart of the audio and video freeze detection method provided in an embodiment of the present application, which may include but is not limited to steps 101 to 107.

[0075] Step 101, obtaining target audio and video;

[0076] Step 102, separating the target audio and video to obtain the target video and the target audio;

[0077] Step 103, extracting frames from the target video according to a preset period to obtain image frames;

[0078] Step 104, sampling the target audio according to a preset period to obtain audio sampling points;

[0079] Step 105, performing spectrum conversion on the audio sampling points to obtain a spectrum graph;

[0080] Step 106, when the extraction time of the image frame is the same as the sampling time of the audio sampling point, the image frame and the spectrum are synthesized to obtain a target test image;

[0081] Step 107, determining a jam detection result of the target audio or video according to the difference between any two target test images; wherein the jam detection result is used to characterize whether the target audio or video has jams or not.

[0082] Steps 101 to 107 shown in the embodiment of the present application are aimed at the problem that the accuracy rate is low due to the fact that only the image information in the video is relied upon when detecting video freeze in the related art. In this embodiment, when the target audio and video are obtained, the target audio and video will be separated first to obtain the target video and the target audio, so that the video and audio can be processed independently. Further, the target video is frame extracted according to the preset period to obtain the image frame. In this way, an image frame will be extracted every other preset period, and the extraction time of any two image frames is different. Further, in addition to extracting the image frame according to the preset period, the target audio is sampled according to the preset period to obtain the audio sampling point, and then the spectrum diagram is obtained. In this way, an audio sampling point will be extracted every other preset period, and the sampling time of any two audio sampling points is different. Since the image frame and the audio sampling point are collected based on the preset period, each extraction moment has and only has one sampling moment that is the same as the extraction moment. Furthermore, when the extraction time of the image frame is the same as the sampling time of the audio sampling point, the image frame and the spectrum diagram are synthesized to obtain a target test diagram, so that the target test diagram can represent the audio and video information corresponding to the target audio and video at a certain moment. Finally, the jam detection result of the target audio and video is determined based on the difference between any two target test diagrams. The target audio and video are first separated by audio and video, and then the target test diagram is generated through a series of operations. The target test diagram can represent the audio and video information corresponding to the target audio and video at a certain moment. Therefore, determining the jam detection result based on the target test diagram is equivalent to determining the jam detection result based on the audio and video information of the target audio and video at different moments. The information is richer and the jam detection accuracy is improved.

[0083] In step 101 of some embodiments, target audio and video are obtained. The target audio and video refers to the audio and video to be detected for jamming. For example, the data information generated by recording and videotaping the business handling of on-site personnel is obtained by shooting the target audio and video through a terminal (such as a mobile phone or a dual recording device). When the above audio and video jamming detection method is applied to the server side, the server side receives the target audio and video from the terminal (such as a mobile phone or a dual recording device).

[0084] In step 102 of some embodiments, the target audio and video are separated into audio and video to obtain the target video and the target audio. A separation tool (such as FFmpeg, Audacity, Adobe Premiere Pro, Final Cut Pro) can be called to separate the target audio and video.

[0085] In one embodiment, after step 102, the audio and video freeze detection method provided by this embodiment may further include:

[0086] Get the duration of the target video and obtain the duration of the target video;

[0087] Get the duration of the target audio and obtain the duration of the target audio;

[0088] Calculate the difference between the target video duration and the target audio duration to obtain an absolute duration difference;

[0089] If the absolute duration difference is greater than a preset duration difference threshold, target prompt information is generated; wherein the target prompt information is used to prompt the re-recording of the target audio and video.

[0090] For example, the target audio and video are parsed into the target audio and the target video, and the duration of the target audio is t1 seconds, and the duration of the target video is t2 seconds. Then the absolute duration difference is calculated: abs(t2–t1). The duration difference threshold is 30 seconds. If abs(t2–t1) is greater than 30 seconds, the target audio and video is considered incomplete and target prompt information needs to be generated.

[0091] The benefit of the above embodiment is that it can prompt the user to re-record the target audio and video in time during the execution of the jamming detection task, thereby improving the feasibility of task parallelism.

[0092] In one embodiment, step 102 may include: demultiplexing the target audio and video through a demultiplexer to obtain an audio packet queue and a video packet queue; performing audio decoding on the audio packet queue through an audio decoder to obtain the target audio; and performing video decoding on the video packet queue through a video decoder to obtain the target video.

[0093] In step 103 of some embodiments, frame extraction is performed on the target video according to a preset period to obtain image frames. The preset period is, for example, 1 second, 2 seconds, etc. For example, assuming that the preset period is 1 second, step 103 includes: frame extraction is performed on the target video every 1 second to obtain the image frame of the 1st second, the image frame of the 2nd second, the image frame of the 3rd second, the image frame of the 4th second, etc.

[0094] In step 104 of some embodiments, the target audio is sampled according to a preset period to obtain audio sampling points. For example, assuming that the preset period is 1 second, step 104 includes: sampling the target audio every 1 second to obtain the audio sampling point of the 1st second, the audio sampling point of the 2nd second, the audio sampling point of the 3rd second, the audio sampling point of the 4th second, etc.

[0095] It should be noted that the preset period corresponding to frame extraction is the same as the preset period corresponding to audio sampling.

[0096] In step 105 of some embodiments, the audio sampling points are spectrally converted to obtain a spectrogram. For example, for each audio sampling point, the audio sampling point is subsampled at a preset sampling rate (such as 16K) to obtain an audio subsampling point; the audio subsampling point is transformed by Fast Fourier Transform (FFT) to obtain a spectrogram. For example, the mel spectrogram is obtained according to the mel spectrum transformation or the MFCC transformation formula. For the audio sampling point of the 1st second, the audio sampling point of the 2nd second, the audio sampling point of the 3rd second, and the audio sampling point of the 4th second in the above example, the spectrogram of the 1st second, the audio sampling point of the 2nd second, the audio sampling point of the 3rd second, and the audio sampling point of the 4th second can be obtained.

[0097] In step 106 of some embodiments, when the extraction time of the image frame is the same as the sampling time of the audio sampling point, the image frame and the spectrogram are synthesized to obtain a target test pattern. For example, the image frame of the first second and the spectrogram of the first second are synthesized to obtain the target test pattern of the first second, the image frame of the second second and the spectrogram of the second second are synthesized to obtain the target test pattern of the second second, the image frame of the third second and the spectrogram of the third second are synthesized to obtain the target test pattern of the third second, the image frame of the fourth second and the spectrogram of the fourth second are synthesized to obtain the target test pattern of the fourth second, and so on.

[0098] In one embodiment, referring to Figure 2 , step 106 may include:

[0099] Step 201, scaling the image frame according to a preset target scale, and scaling the spectrogram according to the target scale, so that the scale of the image frame is the same as the scale of the spectrogram;

[0100] Step 202: When the extraction time of the image frame is the same as the sampling time of the audio sampling point, the spectrum graph is spliced ​​on the left side of the image frame to obtain a target test graph.

[0101] In step 201, assuming that the target scale is 320*240, the scale of the image frame is scaled to 320*240, and the scale of the spectrogram is scaled to 320*240.

[0102] In step 202, for example, the spectrogram of the first second is spliced ​​to the left side of the image frame of the first second to obtain a target test image of the first second with a scale of 640*240; the spectrogram of the second second is spliced ​​to the left side of the image frame of the second second to obtain a target test image of the second second with a scale of 640*240; the spectrogram of the third second is spliced ​​to the left side of the image frame of the third second to obtain a target test image of the third second with a scale of 640*240; the spectrogram of the fourth second is spliced ​​to the left side of the image frame of the fourth second to obtain a target test image of the fourth second with a scale of 640*240; and so on.

[0103] The benefit of the above-mentioned embodiment of steps 201 to 204 is that the scale of the target test image can be standardized, which facilitates the subsequent determination of the jam detection result based on the difference between the target test images, thereby helping to improve the efficiency of jam detection.

[0104] In step 107 of some embodiments, a jam detection result of the target audio and video is determined according to the difference between any two target test images. The jam detection result is used to characterize whether the target audio and video has jams or not.

[0105] In one embodiment, referring to Figure 3 , step 107 may include:

[0106] Step 301, obtaining a target time of each target test chart; wherein the target time, extraction time and sampling time of the same target test chart are the same;

[0107] Step 302, extracting a first target test graph and a second target test graph from each target test graph according to the target time; wherein the target time of the second target test graph is after the target time of the first target test graph;

[0108] Step 303, performing image difference calculation based on the first target test image and the second target test image to obtain a first target difference image;

[0109] Step 304, performing pixel summation according to each pixel point in the first target difference map to obtain a first test pixel sum;

[0110] Step 305, determining a freeze detection result based on the first test pixel and.

[0111] In step 301, in combination with the above example, the target test graph includes the target test graph at the 1st second, the target test graph at the 2nd second, the target test graph at the 3rd second and the target test graph at the 4th second, so the target times are the 1st second, the 2nd second, the 3rd second and the 4th second respectively.

[0112] In step 302, for example, if the first target test image is a target test image with a target time of the first second, then the second target test image is a target test image with a target time of the second. For another example, if the first target test image is a target test image with a target time of the second, then the second target test image is a target test image with a target time of the third second.

[0113] In step 303, specifically, a pixel difference is calculated between each first pixel point in the first target test image and a second pixel point corresponding to the first pixel point in the second target test image to obtain a first target difference image. The second pixel point corresponding to the first pixel point specifically refers to a second pixel point at the same position as the first pixel point.

[0114] In step 304, all pixels in the first target difference image are summed to obtain a first test pixel sum. For example, if the first target difference image has 3 pixels, and the pixels of the pixels are 0.1 pixel, 0.1 pixel, and 0.2 pixel in sequence, then the first test pixel sum is 0.4 pixel.

[0115] Step 305, determine the jam detection result according to the first test pixel sum. For example, if the first test pixel sum is less than or equal to a preset threshold value (such as 0 pixels or 0.3 pixels), a jam detection result is generated to characterize the presence of jams in the target audio and video; if the first test pixel sum is greater than the preset threshold value (such as 0 pixels or 0.3 pixels), a jam detection result is generated to characterize the absence of jams in the target audio and video. For another example, a preset pixel jam mapping table is searched according to the first test pixel sum to obtain a jam detection result. Specifically, first, call the pixel jam mapping table, wherein the pixel jam mapping table is used to indicate the jam detection result corresponding to each pixel interval. Next, compare the first test pixel sum with each pixel interval of the pixel jam mapping table, and use the detection result corresponding to the pixel interval where the first test pixel sum is located as the jam detection result.

[0116] The benefit of the above-mentioned embodiment of step 301 to step 305 is that the jamming detection result can be determined based on the difference between two adjacent target test images, and the operation is simple and flexible.

[0117] In one embodiment, referring to Figure 4 , step 107 may include:

[0118] Step 401, obtaining a target time of each target test chart; wherein the target time, extraction time and sampling time of the same target test chart are the same;

[0119] Step 402, extracting a first target test graph and a second target test graph from each target test graph according to the target time; wherein the target time of the second target test graph is after the target time of the first target test graph;

[0120] Step 403, performing image difference calculation based on the first target test image and the second target test image to obtain a first target difference image;

[0121] Step 404, performing pixel summation according to each pixel point in the first target difference map to obtain a first test pixel sum;

[0122] Step 405, extracting a third target test pattern from each target test pattern according to the target time corresponding to the second target test pattern; wherein the target time of the third target test pattern is after the target time of the second target test pattern;

[0123] Step 406, performing image difference calculation based on the first target test image and the third target test image to obtain a second target difference image;

[0124] Step 407, performing pixel summation according to each pixel point in the second target difference map to obtain a second test pixel sum;

[0125] Step 408, performing image difference calculation based on the second target test image and the third target test image to obtain a third target difference image;

[0126] Step 409, performing pixel summation according to each pixel point in the third target difference map to obtain a third test pixel sum;

[0127] Step 410, performing pixel jamming detection based on the first test pixel sum, the second test pixel sum, and the third test pixel sum to obtain a jamming detection result.

[0128] Steps 401 to 404 are substantially the same as the above-mentioned steps 301 to 304 and will not be described in detail here.

[0129] In step 405, for example, if the second target test pattern is a target test pattern with a target time of the second second, then the third target test pattern is a target test pattern with a target time of the third second. For another example, if the second target test pattern is a target test pattern with a target time of the third second, then the third target test pattern is a target test pattern with a target time of the fourth second.

[0130] In step 406, specifically, a pixel difference is calculated between each first pixel point in the first target test image and a third pixel point corresponding to the first pixel point in the third target test image to obtain a second target difference image. The third pixel point corresponding to the first pixel point specifically refers to a third pixel point at the same position as the first pixel point.

[0131] In step 407, all pixels in the second target difference map are summed to obtain a second test pixel sum. For example, if the second target difference map has 3 pixels and the pixels are 0.1 pixel, 0.1 pixel, and 0.1 pixel in sequence, the second test pixel sum is 0.3 pixel.

[0132] In step 408, specifically, a pixel difference is calculated between each second pixel point in the second target test image and a third pixel point corresponding to the second pixel point in the third target test image to obtain a third target difference image. The third pixel point corresponding to the second pixel point specifically refers to a third pixel point at the same position as the second pixel point.

[0133] In step 409, all pixels in the third target difference map are summed to obtain a third test pixel sum. For example, if the third target difference map has 3 pixels, and the pixels of the pixels are 0.05 pixels, 0.05 pixels, and 0.1 pixels respectively, the third test pixel sum is 0.2 pixels.

[0134] Step 410, pixel jamming detection is performed according to the first test pixel sum, the second test pixel sum, and the third test pixel sum to obtain a jamming detection result. For example, if the first test pixel sum, the second test pixel sum, and the third test pixel sum are all less than or equal to a preset threshold value (such as 0 or 0.5), a jamming detection result is generated to characterize that there is jamming in the target audio and video; if the first test pixel sum, the second test pixel sum, and the third test pixel sum are all greater than a preset threshold value (such as 0 or 0.5), a jamming detection result is generated to characterize that there is no jamming in the target audio and video.

[0135] In one embodiment, referring to Figure 5 , step 410 may include:

[0136] Step 501, construct a jam graph quantity mark, the jam graph quantity mark is 0;

[0137] Step 502, taking the first test pixel sum, the second test pixel sum and the third test pixel sum in turn as the target test pixel sum;

[0138] Step 503, if the target test pixel sum is less than or equal to the preset pixel sum threshold, the number of jamming images is marked with 1;

[0139] Step 504, perform a quantity jam detection according to the quantity mark of the jam graph to obtain a jam detection result.

[0140] In one example, if the number of stuttering graphs is greater than a preset number threshold, a stuttering detection result is generated to characterize the presence of stuttering in the target audio or video; if the number of stuttering graphs is less than or equal to the preset number threshold, a stuttering detection result is generated to characterize the absence of stuttering in the target audio or video.

[0141] The benefit of the above-mentioned embodiment of steps 501 to 504 is that the jamming detection result can be dynamically determined according to the jamming graph quantity mark, with high accuracy and flexibility.

[0142] In one example, assume that there are 5 target test images, denoted as f1, f2, f3, f4 and f5. Then calculate the 5-second image difference, including: f 12 =abs(f1–f2),f 13 =abs(f1–f3),f 14 =abs(f1–f4),f 15=abs(f1-f5),f 23 =abs(f2–f3),f 24 =abs(f2–f4),f 25 =abs(f2–f5),f 34 =abs(f3–f4),f 35 =abs(f3–f5),f 45 = abs(f4–f5), a total of 4+3+2+1=10 target difference maps are obtained, and the pixel sum of each target difference map is counted. Judgment condition 1: If the pixel sum of each target difference map is equal to 0, it is considered to be stuck. Judgment condition 2: If it is less than the pixel sum threshold (for example, 50), the number of target difference maps that meet the conditions is N. If the proportion of N to the total number of target difference maps is greater than the preset proportion (for example, 0.5), it is considered to be stuck and further confirmation is required; if the proportion of N to the total number of target difference maps is less than or equal to the preset proportion.

[0143] In one embodiment, after step 107, the audio and video freeze detection method provided by this embodiment may further include:

[0144] If the jam detection result of the target audio and video indicates that there is no jam in the target audio and video, performing face detection on the first target test image to obtain first face area position information;

[0145] Performing region extraction on the first target difference image according to the first face region position information to obtain a first difference image face region;

[0146] Perform pixel summation on each pixel point in the face area of ​​the first difference image to obtain a pixel sum of the first face area;

[0147] Update the freeze detection result of the target audio and video according to the pixel sum of the first face area.

[0148] The benefit of the above embodiment is that the accuracy of the freeze detection is further improved by utilizing the face area in the target test image to perform freeze detection.

[0149] Specifically, the first target test image can be used for face detection through the existing face detection model. For example, face detection Retinaface is performed on the first target test image to obtain the face frame and the coordinates of the five landmark points P1, P2, P3, P4, and P5, which represent the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth respectively; the eye area is calculated: the centers of the left and right eyes are P1 and P2 respectively, and the side length L is calculated as (P2.x–P1.x) / 3, so the left eye area R_lef=cvRect(P1.xL / 2,P1.yL / 2,L,L), the left eye area Domain R_right = cvRect(P2.xL / 2,P2.yL / 2,L,L); finally calculate the mouth area: the side length L1 = P5.x–P4.x, the area is R_mouth = cvRect(P4.x-L1 / 2,P4.y-L1 / 2,L1,L1); in the first target difference map, the first difference image face area includes the pixels of these three areas, and the pixels of the three areas of the first difference image face area are counted to obtain the pixel sum of the first face area.

[0150] The above-mentioned step of updating the jam detection result of the target audio and video according to the first face area pixel sum may include: if the first face area pixel sum is less than or equal to a preset threshold value (such as 0 pixels or 0.3 pixels), then updating the jam detection result of the target audio and video to characterize the existence of jams in the target audio and video; if the first face area pixel sum is greater than a preset threshold value (such as 0 pixels or 0.3 pixels), then updating the jam detection result of the target audio and video to characterize the absence of jams in the target audio and video.

[0151] The above-mentioned step of updating the jam detection result of the target audio and video according to the first face area pixel sum may also include: calling a pixel jam mapping table, wherein the pixel jam mapping table is used to indicate the jam detection result corresponding to each pixel interval. Then, the first face area pixel sum is compared with each pixel interval of the pixel jam mapping table, and the jam detection result of the target audio and video is updated to the detection result corresponding to the pixel interval where the first face area pixel sum is located.

[0152] In one embodiment, in addition to performing face detection on the first target test image, face detection can also be performed on the second target test image to obtain the second face area position information, and face detection can also be performed on the third target test image to obtain the third face area position information, etc., so as to update the jam detection result based on the face area pixels of each difference image. In this way, the accuracy of the jam detection result update can be further improved.

[0153] In one example, combined with the above example, there are a total of 10 target difference maps, then after performing face area extraction, 10 difference map face areas can be obtained. Judgment condition one: If the sum of the pixels of each difference map face area is equal to 0, it is considered to be stuck. Judgment condition two: If it is less than the pixel sum threshold (such as 50), the number of target difference maps that meet the conditions is N. If the proportion of N in the total number of target difference maps is greater than the preset proportion (such as 0.5), it is considered to be stuck and further confirmation is needed; if the proportion of N in the total number of target difference maps is less than or equal to the preset proportion.

[0154] In summary, the present application can achieve the following technical effects: accurately detect the integrity of audio and video to ensure that dual-recorded videos can be traced back; fast execution speed, less time consumption, timely response, and convenient business completion; reduce manual quality inspection, further reduce costs and increase efficiency, and enhance value.

[0155] See also Figure 6 The embodiment of the present application also provides an audio and video freeze detection device, which can implement the above-mentioned audio and video freeze detection method. Figure 6 This is a module structure block diagram of the audio and video freeze detection device provided in an embodiment of the present application, and the device includes: a data acquisition module 601, a data separation module 602, an image extraction module 603, an audio sampling module 604, a spectrum conversion module 605, an image synthesis module 606 and a result determination module 607. Among them, the data acquisition module 601 is used to acquire the target audio and video; the data separation module 602 is used to separate the audio and video of the target audio and video to obtain the target video and the target audio; the image extraction module 603 is used to extract the frames of the target video according to a preset period to obtain the image frames; the audio sampling module 604 is used to perform audio sampling on the target audio according to a preset period to obtain the audio sampling points; the spectrum conversion module 605 is used to perform spectrum conversion on the audio sampling points to obtain the spectrum diagram; the image synthesis module 606 is used to perform image synthesis on the image frame and the spectrum diagram when the extraction time of the image frame is the same as the sampling time of the audio sampling point to obtain the target test diagram; the result determination module 607 is used to determine the jam detection result of the target audio and video according to the difference between any two target test diagrams; wherein the jam detection result is used to characterize the presence or absence of jam in the target audio and video.

[0156] In one embodiment, after performing audio and video separation on the target audio and video to obtain the target video and target audio, the audio and video freeze detection device also includes a prompt module, which is used to: obtain the duration of the target video to obtain the target video duration; obtain the duration of the target audio to obtain the target audio duration; calculate the difference between the target video duration and the target audio duration to obtain the absolute duration difference; if the absolute duration difference is greater than a preset duration difference threshold, generate a target prompt information; wherein the target prompt information is used to prompt the re-recording of the target audio and video.

[0157] In one embodiment, after determining the freeze detection result of the target audio and video based on the difference between any two target test images, the audio and video freeze detection device also includes a result update module, which is used to: if the freeze detection result of the target audio and video indicates that there is no freeze in the target audio and video, perform face detection on the first target test image to obtain first face area position information; perform area extraction on the first target difference image based on the first face area position information to obtain the first difference image face area; perform pixel summation based on each pixel point in the face area of ​​the first difference image to obtain the first face area pixel sum; and update the freeze detection result of the target audio and video based on the first face area pixel sum.

[0158] It should be noted that the specific implementation of the audio and video freeze detection device is basically the same as the specific implementation example of the above-mentioned audio and video freeze detection method, and will not be repeated here.

[0159] The embodiment of the present application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned audio and video freeze detection method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0160] See also Figure 7 , Figure 7 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0161] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0162] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the audio and video freeze detection method of the embodiment of this application;

[0163] Input / output interface 703, used to implement information input and output;

[0164] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0165] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0166] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0167] An embodiment of the present application also provides a computer-readable storage medium for computer-readable storage, the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned audio and video freeze detection method.

[0168] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0169] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0170] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0171] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0173] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0174] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0175] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0178] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0179] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A method for detecting audio and video freeze, characterized in that: The method comprises: Get the target audio and video; Separating the target audio and video to obtain the target video and the target audio; Extract frames from the target video according to a preset period to obtain image frames; Performing audio sampling on the target audio according to the preset period to obtain audio sampling points; Performing spectrum conversion on the audio sampling points to obtain a spectrum graph; When the extraction time of the image frame is the same as the sampling time of the audio sampling point, performing image synthesis on the image frame and the spectrum diagram to obtain a target test diagram; According to the difference between any two of the target test images, a jam detection result of the target audio and video is determined; wherein the jam detection result is used to characterize whether there is jam or not in the target audio and video.

2. The method according to claim 1, characterized in that The step of determining the jam detection result of the target audio and video according to the difference between any two of the target test images includes: Obtaining a target time of each target test chart; wherein the target time, extraction time and sampling time of the same target test chart are the same; Extracting a first target test graph and a second target test graph from each of the target test graphs according to the target time; wherein the target time of the second target test graph is after the target time of the first target test graph; Performing image difference calculation according to the first target test image and the second target test image to obtain a first target difference image; Perform pixel summation according to each pixel point in the first target difference map to obtain a first test pixel sum; The jamming detection result is determined based on the first test pixel sum.

3. The method according to claim 2, characterized in that The determining the jam detection result of the target audio and video according to the difference between any two of the target test images also includes: Extracting a third target test pattern from each of the target test patterns according to the target time corresponding to the second target test pattern; wherein the target time of the third target test pattern is after the target time of the second target test pattern; Performing image difference calculation based on the first target test image and the third target test image to obtain a second target difference image; Perform pixel summation according to each pixel point in the second target difference map to obtain a second test pixel sum; Performing image difference calculation according to the second target test image and the third target test image to obtain a third target difference image; Perform pixel summation according to each pixel point in the third target difference map to obtain a third test pixel sum; Pixel stuttering detection is performed according to the first test pixel sum, the second test pixel sum, and the third test pixel sum to obtain the stuttering detection result.

4. The method according to claim 3, characterized in that The performing pixel jamming detection according to the first test pixel sum, the second test pixel sum, and the third test pixel sum to obtain the jamming detection result includes: Construct a jam graph quantity mark, wherein the jam graph quantity mark is 0; Taking the first test pixel sum, the second test pixel sum and the third test pixel sum as target test pixel sum in turn; If the target test pixel sum is less than or equal to the preset pixel sum threshold, the number mark of the jam image is increased by 1; A quantity jam detection is performed according to the jam graph quantity mark to obtain the video jam detection result.

5. The method according to any one of claims 1 to 4, characterized in that: When the extraction time of the image frame is the same as the sampling time of the audio sampling point, the image frame and the spectrum diagram are synthesized to obtain a target test diagram, including: Scaling the image frame according to a preset target scale, and scaling the spectrogram according to the target scale, so that the scale of the image frame is the same as the scale of the spectrogram; When the extraction time of the image frame is the same as the sampling time of the audio sampling point, the spectrum diagram is spliced ​​on the left side of the image frame to obtain the target test diagram.

6. The method according to any one of claims 2 to 4, characterized in that: After determining the jam detection result of the target audio and video according to the difference between any two of the target test images, the method further includes: If the jam detection result of the target audio and video indicates that there is no jam in the target audio and video, performing face detection on the first target test image to obtain first face area position information; Performing region extraction on the first target difference image according to the first face region position information to obtain a first difference image face region; Perform pixel summation on each pixel point in the face area of ​​the first difference image to obtain a pixel sum of the first face area; According to the first face area pixel sum, update the jam detection result of the target audio and video.

7. The method according to any one of claims 1 to 4, characterized in that: After the target audio and video are separated into audio and video to obtain the target video and the target audio, the method further includes: Obtaining the duration of the target video to obtain the target video duration; Obtaining the duration of the target audio to obtain the target audio duration; Calculate the difference between the target video duration and the target audio duration to obtain an absolute duration difference; If the absolute duration difference is greater than a preset duration difference threshold, target prompt information is generated; wherein the target prompt information is used to prompt the re-recording of the target audio and video.

8. An audio and video freeze detection device, characterized in that: The device comprises: A data acquisition module, used to acquire target audio and video; A data separation module, used to separate the target audio and video to obtain the target video and the target audio; An image extraction module, used to extract frames from the target video according to a preset period to obtain image frames; An audio sampling module, used to perform audio sampling on the target audio according to the preset period to obtain audio sampling points; A spectrum conversion module, used for performing spectrum conversion on the audio sampling points to obtain a spectrum graph; An image synthesis module, used for performing image synthesis on the image frame and the spectrum diagram to obtain a target test diagram when the extraction time of the image frame is the same as the sampling time of the audio sampling point; The result determination module is used to determine the jam detection result of the target audio and video according to the difference between any two of the target test images; wherein the jam detection result is used to characterize whether the target audio and video has jam or not.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images