Audio and video detection method and device, equipment and medium

Abstract

The invention relates to the technical field of financial science and technology, and discloses an audio and video detection method and device, computer equipment and a storage medium. According to the scheme, firstly, video information of a user is acquired; obtaining a first timestamp when the user dictates the voice information from the audio stream; obtaining a second timestamp of the picture information when the user dictates from the video stream; and when the time recorded by the first timestamp and the time recorded by the second timestamp coincide or the error is smaller than a preset value, judging that the voice information is the user's own spoken recorded by the picture information. Therefore, according to the audio and video content recorded in real time, whether the user answers can be judged, the problem that in a traditional voiceprint recognition scheme, the user needs to read a text in advance is solved, and meanwhile, due to the fact that judgment is conducted in a combined mode through audio and video, interference of environmental noise is less. And through comparison of audio and video contents, whether a breaking problem occurs or not can be judged in an auxiliary manner. The method is higher in efficiency and better in stability.

Description

Technical Field

[0001] The present invention relates to the field of financial technology, and in particular to an audio and video detection method, device, equipment and medium Background Art

[0002] Double recording, i.e. audio and video recording, is mainly used to record the process of users handling business, especially the process of risk warning, to regulate the sales behavior of the financial industry, and also to provide a basis for future disputes. Double recording is an indispensable part of the process of selling products or services in the financial industry, such as banking, securities and insurance.

[0003] In the double recording process, in order to prevent others from "answering on behalf of others", the existing technology adopts voiceprint recognition solutions to ensure that the customer answers in person. The specific means are: before the double recording officially starts, the user needs to complete a reading containing specific keywords. The background system extracts the user's voiceprint information based on the keywords read by the user. After the double recording officially starts, the recorded sound is compared with the voiceprint information. When the comparison results are consistent, it is determined that the double recording link is answered by the user himself.

[0004] Then, the above verification process in the prior art requires the user to read the keyword text in advance, which wastes the user's time. At the same time, the voiceprint recognition solution based on pure audio is also easily interfered by background noise.

[0005] Therefore, the above problems existing in the prior art have yet to be solved. Summary of the invention

[0006] The present invention provides an audio and video detection method, device, equipment and medium to solve the technical problem that the double recording verification process is complicated and easily interfered by noise.

[0007] In a first aspect, an audio and video detection method is provided, comprising:

[0008] Acquire video information of the user, the video information including an audio stream and a video stream, wherein the audio stream includes voice information dictated by the user; the video stream includes picture information when the user dictates the voice information;

[0009] Acquire a first timestamp when the user dictates the voice information from the audio stream;

[0010] Acquire a second timestamp of the picture information when the user dictates from the video stream;

[0011] When the time recorded by the first timestamp coincides with the time recorded by the second timestamp or the error is less than a preset value, it is determined that the voice information is dictated by the user recorded in the picture information.

[0012] In a second aspect, an audio and video detection device is provided, comprising:

[0013] A receiving module, the receiving module is used to obtain the video information of the user, the video information includes an audio stream and a video stream, wherein the audio stream includes the voice information spoken by the user; the video stream includes the picture information when the user orally speaks the voice information;

[0014] A first processing module, the first processing module is used to obtain a first timestamp when the user orally dictates the voice information from the audio stream;

[0015] A second processing module, the second processing module is used to obtain a second timestamp of the picture information when the user speaks from the video stream;

[0016] The third processing module determines that the voice information is spoken by the user recorded in the screen information when the first timestamp obtained by the first processing module coincides with the time recorded by the second timestamp obtained by the second processing module or the error is less than a preset value.

[0017] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned audio and video detection method when executing the computer program.

[0018] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned audio and video detection method are implemented.

[0019] In the scheme implemented by the above-mentioned audio and video detection method, device, computer equipment and storage medium, the user's video information is first obtained, wherein the audio stream includes the voice information dictated by the user; the video stream includes the picture information when the user dictates the voice information; the first timestamp when the user dictates the voice information is obtained from the audio stream; the second timestamp of the picture information when the user dictates is obtained from the video stream; when the time recorded by the first timestamp and the second timestamp coincides or the error is less than a preset value, it is determined that the voice information is dictated by the user himself as recorded in the picture information. In this way, according to the real-time recorded audio and video content, it can be determined whether the answer is from the user himself, which overcomes the problem of the traditional voiceprint recognition scheme that the user needs to read a text in advance. At the same time, because it is a combination of audio and video, it is less affected by environmental noise. By comparing the audio and video content, it can also assist in determining whether a card disconnection problem occurs. The method is more efficient and more stable. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0021] Figure 1 is a schematic diagram of an application environment of an audio and video detection method in one embodiment of the present invention;

[0022] Figure 2 is a flow chart of an audio and video detection method in one embodiment of the present invention;

[0023] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S20;

[0024] Figure 4 yes Figure 3 A schematic flow chart of a specific implementation of step S21;

[0025] Figure 5 yes Figure 2 A schematic flow chart of a specific implementation of step S30;

[0026] Figure 6 Another specific implementation flow diagram of step S30

[0027] Figure 7 is a flow chart of an audio and video detection method in another embodiment of the present invention;

[0028] Figure 8 is a structural schematic diagram of an audio and video detection device in one embodiment of the present invention;

[0029] Fig. 9 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0030] Fig.10 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The audio and video detection method, device, equipment and medium provided by the embodiments of the present invention can be applied in the following aspects: Figure 1 In an application environment, the client communicates with the server through a network. The server can record audio and video of the client's dual recording process through the client. The audio and video detection method provided in the embodiment of the present application can be executed by the client and the result can be sent to the server. It can also be that after the client uploads the audio and video to the server, the server executes the audio and video detection method provided in the embodiment of the present application. When executing the audio and video detection method, the first timestamp when the user starts to dictate the voice information in the audio stream and the second timestamp when the user starts to dictate the voice information in the video stream are detected respectively. If the first timestamp coincides with the second timestamp, it is determined that the voice information is dictated by the user recorded by the picture information. In this way, the process of forcing each user to read a text aloud is avoided, and the service experience is optimized. Secondly, the limitations of the pure voiceprint recognition solution are overcome to ensure that in various scenarios, it can be accurately judged whether the speaker himself has completed the answer. At the same time, judging the time of occurrence of sound and facial expressions helps to solve the judgment of whether the video has a stuck phenomenon. Further, it helps to accurately detect and identify the speaker's content and who said it.

[0033] The client is generally a terminal device specially used for dual recording in a bank or other financial institution, which may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0034] See also Figure 2 As shown, Figure 2 A flowchart of an audio and video detection method provided by an embodiment of the present invention includes the following steps:

[0035] S10: Obtaining the user's video information.

[0036] In this embodiment, the video information includes an audio stream and a video stream, wherein the audio stream includes the voice information dictated by the user; the video stream includes the picture information when the user dictates the voice information. For example, according to insurance regulatory requirements, the insurance application process needs to be double-recorded, and the agent and the user (policyholder or insured) are required to appear on camera at the same time. At the same time, during the double-recording process, the insurance salesperson needs to read out the insurance terms, and the key terms need to be answered affirmatively by the user. The content recorded in this process is the above-mentioned video information, wherein the audio stream records the voice information of the user's answer, and the video stream records the picture information during the user's answer.

[0037] Furthermore, the recorded video information needs to be separated into an audio stream and a video stream. Preferably, lossless audio streams and video streams can be separated from the video information by using existing audio and video tools, thereby increasing the quality of the audio stream and the video stream.

[0038] It should be noted that the method provided in the embodiment of the present application detects the audio stream and the video stream respectively to determine whether the facial activity of the user speaking is consistent with the time of the voice emitted, so as to determine whether the user himself answers in person. The processing of the audio stream and the video stream can be performed synchronously or asynchronously, which is not limited in the embodiment of the present application. As a preferred embodiment, the embodiment of the present application is described by taking the example of first detecting the audio stream and then detecting the video stream, but it does not constitute a limitation on the scheme of the present application.

[0039] S20: Acquire the first timestamp when the user dictates the voice information from the audio stream.

[0040] In this embodiment, it is necessary to confirm the time when the user dictates the voice information from the audio stream. For example, during the dual recording process, after the account manager reads out the terms, the user says "informed and agree" at the 12th second of the audio. Then the first timestamp records that the user uttered the voice information at 12 seconds, and the voice information lasts for two seconds.

[0041] It should be noted that in order to determine the user's voice information from the audio stream, a series of processing is required on the audio stream, such as Figure 3 The detailed description is as follows.

[0042] S21: Determine the time when the user starts speaking in the voice information according to the voice recognition result.

[0043] In this embodiment, it is necessary to use voice recognition technology to determine which sounds are voice information spoken by the user. The specific implementation method of the voice recognition technology is not limited in this embodiment of the application. For ease of understanding, Figure 4 As shown, the preferred implementation is provided as follows.

[0044] S211: Detect human voice from the voice information.

[0045] In this embodiment, an audio quality control model, i.e., a classification model based on YOLO, is used to distinguish different types of sounds such as TTS voice broadcast, noise, and human voice, so as to locate the human voice in the voice information. When the human voice is detected, the start time of the dictation can be determined.

[0046] Preferably, when a human voice is detected, first determine whether the content spoken by the human voice meets the requirements before proceeding to the next step. Otherwise, there is no need to execute. For example, when the account manager asks the user whether he agrees with the terms that have been read out, the user replies "I didn't hear the content clearly, I don't agree", then the content does not meet the relevant regulatory requirements. At this time, the system directly prompts that there is an abnormality in the dual recording, please re-do the relevant process, and there is no need to execute the subsequent steps. The specific steps are described as follows.

[0047] S212: Based on the detected segment containing human voice, a voiceprint extraction model based on the Pyannote framework is used to convert the audio stream into a corresponding mfcc feature matrix.

[0048] S213: Based on the feature matrix, use the ASR speech recognition model to convert the audio into text information.

[0049] In this embodiment, through the above two steps, the human voice information spoken by the user is converted into text information that can be processed by the computer.

[0050] S214: Send the text information to a text error correction model to obtain the text information after error correction.

[0051] In this embodiment, the text correction model can be a text correction model developed by the patentee, which is customized for a specific scenario, such as an insurance scenario text correction model, or a banking scenario text correction model. Through this text correction model, the word or punctuation errors in the text are accurately identified, and the correct suggested text content is given in a targeted manner, further improving the accuracy of speech recognition.

[0052] S215: Compare the corrected text information with the preset text information.

[0053] In this embodiment, the preset text information is the text information of the regulatory requirements pre-stored in the database. For example, according to the regulatory requirements, after the account manager has read out the relevant terms, the user must explicitly reply "I understand and agree to the terms" before the subsequent steps can be executed.

[0054] After completing the above steps, not only the time when the user dictated the voice information is determined, but also the content of the voice information is determined to comply with regulatory requirements. Only then can the subsequent steps be further executed, thereby saving computing power and improving the efficiency of dual recording. Figure 3 As shown, the subsequent steps are further described as follows.

[0055] S22: Record the time when the user starts speaking into the first timestamp.

[0056] In this embodiment, as a preferred solution, after determining that the user's oral content meets the regulatory requirements, the relevant timestamp is recorded, which can save computing power and improve efficiency. It is understandable that the step of recording the timestamp can also be performed simultaneously with the step of text comparison, and the execution order is not limited in this embodiment of the application.

[0057] Through the above steps, it is determined that the user's oral content meets the requirements, and the time when the user's oral voice information is in the audio stream is determined. At this time, the video stream can be further processed to determine whether the above voice information is spoken by the user himself to prevent the "replying on behalf of others" phenomenon. Figure 2 The specific instructions are as follows.

[0058] S30: Acquire a second timestamp of the picture information when the user dictates from the video stream.

[0059] In this embodiment, it is necessary to confirm the screen appearance time of the user's oral voice information from the video stream. For example, during the dual recording process, after the account manager reads out the terms, the user speaks the voice information at the 12th second in the video, and the second timestamp records that the user spoke the voice information at 12 seconds.

[0060] It should be noted that in order to determine which pictures in the video stream are pictures where the user is dictating voice information, a series of processing needs to be performed on the video stream, which is described in detail as follows.

[0061] First, since the dual recording process requires both the account manager and the user to appear in the recording, the user needs to be separated from the video stream as the object to be identified. Figure 5 The specific steps are as follows.

[0062] S311: Obtain user face information from the video stream.

[0063] In this embodiment, the video stream is sent to the high-performance face detection model ReTinaFace model, using mobileNet as the backbone. This face recognition method occupies less memory and computing power, has the advantages of lightweight and fast processing, can complete the face detection task in real time, and output the corresponding face frame information.

[0064] S312: Output a face detection frame, and align and correct the user's face within the face detection frame.

[0065] In this embodiment, the face detection frame only marks the face image of the face, and aligns and corrects the face image to facilitate the subsequent analysis steps. In this step, the recognized facial information may be multiple, for example, one is the facial information of the account manager, and the other is the facial information of the user.

[0066] S313: Compare the facial information with the preset information to determine the facial information of the user.

[0067] In this embodiment, the preset information may be a photo uploaded by the user when registering. By comparing, it is possible to determine which of the multiple facial information is the facial information of the user, and at the same time, it is also possible to determine whether the facial information is the facial information of the user himself by comparing.

[0068] Through the above steps, the detection and recognition of user facial information is realized. On the one hand, it is possible to identify which facial information is the facial information of a specified user from multiple facial information, and on the other hand, the facial information detected is the information of the user himself. Therefore, in the subsequent steps, only the face detection frame of the target user needs to be tracked and recognized, which greatly reduces the amount of calculation and resource usage.

[0069] like Figure 6 As shown, further perform the following method:

[0070] S31: Identify characteristic picture information of the user from the video stream.

[0071] In this embodiment, by executing the aforementioned steps, the face detection frame where the target user is located has been determined. When executing this step, it is only necessary to identify the characteristic screen of the facial information in the face detection frame. Further, the characteristic screen information is the screen information displayed when the user orally narrates the voice information. The embodiment of the present application does not limit which information is selected as the characteristic screen information, and a preferred embodiment is provided as follows (not shown in the figure).

[0072] S301: Obtaining user's expression information from the video stream;

[0073] S302: When the expression information is a speaking action, the expression information is determined as characteristic screen information.

[0074] In this embodiment, expression information is the most intuitive information for judging whether the user is speaking. The preferred recognition method is as follows:

[0075] According to the continuous face detection frame information, continuous face images are obtained from the original video frames, and the expression action recognition model is adopted. Based on the Encoder-Decoder model architecture, expression information is extracted from the face and arranged in a time series, and finally it is obtained whether the user has a speaking action.

[0076] Preferably, the facial expression recognition model is a training data model, and the user's facial expression information is input into the training data model, and the training data model is used to determine whether the user has a speaking action in the screen information. In this way, since the facial expression information is recognized and detected based on continuous screens, it can distinguish the user's normal mouth movement when opening the mouth and dazed, and accurately determine whether the target user has a continuous speaking action. And the facial expression information with a speaking action is determined as the characteristic screen information, that is, in the screen information, the user speaks.

[0077] S32: Determine the time when the user dictates the voice information in the video stream according to the time when the characteristic picture information appears in the video stream.

[0078] In this embodiment, the screen in which the user speaks has been determined by identifying the characteristic screen information, and thus the time when the user dictates the voice information can be determined.

[0079] S33: Record the appearance time of the characteristic screen information into the second timestamp.

[0080] In this embodiment, for example, if it is detected that the user has a speaking expression at the 12th second of the video, the second timestamp records: the characteristic screen information spoken by the user is recorded at the 12th second. The characteristic screen information lasts for two seconds.

[0081] After the above steps, the audio stream and video stream are processed respectively. Next, we only need to compare the two timestamps to determine whether it is the user's own dictation. Figure 2 As shown, the specific steps are as follows.

[0082] S40: When the time recorded by the first timestamp coincides with the time recorded by the second timestamp or the error is less than a preset value, it is determined that the voice information is dictated by the user recorded in the picture information.

[0083] In this embodiment, the first timestamp and the second timestamp respectively record the voice information and picture information when the user speaks. For example, the first timestamp records that the user said "informed, agreed" at the 12th second of the audio stream, and the duration of the voice information is two seconds. The second timestamp records the facial expression information screen of the user speaking at the 12th second of the video stream, and the duration of the facial expression information is two seconds. It can be determined that the sentence "informed, agreed" was spoken by the user himself. It can be seen that the method of this application can determine whether the current content is answered by the user himself by analyzing and comparing the audio stream and the video stream without the user recording the voice in advance, thereby improving the efficiency of dual recording and user experience. In addition, compared with the recognition scheme of pure voiceprint, this scheme is less affected by noise interference. In the traditional scheme, if the environmental noise affects the user's voice, it will interfere with the accuracy of voiceprint recognition. In the scheme of this application, as long as the appearance time of the voice in the audio stream matches the appearance time of the facial expression information of the speech in the video stream, it can be judged, and the anti-interference ability to noise is stronger.

[0084] Furthermore, the method provided by the present application can also help determine whether a network freeze occurs. For example, the method of the present application is executed by the server, and the server detects that the user says "informed, agreed" at the 12th second in the audio stream, and the voice message lasts for 2 seconds. The user's expression message of speaking is detected at the 12.3th second in the video stream, and it lasts for 2 seconds. At this time, it can be determined that the video transmission is frozen, with an error of 0.3 seconds. This helps accurately determine the phenomenon of network freeze.

[0085] At the same time, the method provided in this application can accurately detect and identify the speaker's content and who is speaking.

[0086] For ease of understanding, Figure 7 As shown, the preferred embodiment of the present application provides a complete process as follows:

[0087] S1: Get the user's video information.

[0088] S2: Separate the lossless audio stream from the video information through FFMPEG.

[0089] S3: Separate the human voice from the audio stream.

[0090] S4: Convert the human voice from audio to text information through a voiceprint extraction model.

[0091] S5: Input the text information into the error correction model for error correction.

[0092] S6: Compare the corrected text information with the preset text information to determine whether the text information passes the verification.

[0093] If it passes, the following steps are further performed, otherwise the above steps S1 to S6 are repeated until step S6 is verified to pass.

[0094] S7: Record the first timestamp of the time when the text information appears in the audio stream.

[0095] S8: Separate lossless video stream from video information through FFMPEG.

[0096] S9: Send the video stream to the face recognition model to obtain face frame information.

[0097] S10: Align and correct the face within the range of the face frame information.

[0098] S11: Determine the face frame information where the user is located by comparing the face in the face frame information with the background photo uploaded when the user registers.

[0099] S12: Track and identify the face frame information of the user to determine whether the user has speaking expression information.

[0100] S13: Recording a second timestamp when the user starts to speak.

[0101] S14: Compare the first timestamp and the second timestamp, and when the time when the user starts speaking is consistent with or close to the time when the speaking expression information appears, determine that the content in the current video information is dictated by the user himself.

[0102] The specific implementation methods and intended effects of the above steps S1 to S14 can be found in the above records and will not be repeated here.

[0103] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0104] In one embodiment, an audio and video detection device is provided, and the audio and video detection device corresponds to the audio and video detection method in the above embodiment. Figure 8 As shown, the audio and video detection device includes a receiving module, a first processing module, a second processing module and a third processing module. Each functional module is described in detail as follows:

[0105] The receiving module 101 is used to obtain the video information of the user, and the video information includes an audio stream and a video stream, wherein the audio stream includes the voice information spoken by the user; the video stream includes the picture information when the user speaks the voice information;

[0106] A first processing module 102, the first processing module 102 is used to obtain a first timestamp when the user dictates voice information from the audio stream;

[0107] The second processing module 103 is used to obtain a second timestamp of the screen information when the user speaks from the video stream;

[0108] The third processing module 104 determines that the voice information is spoken by the user recorded in the screen information when the time recorded by the first timestamp obtained by the first processing module 102 coincides with the time recorded by the second timestamp obtained by the second processing module 103 or the error is less than a preset value.

[0109] Preferably, the first processing module 102 is specifically used to: determine the time when the user starts speaking in the voice information according to the voice recognition result; and record the time when the user starts speaking in the first timestamp.

[0110] Preferably, the second processing module 103 is specifically used to: identify the user's characteristic picture information from the video stream; determine the time when the user's oral voice information appears in the video stream according to the time when the characteristic picture information appears in the video stream; and record the appearance time of the characteristic picture information in the second timestamp.

[0111] Preferably, the second processing module 103 is also used to: obtain the user's facial information from the video stream; output a face detection frame, and align and correct the user's face within the face detection frame; the second processing module 103 performs the identification of the user's characteristic picture information from the video stream, including: extracting characteristic picture information from the user's facial picture identified in the face detection frame.

[0112] Preferably, the second processing module 103 is further used to compare the user's facial image information with preset facial image information to determine whether the current user is the preset user.

[0113] Preferably, the second processing module 103 is further used for: acquiring the user's expression information from the video stream; and when the expression information is a speaking action, determining the expression information as characteristic picture information.

[0114] Preferably, the second processing module 103 is further used for: acquiring a training data model; inputting the expression information into the training data model; and when the result output by the training data model is a speaking action, determining the expression information as feature screen information.

[0115] The present invention provides an audio and video detection device, which first separates the audio stream and the video stream in the video information, then extracts the first timestamp when the user dictates the voice information from the audio stream, and obtains the second timestamp when the user dictates the picture information from the video stream. When the time recorded by the first timestamp coincides with the time recorded by the second timestamp or the error is less than a preset value, it is determined that the voice information is the dictation of the user recorded in the picture information. In this way, the problem of traditional voiceprint recognition solutions that require users to read a text in advance is overcome. At the same time, since the audio and video are combined for judgment, it is less affected by environmental noise. By comparing the audio and video content, it can also assist in determining whether a card disconnection problem occurs. It is more efficient and more stable.

[0116] For the specific definition of the audio and video detection device, please refer to the definition of the audio and video detection method above, which will not be repeated here. Each module in the above-mentioned audio and video detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0117] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of an intelligent question-answering processing method based on artificial intelligence.

[0118] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the client side of an intelligent question-answering processing method based on artificial intelligence

[0119] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0120] Acquire the video information of the user, the video information includes an audio stream and a video stream, wherein the audio stream includes the voice information spoken by the user; the video stream includes the picture information when the user speaks the voice information;

[0121] The first timestamp when the user's spoken voice information is obtained from the audio stream;

[0122] Obtaining a second timestamp of the screen information when the user is speaking from the video stream;

[0123] When the time recorded by the first timestamp and the second timestamp coincides or the error is less than a preset value, it is determined that the voice information is dictated by the user recorded in the picture information.

[0124] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0125] Acquire the video information of the user, the video information includes an audio stream and a video stream, wherein the audio stream includes the voice information spoken by the user; the video stream includes the picture information when the user speaks the voice information;

[0126] The first timestamp when the user's spoken voice information is obtained from the audio stream;

[0127] Obtaining a second timestamp of the screen information when the user is speaking from the video stream;

[0128] When the time recorded by the first timestamp and the second timestamp coincides or the error is less than a preset value, it is determined that the voice information is dictated by the user recorded in the picture information.

[0129] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0131] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0132] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An audio and video detection method, characterized in that: include: Acquire video information of the user, the video information including an audio stream and a video stream, wherein the audio stream includes voice information dictated by the user; the video stream includes picture information when the user dictates the voice information; Acquire a first timestamp when the user dictates the voice information from the audio stream; Acquire a second timestamp of the picture information when the user dictates from the video stream; When the time recorded by the first timestamp and the second timestamp coincides or the error is less than a preset value, it is determined that the voice information is dictated by the user recorded in the screen information.

2. The audio and video detection method according to claim 1, characterized in that: The obtaining, from the audio stream, a first timestamp when the user orally dictates the voice information comprises: Determine the time when the user starts speaking in the voice information according to the voice recognition result; The time when the user starts speaking is recorded in the first timestamp.

3. The audio and video detection method according to claim 1, characterized in that: The step of obtaining the second timestamp of the picture information when the user dictates from the video stream includes: Identifying characteristic picture information of a user from the video stream; Determining the time when the user dictates the voice information in the video stream according to the time when the characteristic picture information appears in the video stream; The appearance time of the characteristic screen information is recorded in the second timestamp.

4. The audio and video detection method according to claim 3, characterized in that: Before identifying the characteristic picture information of the user from the video stream, the method further includes: Acquire user's facial information from the video stream; Outputting a face detection frame, and aligning and correcting the user's face within the face detection frame; The identifying characteristic picture information of the user from the video stream includes: The characteristic picture information is extracted from the user's facial picture identified in the face detection frame.

5. The audio and video detection method according to claim 4, characterized in that: The output face detection frame, after aligning and correcting the user's face within the face detection frame, further comprises: The user's facial image information is compared with preset facial image information to determine whether the current user is the preset user.

6. The audio and video detection method according to claim 3, characterized in that: The step of identifying characteristic picture information of the user speaking from the video stream comprises: Acquire the user's facial expression information from the video stream; When the expression information is a speaking action, the expression information is determined as the characteristic screen information.

7. The audio and video detection method according to claim 6, characterized in that: When the expression information is a speaking action, determining the expression information as the characteristic screen information includes: Get the training data model; Inputting the expression information into the training data model; When the result output by the training data model is the speaking action, the expression information is determined as the characteristic screen information.

8. An audio and video detection device, characterized in that: include: A receiving module, the receiving module is used to obtain the video information of the user, the video information includes an audio stream and a video stream, wherein the audio stream includes the voice information spoken by the user; the video stream includes the picture information when the user orally speaks the voice information; A first processing module, the first processing module is used to obtain a first timestamp when the user orally dictates the voice information from the audio stream; A second processing module, the second processing module is used to obtain a second timestamp of the picture information when the user speaks from the video stream; The third processing module determines that the voice information is spoken by the user recorded in the screen information when the first timestamp obtained by the first processing module coincides with the time recorded by the second timestamp obtained by the second processing module or the error is less than a preset value.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the audio and video detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the audio and video detection method according to any one of claims 1 to 7 are implemented.

Images