Virtual digital human dialogue method and system and storage medium

Abstract

The invention relates to a virtual digital human dialogue method and system and a storage medium, and belongs to the technical field of artificial intelligence, the method is applied to a virtual digital human dialogue system, the virtual digital human dialogue system is in communication connection with a camera device and a microphone, and the method comprises the steps: obtaining an image shot by the camera device, and carrying out the face recognition of the image; if the face is recognized in the image, determining the proportion area of the face in the image; when the proportion area is greater than or equal to a preset human face proportion threshold value, starting a microphone to carry out audio signal acquisition; and determining the dialogue state of the virtual digital human according to the collected audio signal. According to the virtual digital human dialogue system, the on-state of the microphone is controlled based on the human face detection result, and the problem of user privacy leakage caused by long-time continuous on of the microphone in the awakening process can be avoided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a virtual digital human dialogue method, system and storage medium. Background Art

[0002] With the development of artificial intelligence technology, virtual digital human (VHDH), as a new form of human-computer interaction, has been widely used in multiple scenarios such as customer service and educational counseling.

[0003] Virtual digital human dialogue system usually refers to a device or system that integrates digital human technology. It can be used to simulate human behavior and interaction and is widely used in customer service, education and training, entertainment and other fields. Virtual digital human dialogue system combines artificial intelligence (AI), speech recognition, natural language processing (NLP), computer vision and virtual reality (VR) and other advanced technologies, enabling "digital human" to interact with users in a more realistic image.

[0004] Most existing virtual digital human dialogue systems rely on the user's active input of a wake-up word to wake them up before they can start a conversation. In order to ensure that the virtual digital human dialogue system can be awakened in time, the microphone needs to be kept on so that the virtual digital human dialogue system can recognize the wake-up word in real time and be awakened in time by the wake-up word. However, this awakening method is prone to leaking user privacy. Summary of the invention

[0005] In view of this, it is necessary to provide a virtual digital human dialogue method, system and storage medium to solve the problem that the traditional wake-up word awakening virtual digital human dialogue system is prone to user privacy leakage.

[0006] In order to solve the above problems, in a first aspect, the present invention provides a virtual digital human dialogue method, the method is applied to a virtual digital human dialogue system, the virtual digital human dialogue system is communicatively connected with a camera device and a microphone, the method comprises: Acquire an image captured by the camera device, and perform face recognition on the image; If it is recognized that the image contains a human face, determining the area occupied by the human face in the image; When the occupied area is greater than or equal to a preset face occupied area threshold, turning on the microphone to collect audio signals; The conversation state of the virtual digital human is determined according to the collected audio signal.

[0007] Optionally, the method further includes: When it is recognized that the image does not contain a human face, or when it is recognized that the image contains a human face but the area occupied by the human face in the image is less than the preset face ratio threshold, the microphone is kept in a turned off state.

[0008] Optionally, determining the conversation state of the virtual digital human according to the collected audio signal includes: Sampling the audio signal at a preset sampling rate to obtain a plurality of audio frames; Determine a target audio frame from the plurality of audio frames; Detecting the human voice confidence of the target audio frame, and acquiring current speaking state information; the current speaking state information includes a speaking state and a non-speaking state; If the human voice confidence of the target audio frame is greater than or equal to the first confidence threshold, and the current speech state information is a non-speech state, the number of speech audio frames is reset to zero and then increased by one, and the current speech state information is set to a speech state; If the human voice confidence of the target audio frame is greater than or equal to the first confidence threshold, and the current speech state information is a speech state, then the number of speech audio frames is increased by one; Returning to the step of determining a target audio frame from the plurality of audio frames until it is determined that the speech ends; Determining whether the number of speech audio frames is greater than a valid speech frame number threshold; If the number of speech audio frames is greater than the effective speech frame number threshold, generating a virtual digital human reply message according to the target audio frame; If the number of speech audio frames is less than or equal to the valid speech frame number threshold, no virtual digital human reply information is generated.

[0009] Optionally, the method further includes: If the voice confidence of the target audio frame is less than the first confidence threshold and greater than the second confidence threshold, and the current speech state information is the speech state, then the number of speech audio frames and the current speech state information are kept unchanged.

[0010] Optionally, the method further includes: If the human voice confidence of the target audio frame is less than or equal to the second confidence threshold, and the current speech state information is a speech state, the number of unspeaking audio frames is reset to zero and then increased by one, and the current speech state information is set to a unspeaking state; If the human voice confidence of the target audio frame is less than or equal to the second confidence threshold, and the current speech state information is a non-speech state, then the number of the non-speech audio frame is increased by one; If the human voice confidence of the target audio frame is less than the first confidence threshold and greater than the second confidence threshold, and the current speech state information is a non-speech state, then the number of the non-speech audio frames and the current speech state information are kept unchanged; The step of determining that the speech is finished includes: When the number of unspoken audio frames is greater than the unspoken frame number threshold, it is determined that the speech ends.

[0011] Optionally, each of the audio frames corresponds to a frame number; the method further includes: If the human voice confidence of the target audio frame is greater than or equal to the first confidence threshold, and the current speech state information is a non-speech state, the frame number of the target audio frame is used as the starting speech frame number; If the human voice confidence of the target audio frame is less than or equal to the second confidence threshold, and the current speech state information is a speech state, taking the frame number of the target audio frame as the end speech frame number; Generating virtual digital human reply information according to the target audio frame includes: The audio frames corresponding to the starting speech frame number to the ending speech frame number are parsed to generate virtual digital human reply information.

[0012] Optionally, the multiple audio frames are stored in a preset buffer, and the method further includes: When it is determined that the speech ends, the audio frames corresponding to the start speech frame number to the end speech frame number are deleted from the preset buffer.

[0013] Optionally, the method further includes: When it is determined that the speech ends, the number of the speech audio frames and the number of the non-speech audio frames are reset to zero.

[0014] In a second aspect, the present invention further provides a virtual digital human dialogue system, which is communicatively connected to a camera device and a microphone: the virtual digital human dialogue system is used to execute the steps in any one of the above-mentioned virtual digital human dialogue methods.

[0015] In a third aspect, the present invention further provides a computer-readable storage medium for storing a computer-readable program, wherein the program or instruction, when executed by a processor, can implement the steps in any one of the above-mentioned virtual digital human dialogue methods.

[0016] The beneficial effects of the present invention are: The virtual digital human dialogue system can obtain images taken by a camera device and perform face recognition on the images; if a face is recognized in the image, the area occupied by the face in the image is determined; when the area occupied is greater than or equal to a preset face ratio threshold, the microphone is turned on to collect audio signals; and the virtual digital human dialogue state is determined based on the collected audio signals. In the present invention, the virtual digital human dialogue system controls the microphone on state based on the face detection result, which can avoid the problem of user privacy leakage caused by the microphone being turned on for a long time during the wake-up process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of an embodiment of a virtual digital human dialogue method provided by the present invention; Figure 2 A schematic diagram of a microphone switch control process provided by the present invention; Figure 3 A schematic diagram of a sound recording thread provided by the present invention; Figure 4 A schematic diagram of a human voice detection thread provided by the present invention; Figure 5 A schematic diagram of the structure of an embodiment of the virtual digital human dialogue system provided by the present invention. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0019] In the description of the embodiments of the present invention, unless otherwise specified, "multiple" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone.

[0020] The descriptions of "first", "second", etc. involved in the embodiments of the present invention are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the technical features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0021] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0022] Reference Figure 1 , shows a flow chart of an embodiment of a virtual digital human dialogue method provided by the present invention, the method is applied to a virtual digital human dialogue system, the virtual digital human dialogue system is communicatively connected with a camera device and a microphone, and the method includes: S101, acquiring an image captured by a camera device and performing face recognition on the image.

[0023] The camera device can remain on, the microphone can remain off, and the virtual digital human dialogue system can obtain the images taken by the camera device in real time and perform face recognition on the images. The virtual digital human dialogue system can perform face recognition through models such as Yolo and OpenCV to obtain the face confidence FaceConfidence. FaceConfidence is a floating point type between 0 and 1. The larger the value, the higher the possibility of a face.

[0024] S102: If it is recognized that the image contains a face, determine the area occupied by the face in the image.

[0025] Furthermore, the virtual digital human dialogue system can compare FaceConfidence with a preset face confidence threshold FaceConfidenceThreshold. If FaceConfidence is greater than or equal to FaceConfidenceThreshold, it indicates that the image contains a face. If FaceConfidence is less than FaceConfidenceThreshold, it indicates that the image does not contain a face. Wherein, FaceConfidenceThreshold is a parameter set based on experience and can be set to 0.5.

[0026] When it is determined that an image contains a face, the area of ​​the face in the image can be calculated. The area of ​​the face can be the area of ​​the face AABB (Axis-Aligned Bounding Box) box. The area of ​​the face AABB box refers to the area of ​​a rectangular box used to represent the face area in computer vision and image processing. Then calculate the ratio of the area of ​​the face to the area of ​​the image, that is, the FaceProportion of the face in the image.

[0027] When it is determined that the image does not contain a human face, the microphone continues to be turned off.

[0028] S103: When the face area is greater than or equal to a preset face area threshold, the microphone is turned on to collect audio signals.

[0029] When it is determined that the image contains a face, the virtual digital human dialogue system can compare FaceProportion with a preset face ratio threshold FaceProportionThreshold, where FaceProportionThreshold is a parameter set based on experience and can be set to 0.2.

[0030] If FaceProportion is greater than or equal to FaceProportionThreshold, it indicates that the face is close enough to the camera device, and the microphone is turned on to collect audio signals.

[0031] If FaceProportion is less than FaceProportionThreshold, it means that the face is far away from the camera device. In this case, keep the microphone off.

[0032] S104, determining the conversation state of the virtual digital human according to the collected audio signal.

[0033] After the audio signal is collected, it can be determined based on the audio signal whether the user is having a conversation with the virtual digital person and whether the virtual digital person needs to respond to the conversation.

[0034] The virtual digital human dialogue system can obtain images taken by a camera device and perform face recognition on the images; if a face is recognized in the image, the area occupied by the face in the image is determined; when the area occupied is greater than or equal to a preset face ratio threshold, the microphone is turned on to collect audio signals; and the virtual digital human dialogue state is determined based on the collected audio signals. In the present invention, the virtual digital human dialogue system controls the microphone on state based on the face detection result, which can avoid the problem of user privacy leakage caused by the microphone being turned on for a long time during the wake-up process.

[0035] Reference Figure 2 , showing a schematic diagram of a microphone switch control process provided by the present invention. The virtual digital human dialogue system may include a face detection module FD module, in which the constants FaceConfidenceThreshold and FaceProportionThreshold set by the user can be stored, and the FD module can perform face recognition on the images taken by the camera device in real time, and determine whether to turn on the microphone according to the face recognition result.

[0036] In one embodiment, S104 specifically includes: sampling and processing the audio signal according to a preset sampling rate to obtain multiple audio frames; determining a target audio frame from the multiple audio frames; detecting the human voice confidence of the target audio frame, and obtaining the current speaking state information; the current speaking state information includes speaking state and non-speaking state; if the human voice confidence of the target audio frame is greater than or equal to the first confidence threshold, and the current speaking state information is the non-speaking state, then the number of speaking audio frames is reset to zero and then increased by one, and the current speaking state information is set to the speaking state; if the human voice confidence of the target audio frame is greater than or equal to the first confidence threshold, and the current speaking state information is the speaking state, then the number of speaking audio frames is increased by one; return to the step of determining a target audio frame from the multiple audio frames until the end of the speech is determined; determine whether the number of speaking audio frames is greater than the valid speaking frame number threshold; if the number of speaking audio frames is greater than the valid speaking frame number threshold, generate virtual digital human reply information according to the target audio frame; if the number of speaking audio frames is less than or equal to the valid speaking frame number threshold, do not generate virtual digital human reply information.

[0037] When the microphone is turned on, the virtual digital human dialogue system can start a thread to collect sound, and then sample the acquired audio signal at a sampling rate of 16kHz to obtain multiple audio frames. After that, each audio frame can be detected in turn to detect whether the audio frame contains human voice (if the human voice confidence is greater than or equal to the first confidence threshold, it is determined to contain human voice), and the number of speech audio frames is recorded according to the human voice detection result. If the number of speech audio frames is greater than the valid speech frame number threshold at the end of the speech, it indicates that the target audio frame contains valid voice information, and the virtual digital human dialogue system needs to parse the voice information contained in the multiple target audio frames determined in turn, and generate reply information based on the analysis results; if the number of speech audio frames is less than or equal to the valid speech frame number threshold at the end of the speech, it indicates that it may be a false trigger, and the virtual digital human dialogue system does not need to parse the multiple target audio frames, nor does it need to generate reply information.

[0038] The virtual digital human dialogue system can analyze the voice information contained in the target audio frame through models such as ASR (speech to text recognition) \LLM (large model conversation structure) \TTS (text to speech), and generate response information based on the analysis results.

[0039] The embodiment of the present invention records the number of speech audio frames and compares the number of speech audio frames with the effective speech frame number threshold after the speech is finished to determine whether the speech is a false trigger, thereby filtering out noise and improving the conversation accuracy of the virtual digital human conversation system, especially in scenes with poor sound reception conditions such as exhibitions and pavilions. In addition, compared with the traditional method of waking up with a wake-up word and then replying, the embodiment of the present invention does not require the user to remember the wake-up word, which reduces the user's memory burden, and can also achieve continuity of conversation with the user without the need for the user to repeatedly wake up.

[0040] In one embodiment, the virtual digital human dialogue method also includes: if the human voice confidence of the target audio frame is less than a first confidence threshold and greater than a second confidence threshold, and the current speech state information is a speech state, then the number of speech audio frames and the current speech state information are kept unchanged.

[0041] If the human voice confidence is less than the first confidence threshold and greater than the second confidence threshold, it cannot be accurately determined whether the human voice is contained. Therefore, no changes are made to the current speech state information and the number of speech audio frames. Instead, the next target audio frame is determined and human voice detection is performed on the next target audio frame.

[0042] In one embodiment, the virtual digital human dialogue method further includes: if the human voice confidence of the target audio frame is less than or equal to the second confidence threshold, and the current speech state information is the speech state, then the number of unspeaked audio frames is reset to zero and then increased by one, and the current speech state information is set to the unspeaked state; if the human voice confidence of the target audio frame is less than or equal to the second confidence threshold, and the current speech state information is the unspeaked state, then the number of unspeaked audio frames is increased by one; if the human voice confidence of the target audio frame is less than the first confidence threshold and greater than the second confidence threshold, and the current speech state information is the unspeaked state, then the number of unspeaked audio frames and the current speech state information are kept unchanged. Then, the step of determining the end of speech can specifically include: when the number of unspeaked audio frames is greater than the unspeaked frame number threshold, then determining the end of speech.

[0043] If the human voice confidence is less than or equal to the second confidence threshold, it is determined that no human voice is included. If the human voice confidence is less than the first confidence threshold and greater than the second confidence threshold, it cannot be accurately determined whether human voice is included, so no changes are made to the current speech state information and the number of unspeaking audio frames.

[0044] The embodiment of the present invention determines whether a speech has ended by comparing the number of unspoken audio frames with a threshold of the number of unspoken frames, thereby improving the recognition accuracy of a complete conversation by a virtual digital human conversation system.

[0045] In one embodiment, each audio frame corresponds to a frame number; the virtual digital human dialogue method further includes: if the human voice confidence of the target audio frame is greater than or equal to the first confidence threshold, and the current speech state information is a non-speech state, then the frame number of the target audio frame is used as the starting speech frame number; if the human voice confidence of the target audio frame is less than or equal to the second confidence threshold, and the current speech state information is a speech state, then the frame number of the target audio frame is used as the ending speech frame number. Then, the step of generating the virtual digital human reply information according to the target audio frame may specifically include: parsing the audio frames corresponding to the starting speech frame number to the ending speech frame number to generate the virtual digital human reply information.

[0046] In one embodiment, multiple audio frames are stored in a preset buffer, and the virtual digital human dialogue method further includes: when it is determined that the speech is finished, deleting the audio frames corresponding to the starting speech frame number to the ending speech frame number from the preset buffer.

[0047] In one embodiment, the virtual digital human dialogue method further includes: when it is determined that the speech is finished, resetting the number of spoken audio frames and the number of unspoken audio frames to zero.

[0048] In one embodiment, when it is determined that the speech has ended, the virtual digital human dialogue system can determine whether the currently detected target audio frame is the last audio frame in the preset buffer. If not, the virtual digital human dialogue system can continue to determine the next target audio frame and perform human voice detection on the next target audio frame.

[0049] In one embodiment, after detecting that the target audio frame contains human voice, the virtual digital human dialogue system can also perform voiceprint detection on the human voice to determine whether the user corresponding to the human voice in the target audio frame has the authority to wake up the virtual digital human, thereby effectively identifying the audio of the dialogue target under complex sound reception conditions such as in an exhibition hall to prevent false triggering.

[0050] Reference Figure 3, showing a schematic diagram of a sound recording thread provided by the present invention. The following constants can be pre-set in the virtual digital human dialogue system: sampling rate FrameSampleCount (the sampling rate can be 16000Hz), effective speaking frame number threshold SpeakingFrameCountLimit (effective speaking frame number threshold can be 10), first confidence threshold SpeakingConfidenceThreshold (the first confidence threshold can be 0.7), second confidence threshold NonSpeakingConfidenceThreshold (the second confidence threshold can be 0.3), non-speaking frame number threshold NonSpeakingFrameCountLimit (non-speaking frame number threshold can be 30). Also, before turning on the microphone, set the following variables as follows: set the initial value of the current speaking status information SpeakingStats to False, that is, set the current speaking status information to the non-speaking state, set the initial value of the non-speaking audio frame number NonSpeakingFrameCount to 0, set the initial value of the speaking audio frame number SpeakingFrameCount to 0, set the initial value of the speaking start frame index StartSpeakingFrameIndex (also called the starting speaking frame number) to 0, set the initial value of the speaking end frame index FinishSpeakingFrameIndex (also called the ending speaking frame number) to 0, and set the initial value of the current frame index I (the frame number of the target audio frame) to 0. When the microphone is turned on, the virtual digital human dialogue system starts a thread to receive sound, and samples the audio information at a sampling rate of 16kHz, and stores the sampled audio frames in the sample buffer SampleBuffer.

[0051] Reference Figure 4 , showing a schematic diagram of a human voice detection thread provided by the present invention. The virtual digital human dialogue system may include a human voice detector, and the human voice detector workflow includes: 1. Detect the audio frames in the sample buffer frame by frame starting from frame 0. The preset sample length of 1 frame is 1600, and go to step 2.

[0052] 2. Perform voice inference on the frame through the voice detection model. Here, the neural network will output the voice confidence SpeakingConfidence (Float type between 0 and 1). There is no restriction on the voice detection model used here. You can use open source models such as Silero vad and go to step 3.

[0053] 3. When the voice confidence is greater than or equal to SpeakingConfidenceThreshold, it is considered that this frame includes human voice. If SpeakingStats is in the non-speaking state at this time, the speaking start frame index StartSpeakingFrameIndex is recorded as the current frame number i, and the speaking frame number SpeakingFrameCount is set to accumulate 1 frame, and the speaking state is set to True.

[0054] 4. When the confidence is greater than NonSpeakingConfidenceThreshold but less than SpeakingConfidenceThreshold, no operation is performed on this frame and the process goes directly to step 7.

[0055] 5. When the voice confidence is lower than NonSpeakingConfidenceThreshold, it is considered that this frame does not include human voice. At this time, it is determined whether SpeakingStats is in a speaking state. If it is in a speaking state, the number of non-speaking frames NonSpeakingFrameCount is reset to 0, and the speech end frame index FinishSpeakingFrameIndex is recorded as the current frame index, the number of non-speaking frames NonSpeakingFrameCount is accumulated by 1, and SpeakingStats is set to False, and then go to step 6.

[0056] 6. Determine whether the speech has ended by judging whether NonSpeakingFrameCount has reached NonSpeakingFrameCountLimit. If not, go to step 7; if so, go to step 8.

[0057] 7. Determine whether the current frame is the last frame in the buffer. If not, process the next frame.

[0058] 8. At this point, it can be determined that the speech has ended by judging whether SpeakingFrameCount exceeds SpeakingFrameCountLimit. If it does not exceed, it may be a false trigger and go to step 10. If it exceeds, go to step 9.

[0059] 9. Take out the samples in the frames from StartSpeakingFrameIndex to FinishSpeakingFrameIndex from SampleBuffer and delete them from SampleBuffer, encode them into 16kHz Wav audio, trigger the digital human Q&A step through callback, and go to step 11.

[0060] 10. Reset the speaking start frame index StartSpeakingFrameIndex and FinishSpeakingFrameIndex, and go to step 7.

[0061] 11. Reset the spoken audio frame count SpeakingFrameCount and the non-spoken audio frame count NonSpeakingFrameCount to 0, and go to step 7.

[0062] In the embodiment of the present invention, by using face detection technology, if a face with a certain coverage area does not appear in the picture, the sound reception function of the microphone is disabled, thereby avoiding privacy issues caused by long-term sound reception.

[0063] The misrecognition rate can be reduced by configuring various preset parameters, such as the effective speech frame number threshold, the non-speech frame number threshold, etc.

[0064] Through the sample buffer of voice input and the multi-threaded frame-by-frame detection callback mechanism, direct conversation can trigger the conversation callback without manual wake-up.

[0065] Reference Figure 5 , showing a structural schematic diagram of an embodiment of the virtual digital human dialogue system provided by the present invention, the virtual digital human dialogue system 10 is communicatively connected with the camera device 20 and the microphone 30: the virtual digital human dialogue system 10 is used to execute the steps in any one of the above-mentioned virtual digital human dialogue methods.

[0066] In one embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by the processor, the steps of any one of the above-mentioned virtual digital human dialogue methods are implemented.

[0067] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0068] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A virtual digital human dialogue method, characterized in that: The method is applied to a virtual digital human dialogue system, wherein the virtual digital human dialogue system is communicatively connected with a camera device and a microphone, and the method comprises: Acquire an image captured by the camera device, and perform face recognition on the image; If it is recognized that the image contains a human face, determining the area occupied by the human face in the image; When the occupied area is greater than or equal to a preset face occupied area threshold, turning on the microphone to collect audio signals; The conversation state of the virtual digital human is determined according to the collected audio signal.

2. The virtual digital human dialogue method according to claim 1, characterized in that: The method further comprises: When it is recognized that the image does not contain a human face, or when it is recognized that the image contains a human face but the area occupied by the human face in the image is less than the preset face ratio threshold, the microphone is kept in a turned off state.

3. The virtual digital human dialogue method according to claim 1, characterized in that: Determining the conversation state of the virtual digital human according to the collected audio signal includes: Sampling the audio signal at a preset sampling rate to obtain a plurality of audio frames; Determine a target audio frame from the plurality of audio frames; Detecting the human voice confidence of the target audio frame, and acquiring current speaking state information; the current speaking state information includes a speaking state and a non-speaking state; If the human voice confidence of the target audio frame is greater than or equal to the first confidence threshold, and the current speech state information is a non-speech state, the number of speech audio frames is reset to zero and then increased by one, and the current speech state information is set to a speech state; If the human voice confidence of the target audio frame is greater than or equal to the first confidence threshold, and the current speech state information is a speech state, then the number of speech audio frames is increased by one; Returning to the step of determining a target audio frame from the plurality of audio frames until it is determined that the speech ends; Determining whether the number of speech audio frames is greater than a valid speech frame number threshold; If the number of speech audio frames is greater than the effective speech frame number threshold, generating a virtual digital human reply message according to the target audio frame; If the number of speech audio frames is less than or equal to the valid speech frame number threshold, no virtual digital human reply information is generated.

4. The virtual digital human dialogue method according to claim 3, characterized in that: The method further comprises: If the voice confidence of the target audio frame is less than the first confidence threshold and greater than the second confidence threshold, and the current speech state information is the speech state, then the number of speech audio frames and the current speech state information are kept unchanged.

5. The virtual digital human dialogue method according to claim 4, characterized in that: The method further comprises: If the human voice confidence of the target audio frame is less than or equal to the second confidence threshold, and the current speech state information is a speech state, the number of unspeaking audio frames is reset to zero and then increased by one, and the current speech state information is set to a unspeaking state; If the human voice confidence of the target audio frame is less than or equal to the second confidence threshold, and the current speech state information is a non-speech state, then the number of the non-speech audio frame is increased by one; If the human voice confidence of the target audio frame is less than the first confidence threshold and greater than the second confidence threshold, and the current speech state information is a non-speech state, then the number of the non-speech audio frames and the current speech state information are kept unchanged; The step of determining that the speech is finished includes: When the number of unspoken audio frames is greater than the unspoken frame number threshold, it is determined that the speech ends.

6. The virtual digital human dialogue method according to claim 5, characterized in that: Each of the audio frames corresponds to a frame number; the method further comprises: If the human voice confidence of the target audio frame is greater than or equal to the first confidence threshold, and the current speech state information is a non-speech state, the frame number of the target audio frame is used as the starting speech frame number; If the human voice confidence of the target audio frame is less than or equal to the second confidence threshold, and the current speech state information is a speech state, taking the frame number of the target audio frame as the end speech frame number; Generating virtual digital human reply information according to the target audio frame includes: The audio frames corresponding to the starting speech frame number to the ending speech frame number are parsed to generate virtual digital human reply information.

7. The virtual digital human dialogue method according to claim 6, characterized in that: The plurality of audio frames are stored in a preset buffer, and the method further comprises: When it is determined that the speech ends, the audio frames corresponding to the start speech frame number to the end speech frame number are deleted from the preset buffer.

8. The virtual digital human dialogue method according to claim 5, characterized in that: The method further comprises: When it is determined that the speech ends, the number of the speech audio frames and the number of the non-speech audio frames are reset to zero.

9. A virtual digital human dialogue system, characterized in that: The virtual digital human dialogue system is communicatively connected with a camera device and a microphone: the virtual digital human dialogue system is used to execute the steps in the virtual digital human dialogue method described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: Used to store computer-readable programs, which, when executed by a processor, can implement the steps of the virtual digital human dialogue method described in any one of claims 1 to 8.

Images