A method and system for enhancing voice interaction perception based on video content

Abstract

The present invention relates to the field of audio-visual interaction technology, and specifically to a method and system for enhancing voice interaction perception based on video content, comprising: a capture module, used to capture command audio and lip video subsequently issued by a user who says a wake-up word in a car; a cloud database, used to store lip dynamic curves corresponding to various control commands that can be executed by a car computer; a tracking module, used to receive the lip video captured in the capture module, and track the current user's lip dynamic curves based on the lip video; the present invention bidirectionally collects the command audio and lip video of the user in the car, and analyzes the control commands to be expressed by the user based on the two types of data, so that when the vehicle is driving in a scene with large noise, the in-vehicle interaction system can bidirectionally identify and determine the control commands to be expressed by the user based on the command audio and lip video, thereby effectively enhancing the interaction effect between the in-vehicle user and the car computer interaction scene, and bringing users a more intelligent car-computer interaction and driving experience.

Description

Technical Field

[0001] The present invention relates to the field of audio-visual interaction technology, and in particular to a method and system for enhancing voice interaction perception based on video content. Background Art

[0002] Car voice control is a convenient in-car interactive technology. When driving, the owner does not need to operate manually, just speak the command, and the system can accurately recognize and respond quickly. It improves driving safety and allows the owner to focus on the road; it also simplifies operation and brings a smarter and more comfortable driving experience.

[0003] The invention patent application with application number 201911170029.3 discloses a voice control method for enhancing information security based on cloud devices, the method comprising: the cloud device assigns corresponding virtual security controls to different voiceprints, different virtual security controls have different voice control permissions, so as to control the vehicle-mounted equipment in the target vehicle with adaptation permissions through voice control. For the user's voice command, the cloud device confirms the corresponding virtual security control based on the voiceprint information obtained from the voice command, and then confirms the corresponding voice control permission, and generates a control command that complies with the voice control permission.

[0004] The application aims to solve the problem that "voice control is increasingly widely used in vehicles, but existing in-vehicle voice control solutions generally assign the same control rights to all passengers. For example, all passengers can control the in-vehicle audio, air conditioning, windows and doors through voice commands. However, such control methods may create safety hazards. For example, child passengers in the vehicle may command the windows or doors to be opened while the vehicle is driving, thereby affecting the normal driving of the vehicle."

[0005] However, in the vehicle voice control scenario, the high-speed driving of the vehicle or the presence of severe noise in the driving environment will inevitably affect the vehicle voice control function, such as the problem that the voice control commands cannot be accurately recognized.

[0006] To this end, a method and system for enhanced speech interaction perception based on video content is proposed. Summary of the invention

[0007] In view of the above-mentioned shortcomings of the prior art, the present invention provides a method and system for enhancing voice interaction perception based on video content, which solves the technical problems raised in the above-mentioned background technology.

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions: In a first aspect, a video content-based enhanced voice interaction perception system includes: A capture module is used to capture the command audio and lip video issued by the user who says the wake-up word in the car; a cloud database is used to store the lip dynamic curves corresponding to various control commands that can be executed by the car computer; a tracking module is used to receive the lip video captured in the capture module, and track the lip dynamic curves of the current user based on the lip video; an analysis module is used to obtain the user's lip dynamic curves tracked by the tracking module, and use the user's lip dynamic curves as the similarity analysis target to perform similarity analysis with each lip dynamic curve stored in the cloud database; an in-vehicle interactive system is used to receive the command audio, convert the command audio into text information, and understand the car computer control instructions corresponding to the text information; a control module is used to receive the car computer control instructions understood by the in-vehicle interactive system, and apply the car computer control instructions to drive the car computer to operate; the control module is the original module of the car computer in the car; The capture module is interactively connected to a jump unit via a wireless network, the capture module is interactively connected to a cloud database and a tracking module via a wireless network, the tracking module is interactively connected to a labeling unit and a logic unit at a lower level via a wireless network, the tracking module is interactively connected to an analysis module and an on-board interactive system via a wireless network, the on-board interactive system is interactively connected to the jump unit via a wireless network, and the on-board interactive system is interactively connected to the control module.

[0009] Furthermore, the capture module is integrated with a vehicle-mounted microphone and a camera. After the capture module captures the command audio subsequently issued by the user who speaks the wake-up word in the vehicle, the noise level of the command audio is simultaneously identified, and after the noise level identification is completed, the command audio is further subjected to noise reduction processing; The noise level recognition logic of the command audio is expressed as: ; Where: NI is the noise level of the command audio; K is the total number of noise frames in the command audio; E nj is the short-time energy of the jth noise frame; L is the total amount of speech frames in the command audio; E sk is the short-time energy of the kth speech frame; Among them, the system user customizes the judgment interval of voice frames and noise frames in the capture module, and distinguishes the voice frames and noise frames in the command audio based on the judgment interval. The noise level NI∈[0, 1] of the command audio, the larger the NI, the greater the noise level in the command audio, and vice versa.

[0010] Furthermore, the noise reduction processing logic of the command audio is: ; Where: The command audio after noise reduction; is the priori signal-to-noise ratio estimate; X(k) is the output of the original command audio after discrete Fourier transform; in, , which is the frequency domain of the speech signal after noise reduction.

[0011] Furthermore, the a priori signal-to-noise ratio estimate The calculation formula is: ; Where: α is the smoothing factor; P dd (k) is the power spectral density of the noise signal in the command audio; ξ min It is a custom lower limit value, which is used to prevent the prior signal-to-noise ratio estimate from being too small; Among them, the power spectral density P of the noise signal in the command audio dd (k) Determination is made based on audio within a specified time threshold after the command audio capture ends.

[0012] Furthermore, the capture module is internally provided with submodules, including: A jump unit, used to trigger a jump action and jump to the operation stage of the in-vehicle interactive system; Among them, the in-vehicle interactive system is the original system of the vehicle computer. A trigger threshold is set in the jump unit. During the operation stage, the jump unit receives the recognition result of the noise level of the command audio in the capture module, and compares the recognition result with the trigger threshold to decide whether to trigger the jump action. When the decision result is yes, the capture module does not perform noise reduction processing on the command audio. Otherwise, the capture module performs noise reduction processing on the command audio.

[0013] Furthermore, the preliminary lip dynamic curve stored in the cloud database is the movement path of the corner of the mouth tracked by taking the corner of the mouth as the tracking target when the control command is spoken in Mandarin, and the movement path of the corner of the mouth is recorded as the lip dynamic curve; The tracking module is provided with submodules at the lower level, including: A labeling unit, used for setting a lip grayscale value determination interval, capturing pixels representing lips in a starting frame of a lip video based on the lip grayscale value determination interval, and labeling pixels at left and right edge positions of the pixels representing lips; A logic unit, used to set the frequency at which the tracking module tracks the dynamic curve of the lips of the current user in the lip video; Among them, the logic of the labeling unit for labeling pixels is: lock the labeling target pixel, set the labeling grayscale value of the labeling target pixel, continuously track the movement results of the labeling target pixel based on the tracking frequency and labeling grayscale value set in the logic unit, and obtain several position coordinates relative to the global picture frame in the lip video, connect adjacent position coordinates with a spline curve, and obtain the current user's lip dynamic curve.

[0014] Furthermore, the similarity analysis logic of the lip dynamic curve in the analysis module is expressed as: Firstly, the two lip dynamic curves are placed in the same two-dimensional space for representation, and then the two lip dynamic curves are discretized into a number of points. Based on the set of two points obtained after discretization, the Euclidean distance between the two is calculated, which is recorded as Q. Then the similarity of the two lip dynamic curves is: , based on the above logic, similarity calculation is performed between each stored lip dynamic curve in the cloud database and the user lip dynamic curve tracked in the tracking module; The dynamic curves of the user's lips tracked by the tracking module are denoted as L1 and L2; Then the similarity calculation result is: S(L1) i 、S(L1) i+1 、S(L1) i+2 , … and S(L2) i 、S(L2) i+1 、S(L2) i+2 , …; Where: S(L1) i is the similarity between the user's lip dynamic curve L1 and the i-th lip dynamic curve in the cloud database; S(L2) i is the similarity between the user's lip dynamic curve L2 and the i-th lip dynamic curve in the cloud database.

[0015] Furthermore, after completing the similarity analysis operation, the analysis module synchronously captures the lip dynamic curve stored in the cloud database with the highest similarity, and further identifies the control instruction corresponding to the lip dynamic curve; The in-vehicle interactive system then converts the command audio that has completed the noise reduction process into text information; When the similarity between the text information and the control instruction is not less than 50%, the control module drives the vehicle computer to operate according to the control instruction; The similarity between the text information and the control instruction is determined by any one of the semantic similarity algorithms.

[0016] In a second aspect, a method for enhancing speech interaction perception based on video content comprises the following steps: Real-time collection of the audio and lip video of the user's subsequent commands after the wake-up word is spoken in the car; Create a cloud database, and store the lip dynamic curves corresponding to the executable control instructions of the vehicle computer based on the cloud database; Identify the noise level of the command audio and decide whether to transmit it to the in-vehicle interactive system based on the identification result; If the decision result is yes, the process ends; if the decision result is no, the command audio is subjected to noise reduction, and the user's lip dynamic curve is tracked in the user's lip video simultaneously, and the tracked user's lip dynamic curve is compared with the lip dynamic curve stored in the cloud database for similarity; The command audio after noise reduction processing is transmitted to the in-vehicle interactive system and converted into text information, and the control command corresponding to the lip dynamic curve stored in the cloud database with the highest lip dynamic curve similarity comparison result is obtained; Identify the similarity between text information and control instructions. When the similarity is not less than 50%, control the vehicle computer to operate based on the control instructions.

[0017] Compared with the known public technology, the technical solution provided by the present invention has the following beneficial effects: The present invention provides a method and system for enhanced voice interaction perception based on video content. During the execution of the method and system, the command audio and lip video of the user in the car are bidirectionally collected, and the control instructions to be expressed by the user are analyzed based on the two types of data. Therefore, when the vehicle is driving in a scene with large noise, the in-vehicle interaction system can bidirectionally identify and determine the control instructions to be expressed by the user based on the command audio and lip video, thereby effectively enhancing the interaction effect between the in-vehicle user and the vehicle-computer interaction scene, and bringing the user a more intelligent vehicle-computer interaction and driving experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 It is a structural diagram of an enhanced voice interaction perception system based on video content; Figure 2 The present invention is a flowchart of a method for enhancing speech interaction perception based on video content. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The present invention will be further described below in conjunction with the embodiments.

[0022] Embodiment 1: The embodiment of the present invention is a system for enhancing voice interaction perception based on video content, such as Figure 1 As shown, including: A capture module is used to capture the command audio and lip video of the user who says the wake-up word in the car; The capture module is integrated with the vehicle microphone and camera. After capturing the command audio issued by the user who says the wake-up word in the car, the capture module simultaneously identifies the noise level of the command audio, and after completing the noise level identification, further performs noise reduction processing on the command audio; The noise level recognition logic of the command audio is expressed as: ; Where: NI is the noise level of the command audio; K is the total number of noise frames in the command audio; E nj is the short-time energy of the jth noise frame; L is the total amount of speech frames in the command audio; E sk is the short-time energy of the kth speech frame; The system end user customizes the judgment interval of the voice frame and the noise frame in the capture module, and distinguishes the voice frame and the noise frame in the command audio based on the judgment interval. The noise level NI∈[0,1] of the command audio, the larger the NI, the greater the noise level in the command audio, and vice versa, the smaller the noise level in the command audio; Through the above logic formula, the noise level of the instruction audio is further calculated, providing operation logic support for the operation of the jump unit.

[0023] The noise reduction processing logic of the command audio is: ; Where: The command audio after noise reduction; is the priori signal-to-noise ratio estimate; X(k) is the output of the original command audio after discrete Fourier transform; in, , that is, the frequency domain of the speech signal after noise reduction; A priori SNR estimate The calculation formula is: ; Where: α is the smoothing factor; P dd (k) is the power spectral density of the noise signal in the command audio; ξ min It is a custom lower limit value, which is used to prevent the prior signal-to-noise ratio estimate from being too small; Among them, the power spectral density P of the noise signal in the command audio dd(k) determining based on audio within a specified time threshold after the command audio capture ends; The command audio is denoised using the above logic formula, further improving the quality of the command audio used in the running phase of the analysis module. This is combined with a comprehensive analysis of the user's lip dynamic curve to make a more accurate determination of the control command the user wants to express.

[0024] The capture module is internally provided with submodules, including: A jump unit, used to trigger a jump action and jump to the operation stage of the in-vehicle interactive system; Among them, the in-vehicle interactive system is the original system of the in-vehicle computer. A trigger threshold is set in the jump unit. During the operation phase of the jump unit, the jump unit receives the recognition result of the noise level of the command audio in the capture module, and decides whether to trigger the jump action based on the comparison between the recognition result and the trigger threshold. When the decision result is yes, the capture module does not perform noise reduction processing on the command audio. Otherwise, the capture module performs noise reduction processing on the command audio. The cloud database is used to store the lip dynamic curves corresponding to each control command that can be executed by the vehicle computer; A tracking module, used to receive the lip video captured by the capture module, and track the dynamic curve of the lips of the current user based on the lip video; The preliminary lip dynamic curve stored in the cloud database is the movement path of the corner of the mouth tracked when the control command is given in Mandarin, with the corner of the mouth as the tracking target. The movement path of the corner of the mouth is recorded as the lip dynamic curve; The tracking module is divided into submodules, including: A labeling unit, used for setting a lip grayscale value determination interval, capturing pixels representing lips in a starting frame of a lip video based on the lip grayscale value determination interval, and labeling pixels at left and right edge positions of the pixels representing lips; A logic unit, used to set the frequency at which the tracking module tracks the dynamic curve of the lips of the current user in the lip video; The logic of the labeling unit for labeling pixels is as follows: locking the labeling target pixel, setting the labeling gray value of the labeling target pixel, and continuously tracking the movement result of the labeling target pixel based on the tracking frequency and labeling gray value set in the logic unit to obtain a number of position coordinates relative to the global frame of the lip video, and connecting adjacent position coordinates with a spline curve to obtain the current user lip dynamic curve; An analysis module is used to obtain the user's lip dynamic curve tracked by the tracking module, take the user's lip dynamic curve as a similarity analysis target, and perform similarity analysis with each lip dynamic curve stored in the cloud database; The similarity analysis logic of the lip dynamic curve in the analysis module is expressed as: Firstly, the two lip dynamic curves are placed in the same two-dimensional space for representation, and then the two lip dynamic curves are discretized into a number of points. Based on the set of two points obtained after discretization, the Euclidean distance between the two is calculated, which is recorded as Q. Then the similarity of the two lip dynamic curves is: , based on the above logic, similarity calculation is performed between each stored lip dynamic curve in the cloud database and the user lip dynamic curve tracked in the tracking module; The dynamic curves of the user's lips tracked by the tracking module are denoted as L1 and L2; Then the similarity calculation result is: S(L1) i 、S(L1) i+1 、S(L1) i+2 , … and S(L2) i 、S(L2) i+1 、S(L2) i+2 , …; Where: S(L1) i is the similarity between the user's lip dynamic curve L1 and the i-th lip dynamic curve in the cloud database; S(L2) i is the similarity between the user's lip dynamic curve L2 and the i-th lip dynamic curve in the cloud database; The vehicle interactive system is used to receive the command audio, convert the command audio into text information, and understand the vehicle control command corresponding to the text information; The control module is used to receive the vehicle computer control instructions understood by the vehicle interactive system, and use the vehicle computer control instructions to drive the vehicle computer to operate; After completing the similarity analysis operation, the analysis module synchronously captures the lip dynamic curve stored in the cloud database with the highest similarity, and further identifies the control instruction corresponding to the lip dynamic curve; The in-vehicle interactive system then converts the command audio that has completed the noise reduction process into text information; When the similarity between the text information and the control instruction is not less than 50%, the control module drives the vehicle computer to operate according to the control instruction; Wherein, the similarity between the text information and the control instruction is determined by any one of the semantic similarity algorithms; The control module is the original module of the vehicle computer; The capture module is interactively connected to a jump unit through a wireless network, the capture module is interactively connected to a cloud database and a tracking module through a wireless network, the tracking module is interactively connected to a labeling unit and a logic unit through a wireless network, the tracking module is interactively connected to an analysis module and an on-board interactive system through a wireless network, the on-board interactive system is interactively connected to the jump unit through a wireless network, and the on-board interactive system is interactively connected to the control module.

[0025] In this embodiment, the capture module operates to capture the command audio and lip video issued by the user who says the wake-up word in the car. The jump unit synchronously triggers the jump action based on the trigger jump logic and jumps to the operation stage of the in-vehicle interactive system. The cloud database stores the lip dynamic curves corresponding to each control instruction that can be executed by the car computer in real time. The tracking module post-operates to receive the lip video captured in the capture module, and tracks the lip dynamic curves of the current user based on the lip video. The labeling unit synchronously sets the lip grayscale value judgment interval, and captures the pixels representing the lips in the starting frame of the lip video based on the lip grayscale value judgment interval, and performs the left and right pixel recognition on the pixels representing the lips. The pixels at the side edge positions are marked, and the logic unit sets the frequency of the tracking module tracking the current user's lip dynamic curve in the lip video in real time. The analysis module then obtains the user's lip dynamic curve tracked by the tracking module, and takes the user's lip dynamic curve as the similarity analysis target. Similarity analysis is performed with each lip dynamic curve stored in the cloud database. The in-vehicle interactive system is linked to operate to receive command audio, convert the command audio into text information, understand the vehicle control instructions corresponding to the text information, and finally receive the vehicle control instructions understood by the in-vehicle interactive system through the control module, and apply the vehicle control instructions to drive the vehicle operation.

[0026] Through the operation of the system in the above embodiment, it effectively serves the driving scenarios with high noise, provides stable, balanced and more fault-tolerant interactive services to users driving the vehicles, and further improves the intelligence level of user-vehicle interaction technology in the field of intelligent driving.

[0027] Embodiment 2: In terms of specific implementation, based on Example 1, this example refers to Figure 2 The enhanced voice interaction perception system based on video content in Example 1 is further described in detail: A method for enhancing speech interaction perception based on video content comprises the following steps: Real-time collection of the audio and lip video of the subsequent commands issued by the user who says the wake-up word in the car; Create a cloud database, and store the lip dynamic curves corresponding to the executable control instructions of the vehicle computer based on the cloud database; Identify the noise level of the command audio and decide whether to transmit it to the in-vehicle interactive system based on the identification result; If the decision result is yes, the process ends; if the decision result is no, the command audio is subjected to noise reduction, and the user's lip dynamic curve is tracked in the user's lip video simultaneously, and the tracked user's lip dynamic curve is compared with the lip dynamic curve stored in the cloud database for similarity; The command audio after noise reduction processing is transmitted to the in-vehicle interactive system and converted into text information, and the control command corresponding to the lip dynamic curve stored in the cloud database with the highest lip dynamic curve similarity comparison result is obtained; Identify the similarity between text information and control instructions. When the similarity is not less than 50%, control the vehicle computer to operate based on the control instructions.

[0028] In summary, during the execution of the method and system in the above embodiments, the user's command audio and lip video are bidirectionally collected, and the control instructions to be expressed by the user are analyzed based on the two types of data. Therefore, when the vehicle is driving in a scene with high noise, the in-vehicle interaction system can bidirectionally identify and determine the control instructions to be expressed by the user based on the command audio and lip video, thereby effectively enhancing the interaction effect between the in-vehicle user and the car-machine interaction scene, and bringing users a more intelligent car-machine interaction and driving experience.

[0029] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An enhanced voice interaction perception system based on video content, characterized in that: include: A capture module is used to capture the command audio and lip video of the user who says the wake-up word in the car; The cloud database is used to store the lip dynamic curves corresponding to each control command that can be executed by the vehicle computer; A tracking module, used to receive the lip video captured by the capture module, and track the dynamic curve of the lips of the current user based on the lip video; An analysis module is used to obtain the user's lip dynamic curve tracked by the tracking module, take the user's lip dynamic curve as a similarity analysis target, and perform similarity analysis with each lip dynamic curve stored in the cloud database; The vehicle interactive system is used to receive the command audio, convert the command audio into text information, and understand the vehicle control command corresponding to the text information; The control module is used to receive the vehicle computer control instructions understood by the vehicle interactive system, and use the vehicle computer control instructions to drive the vehicle computer to operate; The control module is an original module of the vehicle computer.

2. The enhanced voice interaction perception system based on video content according to claim 1, characterized in that: The capture module is integrated with a vehicle-mounted microphone and a camera. After capturing the command audio subsequently issued by the user who speaks the wake-up word in the vehicle, the capture module simultaneously performs noise level recognition on the command audio, and after completing the noise level recognition, further performs noise reduction processing on the command audio; The noise level recognition logic of the command audio is expressed as: ; Where: NI is the noise level of the command audio; K is the total number of noise frames in the command audio; E nj is the short-time energy of the jth noise frame; L is the total amount of speech frames in the command audio; E sk is the short-time energy of the kth speech frame; Among them, the system user customizes the judgment interval of voice frames and noise frames in the capture module, and distinguishes the voice frames and noise frames in the command audio based on the judgment interval. The noise level NI∈[0, 1] of the command audio, the larger the NI, the greater the noise level in the command audio, and vice versa.

3. The enhanced voice interaction perception system based on video content according to claim 2, characterized in that: The noise reduction processing logic of the command audio is: ； Where: The command audio after noise reduction; is the priori signal-to-noise ratio estimate; X(k) is the output of the original command audio after discrete Fourier transform; in, , which is the frequency domain of the speech signal after noise reduction.

4. The enhanced voice interaction perception system based on video content according to claim 3 is characterized in that: The a priori signal-to-noise ratio estimate The calculation formula is: ； Where: α is the smoothing factor; P dd (k) is the power spectral density of the noise signal in the command audio; ξ min It is a custom lower limit value, which is used to prevent the prior signal-to-noise ratio estimate from being too small; Among them, the power spectral density P of the noise signal in the command audio dd (k) Determination is made based on audio within a specified time threshold after the command audio capture ends.

5. The enhanced voice interaction perception system based on video content according to claim 1, characterized in that: The capture module is internally provided with submodules, including: A jump unit, used to trigger a jump action and jump to the operation stage of the in-vehicle interactive system; Among them, the in-vehicle interactive system is the original system of the vehicle computer. A trigger threshold is set in the jump unit. During the operation stage, the jump unit receives the recognition result of the noise level of the command audio in the capture module, and compares the recognition result with the trigger threshold to decide whether to trigger the jump action. When the decision result is yes, the capture module does not perform noise reduction processing on the command audio. Otherwise, the capture module performs noise reduction processing on the command audio.

6. The enhanced voice interaction perception system based on video content according to claim 1, characterized in that: The preliminary lip dynamic curve stored in the cloud database is the movement path of the corner of the mouth tracked when the control command is given in Mandarin, with the corner of the mouth as the tracking target. The movement path of the corner of the mouth is recorded as the lip dynamic curve; The tracking module is provided with submodules at the lower level, including: A labeling unit, used for setting a lip grayscale value determination interval, capturing pixels representing lips in a starting frame of a lip video based on the lip grayscale value determination interval, and labeling pixels at left and right edge positions of the pixels representing lips; A logic unit, used to set the frequency at which the tracking module tracks the dynamic curve of the lips of the current user in the lip video; Among them, the logic of the labeling unit for labeling pixels is: lock the labeling target pixel, set the labeling grayscale value of the labeling target pixel, continuously track the movement results of the labeling target pixel based on the tracking frequency and labeling grayscale value set in the logic unit, and obtain several position coordinates relative to the global picture frame in the lip video, connect adjacent position coordinates with a spline curve, and obtain the current user's lip dynamic curve.

7. The enhanced voice interaction perception system based on video content according to claim 1, characterized in that: The similarity analysis logic of the lip dynamic curve in the analysis module is expressed as: Firstly, the two lip dynamic curves are placed in the same two-dimensional space for representation, and then the two lip dynamic curves are discretized into a number of points. Based on the set of two points obtained after discretization, the Euclidean distance between the two is calculated, which is recorded as Q. Then the similarity of the two lip dynamic curves is: , based on the above logic, similarity calculation is performed between each stored lip dynamic curve in the cloud database and the user lip dynamic curve tracked in the tracking module; The dynamic curves of the user's lips tracked by the tracking module are denoted as L1 and L2; Then the similarity calculation result is: S(L1) i 、S(L1) i+1 、S(L1) i+2 , … and S(L2) i 、S(L2) i+1 、S(L2) i+2 , …; Where: S(L1) i is the similarity between the user's lip dynamic curve L1 and the i-th lip dynamic curve in the cloud database; S(L2) i is the similarity between the user's lip dynamic curve L2 and the i-th lip dynamic curve in the cloud database.

8. The video content-based enhanced voice interaction perception system according to claim 1, characterized in that: After completing the similarity analysis operation, the analysis module synchronously captures the lip dynamic curve stored in the cloud database with the highest similarity, and further identifies the control instruction corresponding to the lip dynamic curve; The in-vehicle interactive system then converts the command audio that has completed the noise reduction process into text information; When the similarity between the text information and the control instruction is not less than 50%, the control module drives the vehicle computer to operate according to the control instruction; The similarity between the text information and the control instruction is determined by any one of the semantic similarity algorithms.

9. The enhanced voice interaction perception system based on video content according to claim 1, characterized in that: The capture module is interactively connected to a jump unit via a wireless network, the capture module is interactively connected to a cloud database and a tracking module via a wireless network, the tracking module is interactively connected to a labeling unit and a logic unit at a lower level via a wireless network, the tracking module is interactively connected to an analysis module and an on-board interactive system via a wireless network, the on-board interactive system is interactively connected to the jump unit via a wireless network, and the on-board interactive system is interactively connected to the control module.

10. A method for enhancing speech interaction perception based on video content, the method being an implementation method of an enhanced speech interaction perception system based on video content as claimed in any one of claims 1 to 9, characterized in that: The following steps are involved: Real-time collection of the audio and lip video of the user's subsequent commands after the wake-up word is spoken in the car; Create a cloud database, and store the lip dynamic curves corresponding to the executable control instructions of the vehicle computer based on the cloud database; Identify the noise level of the command audio and decide whether to transmit it to the in-vehicle interactive system based on the identification result; If the decision result is yes, the process ends; if the decision result is no, the command audio is subjected to noise reduction, and the user's lip dynamic curve is tracked in the user's lip video simultaneously, so as to compare the tracked user's lip dynamic curve with the lip dynamic curve stored in the cloud database for similarity; The command audio after noise reduction processing is transmitted to the in-vehicle interactive system and converted into text information, and the control command corresponding to the lip dynamic curve stored in the cloud database with the highest lip dynamic curve similarity comparison result is obtained; Identify the similarity between text information and control instructions. When the similarity is not less than 50%, control the vehicle computer to operate based on the control instructions.

Images