Gesture-assisted large-screen voice control method and terminal
By judging the interactive status of preset gesture switching in the large education screen and performing voice recognition, combined with gesture confirmation of control commands, the problem of mismanagement and load recognition in voice command control is solved, and the control accuracy and convenience are improved.
Patent Information
- Application Number
- CN202310526992.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-11
- Publication Date
- 2024-11-12
AI Technical Summary
The existing voice command control methods of large-screen educational screens are difficult to distinguish between human-computer voice and human-computer voice, which is prone to mismanagement, and the voice command recognition load is large, which may lead to inaccurate or system overload, and gesture control actions are limited and easy to mismanage.
By determining whether there are preset gestures in the video image, switching to the human-computer interaction state and turning on the microphone, collecting sound signals for voice recognition, controlling the large screen according to the recognition results, and cooperating with the gestures and voice to confirm the interaction state, avoiding incorrect operation.
It reduces the probability of mismanagement of large-screen control, improves the accuracy and efficiency of control, enhances the convenience of use, and avoids normal interactions influencing operations.
Smart Images

Figure CN118942452A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent screen control, and in particular to a gesture-assisted large-screen voice control method and terminal. Background Art
[0002] With the continuous advancement of educational informatization, more and more school classrooms are equipped with educational large screens for multimedia teaching, which improves teaching effectiveness. The current educational large screens are generally installed on the wall next to the blackboard. In order to protect the screen, the placement is usually high. Although the screens have touch functions, it is very inconvenient for teachers to operate. Therefore, in most cases, teachers use a mouse or infrared remote control to control the screen. In recent years, there have also been educational large screen products that support voice command control, as well as educational large screen products that support visual gesture control.
[0003] Among them, the biggest problem with the voice command control method is that the system cannot distinguish between the human-machine voice control state and the human-to-human voice communication state, which is prone to misoperation. In addition, if voice command recognition is performed continuously, the computing load will be significantly increased, and inaccurate recognition or large screen overload may occur. The conventional practice is that before each voice command control, a wake-up word must be said to activate the voice control state, so as to avoid recognizing the human-to-human voice communication state as the human-to-machine voice control state. In addition, voice commands are sometimes ambiguous, and it is difficult to achieve what you see is what you get. In order to eliminate ambiguity, very complicated command expressions are required, which becomes very complicated and reduces the control efficiency. Gesture control methods are divided into static gestures and dynamic gestures. The control actions that can be expressed are very limited, especially dynamic gestures, and it is easy to trigger misoperation. Summary of the invention
[0004] The technical problem to be solved by the present invention is to provide a gesture-assisted large-screen voice control method and terminal to reduce the probability of misoperation of large-screen control.
[0005] In order to solve the above technical problems, a technical solution adopted by the present invention is:
[0006] A gesture-assisted large-screen voice control method, comprising the steps of:
[0007] Determine whether there is a preset gesture in the captured video image, and if so, switch to the human-computer interaction state and turn on the microphone;
[0008] Collecting a sound signal through the microphone, and performing speech recognition on the sound signal to obtain a speech recognition result;
[0009] The large screen is controlled according to the voice recognition result.
[0010] In order to solve the above technical problems, another technical solution adopted by the present invention is:
[0011] A gesture-assisted large-screen voice control terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:
[0012] Determine whether there is a preset gesture in the captured video image, and if so, switch to the human-computer interaction state and turn on the microphone;
[0013] Collecting a sound signal through the microphone, and performing speech recognition on the sound signal to obtain a speech recognition result;
[0014] The large screen is controlled according to the voice recognition result.
[0015] The beneficial effects of the present invention are: setting a preset gesture, if there is a preset gesture in the collected video image, the system mode is switched to the human-computer interaction state, the microphone is turned on to receive the sound signal, and the large screen is controlled according to the voice recognition result in the sound signal, and the human-computer interaction state is switched by gestures to prevent misoperation, and at the same time, after switching to the human-computer interaction state, voice control can be directly performed, which improves the convenience of using the large screen; and through the coordination of gestures and voice, voice recognition is performed only after confirming that the human-computer interaction state has been entered, and the large screen is controlled according to the voice recognition result, which avoids the situation where normal human interaction affects the operation and reduces the probability of misoperation of the large screen. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 A flowchart of a gesture-assisted large-screen voice method according to an embodiment of the present invention;
[0017] Figure 2 A schematic diagram of a preset gesture for switching between human-computer interaction states according to an embodiment of the present invention;
[0018] Figure 3 A schematic diagram of a speech recognition result corresponding to multiple control commands according to an embodiment of the present invention;
[0019] Figure 4 A schematic diagram of a control gesture according to an embodiment of the present invention;
[0020] Figure 5 A schematic diagram of an undo gesture according to an embodiment of the present invention;
[0021] Figure 6 This is a schematic diagram of the structure of a gesture-assisted large-screen voice terminal according to an embodiment of the present invention;
[0022] Description of labels:
[0023] 1. A gesture-assisted large-screen voice terminal; 2. A processor; 3. A memory. DETAILED DESCRIPTION
[0024] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.
[0025] Please refer to Figure 1 , a gesture-assisted large-screen voice control method, comprising the steps of:
[0026] Determine whether there is a preset gesture in the captured video image, and if so, switch to the human-computer interaction state and turn on the microphone;
[0027] Collecting a sound signal through the microphone, and performing speech recognition on the sound signal to obtain a speech recognition result;
[0028] The large screen is controlled according to the voice recognition result.
[0029] From the above description, it can be seen that the beneficial effects of the present invention are: setting a preset gesture, if there is a preset gesture in the collected video image, the system mode is switched to the human-computer interaction state, the microphone is turned on to receive the sound signal, and the large screen is controlled according to the voice recognition result in the sound signal, and the human-computer interaction state is switched by gestures to prevent misoperation. At the same time, after switching to the human-computer interaction state, voice control can be directly performed, which improves the convenience of using the large screen; and through the coordination of gestures and voice, voice recognition is performed only after confirming that the human-computer interaction state has been entered, and the large screen is controlled according to the voice recognition result, which avoids the situation where normal human interaction affects the operation and reduces the probability of misoperation of the large screen.
[0030] Further, performing speech recognition on the sound signal to obtain a speech recognition result and controlling the large screen according to the speech recognition result include:
[0031] Acquire a control command according to the voice recognition result;
[0032] If the number of the control commands is 1, executing the step of controlling the large screen according to the voice recognition result;
[0033] The controlling the large screen according to the speech recognition result comprises:
[0034] The large screen is controlled according to the control command corresponding to the voice recognition result.
[0035] From the above description, it can be seen that if the voice recognition result only corresponds to one control command, the possibility of ambiguity is low. At this time, directly controlling the large screen according to the control command obtained from the voice recognition result can improve the efficiency of controlling the large screen.
[0036] Further, the performing speech recognition on the sound signal to obtain a speech recognition result and the controlling the large screen according to the speech recognition result include:
[0037] Acquire a control command according to the voice recognition result, and if the number of the control commands is greater than 1, acquire a target gesture corresponding to each of the control commands;
[0038] Arrange the target gestures in chronological order to obtain a target gesture sequence;
[0039] Start collecting operation videos and reading frame gestures in the operation videos;
[0040] Arrange the frame gestures in chronological order to obtain a frame gesture sequence;
[0041] Determine whether the target gesture sequence is the same as the frame gesture sequence, and if so, execute the step of controlling the large screen according to the speech recognition result.
[0042] From the above description, it can be seen that when the control command corresponding to the voice recognition result is greater than 1, the probability of an incorrect operation is also increased exponentially. At this time, in addition to obtaining the control command corresponding to the voice recognition result, the target gesture corresponding to the control command is also obtained, and the operation video is obtained at the same time, and the frame gesture in the operation video is read. If the frame gesture is the same as the target gesture, the large screen is controlled according to the voice recognition result, and the control command is obtained through the voice recognition result. The control command is converted into the target gesture, allowing the user to confirm through gestures, effectively avoiding the occurrence of incorrect operations.
[0043] Furthermore, after the target gestures are sorted in chronological order to obtain a target gesture sequence, the method further includes:
[0044] The target gesture sequence is displayed on the large screen.
[0045] From the above description, it can be seen that by displaying the target gesture sequence on the screen, the user can directly perform the corresponding gesture according to the screen for confirmation, so the user does not need to remember the specific operation gesture to complete the confirmation action, which further reduces the probability of misoperation and improves the user experience.
[0046] Further, the step of determining whether the target gesture sequence is the same as the frame gesture sequence, and if so, executing the step of controlling the large screen according to the speech recognition result includes:
[0047] If they are not the same, determine whether the frame gesture is a cancel gesture. If so, exit the human-computer interaction state. Otherwise, return to execute the start of collecting the operation video and read the frame gesture step in the operation video.
[0048] From the above description, it can be seen that if the frame gesture and the target gesture sequence are different, it may be that the user does not want to perform the corresponding operation or the user input is wrong, or there may be a difference in recognition; therefore, at this time, it is first determined whether it is an undo gesture. If it is, it means that the user wants to undo the operation, then exit the human-computer interaction state and wait for the command; if it is not an undo gesture, it means that an input error or a recognition error occurs. At this time, the step of collecting the operation video is repeated, and the user only needs to re-confirm the gesture without re-inputting the voice, which improves the user experience.
[0049] A gesture-assisted large-screen voice control terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:
[0050] Determine whether there is a preset gesture in the captured video image, and if so, switch to the human-computer interaction state and turn on the microphone;
[0051] Collecting a sound signal through the microphone, and performing speech recognition on the sound signal to obtain a speech recognition result;
[0052] The large screen is controlled according to the voice recognition result.
[0053] The beneficial effects of the present invention are: setting a preset gesture, if there is a preset gesture in the collected video image, the system mode is switched to the human-computer interaction state, the microphone is turned on to receive the sound signal, and the large screen is controlled according to the voice recognition result in the sound signal, and the human-computer interaction state is switched by gestures to prevent misoperation, and at the same time, after switching to the human-computer interaction state, voice control can be directly performed, which improves the convenience of using the large screen; and through the coordination of gestures and voice, voice recognition is performed only after confirming that the human-computer interaction state has been entered, and the large screen is controlled according to the voice recognition result, which avoids the situation where normal human interaction affects the operation and reduces the probability of misoperation of the large screen.
[0054] Further, performing speech recognition on the sound signal to obtain a speech recognition result and controlling the large screen according to the speech recognition result include:
[0055] Acquire a control command according to the voice recognition result;
[0056] If the number of the control commands is 1, executing the step of controlling the large screen according to the voice recognition result;
[0057] The controlling the large screen according to the speech recognition result comprises:
[0058] The large screen is controlled according to the control command corresponding to the voice recognition result.
[0059] From the above description, it can be seen that if the voice recognition result only corresponds to one control command, the possibility of ambiguity is low. At this time, directly controlling the large screen according to the control command obtained from the voice recognition result can improve the efficiency of controlling the large screen.
[0060] Further, the performing speech recognition on the sound signal to obtain a speech recognition result and the controlling the large screen according to the speech recognition result include:
[0061] Acquire a control command according to the voice recognition result, and if the number of the control commands is greater than 1, acquire a target gesture corresponding to each of the control commands;
[0062] Arrange the target gestures in chronological order to obtain a target gesture sequence;
[0063] Start collecting operation videos and reading frame gestures in the operation videos;
[0064] Arrange the frame gestures in chronological order to obtain a frame gesture sequence;
[0065] Determine whether the target gesture sequence is the same as the frame gesture sequence, and if so, execute the step of controlling the large screen according to the speech recognition result.
[0066] From the above description, it can be seen that when the control command corresponding to the voice recognition result is greater than 1, the probability of an incorrect operation is also increased exponentially. At this time, in addition to obtaining the control command corresponding to the voice recognition result, the target gesture corresponding to the control command is also obtained, and the operation video is obtained at the same time, and the frame gesture in the operation video is read. If the frame gesture is the same as the target gesture, the large screen is controlled according to the voice recognition result, and the control command is obtained through the voice recognition result. The control command is converted into the target gesture, allowing the user to confirm through gestures, effectively avoiding the occurrence of incorrect operations.
[0067] Furthermore, after the target gestures are sorted in chronological order to obtain a target gesture sequence, the method further includes:
[0068] The target gesture sequence is displayed on the large screen.
[0069] From the above description, it can be seen that by displaying the target gesture sequence on the screen, the user can directly perform the corresponding gesture according to the screen for confirmation, so the user does not need to remember the specific operation gesture to complete the confirmation action, which further reduces the probability of misoperation and improves the user experience.
[0070] Further, the step of determining whether the target gesture sequence is the same as the frame gesture sequence, and if so, executing the step of controlling the large screen according to the speech recognition result includes:
[0071] If they are not the same, determine whether the frame gesture is a cancel gesture. If so, exit the human-computer interaction state. Otherwise, return to execute the start of collecting the operation video and read the frame gesture step in the operation video.
[0072] From the above description, it can be seen that if the frame gesture and the target gesture sequence are different, it may be that the user does not want to perform the corresponding operation or the user input is wrong, or there may be a difference in recognition; therefore, at this time, it is first determined whether it is an undo gesture. If it is, it means that the user wants to undo the operation, then exit the human-computer interaction state and wait for the command; if it is not an undo gesture, it means that an input error or a recognition error occurs. At this time, the step of collecting the operation video is repeated, and the user only needs to re-confirm the gesture without re-inputting the voice, which improves the user experience.
[0073] The above-mentioned gesture-assisted large-screen voice control method and terminal of the present invention can be applied to situations where large screens need to be operated, especially to scenes where human-computer interaction and human-human dialogue are mixed, such as classrooms and live broadcast rooms. The following is an explanation through specific implementation methods.
[0074] Please refer to Figure 1 , Embodiment 1 of the present invention is:
[0075] A gesture-assisted large-screen voice control method, comprising the steps of:
[0076] S1. Determine whether there is a preset gesture in the collected video image. If so, switch to the human-computer interaction state and turn on the microphone;
[0077] Specifically, the camera continuously collects video images, and detects the video images based on gesture detection technology using a pre-trained gesture model. If a preset gesture is detected, the human-computer interaction state is switched and the microphone or microphone array is turned on.
[0078] In an optional implementation, the gesture model is a deep learning model, such as a TRN model (temporal relation Network) or a TSM (temporal shift module) model, and the number of nodes in the model corresponds to the number of gesture types;
[0079] S2. collecting a sound signal through the microphone, and performing speech recognition on the sound signal to obtain a speech recognition result;
[0080] Specifically, a voice activity detection technology is used to detect whether there is a voice signal in the sound signal. If so, voice recognition is performed to obtain a voice recognition result. Otherwise, it is determined whether a preset time length has been reached. If so, the human-computer interaction state is exited, that is, switched to the human-human interaction state and the microphone is turned off; if not, a detection is performed to detect whether there is a voice signal in the sound signal.
[0081] In an optional implementation, a pre-trained voice control command model is used to perform control command recognition on the collected voice signal, and a voice recognition result is output;
[0082] After the human-computer interaction state is triggered, if no voice signal is emitted for a long time, it means that the human-computer interaction state may be triggered by an erroneous operation. At this time, the human-computer interaction state is automatically exited and the human-human interaction state is entered, and the microphone is turned off, which can avoid erroneous operation of the large screen in the future and save power.
[0083] S3, verifying the speech recognition result, and if the verification is successful, executing S4, including:
[0084] S31, obtaining a control command according to the voice recognition result; if there is no matching control command in the voice recognition result, issuing a confirmation message;
[0085] In an optional implementation, the speech recognition result is mapped to a corresponding control command according to a preset semantic understanding rule;
[0086] S32, if the number of the manipulation commands is 1, execute S4; if the number of the manipulation commands is greater than 1, execute S32;
[0087] In an optional implementation, it is determined whether the control command has a next-level control command, for example, if the control command is "settings", the next-level control command may be "screen settings", "security settings", etc.; if so, each next-level control command is obtained, and the control gesture corresponding to each next-level control command is displayed to obtain a control gesture sequence; and the operation video is started to be collected, and the frame gestures in the operation video are read. If there is a frame gesture that is the same as a certain control gesture in the control gesture sequence, the control command corresponding to the control gesture is executed;
[0088] That is, when executing a control command with multiple next-level control commands, it is confirmed with gesture assistance to guide the user to make specific actions for the next operation, reducing the probability of misoperation;
[0089] S32, obtaining a target gesture corresponding to each of the control commands; and sorting the target gestures in chronological order to obtain a target gesture sequence;
[0090] In an optional implementation, the target gesture sequence and the control command corresponding to each target gesture are displayed on a large screen;
[0091] In an optional implementation, the gesture corresponding to each control command is numbered, and the target gesture, the number of the target gesture, and the control command corresponding to the target gesture are displayed on the large screen; the number can be in Arabic numerals, or English letters, etc.; for example, if the user says "open weather", the possible corresponding control actions include "open weather application", "open document named weather" and "open folder named weather", and the system will number the above control actions and display them in a specific area of the screen, such as Figure 3 As shown; the number corresponds to the pre-defined gesture, and the user selects to execute the corresponding control action by making the gesture corresponding to the control action number. For example, the typical Chinese digital gestures are used to correspond to the control action numbers. There are 10 categories in total, corresponding to the numbers G1 to G10, such as Figure 4 As shown;
[0092] S33, start collecting the operation video, and read the frame gestures in the operation video; sort the frame gestures in chronological order to obtain a frame gesture sequence;
[0093] In an optional implementation, a collection time is preset, and after obtaining the target gesture sequence, an operation video with a collection time of the preset collection time is collected; and each frame of the operation video is recognized by using a pre-trained gesture model and gesture recognition technology, and a frame gesture and a number corresponding to the frame gesture are output;
[0094] S34, determining whether the target gesture sequence is the same as the frame gesture sequence, if so, executing S4; otherwise, executing S35;
[0095] In an optional implementation, the target gesture sequence is compared with the frame gesture sequence to see whether the numbers are the same, and the occupancy rate of the storage space is reduced by storing the numbers;
[0096] S35, determining whether the frame gesture is a cancel gesture, if so, exiting the human-computer interaction state or returning to S2; otherwise, returning to execute S33;
[0097] In an optional implementation, please refer to Figure 5 , is an example of an undo gesture;
[0098] S4, controlling the large screen according to the voice recognition result, including: controlling the large screen according to the control command corresponding to the voice recognition result, turning off the microphone and returning to S1 after executing the control command.
[0099] Embodiment 2 of the present invention is:
[0100] Apply the above gesture-assisted large-screen voice control method to actual scenarios:
[0101] Example 1: The teacher uses a large screen based on a gesture-assisted large-screen voice control method in Example 1 to teach students. During the lecture, the teacher wanted to show the students a video, but he could not directly control the screen at close range, and was not proficient in large-screen console computer operation. At this time, he first made a human-computer interaction state switching gesture G1, and the screen working state switched to the human-computer interaction state. Then he issued a voice command to the screen "Play video "xxx"". The screen received the voice command and converted it into text through voice recognition. Then, through semantic analysis, the text was converted into pre-defined operation instructions and sent to the control system. Finally, the system opened the default video player according to the operation instruction and automatically played the video "xxx". After completing the voice command, the system automatically exits the human-computer interaction state. In the application example described, there is no need to pause between making gestures and issuing voice commands, which is very efficient.
[0102] Example 2: The teacher uses an educational large screen based on a gesture-assisted large-screen voice control method in Example 1 of the present invention to teach students. During the course, the teacher wants to search for and open a previously written document "xxx". However, the voice command issued by the teacher is "Search for "xxx"". The system corresponds to multiple control actions after voice recognition and semantic analysis of the voice command. The system is not sure which control action should be performed, so these control actions are displayed on the screen and numbered (such as Figure 3 These manipulation actions and numbers include "1. Browser search "xxx"", "2. Search and open WORD document "xxx"", "3. Search file "xxx"", "4. Search and open PPT document "xxx"". The teacher makes a predefined gesture G2 to the screen, and the system collects the gesture image through the video. After gesture recognition, the corresponding gesture number is 2, and finally the manipulation action corresponding to number 2 "Search and open WORD document "xxx"" is executed. In this application example, the system uses gestures to assist in eliminating the ambiguity of voice commands, increases the degree of freedom of interaction, significantly improves the user experience, and lowers the threshold for use.
[0103] Please refer to Figure 2 , Embodiment 3 of the present invention is:
[0104] A gesture-assisted large-screen voice control terminal 1 includes a processor 2, a memory 3, and a computer program stored in the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, each step in Example 1 is implemented.
[0105] In summary, the present invention provides a gesture-assisted large-screen voice control method and terminal; the human-computer interaction and human-human interaction states are switched through gestures, and the voice is collected and the corresponding control commands are recognized through a microphone or a microphone array in the human-computer interaction state, which saves computing power and avoids many scenarios of misoperation; state switching is necessary in the voice control scenario, otherwise the system does not know which of the teacher's words are control commands to the screen and which are teaching contents to the students. Therefore, before performing voice control, a specific gesture is made to give the system an instruction; at the same time, when voice recognition matches multiple operation commands, the ambiguity is eliminated through gesture confirmation, which further improves the accuracy of the operation and avoids misoperation.
[0106] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A gesture-assisted large-screen voice control method, characterized in that: Includes steps: Determine whether there is a preset gesture in the captured video image, and if so, switch to the human-computer interaction state and turn on the microphone; Collecting a sound signal through the microphone, and performing speech recognition on the sound signal to obtain a speech recognition result; The large screen is controlled according to the voice recognition result.
2. The gesture-assisted large-screen voice control method according to claim 1, characterized in that: The performing speech recognition on the sound signal to obtain a speech recognition result and the controlling the large screen according to the speech recognition result include: Acquire a control command according to the voice recognition result; If the number of the control commands is 1, executing the step of controlling the large screen according to the voice recognition result; The controlling the large screen according to the speech recognition result comprises: The large screen is controlled according to the control command corresponding to the voice recognition result.
3. The gesture-assisted large-screen voice control method according to claim 1, characterized in that: The performing speech recognition on the sound signal to obtain a speech recognition result and controlling the large screen according to the speech recognition result include: Acquire a control command according to the voice recognition result, and if the number of the control commands is greater than 1, acquire a target gesture corresponding to each of the control commands; Arrange the target gestures in chronological order to obtain a target gesture sequence; Start collecting operation videos and reading frame gestures in the operation videos; Arrange the frame gestures in chronological order to obtain a frame gesture sequence; Determine whether the target gesture sequence is the same as the frame gesture sequence, and if so, execute the step of controlling the large screen according to the speech recognition result.
4. The gesture-assisted large-screen voice control method according to claim 3, characterized in that: After the target gestures are sorted in chronological order to obtain a target gesture sequence, the following step further includes: The target gesture sequence is displayed on the screen.
5. The gesture-assisted large-screen voice control method according to claim 3, characterized in that: The step of determining whether the target gesture sequence is the same as the frame gesture sequence, and if so, executing the step of controlling the large screen according to the speech recognition result comprises: If they are not the same, determine whether the frame gesture is a cancel gesture. If so, exit the human-computer interaction state. Otherwise, return to execute the start of collecting the operation video and read the frame gesture step in the operation video.
6. A gesture-assisted large-screen voice control terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: Determine whether there is a preset gesture in the captured video image, and if so, switch to the human-computer interaction state and turn on the microphone; Collecting a sound signal through the microphone, and performing speech recognition on the sound signal to obtain a speech recognition result; The large screen is controlled according to the voice recognition result.
7. A gesture-assisted large-screen voice-controlled terminal according to claim 6, characterized in that: The performing speech recognition on the sound signal to obtain a speech recognition result and the controlling the large screen according to the speech recognition result include: Acquire a control command according to the voice recognition result; If the number of the control commands is 1, executing the step of controlling the large screen according to the voice recognition result; The controlling the large screen according to the speech recognition result comprises: The large screen is controlled according to the control command corresponding to the voice recognition result.
8. The gesture-assisted large-screen voice-controlled terminal according to claim 6, characterized in that: The performing speech recognition on the sound signal to obtain a speech recognition result and controlling the large screen according to the speech recognition result include: Acquire a control command according to the voice recognition result, and if the number of the control commands is greater than 1, acquire a target gesture corresponding to each of the control commands; Arrange the target gestures in chronological order to obtain a target gesture sequence; Start collecting operation videos and reading frame gestures in the operation videos; Arrange the frame gestures in chronological order to obtain a frame gesture sequence; Determine whether the target gesture sequence is the same as the frame gesture sequence, and if so, execute the step of controlling the large screen according to the speech recognition result.
9. A gesture-assisted large-screen voice-controlled terminal according to claim 8, characterized in that: After the target gestures are sorted in chronological order to obtain a target gesture sequence, the following step further includes: The target gesture sequence is displayed on the screen.
10. The gesture-assisted large-screen voice-controlled terminal according to claim 8, characterized in that: The step of determining whether the target gesture sequence is the same as the frame gesture sequence, and if so, executing the step of controlling the large screen according to the speech recognition result comprises: If they are not the same, determine whether the frame gesture is a cancel gesture. If so, exit the human-computer interaction state. Otherwise, return to execute the start of collecting the operation video and read the frame gesture step in the operation video.
Citation Information
Cited By
Voice and gesture multi-mode fusion unmanned platform control method and system
CN121523554A
