A method and system for triggering speech recognition by human posture estimation

Abstract

The present invention provides a method and system for triggering speech recognition by human posture estimation, comprising the following steps: using deep learning to construct a human posture estimation model, using deep learning to construct a dialogue behavior judgment model, using a video acquisition device to collect video information, using a human posture estimation model to detect a human body position and a three-dimensional grid feature map; judging whether the human body position can be extracted; using a dialogue behavior judgment model to perform a binary classification judgment on the three-dimensional grid feature map, identifying and outputting the human posture direction at this time; judging whether the person is speaking to the video acquisition device based on the dialogue behavior; and then starting the speech recognition function. Compared with voice keyword wake-up, it is more conducive to using speech recognition in a relatively noisy environment, and overcomes the problem of misjudgment of voice wake-up triggering, which leads to the problem of speech recognition being mistakenly started. It can provide an interactive experience that is more in line with humans.

Description

Technical Field

[0001] The present invention relates to the field of image and speech processing, and in particular to a method and system for triggering speech recognition by estimating human posture. Background Art

[0002] Today's speech recognition technology is very advanced and mature, but it usually requires saying keywords to trigger speech recognition and give relevant response results. This method does not conform to the natural way of human communication. When people communicate, they generally do not necessarily need to call the other party's name or title to know who they are talking to. In order to solve this problem, a method of triggering speech recognition through human posture estimation is proposed. Summary of the invention

[0003] The present invention aims to solve the technical problems existing in the prior art and provides a method and system for triggering speech recognition by human posture estimation. Compared with voice keyword wake-up, the method is more conducive to using speech recognition in a relatively noisy environment and overcomes the problem of misjudgment of speech wake-up triggering, which leads to the problem of speech recognition being mistakenly started. This enables smart devices or robots to provide an interactive experience that is more in line with human beings.

[0004] The technical solution of the present invention to solve the above technical problem is as follows: a method for triggering speech recognition by estimating human posture,

[0005] It includes the following steps:

[0006] Step 1: Use deep learning to build a human posture estimation model and a dialogue behavior judgment model. The human posture estimation model is trained by reading and processing a piece of original video, and the position of the human body in the original video is detected to obtain a three-dimensional grid feature map of the human body; the dialogue behavior judgment model uses the three-dimensional grid feature map of the human body as a feature input to judge whether a dialogue is in progress;

[0007] Step 2: Use a video acquisition device to collect video information, and decode it frame by frame into time-continuous video frame images, and use a human posture estimation model to detect the human body position and three-dimensional grid feature map in the time-continuous video frame images formed by decoding;

[0008] Step 3: Determine whether the human body position can be extracted;

[0009] Step 4: When the human position information can be extracted, the human position and three-dimensional grid feature map detected by the human posture estimation model in step 2 are used as feature inputs, and the dialogue behavior judgment model is used to perform binary classification judgment on the three-dimensional grid feature map to identify and output the human posture direction at this time;

[0010] Step 5: Determine whether the user is speaking to the video capture device based on the result output by the dialogue behavior judgment model; if the user is speaking to the video capture device, start the speech recognition function.

[0011] Based on the above technical solution, the present invention can also make the following improvements.

[0012] Optionally, when reading and processing the original video in step 1, an end-to-end approach is used to detect the position of the human body in the video to obtain a three-dimensional grid map of the human body.

[0013] Optionally, when judging whether the human body position can be extracted in step 3, if the human body position can be extracted, jump to step 4; if the human body position cannot be extracted, jump to step 2 to re-capture the video information, decode it, and then detect it through the human body posture estimation model.

[0014] Optionally, in step 5, when it is determined that the user is not speaking to the video acquisition device, the voice recognition function is not started.

[0015] Optionally, the human body posture direction output in step 4 includes the angle between the current posture and the video acquisition device. Based on the angle between the current posture and the video acquisition device, a deviation angle range is set. If the current human body posture direction is within the deviation angle range, it is output as speaking to the video acquisition device; otherwise, it is output as not speaking to the video acquisition device.

[0016] The present invention also provides a system for triggering speech recognition by estimating human posture, which adopts any of the above methods for triggering speech recognition by estimating human posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The present invention is a flowchart of a method and system for triggering speech recognition by human posture estimation.

[0018] Figure 2 A schematic diagram of a human posture estimation model of a method and system for human posture estimation triggering speech recognition according to the present invention.

[0019] Figure 3 A schematic diagram of a method for triggering speech recognition by human posture estimation and a dialogue behavior judgment model of the system according to the present invention. DETAILED DESCRIPTION

[0020] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0021] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0022] Figure 1 FIG. 1 is a flow chart of a method and system for triggering speech recognition by estimating human posture according to the present invention. Figure 1 As shown, this embodiment discloses a method for triggering speech recognition by estimating human posture,

[0023] It includes the following steps:

[0024] Step 1: Figure 2 As shown, Figure 2 The figure is a schematic diagram of a human posture estimation model of a method and system for triggering speech recognition by human posture estimation of the present invention. The human posture estimation model is constructed by using deep learning. The human posture estimation model is trained by reading and processing a piece of original video, and the position of the human body in the original video is detected to obtain a three-dimensional grid feature map of the human body; Figure 3 As shown, Figure 3 A schematic diagram of a dialogue behavior judgment model of a method and system for triggering speech recognition by human posture estimation of the present invention. The dialogue behavior judgment model is constructed using deep learning, and the dialogue behavior judgment model uses the human body three-dimensional grid feature map obtained in step 1 as a feature input to judge whether a dialogue is in progress;

[0025] Step 2: Use a video acquisition device to collect video information and decode it frame by frame into time-continuous video frame images;

[0026] Step 3: Determine whether the human body position can be extracted;

[0027] Step 4: When the human position information can be extracted, the human position and three-dimensional grid feature map detected by the human posture estimation model in step 2 are used as feature inputs, and the dialogue behavior judgment model is used to perform binary classification judgment on the three-dimensional grid feature map to identify and output the human posture direction at this time;

[0028] Step 5: Determine whether the user is speaking to the video capture device based on the result output by the dialogue behavior judgment model; if the user is speaking to the video capture device, start the speech recognition function.

[0029] It is understandable that compared with voice keyword wake-up, it is more conducive to using voice recognition in a relatively noisy environment, overcoming the problem of voice wake-up trigger misjudgment, which leads to the problem of voice recognition being mistakenly activated. This allows smart devices or robots to provide a more human-friendly interactive experience.

[0030] In this embodiment, a human posture estimation model is constructed by using the provided original video as a training model. In combination with deep learning, the constructed human posture estimation model is used to detect the position of the human body in the original video to obtain a three-dimensional grid feature map of the human body, and then a dialogue behavior judgment model is constructed. Based on the video information collected and decoded by the camera, it is judged whether there is a human body at the corresponding human body position that is speaking and whether the human body is speaking to the camera, and the voice recognition function is started based on this judgment.

[0031] In a possible implementation, when reading and processing the original video in step 1, an end-to-end approach is used to detect the position of the human body in the video to obtain a three-dimensional grid map of the human body.

[0032] It can be understood that in this embodiment, after obtaining the original video, the position information of the human body in the video is detected in an end-to-end manner, and a three-dimensional grid map of the human body is obtained, and this information is used as the feature input for the next step.

[0033] In a possible implementation, when judging whether the human body position can be extracted in step 3, if the human body position can be extracted, jump to step 4; if the human body position cannot be extracted, jump to step 2 to re-capture the video information, decode it, and then detect it through the human body posture estimation model.

[0034] It can be understood that in this embodiment, there are two detection results of the human body posture estimation model in the detection process, one is that the corresponding human body position can be detected, and the other is that the corresponding human body position cannot be detected. Based on this, the corresponding operation steps and logic of the two detection results are set. First, if the human body position can be extracted, jump to step 4; if the human body position cannot be extracted, jump to step 2 to re-capture the video information, decode it, and then detect it through the human body posture estimation model.

[0035] In a possible implementation, in step 5, when it is determined that the user is not speaking to the video acquisition device, the voice recognition function is not started.

[0036] It is understandable that in this embodiment, if the judgment result is that the person at the human body position is not talking to the camera, the voice recognition function will not be started until it is detected that the person is talking to the camera.

[0037] In a possible implementation, the human body posture direction output in step 4 includes the angle between the current posture and the video capture device. Based on the angle between the current posture and the video capture device, a deviation angle range is set. If the current human body posture direction is within the deviation angle range, it is output as speaking to the video capture device; otherwise, it is output as not speaking to the video capture device.

[0038] It can be understood that in this embodiment, a corresponding deviation angle range value is set for the angle of the human body facing the camera. For example, when the angle between the direction facing the human body and the direction of the line between the camera and the human body is within the deviation angle range, it is judged that the person is speaking to the camera, otherwise it is judged that the person is not speaking to the camera.

[0039] This embodiment also provides a system for triggering speech recognition by estimating human posture, which adopts any of the above methods for triggering speech recognition by estimating human posture.

[0040] It can be understood that, in this embodiment, the system for triggering speech recognition by estimating human posture adopts the method for triggering speech recognition by estimating human posture disclosed in the above embodiments.

[0041] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0042] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0043] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0044] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0045] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0046] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0047] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for triggering speech recognition by human posture estimation, characterized in that: It includes the following steps: Step 1: Use deep learning to build a human posture estimation model and a dialogue behavior judgment model. The human posture estimation model is trained by reading and processing a piece of original video, and the position of the human body in the original video is detected to obtain a three-dimensional grid feature map of the human body; the dialogue behavior judgment model uses the three-dimensional grid feature map of the human body as a feature input to judge whether a dialogue is in progress; Step 2: Use a video acquisition device to collect video information, and decode it frame by frame into time-continuous video frame images, and use a human posture estimation model to detect the human body position and three-dimensional grid feature map in the time-continuous video frame images formed by decoding; Step 3: Determine whether the human body position can be extracted; Step 4: When the human position information can be extracted, the human position and three-dimensional grid feature map detected by the human posture estimation model in step 2 are used as feature inputs, and the dialogue behavior judgment model is used to perform binary classification judgment on the three-dimensional grid feature map to identify and output the human posture direction at this time; Step 5: Determine whether the user is speaking to the video capture device based on the result output by the dialogue behavior judgment model; if the user is speaking to the video capture device, start the speech recognition function.

2. The method for triggering speech recognition by human posture estimation according to claim 1, characterized in that: When reading and processing the original video in step 1, an end-to-end method is used to detect the position of the human body in the video to obtain a three-dimensional grid map of the human body.

3. The method for triggering speech recognition by human posture estimation according to claim 1, characterized in that: When judging whether the human body position can be extracted in step 3, if the human body position can be extracted, jump to step 4; if the human body position cannot be extracted, jump to step 2 to re-collect the video information, decode it, and then detect it through the human body posture estimation model.

4. The method for triggering speech recognition by human posture estimation according to claim 1, characterized in that: In step 5, when it is determined that the user is not speaking to the video acquisition device, the voice recognition function is not started.

5. The method for triggering speech recognition by human posture estimation according to claim 1, characterized in that: The human body posture direction output in step 4 includes the angle between the current posture and the video acquisition device. Based on the angle between the current posture and the video acquisition device, a deviation angle range is set. If the current human body posture direction is within the deviation angle range, it is output as speaking to the video acquisition device; otherwise, it is output as not speaking to the video acquisition device.

6. A system for triggering speech recognition by human posture estimation, characterized in that: It adopts the method of triggering speech recognition by estimating human posture as described in any one of claims 1-5.

Images