AI intelligent control touch screen and input system thereof

Abstract

The invention discloses an AI intelligent control touch screen and an input system thereof, and relates to the technical field of man-machine interaction, the AI intelligent control touch screen comprises a large-screen display, and the large-screen display comprises a touch screen body used for generating induction electric signals; the host comprises a processor and a memory, is used for processing and storing data, and is connected with the large-screen display; the sound pickup is connected with the host and is used for capturing voice input of a user; and the camera is connected with the host and is used for identifying gestures and facial actions of the user. The system supports various interaction modes such as touch, gestures and voice, so that a doctor can interact with the equipment more visually and naturally, the diversified operation modes greatly improve the operation efficiency of the doctor, especially in an emergency, the doctor can make a response quickly, daily work of the doctor is substantially improved, and the system is worthy of popularization and application. Good application prospects and market values are realized.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to an AI intelligent control touch screen and an input system thereof. Background Art

[0002] In the medical industry, especially in operating rooms and doctors' offices, real-time display and manipulation of patients' 3D models are essential for doctors' diagnosis and surgical planning. Traditional medical equipment usually relies on ordinary computer monitors, which can display high-resolution images but lack interactive functions with users, limiting the efficiency of doctors in performing complex operations.

[0003] Existing solutions usually use a single display mode, and doctors need to rely on mouse and keyboard to operate, which makes the operation less intuitive and difficult to respond quickly in emergency situations. In addition, large-screen displays on the market often do not have touch functions, which makes it impossible for doctors to interact with 3D models through intuitive gestures, affecting the timeliness of medical decision-making.

[0004] In order to improve the user experience, some devices have begun to introduce touch screen functions and support multiple interfaces such as USB and HDMI. Such devices are usually in the form of all-in-one machines that can integrate a computer host and a display screen, making them easy to use in limited spaces. However, these all-in-one machines or small hosts are often unsatisfactory in terms of performance, especially when processing highly complex 3D models, which may cause freezes, seriously affecting the doctor's operating experience and work efficiency.

[0005] Therefore, there is an urgent need for AI intelligent control touch screens and their input systems that can support multiple interaction methods, such as touch, voice, gestures, and keyboard and mouse operations while ensuring high performance, so as to better serve the actual needs of medical work. Summary of the invention

[0006] The purpose of the present invention is to provide an AI intelligent control touch screen and an input system thereof to solve the problems raised in the above-mentioned background technology.

[0007] To solve the above problems, the technical solutions adopted by the present invention are as follows:

[0008] AI intelligent control touch screen, used in human-computer interaction device, the human-computer interaction device includes

[0009] A large-screen display, the large-screen display comprising a touch screen body for generating an induced electrical signal;

[0010] A host computer, comprising a processor and a memory, for processing and storing data, and connected to a large-screen display;

[0011] A microphone connected to the host computer for capturing a user's voice input;

[0012] A camera is connected to the host and is used to recognize user gestures and facial movements.

[0013] Preferably, the large-screen display further comprises a chip and a circuit connected thereto and used for converting the electrical signal into a position coordinate signal.

[0014] Preferably, the large-screen display and the host further include multiple interfaces, including but not limited to an HDMI interface, a USB interface and an audio interface;

[0015] The HDMI interface is used to connect the host to the large-screen display and transmit video signals;

[0016] The audio interface is used to connect the microphone to the host for voice input;

[0017] The USB interface is used to connect the camera to the host for gesture recognition.

[0018] AI intelligent touch screen input system, including

[0019] The voice recognition module is used to recognize and process the voice input captured by the microphone. It has a built-in deep learning model and supports custom command settings. Users can start applications or make system settings through voice commands.

[0020] The gesture recognition module is used to recognize and process the gestures captured by the camera, and use computer vision technology to recognize gestures for intelligent control input;

[0021] The control module is used to upload the command signals recognized and processed by the voice recognition module and the gesture recognition module to the host processor, and further convert them into corresponding operations through the processor and feedback them through the touch screen body.

[0022] Preferably, a speech recognition library and a gesture recognition library are installed in the operating system of the host;

[0023] The speech recognition library has a built-in speech recognition algorithm, supports multiple languages ​​and instructions, and can recognize the user's voice commands and respond;

[0024] The gesture recognition library obtains the user's gesture image through a camera, uses image processing technology to recognize specific gestures, and converts them into corresponding instructions.

[0025] Preferably, the speech recognition step of the speech recognition module includes the following:

[0026] S1. Collect the user's voice signal through the microphone, and the signal can be expressed as a time domain waveform x(t);

[0027] S2, preprocessing the signal, including denoising and pre-emphasis;

[0028] The processing formula of the pre-emphasis is: y(t)=x(t)-αx(t-1);

[0029] Among them, the value range of α is 0.95~0.99;

[0030] S3, divide the signal into frames, each frame length is usually 20 to 30 ms, and use a Hamming window for windowing processing. The windowing processing formula is:

[0031]

[0032] S4, using Mel frequency cepstral coefficients to extract features of speech signals;

[0033] S5, use hidden Markov model or deep neural network to perform pattern matching, and calculate a given speech feature sequence, the given speech feature sequence O = (o 1 ,o 2 ,...,o T )’s likelihood:

[0034]

[0035] Among them, λ is the model parameter, q is the state sequence;

[0036] S6. According to the calculated probabilities, select the output sequence with the highest probability to recognize the voice command.

[0037] Preferably, the steps of calculating the Mel-frequency cepstral coefficients include the following:

[0038] S11. Calculate the power spectrum of each frame signal: P(f) = |X(f)| 2 ;

[0039] Where X(f) is the Fourier transform result of the signal;

[0040] S12, mapping the power spectrum to the Mel scale;

[0041] S13. Calculate the discrete cosine transform of each frame to obtain the Mel-frequency cepstrum coefficients:

[0042] Among them, c m is the mth Mel-frequency cepstral coefficient.

[0043] Preferably, the camera used in the gesture recognition module is also used to identify the facial features of the operator and the changes in mouth shape. The facial features are used to unlock the display interface of the touch screen body. The changes in mouth shape are combined with the voice recognition module to timely and accurately reflect the operator's operating intentions through changes in mouth shape.

[0044] Preferably, the camera recognizes the mouth shape of the operator when speaking and stores and uploads it, which specifically includes the following steps:

[0045] Use a camera to capture and record the operator's mouth movements in real time and use a face detection algorithm to locate the mouth area;

[0046] Extract the image of the mouth area through the image processing library;

[0047] Use a deep learning model to analyze the mouth image, train a lip recognition model, input the mouth area image, and output the corresponding phoneme or word;

[0048] Collect and annotate mouth shape datasets for model training;

[0049] The recognized mouth shapes and corresponding texts or phonemes are stored, and a database or a file system can be used to store the data;

[0050] When a person speaks, the camera captures the mouth shape in real time, uses the trained lip shape recognition model to make inferences, matches the recognized lip shape with the previously stored data, and quickly obtains the corresponding text.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] The present invention supports multiple interaction methods such as touch, gesture, and voice, allowing doctors to interact with the device more intuitively and naturally. This diversified operation method greatly improves the doctor's operating efficiency, especially in emergency situations, allowing for quick responses, bringing substantial improvements to the doctor's daily work, and has good application prospects and market value.

[0053] The present invention interacts with the terminal device through voice and gestures, simplifies the operation process, and reduces the learning curve of doctors during use. This friendly user experience can effectively reduce operational errors and improve the quality of medical services.

[0054] The present invention provides a microphone and a camera to facilitate medical staff to change the content output by the large-screen display through voice and gesture input, such as opening a certain software, adjusting the position of a medical model, switching interfaces, etc. The input method is more convenient and efficient, and no manual input is required.

[0055] The present invention sets up a dual-stream network, one stream processes mouth shape features, and the other stream processes audio features. Finally, the information of the two is merged through a feature fusion layer. Feature splicing, weighted summation or the use of an attention mechanism to fuse information can be selected. A joint loss function is used to consider both the loss of speech recognition and the loss of mouth shape recognition in order to optimize the overall performance of the model. The system receives audio and video streams in real time, performs feature extraction respectively, effectively combines mouth shape recognition with speech recognition modules, improves the accuracy and robustness of overall recognition, and achieves the purpose of rapid and accurate response of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a structural block diagram of the human-computer interaction device in the present invention;

[0057] Figure 2 This is a block diagram of the input system of the AI ​​intelligent control touch screen of the present invention.

[0058] In the figure: 100, large screen display; 110, touch screen body; 200, host; 210, processor; 220, memory; 300, microphone; 400, camera; 500, voice recognition module; 600, gesture recognition module; 700, control module. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] Embodiment 1:

[0061] See also Figure 1 As shown, the present invention is an AI intelligent control touch screen, which is applied to a human-computer interaction device, and the human-computer interaction device includes

[0062] A large-screen display 100, wherein the large-screen display 100 includes a touch screen body 110 for generating an induced electrical signal;

[0063] A host 200, comprising a processor 210 and a memory 220, for processing and storing data, and connected to the large-screen display 100;

[0064] A microphone 300, which is connected to the host 200 and is used to capture the user's voice input;

[0065] The camera 400 is connected to the host 200 and is used to recognize the user's gestures and facial movements.

[0066] The large-screen display 100 also includes a chip and a circuit connected thereto for converting the electrical signal into a position coordinate signal.

[0067] The large-screen display 100 and the host 200 further include multiple interfaces, including but not limited to an HDMI interface, a USB interface, and an audio interface;

[0068] The HDMI interface is used to connect the host 200 to the large-screen display 100 to transmit video signals;

[0069] The audio interface is used to connect the microphone 300 to the host 200 for voice input;

[0070] The USB interface is used to connect the camera 400 to the host 200 for gesture recognition.

[0071] As can be seen from the above, the device can be applied to actual scenarios. By placing the device in an operating room or a doctor's office, the touch screen body 110 is provided to facilitate medical staff to adjust the output content of the large-screen display 100 by touch, such as touching and adjusting the position of the patient's three-dimensional model presented on the large-screen display 100, without having to change the orientation of the model by keyboard and mouse;

[0072] Furthermore, the device is provided with a microphone 300 and a camera 400, so that medical staff can change the content output by the large-screen display 100 by voice and gesture input, such as opening a certain software, adjusting the position of the medical model, switching interfaces, etc. The input method is more convenient and efficient, and no manual input is required;

[0073] Among them, the interface of the large-screen display 100 in this device includes but is not limited to an HDMI interface, which is convenient for connecting to an external host 200. It has a better effect when displaying three-dimensional models without any freezes or delays.

[0074] Embodiment 2:

[0075] See also Figure 2 As shown, the AI ​​intelligent touch screen input system includes

[0076] The voice recognition module 500 is used to recognize and process the voice input captured by the microphone 300, has a built-in deep learning model, supports custom command settings, and the user starts an application or performs system settings through voice commands;

[0077] The gesture recognition module 600 is used to recognize and process the gestures captured by the camera 400, and to recognize the gestures using computer vision technology for intelligent control input;

[0078] The control module 700 is used to upload the command signals recognized and processed by the voice recognition module 500 and the gesture recognition module 600 to the processor 210 , and further convert them into corresponding operations through the processor 210 and feed them back through the touch screen body 110 .

[0079] The operating system of the host 200 is installed with a speech recognition library and a gesture recognition library;

[0080] The speech recognition library has a built-in speech recognition algorithm, supports multiple languages ​​and instructions, and can recognize the user's voice commands and respond;

[0081] The gesture recognition library obtains the user's gesture image through the camera 400, uses image processing technology to recognize specific gestures, and converts them into corresponding instructions.

[0082] The speech recognition steps of the speech recognition module 500 include the following:

[0083] S1. Collecting the user's voice signal through the microphone 300. The signal can be represented as a time domain waveform x(t);

[0084] S2, preprocessing the signal, including denoising and pre-emphasis;

[0085] The processing formula of the pre-emphasis is: y(t)=x(t)-αx(t-1);

[0086] Among them, the value range of α is 0.95~0.99;

[0087] S3, divide the signal into frames, each frame length is usually 20 to 30 ms, and use a Hamming window for windowing processing. The windowing processing formula is:

[0088]

[0089] S4, using Mel frequency cepstral coefficients to extract features of speech signals;

[0090] S5, use hidden Markov model or deep neural network to perform pattern matching, and calculate a given speech feature sequence, the given speech feature sequence O = (o 1 ,o 2 ,...,o T )’s likelihood:

[0091]

[0092] Among them, λ is the model parameter, q is the state sequence;

[0093] S6. According to the calculated probabilities, select the output sequence with the highest probability to recognize the voice command.

[0094] The calculation steps of the Mel-frequency cepstral coefficients include the following:

[0095] S11. Calculate the power spectrum of each frame signal: P(f) = |X(f)| 2 ;

[0096] Where X(f) is the Fourier transform result of the signal;

[0097] S12, mapping the power spectrum to the Mel scale;

[0098] S13. Calculate the discrete cosine transform of each frame to obtain the Mel-frequency cepstrum coefficients:

[0099] Among them, c m is the mth Mel-frequency cepstral coefficient.

[0100] The camera 400 used in the gesture recognition module 600 is also used to identify the facial features of the operator and the changes in mouth shape. The facial features are used to unlock the display interface of the touch screen body 110. The changes in mouth shape are combined with the voice recognition module 500 to timely and accurately reflect the operator's operating intentions through changes in mouth shape.

[0101] The camera 400 recognizes the mouth shape of the operator when speaking and stores and uploads it, which specifically includes the following steps:

[0102] Using the camera 400 to capture and record the operator's mouth movements in real time and using a face detection algorithm to locate the mouth area;

[0103] Face detection algorithms include Haar cascade classifier, MTCNN or D lib;

[0104] Extract the image of the mouth area through the image processing library;

[0105] Use a deep learning model to analyze the mouth image, train a lip recognition model, input the mouth area image, and output the corresponding phoneme or word;

[0106] Collect and annotate mouth shape datasets for model training;

[0107] The recognized mouth shapes and corresponding texts or phonemes are stored, and a database or a file system can be used to store the data;

[0108] When a person is speaking, the mouth shape is captured in real time by the camera 400, and the trained mouth shape recognition model is used for inference, and the recognized mouth shape is matched with previously stored data to quickly obtain the corresponding text;

[0109] Specifically, a dual-stream network can be set up, one stream processes the mouth shape features, and the other stream processes the audio features. Finally, the information of the two is merged through a feature fusion layer. Feature concatenation, weighted summation, or an attention mechanism can be selected to fuse the information. A joint loss function is used to consider both the loss of speech recognition and the loss of mouth shape recognition, so as to optimize the overall performance of the model. The system receives audio and video streams in real time, performs feature extraction respectively, and effectively combines mouth shape recognition with speech recognition module 500, thereby improving the accuracy and robustness of overall recognition and achieving the purpose of rapid and accurate response of the system.

[0110] As can be seen from the above, the operator's voice and gesture are recognized, processed and converted 210 respectively by the voice recognition module 500 and the gesture recognition module 600, and in the process of voice recognition, the operator's mouth deformation characteristics are combined, and the control output of the content of the large-screen display 100 is completed through the processor, without the need for manual command input, and the operation is convenient and fast.

[0111] The standard parts used in the present invention can all be purchased from the market, and the special-shaped parts can be customized according to the description and the drawings. The specific connection methods of each part adopt conventional means such as mature bolts, rivets, welding, etc. in the prior art. The machinery, parts and equipment all adopt conventional models in the prior art, and the circuit connection adopts the conventional connection method in the prior art, which will not be described in detail here. The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

[0112] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. "Multiple" means two or more, unless otherwise clearly and specifically defined.

[0113] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0114] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0115] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0116] In the drawings of the embodiments disclosed in the present invention, only the structures related to the embodiments disclosed in the present invention are involved, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other.

[0117] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. AI intelligent touch screen, used in human-computer interaction devices, characterized by: The human-computer interaction device comprises A large-screen display, the large-screen display comprising a touch screen body for generating an induced electrical signal; A host computer, comprising a processor and a memory, for processing and storing data, and connected to a large-screen display; A microphone connected to the host computer for capturing a user's voice input; A camera is connected to the host and is used to recognize user gestures and facial movements.

2. The AI ​​intelligent control touch screen according to claim 1, characterized in that: The large-screen display also includes a chip and a circuit connected thereto and used for converting the electrical signal into a position coordinate signal.

3. The AI ​​intelligent control touch screen according to claim 2, characterized in that: The large-screen display and the host further include multiple interfaces, including but not limited to an HDMI interface, a USB interface and an audio interface; The HDMI interface is used to connect the host to a large-screen display and transmit video signals; The audio interface is used to connect the microphone to the host for voice input; The USB interface is used to connect the camera to the host for gesture recognition.

4. AI intelligent touch screen input system, characterized by: include The voice recognition module is used to recognize and process the voice input captured by the microphone. It has a built-in deep learning model and supports custom command settings. Users can start applications or make system settings through voice commands. The gesture recognition module is used to recognize and process the gestures captured by the camera, and use computer vision technology to recognize gestures for intelligent control input; The control module is used to upload the command signals recognized and processed by the voice recognition module and the gesture recognition module to the host processor, and further convert them into corresponding operations through the processor and feedback them through the touch screen body.

5. The AI ​​intelligent touch screen input system according to claim 4, characterized in that: The host operating system is installed with a speech recognition library and a gesture recognition library; The speech recognition library has a built-in speech recognition algorithm, supports multiple languages ​​and instructions, and can recognize the user's voice commands and respond; The gesture recognition library obtains the user's gesture image through a camera, uses image processing technology to recognize specific gestures, and converts them into corresponding instructions.

6. The AI ​​intelligent touch screen input system according to claim 4, characterized in that: The speech recognition steps of the speech recognition module include the following: S1. Collect the user's voice signal through the microphone, and the signal can be expressed as a time domain waveform x(t); S2, preprocessing the signal, including denoising and pre-emphasis; The processing formula of the pre-emphasis is: y(t)=x(t)-αx(t-1); Among them, the value range of α is 0.95~0.99; S3, divide the signal into frames, each frame length is usually 20 to 30 ms, and use a Hamming window for windowing processing, the windowing processing formula is: S4, using Mel frequency cepstral coefficients to extract features of speech signals; S5, use a hidden Markov model or a deep neural network to perform pattern matching and calculate a given speech feature sequence, wherein the given speech feature sequence O = (o1, o2, ..., o T )’s likelihood: Among them, λ is the model parameter, q is the state sequence; S6. According to the calculated probabilities, select the output sequence with the highest probability to recognize the voice command.

7. The AI ​​intelligent touch screen input system according to claim 6, characterized in that: The calculation steps of the Mel-frequency cepstral coefficients include the following: S11. Calculate the power spectrum of each frame signal: P(f) = |X(f)| 2 ; Where X(f) is the Fourier transform result of the signal; S12, mapping the power spectrum to the Mel scale; S13. Calculate the discrete cosine transform of each frame to obtain the Mel-frequency cepstrum coefficients: Among them, c m is the mth Mel-frequency cepstral coefficient.

8. The AI ​​intelligent touch screen input system according to claim 4, characterized in that: The camera used in the gesture recognition module is also used to identify the facial features of the operator and the changes in mouth shape. The facial features are used to unlock the display interface of the touch screen body. The changes in mouth shape are combined with the voice recognition module to timely and accurately reflect the operator's operating intentions through changes in mouth shape.

9. The AI ​​intelligent touch screen input system according to claim 8, characterized in that: The camera recognizes the mouth shape of the operator when speaking and stores and uploads it, specifically including the following steps: Use a camera to capture the operator's mouth movements in real time and use a face detection algorithm to locate the mouth area; Extract the image of the mouth area through the image processing library; Use a deep learning model to analyze the mouth image, train a lip recognition model, input the mouth area image, and output the corresponding phoneme or word; Collect and annotate mouth shape datasets for model training; The recognized mouth shapes and corresponding texts or phonemes are stored, and a database or a file system can be used to store the data; When a person speaks, the camera captures the mouth shape in real time, uses the trained lip shape recognition model to make inferences, matches the recognized lip shape with the previously stored data, and quickly obtains the corresponding text.

Images