Human-computer interaction method, device and system

Abstract

A human-computer interaction method, device and system, which relates to the field of human-computer interaction technology, helps to output and display the voice content of one or more users to the corresponding output position on the touch screen, and improves the efficiency of touch screen input and the user experience. The human-computer interaction method includes: establishing a correspondence between a first voiceprint and a first output position on the touch screen; receiving a first voice, and when it is determined that the voiceprint of the first voice matches the first voiceprint, identifying the content of the voice, and outputting and displaying the content to the first output position.

Description

Technical Field

[0001] The present application relates to the field of human-computer interaction technology, and in particular to a human-computer interaction method, device and system. Background Art

[0002] Human computer interaction (HCI; or, human machine interaction, HMI) is a discipline that studies the interactive relationship between systems and users. The system here can be various machines, or computerized systems and software. In human-computer interaction involving touch screen input, input can be achieved through touch operation. However, due to cost and technical reasons, the way of inputting on the touch screen through touch operation often results in handwriting ligatures and difficult handwriting control, resulting in difficulty in input and low input efficiency. Summary of the invention

[0003] The human-computer interaction method, device and system provided in the present application, through a multi-modal human-computer interaction method, help to output and display the voice content of one or more users to the corresponding output position on the touch screen, thereby improving the efficiency of touch screen input and the user experience.

[0004] To achieve the above objectives, this application provides the following technical solutions:

[0005] In a first aspect, the present application provides a human-computer interaction method, which is applied to a human-computer interaction system. The method includes: firstly establishing a correspondence between a first voiceprint and a first output position on a touch screen; then, when a first voice is received and it is determined that the voiceprint of the voice matches the first voiceprint, recognizing the content of the voice, and outputting the recognized content and displaying the first output position.

[0006] Through the first aspect of the present application, the voice content input by one or more users can be output and displayed on the output position indicated by the user on the touch screen, avoiding the problem of different users' voice content being mixed and displayed on the touch screen when multiple people use voice input at the same time, thereby improving the efficiency and experience of touch screen input.

[0007] In combination with the first aspect, in a possible implementation, the process of establishing the correspondence between the first voiceprint and the first output position on the touch screen may be: receiving a touch operation and a second voice, and when it is determined that the touch operation matches a predetermined first rule and the second voice matches a predetermined second rule, determining the first output position on the touch screen according to the position of the touch operation, and extracting the first voiceprint from the second voice; and then establishing the correspondence between the first voiceprint and the first output position.

[0008] In combination with the first aspect, in a possible implementation, the process of determining the first output position according to the position of the touch operation may be as follows: determining the starting position and range of the first output position according to a set of contact positions in the position of the touch operation.

[0009] Optionally, the shape indicated by the first output position may be a rectangle, a circle, a rhombus, etc. When the shape indicated by the first output position is a rectangle, the first output position may include the coordinates of the upper left corner, as well as the width and height. Alternatively, the first output position may also include its starting position and ending position, such as the coordinates of the upper left corner and the coordinates of the lower right corner of the first output position.

[0010] Optionally, the system can also output and display the first output position on the touch screen. For example, the first output position is displayed in a frame or other form that can be distinguished from the current background of the touch screen. In addition, when the content of the above-mentioned first voice exceeds the upper limit that can be output at the first output position, the system can also generate a scroll bar in the horizontal or disposal direction of the first output position and display it on the touch screen. In this implementation mode, it can be ensured that the voice content input by the user is recorded, and it is convenient for the user to review any recorded voice content at any time, thereby further enhancing the user experience.

[0011] In conjunction with the first aspect, in a possible implementation, the process of determining whether the touch operation matches the predetermined first rule may be: determining whether the set of contact point positions in the position of the touch operation is consistent with the predetermined position rule. Alternatively, identifying a shape formed by the set of contact point positions in the position of the touch operation, and determining whether the shape matches a predetermined shape.

[0012] In combination with the first aspect, in a possible implementation, the process of determining whether the second voice matches the predetermined second rule may be: identifying the content of the second voice, and determining whether the content matches the predetermined content.

[0013] Optionally, the system can also cancel the correspondence between the first voiceprint and the first output position. In this way, the correspondence between the first voiceprint and other output positions can be established, so that the user corresponding to the first voiceprint can switch to other output positions to input his voice content.

[0014] Optionally, the system can also establish a corresponding relationship between the second voiceprint and the first output position. And, when a third voice is received and it is determined that the voiceprint of the voice matches the second voiceprint, the content of the voice can be output to the blank space of the first output position, or the content of the first voice can be overwritten. In this way, the function of a user supplementing or modifying the voice content (the first voice) of other users through voice (the third voice) can be realized. Therefore, through this implementation method, multiple users can collaborate with each other to input content on the touch screen through voice, thereby improving the efficiency and experience of touch screen input. In addition, optionally, since the third voice and the first voice are sent by different users, the system can also select an output format different from the output format of the first voice when outputting the content of the third voice, so that the display effect is better.

[0015] Optionally, when the second voice is collected by an array microphone, the position of the sound source of the second voice may be calculated, and before the corresponding relationship between the first voiceprint and the first output position is established, it is determined whether the sound source position of the second voice and the position of the touch operation meet a preset condition. If so, the corresponding relationship between the first voiceprint and the first output position is established; otherwise, the corresponding relationship is not established.

[0016] Optionally, the system may also receive an image in front of the touch screen captured by the image collector, and before establishing the correspondence between the first voiceprint and the first output position, analyze and track the content of the image in real time, and determine whether the user who performs the touch operation and the user who emits the second voice are the same user based on the results of the image analysis and tracking. Alternatively, determine whether the user who performs the touch operation and the user who emits the second voice are the same user by combining sound source localization and image tracking. If it is determined that they are the same user, the correspondence is established; otherwise, the correspondence is not established.

[0017] Through the above two methods, when multiple users start voice input at the same time, it is possible to avoid binding the voiceprint of one user to the output position indicated by another user, thereby improving the robustness of the system.

[0018] In a second aspect, the present application provides a human-computer interaction method, which is applied to a computer device, and the method includes: first establishing a correspondence between a first voiceprint and a first output position; then when a first voice is received and it is determined that the voiceprint of the voice matches the above-mentioned first voiceprint, identifying the content of the voice and outputting the content to the above-mentioned first output position.

[0019] In combination with the second aspect, in a possible implementation, the process of establishing the correspondence between the first voiceprint and the first output position may be: receiving a contact position and a second voice, and when it is determined that the contact position matches a predetermined first rule and the second voice matches a predetermined second rule, determining the first output position according to the contact position, and extracting the first voiceprint from the second voice; and then establishing the correspondence between the first voiceprint and the first output position.

[0020] In combination with the second aspect, in a possible implementation manner, the process of determining the first output position according to the contact position may be: determining the starting position and range of the first output position according to a contact position set in the contact position.

[0021] Optionally, the shape indicated by the first output position may be a rectangle, a circle, a rhombus, etc. When the shape indicated by the first output position is a rectangle, the first output position may include the coordinates of the upper left corner, as well as the width and height. Alternatively, the first output position may also include its starting position and ending position, such as the coordinates of the upper left corner and the coordinates of the lower right corner of the first output position.

[0022] Optionally, the computer device may also output the first output position; for example, the first output position may be displayed in a frame or other form that can be distinguished from the current background.

[0023] In conjunction with the second aspect, in a possible implementation, the process of determining whether the contact position matches the predetermined first rule may be: determining whether the contact position set in the contact position is consistent with the predetermined position rule. Alternatively, identifying a shape formed by the contact position set in the contact position, and determining whether the shape matches a predetermined shape.

[0024] In conjunction with the second aspect, in a possible implementation, the process of determining whether the second voice matches the predetermined second rule may be: identifying the content of the second voice, and determining whether the content matches the predetermined content.

[0025] Optionally, the computer device may also cancel the correspondence between the first voiceprint and the first output position.

[0026] Optionally, the computer device may also establish a corresponding relationship between the second voiceprint and the first output position. Furthermore, when a third voice is received and the voiceprint of the third voice matches the second voiceprint, the content of the third voice may be output to a blank space of the first output position, or overwrite the content of the first voice.

[0027] Optionally, when the second voice is collected by an array microphone, the position of the sound source of the second voice can also be calculated, and before establishing the correspondence between the first voiceprint and the first output position, it is determined whether the sound source position of the second voice and the contact position meet preset conditions.

[0028] Optionally, the computer device may also receive an image in front of the touch screen captured by the image collector, and before establishing the correspondence between the first voiceprint and the first output position, analyze and track the content of the image in real time, and determine that the user who performs the touch operation and the user who emits the second voice are the same user based on the results of the image analysis and tracking. Alternatively, determine that the user who performs the touch operation and the user who emits the second voice are the same user by combining sound source localization and image tracking.

[0029] The explanation of the relevant contents and description of the beneficial effects of the technical solutions provided by the second aspect and any possible design thereof can refer to the technical solutions provided by the above-mentioned first aspect or its corresponding possible designs, and will not be repeated here.

[0030] In a third aspect, the present application provides a human-computer interaction system, which can be used to execute any of the methods provided in the first aspect or the second aspect. The human-computer interaction system can include a touch screen and a processor.

[0031] The touch screen is used to receive a touch operation and send the position of the touch operation to the processor.

[0032] The processor executes any human-computer interaction method provided in the second aspect. The explanation of the relevant contents and description of the beneficial effects of any possible technical solution implemented by the processor can refer to the technical solution provided by the second aspect or its corresponding possible design, which will not be repeated here.

[0033] Optionally, the human-computer interaction system may further include a voice collector for collecting the first voice, the second voice and the third voice, and sending the first voice, the second voice and the third voice to the processor.

[0034] Optionally, the human-computer interaction system may further include an image collector for collecting images near the front of the touch screen and sending them to the processor. In the third aspect, the description of possible technical solutions and beneficial effects performed by the processor and the touch screen can refer to the technical solutions provided by the first aspect or the second aspect or their corresponding possible designs, which will not be repeated here.

[0035] In a fourth aspect, the present application provides a computer device, which can be used to execute any one of the methods provided in the second aspect above. In this case, the computer device can specifically be a processor or a device including a processor.

[0036] In a possible design, the device can be divided into functional modules according to any of the methods provided in the second aspect. In this implementation, the computer device includes a speech processing unit and a comprehensive processing unit.

[0037] The voice processing unit is used to receive the first voice and recognize the content of the voice when it is determined that the voiceprint of the first voice matches the first voiceprint.

[0038] The comprehensive processing unit is used to establish a corresponding relationship between the first voiceprint and the first output position; and is also used to output the content of the first voice to the first output position.

[0039] The computer device also includes a touch position processing unit for receiving a touch position generated by a touch operation; and for determining a first output position according to the touch position when it is determined that the touch position matches a predetermined first rule.

[0040] The speech processing unit is further configured to receive a second speech, and extract a first voiceprint from the speech when it is determined that the speech matches a predetermined second rule.

[0041] In the above process, when the contact position processing unit determines the first output position according to the contact position, it is specifically used to: determine the starting position and range of the first output position according to the contact position set in the contact position. Optionally, the comprehensive processing unit is also used to output the first output position.

[0042] Optionally, the comprehensive processing unit is further used to release the correspondence between the first voiceprint and the first output position.

[0043] Optionally, the voice processing unit is further used to receive a third voice, and when it is determined that the voiceprint of the third voice matches the second voiceprint, identify the content of the third voice. The comprehensive processing unit is further used to establish a corresponding relationship between the second voiceprint and the first output position; and to output the content of the third voice to a blank space of the first output position, or to overwrite the content of the first voice. At this time, therefore, when outputting the third voice, the comprehensive processing unit is also used to select an output format different from that when outputting the content of the first voice.

[0044] Optionally, when the second voice received by the voice processing unit is collected by an array microphone, the voice processing unit is further used to calculate the position of the sound source of the second voice. In this case, the comprehensive processing unit is further used to determine whether the position of the sound source of the second voice and the contact point position meet a preset condition before establishing the correspondence between the first voiceprint and the first output position.

[0045] Optionally, the computer device may further include an image processing unit for receiving an image. The image processing unit is further configured to analyze and track the image in real time before the integrated processing unit establishes the corresponding relationship between the first voiceprint and the first output position, and then determine whether the user who performs the touch operation and the user who issues the first voice are the same user based on the analysis and tracking results.

[0046] In another possible design, the computer device includes: a memory and one or more processors; the memory and the processor are coupled. The above-mentioned memory is used to store computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the computer device, the computer device executes the human-computer interaction method as described in the second aspect and any possible design thereof.

[0047] In a fifth aspect, the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on a human-computer interaction system, the human-computer interaction system implements the human-computer interaction method as described in any possible design method provided in the first aspect or the second aspect.

[0048] In a sixth aspect, the present application provides a computer program product, which, when executed on a human-computer interaction system, enables the human-computer interaction system to implement a human-computer interaction method as described in any possible design method provided in the first aspect or the second aspect.

[0049] For the specific description of the second to sixth aspects and their various implementations in the present application, reference may be made to the detailed description in the first aspect and its various implementations; and for the beneficial effects of the second to sixth aspects and their various implementations, reference may be made to the beneficial effects analysis in the first aspect and its various implementations, which will not be repeated here.

[0050] In this application, the name of the human-computer interaction system does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.

[0051] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A hardware structure diagram of the human-computer interaction system provided in an embodiment of the present application.

[0053] Figure 2 A structural diagram of a human-computer interaction system provided in an embodiment of the present application Figure 1 .

[0054] Figure 3 A structural diagram of a human-computer interaction system provided in an embodiment of the present application Figure 2 .

[0055] Figure 4 A flowchart of a human-computer interaction method provided in an embodiment of the present application.

[0056] Figure 5A-Figure 5C A schematic diagram of determining an output position according to a touch operation provided in an embodiment of the present application.

[0057] Fig. 6A and Figure 6B A schematic diagram of a method for calculating the location of a user who utters a first voice provided in an embodiment of the present application.

[0058] Figure 6C A schematic diagram of a method for determining the correspondence between speech and facial features provided in an embodiment of the present application.

[0059] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of the present application.

[0060] Figure 8 A schematic diagram of the structure of a computer program product provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0062] In the embodiments of the present application, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, "plurality" means two or more.

[0063] The embodiments of the present application provide a human-computer interaction method, device and system. The human-computer interaction system acquires a touch operation and a first voice, and when it is determined that the touch operation meets a predetermined first rule and the first voice meets a predetermined second rule, establishes a correspondence between the voiceprint of the first voice and the output position; wherein the output position refers to the output position on the touch screen determined according to the touch operation; then acquires a second voice, and determines whether the voiceprint of the second voice matches the voiceprint of the first voice. If they match, the text content corresponding to the second voice is output and displayed at the output position on the touch screen.

[0064] Optionally, in the above process, before establishing the correspondence between the voiceprint of the first voice and the output position, it is further possible to determine whether the position of the sound source of the first voice and the position of the touch operation meet a preset condition, and only if the determination is yes, the above correspondence is established. The above determination is to further confirm whether the user who performs the touch operation and the user who issues the first voice are the same user, thereby improving the accuracy of establishing the correspondence and ultimately improving the robustness of the system.

[0065] Also optionally, before establishing the correspondence between the voiceprint of the above-mentioned first voice and the output position, a camera can be used to capture images near the front of the touch screen, and real-time analysis and tracking of the captured images can be used to determine whether the user performing the touch operation and the user who issued the first voice are the same user. This process can also improve the accuracy of establishing the corresponding relationship, thereby improving the robustness of the system.

[0066] Through the embodiments of the present application, one or more users can implement touch screen input in the form of voice, thereby improving input efficiency; and the voice content input by each user can be output and displayed at the output position indicated by the user on the touch screen, avoiding the problem of confusing the voice content of different users and displaying them on the touch screen, thereby also improving the input experience.

[0067] The above-mentioned human-computer interaction method may be implemented by an application installed on a device, such as a human-computer interaction application.

[0068] The above application can be an embedded application installed in the device (i.e., a system application of the device) or a downloadable application. Among them, an embedded application is an application provided as part of the implementation of a device (such as a mobile phone). A downloadable application is an application that can provide its own Internet protocol multimedia subsystem (IMS) connection. The downloadable application is an application that can be pre-installed in the device or a third-party application that can be downloaded and installed in the device by the user.

[0069] Figure 1 , is the hardware structure of the human-computer interaction system provided in the embodiment of the present application. Figure 1 As shown, the human-computer interaction system 10 includes a processor 11, a memory 12, a touch screen 13, and a voice collector 14. Optionally, the human-computer interaction system 10 may also include an image collector 15.

[0070] The processor 11 is the control center of the human-computer interaction system 10, and can be a general-purpose central processing unit (CPU) or other general-purpose processors, such as a graphics processing unit (GPU). The general-purpose processor can be a microprocessor or any conventional processor. As an example, the processor 11 can include one or more CPUs, such as Figure 1 0 and CPU 1 shown in FIG. Optionally, the processor 11 may also include one or more GPUs, such as Figure 1 GPU0 and GPU1 are shown.

[0071] The memory 12 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 12 may be independent of the processor 11, or may be connected to the processor 11 via a bus, or may be integrated with the processor 11. The memory 12 is used to store data, instructions or program codes. When the processor 11 calls and executes the instructions or program codes stored in the memory 12, the human-computer interaction method provided in the embodiment of the present application can be implemented.

[0072] The touch screen 13 may specifically include a touch panel 131 and a display screen 132 .

[0073] The touch panel 131 may be implemented by various types such as resistive, capacitive, infrared, and surface acoustic wave. The touch panel 131 is used to collect touch events of the user on or near it (such as the user's operation on or near the touch panel using any suitable object such as a finger, a stylus, etc.), and send the collected touch information to other devices (such as the processor 11). The touch event of the user near the touch panel may be referred to as floating touch; floating touch may mean that the user does not need to directly touch the touch panel in order to select, move, or drag a target (such as an icon, etc.), but only needs the user to be near the device in order to perform the desired function.

[0074] The display screen 132 may be configured in the form of a liquid crystal display screen, an organic light emitting diode, etc. The touch panel 131 may be covered on the display screen 132. When the touch panel 131 detects a touch event on or near it, it is transmitted to the processor 11 to determine the type of the touch event. The processor 11 may provide a corresponding visual output on the display screen 132 according to the type of the touch event. The display screen 132 is used to display information input by a user or information provided to a user.

[0075] The voice collector 14 is also called a "microphone" or a "microphone", etc. It can be a single microphone; or optionally, it can also be a microphone array. The voice collector 14 is used to receive voice signals, convert the voice signals into electrical signals, and then send them to other devices (such as the processor 11) for processing. When the voice collector 14 is a microphone array, it is also used to locate the position of the sound source.

[0076] The image collector 15 may be an imaging device such as CCD or CMOS, also known as a “camera.” The image collector 15 is used to collect images and send the collected image data to other devices (such as the processor 11) for processing.

[0077] The processor 11, memory 12, touch screen 13, voice collector 14 and image collector 15 can be integrated into one device. In this implementation, the human-computer interaction system 10 can be an electronic whiteboard, a smart phone, a laptop with a touch screen, a computer with a touch screen, a tablet, a netbook, a car-mounted terminal device, etc. For example, if it is an electronic whiteboard, refer to Figure 2 As shown, the above human-computer interaction application can be run in the electronic whiteboard 20. Optionally, the human-computer interaction system 10 can also include a touch pen 21, and the touch pen 21 is used to input touch operations on the touch screen 13 of the electronic whiteboard 20.

[0078] In addition, the processor 11, memory 12, touch screen 13, voice collector 14 and image collector 15 may also be integrated on different devices. In this implementation, the human-computer interaction system 10 may include multiple devices to execute the human-computer interaction method provided in the embodiment of the present application. Figure 3 The human-computer interaction system 10 shown may include: an electronic whiteboard 20, a computer 32 and a projector 33. Optionally, the human-computer interaction system 10 may also include a touch pen 21, which is used to input touch operations on the touch screen 13 of the electronic whiteboard 20. Among them, the processor 11 may be a processor of the computer 32. The memory 12 may be a memory of the computer 32. At this time, the above-mentioned human-computer interaction application can be run in the computer 32. In addition, the touch screen 13 may be a touch screen of the electronic whiteboard 20. The voice collector 14 may be integrated in the electronic whiteboard 20. Alternatively, the voice collector 14 may also be integrated in the computer 32, the projector 33 or the touch pen 21, and the implementation of this application is not limited to this. The image collector 15 may be integrated in the electronic whiteboard 20.

[0079] It should be noted that if the above-mentioned voice collector 14 is a microphone array, its integration position and the touch screen need to satisfy a certain relationship; specifically, its integration position can be above or below the touch screen, parallel to the horizontal direction of the touch screen, and its midpoint coincides with the midpoint of the horizontal direction of the touch screen. For example, it can be integrated on the upper edge or lower edge of the touch screen (such as Figure 2 or Figure 3 In addition, the deployment position can also be on the left or right side of the touch screen, parallel to the vertical direction of the touch screen, and the midpoint thereof coincides with the midpoint of the vertical direction of the touch screen.

[0080] Figure 1 The structure shown in the figure does not constitute a limitation on the human-computer interaction system 10. Figure 1 In addition to the components shown, the human-computer interaction system 10 may include more or fewer components than shown, or combine certain components, or arrange the components differently; the above description of the human-computer interaction system 10 is only an exemplary description and does not constitute a limitation of this embodiment.

[0081] The human-computer interaction method provided in the embodiment of the present application is described below with reference to the accompanying drawings.

[0082] Please refer to Figure 4 , Figure 4 A flow chart of a human-computer interaction method provided in an embodiment of the present application is shown. In this embodiment, a scenario in which multiple users discuss in a conference room (an electronic whiteboard is deployed in the conference room) and some users want to record some content on the electronic whiteboard in voice is taken as an example. The human-computer interaction method includes the following but is not limited to the following steps:

[0083] S402: The touch screen receives a touch operation and determines position information of the touch operation.

[0084] When user A performs a touch operation on the touch screen with a finger or a touch pen, the touch screen senses the operation and obtains the position information of the touch operation. The position information of the touch operation may be the position of the touch point of the finger or the touch pen on the touch screen when user A performs the touch operation, for example, the coordinates of the touch point generated by the touch operation of user A on the touch screen.

[0085] The position information of the touch operation can be the position of a single touch point on the touch screen, for example, the touch point position generated by user A touching the screen with a single finger; the position information of the touch operation can also be the positions of multiple touch points on the touch screen, for example, the two touch point positions generated by user A touching the screen with the index fingers of his left hand and right hand at the same time.

[0086] For another example, user A draws a line on the touch screen with a finger or a touch pen. During the touch operation, the touch screen obtains the position information of the touch operation multiple times in succession. Each obtained position information includes a single touch point position, which represents the touch point position generated at different times during the sliding process of user A drawing a line.

[0087] S404: The touch screen sends the position information of the touch operation to the processor.

[0088] Specifically, the touch screen may send the position of the touch point generated by the detected touch operation to the processor periodically, triggered, or in real time. For example, the touch screen may send a frame of data to the processor in each cycle. The frame of data includes the position of the touch point detected in the cycle; it is understandable that when no touch point is detected in the cycle, the position of the touch point will not be included in the sent frame of data.

[0089] Since the touch screen may send the position of a single or multiple touch points within a detection period to the processor at a certain time period; in order to maintain the response sensitivity and real-time performance of the touch operation, the above time period is usually very short (for example, ms level). Therefore, during the continuous process of user A completing a complete touch operation, the touch screen will usually send the position information of the touch operation to the processor multiple times.

[0090] Optionally, in some implementations, when the touch screen sends the position information of the touch operation to the processor, it also carries the touch state information corresponding to each contact point position in the position information, such as "pressed", "lifted" or "moved" state.

[0091] S406: The processor receives the position information of the touch operation, and determines whether the touch operation matches a predetermined first rule.

[0092] The processor receives the position information of the touch operation in real time.

[0093] The processor usually receives the location information of the touch operation multiple times during the process of user A completing a complete touch operation. Therefore, when the processor receives the location information of the touch operation, it will first determine whether the touch operation of a certain user (user A is taken as an example here) is completed. Specifically, the processor can determine whether the touch operation is completed by tracking the touch operation. For example, the processor starts timing each time it receives the location information of the touch operation. If the location information of the touch operation is not received again within a predetermined time interval, it is determined that the current touch operation is completed; and from the moment when the current touch operation is determined to be completed, the location information of the touch operation received subsequently is used as the location information of the next touch operation. If the location information of the touch operation is received again within a predetermined time interval, it is considered that the current touch operation has not been completed, and the location information received again is used as the location information of the current touch operation. Then, the timing starts from the moment when the location information of the touch operation is received again, and the judgment continues according to the above process; until after receiving the location information of a certain touch operation, the location information of the touch operation is not received again within a predetermined time interval, then it is considered that the touch operation is completed. Among them, the predetermined time interval is set according to the experience value or the actual situation, and is not limited in this application. The processor may also determine whether the touch operation is finished according to the touch state information carried in the position information of the touch operation. For example, when the touch state is "pressed" or "moved", the touch operation is judged to be unfinished; when the touch state is "lifted", the touch operation is judged to be completed.

[0094] The above method for the processor to determine whether the touch operation is completed is only an exemplary method. Depending on the different driver implementations of different touch screen manufacturers, the method for the processor to determine whether the touch operation is completed may also be different, and this embodiment of the application does not limit this.

[0095] When the processor determines that the touch operation is completed, it then determines whether the touch operation matches the predetermined first rule. Among them, the first rule is an arbitrary rule about touch operation that is predefined or configured, and the rule is used to determine whether user A wants to start voice input. Exemplarily, the predetermined first rule can be "any point on the touch screen"; it can also be "two points on the touch screen, and the line connecting the two points is parallel to the horizontal direction of the touch screen, and the distance between the two points is not less than the preset value M"; it can also be "a touch track, and the touch track is parallel to the horizontal direction of the touch screen, and the length is not less than the preset value". The processor can determine whether one or more contact point position sets generated by the received touch operation meet the conditions defined by the above-mentioned first rule. If so, it can be considered that the touch operation matches the predetermined first rule.

[0096] In addition, the predetermined first rule may also be an image containing a certain shape. In this implementation, the processor may use image matching technology to compare the shape formed by the position information of the touch operation received during the current touch operation with the predetermined shape. If the comparison results are consistent, it can be considered that the touch operation matches the predetermined first rule.

[0097] The content of the predetermined first rule and the method of determining whether the touch operation matches the predetermined first rule may be other content and methods other than the above examples, and this application does not limit them.

[0098] In this step, if the processor determines that the touch operation matches the predetermined first rule, it preliminarily determines that user A wants to start voice input and continues to execute subsequent steps in this method; conversely, if the processor determines that user A's touch operation does not match the predetermined first rule, it is considered that user A is performing other touch operations and the process ends.

[0099] S408: The processor determines an output position on the touch screen according to the position information of the touch operation.

[0100] As described in S406, when the processor determines that the touch operation meets the predetermined first rule, it can determine that user A wants to start voice input, and can also determine the output position on the touch screen indicated by user A from the position information of the touch operation. The output position is used to indicate the starting position and range of the voice content output on the touch screen, that is, the position and size of the output box displaying the voice content. The following takes the case where the shape of the output box is a rectangle as an example to explain the content contained in the output position and the method of determining it.

[0101] In the first implementation, the output position includes the starting position of the output box, and the width and height of the output box. If the output order from left to right and from top to bottom is taken as an example, the starting position of the output box can be the coordinates of the upper left corner of the output box. The output position can be determined by taking the point in the upper left corner of the touch screen as the origin, the side with the origin as the endpoint as the horizontal axis, and the other side with the origin as the endpoint as the vertical axis. From the set of contacts contained in the received position information of the touch operation, select the contact corresponding to the minimum horizontal coordinate or the contact corresponding to the minimum vertical coordinate as the first contact, and select the contact corresponding to the maximum horizontal coordinate or the contact corresponding to the maximum vertical coordinate as the second contact. For example Figure 5A-Figure 5BIn the touch point set shown in FIG. 1 , the processor can use touch point A or B as the first touch point and touch point C or D as the second touch point. Then the first touch point is used as the starting position of the output box, and the horizontal coordinate difference between the second touch point and the first touch point is used as the width of the output box, and the vertical coordinate difference between the second touch point and the first touch point is used as the height of the output box. When touch point A is used as the first touch point and touch point C is used as the second touch point, the starting position of the output box, as well as its width and height are as follows: Figure 5A When contact point A is the first contact point and contact point D is the second contact point, the starting position of the output box, as well as its width and height are as shown in Figure 5B When contact B is used as the first contact and contact C or D is used as the second contact, the same logic can be applied.

[0102] The output position can also be determined by taking the point at the upper left corner of the touch screen as the origin, and taking the point consisting of the smallest horizontal coordinate and the smallest vertical coordinate (which may not be the actual touch point) as the first touch point, and the point consisting of the largest horizontal coordinate and the largest vertical coordinate as the second touch point (which may not be the actual touch point). Then, the starting position of the output box, as well as its width and height, can be determined in a similar manner. For example, Figure 5A-Figure 5C In the set of touch points shown in FIG. 1 , the point (xa, yb) formed by the horizontal coordinate xa of touch point A and the vertical coordinate yb of touch point B is taken as the first touch point. The point (xd, yc) formed by the horizontal coordinate xd of touch point D and the vertical coordinate yc of touch point C is taken as the second touch point. At this time, the starting position of the output box, as well as its width and height are as follows: Figure 5C shown.

[0103] In the second implementation, the output position includes the starting position and the ending position of the output box. If the output order from left to right and from top to bottom is still taken as an example, the starting and ending positions of the output box are the coordinates of the upper left corner and the lower right corner of the output box, respectively. Similar to the introduction in the first implementation, the processor can determine the coordinates of the upper left corner and the lower right corner of the output box from the set of contacts contained in the position information of the received touch operation. For example, the coordinates of the first contact and the second contact determined in the first implementation can be used as the coordinates of the upper left corner and the lower right corner of the output box, respectively, to determine the starting position and the ending position of the output box. The specific process will not be repeated here.

[0104] Optionally, the system may define the height of the output box as a preset value. In this implementation, this step only needs to determine the starting position and width of the output box. Alternatively, the system may define both the height and width of the output box as preset values. In this implementation, this step only needs to determine the starting position of the output box.

[0105] The content and determination method of the above output position are only exemplary descriptions; in some implementations, the system may also define the shape of the output box for displaying the voice content as other shapes other than a rectangle, such as a diamond, a circle, etc. Accordingly, the content and determination method of the output position may be other implementations other than the first and second implementations described above, and the embodiments of the present application do not limit this.

[0106] Optionally, the processor may also call a rendering instruction to generate the output position according to the determined output position (i.e., the position and size of the output box), so that the output position can be displayed on the display screen. For example, the output position may be generated and output in the form of a border, and the border of the output box may be as follows: Figure 5A-Figure 5C The black dashed frame shown may also be a dashed frame of other colors, or a black solid frame or a solid frame of other colors. The processor may display the output position by changing the background color of the output position, or by any method that can distinguish the output position from the background currently displayed on the display screen. This embodiment of the application is not limited to this.

[0107] S410-S412, the voice collector collects and sends the first voice to the processor.

[0108] Specifically, the voice collector collects the first voice signal in real time, and sends the collected first voice to the processor.

[0109] It can be understood that, since the voice signal collected by the voice collector comes from the venue, the first voice may be sent by user A in the venue, or may be sent by other users in the venue.

[0110] S414: The processor receives the first speech and determines whether the first speech matches a predetermined second rule.

[0111] In this embodiment, the second rule is used to determine whether the user who utters the first voice, such as user A, wants to activate voice input.

[0112] The predetermined second rule may be a text of specific content, such as “start voice input”.

[0113] The method for determining whether the first voice matches the predetermined second rule may be to first convert the received first voice signal into text using voice recognition technology, and then use a text matching method to determine whether the content of the received first voice is consistent with the above-mentioned specific content. If consistent, it is determined that the first voice matches the predetermined second rule; otherwise, it can be determined as not matching. It should be understood that the text matching method adopted by the processor may not only be a literal match, but also a semantic match. For example, if the content of the first voice received by the processor is "start voice input", the processor determines that the content of the voice matches the above-mentioned specific content "start voice input" at the semantic level, and thus determines that the first voice matches the predetermined second rule.

[0114] The content of the above-mentioned predetermined second rule and the method of determining whether the first voice matches the predetermined second rule may be other content and methods other than the above-mentioned examples, and are not limited in the embodiments of the present application.

[0115] In this step, if the processor determines that the first voice matches the predetermined second rule, and in combination with the above step S406, the processor has received a touch operation that meets the first rule, then it is considered that the first voice is issued by user A, that is, it is considered that the touch operation received in step S406 and the first voice received in this step are respectively performed and issued by the same user, and it is confirmed that user A wants to start voice input, and then the subsequent steps of this method are continued. If the processor determines that the first voice does not match the predetermined second rule, then it is considered that the first voice is not issued by user A, that is, it is considered that the touch operation received in step S406 and the first voice received in this step are not performed and issued by the same user, for example, it may be a voice randomly issued by other users in the venue. At this time, the processor confirms that it is not user A who wants to start voice input, and will not execute the subsequent steps.

[0116] S416: The processor establishes a corresponding relationship between the voiceprint of the first voice and the output position.

[0117] Specifically, the processor performs denoising and other processing on the first voice signal, extracts voiceprint features of the first voice from the processed signal, such as acoustic or language features, and establishes a corresponding relationship between the voiceprint of the first voice, i.e., the voiceprint feature of user A, and the output position determined in S408, i.e., the output position indicated by the touch operation of user A. The specific process of establishing the corresponding relationship may include: using the voiceprint feature data of user A as a key and the output position on the touch screen determined in S405 (assuming that the output position is (x1, y1, w1, h1)) as a value, and saving the voiceprint library shown in Table 1 in a dictionary or hash table manner, so that in subsequent steps, the output position of the touch screen corresponding to user A can be determined by the voiceprint of user A.

[0118] Table 1

[0119] user Voiceprint(key) Output position (value) A Voiceprint feature A (x1,y1,w1,h1)

[0120] S418-S420, the voice collector collects the second voice and sends the second voice to the processor.

[0121] During the conference, the voice collection device will collect the voices of different people. Therefore, the second voice may be sent by user A or other users.

[0122] S422: The processor receives a second voice, and determines whether a voiceprint of the second voice matches a voiceprint of the first voice.

[0123] As described above, since the second voice may be sent by any user, in some scenarios, user A does not want to be disturbed by other users' voices when outputting the voice to the output location. Therefore, when the processor receives the second voice, it will determine whether the second voice is sent by user A.

[0124] Specifically, the processor can process the second voice in a manner similar to that of the first voice in S414, perform denoising and other processing on the second voice, and then extract its voiceprint features; then use voiceprint recognition technology to determine whether the voiceprint of the second voice matches the voiceprint of user A in the voiceprint library shown in Table 1, so as to determine whether the second voice is issued by user A.

[0125] If the judgment result of this step is yes, it means it is the voice of user A, and then step S424 is executed. If the judgment result is no, it means it is not the voice of user A, then the voice signal can be discarded, and the process of collecting and judging the voice signal is re-executed in step S418.

[0126] S424-S426, the processor recognizes the content of the second voice, and outputs and displays the content at the output position on the touch screen.

[0127] The processor can use voice recognition technology to convert the second voice signal into text content, and then output and display the text content to the output position on the touch screen; wherein the output position refers to the output position corresponding to the voiceprint of the first voice established in S416.

[0128] The process of the processor outputting the second voice content can be: according to the output position information and the text content, calling the rendering instruction to generate a text window containing the second voice content, and then merging the generated text window with other windows (such as the system background window) according to the output position information to generate a pixel array, and then outputting the pixel array to a predetermined storage area (such as the system's frame buffer). In this way, when the system sends a display signal (such as a vertical synchronization signal), the content of the predetermined storage area can be displayed on the touch screen. The above-mentioned way in which the processor outputs the second voice content is only an exemplary description. Depending on the actual display mechanism of the operating system run by the processor, the way the processor outputs the second voice content may also be different, and the embodiments of the present application are not limited to this.

[0129] After executing this step, the embodiment of the present application can continue to collect the third voice, and adopt the same processing and judgment process for the third voice as the second voice; in this way, after user A successfully starts the voice input function, the content can be continuously output to the output box indicated by user A through voice input (the output position information indicated by the above-mentioned touch operation).

[0130] It is understandable that if the voice content actually input by user A exceeds the range of the output box indicated by the output position information, a scroll bar in the horizontal or vertical direction of the output box can be generated according to the situation to ensure that the voice content input by user A is recorded, and all the recorded content can be displayed through the scroll bar. In this case, the processor can output the most recently input content at the output position by default. When it is detected that the user is operating the scroll bar, the corresponding part of the voice text content is output according to the position of the operated scroll bar, so that the user can flip through the content he wants to see.

[0131] In addition, in practical applications, the voice collector may use an array microphone. Since the array microphone can locate the sound source, in another implementation, the following step 415 may be included between the above steps S414 and S416.

[0132] S415. The processor determines whether the location of the sound source of the first voice and the location information of the touch operation meet a preset condition.

[0133] The position of the user performing the touch operation is usually near the position of the touch operation. Therefore, the processor can verify whether the user who issued the first voice is user A by determining whether the position of the sound source to which the first voice belongs and the position information of the touch operation meet the preset conditions (that is, further verify whether the user performing the touch operation received in step S406 and the user who issued the first voice received in step S414 are the same user).

[0134] Specifically, the processor receives the first speech collected by the array microphone, calculates the sound source position of the first speech according to the first speech, and obtains the position of the sound source relative to the array microphone. Figure 2 or Figure 3 ), usually the array microphone is integrated on the upper edge or the lower edge of the touch screen (that is, the integration position of the array microphone satisfies the requirement of being parallel to the horizontal direction of the touch screen, and its midpoint coincides with the midpoint of the horizontal direction of the touch screen). Therefore, the position of the sound source relative to the array microphone can also be used as the position of the sound source relative to the touch screen; the position includes the vertical distance of the sound source relative to the touch screen, and the horizontal distance relative to the midpoint of the horizontal direction of the touch screen.

[0135] Assuming that the array microphone for collecting the first speech is an array microphone including n microphones (MC1 to MCn), the process of calculating the position of the sound source using the array microphone is as follows:

[0136] refer to Fig. 6A , MC1 and MC2 are used as the sub-microphone array at the left end of the microphone array, and C1 is the midpoint of the sub-microphone array at the left end; similarly, MC3 and MC4 are used as the sub-microphone array at the right end, and C2 is the midpoint of the sub-microphone array at the right end.

[0137] First, using the time difference between the sound source reaching MC1 and MC2 (i.e., the time difference between MC1 and MC2 collecting speech signals) and the distance between MC1 and MC2, the angle between the line connecting the sound source and C1 and the array microphone is calculated to be α1; similarly, the angle between the line connecting the sound source and C2 and the array microphone can be calculated to be α2.

[0138] Further, refer to Figure 6B Since the distance between C1 and C2 is known, the vertical distance H of the sound source relative to the array microphone (i.e., the vertical distance relative to the touch screen) and the horizontal distance W relative to the midpoint of the array microphone (i.e., the horizontal distance relative to the midpoint of the touch screen in the horizontal direction) can be calculated based on the trigonometric function relationship.

[0139] It is understandable that the vertical distance of the position of the touch operation relative to the touch screen is 0, and the horizontal distance relative to the midpoint of the horizontal direction of the touch screen can be obtained according to the horizontal coordinate in the position information of the touch operation; therefore, the vertical distance between the sound source and the position of the touch operation (i.e., the vertical distance of the sound source relative to the touch screen) and the horizontal distance between the sound source and the position of the touch operation can be obtained with the touch screen as a reference. It is then determined that if the vertical distance between the sound source and the position of the touch operation does not exceed a predetermined range (e.g., set to 0.5m based on an empirical value), and the horizontal distance between the sound source and the position of the touch operation does not exceed a predetermined range (e.g., set to 0.5m based on an empirical value), then it is considered that the user who issued the first voice is user A (i.e., when the result of this step is yes, it is considered that the user who performs the touch operation and the user who issues the first voice are the same user); otherwise, it is considered that the user who issued the first voice is not user A (i.e., when the result of this step is no, it is considered that the user who performs the touch operation and the user who issues the first voice are not the same user).

[0140] Optionally, the above-mentioned method of determining that the user who performs the touch operation and the user who utters the first voice are not the same user can also be implemented by collecting and tracking the image in front of the touch screen. Assume that in the embodiment of the present application, the camera is deployed on the touch screen, for example, at the center of the upper edge of the touch screen (for example Figure 2 or Figure 3 The position shown in the figure) can capture images near the front of the touch screen and send them to the processor in real time; the processor analyzes and tracks the images captured and sent by the camera in real time, such as analyzing and tracking the body movements and lip movement information of the person in the image, so as to determine whether the user performing the touch operation is the same user as the user who uttered the first voice.

[0141] In addition, it is also possible to determine whether the user performing the touch operation and the user emitting the first voice are the same user by combining the position calculation of the sound source of the first voice and image tracking. In this case, assuming that the camera is deployed at the midpoint of the array microphone, such as Figure 6CAs shown. At this time, the processor can identify the facial features of the user performing the touch operation through a machine learning algorithm, such as the facial feature Face-A of user A, and bind the Face-A to the position indicated by the touch operation of user A, such as (x1, y1, w1, h1). Then calculate the angle a1-A of the facial feature of user A, i.e. Face-A, relative to the camera. The angle a2-A of the sound source of the first voice relative to the midpoint of the array microphone (i.e., the camera) can be obtained through the above sound source position calculation. Then, by comparing the above a1-A and a2-A, if the difference between the two is within a predetermined range, it is considered that the first voice and Face-A are features belonging to the same user, so it can be determined that the user performing the above touch operation and the user emitting the first voice are the same user, and thus the voiceprint of the first voice can be matched with (x1, y1, w1, h1).

[0142] When another user at the venue starts voice input at the same time as user A, the above method can avoid establishing a correspondence between the voiceprint of user A and the output position indicated by the other user, thereby improving the robustness of the system.

[0143] It should also be noted that the detection and judgment of user A's touch operation in the above steps S402-S406, and the collection and judgment of user A's voice signal in steps S410-S414, are two conditions for determining whether user A wants to start voice input. There is no order of priority for determining the two conditions, that is, there is no existing order for the execution of steps S402-S406 and steps S410-S414. The detection and judgment of the touch operation can be performed first, or the collection and judgment of the voice signal can be performed first.

[0144] As can be seen from the above description, in the embodiment of the present application, when the terminal device determines that user A wants to start voice input based on the touch operation and voice input of user A, the voiceprint of user A is matched with the text output position indicated by user A. In this way, when there are multiple people speaking in the venue (including user A), only the voice content of user A, rather than the voice content of other users in the venue, can be displayed at the text output position indicated by the above user A (rather than other positions on the touch screen) through voiceprint matching.

[0145] In actual applications, there may be a scenario where user A wants to change the output position of the voice input after starting the voice input at the specified output position. For example, if user A wants to change the output position from (x1, y1, w1, h1) to (x3, y3, w3, h3), the present application can complete the binding of user A's voiceprint with (x3, y3, w3, h3) through a similar process, thereby outputting user A's voice content to the output box indicated by (x3, y3, w3, h3). In this scenario, the voiceprint library saved by the system will be updated from the content shown in Table 1 above to the content shown in Table 2 shown below. Specifically, the system can implement this scenario in the following two possible ways:

[0146] The first possible implementation method may be: user A may instruct the system to cancel the correspondence between user A's voiceprint and the output position (x1, y1, w1, h1) by emitting a voice that meets the third rule (for example, the voice content matches "turn off voice input"). Specifically, when the processor detects a voice that meets the third rule, it extracts the voiceprint of the voice, and compares the voiceprint with the voiceprints in the voiceprint library saved by the system one by one, and then deletes the voiceprint that can match the voiceprint and its corresponding output position from the voiceprint library. Then, user A instructs the system to start voice input again through a similar process as described above (performing a touch operation that meets the first rule and emitting a voice that meets the second rule) to establish a correspondence between the output position (x3, y3, w3, h3) newly specified by his voiceprint.

[0147] The second possible implementation method may be: user A directly instructs the system to restart voice input through a similar process as described above, and establishes a corresponding relationship between his voiceprint and the newly specified output position (x3, y3, w3, h3). In this implementation method, when the processor receives a voice that meets the second rule, it determines that the voiceprint of the voice already exists in the voiceprint table saved by the system, and directly updates the output position corresponding to the voiceprint, thereby updating the output position corresponding to the voiceprint of user A to (x3, y3, w3, h3).

[0148] Table 2

[0149] user Voiceprint(key) Output position (value) A Voiceprint feature A (x3,y3,w3,h3)

[0150] It is understandable that, as mentioned above, there are usually multiple people discussing at the venue. Therefore, not only user A operates on the touch screen, but other users, such as user B, can also operate on the same touch screen while user A starts and executes voice input. Based on the same implementation, the present application can also realize the simultaneous display of user B's voice output on the output position indicated by user B on the touch screen. Specifically, the processor will add the corresponding relationship between the voiceprint of user B and the output position indicated by user B on the basis of the above Table 2. At this time, the voiceprint library saved by the system is as shown in Table 3.

[0151] Table 3

[0152] user Voiceprint(key) Output position (value) A Voiceprint feature A (x3,y3,w3,h3) B Voiceprint feature B (x2,y2,w2,h2)

[0153] It should be noted that when user A and user B start voice input at the same time, the above step S415 is needed to further verify whether the user performing the touch operation and the user emitting the voice are the same user, so as to avoid establishing a correspondence between the voiceprint of user A and the output position indicated by user B, thereby improving the robustness of the system. In this case, in S422, the processor can compare the voiceprint of the second voice with the voiceprints in the voiceprint library shown in Table 3 one by one. If it can match one of the voiceprints, the judgment result of S422 is considered to be yes. On the contrary, if the second voice cannot match any of the voiceprints in the voiceprint library stored in the system, the judgment result of S422 is considered to be no, and the second voice is not output.

[0154] Therefore, the embodiment of the present application identifies the user identity corresponding to the second voice through voiceprint matching, which can avoid the problem of different users' voice contents being mixed and displayed on the touch screen when multiple people use voice input at the same time because the identities of the users who make the voices cannot be distinguished.

[0155] In addition, in actual applications, user A has released the binding relationship with a specific output position, and user C may want to modify or supplement the content at the output position historically specified by user A, such as the content in the output position (x1, y1, w1, h1). At this time, the system establishes a corresponding relationship between user C's voiceprint and the output position (x1, y1, w1, h1) based on the same implementation, and appends the voice content of user C to the output box (x1, y1, w1, h1) after the existing content. Or the system establishes a corresponding relationship between user C's voiceprint and the output position (x1', y1', w1', h1') based on the same implementation, and outputs the voice content of user C in the output box (x1', y1', w1', h1'); wherein (x1', y1', w1', h1') is an output box superimposed on (x1, y1, w1, h1). Therefore, in the first possible scenario, the content of the voiceprint library saved by the system may be as shown in Table 4A-1 or Table 4A-2.

[0156] Table 4A-1

[0157] user Voiceprint(key) Output position (value) A Voiceprint feature A (x3,y3,w3,h3) B Voiceprint feature B (x2,y2,w2,h2) C Voiceprint feature C (x1,y1,w1,h1)

[0158] Table 4A-2

[0159] user Voiceprint(key) Output position (value) A Voiceprint feature A (x3,y3,w3,h3) B Voiceprint feature B (x2,y2,w2,h2) C Voiceprint feature C (x1',y1',w1',h1')

[0160] In addition, there may be a scenario where user C modifies or supplements the content at the current output position of user A, such as (x3, y3, w3, h3). Similarly, the system can establish a corresponding relationship between user C and the output position (x3, y3, w3, h3). Since the voiceprints of user A and user C correspond to the same output position, user A and user C can input voice content in (x3, y3, w3, h3) at the same time. At this time, the system can output the voices of user A and user C to (x3, y3, w3, h3) in sequence according to the order of the acquisition time of the voices of user A and user C. In addition, the system can also implement a scenario where user C supplements or modifies the content input by user A by establishing a corresponding relationship between user C and the output position (x3', y3', w3', h3') superimposed on (x3, y3, w3, h3). This process is similar to the implementation description in the first possible scenario above, so it will not be repeated. Therefore, in the second possible scenario, the content of the voiceprint library saved by the system may be as shown in Table 4B-1 or Table 4B-2.

[0161] Table 4B-1

[0162]

[0163]

[0164] Table 4B-2

[0165] user Voiceprint(key) Output position (value) A Voiceprint feature A (x3,y3,w3,h3) B Voiceprint feature B (x2,y2,w2,h2) C Voiceprint feature C (x3',y3',w3',h3')

[0166] Optionally, in the above two possible scenarios, the system may distinguish and display the voice content input by user A and user C. Specifically, the content supplemented or superimposed by user C may be displayed on the touch screen in a format different from the voice content of user A, for example, the voice content of user C may be displayed in a different color, font or other format from that of user A's voice text. The present application does not limit the way in which the system displays the voice content of different users in different formats.

[0167] It can be seen that the embodiments of the present application can also allow multiple users to input content on the touch screen through voice in a collaborative manner, thereby improving the input efficiency and experience.

[0168] The various implementation scenarios in the venue described above do not constitute a limitation of the embodiments of the present application. It is understandable that the embodiments of the present application can also be applied to other scenarios in which touch screen input is achieved through voice, such as in educational scenarios, teachers and students independently or collaboratively input content on an interactive electronic blackboard by voice, or in family scenarios, for example, family members independently or collaboratively input content on a tablet, a touch-enabled computer display, or an electronic whiteboard by voice.

[0169] The embodiment of the present application can divide the human-computer interaction system according to the above method example. Figure 1 , Figure 1 The human-computer interaction system shown can be used to perform a human-computer interaction method, for example, to perform Figure 4 The human-computer interaction system comprises: a touch screen 13, a processor 11, and a memory 12.

[0170] The touch screen 13 is used to receive a touch operation and send position information of the touch operation to the processor 11 .

[0171] The processor 11 is configured to establish a correspondence between a first voiceprint and a first output position on the touch screen 13 .

[0172] The processor 11 is further configured to receive a first voice, and when determining that the voiceprint of the voice matches the first voiceprint, identify the content of the voice, and output and display the content of the voice at the first output position.

[0173] The memory 12 is used to store data, instructions or program codes. When the processor 11 calls and executes the instructions or program codes stored in the memory 12, the human-computer interaction method provided in the embodiment of the present application can be implemented.

[0174] In the above process, when the processor 11 establishes the correspondence between the first voiceprint and the first output position on the touch screen 13, it is specifically used to: receive the position information of the touch operation, and receive the second voice; and when it is determined that the touch operation matches the predetermined first rule and the second voice matches the predetermined second rule, determine the first output position on the touch screen 13 according to the position of the touch operation, extract the first voiceprint from the second voice, and then establish the correspondence between the first voiceprint and the first output position on the touch screen 13. For example, in combination with Figure 4 , the touch screen 13 can be used to execute S402 and S404, and the processor 11 can be used to execute S406, S408, S414, S416, S422, S424 and S426. In the above process, when the processor 11 determines the first output position on the touch screen 13 according to the position of the touch operation, it is specifically used to: determine the starting position and range of the first output position on the touch screen 13 according to the set of contact point positions in the position of the touch operation. Optionally, the processor 11 is also used to output and display the first output position on the touch screen 13.

[0175] Optionally, the processor 11 is further configured to cancel the correspondence between the first voiceprint and the first output position on the touch screen 13 .

[0176] Optionally, the processor 11 is further configured to establish a correspondence between the second voiceprint and the first output position on the touch screen 13. At this time, when the processor 11 receives the third voice and determines that the voiceprint of the third voice matches the second voiceprint, it identifies the content of the third voice, and outputs and displays the content to a blank space of the first output position of the touch screen 13 or covers the content of the first voice. Optionally, in this case, since the processor 11 is also configured to select an output format different from that of the first voice when outputting the content of the third voice.

[0177] Optionally, when the second voice received by the processor 11 is collected by an array microphone, it is also used to calculate the position of the sound source of the second voice, and before establishing the correspondence between the first voiceprint and the first output position, it is also used to determine whether the sound source position of the second voice and the position of the touch operation meet a preset condition. Figure 4 , processor 11 can be used to execute S415.

[0178] Optionally, the processor 11 is also used to receive an image near the front of the touch screen 13, and after determining that the touch operation matches a predetermined first rule and the first voice matches a predetermined second rule, it is also used to analyze and track the image in real time, and determine whether the user performing the touch operation and the user emitting the first voice are the same user based on the results of the image analysis and tracking.

[0179] Optionally, the touch screen 13 is also used to display text content, output position or scroll bar according to instructions of the processor 11.

[0180] Optionally, the voice collector 14 can be used to collect voice and send the collected voice to the processor 11. Figure 4 The voice collector 14 can be used to execute S410, S412, S418 and S420.

[0181] Optionally, the image collector 15 may be used to collect images in front of the touch screen 13 and send the images to the processor 11 in real time.

[0182] For the detailed description of the above optional methods, please refer to the above method embodiments, which will not be repeated here. In addition, the explanation of any human-computer interaction system provided above and the description of the beneficial effects can refer to the above corresponding method embodiments, which will not be repeated here.

[0183] In addition, the embodiment of the present application can divide the above-mentioned processor, or a computer device including a processor, into functional modules according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0184] like Figure 7 FIG. 1 is a schematic diagram of a processor or computer device provided in an embodiment of the present application. The processor or computer device is used to execute the above-mentioned human-computer interaction method, for example, to execute Figure 4 The method shown in the figure. The processor or computer device may include a speech processing unit 702 and a comprehensive processing unit 704.

[0185] The speech processing unit 702 is used to receive the first speech, and recognize the content of the second speech when it is determined that the voiceprint of the first speech matches the first voiceprint.

[0186] The integrated processing unit 704 establishes a correspondence between the first voiceprint and the first output position; and is also used to output the content of the first voice to the first output position.

[0187] The processor or computing device further comprises a touch position processing unit 701 for receiving a touch position generated by a touch operation and for determining a first output position according to the touch position when determining that the touch position matches a predetermined first rule.

[0188] The speech processing unit 702 is further configured to receive a second speech, and extract a first voiceprint from the first speech when it is determined that the second speech matches a predetermined second rule.

[0189] Combination Figure 4 The contact position processing unit 701 can be used to execute S406 and S408, the speech processing unit 702 can be used to execute S414, S422 and S424, and the integrated processing unit 704 can be used to execute S416 and S426.

[0190] In the above process, when the contact position processing unit 701 determines the first output position according to the contact position, it is specifically used to: determine the starting position and range of the first output position according to the contact position set in the contact position. Optionally, the comprehensive processing unit 705 is also used to output the first output position.

[0191] Optionally, the comprehensive processing unit 704 is further configured to cancel the correspondence between the first voiceprint and the first output position.

[0192] Optionally, the voice processing unit 702 is further configured to receive a third voice, and when it is determined that the voiceprint of the third voice matches the second voiceprint, identify the content of the third voice. The comprehensive processing unit 704 is further configured to establish a corresponding relationship between the second voiceprint and the first output position, and output the content of the third voice to the first output position, for example, to a blank space of the first output position or to cover the content of the first voice.

[0193] Optionally, when the second voice received by the voice processing unit 702 is collected by an array microphone, it is also used to calculate the position of the sound source of the second voice. In this case, the comprehensive processing unit 704 is also used to determine whether the position of the sound source of the second voice and the contact point position meet a preset condition before establishing the corresponding relationship between the identifier of the first voiceprint and the first output position. Figure 4 , the integrated processing unit 704 can be used to execute S415.

[0194] Optionally, the processor or computer device may further include an image processing unit 703 for receiving an image, and analyzing and tracking the image in real time before the integrated processing unit 704 establishes a correspondence between the identifier of the first voiceprint and the first output position, and then determining whether the user who performs the touch operation and the user who issues the first voice are the same user based on the analysis and tracking results.

[0195] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0196] As an example, combining Figure 1 The functions implemented by the touch position processing unit 701, the voice processing unit 7022, the comprehensive processing unit 704 and the image processing unit 703 in the computer device are similar to those Figure 1 The function of processor 11 in is the same.

[0197] For the detailed description of the above optional methods, please refer to the above method embodiments, which will not be repeated here. In addition, the explanation of any processor or computer device provided above and the description of the beneficial effects can refer to the above corresponding method embodiments, which will not be repeated here.

[0198] Another embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a human-computer interaction system or a computer device, the human-computer interaction system or the computer device executes each step executed by the human-computer interaction system or the computer device in the method flow shown in the above method embodiment.

[0199] In some embodiments, the disclosed methods may be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of manufacture.

[0200] Figure 8 A conceptual partial view of a computer program product provided by an embodiment of the present application is schematically shown, wherein the computer program product includes a computer program for executing a computer process on a computing device.

[0201] In one embodiment, the computer program product is provided using a signal bearing medium 90. The signal bearing medium 90 may include one or more program instructions which, when executed by one or more processors, may provide the above-described Figure 4 Thus, for example, reference to Figure 4One or more features of S402 to S426 may be undertaken by one or more instructions associated with the signal bearing medium 90. In addition, Figure 8 The program instructions in also describe example instructions.

[0202] In some examples, the signal bearing medium 90 may include a computer readable medium 91 such as, but not limited to, a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, a memory, a read-only memory (ROM) or a random access memory (RAM), and the like.

[0203] In some implementations, the signal bearing medium 90 may include a computer recordable medium 92 such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, or the like.

[0204] In some embodiments, signal bearing medium 90 may include communication medium 93 such as, but not limited to, digital and / or analog communication media (eg, fiber optic cables, waveguides, wired communication links, wireless communication links, etc.).

[0205] The signal bearing medium 90 may be communicated by a wireless form of communication medium 93 (eg, a wireless communication medium conforming to the IEEE 802.11 standard or other transmission protocol). The one or more program instructions may be, for example, computer executable instructions or logic implemented instructions.

[0206] In some examples, such as for Figure 4 The described human-computer interaction system or computer device may be configured to provide various operations, functions, or actions in response to one or more program instructions in computer-readable media 91 , computer-recordable media 92 , and / or communication media 93 .

[0207] It should be understood that the arrangement described here is only for illustrative purposes. Thus, it will be appreciated by those skilled in the art that other arrangements and other elements (e.g., machines, interfaces, functions, sequences, and functional groups, etc.) can be used instead, and some elements can be omitted together according to the desired result. In addition, many of the described elements can be implemented as discrete or distributed components, or in any appropriate combination and position to combine other components to implement functional entities.

[0208] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer execution instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more servers that can be integrated with a medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0209] The above is only a specific implementation of the present application. Those skilled in the art may conceive of changes or substitutions based on the specific implementation provided by the present application, which should all be included in the protection scope of the present application.

Claims

1. A method for human-computer interaction, characterized in that: Applied to a human-computer interaction system, the method comprises: Receiving touch operation and second voice; When it is determined that the touch operation matches a predetermined first rule, and the second voice matches a predetermined second rule; the first rule is used to determine the start of voice input according to the touch operation, and the second rule is used to determine the start of voice input according to the second voice; Determining a first output position on the touch screen according to the position of the touch operation; extracting a first voiceprint from the second voice; Establishing a correspondence between a first voiceprint and a first output position on the touch screen; A first voice is received, and when it is determined that the voiceprint of the first voice matches the first voiceprint, the content of the first voice is recognized, and the content of the first voice is output and displayed to the first output location.

2. The method according to claim 1, characterized in that The step of determining a first output position on the touch screen according to the position of the touch operation includes: The starting position and range of the first output position on the touch screen are determined according to a set of contact point positions in the positions of the touch operation.

3. The method according to claim 1 or 2, characterized in that include: The area indicated by the first output position is output and displayed on the touch screen in a manner distinguishable from a current background on the touch screen.

4. The method according to claim 1, characterized in that Before establishing the correspondence between the first voiceprint and the first output position on the touch screen, the method further includes: The position of the sound source of the second voice is calculated, and it is determined whether the position and the position of the touch operation meet a preset condition.

5. The method according to any one of claims 1, 2 and 4, characterized in that: Also includes: The correspondence between the first voiceprint and the first output position on the touch screen is cancelled.

6. The method according to claim 5, characterized in that Also includes: Establishing a correspondence between the second voiceprint and the first output position on the touch screen; A third voice is received, and when it is determined that the voiceprint of the third voice matches the second voiceprint, the content of the third voice is recognized, and the content of the third voice is output and displayed in a blank space of the first output position.

7. The method according to claim 5, characterized in that Also includes: Establishing a correspondence between the second voiceprint and the first output position on the touch screen; A third voice is received, and when it is determined that the voiceprint of the third voice matches the second voiceprint, the content of the third voice is recognized, and the content of the third voice is output and displayed to the first output location, covering the content of the first voice.

8. The method according to claim 6 or 7, characterized in that The step of outputting and displaying the content of the third voice to the first output position on the touch screen includes: The content of the third voice is output and displayed in a format different from the format in which the content of the first voice is output and displayed.

9. A method of human-computer interaction, characterized in that: Applied to a computer device, the method comprises: Receiving a touch point position and a second voice; the touch point position is generated by a touch operation; When it is determined that the touch operation matches a predetermined first rule, and the second voice matches a predetermined second rule; the first rule is used to determine the start of voice input according to the touch operation, and the second rule is used to determine the start of voice input according to the second voice; Determining a first output position on the touch screen according to the position of the touch operation; extracting a first voiceprint from the second voice; Establishing a correspondence between a first voiceprint and a first output position; A first voice is received, and when it is determined that the voiceprint of the first voice matches the first voiceprint, content of the first voice is recognized, and the content of the first voice is output to the first output location.

10. The method according to claim 9, characterized in that The step of determining a first output position on the touch screen according to the position of the touch operation includes: A starting position and a range of the first output position are determined according to a set of touch positions in the touch positions.

11. The method according to claim 9 or 10, characterized in that The method includes outputting the area indicated by the first output position in a manner distinguishable from a current background.

12. The method according to claim 9, characterized in that Before establishing the correspondence between the first voiceprint and the first output position, the method further includes: The position of the sound source of the second voice is calculated, and it is determined whether the position and the contact point position meet a preset condition.

13. The method according to any one of claims 9, 10 and 12, characterized in that: Also includes: The correspondence between the first voiceprint and the first output position is cancelled.

14. The method according to claim 13, characterized in that Also includes: Establishing a correspondence between the second voiceprint and the first output position; A third voice is received, and when it is determined that the voiceprint of the third voice matches the second voiceprint, the content of the third voice is recognized, and the content of the third voice is output to a blank space of the first output position.

15. The method according to claim 14, characterized in that Also includes: Establishing a correspondence between the second voiceprint and the first output position; A third voice is received, and when it is determined that the voiceprint of the third voice matches the second voiceprint, the content of the third voice is recognized, and the content of the third voice is output to the first output location, overwriting the content of the first voice.

16. The method according to claim 14 or 15, characterized in that Outputting the content of the third voice to the first output location includes: The content of the third speech is output in a format different from the format in which the content of the first speech is output.

17. A human-computer interaction system, characterized in that: The human-computer interaction system includes a touch screen and a processor; The touch screen is used to receive a touch operation; The processor is used to receive a second voice; and is also used to determine a first output position on the touch screen according to a position of the touch operation when determining that the touch operation matches a predetermined first rule, and when the second voice matches a predetermined second rule; the first rule is used to determine to start voice input according to the touch operation, and the second rule is used to determine to start voice input according to the second voice; and is also used to extract a first voiceprint from the second voice, and establish a corresponding relationship between the first voiceprint and the first output position on the touch screen; The processor is further configured to receive a first voice, and when determining that the voiceprint of the first voice matches the first voiceprint, identify the content of the first voice, and output and display the content of the first voice to the first output location.

18. The system of claim 17, wherein: When the processor determines the first output position on the touch screen according to the position of the touch operation, it is specifically used to: The starting position and range of the first output position on the touch screen are determined according to a set of contact point positions in the positions of the touch operation.

19. The system according to claim 17 or 18, characterized in that: The processor is further configured to output and display the area indicated by the first output position on the touch screen in a manner distinguishable from a current touch screen background.

20. The system of claim 17, wherein: Before establishing the correspondence between the first voiceprint and the first output position on the touch screen, the processor is further configured to: The position of the sound source of the second voice is calculated, and it is determined whether the position and the position of the touch operation meet a preset condition.

21. The system according to any one of claims 17 to 20, characterized in that: The processor is further configured to: The correspondence between the first voiceprint and the first output position on the touch screen is cancelled.

22. The system of claim 21, wherein: The processor is further configured to: Establishing a correspondence between the second voiceprint and the first output position on the touch screen; A third voice is received, and when it is determined that the voiceprint of the third voice matches the second voiceprint, the content of the third voice is recognized, and the content of the third voice is output and displayed in a blank space of the first output position.

23. The system of claim 22, wherein: The processor is further configured to: Establishing a correspondence between the second voiceprint and the first output position on the touch screen; A third voice is received, and when it is determined that the voiceprint of the third voice matches the second voiceprint, the content of the third voice is recognized, and the content of the third voice is output and displayed to the first output location, covering the content of the first voice.

24. The system according to claim 22 or 23, characterized in that When the processor outputs and displays the content of the third voice, the processor is specifically configured to: output and display the content of the third voice in a format different from that when the processor outputs and displays the content of the first voice.

25. A computer device, characterized in that: The computer device comprises: a speech processing unit and a comprehensive processing unit; wherein, The speech processing unit is configured to receive a first speech, and when it is determined that the voiceprint of the first speech matches the first voiceprint, identify the content of the first speech; The comprehensive processing unit is used to receive a touch operation and a second voice; and is also used to determine a first output position on the touch screen according to the position of the touch operation when determining that the touch operation matches a predetermined first rule and the second voice matches a predetermined second rule; the first rule is used to determine to start voice input according to the touch operation, and the second rule is used to determine to start voice input according to the second voice; and is also used to extract a first voiceprint from the second voice, and establish a corresponding relationship between the first voiceprint and the first output position on the touch screen; and is also used to output the content of the first voice to the first output position.

26. The computer device of claim 25, wherein: Also included is a contact processing unit; The touch point processing unit is used to receive a touch point position, and when it is determined that a touch point position set in the touch point position matches a predetermined first rule, determine the first output position according to the touch point position; The speech processing unit is further configured to receive a second speech, and extract a first voiceprint from the second speech when it is determined that the second speech matches a predetermined first rule.

27. The computer device of claim 26, wherein: When determining the first output position according to the contact position, the contact processing unit is specifically configured to: determine a starting position and a range of the first output position according to a contact position set in the contact position.

28. The computer device of claim 26, wherein: The speech processing unit is further used to calculate the position of the sound source of the second speech; The comprehensive processing unit is further used to determine whether the position of the sound source of the second voice and the contact point position meet preset conditions before establishing the corresponding relationship between the first voiceprint and the first output position.

29. The computer device according to any one of claims 25 to 28, characterized in that: The integrated processing unit is also used for: The correspondence between the first voiceprint and the first output position is cancelled.

30. A computer device, characterized in that: include: A memory and one or more processors; the memory and the processor are coupled; The memory is used to store computer program codes, which include computer instructions. When the computer instructions are executed by the computer device, the computer device executes the human-computer interaction method as described in any one of claims 1 to 9.

Images