Virtual keyboard input method and device, electronic equipment, related medium and product

Abstract

The present disclosure provides a virtual keyboard input method, device, electronic equipment, related medium and product. The virtual keyboard input method comprises: establishing a first cross-process communication between a host application and a keyboard extension; displaying a third-party virtual keyboard in a native keyboard container through the keyboard extension, the third-party virtual keyboard comprising an auxiliary input area and a first keyboard area, the size and position of the native keyboard container being fixed; hiding the first keyboard area, displaying a second keyboard area in a custom keyboard container through the host application, the size and position of the custom keyboard container being variable; in response to receiving operation information for the second keyboard area, sending the operation information from the host application to the keyboard extension through the first cross-process communication, and displaying a response result for the operation information in the auxiliary input area through the keyboard extension. The present disclosure can improve the information transmission efficiency of the application and the input method input efficiency when the application calls the third-party input method.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and in particular to a virtual keyboard input method, apparatus, electronic device, related medium, and product. Background Technology

[0002] With the widespread use of smartphones and other electronic devices, various applications (Apps) have flourished, providing users with a rich and diverse range of functional experiences. In application usage scenarios, there are often situations where a virtual keyboard needs to be invoked for text input. For example, a user might need to invoke a virtual keyboard to send comments while using a video application, or they might need to invoke a virtual keyboard to communicate while using a game application.

[0003] In some current operating systems (such as iOS), third-party input methods are subject to similar restrictions as the system input methods. Specifically, the virtual keyboard corresponding to a third-party input method must be displayed within the native keyboard container provided by the operating system. However, the size and position of the native keyboard container are fixed and cannot be directly adjusted by developers. In some situations (such as landscape mode), the virtual keyboard displayed within the native keyboard container can significantly obscure the screen. Objects need to be frequently adjusted to see the obscured key information, resulting in low input efficiency and inefficient information transmission. Summary of the Invention

[0004] This disclosure provides a virtual keyboard input method, device, electronic device, related medium, and product, which can improve the information transmission efficiency of the application and the input efficiency of the input method when the application calls a third-party input method.

[0005] According to one aspect of this disclosure, a virtual keyboard input method is provided, comprising: Establish the first cross-process communication between the host application and the keyboard extension of the third-party keyboard application; A third-party virtual keyboard is displayed in the native keyboard container through the keyboard extension. The third-party virtual keyboard includes an auxiliary input area and a first keyboard area. The size and position of the native keyboard container are fixed. The first keyboard area is hidden, and a second keyboard area is displayed in a custom keyboard container through the host application. The size and position of the custom keyboard container are variable. In response to receiving operation information for the second keyboard area, the operation information is sent from the host application to the keyboard extension via the first inter-process communication, and the response result for the operation information is displayed in the auxiliary input area via the keyboard extension.

[0006] According to one aspect of this disclosure, a virtual keyboard input device is provided, comprising: The communication establishment module is used to establish the first inter-process communication between the host application and the keyboard extension of the third-party keyboard application; A keyboard display module is used to display a third-party virtual keyboard in a native keyboard container through the keyboard extension. The third-party virtual keyboard includes an auxiliary input area and a first keyboard area. The size and position of the native keyboard container are fixed. The keyboard display module is also used to hide the first keyboard area and display the second keyboard area in a custom keyboard container through the host application, wherein the size and position of the custom keyboard container are variable; An input response module is configured to, in response to receiving operation information for the second keyboard area, send the operation information from the host application to the keyboard extension via the first inter-process communication, and display the response result for the operation information in the auxiliary input area via the keyboard extension.

[0007] Optionally, the communication establishment module is further configured to: The host application sends a call request to the keyboard extension; Through the keyboard extension, in response to the call request, a socket connection request is generated based on the port number of the host application, and the socket connection request is sent to the host application; Through the keyboard extension, a response to the socket connection request is received from the host application, thereby establishing socket communication as the first inter-process communication.

[0008] Optionally, the virtual keyboard input device further includes a verification module for: The host application generates a signature based on the application identifier of the host application, generates a data packet to be verified based on the application identifier and the signature, and sends the data packet to be verified to the keyboard extension through the first inter-process communication. The application identifier and the signature are obtained from the data packet to be verified through the keyboard extension, and the signature is verified based on the application identifier. The keyboard display module is also used for: When the signature verification is successful, the third-party virtual keyboard is displayed in the native keyboard container via the keyboard extension.

[0009] Optionally, the verification module is further configured to: The signature is generated based on the port number, the application identifier, the predefined key, and the timestamp; The data packet to be verified is generated based on the application identifier, the timestamp, and the signature. The application identifier, the timestamp, and the signature are obtained from the data packet to be verified. The signature is verified based on the port number, the application identifier, the predetermined key, and the timestamp.

[0010] Optionally, the host application's port number includes N, where N is an integer greater than 1; the communication establishment module is further used for: Execute at least one iteration of the loop, and in the i-th iteration, perform the following processing: Generate the i-th socket connection request based on the i-th port number of the host application, and send the i-th socket connection request to the host application; If no response to the i-th socket connection request is received after the timeout period, and i is less than N, the loop enters the (i+1)-th iteration. When a response to the i-th socket connection request is received after the timeout period, socket communication is established as the first inter-process communication.

[0011] Optionally, the keyboard display module is further configured to: When the size of the native keyboard container exceeds a first threshold, the first keyboard area is hidden. The second keyboard area is displayed in the custom keyboard container by the host application, wherein the size of the custom keyboard container does not exceed the first threshold.

[0012] Optionally, the keyboard display module is further configured to: Identify key subjects on the screen where the virtual keyboard input device is applied; When the native keyboard container obscures the identified key subject, the first keyboard area is hidden; The second keyboard area is displayed in the custom keyboard container through the host application, so that the custom keyboard container does not obscure the key subject.

[0013] Optionally, the keyboard display module is further configured to: Determine the non-critical subject area on the screen that does not contain the critical subject; A reference line is generated based on the key subject, and the non-key subject area is divided based on the reference line to obtain multiple divided areas; Based on the area of ​​each of the multiple divided regions, a target region is determined among the multiple divided regions; The custom keyboard container is determined in the target area, and the second keyboard area is displayed in the custom keyboard container.

[0014] Optionally, the keyboard display module is further configured to: Determine the first semantics before the screen of the virtual keyboard input device is displayed, prior to the display of the third-party virtual keyboard; Determine the second semantic meaning after the third-party virtual keyboard is displayed on the screen; When the matching degree between the first semantic and the second semantic is less than the second threshold, the first keyboard area is hidden, and the second keyboard area is displayed in the custom keyboard container through the host application.

[0015] Optionally, the keyboard display module is further configured to: When the matching degree between the first semantic and the second semantic is less than the second threshold, the first keyboard area is hidden; The host application displays the custom keyboard container at a predetermined size and position, and displays the second keyboard area within the custom keyboard container; Determine the third semantics of the screen after displaying the second keyboard area; When the matching degree between the first semantic and the third semantic is less than the second threshold, adjust the size and position of the custom keyboard container, and return to the step of obtaining the third semantic after the screen displays the second keyboard area, until the matching degree between the first semantic and the third semantic is not less than the second threshold.

[0016] Optionally, the operation information is at least one character selected in the second keyboard area, and the response result is a candidate word corresponding to the at least one character; the input response module is further configured to: In response to receiving the selected at least one character, the at least one character is sent from the host application to the keyboard extension via the first inter-process communication; Establish a second cross-process communication between the keyboard extension and the third-party keyboard application; The keyboard extension allows the at least one character to be sent to the third-party keyboard application via the second inter-process communication, and the candidate word to be received from the third-party keyboard application. The candidate words are displayed in the auxiliary input area via the keyboard extension.

[0017] Optionally, the operation information is at least one character selected in the second keyboard area, and the response result is a candidate word corresponding to the at least one character; the screen on which the virtual keyboard input device is applied further includes an upper screen area; the virtual keyboard input device further includes an upper screen module for: In response to the selection operation of the candidate word, the selected candidate word is displayed in the upper screen area.

[0018] Optionally, the input response module is further configured to: In response to an operation on the second keyboard area, determine the trigger position of the operation on the second keyboard area; Based on the key layout within the custom keyboard container and the scaling ratio of the custom keyboard container, the key corresponding to the trigger position within the custom keyboard container is determined. Based on the determined key press, the operation information is generated; The operation information is sent from the host application to the keyboard extension via the first inter-process communication.

[0019] Optionally, the native keyboard container is placed along the lower edge of the screen on which the virtual keyboard input device is applied; the keyboard display module is further configured to: The native keyboard container is height-constrained to a second height from a first height, such that the first keyboard area is hidden below the bottom edge of the screen. The first height is the common height of the auxiliary input area and the first keyboard area, and the second height is the height of the auxiliary input area.

[0020] Optionally, the host application embeds a proxy toolkit; the communication establishment module is further configured to: The first cross-process communication between the host application and the keyboard extension is established using the communication tools in the proxy toolkit. The keyboard display module is also used for: The second keyboard area is displayed in the custom keyboard container using the keyboard display tool in the proxy toolkit.

[0021] Optionally, the virtual keyboard input device further includes a synchronized skinning module, used for: In response to an update of the host application's display resources, obtain the first updated display resources of the host application; The host application queries the display resource database to obtain the second updated display resource of the third-party virtual keyboard, and the second updated display resource matches the first updated display resource. The second updated display resource is sent from the host application to the keyboard extension via the first inter-process communication; The keyboard extension updates the display resources of the third-party virtual keyboard according to the second updated display resources.

[0022] According to one aspect of this disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the virtual keyboard input method as described above.

[0023] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the virtual keyboard input method as described above. According to one aspect of this disclosure, a computer program product is provided, the computer program product including a computer program that is read and executed by a processor of an electronic device, causing the electronic device to perform the virtual keyboard input method as described above.

[0024] To overcome the limitations imposed by third-party input methods' virtual keyboards being displayed within a fixed-size and fixed-position native keyboard container, this embodiment hides the first keyboard area of ​​the third-party virtual keyboard and displays a second keyboard area within a custom keyboard container with adjustable size and position, replacing the first keyboard area. Since the size and position of the custom keyboard container are adjustable, i.e., the size and position of the second keyboard area are also adjustable, the obstruction of key information on the screen is reduced, thus eliminating the need for frequent adjustments to see obscured information. Because the custom keyboard container is introduced outside the existing native keyboard container mechanism, content entered through the second keyboard area cannot be transmitted to the third-party input method through inherent mechanisms. Therefore, this embodiment establishes cross-process communication between the host application and the keyboard extension of the third-party keyboard application. Through this cross-process communication, operation information targeting the second keyboard area in the custom keyboard container is sent to the keyboard extension, enabling the keyboard extension to respond to the operation information, obtain a response result, and display the response result in the auxiliary input area. This achieves the effect of transmitting the information required by the input method without using the original native keyboard container mechanism. The embodiments disclosed herein can reduce screen obstruction of the host application when the host application needs to call a third-party input method, thereby improving the information transmission efficiency of the host application and the input efficiency of the input method.

[0025] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0026] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure. Figure 1 This is a system architecture diagram of the virtual keyboard input method provided in this embodiment of the disclosure. Figure 2 This is a schematic diagram illustrating the application of the virtual keyboard input method provided in this embodiment to a video playback scenario; Figure 3 This is a schematic diagram illustrating the application of the virtual keyboard input method provided in this embodiment to a game battle scenario; Figure 4 This is a schematic diagram illustrating the application of the virtual keyboard input method provided in this embodiment of the present disclosure to an instant messaging scenario; Figure 5 This is a general flowchart of the virtual keyboard input method provided in the embodiments of this disclosure; Figure 6 This is a schematic diagram of the communication architecture provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram of a third-party virtual keyboard provided in an embodiment of this disclosure; Figure 8 This is a schematic diagram of the native keyboard container provided in an embodiment of this disclosure; Figure 9 This is a schematic diagram of the virtual keyboard type switching process provided in an embodiment of this disclosure; Figure 10 This is a schematic diagram illustrating the process of switching from non-floating mode to floating mode provided in an embodiment of this disclosure; Figure 11 This is a schematic diagram illustrating the process of receiving operation information and displaying response results provided in an embodiment of this disclosure; Figure 12 It is about Figure 5 A detailed flowchart of step 510; Figure 13 Is Figure 12 A flowchart showing the verification process following steps 1230; Figure 14 It is about Figure 13 A detailed flowchart of the verification process; Figure 15 It is about Figure 12 A detailed flowchart for establishing socket communication in China; Figure 16 It is about Figure 5 A detailed flowchart of step 530; Figure 17 This is a schematic diagram illustrating the process of entering floating mode when the size of the native keyboard container exceeds a first threshold, as provided in an embodiment of this disclosure. Figure 18 It is about Figure 5 Another specific flowchart for step 530; Figure 19 This is a schematic diagram of the process of entering floating mode when the native keyboard container obscures the key subject, provided in an embodiment of this disclosure; Figure 20 It is about Figure 19 A detailed flowchart of step 1830; Figure 21 This is a schematic diagram illustrating the process of determining a target area and displaying a custom keyboard container and a second keyboard area according to an embodiment of this disclosure; Figure 22 It is about Figure 5 Another specific flowchart for step 530; Figure 23 It is about Figure 22 A detailed flowchart of step 2230; Figure 24 It is about Figure 5 Another specific flowchart for step 530; Figure 25 This is a schematic diagram of the virtual keyboard display mode switching process provided in an embodiment of this disclosure; Figure 26 This is another schematic diagram of the virtual keyboard display mode switching process provided in the embodiments of this disclosure; Figure 27 It is about Figure 5 A detailed flowchart of step 540; Figure 28 This is another schematic diagram of the communication architecture provided in the embodiments of this disclosure; Figure 29 Is Figure 5 A detailed flowchart of the synchronized skin change after step 520; Figure 30 This is a schematic diagram of the synchronous skin-changing process provided in an embodiment of this disclosure; Figure 31 This is another schematic diagram of the synchronous skin-changing process provided in the embodiments of this disclosure; Figure 32This is a flowchart illustrating a floating keyboard scenario provided in an embodiment of this disclosure; Figure 33 This is a schematic diagram of the communication architecture for a synchronized skin-changing scenario provided in an embodiment of this disclosure; Figure 34 This is a schematic diagram of the socket communication mechanism provided in an embodiment of this disclosure; Figure 35 This is a schematic diagram of the data transmission mechanism provided in an embodiment of this disclosure; Figure 36 This is a block diagram of the virtual keyboard input device provided in the embodiments of this disclosure; Figure 37 This is a terminal structure diagram of the implementation of the virtual keyboard input method provided in this embodiment of the disclosure; Figure 38 This is a server structure diagram of the implementation of the virtual keyboard input method provided in this embodiment. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0028] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations: 1) Virtual Keyboard: Also known as a software keyboard. A virtual keyboard is not a separate hardware keyboard, but rather something displayed on the terminal screen. It simulates a hardware keyboard through software, allowing users to input information via touch or mouse clicks, achieving the same input effect as a hardware keyboard.

[0029] 2) Host App: This refers to the application that calls the keyboard extension. This disclosure does not limit the type of host app; for example, it can be a video application, a game application, or an instant messaging application.

[0030] 3) Third-party keyboard application: also known as a third-party input method application. In this disclosure embodiment, it is used to package and distribute keyboard extensions, i.e., as a container application (Container App) for keyboard extensions.

[0031] 4) Keyboard Extension: This is a type of App Extension. App Extensions allow specific functionality to be separated from a container application, making it usable within the host application. In this embodiment, a keyboard extension is used to separate virtual keyboard functionality from a third-party keyboard application, enabling its use within the host application. App Extensions cannot be packaged and distributed independently; the container application is used to package and distribute them. The container application can provide additional resources, perform complex calculations, or retrieve data from the network that can be utilized by the app extension. App Extensions cannot run independently as a complete application; they must be attached to the host application. The lifecycle of an app extension begins when the host application invokes it and ends when the host application stops invoking it.

[0032] 5) Keyboard container: refers to a specific area on the screen used to hold the virtual keyboard view. This disclosure relates to native keyboard containers and custom keyboard containers. A native keyboard container is a keyboard container provided by the operating system, whose size and position are preset and fixed by the operating system to regulate the presentation and use of the virtual keyboard within the operating system. A custom keyboard container, on the other hand, is an area specified according to actual needs; therefore, the size and position of a custom keyboard container are variable.

[0033] 6) Inter-Process Communication (IPC): Used to transfer data or signals between multiple processes (or threads). Each process has its own independent system resources and is isolated from others. Inter-process communication is needed to enable resource access and coordination between different processes.

[0034] In some current operating systems (such as iOS), third-party input methods are subject to similar restrictions as the system input methods. Specifically, the virtual keyboard corresponding to a third-party input method must be displayed within the native keyboard container provided by the operating system. However, the size and position of the native keyboard container are fixed and cannot be directly adjusted by developers. In some situations (such as landscape mode), the virtual keyboard displayed within the native keyboard container can significantly obscure the screen. Objects need to be frequently adjusted to see the obscured key information, resulting in low input efficiency and inefficient information transmission.

[0035] In view of this, the present disclosure provides a virtual keyboard input method, device, electronic device, related medium and product, which can improve the information transmission efficiency of the application and the input efficiency of the input method when the application calls the third-party input method.

[0036] System architecture and scenario description of the embodiments disclosed herein Figure 1 This is a system architecture diagram of the virtual keyboard input method according to an embodiment of the present disclosure. It includes a terminal 140, an Internet 130, a gateway 120, a server 110, etc.

[0037] Terminal 140 is a device with a host application and a third-party keyboard application installed. It can take various forms, including desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple desktop computers connected via a local area network, sharing a single monitor, can work collaboratively to form a single terminal 140. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0038] Server 110 refers to a computer system capable of providing certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 110 can also communicate with the Internet 130 via wired or wireless means to exchange data.

[0039] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0040] In one embodiment of this disclosure, the virtual keyboard input method can be implemented via terminal 140.

[0041] For example, during the execution of the host application, terminal 140 establishes a first inter-process communication between the host application and a keyboard extension of a third-party keyboard application. Terminal 140 displays a third-party virtual keyboard within a native keyboard container via the keyboard extension. The third-party virtual keyboard includes an auxiliary input area and a first keyboard area, with the size and position of the native keyboard container fixed. Terminal 140 hides the first keyboard area and displays a second keyboard area within a custom keyboard container via the host application. The size and position of the custom keyboard container are variable. In response to receiving operation information for the second keyboard area, terminal 140 sends the operation information from the host application to the keyboard extension via the first inter-process communication and displays the response result for the operation information in the auxiliary input area via the keyboard extension. Thus, only terminal 140 is needed to independently implement the virtual keyboard input method provided in this embodiment, which has a wide range of applications, such as scenarios with weak or no network coverage.

[0042] In one embodiment of this disclosure, the virtual keyboard input method can be implemented jointly by terminal 140 and server 110.

[0043] For example, server 110 can be a backend server for a third-party keyboard application. In addition to establishing a first inter-process communication (IPC) between the host application and the keyboard extension, terminal 140 also establishes a second IPC between the keyboard extension and the third-party keyboard application. Based on this second IPC, the third-party keyboard application can provide additional capabilities to the keyboard extension. For example, the keyboard extension can send operation information to the third-party keyboard application via the second IPC; the third-party keyboard application sends the operation information to server 110; server 110 determines the response result based on the operation information and sends the response result to the third-party keyboard application; the third-party keyboard application then sends the response result to the keyboard extension via the second IPC. Thus, the task of determining the response result is delegated to server 110. Since server 110 typically has abundant computing and data resources, it can improve the accuracy of data processing and return accurate response results. Therefore, by collaborating between terminal 140 and server 110 to complete the input task, the accuracy of the input task can be improved, while also reducing the local computing and storage pressure on terminal 140.

[0044] For example, server 110 is connected to a display resource database. When the host application of terminal 140 completes the display resource update, it sends a request to server 110 based on the first updated display resource to query the display resource database and obtain the second updated display resource of the third-party virtual keyboard. Then, the host application sends the second updated display resource to the keyboard extension via the first inter-process communication, and the keyboard extension updates the display resource of the third-party virtual keyboard based on the second updated display resource. In this way, by using the display resource database connected to server 110, the host application and the third-party virtual keyboard can synchronize their skinning, thereby ensuring the similarity of the overall display style and improving the user's input experience.

[0045] The embodiments disclosed herein can be applied to various scenarios, such as Figure 2 The video playback scene shown Figure 3 The game battle scene shown Figure 4 The instant messaging scenarios shown will be explained separately below.

[0046] (a) Video playback scenario Taking a video application as an example of the host application in terminal 140, the user may need to invoke a virtual keyboard to send comments while watching a video through the video application. In the solutions provided by related technologies, regardless of whether the video application uses a system input method or a third-party input method, the virtual keyboard is displayed through a native keyboard container. Figure 2 The image shows a virtual keyboard displayed within a native keyboard container, comprising an auxiliary input area 210 and a keyboard area 220 (i.e., a first keyboard area). However, this virtual keyboard display obscures key information in the video frame, preventing the user from seeing the obscured information and resulting in low input and information transmission efficiency. For example, if the user wants to send a comment about a person's clothing in a video, the virtual keyboard will obscure the clothing, allowing the user to see only the person's head. If the user forgets the clothing, they must then manually close the virtual keyboard to reveal the obscured clothing and then reopen the virtual keyboard for input, a cumbersome process.

[0047] To address this, in this embodiment, the keyboard area 220 is hidden, and the keyboard area 230 (i.e., the second keyboard area) is displayed in a custom keyboard container via a video application. Since the keyboard area 230 does not obscure the clothing of the person in the video frame, the user can simultaneously view the clothing in the video frame and perform input operations in the keyboard area 230, achieving synchronous viewing and input. This greatly improves input efficiency and information transmission efficiency, avoiding unnecessary activation and deactivation of the virtual keyboard. It is worth noting that although... Figure 2 The display method of the keyboard area has been changed, but the auxiliary input area 210 and the keyboard area 230 are still displayed on the screen at the same time, which can realize the complete virtual keyboard input function and ensure the effectiveness and availability of the virtual keyboard input function. That is, the keyboard area 230 is used to receive operation information, and the auxiliary input area 210 is used to display the response result.

[0048] (II) Game Battle Scene Taking the host application in terminal 140 as a game application as an example, object A, while playing a game through the game application, may need to invoke a virtual keyboard to communicate with other objects (such as teammates in a game, referred to here as object B). In the solutions provided by related technologies, regardless of whether the game application uses the system input method or a third-party input method, the virtual keyboard will be displayed through the native keyboard container. Figure 3 The image shows a virtual keyboard displayed within a native keyboard container, comprising an auxiliary input area 310 and a keyboard area 320 (i.e., a first keyboard area). However, this virtual keyboard display obscures key information in the game screen, preventing object A from seeing this obscured information, thus leading to low input efficiency and low information transmission efficiency. For example, object A might want to communicate with object B based on the real-time battle situation, but after the game application invokes the virtual keyboard, it obscures key information such as the location of object A's virtual character and the minimap, preventing object A from understanding the real-time battle situation. In game battles, the situation changes rapidly; if object A uses the virtual keyboard to communicate, the content may be outdated and untimely, failing to adapt to the constantly updated battle situation. For example, object A sends the message "No enemy location detected" to object B by typing on the virtual keyboard. However, the enemy's location is already displayed on the minimap, so the message "No enemy location detected" is inaccurate and may have a negative impact on the game, such as causing object B to misjudge the situation.

[0049] To address this, in this embodiment, the keyboard area 320 is hidden, and the keyboard area 330 (i.e., the second keyboard area) is displayed in a custom keyboard container through a game application. Since the keyboard area 330 does not obscure key information such as the location of the virtual character controlled by object A or the minimap, object A can obtain real-time information about the battle situation and perform input operations in the keyboard area 330 to communicate with object B, greatly improving input efficiency and information transmission efficiency, and facilitating effective communication. For example, object A might initially intend to send the message "No enemy location detected" to object B by typing in the keyboard area 330, but if object A discovers that the enemy's location is already displayed on the minimap, then object A can send the message "Enemy location is XXXX" by typing in the keyboard area 330, thus promptly alerting object B.

[0050] (III) Instant Messaging Scenarios Taking the host application in terminal 140 as an example of an instant messaging application, object A needs to invoke a virtual keyboard to communicate instantly with other objects (here named object B) while using the instant messaging application. In the solutions provided by related technologies, regardless of whether the instant messaging application calls the system input method or a third-party input method, the virtual keyboard will be displayed through the native keyboard container. Figure 4 The image shows a virtual keyboard displayed within the native keyboard container, comprising an auxiliary input area 410 and a keyboard area 420 (i.e., the first keyboard area). However, because the virtual keyboard occupies a large area, the area left for instant messaging messages is small, resulting in many historical instant messaging messages not being displayed. In other words, the virtual keyboard indirectly obscures historical instant messaging messages. In an instant messaging scenario, historical instant messaging messages are likely crucial information needed by object A for input. If historical instant messaging messages are obscured, it leads to low input efficiency and low information transmission efficiency. For example, object A is object B's assistant. On January 1st, object B sends object A an instant messaging message: "Please help me book a flight for January 2nd, and remind me to bring the project proposal for xx." On January 2nd, object A invokes the virtual keyboard in the instant messaging application to send the instant messaging message: "Please proceed to xx airport for boarding." At this time, only the instant messaging message "Please proceed to xx airport for boarding" is displayed on the screen, and the historical instant messaging message "Please help me book a flight for January 2nd, and remind me to bring the project proposal for xx" is not displayed. If object A forgets the specific content that needs to be reminded, object A can only operate by hiding the virtual keyboard to restore the obscured historical instant messaging messages, and then operate by bringing up the virtual keyboard to input them, which is a very cumbersome operation.

[0051] To address this, in this embodiment, keyboard area 420 is hidden, and keyboard area 430 (i.e., the second keyboard area) is displayed in a custom keyboard container via an instant messaging application. Since keyboard area 430 does not obscure the historical instant messaging message "Please help me book a flight for January 2nd, and remind me to bring the project proposal for xx project," object A can perform input operations in keyboard area 430 based on historical instant messaging messages to send the instant messaging message "Please bring the project proposal for xx project," greatly improving input efficiency and information transmission efficiency. Applying the solution provided in this embodiment to instant messaging scenarios helps relevant objects understand the context, thereby achieving more accurate and effective input.

[0052] It is worth noting that, Figure 1 The system architecture shown and Figures 2 to 4 The scenarios shown are for the purpose of more clearly illustrating the technical solutions of the embodiments of this disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of this disclosure. As those skilled in the art will know, with the evolution of virtual keyboard input technology and the emergence of new business scenarios, the technical solutions provided by the embodiments of this disclosure are also applicable to similar technical problems.

[0053] General Description of Embodiments in this Disclosure This disclosure provides a virtual keyboard input method. The virtual keyboard input method can be executed on terminal 140, or partly on terminal 140 and partly on server 110. There is no limitation on this.

[0054] like Figure 5 As shown, a virtual keyboard input method according to an embodiment of this disclosure includes: Step 510: Establish the first cross-process communication between the host application and the keyboard extension of the third-party keyboard application; Step 520: Display a third-party virtual keyboard in the native keyboard container via keyboard extension. The third-party virtual keyboard includes an auxiliary input area and a first keyboard area. The size and position of the native keyboard container are fixed. Step 530: Hide the first keyboard area and display the second keyboard area in a custom keyboard container through the host application. The size and position of the custom keyboard container are variable. Step 540: In response to receiving operation information for the second keyboard area, the operation information is sent from the host application to the keyboard extension via the first inter-process communication, and the response result for the operation information is displayed in the auxiliary input area via the keyboard extension. Steps 510-540 are described in detail below.

[0055] In step 510, the first cross-process communication is established between the host application and the keyboard extension of the third-party keyboard application.

[0056] Here, the terminal has a host application and a third-party keyboard application installed. The host application refers to the application that calls the keyboard extension. This embodiment does not limit the type of the host application; for example, it could be a video application, a game application, or an instant messaging application. The third-party keyboard application, also known as a third-party input method application, is used in this embodiment to package and distribute the keyboard extension, i.e., it serves as a container application for the keyboard extension. It is worth noting that the third-party keyboard application is different from the native keyboard component provided by the operating system (i.e., the system input method). Furthermore, in this embodiment, "native" means provided by the operating system and follows the original architecture, interfaces, and specifications of the operating system. For example, the native keyboard component is built into the operating system and does not require additional installation; the native keyboard container is managed by the operating system. For some operating systems (such as iOS), the size and position of the native keyboard container are fixed and cannot be directly adjusted.

[0057] In response to the host application calling the keyboard extension of a third-party keyboard application, inter-process communication (local inter-process communication) is established between the host application and the keyboard extension; for ease of distinction, this is referred to as the first inter-process communication. For ease of understanding, taking the iOS operating system as an example, this disclosure provides the following embodiments: Figure 6 The diagram illustrates the communication architecture, showing the initial cross-process communication established between the host application and the keyboard extension. Since the keyboard extension is packaged and distributed along with the third-party keyboard application, therefore... Figure 6 In iOS, keyboard extensions and third-party keyboard applications belong to the same app group. An app group is an application group identifier in the iOS operating system used to enable collaboration and data sharing among multiple applications from the same developer.

[0058] The embodiments disclosed herein do not limit the communication method used for the first inter-process communication. Taking the iOS operating system as an example, UIPasteboard, URL Scheme, Universal Link, App Group, etc. can be used, which will be described below.

[0059] 1) UIPasteboard: UIPasteboard is the application programming interface (API) for the clipboard in the iOS operating system. When some content is copied in one process through UIPasteboard, other processes can read this content. Therefore, UIPasteboard can be used to achieve the first cross-process communication.

[0060] 2) URL Scheme / Universal Link: A URL Scheme is a protocol for jumping to other processes via hyperlinks. A Universal Link can be understood as an upgraded version of a URL Scheme. Processes can be brought to the foreground through URL Scheme / Universal Links. URL Scheme / Universal Links can also carry a certain amount of text to the process that needs to be invoked.

[0061] 3) App Group: Allows processes within the same App Group to send notifications to each other or read and write files in the same directory. This requires the host application and the keyboard extension to belong to the same App Group.

[0062] In step 520, a third-party virtual keyboard is displayed in the native keyboard container through keyboard extension. The third-party virtual keyboard includes an auxiliary input area and a first keyboard area. The size and position of the native keyboard container are fixed.

[0063] After the initial inter-process communication between the host application and the keyboard extension is successfully established, the keyboard extension displays the third-party virtual keyboard. Due to operating system limitations, the keyboard extension can only display the third-party virtual keyboard within the native keyboard container. For example, the keyboard extension may create and display the third-party virtual keyboard using native keyboard classes provided by the operating system (such as the UIInputViewController class provided by iOS). When a virtual keyboard created using a native keyboard class is displayed, its view hierarchy is bound to the native keyboard container.

[0064] Third-party virtual keyboards consist of an auxiliary input area and a keyboard area. For ease of distinction, the keyboard area in a third-party virtual keyboard will be referred to as the first keyboard area. The definitions of the auxiliary input area and the keyboard area are explained below. The keyboard area is used to receive operation information. Types of keyboard areas include key input areas, handwriting input areas, and gesture input areas. Key input areas can be further subdivided into 9-key key input areas, 26-key key input areas, etc., each type of keyboard area corresponding to a keyboard input mode. Taking the key input area as an example, it includes several keys for input, such as character input keys (letter keys, number keys, punctuation keys, etc.) and character control keys (space key, delete key, enter / newline key, etc.). The auxiliary input area can include several function entries for auxiliary input functions, such as emoji input function entry, microphone input function entry, and keyboard input mode function entry (used to set 9-key key input mode, 26-key key input mode, or other input modes). These function entries can be represented through controls (such as buttons) or other forms. The auxiliary input area can also be used to display response results, such as candidate words corresponding to at least one character selected in the keyboard area. As an example, embodiments of this disclosure provide, for example, Figure 7 The diagram shown illustrates a third-party virtual keyboard. Figure 7 The diagram illustrates a native keyboard container 710 and a third-party virtual keyboard displayed within it. The third-party virtual keyboard includes an auxiliary input area 720 and a keyboard area 730 (taking a 9-key input area as an example). It's worth noting that the third-party virtual keyboard can either completely fill the native keyboard container or only occupy a portion of it, depending on its specific design. Figure 7 Taking the latter case as an example. The function entry for auxiliary input can be displayed in the auxiliary input area, such as... Figure 7 The function entry points for functions 1 to 4 are displayed; they can also be shown in the area of ​​the native keyboard container 710 that is not occupied by a third-party virtual keyboard, such as... Figure 7 The function entry points for functions 5 to 6 are as follows: function 5 includes virtual keyboard type switching, and function 6 includes microphone input.

[0065] The native keyboard container is managed by the operating system, therefore its size and position are fixed. The size is determined by the width and height, and can be represented as (width × height); the position is determined by the horizontal and vertical coordinates, and can be represented as (x, y). Taking the iOS operating system as an example, the frame property of the native keyboard container is fixed, and the frame property includes x, y, width, and height. x represents the horizontal coordinate of the top-left corner of the native keyboard container, y represents the vertical coordinate of the top-left corner of the native keyboard container, width represents the width of the native keyboard container, and height represents the height of the native keyboard container. As an example, this disclosure provides embodiments such as... Figure 8 The diagram shows a native keyboard container 810. The parent view of the native keyboard container 810 is located at (0, 0) and has a size of (550×400). Therefore, the position of the native keyboard container 810 is (0, 150) and its size is (550×250).

[0066] In one embodiment of this disclosure, after establishing a first inter-process communication between the host application and the keyboard extension of a third-party keyboard application, the virtual keyboard input method further includes: when the establishment of the first inter-process communication between the host application and the keyboard extension fails, displaying a native virtual keyboard in a native keyboard container through a native keyboard component.

[0067] Here, when the first inter-process communication between the host application and the keyboard extension fails, it indicates that the keyboard extension is unavailable. At this point, the host application can invoke the native keyboard component (i.e., the system input method) to display the native virtual keyboard within the native keyboard container. Similar to third-party virtual keyboards, the native virtual keyboard also includes two areas: an auxiliary input area and a keyboard area. (See also...) Figure 7 As shown. By employing the above methods, the availability and overall robustness of virtual keyboard input can be guaranteed. That is, even when third-party input methods are unavailable, the system input method can at least be guaranteed to be available, thus meeting input needs.

[0068] In one embodiment of this disclosure, establishing a first inter-process communication between a host application and a keyboard extension of a third-party keyboard application includes: in response to receiving an operation that invokes a virtual keyboard in the host application, establishing a first inter-process communication between the host application and the keyboard extension of the third-party keyboard application.

[0069] Here, when an operation to invoke the virtual keyboard is received in the host application, the host application can call the keyboard extension, thereby establishing the first inter-process communication between the host application and the keyboard extension. For example, when an operation to invoke the virtual keyboard is received in the host application, the host application will call the keyboard extension by default, where the default call object of the host application can be set; or, when an operation to invoke the virtual keyboard is received in the host application, the host application will call the call object of the last call, which is the keyboard extension. This embodiment of the disclosure does not limit the triggering method of invoking the virtual keyboard. For example, if the host application displays an input field (such as an input box), then the triggering operation targeting the input field can be determined as the operation to invoke the virtual keyboard.

[0070] In one embodiment of this disclosure, establishing a first inter-process communication between a host application and a keyboard extension of a third-party keyboard application includes: in response to receiving an operation to invoke a virtual keyboard in the host application, displaying a native virtual keyboard in a native keyboard container via a native keyboard component; and in response to receiving a virtual keyboard type switching operation, establishing a first inter-process communication between the host application and the keyboard extension of the third-party keyboard application.

[0071] Here, when an operation to invoke the virtual keyboard is received in the host application, the native keyboard component can be invoked through the host application. For example, when an operation to invoke the virtual keyboard is received in the host application, the host application will invoke the native keyboard component by default. The default invoked object of the host application can be set. Alternatively, when an operation to invoke the virtual keyboard is received in the host application, the host application will invoke the invoked object of the last invocation, which is the native keyboard component.

[0072] After the native keyboard component is invoked, the native virtual keyboard is displayed in the native keyboard container. Then, when a virtual keyboard type switching operation is received, the host application invokes the keyboard extension, thereby establishing the first inter-process communication between the host application and the keyboard extension. The virtual keyboard type switching operation is used to indicate switching the host application's invocation object from the native keyboard component to the keyboard extension, or in other words, to indicate switching the native virtual keyboard to a third-party virtual keyboard. This disclosure does not limit the triggering method of the virtual keyboard type switching operation; for example, a function entry point for the virtual keyboard type switching function can be displayed in the native keyboard container, and the virtual keyboard type switching operation can be triggered through this function entry point. Figure 9As shown, the virtual keyboard type switching operation can be triggered through the function entry point of the virtual keyboard type switching function (i.e., function 5). For example, in response to receiving a trigger operation for the function entry point of function 5, all input method options are displayed, specifically including system input method (corresponding to the native keyboard component) options and third-party input method (corresponding to keyboard extension) options. Then, in response to receiving a trigger operation for the displayed third-party input method options, the host application calls the keyboard extension, thereby establishing the first inter-process communication between the host application and the keyboard extension. The virtual keyboard type switching operation includes the aforementioned trigger operation for the function entry point of function 5 and the trigger operation for the third-party input method options.

[0073] Furthermore, embodiments of this disclosure can also support switching between keyboard extensions of different third-party keyboard applications. For example, in response to receiving an operation to invoke a virtual keyboard in the host application, a first inter-process communication is established between the host application and the keyboard extension of third-party keyboard application A, and the third-party virtual keyboard A is displayed in the native keyboard container through the keyboard extension of third-party keyboard application A. Then, in response to receiving a virtual keyboard type switching operation, a first inter-process communication is established between the host application and the keyboard extension of third-party keyboard application B, and the third-party virtual keyboard B is displayed in the native keyboard container through the keyboard extension of third-party keyboard application B. Here, the virtual keyboard type switching operation is used to indicate switching the calling object of the host application from the keyboard extension of third-party keyboard application A to the keyboard extension of third-party keyboard application B, or in other words, to indicate switching third-party virtual keyboard A to third-party virtual keyboard B.

[0074] The above embodiments illustrate several scenarios in which the host application invokes the keyboard extension. Specifically, the host application can invoke the keyboard extension when initially launching the virtual keyboard, or it can switch to the keyboard extension subsequently, providing diverse virtual keyboard options to meet the user's preferences. It is worth noting that these scenarios do not constitute a limitation on the technical solutions provided in this disclosure; that is, this disclosure can also be applied to other possible situations.

[0075] In step 530, the first keyboard area is hidden, and the second keyboard area is displayed in a custom keyboard container through the host application. The size and position of the custom keyboard container are variable.

[0076] After displaying a third-party virtual keyboard within the native keyboard container via keyboard extension, considering that the third-party virtual keyboard may obscure key information on the screen, affecting input efficiency and information transmission efficiency, the first keyboard area of ​​the third-party virtual keyboard is hidden to reduce the obscuring of key information on the screen.

[0077] Because the first keyboard area in the third-party virtual keyboard is hidden, operation information cannot be received through the first keyboard area. Therefore, a second keyboard area is displayed in a custom keyboard container by the host application to receive operation information through the second keyboard area. Unlike the native keyboard container, the size and position of the custom keyboard container are variable, so the second keyboard area can be displayed in an appropriate position and at an appropriate size, thereby minimizing the obstruction of key information on the screen. In one embodiment of this disclosure, the keyboard extension can create and display a third-party virtual keyboard through a custom keyboard class, wherein the custom keyboard class supports arbitrary adjustment of the custom keyboard container.

[0078] It is worth noting that step 530 can be executed immediately after step 520, or step 530 can be executed when certain conditions are met.

[0079] For ease of understanding, the virtual keyboard display mode corresponding to step 520 is called non-floating mode, and the virtual keyboard display mode corresponding to step 530 is called floating mode. Specifically, non-floating mode refers to displaying a third-party virtual keyboard containing an auxiliary input area and a first keyboard area within the native keyboard container, while floating mode refers to displaying a third-party virtual keyboard with its first keyboard area hidden within the native keyboard container, while simultaneously displaying a second keyboard area within a custom keyboard container. Taking a video playback scenario as an example... Figure 10 The process of switching from non-floating mode to floating mode is shown.

[0080] In one embodiment of this disclosure, the native keyboard container is placed along the lower edge of the screen where the virtual keyboard input method is applied; hiding the first keyboard area includes: constraining the height of the native keyboard container to a second height from a first height, such that the first keyboard area is hidden below the lower edge of the screen, wherein the first height is the common height of the auxiliary input area and the first keyboard area, and the second height is the height of the auxiliary input area.

[0081] First, let's introduce the concept of Auto Layout. Auto Layout is a layout system that provides a declarative way to define the constraints between views, rather than relying on fixed sizes and positions, thus enabling the creation of responsive user interfaces. Constraints are the core concept of Auto Layout; specifically, they are rules used to describe the positional and size relationships between views. For example, the left margin of one view may be equal to the right margin of another view, or the height of one view may be half the height of another view.

[0082] Although the size and position of the native keyboard container are fixed attributes and cannot be directly adjusted, the height of the native keyboard container can be indirectly adjusted through constraints in this embodiment. For example, when the native keyboard container is placed along the bottom edge of the screen where the virtual keyboard input method is applied, a height constraint is applied to the native keyboard container (e.g., using the NSLayoutConstraint class) to constrain its height from a first height to a second height. The first height is the common height of the auxiliary input area and the first keyboard area, and the second height is the height of the auxiliary input area. For example, if the height of the auxiliary input area is 50 and the height of the first keyboard area is 200, then the first height is 50 + 200 = 250, and the second height is 50. Since the auxiliary input area is usually located above the first keyboard area in the height direction, constraining the height of the native keyboard container will cause the first keyboard area to be hidden below the bottom edge of the screen. It is worth noting that the native keyboard container may also be placed along the left, right, or top edge of the screen where the virtual keyboard input method is applied, and is not limited to the bottom edge.

[0083] In the above method, by using automatic layout and its core concept constraints, the height of the native keyboard container can be indirectly adjusted by declaratively defining constraint relationships, thereby hiding the first keyboard area and releasing the screen space occupied by the first keyboard area.

[0084] In one embodiment of this disclosure, displaying a second keyboard area in a custom keyboard container via a host application includes: displaying the second keyboard area in the custom keyboard container with a predetermined transparency via the host application.

[0085] Here, the transparency of the second keyboard area can be preset, and the keyboard will be displayed with the preset transparency, such as completely opaque or partially transparent. For example, in scenarios where fast input is required and background information is less important, the transparency can be set to completely opaque to highlight the second keyboard area; in scenarios where background information is needed for input, the transparency can be set to partially transparent, allowing the input object to see the background information through the second keyboard area for better input.

[0086] In one embodiment of this disclosure, after the second keyboard area is displayed in a custom keyboard container by the host application, the virtual keyboard input method further includes at least one of the following: changing the size of the custom keyboard container to scale the second keyboard area in response to a scaling operation on the second keyboard area; and changing the position of the custom keyboard container to translate the second keyboard area in response to a translation operation on the second keyboard area.

[0087] After displaying the second keyboard area within a custom keyboard container, scaling and / or panning of the second keyboard area can be supported. For example, in response to scaling the second keyboard area, while keeping the position of the custom keyboard container unchanged, the size of the custom keyboard container can be changed to scale the second keyboard area; in response to panning the second keyboard area, while keeping the size of the custom keyboard container unchanged, the position of the custom keyboard container can be changed to pan the second keyboard area.

[0088] The second keyboard area supports scaling, allowing users to flexibly adjust its size based on their eyesight, operating habits, or the characteristics of their device screen. For example, when typing on a small screen, users can enlarge the second keyboard area to make keys easier to click and reduce accidental presses; while on a large screen, shrinking the second keyboard area frees up more space for other content. For instance, when typing while viewing a document on a tablet, the second keyboard area can be shrunk to facilitate simultaneous browsing of the document content.

[0089] The second keyboard area also supports panning, allowing the object to move the second keyboard area to the most suitable position on the screen for operation. For example, when the object is operating the phone with one hand, it can pan the second keyboard area to an area that is easy to operate with one hand, such as the bottom of the screen near the thumb, which greatly improves the convenience of input, especially suitable for fast input in scenarios such as mobile office and mobile social networking.

[0090] In step 540, in response to receiving operation information for the second keyboard area, the operation information is sent from the host application to the keyboard extension via the first inter-process communication, and the response result for the operation information is displayed in the auxiliary input area via the keyboard extension.

[0091] The second keyboard area displayed by the host application is used for objects to perform input operations. When the host application receives operation information for the second keyboard area, it sends the operation information from the host application to the keyboard extension via a first inter-process communication, so that the keyboard extension can determine the response result to the operation information. Taking key input as an example, the operation information is at least one character selected in the second keyboard area, and the response result is a candidate word corresponding to at least one character; taking handwriting input as an example, the operation information is a handwritten image received in the second keyboard area, and the response result is a candidate word obtained by image recognition of the handwritten image; taking gesture input as an example, the operation information is a gesture operation received in the second keyboard area, and the response result is a candidate word obtained by gesture recognition of the gesture operation.

[0092] After the keyboard extension determines the response to the operation information, the response is displayed in the auxiliary input area, thus realizing the complete virtual keyboard function. For example... Figure 11As shown, the operation information is to select at least one character in the second keyboard area, in the order of "x", "i", "e", "x", "i", "e", and the response results include candidate words such as "thank you", "take a break", "thank you", "shoes", and "some".

[0093] In one embodiment of this disclosure, the keyboard extension integrates an input method kernel; after receiving operation information, the keyboard extension determines the response result to the operation information through the input method kernel.

[0094] The input method kernel is the core component of an input method, responsible for implementing its basic functions and logic. This includes, but is not limited to, modules such as keyboard input parsing, text encoding conversion, candidate word generation, language model processing, and dictionary management. Specifically, the keyboard input parsing module converts operation information into an internal encoding format that the computer can understand, such as ASCII or other character encodings; the text encoding conversion module converts the encoding output by the keyboard input parsing module into an encoding format that the input method can process, such as UTF-8; the candidate word generation module generates and sorts multiple candidate words based on the operation information (which conforms to the encoding format that the input method can process), combined with the language model and dictionary; the language model processing module predicts candidate words based on the operation information; and the dictionary management module manages custom or historically selected candidate words. When the keyboard extension integrates the input method kernel, the keyboard extension, after receiving operation information, determines the response result based on the input method kernel.

[0095] In this way, the keyboard extension can independently determine the response result and quickly feed it back to the host application, which helps to improve input efficiency. At the same time, it has a wide range of applications and does not depend on a specific network environment, and can still be implemented in weak network or no network environment.

[0096] In one embodiment of this disclosure, the operation information is at least one character selected in the second keyboard area, and the response result is a candidate word corresponding to at least one character; the screen on which the virtual keyboard input method is applied also includes an upper screen area; after displaying the response result for the operation information in the auxiliary input area through keyboard extension, the virtual keyboard input method further includes: in response to the selection operation of the candidate word, displaying the selected candidate word in the upper screen area.

[0097] Here, the operation information refers to the selection of at least one character in the second keyboard area. The response result is a candidate word corresponding to at least one character. After displaying the candidate word corresponding to at least one character in the auxiliary input area, if a selection operation for a candidate word is received, the selected candidate word is displayed in the upper screen area, which can be the area where the input entry (such as the input box) is located. The selection operation for a candidate word can refer to a trigger operation (such as a click operation) on any of the displayed candidate words, in which case the selected candidate word is the triggered candidate word; or it can refer to a trigger operation on the confirmation key in the second keyboard area, in which case the selected candidate word is the first candidate word in the displayed list. The confirmation key can be the space bar or the Enter key, etc.

[0098] In the above method, by responding to the selection operation of candidate words, the selected candidate words are displayed on the upper screen area. The complete input method on-screen function can be realized through the separate auxiliary input area and the second keyboard area. The operation method is similar to that of traditional input methods, which can improve ease of use and reduce the learning and use threshold of this embodiment.

[0099] In one embodiment of this disclosure, in response to receiving operation information for a second keyboard area, sending the operation information from the host application to the keyboard extension via a first inter-process communication includes: in response to an operation on the second keyboard area, determining the trigger position of the operation on the second keyboard area; determining the key corresponding to the trigger position within the custom keyboard container based on the key layout within the custom keyboard container and the scaling ratio of the custom keyboard container; generating operation information based on the determined key; and sending the operation information from the host application to the keyboard extension via the first inter-process communication.

[0100] Here, in response to an operation on the second keyboard area, the trigger position of the operation on the second keyboard area (referred to as the first trigger position) is determined. This can be achieved by using the event handling mechanism provided by the operating system to obtain the position of the operation on the screen, which is then used as the first trigger position. Since the keys within the custom keyboard container may be custom and may be scaled, the key layout within the custom keyboard container (including the position range of each key within the custom keyboard container before scaling) and the scaling ratio of the custom keyboard container need to be considered. For example, based on the scaling ratio of the custom keyboard container, the first trigger position is converted into a second trigger position within the custom keyboard container before scaling. The second trigger position is then compared with the position range of each key in the key layout, and the key corresponding to the position range falling within the second trigger position is determined as the target key (i.e., the determined key). Then, based on the target key, operation information (such as the character corresponding to the target key) is determined, and this operation information is sent from the host application to the keyboard extension via a first inter-process communication.

[0101] The above method, combining the key layout and scaling ratio of a custom keyboard container, determines the target key, accurately identifying the specific key pressed by the object and determining the object's intent. It is applicable to different key layouts and scaling ratios. For example, on terminal screens with different resolutions, the custom keyboard container typically scales based on the object's scaling operation. In this case, the embodiments of this disclosure can still accurately identify the key, improving input accuracy.

[0102] In one embodiment of this disclosure, the host application embeds a proxy toolkit; operations performed by the host application in this embodiment can be performed by the proxy toolkit.

[0103] Here, a proxy toolkit corresponding to a third-party keyboard application can be pre-embedded in the host application. This proxy toolkit, such as a Software Development Kit (SDK), integrates various tools for implementing the embodiments of this disclosure and can be directly invoked by the host application. For example, the proxy toolkit provides a communication tool, which the host application invokes to establish a first inter-process communication between the host application and the keyboard extension; the proxy toolkit also provides a keyboard display tool, which the host application invokes to display a second keyboard area in a custom keyboard container.

[0104] In the above approach, the proxy toolkit integrates various tools for implementing specific functions. These tools can be directly called by the host application, eliminating the need to repeatedly write code in the host application to implement these functions, greatly improving code reusability and reducing redundant code. The proxy toolkit provides the host application with a rich set of functions. Through simple calls, the host application can easily obtain functions that it did not originally have, facilitating rapid expansion of application functionality. The proxy toolkit encapsulates the underlying logic for implementing complex functions. Developers of the host application only need to focus on how to call these tools to achieve business goals, without needing to delve into the underlying implementation details, which can reduce development difficulty, shorten the development cycle, and improve development efficiency.

[0105] In steps 510-540 above, to overcome the limitation of the virtual keyboard of the third-party input method being displayed in a native keyboard container with fixed size and position, this embodiment hides the first keyboard area in the third-party virtual keyboard and displays a second keyboard area in a custom keyboard container with adjustable size and position to replace the first keyboard area. Since the size and position of the custom keyboard container are adjustable, that is, the size and position of the second keyboard area are adjustable, the obstruction of key information on the screen can be reduced, so that the object does not need to be frequently adjusted to see the obscured information. Since the custom keyboard container is introduced outside the mechanism of the original native keyboard container, the content entered through the second keyboard area cannot be transmitted to the third-party input method and obtain a response through the inherent mechanism. Therefore, this embodiment establishes cross-process communication between the host application and the keyboard extension of the third-party keyboard application. Through this cross-process communication, the operation information for the second keyboard area in the custom keyboard container is sent to the keyboard extension, so that the keyboard extension can respond to the operation information, obtain the response result, and display the response result in the auxiliary input area, thereby achieving the effect of transmitting the information that the input method needs to transmit without using the original native keyboard container mechanism. The embodiments disclosed herein can reduce screen obstruction of the host application when the host application needs to call a third-party input method, thereby improving the information transmission efficiency of the host application and the input efficiency of the input method.

[0106] Below, we will focus on providing a more detailed description of some of the steps.

[0107] Detailed description of step 510 Figure 5 Step 510 shown describes the process of establishing the first inter-process communication between the host application and the keyboard extension. The communication methods used include UIPasteboard, URL Scheme, Universal Link, App Group, etc. Next, we will take socket communication as an example for explanation.

[0108] In one embodiment of this disclosure, reference is made to Figure 12 , Figure 5 Step 510 shown includes: Step 1210: Send a call request to the keyboard extension through the host application; Step 1220: In response to the call request via keyboard extension, generate a socket connection request based on the host application's port number and send the socket connection request to the host application. Step 1230: Receive a response to the socket connection request from the host application via keyboard extension, thereby establishing socket communication as the first inter-process communication.

[0109] Steps 1210 to 1230 are described in detail below.

[0110] In step 1210, a call request is sent to the keyboard extension via the host application.

[0111] When the host application needs to invoke the keyboard extension, it sends an invocation request to the keyboard extension to start the keyboard extension.

[0112] In step 1220, in response to the call request via keyboard extension, a socket connection request is generated based on the host application's port number and sent to the host application.

[0113] Here, we take socket communication as an example. The keyboard extension is pre-configured with the host application's port number. After receiving a call request, the keyboard extension generates a socket connection request based on the host application's port number and sends the socket connection request to the host application. It is worth noting that the generation process of the socket connection request requires not only the host application's port number but also the host application's Internet Protocol (IP) address. Considering that the host application and the keyboard extension are located on the same terminal, the host application's IP address can be assumed to be the loopback address, i.e., Localhost.

[0114] In step 1230, a response to the socket connection request is received from the host application via keyboard extension, thereby establishing socket communication as the first inter-process communication.

[0115] After receiving a socket connection request from the keyboard extension, the host application returns a response to the socket connection request. The keyboard extension receives the response to the socket connection request, thereby establishing socket communication between the host application and the keyboard extension. This socket communication is the first inter-process communication.

[0116] In one embodiment of this disclosure, after sending a socket connection request to the host application, the virtual keyboard input method further includes: if no response to the socket connection request is received after a timeout period, determining that the first inter-process communication between the host application and the keyboard extension has failed.

[0117] A timeout period is set here. If no response to the socket connection request is received after the timeout period, it is determined that the first inter-process communication between the host application and the keyboard extension has failed. This avoids indefinite waiting and allows other measures to be taken after the first inter-process communication fails, such as the host application calling the native keyboard component.

[0118] In steps 1210 to 1230 above, socket communication is established through interaction between the host application and the keyboard extension. Compared with other communication methods, socket communication has the following advantages: 1) Strong real-time performance: both parties can transmit data in real time, avoiding the additional performance overhead caused by polling; 2) One-to-one communication between the host application and the keyboard extension: third parties cannot intervene, ensuring strong security and eliminating the risk of privacy leakage; 3) Support for bidirectional communication: the host application can send data to the keyboard extension, and the keyboard extension can also send data to the host application; 4) No restrictions on the format of the data transmitted during the communication process, ensuring strong universality; 5) No need to restrict the host application and the keyboard extension to belong to the same App Group, ensuring strong universality.

[0119] In one embodiment of this disclosure, reference is made to Figure 13 , Figure 12 Following step 1230 shown, the following is also included: Step 1310: Generate a signature based on the application identifier of the host application through the host application, generate a data packet to be verified based on the application identifier and the signature, and send the data packet to be verified to the keyboard extension through the first inter-process communication. Step 1320: Obtain the application identifier and signature from the data packet to be verified through keyboard extension, and verify the signature based on the application identifier; Reference Figure 13 , Figure 12 Step 520 shown can be updated to step 1330: Step 1330: When the signature verification is successful, display a third-party virtual keyboard in the native keyboard container via keyboard extension.

[0120] Steps 1310 to 1330 are described in detail below.

[0121] In step 1310, a signature is generated based on the application identifier of the host application, a data packet to be verified is generated based on the application identifier and the signature, and the data packet to be verified is sent to the keyboard extension through the first inter-process communication.

[0122] After the initial inter-process communication between the host application and the keyboard extension is successfully established, the host application generates a signature based on its application identifier (such as the Bundle ID in iOS). The signature algorithm used is not limited; for example, it can be a hash algorithm. Besides the host application identifier, other factors may be considered in generating the signature, and there are no restrictions on these factors.

[0123] Then, the host application generates a data packet to be verified based on the application identifier and signature, and sends the data packet to the keyboard extension via the first inter-process communication.

[0124] In step 1320, the application identifier and signature are obtained from the data packet to be verified through keyboard extension, and the signature is verified based on the application identifier.

[0125] After receiving the data packet to be verified, the keyboard extension obtains the application identifier and signature from the data packet and verifies the signature based on the application identifier using the same signature algorithm as in step 1310.

[0126] Taking a hash algorithm as an example, in step 1310, the host application performs a hash operation on the application identifier of the host application (the hash algorithm used is not limited, such as MD5 algorithm, SHA-1 algorithm, etc.) to obtain the first signature; in step 1320, the keyboard extension performs a hash operation on the application identifier obtained from the data packet to be verified to obtain the second signature, and determines that the signature verification is successful when the second signature is the same as the first signature, and determines that the signature verification fails when the second signature is different from the first signature.

[0127] In addition to referencing the application identifier of the host application, a key can also be introduced during the signing process. For example, in step 1310, the host application performs a hash operation on the predetermined key and the application identifier of the host application to obtain a first signature; in step 1320, the keyboard extension performs a hash operation on the predetermined key and the application identifier obtained from the data packet to be verified to obtain a second signature, and determines that the verification is successful if the second signature is the same as the first signature, and determines that the verification fails if the second signature is different from the first signature. The predetermined key can be negotiated in advance.

[0128] In step 1330, when the signature verification is successful, a third-party virtual keyboard is displayed in the native keyboard container via keyboard extension.

[0129] When the signature verification is successful, it proves that the first inter-process communication is trustworthy, so subsequent steps can be executed; when the signature verification fails, it proves that although the first inter-process communication is usable, it is not trustworthy, so we can try to re-establish the first inter-process communication, or call the native keyboard component through the host application.

[0130] It is worth noting that during the transmission of operation information and response results through the first inter-process communication, there is no need to perform signature-verification, thus saving communication resources.

[0131] In steps 1310 to 1330 above, after the first inter-process communication between the host application and the keyboard extension is successfully established, the host application signs based on the application identifier, and the keyboard extension verifies the signature based on the application identifier. Through this signature-verification mechanism, the privacy and security of the first inter-process communication can be ensured, and the accuracy of the data transmitted through the first inter-process communication can also be ensured. That is, the response result displayed in the auxiliary input area corresponds to the operation information of the object input, thereby improving input efficiency and input accuracy.

[0132] In one embodiment of this disclosure, reference is made to Figure 14 , Figure 13 Step 1310 shown can be updated to step 1410: Step 1410: Through the host application, generate a signature based on the host application's port number, application identifier, predefined key, and timestamp, and send the data packet to be verified to the keyboard extension via the first inter-process communication; Reference Figure 14 , Figure 13 Step 1320 shown can be updated to step 1420: Step 1420: Obtain the application identifier, timestamp, and signature from the data packet to be verified via keyboard extension. Verify the signature based on the port number, application identifier, pre-defined key, and timestamp.

[0133] Steps 1410 to 1420 are described in detail below.

[0134] In step 1410, a signature is generated by the host application based on the host application's port number, application identifier, predefined key, and timestamp. A data packet to be verified is generated based on the application identifier, timestamp, and signature, and the data packet to be verified is sent to the keyboard extension via the first inter-process communication.

[0135] For example, the host application performs a hash operation on the predefined key, the host application's port number, the host application's application identifier, and the timestamp (referring to the current timestamp) to obtain a first signature. Based on the application identifier, the timestamp, and the first signature, it generates a data packet to be verified and sends the data packet to be verified to the keyboard extension through the first inter-process communication.

[0136] In step 1420, the application identifier, timestamp, and signature are obtained from the data packet to be verified via keyboard extension. The signature is verified based on the port number, application identifier, pre-defined key, and timestamp.

[0137] For example, the keyboard extension obtains the application identifier, timestamp, and first signature from the data packet to be verified. It then performs a hash operation on the predetermined key, the host application's port number (i.e., the port number used by the keyboard extension to establish socket communication), the application identifier obtained from the data packet to be verified, and the timestamp obtained from the data packet to be verified, to obtain a second signature. The keyboard extension then compares the second signature with the first signature obtained from the data packet to be verified. If the second signature matches the first signature, the verification is successful; if the second signature does not match the first signature, the verification fails.

[0138] In steps 1410 to 1420 above, the hash operation takes into account multiple factors such as port number, application identifier, timestamp, and pre-defined key. The fusion of multiple factors makes the signature generation more complex and difficult to crack, making the entire signature-verification process more rigorous and reliable. For example, it is difficult for an attacker to forge a valid signature without knowing the key, and the timestamp can be used to resist replay attacks.

[0139] In one embodiment of this disclosure, reference is made to Figure 15 , Figure 12 Step 1220 shown includes: Step 1510: Generate the i-th socket connection request based on the i-th port number of the host application, and send the i-th socket connection request to the host application; Step 1520: If no response to the connection request for the i-th socket is received after the timeout period, and i is less than N, proceed to the (i+1)-th round of the loop. Figure 15 (This is represented as executing i=i+1). Reference Figure 15 , Figure 12 Step 1230 shown can be updated as follows: Step 1530: When a response to the i-th socket connection request is received after the timeout period, socket communication is established as the first inter-process communication.

[0140] Steps 1510 to 1530 are described in detail below.

[0141] In step 1510, an i-th socket connection request is generated based on the i-th port number of the host application, and the i-th socket connection request is sent to the host application.

[0142] If the host application has N port numbers, and N is an integer greater than 1, the keyboard extension can execute at least one round of looping to establish socket communication between the host application and the keyboard extension. For ease of understanding, we will take the i-th round of looping as an example, where the initial value of i is 1.

[0143] In the i-th iteration, the keyboard extension generates the i-th socket connection request based on the i-th port number of the host application and sends the i-th socket connection request to the host application.

[0144] In step 1520, if no response to the connection request for the i-th socket is received after the timeout period and i is less than N, the (i+1)-th iteration of the loop begins.

[0145] If no response to the i-th socket connection request is received within the timeout period after the keyboard extension sends the i-th socket connection request to the host application, it proves that the i-th port number of the host application is occupied by another process. Therefore, further judgment is performed: when i is less than N, it proves that the host application has other port numbers that it has not tried to connect to, so it enters the (i+1)-th round of the loop; when i equals N, it proves that all port numbers of the host application are occupied, so the loop ends. At this time, the socket communication between the host application and the keyboard extension fails to be established.

[0146] In step 1530, when a response to the i-th socket connection request is received after the timeout period, socket communication is established as the first inter-process communication.

[0147] If a response to the i-th socket connection request is received within the timeout period after the keyboard extension sends the i-th socket connection request to the host application, the loop ends, and the socket communication between the host application and the keyboard extension is successfully established.

[0148] In steps 1510 to 1530 above, the keyboard extension continuously attempts to establish socket communication through various port numbers of the host application. If a port number of the host application cannot be used to establish socket communication due to reasons such as being occupied by other processes, it will automatically try the next port number, rather than considering the socket connection establishment a failure due to a single port number problem. This gives the keyboard extension and the host application more opportunities to establish socket communication. In addition, by setting a timeout duration, it will not wait indefinitely when attempting to establish socket communication, avoiding the waste of a lot of computing resources due to waiting for invalid connections.

[0149] Detailed description of step 530 Figure 5 Step 530 shown describes the process of switching from non-floating mode to floating mode. This switching process can be performed directly or under specific conditions. Several examples of the latter will be explained below.

[0150] In one embodiment of this disclosure, reference is made to Figure 16 , Figure 5 Step 530 shown includes: Step 1610: When the size of the native keyboard container exceeds the first threshold, hide the first keyboard area; Step 1620: Display the second keyboard area in a custom keyboard container through the host application, wherein the size of the custom keyboard container does not exceed the first threshold.

[0151] Steps 1610 to 1620 are described in detail below.

[0152] In step 1610, when the size of the native keyboard container exceeds the first threshold, the first keyboard area is hidden.

[0153] Here, considering that if the size of the native keyboard container is too large, it may obscure too much key information on the screen, the first keyboard area can be hidden when the size of the native keyboard container exceeds the first threshold; when the size of the native keyboard container does not exceed the first threshold, the third-party virtual keyboard, including the auxiliary input area and the first keyboard area, continues to be displayed, that is, the non-floating mode remains unchanged.

[0154] The statement that the size of the native keyboard container exceeds the first threshold can be interpreted as: the product of the width and height of the native keyboard container exceeds the first threshold; or, the width of the native keyboard container exceeds the first threshold; or, the height of the native keyboard container exceeds the first threshold. For example, if the size of the native keyboard container exceeds the first threshold, it means that the height of the native keyboard container exceeds the first threshold, and the first threshold is 200. Figure 17 As shown, the first keyboard area is hidden because the height of the native keyboard container, 250, exceeds the first threshold, 200.

[0155] In step 1620, a second keyboard area is displayed in a custom keyboard container by the host application, wherein the size of the custom keyboard container does not exceed a first threshold.

[0156] Here, the second keyboard area is displayed within a custom keyboard container by the host application, and the size of the custom keyboard container does not exceed a first threshold. This results in a smaller custom keyboard container compared to the native keyboard container, reducing the obstruction of key information on the screen.

[0157] The size of the custom keyboard container can be preset or proportionally scaled down from the size of the native keyboard container; there are no restrictions on this. Similarly, there are no restrictions on the position of the custom keyboard container; it can be placed anywhere except where the native keyboard container would be, provided that the custom keyboard container does not overlap with the native keyboard container.

[0158] like Figure 17 As shown, the height of the custom keyboard container is 150, which does not exceed the first threshold of 200.

[0159] In steps 1610 to 1620 above, considering that the native keyboard container will obscure too much key information on the screen when it is too large, the non-floating mode is switched to the floating mode, that is, the first keyboard area is hidden and the second keyboard area is displayed in a custom keyboard container that is smaller than the native keyboard container. In this way, the obstruction of key information on the screen can be reduced.

[0160] In one embodiment of this disclosure, reference is made to Figure 18 , Figure 5 Step 530 shown includes: Step 1810: Identify the key subjects on the screen where the virtual keyboard input method is applied; Step 1820: When the native keyboard container obscures the identified key subject, hide the first keyboard area; Step 1830: Display the second keyboard area in the custom keyboard container through the host application, so that the custom keyboard container does not obscure the key subject.

[0161] Steps 1810 to 1830 are described in detail below.

[0162] In step 1810, key subjects on the screen to which the virtual keyboard input method is applied are identified.

[0163] Here, target recognition is performed on the screen where the virtual keyboard input method is applied to obtain the key subjects on the screen. The type of target to be recognized is not limited, such as faces, bodies, animals, etc. Similarly, the target recognition algorithm used is not limited; it can be feature-based algorithms (such as scale-invariant feature transform algorithms, accelerated robust feature algorithms, histogram of oriented gradients algorithms, etc.) or deep learning-based algorithms (such as convolutional neural network algorithms, etc.).

[0164] In step 1820, the first keyboard area is hidden when the native keyboard container obscures the identified key subject.

[0165] Step 1810 determines the area occupied by the key subject on the screen. Since the position and size of the native keyboard container are known, it's possible to determine whether the native keyboard container obscures the key subject. For example, if the area occupied by the native keyboard container and the key subject on the screen overlaps, it's determined that the native keyboard container obscures the key subject; if the area occupied by the native keyboard container and the key subject on the screen does not overlap, it's determined that the native keyboard container does not obscure the key subject.

[0166] When the native keyboard container obscures the identified key subject (i.e., the native keyboard container obscures key information on the screen), the first keyboard area is hidden; when the native keyboard container does not obscure the identified key subject, the third-party virtual keyboard containing the auxiliary input area and the first keyboard area continues to be displayed, that is, the non-floating mode remains unchanged.

[0167] In one embodiment of this disclosure, steps 1810 to 1830 may be executed after step 520; or, step 1810 may be executed before step 520, and steps 1820 to 1830 may be executed after step 520. The latter approach enables accurate identification of key subjects before displaying the third-party virtual keyboard, avoiding the inability to identify or inaccurate identification caused by obscuring key subjects.

[0168] In step 1830, the second keyboard area is displayed in a custom keyboard container through the host application, so that the custom keyboard container does not obscure the key subject.

[0169] Since the area occupied by the key subject on the screen can be determined through step 1810, a second keyboard area can be displayed in a custom keyboard container so that the custom keyboard container does not obscure the key subject. For example, an area outside the area occupied by the key subject on the screen can be arbitrarily selected as the custom keyboard container.

[0170] like Figure 19 As shown, the key subject region 1910 where the identified key subject (human body) is located is shown. It can be found that there is an overlapping area between the original keyboard container and the key subject region 1910. However, after switching from non-floating mode to floating mode, there is no overlapping area between the custom keyboard container and the key subject region 1910.

[0171] In steps 1810 to 1830 above, by identifying key subjects on the screen where the virtual keyboard input method is applied, it is possible to accurately and effectively determine whether the native keyboard container obscures the key subjects. When the native keyboard container obscures the key subjects, the first keyboard area is hidden, while the second keyboard area is displayed in the custom keyboard container that does not obscure the key subjects. This reduces the obstruction of key information on the screen by the native keyboard container (or reduces the obstruction of key information on the screen by the third-party virtual keyboard), while avoiding the obstruction of key information on the screen by the custom keyboard container (or avoiding the obstruction of key information on the screen by the second keyboard area).

[0172] In one embodiment of this disclosure, reference is made to Figure 20 , Figure 19 Step 1830 shown includes: Step 2010: Identify non-critical subject areas on the screen that do not contain critical subjects; Step 2020: Generate reference lines based on key subjects, and divide non-key subject areas based on reference lines to obtain multiple divided areas; Step 2030: Based on the area of ​​each of the multiple divided regions, determine the target region among the multiple divided regions; Step 2040: Display a custom keyboard container in the target area, and display a second keyboard area in the custom keyboard container.

[0173] Steps 2010 to 2040 are described in detail below.

[0174] In step 2010, non-critical subject areas on the screen that do not contain critical subjects are identified.

[0175] Identifying the key subject also means being able to determine the key subject area where the key subject is located. Based on this, the area on the screen other than the key subject area is determined as the non-key subject area.

[0176] In step 2020, reference lines are generated based on key subjects, and non-key subject areas are divided based on the reference lines to obtain multiple divided areas.

[0177] For example, reference lines can be obtained by extending the edges of the key subject area, or by extending the longitudinal and transverse center lines of the key subject. Then, non-key subject areas are divided based on the reference lines to obtain multiple subdivided areas.

[0178] like Figure 21 As shown, one reference line can be obtained by extending the vertical centerline of the key entity, and another reference line can be obtained by extending the horizontal centerline of the key entity. Based on these two reference lines, the non-key entity area can be divided into multiple subdivided areas, such as... Figure 21 Regions 1 to 4 in the middle.

[0179] In step 2030, the target region is determined among the multiple partitioned regions based on the area of ​​each of the partitioned regions.

[0180] For example, determine the area of ​​each divided region and designate the region with the largest area as the target region. The reasons for choosing the largest region as the target region are as follows: From a spatial perspective, a larger region provides more space for the custom keyboard container, preventing it from becoming too small and causing an overcrowded layout in the second keyboard area, thus providing a more comfortable input experience and reducing accidental touches; from a visual perspective, a larger region attracts more attention and is easier for the user to locate during operation, meaning the user can find the second keyboard area more quickly when needed; and from the perspective of avoiding interference, a larger region can accommodate custom keyboard containers that are further away from the key subject, reducing interference and making it easier for the user to view the key subject.

[0181] like Figure 21 As shown, among regions 1 to 4, region 1 has the largest area, so region 1 is determined as the target region.

[0182] In step 2040, a custom keyboard container is displayed in the target area, and a second keyboard area is displayed in the custom keyboard container.

[0183] Here, a custom keyboard container is displayed in the target area, for example in... Figure 21 In the middle, a custom keyboard container is located in the lower left corner of the target area. Then, a second keyboard area is displayed within the custom keyboard container.

[0184] In steps 2010 to 2040 above, reference lines are generated based on the key subject, and the non-key subject area is divided based on the reference lines to obtain multiple divided areas. The area with the largest area among these divided areas is used as the target area to accommodate the custom keyboard container. This approach serves several purposes: firstly, it avoids the custom keyboard container becoming too small due to space constraints, resulting in an overly crowded layout of the second keyboard area, thus providing a more comfortable input experience and reducing accidental touches; secondly, it makes it easier for the user to notice and locate the second keyboard area; and thirdly, it reduces interference from the second keyboard area on the key subject, making it easier for the user to view the key subject when performing input operations through the second keyboard area.

[0185] In one embodiment of this disclosure, reference is made to Figure 22 , Figure 5 Step 530 shown includes: Step 2210: Determine the first semantics before the screen displays the third-party virtual keyboard when the virtual keyboard input method is applied; Step 2220: Determine the second semantics after the third-party virtual keyboard is displayed on the screen; Step 2230: When the matching degree between the first semantic and the second semantic is less than the second threshold, hide the first keyboard area and display the second keyboard area in the custom keyboard container through the host application.

[0186] Steps 2210 to 2230 are described in detail below.

[0187] In step 2210, the first semantics of the virtual keyboard input method application before the screen displays a third-party virtual keyboard are determined.

[0188] Before displaying a third-party virtual keyboard on the screen, image semantic analysis is performed on the image displayed on the screen to obtain the first semantic meaning. Image semantic analysis refers to using computer algorithms to enable a computer to understand and interpret images, obtaining semantic information from them. This includes recognizing high-level concepts such as objects, scenes, and actions within the image. This disclosure does not limit the specific method of image semantic analysis; for example, it can be implemented using deep learning models, such as visual language models.

[0189] In step 2220, the second semantics after the third-party virtual keyboard is displayed on the screen are determined.

[0190] After displaying a third-party virtual keyboard on the screen, image semantic analysis is performed on the image displayed on the screen to obtain the second semantic meaning.

[0191] In step 2230, when the matching degree between the first semantic and the second semantic is less than the second threshold, the first keyboard area is hidden, and the second keyboard area is displayed in a custom keyboard container through the host application.

[0192] Here, the first semantic and the second semantic are matched to obtain the matching degree. The matching method is not limited; for example, both the first and second semantics can be converted into vectors using a word vector model (such as the Word2Vec model), and the similarity between the vectors can be determined as the matching degree between the first and second semantics. Alternatively, the edit distance between the first and second semantics (such as the Levenshtein distance) can be determined, and the matching degree between the first and second semantics can be determined based on the edit distance. Here, the matching degree and the edit distance are negatively correlated.

[0193] When the matching degree between the first semantic and the second semantic is less than the second threshold, it is proven that the display of the third-party virtual keyboard has seriously changed the semantics of the screen (i.e., seriously obscured key information on the screen). Therefore, the first keyboard area is hidden, and the second keyboard area is displayed in a custom keyboard container through the host application. When the matching degree between the first semantic and the second semantic is not less than the second threshold, it is proven that the display of the third-party virtual keyboard has slightly changed the semantics of the screen or has not changed the semantics of the screen. Therefore, the third-party virtual keyboard, including the auxiliary input area and the first keyboard area, continues to be displayed, that is, the non-floating mode remains unchanged.

[0194] In steps 2210 to 2230 above, the first semantic and the second semantic are matched to obtain the matching degree. When the matching degree between the first semantic and the second semantic is less than the second threshold, it proves that the display of the third-party virtual keyboard has seriously changed the semantics of the screen. Therefore, the non-floating mode is switched to the floating mode to restore the first semantic as much as possible, that is, to ensure that the display of the virtual keyboard does not affect the semantic output of the screen, so that the object can obtain accurate information from the screen.

[0195] In one embodiment of this disclosure, reference is made to Figure 23 , Figure 22 Step 2230 shown includes: Step 2310: When the matching degree between the first semantic and the second semantic is less than the second threshold, hide the first keyboard area; Step 2320: Display a custom keyboard container according to a predetermined size and position through the host application, and display a second keyboard area within the custom keyboard container; Step 2330: Determine the third semantics of the screen after displaying the second keyboard area; Step 2340: When the matching degree between the first semantic and the third semantic is less than the second threshold, adjust the size and position of the custom keyboard container, and return to the step of obtaining the third semantic after the second keyboard area is displayed on the screen, until the matching degree between the first semantic and the third semantic is not less than the second threshold.

[0196] Steps 2320 to 2340 are described in detail below.

[0197] In step 2320, a custom keyboard container is displayed by the host application according to a predetermined size and position, and a second keyboard area is displayed in the custom keyboard container.

[0198] Here, the size and position of the custom keyboard container can be predetermined (preset). While hiding the first keyboard area, the custom keyboard container can be displayed according to the predetermined size and position, and the second keyboard area can be displayed in the custom keyboard container.

[0199] In step 2330, a third semantic is determined after the screen displays the second keyboard area.

[0200] After the second keyboard area is displayed on the screen, image semantic analysis is performed on the image displayed on the screen to obtain the third semantic.

[0201] In step 2340, when the matching degree between the first semantic and the third semantic is less than the second threshold, the size and position of the custom keyboard container are adjusted, and the step of obtaining the third semantic after the second keyboard area is displayed is returned until the matching degree between the first semantic and the third semantic is not less than the second threshold.

[0202] Here, the first semantic and the third semantic are matched to obtain the matching degree between the first semantic and the third semantic. When the matching degree between the first semantic and the third semantic is less than the second threshold, it proves that the display of the second keyboard area has seriously changed the semantics of the screen (i.e., seriously obscured the key information on the screen), and the size and position of the custom keyboard container are not appropriate. Therefore, the size and position of the custom keyboard container are adjusted, and step 2330 is returned to be executed until the matching degree between the first semantic and the third semantic is not less than the second threshold.

[0203] There are no restrictions on how to adjust the size and position of the custom keyboard container. For example, it can be adjusted randomly within a preset range, or it can be adjusted to an unused combination given N preset combinations of size and position (e.g., size 1-position 1, size 2-position 2, ..., size N-position N).

[0204] In steps 2310 to 2340 above, a custom keyboard container is first displayed at a predetermined size and position, and a second keyboard area is displayed within the custom keyboard container. Considering that the predetermined size and position may not be suitable for the current situation, a third semantic meaning of the screen after displaying the second keyboard area is determined. When the matching degree between the first semantic meaning and the third semantic meaning is less than a second threshold, it proves that the predetermined size and position are not suitable, and the size and position of the custom keyboard container are adjusted. In this way, the size and position of the custom keyboard container can be intelligently adjusted, ensuring that the display of the second keyboard area does not significantly change the semantic meaning of the screen, making it easier for objects to obtain accurate information from the screen.

[0205] In one embodiment of this disclosure, reference is made to Figure 24 , Figure 5 Step 530 shown can be updated to step 2410: In step 2410, in response to receiving a virtual keyboard display mode switching operation, the first keyboard area is hidden, and the second keyboard area is displayed in a custom keyboard container through the host application.

[0206] Here, step 530 can be executed manually, that is, in response to receiving a virtual keyboard display mode switching operation, the first keyboard area is hidden, and the second keyboard area is displayed in a custom keyboard container through the host application.

[0207] This disclosure does not limit the triggering method for the virtual keyboard display mode switching operation. For example, the function entry for the virtual keyboard display mode switching function can be displayed in the native keyboard container, and the virtual keyboard display mode switching operation can be triggered through this function entry. Figure 25 As shown, the virtual keyboard display mode switching operation can be triggered through the function entry point of the virtual keyboard display mode switching function (i.e., function 6). For example, in response to receiving a trigger operation for the function entry point of function 6, both the non-floating mode option and the floating mode option are displayed. Then, in response to receiving a trigger operation for the displayed floating mode option, the non-floating mode is switched to floating mode. The virtual keyboard display mode switching operation includes the aforementioned trigger operation for the function entry point of function 6 and the trigger operation for the floating mode option.

[0208] For example, the virtual keyboard display mode switching operation can be triggered through the input field displayed by the host application. Figure 26 As shown, in response to a first trigger operation received for the input field (i.e., the input field displaying "Please enter a comment"), the system enters non-floating mode. Then, in response to a second trigger operation received for the input field (i.e., a virtual keyboard display mode switching operation), the system switches from non-floating mode to floating mode. The first and second trigger operations are different; for example, the first trigger operation might be a click operation, and the second trigger operation might be a long press operation; or the first trigger operation might be a single click operation, and the second trigger operation might be a double click operation.

[0209] The manual triggering method provided in step 2410 above grants the object the right to actively control the switching of the virtual keyboard display mode. The object can choose a suitable virtual keyboard display mode according to its own habits and ease of operation, ensuring the necessity and accuracy of the virtual keyboard display mode switching. Of course, in addition to supporting switching from non-floating mode to floating mode, this embodiment of the disclosure also supports switching from floating mode to non-floating mode.

[0210] Detailed description of step 540 In step 540, the keyboard extension can determine the response to the operation information independently, or it can determine the response to the operation information by means of additional resources provided by a third-party keyboard application. The latter case will be illustrated below.

[0211] In one embodiment of this disclosure, reference is made to Figure 27 , Figure 5Step 540 shown includes: Step 2710: In response to receiving at least one selected character, send at least one character from the host application to the keyboard extension via a first inter-process communication; Step 2720: Establish a second cross-process communication between the keyboard extension and the third-party keyboard application; Step 2730: Send at least one character to a third-party keyboard application via keyboard extension using second inter-process communication, and receive candidate words from the third-party keyboard application; Step 2740: Display candidate words in the auxiliary input area via keyboard extension.

[0212] Steps 2710 to 2740 are described in detail below.

[0213] In step 2710, in response to receiving at least one selected character, at least one character is sent from the host application to the keyboard extension via a first inter-process communication.

[0214] Here, taking the operation information as the selection of at least one character in the second keyboard area as an example, at least one character is first sent from the host application to the keyboard extension via a first inter-process communication.

[0215] In step 2720, a second cross-process communication is established between the keyboard extension and the third-party keyboard application.

[0216] Here, there are no restrictions on the communication method for the second inter-process communication; for example, UIPasteboard, URLScheme, Universal Link, App Group, etc., can be used. The second inter-process communication can be the same as or different from the first inter-process communication in terms of communication method; there are no restrictions on this.

[0217] like Figure 28 As shown, since the keyboard extension and the third-party keyboard application belong to the same App Group, a second cross-process communication can be established through the communication methods provided by the App Group.

[0218] It is worth noting that step 2720 can be executed after step 2710 or before step 2710, for example, simultaneously with step 510, and there is no limitation on this.

[0219] In step 2730, at least one character is sent to a third-party keyboard application via keyboard extension using a second inter-process communication, and candidate words are received from the third-party keyboard application.

[0220] Here, the keyboard extension sends at least one character to a third-party keyboard application via a second inter-process communication. The third-party keyboard application determines a candidate word corresponding to the at least one character and sends the candidate word to the keyboard extension via the second inter-process communication.

[0221] The method by which a third-party keyboard application determines candidate words corresponding to at least one character is not limited. For example, a third-party keyboard application can send at least one character to a server (third-party keyboard server), so that the server can determine candidate words corresponding to at least one character based on dictionary data and / or language models. In this way, there is no need to deploy dictionary data and / or language models locally on the keyboard extension (on the terminal), which can reduce the storage and computing pressure on the terminal.

[0222] In step 2740, candidate words are displayed in the auxiliary input area via keyboard expansion.

[0223] In steps 2710 to 2740 above, candidate words corresponding to at least one character are determined by using additional resources provided by a third-party keyboard application. The keyboard extension is mainly used as a relay and does not need to perform complex calculation tasks, which makes the keyboard extension more streamlined and easier to deploy and call.

[0224] Of course, the above is merely an example application of the second inter-process communication and does not constitute a limitation on the embodiments of this disclosure. For example, a keyboard extension can obtain dictionary data from a third-party keyboard application through the second inter-process communication and determine candidate words corresponding to at least one character based on the dictionary data. This ensures the real-time performance and accuracy of the local dictionary data of the keyboard extension, thereby improving the accuracy of the determined candidate words.

[0225] Synchronized Skin Renewal Program In one embodiment of this disclosure, synchronized skinning between the host application and a third-party virtual keyboard can be achieved. See also... Figure 29 , Figure 5 Following step 520, the following is also included: Step 2910: In response to the host application's display resource update, obtain the host application's first updated display resources; Step 2920: Query the display resource database through the host application to obtain the second updated display resource of the third-party virtual keyboard. The second updated display resource matches the first updated display resource. Step 2930: Send the second updated display resources from the host application to the keyboard extension via the first inter-process communication; Step 2940: Update the display resources of the third-party virtual keyboard according to the second updated display resources via keyboard extension.

[0226] Steps 2910 to 2940 are described in detail below.

[0227] In step 2910, in response to the host application's display resource update, the first updated display resource of the host application is obtained.

[0228] Here, display resources refer to the various elements and data used in the host application to build and render the visual parts of the user interface; display resources can also be called themes or skins. When the host application updates its display resources, the system retrieves the first updated display resources from the host application.

[0229] In step 2920, the host application queries the display resource database to obtain the second updated display resource of the third-party virtual keyboard, and the second updated display resource matches the first updated display resource.

[0230] Here, the host application queries the display resource database based on the first updated display resource to obtain the second updated display resource for the third-party virtual keyboard. The display resource database includes multiple candidate display resources for the third-party virtual keyboard, and the second updated display resource refers to the candidate display resource in the database that matches the first updated display resource. For example, a one-to-one matching relationship can be pre-defined between the multiple candidate display resources of the host application and the multiple candidate display resources of the third-party virtual keyboard. After determining the first updated display resource of the host application, the second updated display resource matching the first updated display resource can be determined based on the pre-defined matching relationship.

[0231] In one embodiment of this disclosure, the display resource database may be located in the cloud, for example, connected to a server. This reduces the storage pressure on the terminal's local storage. Of course, the possibility of storing the display resource database locally on the terminal is not excluded.

[0232] In one embodiment of this disclosure, obtaining the second updated display resource of a third-party virtual keyboard by querying the display resource database through a host application includes: querying the display resource database based on the metadata corresponding to the first updated display resource through the host application to obtain the second updated display resource corresponding to the metadata.

[0233] The metadata corresponding to a display resource refers to the data used to describe the display resource. Metadata can be preset and its data type is not limited. For example, metadata includes, but is not limited to, display resource type, display resource identifier, terminal model (referring to the terminal model to which the display resource is applicable), and operating system version (referring to the operating system version to which the display resource is applicable). Therefore, matching the first updated display resource with the second updated display resource can refer to matching the metadata corresponding to each of them.

[0234] Taking metadata as an example of resource type, suppose the resource type corresponding to the first updated display resource is "Vacation Theme". The display resource database includes candidate display resources A, B, and C from a third-party virtual keyboard. Candidate display resource A corresponds to the resource type "Ink Painting Theme", candidate display resource B corresponds to the resource type "Vacation Theme", and candidate display resource C corresponds to the resource type "Anime Theme". Since candidate display resource B matches (is the same as) the first updated display resource in terms of resource type, candidate display resource B is determined as the second updated display resource. This ensures consistency in resource type between the first and second updated display resources.

[0235] Taking metadata including resource type and terminal model as an example, suppose the resource type corresponding to the first updated display resource is "vacation theme" and the terminal model corresponding to the first updated display resource is "mobile phone". The display resource database includes candidate display resource B and candidate display resource D for a third-party virtual keyboard. Candidate display resource B corresponds to the resource type "vacation theme" and the terminal model corresponding to candidate display resource B is "mobile phone". Candidate display resource D corresponds to the resource type "vacation theme" and the terminal model corresponding to candidate display resource D is "tablet". Since candidate display resource B matches the first updated display resource in both resource type and terminal model (they are the same), candidate display resource B is determined as the second updated display resource. In this way, the consistency of the resource type between the first and second updated display resources is ensured, while also ensuring that the second updated display resource can be displayed normally on a mobile phone.

[0236] The above method utilizes metadata to query the display resource database, essentially establishing an indexing mechanism for it. The display resource database may contain a large number of candidate display resources; by filtering and selecting through metadata, matching candidate display resources can be quickly located without traversing the entire database, significantly improving query efficiency.

[0237] In one embodiment of this disclosure, the second updated display resource of the third-party virtual keyboard is obtained by querying the display resource database through the host application, including: matching the first updated display resource with a plurality of candidate display resources in the display resource database through the host application; and determining the candidate display resources whose matching degree with the first updated display resource is greater than a third threshold as the second updated display resource.

[0238] Here, the updated display resource is matched with each candidate display resource in the display resource database to obtain the matching degree between the updated display resource and each candidate display resource. Considering that display resources are usually presented in the form of images, the matching degree between display resources can be calculated by calculating the matching degree between images. For example, statistical features of display resources can be extracted, and the matching degree between the statistical features of different display resources can be calculated as the matching degree between different display resources. Statistical features include color histogram features, texture features, etc., where texture features can be extracted through the gray-level co-occurrence matrix. Alternatively, semantic features of display resources can be extracted using a deep learning model, and the matching degree between the semantic features of different display resources can be calculated as the matching degree between different display resources.

[0239] Then, candidate display resources whose matching degree with the first updated display resource is greater than the third threshold are determined as the second updated display resource. If there are multiple candidate display resources whose matching degree with the first updated display resource is greater than the third threshold, the candidate display resource with the highest matching degree is determined as the second updated display resource.

[0240] In the above method, by matching the first updated display resource with multiple candidate display resources in the display resource database, and then filtering the candidate display resources based on the matching degree to obtain the second updated display resource, it is possible to ensure that the second updated display resource is similar to the first updated display resource in terms of display style, thereby improving the visual experience. Furthermore, it can be implemented without annotating metadata, making it widely applicable and highly versatile.

[0241] In step 2930, the second updated display resources are sent from the host application to the keyboard extension via the first inter-process communication.

[0242] Here, the host application sends the second updated display resource to the keyboard extension via the first inter-process communication. In this way, the transmission of the second updated display resource is implemented locally on the terminal. The keyboard extension can obtain the second updated display resource without initiating a request to the outside world (i.e., outside the terminal), reducing the number of interaction steps with the outside world and improving the success rate of the keyboard extension obtaining the second updated display resource.

[0243] In step 2940, the display resources of the third-party virtual keyboard are updated according to the second updated display resources through keyboard extension.

[0244] Here, the keyboard extension updates the display resources of the third-party virtual keyboard based on the second updated display resources. The method of updating the display resources is not limited; for example, it can be an update of the background image, key icons, key animations, font styles, etc. (See reference...) Figure 30After the first update of the host application, the displayed resources are arranged with vacation patterns according to a specific rule (see...). Figure 30 The background image of pattern 3010 in the third-party virtual keyboard, after the second update, also has a background image with vacation patterns arranged in a specific pattern.

[0245] It is worth noting that the third-party virtual keyboard in step 2940 can refer to the third-party virtual keyboard before the first keyboard area is hidden (see [reference]). Figure 30 This can also refer to a third-party virtual keyboard hidden behind the first keyboard area. In the latter case, the host application can also update the display resources of the second keyboard area based on the updated display resources, such as... Figure 31 As shown, this ensures that the host application, auxiliary input area, and second keyboard area have a similar display style, thereby improving the visual experience.

[0246] In steps 2910 to 2940 above, when the host application updates its display resources, the second updated display resources of the third-party virtual keyboard are obtained by querying the first updated display resources of the host application, and the display resources of the third-party virtual keyboard are updated according to the second updated display resources, thereby ensuring that the host application and the third-party virtual keyboard are similar in display style, thus improving the visual experience; at the same time, it can realize automatic synchronous skinning, saving the additional time cost of manual skinning.

[0247] Steps 2910 to 2940 described above describe synchronous skinning from the host application to the third-party virtual keyboard. In one embodiment of this disclosure, synchronous skinning from the third-party virtual keyboard to the host application can also be achieved, thereby effectively expanding the application scenarios of synchronous skinning. For example, in response to an update of the display resources of the third-party virtual keyboard, the third updated display resource of the third-party virtual keyboard is obtained; the display resource database is queried through the keyboard extension (unlike the above, the display resource database here includes multiple candidate display resources of the host application) to obtain the fourth updated display resource of the host application, which matches the third updated display resource; the fourth updated display resource is sent from the keyboard extension to the host application through the first inter-process communication; and the host application updates the display resources according to the fourth updated display resource.

[0248] Example of the specific usage process of the virtual keyboard input method in this embodiment of the disclosure Next, taking a terminal using the iOS operating system as an example, we will illustrate a specific usage example of the virtual keyboard input method of this disclosure embodiment.

[0249] The embodiments disclosed herein are mainly applied to scenarios where a certain application (i.e., the host application) on a terminal invokes a virtual keyboard for typing, such as a floating keyboard scenario and a synchronized skinning scenario, which will be described separately below.

[0250] (a) Floating keyboard scenario like Figure 32 As shown, in response to a click on an input field displayed by the host application, the host application invokes a virtual keyboard. Invoking the virtual keyboard here refers to launching the third-party keyboard SDK embedded in the host application (corresponding to the proxy toolkit mentioned above). After launching, the third-party keyboard SDK notifies the keyboard extension to attempt to establish socket communication with it. Once the socket communication between the third-party keyboard SDK and the keyboard extension is successfully established, the keyboard extension displays the third-party virtual keyboard within the native keyboard container. Figure 32 (Not shown in the image) A third-party virtual keyboard includes an auxiliary input area and a first keyboard area. The size and position of the native keyboard container are fixed. The third-party keyboard SDK displays (e.g., around the input box) a prompt to long-press the input box to switch to floating mode. In response to a long-press operation on the input box, the keyboard extension constrains the height of the native keyboard container to hide the first keyboard area. Simultaneously, the third-party keyboard SDK displays a second keyboard area within a custom keyboard container, the size and position of which are variable. This achieves the switching from non-floating mode to floating mode. Floating mode reduces obstruction of key information on the screen and makes it easier for objects to perform input operations on the second keyboard area. If subsequent operation information for the second keyboard area is received, the operation information is sent from the third-party keyboard SDK to the keyboard extension via socket communication. The keyboard extension then displays the response to the operation information in the auxiliary input area.

[0251] (II) Synchronous Skin Changing Scenarios like Figure 33As shown, after the host application completes the skinning, it determines the metadata corresponding to the first updated theme skin data (i.e., the theme skin data of the currently used theme skin, corresponding to the first updated display resource mentioned above). Based on this metadata, the host application requests the second updated theme skin data (corresponding to the second updated display resource mentioned above) of the third-party virtual keyboard from the theme skin API. The theme skin API requests the server to query the theme skin database based on this metadata to obtain the second updated theme skin data. The theme skin database includes multiple candidate theme skin data for the third-party virtual keyboard. The host application sends the second updated theme skin data to the keyboard extension via socket communication. When the keyboard extension needs to display the third-party virtual keyboard, it displays the third-party virtual keyboard based on the second updated theme skin data, thus achieving the skinning of the third-party virtual keyboard. It is worth noting that the operations performed by the host application can be executed through the embedded third-party keyboard SDK.

[0252] In terms of underlying implementation, the embodiments disclosed herein mainly involve the following aspects.

[0253] (a) Socket communication mechanism like Figure 34 As shown, this embodiment of the disclosure implements a local socket communication mechanism, in which the keyboard extension acts as the client and the host application acts as the server. When the virtual keyboard is invoked, the keyboard extension requests the host application to establish socket communication (through a request-response method); when the virtual keyboard is dismissed, the host application notifies the keyboard extension to disconnect the socket communication (i.e., close the connection). Figure 34 In this example, the host application uses port number 8080.

[0254] (ii) Data transmission mechanism like Figure 35 As shown, this disclosure defines a mechanism for transmitting data based on socket communication. The host application and the keyboard extension can transmit data to each other in accordance with this mechanism. The transmitted data may include theme skin data, dictionary data, or other business-defined data. This mechanism can ensure the efficiency and stability of data transmission.

[0255] Next, we will explain in top-down order. Figure 35The data transmission mechanism is illustrated. First, a socket message (LocalSocketMsg) is defined. LocalSocketMsg consists of three parts: message type (MsgType), sender's application identifier (bundle id), and message body (Body). Then, LocalSocketMsg is encoded into a data packet body using an encoding / decoding protocol. The data packet body and data packet length together constitute the data packet used for transmission in socket communication. The encoding / decoding protocol, such as NSCoding, provides a standard way to convert data into a format suitable for storage or transmission and is widely used in the iOS operating system. Figure 35 It also shows the Transmission Control Protocol (TCP), which is the transport layer protocol on which socket communication relies. Of course, transport layer protocols are not limited to TCP.

[0256] (III) Verification Mechanism This disclosure also implements a verification mechanism. For example... Figure 34 As shown, after the host application establishes socket communication with the keyboard extension, if the host application receives a detection data packet (i.e., a ping data packet) from the keyboard extension, the host application generates a signature based on the host application's port number, application identifier, predefined key, and timestamp. Then, based on the host application's application identifier, timestamp, and signature, it generates a data packet to be verified and sends the data packet to the keyboard extension. Figure 34 (Only the process of sending the application identifier of the host application is shown). After receiving the data packet to be verified, the keyboard extension verifies the signature using the same signature rules. Only when the signature verification is successful does it transmit theme skin data, dictionary data, or other business-defined data to the host application via socket communication. This enhances privacy and security during socket communication.

[0257] The signature rules are as follows: Signature = HASH(Host application's port number + Host application's application identifier + Pre-defined key + Timestamp).

[0258] (iv) SDK encapsulation To further improve the ease of access for host applications, the aforementioned socket communication mechanism, data transmission mechanism, and verification mechanism can be encapsulated into an SDK in the form of a Framework suitable for the iOS platform. By embedding this SDK in the host application, the host application can more easily access the communication framework of this disclosure embodiment and utilize its efficient and secure data transmission capabilities.

[0259] Description of apparatus and devices according to embodiments of this disclosure It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0260] It should be noted that in all specific embodiments of this disclosure, when processing based on operational information is required, permission or consent from the relevant parties is obtained first. Furthermore, the collection, use, and processing of operational information comply with relevant laws, regulations, and standards. In addition, when embodiments of this disclosure require obtaining operational information, separate permission or consent from the relevant parties is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the separate permission or consent of the relevant parties is the necessary operational information required for the normal operation of the embodiments of this disclosure obtained.

[0261] Reference Figure 36 , Figure 36 This is a schematic diagram of the structure of the virtual keyboard input device 3600 provided in an embodiment of the present disclosure. The virtual keyboard input device 3600 includes: The communication establishment module 3610 is used to establish the first inter-process communication between the host application and the keyboard extension of the third-party keyboard application; The keyboard display module 3620 is used to display a third-party virtual keyboard in the native keyboard container through keyboard extension. The third-party virtual keyboard includes an auxiliary input area and a first keyboard area. The size and position of the native keyboard container are fixed. The keyboard display module 3620 is also used to hide the first keyboard area and display the second keyboard area in a custom keyboard container through the host application. The size and position of the custom keyboard container are variable. The input response module 3630 is used to respond to receiving operation information for the second keyboard area, send the operation information from the host application to the keyboard extension through the first inter-process communication, and display the response result for the operation information in the auxiliary input area through the keyboard extension. In some embodiments, the communication establishment module 3610 is further configured to: Send a call request to the keyboard extension via the host application; Through keyboard extension, in response to a call request, a socket connection request is generated based on the host application's port number, and the socket connection request is sent to the host application; By using keyboard extensions, responses to socket connection requests are received from the host application, thereby establishing socket communication as the first inter-process communication.

[0262] In some embodiments, the virtual keyboard input device 3600 further includes a verification module (not shown), used for: A signature is generated based on the application identifier of the host application, a data packet to be verified is generated based on the application identifier and the signature, and the data packet to be verified is sent to the keyboard extension through the first inter-process communication. By using keyboard extension, the application identifier and signature are obtained from the data packet to be verified, and the signature is verified based on the application identifier. The keyboard display module 3620 is also used for: When the signature verification is successful, a third-party virtual keyboard is displayed in the native keyboard container via keyboard extension.

[0263] In some embodiments, the verification module is further configured to: Generate a signature based on the port number, application identifier, pre-defined key, and timestamp; Generate a data packet to be verified based on the application identifier, timestamp, and signature; The application identifier, timestamp, and signature are obtained from the data packet to be verified. The signature is verified based on the port number, application identifier, pre-defined key, and timestamp.

[0264] In some embodiments, the host application's port number includes N, where N is an integer greater than 1; the communication establishment module 3610 is further configured to: Execute at least one iteration of the loop, and in the i-th iteration, perform the following processing: Generate the i-th socket connection request based on the i-th port number of the host application, and send the i-th socket connection request to the host application; If no response to the connection request for the i-th socket is received after the timeout period, and i is less than N, the loop enters the (i+1)-th iteration. When a response to the i-th socket connection request is received after the timeout period, socket communication is established as the first inter-process communication.

[0265] In some embodiments, the keyboard display module 3620 is further configured to: When the size of the native keyboard container exceeds a first threshold, hide the first keyboard area; A second keyboard area is displayed in a custom keyboard container via the host application, wherein the size of the custom keyboard container does not exceed a first threshold.

[0266] In some embodiments, the keyboard display module 3620 is further configured to: Identify key subjects on the screen of a virtual keyboard input device application; When the native keyboard container obscures the identified key subject, hide the first keyboard area; By using the host application, a second keyboard area can be displayed within a custom keyboard container, ensuring that the custom keyboard container does not obscure the main content.

[0267] In some embodiments, the keyboard display module 3620 is further configured to: Identify non-critical subject areas on the screen that do not contain critical subjects; Reference lines are generated based on key subjects, and non-key subject areas are divided based on the reference lines to obtain multiple divided areas; Based on the area of ​​each of the multiple subdivided regions, the target region is determined among the multiple subdivided regions; Define a custom keyboard container in the target area, and display the second keyboard area within the custom keyboard container.

[0268] In some embodiments, the keyboard display module 3620 is further configured to: Determine the first semantics before the screen displays a third-party virtual keyboard for the virtual keyboard input device application; Determine the second semantic meaning after the third-party virtual keyboard is displayed on the screen; When the matching degree between the first semantic and the second semantic is less than the second threshold, the first keyboard area is hidden, and the second keyboard area is displayed in a custom keyboard container through the host application.

[0269] In some embodiments, the keyboard display module 3620 is further configured to: When the matching degree between the first semantic and the second semantic is less than the second threshold, the first keyboard area is hidden; The host application displays a custom keyboard container at a predetermined size and position, and a second keyboard area is displayed within the custom keyboard container. Determine the third semantics of the screen after displaying the second keyboard area; When the matching degree between the first semantic and the third semantic is less than the second threshold, adjust the size and position of the custom keyboard container, and return to the step of obtaining the third semantic after the second keyboard area is displayed on the screen, until the matching degree between the first semantic and the third semantic is not less than the second threshold.

[0270] In some embodiments, the operation information is at least one character selected in the second keyboard area, and the response result is a candidate word corresponding to at least one character; the input response module 3630 is further configured to: In response to receiving at least one selected character, at least one character is sent from the host application to the keyboard extension via a first inter-process communication; Establish a second cross-process communication between keyboard extensions and third-party keyboard applications; Through keyboard extension, at least one character is sent to a third-party keyboard application by means of a second cross-process communication, and candidate words are received from the third-party keyboard application; Display candidate words in the auxiliary input area using keyboard extension.

[0271] In some embodiments, the operation information is at least one character selected in the second keyboard area, and the response result is a candidate word corresponding to at least one character; the screen on which the virtual keyboard input device 3600 is applied also includes an upper screen area; the virtual keyboard input device 3600 further includes an upper screen module (not shown), used for: In response to the selection of candidate words, the selected candidate words are displayed in the upper screen area.

[0272] In some embodiments, the input response module 3630 is further configured to: In response to an operation on the second keyboard area, determine the trigger position of the operation on the second keyboard area; Based on the key layout within the custom keyboard container and the scaling ratio of the custom keyboard container, determine the key corresponding to the trigger position within the custom keyboard container; Based on the selected key presses, generate operation information; Operation information is sent from the host application to the keyboard extension via first inter-process communication.

[0273] In some embodiments, the native keyboard container is placed along the lower edge of the screen where the virtual keyboard input device 3600 is applied; the keyboard display module 3620 is further configured to: The native keyboard container is height-constrained, so that the height of the native keyboard container is constrained from a first height to a second height, so that the first keyboard area is hidden below the bottom edge of the screen. The first height is the common height of the auxiliary input area and the first keyboard area, and the second height is the height of the auxiliary input area.

[0274] In some embodiments, the host application embeds a proxy toolkit; the communication establishment module 3610 is also used for: The first cross-process communication between the host application and the keyboard extension is established using the communication tools in the proxy toolkit. The keyboard display module 3620 is also used for: Display a second keyboard area within a custom keyboard container using the keyboard display tool in the proxy toolkit.

[0275] In some embodiments, the virtual keyboard input device 3600 further includes a synchronized skinning module (not shown), used for: In response to updates to the host application's display resources, display resources after obtaining the first update from the host application; The host application queries the display resource database to obtain the second updated display resource of the third-party virtual keyboard, and the second updated display resource matches the first updated display resource. The second updated display resources are sent from the host application to the keyboard extension via the first inter-process communication; By extending the keyboard, the display resources of the third-party virtual keyboard are updated according to the second updated display resources.

[0276] Reference Figure 37 , Figure 37 The structural block diagram of a portion of the terminal 140 implementing the virtual keyboard input method of this embodiment includes: a radio frequency (RF) circuit 3710, a memory 3715, an input unit 3730, a display unit 3740, a sensor 3750, an audio circuit 3760, a wireless fidelity (WiFi) module 3770, a processor 3780, and a power supply 3790, among other components. Those skilled in the art will understand that... Figure 37 The terminal structure shown does not constitute a limitation on mobile phones or computers and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0277] The RF circuit 3710 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 3780; in addition, it transmits uplink data to the base station.

[0278] The memory 3715 can be used to store software programs and modules, and the processor 3780 executes various terminal functions and data processing by running the software programs and modules stored in the memory 3715.

[0279] The input unit 3730 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 3730 may include a touch panel 3731 and other input devices 3732.

[0280] Display unit 3740 can be used to display input or provided information, as well as various menus of the terminal. Display unit 3740 may include display panel 3741.

[0281] Audio circuitry 3760, speaker 3761, and microphone 3762 provide an audio interface.

[0282] In this embodiment, the processor 3780 included in the terminal can execute the virtual keyboard input method of the previous embodiment. The terminals disclosed in this embodiment include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The embodiments of this invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0283] Reference Figure 38 , Figure 38 The structural block diagram of a server portion implementing the virtual keyboard input method of this disclosure embodiment shows that the server 110 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 3822 (e.g., one or more processors) and a memory 3832, and one or more storage media 3830 (e.g., one or more mass storage devices) storing application programs 3842 or data 3844. The memory 3832 and storage media 3830 can be temporary or persistent storage. The program stored in the storage media 3830 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server 3800. Furthermore, the CPU 3822 may be configured to communicate with the storage media 3830 and execute the series of instruction operations in the storage media 3830 on the server 3800.

[0284] Server 3800 may also include one or more power supplies 3826, one or more wired or wireless network interfaces 3850, one or more input / output interfaces 3858, and / or one or more operating systems 3841, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0285] The processor in server 3800 can be used to execute the virtual keyboard input method of the present disclosure embodiments.

[0286] This disclosure also provides a computer-readable storage medium for storing a computer program for executing the virtual keyboard input methods of the foregoing embodiments.

[0287] This disclosure also provides a computer program product comprising a computer program. The processor of an electronic device reads and executes the computer program, causing the electronic device to perform the virtual keyboard input method described above.

[0288] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0289] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0290] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0291] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0292] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0293] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0294] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions (computer programs) to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0295] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects. In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0296] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A virtual keyboard input method, characterized in that, include: Establish the first cross-process communication between the host application and the keyboard extension of the third-party keyboard application; A third-party virtual keyboard is displayed in the native keyboard container through the keyboard extension. The third-party virtual keyboard includes an auxiliary input area and a first keyboard area. The size and position of the native keyboard container are fixed. The first keyboard area is hidden, and a second keyboard area is displayed in a custom keyboard container through the host application. The size and position of the custom keyboard container are variable. In response to receiving operation information for the second keyboard area, the operation information is sent from the host application to the keyboard extension via the first inter-process communication, and the response result for the operation information is displayed in the auxiliary input area via the keyboard extension.

2. The virtual keyboard input method according to claim 1, characterized in that, The establishment of the first cross-process communication between the host application and the keyboard extension of the third-party keyboard application includes: The host application sends a call request to the keyboard extension; Through the keyboard extension, in response to the call request, a socket connection request is generated based on the port number of the host application, and the socket connection request is sent to the host application; Through the keyboard extension, a response to the socket connection request is received from the host application, thereby establishing socket communication as the first inter-process communication.

3. The virtual keyboard input method according to claim 2, characterized in that, After establishing socket communication as the first inter-process communication by receiving a response to the socket connection request from the host application via the keyboard extension, the virtual keyboard input method further includes: The host application generates a signature based on the application identifier of the host application, generates a data packet to be verified based on the application identifier and the signature, and sends the data packet to be verified to the keyboard extension through the first inter-process communication. The application identifier and the signature are obtained from the data packet to be verified through the keyboard extension, and the signature is verified based on the application identifier. The method of displaying a third-party virtual keyboard within the native keyboard container via the keyboard extension includes: When the signature verification is successful, the third-party virtual keyboard is displayed in the native keyboard container via the keyboard extension.

4. The virtual keyboard input method according to claim 3, characterized in that, The step of generating a signature based on the application identifier of the host application, and generating a data packet to be verified based on the application identifier and the signature, includes: The signature is generated based on the port number, the application identifier, the predefined key, and the timestamp; The data packet to be verified is generated based on the application identifier, the timestamp, and the signature. The step of obtaining the application identifier and the signature from the data packet to be verified, and verifying the signature based on the application identifier, includes: The application identifier, the timestamp, and the signature are obtained from the data packet to be verified. The signature is verified based on the port number, the application identifier, the predetermined key, and the timestamp.

5. The virtual keyboard input method according to claim 2, characterized in that, The host application has N port numbers, where N is an integer greater than 1; The step of generating a socket connection request based on the port number of the host application and sending the socket connection request to the host application includes: Execute at least one iteration of the loop, and in the i-th iteration, perform the following processing: Generate the i-th socket connection request based on the i-th port number of the host application, and send the i-th socket connection request to the host application; If no response to the i-th socket connection request is received after the timeout period, and i is less than N, the loop enters the (i+1)-th iteration. The step of receiving a response to the socket connection request from the host application to establish socket communication as the first inter-process communication includes: When a response to the i-th socket connection request is received after the timeout period, socket communication is established as the first inter-process communication.

6. The virtual keyboard input method according to claim 1, characterized in that, The step of hiding the first keyboard area and displaying the second keyboard area in a custom keyboard container through the host application includes: When the size of the native keyboard container exceeds a first threshold, the first keyboard area is hidden. The second keyboard area is displayed in the custom keyboard container by the host application, wherein the size of the custom keyboard container does not exceed the first threshold.

7. The virtual keyboard input method according to claim 1, characterized in that, The step of hiding the first keyboard area and displaying the second keyboard area in a custom keyboard container through the host application includes: Identify key subjects on the screen to which the virtual keyboard input method is applied; When the native keyboard container obscures the identified key subject, the first keyboard area is hidden; The second keyboard area is displayed in the custom keyboard container through the host application, so that the custom keyboard container does not obscure the key subject.

8. The virtual keyboard input method according to claim 7, characterized in that, Displaying the second keyboard area within the custom keyboard container, such that the custom keyboard container does not obscure the key element, includes: Determine the non-critical subject area on the screen that does not contain the critical subject; A reference line is generated based on the key subject, and the non-key subject area is divided based on the reference line to obtain multiple divided areas; Based on the area of ​​each of the multiple divided regions, a target region is determined among the multiple divided regions; The custom keyboard container is displayed in the target area, and the second keyboard area is displayed in the custom keyboard container.

9. The virtual keyboard input method according to claim 1, characterized in that, The step of hiding the first keyboard area and displaying the second keyboard area in a custom keyboard container through the host application includes: Determine the first semantics before the third-party virtual keyboard is displayed on the screen where the virtual keyboard input method is applied; Determine the second semantic meaning after the third-party virtual keyboard is displayed on the screen; When the matching degree between the first semantic and the second semantic is less than the second threshold, the first keyboard area is hidden, and the second keyboard area is displayed in the custom keyboard container through the host application.

10. The virtual keyboard input method according to claim 9, characterized in that, When the matching degree between the first semantic and the second semantic is less than a second threshold, the first keyboard area is hidden, and the second keyboard area is displayed in the custom keyboard container through the host application, including: When the matching degree between the first semantic and the second semantic is less than the second threshold, the first keyboard area is hidden; The host application displays the custom keyboard container at a predetermined size and position, and displays the second keyboard area within the custom keyboard container; Determine the third semantics of the screen after displaying the second keyboard area; When the matching degree between the first semantic and the third semantic is less than the second threshold, adjust the size and position of the custom keyboard container, and return to the step of obtaining the third semantic after the screen displays the second keyboard area, until the matching degree between the first semantic and the third semantic is not less than the second threshold.

11. The virtual keyboard input method according to claim 1, characterized in that, The operation information is at least one character selected in the second keyboard area, and the response result is a candidate word corresponding to the at least one character; The step of responding to receiving operation information for the second keyboard area, sending the operation information from the host application to the keyboard extension via the first inter-process communication, and displaying the response result for the operation information in the auxiliary input area via the keyboard extension includes: In response to receiving the selected at least one character, the at least one character is sent from the host application to the keyboard extension via the first inter-process communication; Establish a second cross-process communication between the keyboard extension and the third-party keyboard application; The keyboard extension allows the at least one character to be sent to the third-party keyboard application via the second inter-process communication, and the candidate word to be received from the third-party keyboard application. The candidate words are displayed in the auxiliary input area via the keyboard extension.

12. The virtual keyboard input method according to claim 1, characterized in that, The operation information is at least one character selected in the second keyboard area, and the response result is a candidate word corresponding to the at least one character; the screen on which the virtual keyboard input method is applied also includes an upper screen area; After displaying the response result to the operation information in the auxiliary input area via the keyboard extension, the virtual keyboard input method further includes: In response to the selection operation of the candidate word, the selected candidate word is displayed in the upper screen area.

13. The virtual keyboard input method according to claim 1, characterized in that, The step of responding to receiving operation information for the second keyboard area and sending the operation information from the host application to the keyboard extension via the first inter-process communication includes: In response to an operation on the second keyboard area, determine the trigger position of the operation on the second keyboard area; Based on the key layout within the custom keyboard container and the scaling ratio of the custom keyboard container, the key corresponding to the trigger position within the custom keyboard container is determined. Based on the determined key press, the operation information is generated; The operation information is sent from the host application to the keyboard extension via the first inter-process communication.

14. The virtual keyboard input method according to claim 1, characterized in that, The native keyboard container is placed along the bottom edge of the screen where the virtual keyboard input method is applied; Hiding the first keyboard area includes: The native keyboard container is height-constrained to a second height from a first height, such that the first keyboard area is hidden below the bottom edge of the screen. The first height is the common height of the auxiliary input area and the first keyboard area, and the second height is the height of the auxiliary input area.

15. The virtual keyboard input method according to claim 1, characterized in that, The host application has an embedded proxy toolkit; The establishment of the first cross-process communication between the host application and the keyboard extension of the third-party keyboard application includes: The first cross-process communication between the host application and the keyboard extension is established using the communication tools in the proxy toolkit. The step of displaying the second keyboard area in a custom keyboard container through the host application includes: The second keyboard area is displayed in the custom keyboard container using the keyboard display tool in the proxy toolkit.

16. The virtual keyboard input method according to claim 1, characterized in that, After hiding the first keyboard area and displaying the second keyboard area in a custom keyboard container through the host application, the virtual keyboard input method further includes: In response to an update of the host application's display resources, obtain the first updated display resources of the host application; The host application queries the display resource database to obtain the second updated display resource of the third-party virtual keyboard, and the second updated display resource matches the first updated display resource. The second updated display resource is sent from the host application to the keyboard extension via the first inter-process communication; The keyboard extension updates the display resources of the third-party virtual keyboard according to the second updated display resources.

17. A virtual keyboard input device, characterized in that, include: The communication establishment module is used to establish the first inter-process communication between the host application and the keyboard extension of the third-party keyboard application; A keyboard display module is used to display a third-party virtual keyboard in a native keyboard container through the keyboard extension. The third-party virtual keyboard includes an auxiliary input area and a first keyboard area. The size and position of the native keyboard container are fixed. The keyboard display module is also used to hide the first keyboard area and display the second keyboard area in a custom keyboard container through the host application, wherein the size and position of the custom keyboard container are variable; An input response module is configured to, in response to receiving operation information for the second keyboard area, send the operation information from the host application to the keyboard extension via the first inter-process communication, and display the response result for the operation information in the auxiliary input area via the keyboard extension.

18. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the virtual keyboard input method according to any one of claims 1 to 16.

19. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the virtual keyboard input method according to any one of claims 1 to 16.

20. A computer program product, the computer program product comprising a computer program, characterized in that, The computer program is read and executed by the processor of the electronic device, causing the electronic device to perform the virtual keyboard input method according to any one of claims 1 to 16.

Images