An input mode control method and an electronic device

Abstract

The application discloses an input mode control method and an electronic device. The method comprises the following steps: in response to detecting an input position event of an operating body to a display unit, acquiring a current interface image of the display unit; marking the current interface image based on the input position event to generate a marked image; based on the marked image, using a model to identify an interface region type to which an input position of the input position event in the current interface image belongs; based on the interface region type, determining an input mode corresponding to the input position event; and performing corresponding processing on an input track corresponding to the operating body according to the input mode.

Description

Technical Field

[0001] This application relates to the field of control technology, and in particular to an input mode control method and electronic device. Background Technology

[0002] When using electronic devices such as tablets or laptops, users often need to input information into the application interface displayed on the screen. This includes inputting interactive information into input boxes or annotating the interface content. However, to accurately input different types of information, users need to manually switch input modes according to their needs, resulting in a high degree of complexity in the information input. Summary of the Invention

[0003] On the one hand, this application provides an input mode control method, including:

[0004] In response to detecting an input position event of the operator on the display unit, the current interface image of the display unit is acquired;

[0005] Based on the input location event, the current interface image is marked, and a marked image is generated;

[0006] Based on the marked image, the model is used to identify the interface region type to which the input location of the input location event belongs in the current interface image;

[0007] Based on the interface area type, determine the input mode corresponding to the input location event;

[0008] According to the input pattern, the input trajectory corresponding to the operation body is processed accordingly.

[0009] In one possible implementation, generating a labeled image based on the input location event to label the current interface image includes:

[0010] Determine the input position corresponding to the input position event;

[0011] Based on the input location, a visual marker is generated in the current interface image to indicate the input location, thereby converting the input location into a visual location cue in the current interface image, resulting in a marker image.

[0012] In another possible implementation, the step of using a model to identify the interface region type to which the input position event belongs in the current interface image includes:

[0013] The marked image is input into a multimodal model, which identifies the interface region type to which the input location belongs based on the interface visual features in the marked image and the contextual semantics of the location indicated by the visual marker.

[0014] In another possible implementation, the step of identifying the interface region type to which the input location belongs based on the interface visual features in the marked image and the contextual semantics of the location indicated by the visual marker by the multimodal model includes:

[0015] The location of the visual marker in the marker image is determined using the multimodal model.

[0016] Based on the position of the visual marker in the marked image, the multimodal model determines candidate image regions including the visual marker from the marked image, and determines the semantic features of the candidate image regions and the contextual semantic features in the marked image associated with the candidate image regions.

[0017] Based on the visual features of the marked image, the semantic features of the candidate image region, and the contextual semantic features, the multimodal model is used to identify the type of interface region to which the input location belongs.

[0018] In yet another possible implementation, the interface area type belongs to one of the first type and the second type;

[0019] Wherein, when the interface area type is the first type, the input mode is the first input mode;

[0020] When the interface area type is the second type, the input mode is the second input mode;

[0021] The step of performing corresponding processing on the input trajectory corresponding to the operation body according to the input pattern includes:

[0022] In response to the input mode being the first input mode, the input trajectory of the operating body is identified to obtain text content, and the text content is input into the target input area corresponding to the input position;

[0023] In response to the input mode being the second input mode, the input trajectory of the operator is stored as annotation content associated with the current interface image.

[0024] In yet another possible implementation, the input position event of the operating body for the display unit includes at least one of the following:

[0025] The touch event where the operating body touches the display unit;

[0026] The proximity event of the operator entering the sensing range corresponding to the display unit;

[0027] The hovering event where the operator hovers within a preset distance above the display unit.

[0028] In yet another possible implementation, the input mode control method further includes:

[0029] Based on the input location corresponding to the input location event and the current interface image, match historical input location events;

[0030] In response to a matched historical input location event, the historical interface region type corresponding to the historical input location event is determined as the interface region type to which the input location of the input location event belongs in the current interface image;

[0031] The historical interface region type corresponding to the historical input location event is the interface region type identified by the model based on the historical input location and historical interface image corresponding to the historical input location event.

[0032] In yet another possible implementation, the input mode control method further includes:

[0033] In response to obtaining negative feedback information on the input pattern, the input location event, the current interface image, and the interface region type are determined as negative sample data, and the negative feedback information indicates that the determined input pattern is incorrect;

[0034] The model is optimized based on the negative sample data.

[0035] In yet another possible implementation, the visual marker includes at least one of the following:

[0036] An arrow used to indicate the input location;

[0037] Graphical markers used to circle the input location;

[0038] Highlighted markers used to indicate the input location;

[0039] Color markers used to indicate the input location.

[0040] In another aspect, this application also provides an electronic device, including: a sensing unit, a processor, and a display unit;

[0041] The display unit is used to output interface images;

[0042] The sensing unit is used to detect input position events of the operating body to the display unit;

[0043] The processor is configured to, in response to the input position event, acquire the current interface image currently output by the display unit; mark the current interface image based on the input position event to generate a marked image; based on the marked image, use a model to identify the interface region type to which the input position of the input position event in the current interface image belongs; determine the input mode corresponding to the input position event based on the interface region type; and perform corresponding processing on the input trajectory corresponding to the operating body according to the input mode. Attached Figure Description

[0044] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0045] Figure 1 A flowchart illustrating the input mode control method provided in this application;

[0046] Figure 2 Another flowchart illustrating the input mode control method provided in this application;

[0047] Figure 3 Another flowchart illustrating the input mode control method provided in this application;

[0048] Figure 4 An example diagram of the current interface image provided in this application;

[0049] Figure 5 Based on Figure 4 An example image of a labeled image generated from the current interface image in the diagram;

[0050] Figure 6 Another flowchart illustrating the input mode control method provided in this application;

[0051] Figure 7 A schematic diagram of the component architecture of the electronic device provided in this application. Detailed Implementation

[0052] The solution proposed in this application can automatically determine the current input mode in response to the input position event of the operator, thereby eliminating the need for the user to manually switch the input mode before performing information input, thus improving the convenience and flexibility of information input.

[0053] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0054] As will be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0055] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0056] like Figure 1 This illustration shows a flowchart of an input mode control method provided in this application. The method of this embodiment is applied to an electronic device, which can be a mobile phone, tablet, laptop, or desktop computer, etc., without specific limitations. The method of this embodiment may include:

[0057] S101, in response to detecting an input position event of the operator on the display unit, the current interface image of the display unit is acquired.

[0058] The operator is used to input information into the display unit of the electronic device. For example, the operator can be a stylus or a finger, etc., without any specific restrictions.

[0059] The input position event indicates that the operator currently has an input requirement to input information to the display unit. In this application, the input position event can have multiple possibilities, without any specific limitation.

[0060] For example, in one instance, the input location event may include at least one of the following:

[0061] Touch events where the operating body touches the display unit;

[0062] The proximity event when the operator enters the sensing range corresponding to the display unit;

[0063] The hovering event where the operator hovers within a preset distance above the display unit.

[0064] The display unit can be equipped with sensors such as capacitive proximity sensors or optical proximity sensors. If the sensor detects the object before it touches the display unit, it means that the object has entered the corresponding sensing range of the display unit, i.e., there is an approach event of the object.

[0065] A hovering event can be an event in which the operator is within a preset distance above the display unit and remains there for a set duration. Hovering events can be detected by sensors such as the capacitive proximity sensor mentioned above, or they can be determined by other methods, without any specific restrictions.

[0066] In this application, the current interface image is the interface image currently displayed by the display unit when the input position event is detected.

[0067] There are multiple ways to obtain the current interface image. For example, a screenshot of the interface currently displayed by the display unit can be obtained. This application does not limit the specific implementation method for obtaining the current interface image.

[0068] S102, mark the current interface image based on the input location event, and generate a marked image.

[0069] The marked image indicates the input location of the input location event within the current interface image.

[0070] For example, since the marker image is obtained by marking the current interface content, the marker image includes the current interface image and indication information that indicates the input position of the input position event in the current interface image.

[0071] S103, based on the marked image, uses a model to identify the interface region type to which the input position of the input position event belongs in the current interface image.

[0072] In this application, the model can be a machine learning model or other artificial intelligence model, without limitation.

[0073] The model can be located locally on the electronic device; or it can be located on a server, cloud platform, or other device outside the electronic device, without any restrictions.

[0074] The interface region type is used to characterize the interactive behaviors supported by the input position in the current interface image corresponding to the input position event. These interactive behaviors may include, but are not limited to, information input, selection, or event triggering.

[0075] For example, the interface area types that can be classified from multiple different areas in the interface image output by the display unit may include: input box areas that support information input, non-input box areas that do not support information input, information display areas for information display, and link address areas that trigger page jumps by users, etc., without any specific restrictions.

[0076] S104, based on the interface area type, determine the input mode corresponding to the input position event.

[0077] Input mode is used to characterize the processing method of an electronic device in response to an input operation. Accordingly, for the same input information, the electronic device performs different processing methods under different input modes.

[0078] For example, different input modes can correspond to different data editing methods, storage rules, and input flow directions. For instance, input modes may include, but are not limited to, a data input mode for using input data generated by input operations as interface input content, and a comment input mode for processing input data generated by input operations as comment content.

[0079] S105, according to the input mode, perform corresponding processing on the input trajectory corresponding to the operation body.

[0080] For example, there are corresponding processing methods for each input mode. Therefore, the input data generated by the input trajectory can be processed according to the processing method corresponding to the input mode.

[0081] As can be seen from the above, in this application, upon detecting an input position event of the operator on the display unit, the electronic device obtains the current interface image of the display unit. After marking the current interface image based on the input position event, it uses a model to identify the interface region type to which the input position event belongs in the current interface image based on the generated marked image. On this basis, the electronic device can automatically determine a suitable input mode based on the interface region type identified by the model and perform corresponding processing on the input trajectory of the operator based on that input mode. This achieves automatic recognition and switching of the input mode, eliminating the need for the user to manually switch input modes and improving the convenience of information input.

[0082] Moreover, since the appropriate input mode can be automatically determined in response to the input position event of the operator, the user can directly use the operator to perform subsequent input without interrupting the input operation of the operator to switch input modes, which naturally improves the smoothness of the user's input operation through the operator.

[0083] In addition, this application can determine the interface area type to which the input position belongs by leveraging the model's visual understanding capabilities. Based on the interface area type, the input mode can be determined without relying on the application layer control interface. It can be applied to the switching and control of input modes in different applications and has good cross-application compatibility.

[0084] In this application, there are several possible implementations for generating labeled images. The following example uses one implementation method, combined with... Figure 2 Please provide an explanation. For example... Figure 2This illustration shows another flowchart of the input mode control method provided in this application. The method of this embodiment can be applied to the aforementioned electronic devices. The method of this embodiment may include:

[0085] S201, in response to detecting an input position event of the operator on the display unit, the current interface image of the display unit is acquired.

[0086] S202, Determine the input position corresponding to the input position event.

[0087] Here, the input position refers to the coordinate position of the input position event within the display unit. Since the current interface image is an image corresponding to the interface currently output by the display unit, the coordinate position of the input position event within the display unit is also the input position of the input position event within the current interface image.

[0088] The input position can be determined by a sensor in the display unit, and this application does not limit the specific implementation method for determining the input position.

[0089] S203, Based on the input location, a visual marker is generated in the current interface image to indicate the input location, so as to convert the input location into a visual location cue in the current interface image, and obtain a marker image.

[0090] The marked image is the current interface image with the visual mark.

[0091] The specific form of the visual markers generated in the current interface image can be varied and is not limited.

[0092] For example, in one possible implementation, the visual marker may include at least one of the following:

[0093] An arrow used to indicate the input location;

[0094] Graphical markers used to circle the input location;

[0095] Highlighted markers used to indicate the input location;

[0096] And a color mark used to indicate the input location.

[0097] The graphic markers can be circles, squares, or other closed shapes, with no specific restrictions. Highlighted markers can be dots or other forms of markers with brightness higher than the brightness of any pixel in the current interface image. Color markers can take many forms; for example, they can be markers of a color other than those present in the current interface image, such as red or green.

[0098] In practical applications, visual marking can be a combination of the above marking methods. For example, while using graphic marking to circle the input position, the graphic marking can be set to have a specified color and be highlighted, so that the graphic marking can also serve as a highlight marking point and a color marking function.

[0099] S204, Based on the marked image, use the model to identify the interface region type to which the input location event belongs in the current interface image.

[0100] In this embodiment, the model can have several possibilities, as described above.

[0101] In one example, to more accurately identify the type of interface region corresponding to the input location in the current interface image, the model can employ a multimodal model. A multimodal model is an artificial intelligence model capable of simultaneously processing multiple types of information, such as text, images, and audio. In this application, using a multimodal model to process the marked image enables analysis of the image's features from multiple dimensions, thereby more accurately determining the type of the interface region.

[0102] In one possible implementation, the multimodal model analyzes not only the visual features of the marked image but also its semantic features. Specifically, the marked image is input into the multimodal model, which identifies the interface region type to which the input location belongs based on the interface visual features in the marked image and the contextual semantics of the location indicated by the visual marker.

[0103] The visual features of the marked image refer to the interface features exhibited by the current interface image corresponding to the marked image. For example, these visual features are used to characterize at least one of the following features of the current interface image: interface content, structural texture, color distribution, and spatial relationships.

[0104] The contextual semantics of the location indicated by the visual marker include the semantic features in the current interface image corresponding to the location indicated by the visual marker, as well as the contextual semantic features associated with the location indicated by the visual marker.

[0105] The contextual semantic features associated with the location indicated by the visual marker may include, but are not limited to, the interface type corresponding to the current interface image, the distribution relationship between elements such as controls in the interface, and the function, attributes, and displayed text information of at least one element within a defined range of the location indicated by the visual marker. The interface type may represent the type of application to which the interface belongs or the specific application to which the interface belongs. For example, based on the interface type, it can be determined that the current interface image is an image of a browser interface or an image of the application interface of a game application.

[0106] It is understandable that combining the contextual semantics of the location indicated by the visual marker in the marked image with the interface visual features of the marked image can more comprehensively reflect the interface region type corresponding to the location indicated by the visual marker (i.e., the input location) from two dimensions.

[0107] The multimodal model's identification of the interface region type can be implemented in various ways, without any specific limitations. For example, in one possible implementation, the multimodal model can identify the interface region type to which the input location belongs in the following manner:

[0108] First, a multimodal model is used to determine the location of the visual marker in the marker image;

[0109] Secondly, based on the position of visual markers in the labeled image, a multimodal model is used to determine candidate image regions including visual markers from the labeled image, and to determine the semantic features of the candidate image regions and the contextual semantic features associated with the candidate image regions in the labeled image.

[0110] Then, based on the visual features of the labeled image interface, the semantic features of the candidate image region, and the contextual semantic features, a multimodal model is used to identify the interface region type to which the input location belongs.

[0111] There are several ways to determine the location of visual markers. For example, multimodal models can determine the features of visual markers through pre-training or continuous learning. Based on this, visual markers can be identified from a marked image, and thus their location in the marked image can be determined.

[0112] For example, this application can also obtain cue words, which are used to indicate the visual features of the visual marker. For instance, the cue words can be used to determine the interface region type corresponding to the location indicated by the red arrow in the image. Accordingly, based on the features of the keywords in the cue words used to indicate the visual marker, cross-modal attention calculation can be performed with the visual features corresponding to different image patches in the marked image, and the location of the image patch with the highest attention value can be determined as the location of the visual marker.

[0113] The size of the candidate image region can be set as needed, without any specific restrictions.

[0114] The semantic features of a candidate image region can reflect the interface elements to which the candidate image region belongs, the functions and attributes of the interface elements, and the text information contained within the candidate image region. For example, the semantic features of a candidate image region can characterize whether there are interface elements such as rectangular borders that match input boxes, and the semantics of the text contained within the candidate image region.

[0115] Among them, the contextual semantic features associated with the candidate image area in the marked image are similar to the aforementioned contextual semantic features. For example, the contextual semantic features associated with the candidate image area can be the interface type corresponding to the current interface image, the distribution relationship between elements such as controls in the interface, and the function, attributes, and displayed text information of at least one element within the set range of the candidate image area.

[0116] In this implementation, the powerful visual recognition and semantic understanding capabilities of the multimodal model are utilized to identify the interface region type to which the input position corresponding to the input position event belongs in a complex interface, thereby enabling a more accurate identification of the interface region type to which the input position corresponding to the input position event belongs in the current interface image.

[0117] S205, based on the interface area type, determine the input mode corresponding to the input position event.

[0118] S206, Based on the input mode, perform corresponding processing on the input trajectory corresponding to the operating body.

[0119] For the above steps S205 and S206, please refer to the relevant descriptions in the previous embodiments.

[0120] In this embodiment, after obtaining the current interface image in response to the detection of an input location event, this application determines the input location corresponding to the input location event, and generates a visual marker in the current interface image to indicate the input location based on the input location, so as to convert the input location into a visual location prompt in the current interface image. This allows the generated marker image to visually and intuitively indicate the input location of the input location event, enabling the model to more accurately determine the input location of the input location event in the current interface image, and further enabling a more accurate analysis of the interface area type corresponding to the input location and ultimately determining the appropriate input mode.

[0121] In any of the above embodiments of this application, the interface area type can be divided in multiple ways, and the types that the interface area type can be divided into will also be different when the application scenario is different.

[0122] In one possible scenario, the interface area type can belong to either the first type or the second type. Accordingly, if the interface area type is the first type, the determined input mode is the first input mode; if the interface area type is the second type, the determined input mode is the second input mode.

[0123] The first input mode is used to perform text conversion input based on the input operation (or input trajectory) of the operator. In this first input mode, the input data generated by the input operation of the operator is input into the target input area corresponding to the input position where the input position event is located.

[0124] The second input mode is used to convert the input operations (input trajectory) of the operator into annotation content. In this second input mode, the input data generated by the operator's input operations will be used as annotation content associated with the current interface image.

[0125] Based on this, in response to the first input mode, the input trajectory of the operator can be recognized to obtain text content, and the text content can be input into the target input area corresponding to the input position of the input position event. In response to the second input mode, the input trajectory of the operator is stored as annotation content associated with the current interface image.

[0126] Specifically, the target input area corresponding to the input location of this input location event belongs to the input box for text input in the interface output by the display unit. For example, this input box can be a search bar in a browser interface or application interface for entering search keywords; an information input bar in an instant messaging application for entering interactive information; or a question input area in a dialog window, etc., without any specific restrictions.

[0127] Specifically, by storing the input trajectory of the operator as annotation content associated with the current interface image, the annotation content corresponding to the input trajectory is synchronously displayed each time the current interface image is shown. Correspondingly, while storing the input trajectory of the operator as annotation content associated with the current interface image, the input trajectory of the operator is also displayed as annotation content on top of the current interface image.

[0128] In one example, if the input position corresponding to the input location event of the operator is within the input field of the current interface image, it means the user wants to enter text content into that input field. In this case, the electronic device is suitable for the first input mode. Conversely, if the input position corresponding to the input location event of the operator is outside the input field, it means the user needs to annotate the content in the current interface image. In this case, the electronic device is suitable for the second input mode.

[0129] To facilitate understanding of this example, combined with Figure 3 Let's take a specific implementation as an example for illustration. For example... Figure 3 This illustrates another flowchart of the input mode control method provided in this application. The method in this embodiment may include:

[0130] S301, in response to detecting an input position event of the operating body on the display unit, the current interface image of the display unit is acquired.

[0131] S302, Determine the input position corresponding to the input position event.

[0132] S303, Based on the input location, generate a visual marker in the current interface image to indicate the input location, so as to convert the input location into a visual location cue in the current interface image, and obtain a marker image.

[0133] In this embodiment, for ease of understanding, one method of generating a marker image is used as an example. However, other methods of generating the marker image are also applicable to this embodiment and will not be described in detail here.

[0134] S304, Based on the marked image, the model is used to identify the interface region type to which the input location of the input location event belongs in the current interface image.

[0135] In this embodiment, the interface area type can be divided into two types: input box area and non-input box area. Therefore, the interface area type to which the input position belongs is one of the input box area and non-input box area.

[0136] The input box area refers to the region in the interface image where the input box is located. As mentioned earlier, the input box can be a search bar or information input field for entering search terms, etc., without any specific restrictions. The non-input box area is used to represent other areas in the interface image besides the region where the input box is located.

[0137] The specific implementation for determining the type of the interface area can be found in the relevant descriptions in the previous embodiments, and there are no restrictions on it.

[0138] For example:

[0139] like Figure 4 This is an example diagram showing the current interface image displayed in this application. Figure 5 Based on Figure 4 The image shown is an example of a marker image generated from the current interface image.

[0140] exist Figure 4 The current interface image shown includes a search bar for entering search terms, as well as the interface area outside the search bar.

[0141] Assuming the input position of the operator's input position event is within the search bar of the current interface image, then using the solution of this application, a visual marker can be added to the search bar of the current interface image based on the input position. For example, in Figure 5For example, the visual marker is a red circle (501).

[0142] Based on this, after inputting the marked image into the multimodal model, the multimodal model can identify the location of the red circle 501 and, based on the location, determine the candidate image region including that location from the marked image. For example, the multimodal model can divide the marked image into multiple image blocks of the same size and, based on the location of the visual marker, determine the candidate image block including that visual marker. On this basis, the multimodal model analyzes the interface visual features of the marked image and the contextual semantic features associated with the candidate image region in the marked image. It also combines the visual features (e.g., possible border shape, background color, and layout position of the input box), semantic features (e.g., the content of placeholder text that may be included in the input box), and the related layout relationships between the input box and its surrounding elements (e.g., controls, labels, and icons) and the possible types of surrounding elements, which are pre-learned by the multimodal model, to determine whether the input position of the operator's input position event is in the input field. Figure 5 If the search bar is present, the interface area type is determined to be an input box area; otherwise, the interface area type is determined to be a non-input box area.

[0143] S305, in response to the interface area type being an input box area, determine the first input mode corresponding to the input position event.

[0144] S306, in response to the first input mode, recognizes the input trajectory of the operating body and inputs the recognized text content into the input box to which the input position belongs.

[0145] S307, in response to the interface area type being a non-input box area, determine the second input mode corresponding to the input position event.

[0146] S308, in response to the second input mode, stores the input trajectory of the operator as annotation content associated with the current interface image.

[0147] In any of the above embodiments of this application, considering that each time the model is called to determine the interface region type, it consumes a lot of time and resources, therefore, in order to reduce the number of times the model is called, thereby reducing resource consumption and the time required to determine the interface region type, this application can also match historical input position events based on the input position corresponding to the input position event and the current interface image before generating the marked image. If a historical input position event is matched, the historical interface region type corresponding to the matched historical input position event can be directly determined as the interface region type to which the input position of the input position event belongs.

[0148] Among them, the historical interface region type corresponding to the historical input location event is the interface region type identified by the model based on the historical input location and historical interface image corresponding to the historical input location event.

[0149] To facilitate understanding, the following will be combined with... Figure 6 Please provide an explanation. For example... Figure 6 This is another flowchart illustrating the input mode control method provided in this application. The method in this embodiment may include:

[0150] S601, in response to detecting an input position event of the operating body on the display unit, the current interface image of the display unit is acquired.

[0151] S602, Determine the input position corresponding to the input position event.

[0152] The above steps can be found in the relevant descriptions of the previous embodiments, and will not be repeated here.

[0153] S603, based on the input position corresponding to the input position event and the current interface image, matches historical input position events.

[0154] In this application, the historical input location event refers to an input location event obtained before the current time, and whose corresponding historical interface region type was determined by the model in the past. The historical interface region type corresponding to this historical input location event is the interface region type identified by the model based on the historical input location and historical interface image corresponding to this historical input location event. The historical interface image is the interface image of the display unit obtained when the historical input location event was obtained.

[0155] The specific implementation of using the model to identify the type of historical interface region corresponding to the historical input location event is similar to the process of using the model to determine the type of interface region corresponding to the input location event, and will not be repeated here.

[0156] It is understood that before obtaining the input location event, this application may store at least one historical input location event and its corresponding historical input location, historical interface image, and historical interface region type. Based on this, this application needs to determine the historical input location event that matches the input location and the current interface image, that is, the historical interface image matches the current interface image, and the historical input location matches the input location of the input location event.

[0157] For example, this application can determine historical input position events where the similarity between a historical interface image and the current interface image exceeds a first preset threshold, and the coordinate distance between a historical input position and the input position corresponding to the input position event is less than a second preset threshold.

[0158] S604. Determine whether there is a historical input position event that matches the input position and the current interface image. If yes, proceed to step S605; otherwise, proceed to step S606.

[0159] S605, determine the historical interface region type corresponding to the matched historical input location event as the interface region type to which the input location event belongs in the current interface image, and execute step S608.

[0160] If there exists a historical input location event that matches the input location of the current interface image, it means that a similar input location event at the same interface corresponding to the current interface image was processed in the past. Therefore, the historical interface region type corresponding to the historical input location event can be directly used as the interface region type corresponding to the current input location event.

[0161] S606, based on the input position event, marks the current interface image and generates a marked image.

[0162] S607, based on the marked image, uses a model to identify the interface region type to which the input position of the input position event belongs in the current interface image.

[0163] The above steps S606 and S607 can be found in the relevant descriptions of the previous embodiments, and will not be repeated here.

[0164] Specifically, after determining the interface region type corresponding to the input location event using the model, this application can treat the input location event as a historical input location event and store the input location event and its corresponding current interface image, input location, and interface region type.

[0165] S608 determines the input mode corresponding to the input position event based on the interface area type.

[0166] S609, according to the input mode, performs corresponding processing on the input trajectory corresponding to the operating body.

[0167] The above steps S608 and S609 can be found in the relevant descriptions of the previous embodiments, and will not be repeated here.

[0168] In this embodiment, after obtaining the input position event of the operator, if a historical input position event is matched with the current interface image based on the input position corresponding to the input position event, the historical interface region type determined by the model based on the historical input position and historical interface image corresponding to the historical input position event can be obtained. This historical interface region type is used as the interface region type to which the input position of the input position event in the current interface image belongs. Therefore, it is not necessary to repeatedly call the model for identification for input position events at approximately the same position in the same interface image. This not only improves the efficiency of determining the interface region type and input pattern, but also reduces the resource consumption generated by calling and running the model.

[0169] In any of the above embodiments of this application, if the user finds that the input pattern is not the correct input pattern they expect, the user can also input negative feedback information regarding the input pattern. This negative feedback information indicates that the determined input pattern is incorrect.

[0170] For example, upon determining the input mode, this application can also output a prompt message to indicate that the input mode has been entered (e.g., the first input mode or the second input mode), so that the user can be promptly informed of the current input mode. If the prompt message determines that the input mode is incorrect, the user can input negative feedback information for that input mode.

[0171] For example, after an electronic device determines an input mode, it will process the input trajectory accordingly, or after processing the input trajectory, the user can also determine the current input mode based on the processing performed by the electronic device. In this case, if the user believes that the determined input mode is incorrect, they can also input negative feedback information for that input mode.

[0172] There are several possibilities for how negative feedback information can be input for this input mode. For example, after determining the input mode, the electronic device can output a feedback prompt bar or a feedback collection window to obtain user feedback on the determined input information, without any specific restrictions.

[0173] Accordingly, in response to obtaining negative feedback information on the input pattern, the input location event, the current interface image, and the interface region type can be identified as negative sample data; based on the negative sample data, the model is optimized to improve the accuracy of the model in determining the interface region type.

[0174] Understandably, if there is too little negative sample data, the effect on model optimization will be limited. Therefore, this application does not perform model optimization every time a negative sample data is collected. For example, model optimization based on negative sample data may be performed when the number of collected negative sample data exceeds a set threshold, or when the time since the last model optimization exceeds a set time, or when a model optimization command is detected. There are no specific restrictions on the specifics.

[0175] If the model is deployed locally, it can be trained based on the currently collected negative sample data.

[0176] If the model is deployed on a server or cloud device node outside of electronic devices, this application can construct prompts based on negative sample data to instruct the model to optimize based on the negative sample data. Alternatively, based on negative sample data, it can instruct the model to relearn the rules for recognizing interface region types to improve the accuracy of the model in recognizing interface region types. Of course, there are other ways to optimize the model, without any specific limitations.

[0177] This application also provides an electronic device in its embodiments. For example... Figure 7 As shown, it illustrates a schematic diagram of the composition structure of the electronic device, which includes at least: a sensing unit 701, a processor 702, and a display unit 703;

[0178] The display unit 703 is used to output interface images.

[0179] The sensing unit 701 is used to detect input position events of the operating body to the display unit.

[0180] The processor 702 is configured to, in response to an input position event, acquire the current interface image currently output by the display unit; mark the current interface image based on the input position event and generate a marked image; based on the marked image, use a model to identify the interface region type to which the input position of the input position event in the current interface image belongs; determine the input mode corresponding to the input position event based on the interface region type; and perform corresponding processing on the input trajectory corresponding to the operating body according to the input mode.

[0181] The sensing unit can be disposed on the display unit, and the sensing unit can be at least one of a pressure sensor and an image sensor, without any specific limitation.

[0182] Furthermore, the electronic device may also have a memory 704 for storing programs required for the processor to perform operations.

[0183] Of course, the electronic device can also have more than Figure 7 There are no restrictions on the number of components, whether more or fewer.

[0184] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the input mode control methods provided in this application.

[0185] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the input mode control methods provided in this application.

[0186] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0188] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0189] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An input mode control method, comprising: In response to detecting an input position event of the operator on the display unit, the current interface image of the display unit is acquired; Based on the input location event, the current interface image is marked, and a marked image is generated; Based on the marked image, the model is used to identify the interface region type to which the input location of the input location event belongs in the current interface image; Based on the interface area type, determine the input mode corresponding to the input location event; According to the input pattern, the input trajectory corresponding to the operation body is processed accordingly.

2. The input mode control method according to claim 1, wherein generating a marked image based on the input position event marking the current interface image includes: Determine the input position corresponding to the input position event; Based on the input location, a visual marker is generated in the current interface image to indicate the input location, thereby converting the input location into a visual location cue in the current interface image, resulting in a marked image.

3. The input mode control method according to claim 2, wherein the step of using a model to identify the interface region type to which the input position event belongs in the current interface image includes: The marked image is input into a multimodal model, which identifies the interface region type to which the input location belongs based on the interface visual features in the marked image and the contextual semantics of the location indicated by the visual marker.

4. The input mode control method according to claim 3, wherein the step of identifying the interface region type to which the input position belongs based on the interface visual features in the marked image and the contextual semantics of the position indicated by the visual marker by the multimodal model includes: The location of the visual marker in the marker image is determined using the multimodal model. Based on the position of the visual marker in the marked image, the multimodal model determines candidate image regions including the visual marker from the marked image, and determines the semantic features of the candidate image regions and the contextual semantic features in the marked image associated with the candidate image regions. Based on the interface visual features of the marked image, the semantic features of the candidate image region, and the contextual semantic features, the multimodal model is used to identify the interface region type to which the input location belongs.

5. The input mode control method according to claim 1, wherein the interface area type belongs to one of the first type and the second type; wherein When the interface area type is the first type, the input mode is the first input mode; When the interface area type is the second type, the input mode is the second input mode; The step of performing corresponding processing on the input trajectory corresponding to the operation body according to the input pattern includes: In response to the input mode being the first input mode, the input trajectory of the operating body is identified to obtain text content, and the text content is input into the target input area corresponding to the input position; In response to the input mode being the second input mode, the input trajectory of the operator is stored as annotation content associated with the current interface image.

6. The input mode control method according to claim 1, wherein the input position event of the operating body to the display unit includes at least one of the following: The touch event where the operating body touches the display unit; The proximity event of the operator entering the sensing range corresponding to the display unit; The hovering event where the operator hovers within a preset distance above the display unit.

7. The input mode control method according to claim 1, further comprising: Based on the input location corresponding to the input location event and the current interface image, match historical input location events; In response to a matched historical input location event, the historical interface region type corresponding to the historical input location event is determined as the interface region type to which the input location of the input location event belongs in the current interface image; The historical interface region type corresponding to the historical input location event is the interface region type identified by the model based on the historical input location and historical interface image corresponding to the historical input location event.

8. The input mode control method according to claim 1, further comprising: In response to obtaining negative feedback information on the input pattern, the input location event, the current interface image, and the interface region type are determined as negative sample data, and the negative feedback information indicates that the determined input pattern is incorrect; The model is optimized based on the negative sample data.

9. The input mode control method according to claim 2, wherein the visual marker includes at least one of the following: An arrow used to indicate the input location; Graphical markers used to circle the input location; Highlighted markers used to indicate the input location; Color markers used to indicate the input location.

10. An electronic device, comprising: Sensing unit, processor, and display unit; The display unit is used to output interface images; The sensing unit is used to detect input position events of the operating body to the display unit; The processor is configured to, in response to the input position event, acquire the current interface image currently output by the display unit; Based on the input location event, the current interface image is marked, and a marked image is generated; Based on the marked image, the model is used to identify the interface region type to which the input location of the input location event belongs in the current interface image; Based on the interface area type, determine the input mode corresponding to the input location event; According to the input pattern, the input trajectory corresponding to the operation body is processed accordingly.

Images