Control method and device, storage medium and electronic equipment

Abstract

The invention discloses a control method and device, a storage medium and electronic equipment, and relates to the technical field of computer vision, and the method comprises the steps that at least one function area in a first interface is determined, and the function area is an effective control range for triggering function operation; in response to an input operation of the input device, determining at least one target function area based on a preset association table, and executing a function operation corresponding to the target function area; the target function area is one of the at least one function area, and the preset association table is a comparison table of the function area and the input operation. According to the technical scheme, the association table of the input operation of the input device and the function area in the first interface is established, when the input device inputs the operation, the operation corresponding to the target function area can be directly executed, and the problems that an existing physical device is low in virtual interface control speed and accuracy and cannot meet complex operation requirements are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a control method and device, a storage medium and an electronic device. BACKGROUND

[0002] A cloud phone is a virtual phone device running in the cloud. The principle is to allocate CPU, memory and other resources for the cloud phone on a server using virtualization technology such as a virtual machine, install a corresponding operating system, and virtually assign a WIFI Media Access Control (WIFI MAC) address, an International Mobile Equipment Identity (IMEI) and other information, thereby virtually simulating a phone.

[0003] In related technologies, a user can operate a cloud phone on a terminal such as a personal computer (PC), a mobile phone or a tablet. The operation mode is to simulate touch control by using a mouse, such as simulating single-point touch control by clicking a mouse, and simulating finger sliding by dragging a mouse, so as to achieve precise control of the interface of the cloud phone.

[0004] However, the touch control that can be simulated by mouse clicking is only single-point touch control. In addition to single-point touch control, the operating system of the cloud phone also supports multi-point touch control and other touch operations. Mouse clicking cannot simulate multi-point touch control. Moreover, the user dragging a mouse to click the interface of the cloud phone is a virtual operation, not a physical operation, which results in a gap between the speed and accuracy of the simulated touch control and the speed and accuracy of the physical touch control when using an actual phone. SUMMARY

[0005] Therefore, the present application provides a control method and device, a storage medium and an electronic device to improve the speed and accuracy of interface operation.

[0006] In a first aspect, the present application provides a control method, comprising: determining at least one function area in a first interface; the function area is an effective control range of triggering a function operation; in response to an input operation of an input device, determining at least one target function area based on a preset association table, and executing a function operation corresponding to the target function area; the target function area is one of the at least one function area; the preset association table is a comparison table of the function area and the input operation.

[0007] In the embodiment, by establishing the association table of the input operation of the input device and the function area in the first interface, when the input device inputs the operation, the operation corresponding to the target function area can be directly executed, thereby solving the problems of low speed, low accuracy and inability to meet complex operation requirements of the physical device in controlling the virtual interface.

[0008] In some embodiments of the present disclosure, the determining of the at least one function area in the first interface comprises: collecting a plurality of image data; the plurality of image data is the screenshot data of the first interface in different image states; inputting the plurality of image data into a semantic segmentation model to perform semantic segmentation and obtain a plurality of segmentation maps; determining a virtual control area according to the plurality of segmentation maps; detecting the function area in the virtual control area.

[0009] In some embodiments of the present disclosure, the determining of the virtual control area according to the plurality of segmentation maps comprises: performing pixel-level superposition on the plurality of segmentation maps to obtain a total segmentation map; calculating a first proportion of each pixel point in the total segmentation map; the first proportion is a ratio of the number of frames in which the pixel point belongs to the virtual control to the total number of frames in the plurality of segmentation maps; fitting the pixel points with the first proportion greater than a first proportion threshold into a regular pattern to obtain the virtual control area.

[0010] In some embodiments of the present disclosure, the detecting of the function area in the virtual control area comprises: performing sampling on the virtual control area according to a first sampling density to obtain a plurality of first test points; performing a first touch operation; the first touch operation is used to trigger the function operation of a single first test point; calculating a first state difference of a target object after the first touch operation is completed; the target object is an object in the first interface that performs the function operation; deleting the first test point in a case where the first state difference is within a first error range; retaining the first test point in a case where the first state difference is not within the first error range; generating the function area according to the retained first test points.

[0011] In some embodiments of the present disclosure, the generating of the function area according to the retained first test points comprises: Calculate the first similarity between the first state differences of any two first test points; If the first similarity is greater than or equal to the first similarity threshold, the two first test points are classified into the same functional region. If the first similarity is less than the first similarity threshold, the two first test points are assigned to different functional regions.

[0012] In some embodiments of this disclosure, generating the functional region based on the retained first test point includes: Based on the retained first test point, generate a connected component; According to the second sampling density, sampling is performed on the boundary between two adjacent connected regions to obtain multiple second test points; the second sampling density is greater than the first sampling density. Perform a second touch operation; the second touch operation is used to trigger a functional operation on a single second test point; After the second touch operation is completed, the second state difference of the target object is calculated; The functional area is determined based on the second state difference.

[0013] In some embodiments of this disclosure, performing the functional operation corresponding to the target functional area includes: Generate touch events for the target functional area; Invoke the touch framework; The touch event is reported through the touch framework; the touch event is used to trigger the first application to perform the function operation corresponding to the target functional area; the first application is the application that provides the first interface.

[0014] In some embodiments of this disclosure, the method further includes: In response to an input operation on an input device, multiple target functional areas are determined, and multiple functional operations corresponding to the target functional areas are executed simultaneously.

[0015] In some embodiments of this disclosure, generating the touch event for the target functional area includes: The target functional area is divided into multiple sub-regions; Based on the distance between the sub-region and the center point of the target functional region, multiple sub-regions are copied; the number of copies is negatively correlated with the distance. The copied sub-regions are stored in the first array; In the first array, a target sub-region is selected; the target sub-region is any one of the sub-regions in the first array. A touch event is generated for the target sub-region.

[0016] In some embodiments of this disclosure, after the step of generating the touch event of the target sub-region, the method includes: Remove the target sub-region from the first array and store it in the second array; Set a counter; the counter is used to record the number of times touch events are reported; If the value of the counter equals a preset threshold, the target sub-region is removed from the second array and stored in the first array.

[0017] In some embodiments of this disclosure, after the step of obtaining the multi-frame segmentation map, the method further includes: Preprocessing is performed on the segmented images of multiple frames; the preprocessing includes smoothing and erosion.

[0018] Secondly, this application provides a control device, comprising: A first processing module is used to determine at least one functional area in a first interface; the functional area is an effective control range for triggering functional operations. The second processing module is configured to respond to an input operation on the input device, determine at least one target functional area based on a preset association table, and execute the functional operation corresponding to the target functional area; the target functional area is one of the at least one functional area; the preset association table is a lookup table between the functional area and the input operation.

[0019] Thirdly, this application provides an electronic device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0020] Fourthly, this application provides a computer-readable storage medium storing computer instructions for causing the computer to perform the method described in the first aspect.

[0021] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0022] By employing the above technical solutions, this application provides a control method, device, storage medium, and electronic device, relating to the field of computer vision technology. The method includes: determining at least one functional area in a first interface, where the functional area is the effective control range for triggering a functional operation. In response to an input operation on an input device, based on a preset association table, determining at least one target functional area and executing the functional operation corresponding to the target functional area; the target functional area is one of the at least one functional area, and the preset association table is a lookup table between functional areas and input operations. By applying the technical solution of this application, an association table between the input operation of the input device and the functional areas in the first interface is established. When an input operation is performed on the input device, the operation corresponding to the target functional area can be directly executed, solving the problems of low speed, low accuracy, and inability to meet complex operation requirements of existing physical devices for virtual interface manipulation.

[0023] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating the control method provided in an embodiment of this disclosure; Figure 2 A schematic diagram of a game interface provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of the structure of the semantic segmentation model provided in the embodiments of this disclosure; Figure 4 A schematic diagram of test points for the virtual control provided in the embodiments of this disclosure; Figure 5 A schematic diagram of the structure of the control device provided in the embodiments of this disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0027] The embodiments of this application will now be described in more detail with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0028] Embodiments of this disclosure are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0029] This disclosure is not exhaustive, but merely illustrative of some embodiments, and is not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined, for example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0030] In each of the disclosed embodiments, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0031] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure.

[0032] In this disclosure, unless otherwise stated, elements expressed in the singular form, such as "a," "an," "the," "the," "the," "the," "the," "the," "this," etc., can mean "one and only one," or "one or more," "at least one," etc. For example, when using articles such as "a," "an," "the," etc. in translation, the noun following the article can be understood as either a singular or a plural expression.

[0033] In some embodiments, the terms “in response to…”, “in response to determining…”, “in the case of…”, “when…”, “if…”, “if…”, etc., can be used interchangeably.

[0034] In some embodiments, the terms “greater than,” “greater than or equal to,” “not less than,” “more than,” “more than or equal to,” “not less than,” “higher than,” “higher than or equal to,” “not lower than,” and “above” can be used interchangeably, as can the terms “less than,” “less than or equal to,” “not greater than,” “less than,” “less than or equal to,” “not more than,” “lower than,” “lower than or equal to,” “not higher than,” and “below”.

[0035] The prefixes such as "first" and "second" in the embodiments of this disclosure are only for distinguishing different descriptive objects and do not constitute restrictions on the position, order, priority, number or content of the descriptive objects. For the description of the descriptive objects, please refer to the description in the claims or the context of the embodiments. The use of prefixes should not constitute unnecessary restrictions.

[0036] In the embodiments disclosed herein, "multiple" refers to two or more.

[0037] In the embodiments disclosed herein, terms such as “import”, “input”, and “read in” can be used interchangeably.

[0038] In some embodiments, devices, etc., can be interpreted as physical or virtual, and their names are not limited to the names recorded in the embodiments. Terms such as “device”, “equipment”, “circuit”, “network element”, “node”, “function”, “unit”, “section”, “system”, “network”, “chip”, “chip system”, “entity”, and “subject” can be used interchangeably.

[0039] In some embodiments, the terms "terminal", "terminal device", "user equipment (UE)", "user terminal", "mobile station (MS)", "mobile terminal (MT)", "subscriber station", "mobile unit", "subscriber unit", "wireless unit", "remote unit", "mobile device", "wireless device", "wireless communication device", "remote device", "mobile subscriber station", "access terminal", "mobile terminal", "wireless terminal", "remote terminal", "handset", "useragent", "mobile client", and "client" can be used interchangeably.

[0040] A cloud phone is a virtual mobile phone device that runs in the cloud. The principle is to use virtualization technologies such as virtual machines on the server to allocate resources such as CPU and memory to the cloud phone, install the corresponding operating system, and virtualize information such as Wi-Fi Media Access Control (Wi-Fi MAC) and International Mobile Equipment Identity (IMEI) to create a virtual simulated mobile phone.

[0041] In related technologies, users can operate cloud phones on terminals such as personal computers (PCs), mobile phones, and tablets. The operation method is to simulate touch operation with a mouse, such as simulating single-point touch on the screen by clicking the mouse and simulating finger swiping by dragging the mouse, thereby achieving precise control of the cloud phone interface.

[0042] However, mouse clicks can only simulate single-point touch, while cloud phone operating systems support complex touch operations such as multi-touch in addition to single-point touch. Mouse clicks cannot simulate these complex touch operations. Furthermore, when a user drags a mouse to click on the cloud phone's interface, it is a virtual operation, not a physical one, resulting in a difference in the speed and accuracy of the simulated touch compared to the speed and accuracy of physical touch when using a real physical phone.

[0043] To address the issues of low speed, low accuracy, and inability to meet complex operational requirements in existing physical devices for virtual interface control, embodiments of this application provide a control method, such as... Figure 1 The diagram shown is a flowchart of the control method provided in an embodiment of this application, including the following steps: S101: Determine at least one functional area in the first interface.

[0044] The first interface is an interactive, visual interface presented to the user. The functional area is the effective control range for triggering functional operations within the first interface.

[0045] For example, the first interface is the game interface when the cloud phone runs a game application, such as... Figure 2 The diagram shown is a schematic representation of a game interface provided in an embodiment of this application. The game interface includes virtual controls for controlling target objects in the game to perform functional operations. Target objects include virtual characters, virtual animals, virtual airplanes, virtual tanks, etc. The functional area is the effective control range within the virtual controls that triggers functional operations. Functional operations are operations that trigger the target object to perform preset actions, including movement, skill release, and item use.

[0046] Different virtual controls have different functions, and each virtual control possesses one or more functions. For example... Figure 2 Taking the game interface shown as an example, Figure 2 The virtual control in the lower right corner has a single function, serving only to trigger specific skills. Figure 2 The virtual control in the lower left corner is multifunctional and can be used to move target objects in the game through different operation directions. Figure 2 The virtual control in the upper right corner is a functional control; triggering it will take you to the function settings interface.

[0047] S102: In response to an input operation on the input device, determine at least one target functional area based on a preset association table, and execute the functional operation corresponding to the target functional area.

[0048] The input device is a device capable of performing human-computer interaction, and can include various forms, including but not limited to keyboards, gamepads, and touchpads. The target functional area is one of at least one functional area. The preset association table is a mapping table between functional areas and input operations.

[0049] If an input operation is pre-associated with a target functional area, the function operation corresponding to that single target functional area is executed. If an input operation is pre-associated with multiple target functional areas, the function operations corresponding to multiple target functional areas are executed simultaneously.

[0050] For example, in a scenario where a personal computer controls a cloud phone, the physical key events of the keyboard on the personal computer can be mapped to the coordinates of functional areas on the cloud phone interface. When a physical key is triggered, the function operation corresponding to the triggering of one or more functional areas is executed. By triggering key operations on the keyboard of the personal computer, single-point or multi-point touch operations on the cloud phone can be completed, realizing indirect physical operation and thus improving the response speed and accuracy of touch.

[0051] The scenarios where users frequently operate cloud phones are mostly gaming scenarios. Figure 2 In the action game interface shown, the game application itself provides virtual touch controls on the game interface, which can map the key events of the personal computer keyboard to the functional areas of one or more virtual controls in the game interface, realizing the simulation of single-point touch or multi-point touch of the personal computer keyboard keys.

[0052] In this embodiment, by establishing a relationship table between the input operation of the input device and the functional area in the first interface, when the input device performs an input operation, the operation corresponding to the target functional area can be directly executed, thus solving the problems of low operation speed, low accuracy, and inability to meet complex operation requirements of existing physical devices for virtual interface control.

[0053] It should be noted that this embodiment describes the implementation logic and operation steps of the control method using a scenario where a personal computer controls a cloud phone. However, it is understood that the technical concept of this method (such as precise detection of functional areas, pre-association mapping between input operations and functional areas, and efficient conversion and execution of cross-device instructions) is not limited to the cloud phone control scenario, and its control principle and technical framework can be flexibly transferred to other scenarios that require precise control.

[0054] In some embodiments, when a first interface is displayed, the user can manually define a functional area through the first interface, mapping the input operations of the input device to the manually defined functional area. However, this method has a high learning cost and is time-consuming, resulting in a poor user experience. Furthermore, the manually defined functional area may not be standardized, and the defined area may easily exceed the control range of the actual functional area, causing touch operation to malfunction. Therefore, in this embodiment, at least one functional area in the first interface can be determined through semantic segmentation models and functional testing.

[0055] In some embodiments, multiple frames of image data can be acquired, input into a semantic segmentation model, and semantic segmentation can be performed to obtain a multi-frame segmentation map. Based on the multi-frame segmentation map, a virtual control region is determined, and within the virtual control region, a functional region is detected.

[0056] The multi-frame image data consists of screenshots of the first interface under different image states, obtained by performing a screenshot operation on the first interface. The virtual control area is the area corresponding to the virtual control on the first interface. The segmentation map is the output of the semantic segmentation model, essentially an image with the same size as the input image data, with each pixel labeled with a category. In this embodiment, the semantic information categories include two types: virtual controls and non-virtual controls. The segmentation map is equivalent to binarized image data, meaning that the pixel area corresponding to the virtual control can be directly regarded as the foreground, and the pixel area corresponding to the non-virtual control can be regarded as the background.

[0057] In some embodiments, the semantic segmentation model is an end-to-end convolutional neural network for pixel-level prediction. For example... Figure 3 The diagram shown is a structural schematic of the semantic segmentation model provided in this embodiment. The semantic segmentation model includes an input layer, an output layer, a convolutional layer (conv), a max-pooling layer (Maxpool), an upsampling layer (upsample), a deconvolutional layer (deconv), and a concatenation layer (Concat) that stacks layers by channel. The convolutional layer includes batch normalization (BN) and activation functions, such as the rectified linear unit (ReLU) function.

[0058] In this embodiment, the main task of the semantic segmentation model is to segment virtual control regions from non-virtual control regions. Constraints can be applied during the training of the semantic segmentation model, resulting in clear and simple semantics. Furthermore, this embodiment replaces some conventional fully connected layers with convolutional layers, making the semantic segmentation model simple and effective. It can take input image data of any size without scaling the image data, preserving the original information, and outputting pixel-level dense predictions of the same size as the original image data.

[0059] It is understandable that upsampling can be considered, to some extent, as deconvolution. For example... Figure 3 As shown, max pooling reduces the size of image data; after four max pooling iterations, the image data size is reduced by a factor of 16. To obtain a segmentation map of the same size as the original image data, the semantic segmentation model adds upsampling and deconvolution to gradually restore the image data size and compensate for the size loss caused during pooling.

[0060] Deconvolution and convolution operate similarly, both involving multiplication and addition. The difference lies in their operation logic: convolution compresses features from many to one by operating on multiple sets of input features with the kernel, while deconvolution expands features from one set of input features with the kernel. In terms of propagation, the forward and backward propagation of deconvolution is simply the reverse of the forward and backward propagation of convolution.

[0061] Furthermore, while semantic segmentation can be achieved through upsampling and deconvolution, the resulting segmentation maps are usually coarse. Therefore, semantic segmentation models introduce skip connections, which connect the feature maps extracted at different stages from the bottom, middle and high layers of the network layer by layer. This improves the ability to grasp local details and global semantics, and fully utilizes the semantic information at each level while preserving high-resolution information, thereby improving segmentation accuracy.

[0062] After upsampling the feature map, the restored feature map is the one extracted by the convolutional kernel in the previous convolutional layer. Due to accuracy limitations, it is difficult to accurately restore the details of the original image. Therefore, as... Figure 3 As shown, the convolutional features after the second max pooling are incorporated into the feature map after the second upsampling. The mid-level features (the output of the convolutional layer after the second max pooling) achieve a better balance between information content and size, retaining sufficient local detail while possessing appropriate semantic abstraction capabilities. By incorporating them into the feature map after the second upsampling, it is equivalent to supplementing the high-level features with refined detail information (similar to filling in missing details during interpolation), thereby effectively alleviating the detail blurring problem caused by simple upsampling.

[0063] In some embodiments, to reduce the false detection rate, preprocessing can be performed on the multi-frame segmentation map after it is obtained. Preprocessing includes smoothing and erosion.

[0064] In some embodiments, the segmentation map contains semantic information about each pixel in the image data. This semantic information can represent the category of the pixel, including virtual control categories and non-virtual control categories. To address this, the detection accuracy of virtual control regions can be improved by pixel-level overlay of multiple frame segmentation maps. Specifically, multiple frame segmentation maps are pixel-level overlaid to obtain a total segmentation map. A first proportion is calculated for each pixel in the total segmentation map, and pixels with a first proportion greater than a first proportion threshold are fitted into a regular shape to obtain the virtual control region.

[0065] The first ratio is the ratio of the number of frames in the multi-frame segmentation map where a pixel belongs to a virtual control to the total number of frames.

[0066] For example, taking a scenario where a personal computer controls a cloud phone, the cloud phone can launch a game application in a virtual environment. By performing semantic segmentation on the game interface through a semantic segmentation model, the virtual control area can be initially detected.

[0067] When the game application is running, the cloud phone enters the game interface for controlling combat actions and randomly clicks on various positions on the interface to change the game interface, reducing the bias of single image data on semantic segmentation. During this process, screenshots are continuously taken to obtain multiple frames of image data. Semantic segmentation model is used to perform semantic segmentation on each frame of image data to obtain the semantic information of each pixel in each frame of image data.

[0068] Because the game screen can affect virtual controls—for example, in cases where some virtual controls are semi-transparent, the game screen itself can be seen through the virtual controls—virtual controls have a certain false detection rate during semantic segmentation.

[0069] To address this, the same semantic segmentation can be performed on all multiple frames of image data obtained from the screenshot to obtain a multi-frame segmentation map. This multi-frame segmentation map can then be smoothed to reduce the false detection rate. Furthermore, erosion processing can be applied to the multi-frame segmentation map to eliminate noise and burrs on the outlines of virtual controls.

[0070] The eroded multi-frame segmentation images are superimposed pixel-by-pixel to obtain a single overall segmentation image. Each pixel in the overall segmentation image possesses multi-dimensional semantic information; that is, each pixel retains the category information from the multi-frame segmentation images (the determination result of whether the pixel belongs to the virtual control class in each frame segmentation image). For a single pixel, based on its semantic information, a first proportion is calculated out of the total number of frames that were determined to be virtual controls in the multi-frame segmentation images. If this first proportion is greater than a preset first proportion threshold, the pixel is determined to belong to the virtual control class. If the first proportion is less than or equal to the preset first proportion threshold, the pixel is determined to belong to the non-virtual control class.

[0071] After pixel-level semantic determination, regular shape fitting is performed on the pixels of the identified virtual control classes. The fitted virtual control region has a shape that is basically consistent with the actual virtual control shape. Based on the shape of the actual virtual control, most of the fitted virtual control regions are circular, and a few are rectangular.

[0072] It should be noted that due to the initial erosion process, the size of the fitted virtual control area will be slightly smaller than the actual virtual control size. However, there is no significant visual difference between the two, and it does not affect user experience. Furthermore, this slightly smaller size ensures the accuracy of the virtual control area fitting, effectively preventing the fitted area from exceeding the touch range of the actual virtual control.

[0073] In some embodiments, the function of a virtual control can be identified and its functional area determined by simulating a user's touch operation on the virtual control in a first interface. To do this, the virtual control area can be sampled according to a first sampling density to obtain multiple first test points. A first touch operation is executed, which triggers a functional operation at a single first test point. After the first touch operation is completed, a first state difference of the target object is calculated, and the functional area is determined based on the first state difference.

[0074] The first sampling density is the sampling density of the virtual control area. It can perform low-density coarse sampling of the virtual control area without traversing every pixel of the virtual control area, which can greatly reduce the amount of computation for sampling and testing and improve the efficiency of functional area detection.

[0075] The target object is the object performing the functional operation in the first interface. The first state difference refers to the state change characteristics of the target object before and after the first touch operation is triggered. For example, with Figure 2 Taking the game interface shown as an example, the first state difference may include changes in the coordinates of the target object, changes in the amount of item usage, and the health bar.

[0076] In some embodiments, after calculating the first state difference of the target object, a first error range can be determined. If the first state difference is within the first error range, and it is determined that the first test point cannot trigger a functional operation, then the first test point is deleted. If the first state difference is not within the first error range, and it is determined that the first test point can trigger a functional operation, then the first test point is retained. Finally, a functional region is generated based on the retained first test points.

[0077] By calculating the first-state differences of the target object, invalid test points without actual control functions can be accurately eliminated, such as the unresponsive center area of ​​virtual buttons and non-control interface elements. Only valid test points that can trigger functional operations are retained, ensuring that the subsequently generated functional areas match the actual control requirements and avoiding invalid mapping.

[0078] In some embodiments, for a retained first test point, a first similarity can be calculated between the first state differences of any two first test points. If the first similarity is greater than or equal to a first similarity threshold, and it is determined that the two first test points can trigger the same functional operation, then the two first test points are assigned to the same functional region. If the first similarity is less than the first similarity threshold, and it is determined that the two first test points can trigger different functional operations, then the two first test points are assigned to different functional regions.

[0079] By calculating the similarity of the differences in the first state, the retained first test points are categorized. Test points that trigger the same or similar control effects, such as the target object moving in the same direction or releasing the same type of skill, can be classified into the same functional area based on the first similarity threshold. Test points that trigger different control effects can be classified into different functional areas based on the first similarity threshold, ensuring the consistency between functional areas and control logic. This achieves precise differentiation of sub-functions within the virtual control, providing a clear correspondence between subsequent input operations from input devices and the mapping of functional areas, avoiding functional confusion.

[0080] In some embodiments, for the retained first test points, connected regions can be generated based on the retained first test points. Sampling is performed on the boundaries of two adjacent connected regions according to a second sampling density to obtain multiple second test points. A second touch operation is executed, which triggers a functional operation on a single second test point. After the second touch operation is completed, a second state difference of the target object is calculated, and a functional region is determined based on the second state difference.

[0081] The second sampling density is the sampling density of the boundary of the connected region, and the second sampling density is greater than the first sampling density. The second state difference is the state change feature of the target object before and after the second touch operation is triggered. The steps for determining the functional region based on the second state difference can refer to the steps for determining the functional region based on the first state difference, and will not be elaborated here.

[0082] In this embodiment, high-density fine sampling and boundary sampling tests are introduced on top of coarse sampling to compensate for the ambiguity in functional area boundary positioning caused by coarse sampling. The second sampling density is greater than the first sampling density, which can perform pixel-level dense detection of connected area boundaries, accurately capturing the boundaries of different functional areas. Furthermore, through second state difference verification, it is further confirmed whether the second test point can trigger a functional operation and the corresponding function affiliation, ensuring that the boundary of the functional area matches the actual effective touch range, avoiding cross-function mis-triggering due to unclear boundaries, and improving the accuracy of functional area positioning.

[0083] For example, following the above embodiment of controlling a cloud phone with a personal computer, for Figure 2 The game interface shown has various virtual controls with different functions. In order to distinguish the functions of the virtual controls and to ensure that the final fitted virtual control area is the area used to control the target object in the game, rather than the area of ​​the function interface trigger control, the fitted virtual control area can be tested by simulating the user's touch operation on the virtual controls in the game interface. Invalid test points are eliminated through testing, and the functions of each virtual control are identified, thereby obtaining the functional area that can trigger specific function operations.

[0084] Specifically, within the game interface, object detection algorithms and similarity algorithms can be used to detect target objects. First, object detection algorithms are used to detect common control objects in the game interface, such as virtual characters, virtual animals, virtual airplanes, and virtual tanks. Then, similarity algorithms are used to calculate the similarity between these control objects across multiple game frames, grouping control objects with similarity scores greater than a similarity threshold into the same control object. The control object that appears most frequently across multiple game frames is selected as the target object that the user actually controls.

[0085] like Figure 4 The diagram shown illustrates the test points of the virtual control provided in this embodiment. Uniform coarse sampling is performed in each fitted virtual control region to obtain multiple test points. Since the number of coarsely sampled test points is small, the subsequent computational load can be reduced.

[0086] At each test point, a click operation is triggered, and the state difference of the target object before and after the click operation is triggered is calculated.

[0087] If, after a click is triggered at a test point, the state difference is within a preset error range, then the test point is considered unsuitable for triggering game control functions and is deleted. For example, the movement button in the game screen ( Figure 2 The center position (bottom left corner) usually has no additional control response after clicking, and the state difference is close to 0.

[0088] If the state differences are all within the preset error range after triggering the click operation at all test points, it indicates that the target object has no response. This means that the fitted virtual control is a functional control, not an object control. Therefore, the fitted virtual control area should be deleted.

[0089] If, after a click operation is triggered at a certain test point, the state difference is not within the preset error range, then the test point is considered to be able to trigger the game's control function, and the test point is retained.

[0090] For all retained test points, calculate the similarity between the state differences of any two test points. If the similarity is greater than or equal to a preset similarity threshold, it means that the two test points can trigger the same functional operation. If the similarity is less than the preset similarity threshold, it means that the two test points trigger different functional operations.

[0091] Test points belonging to the same functional operation are aggregated to form a connected region. Then, the boundary between two adjacent connected regions is found, and pixel-level fine sampling is performed at the boundary to obtain multiple new test points. For multiple new test points, the above state difference calculation and function determination operations are repeated until each functional region with accurate boundaries is obtained.

[0092] After defining the functional areas, key events from the PC keyboard can be mapped to these areas, establishing a connection between keyboard key events and functional areas. This is then integrated into the cloud phone's game assistant, meaning each function of each virtual control is assigned a PC keyboard key. When a user initially plays the game, they can select the desired keyboard keys for the selected virtual control's functional area, completing the personalized mapping settings.

[0093] In single-point touch scenarios, if a keyboard key corresponds to a function area, when a user presses the key on a personal computer's keyboard, the personal computer can report the keyboard key event to a cloud phone. The cloud phone then maps the keyboard key event to a touch event for the corresponding function area, thereby achieving single-point touch.

[0094] In multi-touch scenarios, if a keyboard key corresponds to multiple function areas, when a user presses a key on a personal computer's keyboard, the personal computer can report the keyboard key event to a cloud phone. The cloud phone then synchronously maps the keyboard key event to touch events for multiple corresponding function areas, thereby achieving multi-touch.

[0095] In some embodiments, for performing a function operation corresponding to the target functional area, a touch event of the target functional area can be generated, a touch framework can be invoked, and the touch event can be reported through the touch framework. The touch event is used to trigger a first application to perform a function operation corresponding to the target functional area. The first application is an application that provides a first interface.

[0096] By directly calling the native touch framework of the system to which the first application belongs, the relay process of the touch screen is bypassed, reducing the latency of command transmission and parsing, and improving the response speed of function operation.

[0097] For example, following the above embodiment of controlling a cloud phone with a personal computer, the cloud phone does not have a physical touchscreen, but it has a control operating system. Therefore, the cloud phone can call its own operating system's touch framework to directly transmit simulated touch events. For instance, when the cloud phone is configured with an Android system, the sendPointerSync() method in the Instrumentation class can be used to distribute touch events, reporting them to the game application corresponding to the game interface to perform functional operations, bypassing the touchscreen's relay stage and improving response speed.

[0098] In some embodiments, to improve the simulation accuracy, after determining the target functional area, the target functional area can be divided into multiple sub-regions. Based on the distance between each sub-region and the center point of the target functional area, the multiple sub-regions are copied, and the copied sub-regions are stored in a first array. In the first array, a target sub-region is selected, and a touch event for that target sub-region is generated to trigger the execution of the corresponding functional operation.

[0099] The number of copies is negatively correlated with distance; that is, the closer the distance to the center point of the target functional area, the more copies are made. The target sub-region is any sub-region in the first array.

[0100] In this embodiment, by segmenting sub-regions, touch events are generated by randomly selecting target sub-regions, achieving randomness in touch position and avoiding triggering touch events at fixed positions, thus circumventing the application's anti-cheating detection mechanism. Furthermore, sub-regions are copied according to their distance from the center point of the target functional area, ensuring that the randomly selected target sub-regions are mostly efficient touch areas of the functional area, reducing the probability of operation failure due to edge areas.

[0101] In some embodiments, after a touch event is generated for a target sub-region, the target sub-region can be removed from the first array and stored in the second array. A counter is set to record the number of times the touch event is reported. When the counter value equals a preset threshold, the target sub-region is removed from the second array and stored in the first array.

[0102] By scheduling sub-regions using a dual array, the randomness of touch positions is further optimized, avoiding frequent selection of a single sub-region and ensuring more uniform randomness of touch positions during long-term operation.

[0103] For example, following the above embodiment of controlling a cloud phone with a personal computer, the mapping of keyboard keys to virtual control functions on a personal computer is mainly applied to game scenarios. Game manufacturers have configured anti-cheat detection mechanisms to combat cheating. If the same function of the same virtual control is simulated at the same location (such as the midpoint), the regularity of the operation is easily judged by the system as script behavior, thus facing the risk of being targeted by cheating.

[0104] Therefore, to avoid being identified as a script by the game's anti-cheat system due to fixed trigger locations, a randomness mechanism can be added. When mapping the same function on the same virtual control to personal computer keyboard keys, randomness can be added to improve the simulation.

[0105] For functional areas on virtual controls, these areas can be divided into multiple sub-areas according to preset basic units, and these sub-areas can be numbered. Then, based on the distance of each sub-area from the center point of the functional area, different sub-areas are copied to generate multiple sub-areas with the same number. Sub-areas closer to the center point are copied more times, increasing their hit rate and thus improving the simulation accuracy. By copying more areas near the center, the probability of selecting a sub-area near the center is higher during subsequent random selections, which both conforms to real-world habits and ensures the hit rate of user operations.

[0106] After copying the sub-regions, all numbered sub-regions are stored in the first array. When a user presses a key on the PC keyboard, the PC reports the keyboard key event to the cloud phone. After receiving the keyboard key event, the cloud phone first determines the functional area corresponding to the keyboard key event, calls the first array corresponding to the functional area, and uses a random algorithm to randomly select a number from the first array. Then, it reports the touch event at the pixel position corresponding to the sub-region with that number.

[0107] Simultaneously, all sub-regions with that number are removed from the first array and added to the second array. A counter is set for each sub-region, incrementing the counter by one for each reported touch event. When the counter reaches a preset threshold, all sub-regions with that number are removed from the second array and added back to the first array. This ensures randomness while also guaranteeing a higher probability of selecting a sub-region near the center, and avoids repeatedly selecting the same sub-region within a short period, thus balancing anti-detection measures and user experience.

[0108] In the above embodiments, the virtual control area is initially segmented by semantic segmentation of the game screen, followed by smoothing and erosion processing to obtain the virtual control area. Then, invalid test points are eliminated through testing, and the various functional areas of the virtual control are identified. When dealing with a massive number of games, this method can efficiently detect the various functional areas of the virtual control without requiring the user to manually define the virtual control area, resulting in a low learning cost and short processing time.

[0109] Mapping PC keyboard key events to the functional areas of virtual controls enables both single-point and multi-point touch. Pressing a PC keyboard key is a physical operation, offering speed and precision, thus elevating the virtual operation of cloud phones to the level of physical operation. Furthermore, adding randomness to the mapping of PC keyboard key events to the functional areas of virtual controls improves the simulation's fidelity and circumvents anti-cheat detection mechanisms.

[0110] In existing scenarios where PCs control cloud phones, users operate the cloud phone with a mouse. However, in games with frequent controls, this results in slow operation speed, low accuracy, and an inability to meet complex operational needs, leading to a poor user experience. Compared to using a professionally configured game controller, mapping the keys on a PC keyboard to the game's virtual controls not only eliminates the need for additional hardware costs but also ensures compatibility with most PCs equipped with keyboards, making it an economical and versatile solution.

[0111] Furthermore, this embodiment provides a semi-automated functional area detection process, which can efficiently identify the functional areas of virtual controls and has good versatility. It eliminates the need for users to manually set the functional areas of virtual controls, reducing labor costs, improving adaptation efficiency, and avoiding human error.

[0112] According to embodiments of this disclosure, a control device is also provided. For example... Figure 5 The diagram shown is a schematic representation of a control device provided in a disclosed embodiment. The device includes: The first processing module 501 is used to determine at least one functional area in the first interface. The functional area is the effective control range for triggering functional operations.

[0113] The second processing module 502 is used to respond to an input operation on the input device, determine at least one target functional area based on a preset association table, and execute the functional operation corresponding to the target functional area.

[0114] The target functional area is one of at least one functional area. The preset association table is a mapping table between functional areas and input operations.

[0115] In some embodiments, the first processing module 501 is used to acquire multi-frame image data, input the multi-frame image data into a semantic segmentation model, perform semantic segmentation to obtain a multi-frame segmentation map, determine a virtual control region based on the multi-frame segmentation map, and detect a functional region within the virtual control region.

[0116] Among them, the multi-frame image data are screenshots of the first interface under different image states.

[0117] In some embodiments, the first processing module 501 is configured to perform preprocessing on the multi-frame segmentation map. The preprocessing includes smoothing and erosion.

[0118] In some embodiments, the first processing module 501 is used to perform pixel-level overlay on the multi-frame segmented images to obtain a total segmented image. A first proportion of each pixel in the total segmented image is calculated, and pixels with a first proportion greater than a first proportion threshold are fitted into a regular shape to obtain a virtual control region.

[0119] The first ratio is the ratio of the number of frames in the segmented image of multiple frames where the pixel belongs to the virtual control to the total number of frames.

[0120] In some embodiments, the first processing module 501 is configured to sample the virtual control area according to a first sampling density to obtain a plurality of first test points. A first touch operation is executed, which triggers a functional operation on a single first test point. After the first touch operation is completed, a first state difference of the target object is calculated. If the first state difference is within a first error range, the first test point is deleted. If the first state difference is not within the first error range, the first test point is retained. A functional area is generated based on the retained first test points.

[0121] The target object is the object that performs the functional operation in the first interface.

[0122] In some embodiments, the first processing module 501 is configured to calculate a first similarity between the first state differences of any two first test points. If the first similarity is greater than or equal to a first similarity threshold, the two first test points are assigned to the same functional region. If the first similarity is less than the first similarity threshold, the two first test points are assigned to different functional regions.

[0123] In some embodiments, the first processing module 501 is configured to generate connected regions based on the retained first test points. Sampling is performed on the boundaries of two adjacent connected regions according to a second sampling density to obtain multiple second test points. A second touch operation is executed, which triggers a functional operation on a single second test point. After the second touch operation is completed, a second state difference of the target object is calculated, and a functional region is determined based on the second state difference.

[0124] The second sampling density is greater than the first sampling density.

[0125] In some embodiments, the second processing module 502 is used to generate touch events for the target functional area, invoke a touch framework, and report the touch events through the touch framework. The touch events are used to trigger a first application to perform the functional operation corresponding to the target functional area. The first application is the application that provides the first interface.

[0126] In some embodiments, the second processing module 502 is configured to determine multiple target functional areas in response to an input operation on an input device, and simultaneously execute functional operations corresponding to the multiple target functional areas.

[0127] In some embodiments, the second processing module 502 is configured to divide the target functional area into multiple sub-regions, and copy the multiple sub-regions according to the distance between the sub-regions and the center point of the target functional area. The copied sub-regions are then stored in a first array. From the first array, a target sub-region is selected, and a touch event for the target sub-region is generated.

[0128] The number of copies is negatively correlated with distance. The target sub-region is any sub-region in the first array.

[0129] In some embodiments, the second processing module 502 is configured to, after the step of generating the touch event of the target sub-region, remove the target sub-region from the first array and store it in the second array. A counter is set to record the number of times the touch event is reported. When the value of the counter equals a preset threshold, the target sub-region is removed from the second array and stored in the first array.

[0130] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0131] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0132] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 602 or loaded from storage unit 608 into RAM (Random Access Memory) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An I / O (Input / Output) interface 605 is also connected to the bus 604.

[0133] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0134] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as control methods. In some embodiments, the control methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the aforementioned control method by any other suitable means (e.g., by means of firmware).

[0135] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0139] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0140] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0141] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0142] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0143] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A control method, characterized in that, include: Identify at least one functional area in the first interface; The functional area is the effective control range for triggering functional operations; In response to an input operation on an input device, at least one target functional area is determined based on a preset association table, and the functional operation corresponding to the target functional area is executed; the target functional area is one of the at least one functional area. The preset association table is a mapping table between the functional areas and input operations.

2. The method according to claim 1, characterized in that, Determining at least one functional area in the first interface includes: Collect multiple frames of image data; the multiple frames of image data are screenshots of the first interface under different image states; The image data from multiple frames is input into the semantic segmentation model for semantic segmentation to obtain a multi-frame segmentation map. Based on the segmentation map described in multiple frames, the virtual control area is determined; In the virtual control area, the functional area is detected.

3. The method according to claim 2, characterized in that, The step of determining the virtual control region based on the segmentation map of multiple frames includes: The segmented images from multiple frames are then superimposed pixel-by-pixel to obtain the final segmented image. Calculate the first proportion of each pixel in the overall segmentation image; the first proportion is the ratio of the number of frames in the segmentation image in multiple frames to the number of frames to which the pixel belongs to the virtual control to the total number of frames; The pixels with a ratio greater than the first ratio threshold are fitted into a regular shape to obtain the virtual control area.

4. The method according to claim 3, characterized in that, The detection of the functional area within the virtual control area includes: According to the first sampling density, sampling is performed on the virtual control area to obtain multiple first test points; Execute a first touch operation; the first touch operation is used to trigger a functional operation of a single first test point; After the first touch operation is completed, the first state difference of the target object is calculated; the target object is the object that performs the functional operation in the first interface. If the first state difference is within the first error range, delete the first test point; If the first state difference is not within the first error range, the first test point is retained; The functional area is generated based on the retained first test point.

5. The method according to claim 4, characterized in that, The step of generating the functional region based on the retained first test point includes: Calculate the first similarity between the first state differences of any two first test points; If the first similarity is greater than or equal to the first similarity threshold, the two first test points are classified into the same functional region. If the first similarity is less than the first similarity threshold, the two first test points are assigned to different functional regions.

6. The method according to claim 5, characterized in that, The step of generating the functional region based on the retained first test point includes: Based on the retained first test point, generate a connected component; According to the second sampling density, sampling is performed on the boundary between two adjacent connected regions to obtain multiple second test points; the second sampling density is greater than the first sampling density. Perform a second touch operation; the second touch operation is used to trigger a functional operation on a single second test point; After the second touch operation is completed, the second state difference of the target object is calculated; The functional area is determined based on the second state difference.

7. The method according to claim 1 or 2, characterized in that, The execution of the function operation corresponding to the target functional area includes: Generate touch events for the target functional area; Invoke the touch framework; The touch event is reported through the touch framework; the touch event is used to trigger the first application to perform the function operation corresponding to the target functional area; the first application is the application that provides the first interface.

8. The method according to claim 1, characterized in that, The method further includes: In response to an input operation on an input device, multiple target functional areas are determined, and multiple functional operations corresponding to the target functional areas are executed simultaneously.

9. The method according to claim 7, characterized in that, The generation of touch events for the target functional area includes: The target functional area is divided into multiple sub-regions; Based on the distance between the sub-region and the center point of the target functional region, multiple sub-regions are copied; the number of copies is negatively correlated with the distance. The copied sub-regions are stored in the first array; In the first array, a target sub-region is selected; the target sub-region is any one of the sub-regions in the first array. A touch event is generated for the target sub-region.

10. The method according to claim 9, characterized in that, After the step of generating the touch event for the target sub-region, the following steps are included: Remove the target sub-region from the first array and store it in the second array; Set a counter; the counter is used to record the number of times touch events are reported; If the value of the counter equals a preset threshold, the target sub-region is removed from the second array and stored in the first array.

11. The method according to claim 2, characterized in that, After the step of obtaining the multi-frame segmentation map, the method further includes: Preprocessing is performed on the segmented images of multiple frames; the preprocessing includes smoothing and erosion.

12. A control device, characterized in that, include: The first processing module is used to determine at least one functional area in the first interface; The functional area is the effective control range for triggering functional operations; The second processing module is configured to respond to an input operation on the input device, determine at least one target functional area based on a preset association table, and execute the functional operation corresponding to the target functional area; the target functional area is one of the at least one functional area. The preset association table is a mapping table between the functional areas and input operations.

13. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-11.

14. A computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

15. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images