Control triggering method, electronic equipment, storage medium and vehicle

Abstract

The invention relates to a control triggering method, electronic equipment, a storage medium and a vehicle, and relates to the field of interaction.The method comprises the steps that a control triggering request is received, and shielding information of a target control in a screen is determined according to target control information of the target control corresponding to the control triggering request; and according to the shielding information, performing a triggering operation on the target control. According to the method, the shielding information of the target control in the screen is determined according to the target control information of the target control corresponding to the control triggering request, and then the target control is triggered according to the shielding information, so that the success rate of triggering the screen control can be improved.

Description

Technical Field

[0001] The present application relates to the field of interactive technology, and in particular to a control triggering method, electronic equipment, storage medium and vehicle. Background Art

[0002] With the development of electronic information technology, mobile phones, tablet computers and other devices are widely used in people's lives. Users need to click on the screen to operate the device. In this process, a large number of tedious click operations often occur to trigger the corresponding controls. In order to simplify user operations, some solutions for automatically triggering controls are provided in the relevant technology. By comparing the image of the control and the current screen image to determine the triggering operation position, the control is automatically triggered. However, this solution often fails to trigger, resulting in a decrease in the success rate of triggering the screen control. Summary of the invention

[0003] The embodiments of the present application provide a control triggering method, an electronic device, a storage medium and a vehicle, which improve the success rate of triggering screen controls to at least partially solve the above-mentioned technical problems.

[0004] In order to achieve the above-mentioned object, according to a first aspect of the present application, a control triggering method is provided, the method is applied to an electronic device, the electronic device is provided with a screen, the method comprises:

[0005] receiving a control trigger request, and determining, according to target control information of a target control corresponding to the control trigger request, shielding information of the target control on the screen;

[0006] A trigger operation is performed on the target control according to the shielding information.

[0007] According to a second aspect of the present application, an electronic device is provided, which is applied to an electronic device, wherein the electronic device is provided with a screen, including:

[0008] A transceiver module, configured to receive a control trigger request, and determine the shielding information of the target control on the screen according to the target control information corresponding to the control trigger request;

[0009] A trigger module is used to trigger the target control according to the shielding information.

[0010] According to a third aspect of the present application, an electronic device is also provided, comprising a processor, wherein the processor is connected to a memory, the memory stores a computer program, and the processor is used to run the computer program in the memory to execute any of the above-mentioned control triggering methods.

[0011] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned control triggering methods is performed.

[0012] According to a fifth aspect of the present application, a computer program product is provided, which includes a computer program, and the computer program is executed by a processor to implement any of the above-mentioned control triggering methods.

[0013] According to a sixth aspect of the present application, a vehicle is provided, comprising the electronic device as described above, or comprising the electronic device as described above, or executing the control triggering method as described above.

[0014] The control triggering method provided in the embodiment of the present application can, after receiving a control triggering request, determine the occlusion information of the target control on the screen according to the target control information corresponding to the control triggering request, and then trigger the target control according to the occlusion information, thereby avoiding the inability to trigger the target control due to occlusion, and improving the success rate of triggering the screen control.

[0015] Other features and advantages of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0017] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same figure numbers represent the same parts in the following description.

[0018] Figure 1 is a flowchart of a control triggering method provided in one embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of a screen scene provided in one embodiment of the present invention;

[0020] Figure 3 is a schematic structural diagram of an electronic device provided in one embodiment of the present invention;

[0021] Figure 4 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0023] With the development of electronic information technology, mobile phones, tablet computers and other devices are widely used in people's lives. These devices have screens that can interact with users. Users can click on some positions on the screen with their fingers or mice to trigger the controls on the screen to operate the device. When users click on the device screen to operate, a large number of tedious operations may occur. The process of users performing repeated mechanical operations is time-consuming and laborious, which will reduce the user experience. Therefore, some solutions for automatically triggering controls are provided in related technologies:

[0024] Acquire real-time screen information, set a matrix frame on the screen information, load the control image to be controlled in the matrix frame, make the loaded control image overlap with the matrix frame, and determine whether the control image on the matrix frame overlaps with the screen image. If so, generate a triggered operation position. Otherwise, start acquiring real-time screen information again, control the clicker to move toward the generated operation position, and automatically trigger the control.

[0025] The inventors discovered that: there may be more than one display content, such as a control, on the screen, and the control may be blocked by other display content, such as pop-up windows, pictures, and interfaces, resulting in the inability to automatically trigger the control.

[0026] Secondly, controlling the clicker to move toward the generated operation position and perform the trigger operation requires a certain amount of movement time, resulting in a slow trigger speed and the need to install clicker hardware, which affects the user experience.

[0027] In order to solve the above problems, the embodiments of the present application propose a control triggering method, an electronic device, a storage medium, a vehicle, etc. The embodiments of the present application determine the occlusion information of the target control on the screen according to the target control information corresponding to the control trigger request, and automatically trigger the target control in combination with the occlusion information, thereby improving the success rate of the triggering operation.

[0028] Specifically, the control triggering method in the present application can be applied to an electronic device, which is provided with a screen, and the control can be displayed on the screen to interact with the user. The electronic device can be a mobile terminal such as a mobile phone, a tablet, a computer, or a vehicle equipped with a screen, such as a car.

[0029] In the following, each embodiment is described in detail by taking the electronic device as an example in which the execution subject of the control triggering method is an electronic device.

[0030] Correspondingly, if Figure 1 As shown, the control triggering method may include the following steps:

[0031] S10, receiving a control trigger request, and determining, according to target control information of a target control corresponding to the control trigger request, shielding information of the target control on the screen;

[0032] In this embodiment, the electronic device can receive a control trigger request. The control trigger request is input in advance or in real time by the user based on his or her operation requirements for the controls displayed on the screen of the electronic device. The control trigger request can determine the target control information that needs to be triggered, and can also determine the operation information for the target control, such as the operation type, operation time, number of operations, etc.

[0033] After receiving a control trigger request, the target control information of the target control corresponding to the control trigger request can be determined according to the control trigger request. The target control is the control that is to be automatically triggered by the control trigger request, and is also the control that will appear on the screen of the electronic device. The target control is information related to the target control, and the target control can be determined on the screen through the target control information. After finding the corresponding target control on the screen according to the target control information, the occlusion information of the target control on the screen can be obtained. Since being blocked and not being blocked are relative, the occlusion information in this embodiment can characterize the blocked or not blocked situation of the target control on the screen.

[0034] In this embodiment, the occlusion information may include at least one occlusion type, and the occlusion type is used to distinguish different occlusion degrees or different non-occlusion degrees of the target control on the screen.

[0035] In some embodiments, the occlusion information may further include an occlusion area, and the occlusion area may be used to represent a specific occluded area or an unoccluded area of ​​the target control on the screen.

[0036] S20. Trigger the target control according to the shielding information.

[0037] In this embodiment, the target control is triggered in combination with the occlusion information to ensure the timing or position of the automatic triggering of the target control. The trigger operation may include a click operation, a double-click operation, a long press operation, a shortcut key operation, etc. Optionally, the control trigger request may also include operation information for the target control, and the operation information may include the operation type, operation time, operation number, etc. Based on the occlusion information, the trigger operation corresponding to the operation information is performed on the target control to improve the trigger success rate.

[0038] In some embodiments, the time when the target control is not obstructed can be determined based on the occlusion information. During the unobstructed time, the overlapping area of ​​the screen image with the image corresponding to the target control is determined. Since the target control is not obstructed, the overlapping area can be easily determined. The overlapping area is the display area where the target control is located. According to the center coordinates of the overlapping area, the triggered operation position is generated, and the trigger operation is performed on the operation position, so as to successfully realize the automatic triggering of the target control and improve the trigger success rate.

[0039] Optionally, triggering the target control according to the shielding information includes:

[0040] Determining an operation position of the target control according to the occlusion information;

[0041] A trigger operation is performed on the target control according to the operation position.

[0042] In this embodiment, the unobstructed area in the target control can be determined based on the occlusion information. The unobstructed area belongs to the unobstructed area on the target control and is also the position where the target control can be displayed on the screen. Then, the unobstructed area can be set as the operation position belonging to the coordinate system where the screen is located, and the corresponding trigger operation is performed for the operation position, so as to realize the trigger operation of the target control. In this way, the trigger operation of the target control can be realized at any time when the target control is not completely obstructed, thereby further improving the trigger success rate and efficiency.

[0043] Optionally, triggering the target control according to the operation position includes:

[0044] The control trigger request is simulated through the Android accessibility service to perform a trigger operation on the operation position.

[0045] In this embodiment, the control trigger request can be simulated through the Android accessibility service, and the trigger operation can be performed based on the operation position, thereby realizing the trigger operation of the target control on the screen. The Android accessibility service is a system-level interface that can simulate operations, which is generally used to help people with disabilities use mobile phones. In this embodiment, it is used to perform the task of automatically triggering the target control. It simulates the user operation to control the electronic device by simulating the control trigger request.

[0046] Specifically, the operation position will be set as the screen coordinates triggered by the simulated gesture, and the operation position will be passed into the Android accessibility service as a simulation parameter, so that the Android accessibility service simulates the gesture to trigger the operation position, so as to achieve the trigger operation of the target control. Optionally, the operation information included in the control trigger request can also be passed into the Android accessibility service as a simulation parameter to simulate the gesture to perform the trigger operation corresponding to the control trigger request on the target control. Optionally, when it is determined according to the operation information that the target control needs to be repeatedly triggered, such as continuously clicking the target control, the average duration information obtained by statistics of people manually clicking the screen control quickly, the minimum interval duration information supporting repeated clicks obtained by statistics, and other information can also be used as simulation parameters, and passed into the Android accessibility service together with the operation position, operation information, etc., to simulate gestures to perform repeated trigger operations.

[0047] In this way, there is no need to install an additional clicker to trigger the target control. The trigger operation can be directly performed on the operation position, which saves the movement time of the clicker and improves the trigger efficiency.

[0048] In some embodiments, the occlusion information includes a specific unoccluded area of ​​the target control, and then the coordinates of its display center are determined based on the specific unoccluded area and set as the setting operation device.

[0049] In this embodiment, the occlusion information includes at least one of unoccluded, partially occluded, and fully occluded;

[0050] The step of determining the operation position of the target control according to the shielding information includes:

[0051] When the occlusion information indicates that the target control is not occluded, the operation position is the control center coordinates of the target control;

[0052] In a case where the occlusion information indicates that the part is occluded, determining the operation position according to an unoccluded area of ​​the target control;

[0053] When the occlusion information indicates that the entire control is occluded, the target control cannot be triggered.

[0054] In this embodiment, different degrees of occlusion are classified to obtain three types of occlusion: unoccluded, partially occluded, and completely occluded. According to the target control information, it can be determined that the occlusion information of the target control includes at least one of the three types of occlusion to characterize the degree of occlusion of the target control. Then, according to the degree of occlusion of the target control, the operation position corresponding to the trigger operation on the screen can be determined. By determining the operation position by the occlusion type in the occlusion information, the success rate of the trigger and the efficiency of the trigger operation can be improved.

[0055] Wherein, when the occlusion information indicates that the object is not occluded, the operation position is the control center coordinate of the target control;

[0056] In a case where the occlusion information indicates that the part is occluded, determining the operation position according to an unoccluded area of ​​the target control;

[0057] When the occlusion information indicates that the entire control is occluded, the target control cannot be triggered.

[0058] In this embodiment, if the occlusion type is unobstructed, the target control is completely displayed on the screen, and the control center coordinates of the target control can be set as the operation position. The control center coordinates are the position coordinates of the center of the target control on the screen. When the occlusion information is unobstructed, the coordinates will definitely not be obscured by other windows. When the control trigger request is simulated based on the Android accessibility service and the control center coordinates of the target control are used as the operation position for the trigger operation, it is equivalent to directly clicking the target control on the screen, thereby realizing the trigger operation.

[0059] If the occlusion information is partially occluded, the target control is partially occluded, and the target control includes an occluded area and an unoccluded area. The occluded area is the occluded area of ​​the target control, and the unoccluded area is the unoccluded area of ​​the target control. On the target control, except for the occluded area, there is an unoccluded area. In the case where the occlusion information is partially occluded, the relevant information of the unoccluded area of ​​the target control is obtained to set the operation position so that the operation position can fall on the unoccluded area of ​​the target control, so that the Android accessibility service can be called to simulate the control trigger request, and the target control is touched at the operation position, so that the target control can be triggered. In the case where the occlusion type is fully occluded, the target control cannot be touched, the target control cannot be triggered, and the operation position is not set.

[0060] In this way, the accurate operation position can be determined through the shielding type of the target control to trigger the target control, which can improve the success rate of the trigger and the efficiency of the trigger operation.

[0061] Furthermore, the display center coordinates of the unblocked area may be determined and set as the operation position.

[0062] In this embodiment, the display center coordinates of the unobstructed area refer to the coordinates of the center of the unobstructed area on the screen. The display center coordinates must be unobstructed, so they can be set as the operation position. After the display center coordinates are set as the operation position, the Android accessibility service can simulate a control trigger request and perform a trigger operation on the display center coordinates of the unobstructed area of ​​the target control, so that the target control can be successfully triggered when it is partially obstructed, thereby improving the success rate of control triggering.

[0063] Optionally, when the occlusion information of the target control includes the occlusion type of partial occlusion, an occlusion mark can also be set for the unobstructed area, so that the unobstructed area includes at least one occlusion mark of right occlusion, left occlusion, upper occlusion and lower occlusion. According to the occlusion mark of the target control, the orientation of the unobstructed area relative to the control center coordinates of the target control can be determined when the target control is partially obstructed, and then the display center coordinates of the unobstructed area can be determined based on this orientation and set as the operation position. In this way, in the case of partial occlusion, the operation position that can successfully trigger the target control can be quickly determined, thereby improving the success rate and efficiency of control triggering.

[0064] Further, the determining the display center coordinates of the unblocked area includes:

[0065] The coordinates of the corner points of the control in the unblocked area are obtained, and the coordinates of the display center are calculated according to the coordinates of the corner points of the control.

[0066] In this embodiment, the control corner coordinates of the target control and the window corner coordinates of each window in the screen can be obtained to determine the unobstructed area of ​​the target control, and then determine the control corner coordinates belonging to the unobstructed area. The control corner coordinates are the coordinates of the corners on the target control outline in the screen, and the window corner coordinates are the coordinates of the corners on the window outline in the screen. The control corner coordinates of the unobstructed area are the control corner coordinates belonging to the unobstructed area. According to the control corner coordinates of the unobstructed area of ​​the target control, the display center coordinates of the unobstructed area in the screen can be determined, and then the display center coordinates can be set as the operation position for triggering operation, so that the target control can be successfully triggered in the case of partial obstruction, further improving the success rate of control triggering.

[0067] Further, the determining, according to the target control information of the target control corresponding to the control trigger request, the shielding information of the target control on the screen includes:

[0068] Based on the target control information, determining node information of the target control on the screen;

[0069] Determine the occlusion information according to the node information and the window information of the target window on the screen;

[0070] The target window includes windows in the screen except the window to which the target control belongs.

[0071] In this embodiment, based on the target control information, the node information of the target control on the screen can be obtained to represent the relevant information of the target control on the screen. In some embodiments, the target control information itself includes this node information, so this node information can be extracted from the target control information.

[0072] However, in some embodiments, the target control information needs to be determined according to the control trigger request through the cloud and local semantic algorithm modules. There is a certain delay between the time when the cloud and local semantic algorithm modules obtain screen-related information, such as scene information, and the time when the target control information is output, and the control on the screen may change, such as disappearing, changing position, etc. At this time, the control position, hierarchy and other node information of the target control information will become invalid and will not match the actual information on the current screen. For example, the target control is no longer displayed or its position has changed, so it is necessary to determine the real-time node information of the target control on the screen through the target control information.

[0073] Specifically, the interface information corresponding to each node on the current screen can be queried. The nodes are screen interface elements such as controls, text boxes, menus, icons, etc. in each window on the screen. Each node corresponds to interface information such as node identification, window level to which the node belongs, and node position. Based on the target control information, the target node matching the target control is determined from each node on the current screen, and the interface information of the target node on the screen is obtained as the node information of the target control on the screen to obtain the real-time node information of the target control. The method of judging the target node corresponding to the target control through the target control information can also be used to judge whether the target control is still on the screen to further improve the trigger success rate.

[0074] The window information of all windows in the current screen is obtained through the accessibility service, including but not limited to the window level, type, coordinates, etc. In this embodiment, the window information that needs to be obtained is the window information of the target window, and the target window includes windows in the screen except the window to which the target control belongs, and the window information is the relevant information of the target window in the screen.

[0075] Windows other than the window to which the target control belongs may obstruct the target control. Therefore, it is possible to determine whether the target control is at least one of the following obstruction types: unobstructed, partially obstructed, and fully obstructed, based on the node information of the target control on the screen and the window information of the target window, and use this as the obstruction information to accurately determine the obstruction information and further improve the success rate of the triggering operation.

[0076] Further, the node information of the target control includes the first level and / or control position of the window to which the target control belongs, and the window information of the target window may include the second level and / or window position of the target window. According to the node information and the window information of the target window in the screen, at least one occlusion type is determined as the occlusion information, including:

[0077] If the first level is greater than the second level, the occlusion information of the target control is determined to be unoccluded.

[0078] In this embodiment, based on the first level of the window to which the node corresponding to the target control in the node information belongs, and the second level of the target window in the window information, or based on the node position corresponding to the target control in the node information, and the window position of the target window in the window information, or based on the first level, the second level, the node position and the window position, it is possible to identify the situation where the target control is obscured by other windows, which is used to determine at least one obstruction type of the target control and set it as the obstruction information.

[0079] In this embodiment, the node information includes the first level of the window to which the target control corresponding node belongs, and the window information includes the second level of the target window. The first level and the second level are compared first. If the second level of each target window is smaller than the first level, it means that the level shown by the target control is the largest on the screen, and it is displayed on the top layer of other target windows and will not be blocked. Therefore, it can be directly determined that the blocking information of the target control is the blocking type of not being blocked.

[0080] In an alternative scheme of the present embodiment, the node information may include the control position of the target control, and the window information includes the window position of the target window. The occlusion type of the target control is determined based on the control position and the window position. If the screen area corresponding to the control position does not overlap with the screen area corresponding to the window position, it indicates that no window will obstruct the target control regardless of the level, and it can be determined that the occlusion information of the target control includes the occlusion type of not being obstructed.

[0081] In this way, according to the first level and the second level, or according to the control position and the window position, it is prioritized to determine whether the target control is not blocked and set it as the blocking information. When it is not blocked, the execution process of the trigger operation is simpler, and the center coordinates of the control can be directly set as the operation position, thereby saving calculation amount and improving trigger efficiency.

[0082] Furthermore, the method further comprises:

[0083] If the first level is smaller than the second level, then if the first level is smaller than the second level, the shielding information is determined according to the control position and the window position.

[0084] In this embodiment, if the second level of the target window is greater than the first level, it is not possible to prioritize the occlusion information of the target control as being of the occlusion type of being unoccluded, so the occlusion type of the occlusion information can be further determined based on the control position in the node information and the window position in the window information. This allows the occlusion information of the target control to be quickly and accurately determined for triggering operations, further improving the success rate and efficiency of triggering.

[0085] Further, the control position includes control center coordinates and multiple control corner point coordinates, and the window position includes multiple window corner point coordinates; and determining the occlusion type according to the control position in the node information and the window position in the window information includes:

[0086] If the center coordinates of the control are not within the range enclosed by the coordinates of the multiple window corner points, determining that the occlusion information is not occluded;

[0087] If the coordinates of the corner points of the plurality of controls are all within the range enclosed by the coordinates of the corner points of the plurality of windows, then determining that the occlusion information is all occluded;

[0088] If the coordinates of the multiple control corner points are partially within the range surrounded by the multiple window corner point coordinates, and the center coordinates of the control are within the range surrounded by the multiple window corner point coordinates, then it is determined that the occlusion information is that the portion is occluded.

[0089] In this embodiment, the control position includes the control center coordinates and control corner coordinates of the target control on the screen, the control corner coordinates are the coordinates of the corners formed by the target control outline on the screen, and the window position includes the window corner coordinates of the target window on the screen, the window corner coordinates are the coordinates of the corners formed by the target window outline on the screen. First, according to the control center coordinates in the control position and the window corner coordinates in the window position, it can be determined whether the occlusion information is not occluded.

[0090] The center coordinates of the control are the coordinates of the center of the target control on the screen. The center coordinates of the control are generally the default operation position for triggering the target control. By comparing the center coordinates of the control with the coordinates of the window corners, it can be determined whether the center coordinates of the control are within the range surrounded by multiple window corner coordinates. The range surrounded by multiple window corner coordinates is the range where the target window is located. If the center coordinates of the control are not within the range surrounded by multiple window corner coordinates, it means that the operation position triggered by the center coordinates of the control as the target can accurately imitate the user's triggering operation on the target control, and then it can be determined whether the occlusion information is not occluded. On the contrary, if the center coordinates of the control are within the range surrounded by multiple window corner coordinates, it means that its occlusion information is definitely not occluded.

[0091] After determining that the occlusion information is definitely not unoccluded, it is possible to further determine whether the occlusion information is partially or completely occluded based on the control corner coordinates and the window corner coordinates. Continue to use the control corner coordinates in the control position to compare with the window corner coordinates to determine whether the occlusion information is partially or completely occluded. If the control corner coordinates are all within the range surrounded by multiple window corner coordinates, it is determined that the occlusion information is unoccluded. If the control corner coordinates are partially within the range surrounded by the window corner coordinates, it is determined that the occlusion information is partially occluded.

[0092] Optionally, for the case of partial occlusion, the occlusion information can be further subdivided to add occlusion marks for the unobstructed areas in the partial occlusion case. The control corner coordinates can include at least one of the upper left corner coordinates, lower left corner coordinates, upper right corner coordinates and lower right corner coordinates. Based on the control corner coordinates in each direction, the occlusion mark of the unobstructed area can be determined in the case of partial occlusion, and it can also be used as occlusion information.

[0093] Specifically, if the coordinates of the upper left corner and / or the lower left corner are within the range, it is marked as the left side is blocked; if the coordinates of the upper right corner and / or the lower right corner are within the range, it is marked as the right side is blocked; if the coordinates of the lower left corner and / or the lower right corner are within the range, it is marked as the lower side is blocked; if the coordinates of the upper left corner and / or the upper right corner are within the range, it is marked as the upper side is blocked. If the number of occlusion marks in the unobstructed area is less than the preset threshold in the case of partial occlusion, if it is equal to or exceeds the preset threshold, it also indicates that the target control is completely blocked.

[0094] For example, based on the above, refer to Figure 2 , the process of determining the occlusion information according to the control center coordinates, control corner coordinates, and window corner coordinates can be as follows:

[0095] According to the window corner coordinates, determine the window horizontal axis starting value x1, end value x2 and vertical axis starting value y1, end value y2 of the target window as the range surrounded by the window corner coordinates, and compare the control center coordinates (x0, y0) with the window horizontal axis starting value x1, end value x2 and vertical axis starting value y1, end value y2 respectively. If the control center coordinates (x0, y0) are greater than the window horizontal axis starting value x1, less than the window horizontal axis end value x2, greater than the window vertical axis starting value y1 and less than the window vertical axis end value y2, if all are not satisfied, the control center coordinates are not in the range surrounded by the window corner coordinates, indicating that the occlusion information of the target control is not blocked, and the target control belongs to Figure 2 The control type in 1.

[0096] If it is partially satisfied, it means that the center coordinates of the control are within the range enclosed by the window corner coordinates, and the occlusion information of the target control needs to be further determined. The control horizontal axis starting point value x3 can be determined based on the upper left corner coordinates and the lower left corner coordinates in the control corner coordinates. If the control horizontal axis starting point value x3 is greater than the window horizontal axis starting point value x1, it is determined that the upper left corner coordinates and the lower left corner coordinates are within the above range, and the left side occlusion mark is added to the target control to indicate whether the target control belongs to the following example: Figure 2 Based on the above similar operations, it is possible to determine whether the coordinates of the upper right corner point and the lower right corner point are within the above range, so as to determine whether to add an obstruction mark on the right side of the target control to indicate whether the target control belongs to the following example: Figure 2 Control type 3; Based on the above similar operations, it can be determined whether the coordinates of the lower left corner point and the lower right corner point are within the above range to determine whether to add an obstruction mark for the target control to indicate whether the target control belongs to Figure 2 Based on the above similar operations, it can be determined whether the coordinates of the upper left corner point and the upper right corner point are within the above range to determine whether to add an obstruction mark for the target control to indicate whether the target control belongs to the following: Figure 2 The control type is 5.

[0097] If the occlusion mark of the target control includes at least one of the following: the upper side is occluded, the lower side is occluded, the left side is occluded, and the right side is occluded, and the number of occlusion marks is less than four, it means that the coordinates of all the corner points of the control are partially within the above range, and the occlusion information is partially occluded. If the occlusion mark of the target control includes all of the following: the upper side is occluded, the lower side is occluded, the left side is occluded, and the right side is occluded, it means that the coordinates of all the corner points of the control are within the above range, and it is determined that the occlusion information is completely occluded, indicating that the target control is Figure 2 The control type is 6.

[0098] In this way, through the control center coordinates and multiple control corner point coordinates in the control position, and the window position including multiple window corner point coordinates, it is possible to accurately and quickly determine the occlusion type of the target control, including unoccluded, partially occluded, and fully occluded, thereby quickly and accurately determining the operation position and further improving the triggering efficiency.

[0099] Optionally, based on any of the above embodiments, the above method may further include determining the target control information according to the control trigger request and the scene information of the screen.

[0100] In this embodiment, the control trigger request includes information about the control, which is used to determine whether there is a target control that the user wants to control on the screen. After receiving the control trigger request, it is necessary to obtain the scene information on the screen. Based on the scene information on the screen, the target control information related to the target control corresponding to the control trigger request can be determined. The target control information can be extracted based on at least one of the character information, text information, image information and audio information of each control in the screen scene information. The target control information can point to at least one control in the corresponding scene on the screen, which is also the target control that the control trigger request wants to trigger. Specifically, the target control information can include at least one of the control name, control identifier, control text, control position, control image and control node.

[0101] In this way, the target control information corresponding to the target control of the control trigger request can be quickly matched through the scene information of the screen, thereby improving the triggering efficiency.

[0102] Further, the scene information includes at least one of character information, text information, image information and audio information corresponding to the plurality of controls in the screen; and determining the target control information according to the control trigger request and the scene information of the screen includes:

[0103] Obtaining request information corresponding to the control trigger request through a cloud and / or local semantic algorithm module, and matching the request information with the scene information to obtain target control information corresponding to the request information;

[0104] The request information includes at least one of character information, text information, image information and audio information.

[0105] In this embodiment, the control trigger request may be at least one of a voice request, a document request, and an image request, so that the user can generate a control trigger request through a trigger operation that the user wants to perform through voice interaction, document interaction, image interaction, etc. The scene information may include at least one of character information, text information, image information, and audio information corresponding to multiple controls on the screen.

[0106] The scene information of the screen is first uploaded to the cloud and / or local semantic algorithm module. The cloud and local semantic algorithm modules can use the scene data as the data source, and then upload the control trigger request to the cloud and / or local semantic algorithm module. The cloud and / or local semantic algorithm module has higher computing resources, and it will search for the target control information that matches the control trigger request from the scene data and return it.

[0107] Specifically, when a control trigger request is received, a variety of request information can be obtained according to the control trigger request through the cloud and / or local semantic algorithm module, and a variety of request information can also be converted according to the preprocessing of the control trigger request. For example, semantic recognition can be performed on the image in the image request to obtain the text with corresponding semantics, and the document request can be divided into documents to obtain a number of text information, image information, etc., and text recognition can be performed on the voice request to obtain the corresponding text, etc., so that the request information corresponding to the control trigger request includes at least one of character information, image information and audio information. The request information is matched with the character information, image information, audio information, etc. corresponding to each control in the scene information, and the matched control in the scene information is used as the target control, and the target control information is obtained based on at least one of the character information, image information and audio information corresponding to this control.

[0108] It should be noted that attention should be paid to the timeliness of scene data transmission. The scene data on the current screen needs to be transmitted in real time, or the scene data should be re-acquired and uploaded to the cloud and / or local semantic algorithm module at regular intervals to ensure that the scene data is valid.

[0109] Optionally, when generating scene data, it is also possible to synchronously establish associations between each control and each node on the screen, or use the control node where the control is located at that moment as one of the attribute information of the control to participate in the generation of scene data, so that the scene data includes the control node of the control, and the target control information obtained will also include the control node. Through the above two methods, after obtaining the target control information based on the control trigger request, the target node can be searched on the screen through the target control information to cope with the change of control position caused by delay and the situation where the control is small, and the occlusion information corresponding to the target control information can be accurately obtained through the target node to improve the success rate of control triggering.

[0110] In this way, by processing the matching of control trigger requests and screen scene information through cloud and / or local semantic algorithm modules, more computing resources can be utilized to improve the accuracy of determining target control information.

[0111] Furthermore, before determining the target control information according to the control trigger request and the scene information of the screen, the method further includes:

[0112] For each window in the screen, the attribute information of all controls in the window is obtained through the Android accessibility service, and the scene information is integrated; wherein the attribute information includes at least one of a control name, a control identifier, a control text, a control position, a control image, and a control node.

[0113] In this embodiment, the attribute information of all controls displayed in the window is traversed one by one through the Android accessibility service. The attribute information may include but is not limited to at least one of the control identification, control text, control position, control image and control node, and then the attribute information of all controls is integrated into a structured scene data. The attribute information of each control in each window can be obtained through the Android accessibility service, and these data can also be integrated using certain rules to form structured scene data to facilitate the subsequent matching of target control information, further improving the triggering efficiency.

[0114] This embodiment also provides an electronic device, which can be integrated into a vehicle, for example Figure 3 As shown, the electronic device may include:

[0115] The transceiver module 1001 is used to receive a control trigger request, and determine the shielding information of the target control on the screen according to the target control information corresponding to the control trigger request;

[0116] The trigger module 1002 is used to trigger the target control according to the shielding information.

[0117] Optionally, the trigger module 1002 is further configured to:

[0118] Determining an operation position of the target control according to the occlusion information;

[0119] A trigger operation is performed on the target control according to the operation position.

[0120] Optionally, the trigger module 1002 is further configured to:

[0121] The control trigger request is simulated through the Android accessibility service to perform a trigger operation on the operation position.

[0122] Optionally, the occlusion information includes at least one of unoccluded, partially occluded and fully occluded, and the trigger module 1002 is further used to: determine the operation position according to the occlusion type.

[0123] Optionally, the trigger module 1002 is further configured to:

[0124] When the occlusion information indicates that the target control is not occluded, the operation position is the control center coordinates of the target control;

[0125] In a case where the occlusion information indicates that the part is occluded, determining the operation position according to an unoccluded area of ​​the target control;

[0126] When the occlusion information indicates that the entire control is occluded, the target control cannot be triggered.

[0127] Optionally, the trigger module 1002 is further configured to:

[0128] Determine the display center coordinates of the unblocked area, and set the display center coordinates as the operation position;

[0129] The unobstructed area includes at least one of the right side being unobstructed, the left side being unobstructed, the upper side being unobstructed, and the lower side being unobstructed.

[0130] Optionally, the trigger module 1002 is further configured to:

[0131] The coordinates of the corner points of the control in the unblocked area are obtained, and the coordinates of the display center are calculated according to the coordinates of the corner points of the control.

[0132] Optionally, the trigger module 1002 is further configured to:

[0133] Based on the target control information, determining node information of the target control on the screen;

[0134] Determine the occlusion information according to the node information and the window information of the target window on the screen;

[0135] The target window includes windows in the screen except the window to which the target control belongs.

[0136] Optionally, the node information includes the first level and / or control position of the window to which the target control belongs; the window information includes the second level and / or window position of the target window, and the trigger module 1002 is also used to: if the first level is greater than the second level, determine that the occlusion information of the target control is not occluded.

[0137] Optionally, the trigger module 1002 is further configured to:

[0138] If the first level is smaller than the second level, the shielding information is determined according to the control position and the window position.

[0139] Optionally, the control position includes the control center coordinates and multiple control corner point coordinates, and the window position includes multiple window corner point coordinates; the triggering module 1002 is further used to:

[0140] If the center coordinates of the control are not within the range enclosed by the coordinates of the multiple window corner points, determining that the occlusion information is not occluded;

[0141] If the coordinates of the corner points of the plurality of controls are all within the range enclosed by the coordinates of the corner points of the plurality of windows, then determining that the occlusion information is all occluded;

[0142] If the coordinates of the multiple control corner points are partially within the range surrounded by the multiple window corner point coordinates, and the center coordinates of the control are within the range surrounded by the multiple window corner point coordinates, then it is determined that the occlusion information is that the portion is occluded.

[0143] Optionally, the transceiver module 1001 is further configured to:

[0144] The target control information is determined according to the control trigger request and the scene information of the screen.

[0145] Optionally, the scene information includes at least one of character information, text information, image information and audio information corresponding to the multiple controls in the screen; the target control information is determined according to the control trigger request and the scene information of the screen, and the transceiver module 1001 is further used to:

[0146] Obtaining request information corresponding to the control trigger request through a cloud and / or local semantic algorithm module, and matching the request information with the scene information to obtain target control information corresponding to the request information;

[0147] The request information includes at least one of character information, text information, image information and audio information.

[0148] Optionally, the transceiver module 1001 is further configured to:

[0149] For each window in the screen, the attribute information of all controls in the window is obtained through the Android accessibility service, and the scene information is integrated; wherein the attribute information includes at least one of a control name, a control identifier, a control text, a control position, a control image, and a control node.

[0150] Optionally, the control trigger request includes at least one of a voice request, a document request and an image request.

[0151] Optionally, the target control information includes at least one of a control name, a control identifier, a control text, a control position, a control image, and a control node.

[0152] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0153] In this embodiment, after receiving a control trigger request, the occlusion information of the target control on the screen can be determined based on the target control information corresponding to the control trigger request, and then the target control can be triggered based on the occlusion information to avoid the inability to trigger the target control due to occlusion, thereby improving the success rate of triggering the screen control.

[0154] Accordingly, an embodiment of the present application further provides an electronic device, such as Figure 4 As shown, Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 1100 also includes a processor 1101 having one or more processing cores, a memory 1102 having one or more computer-readable storage media, and a computer program stored in the memory 1102 and executable on the processor. The processor 1101 is electrically connected to the memory 1102. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or arrange components differently.

[0155] The processor 1101 is the control center of the electronic device 1100, and uses various interfaces and lines to connect various parts of the entire electronic device 1100. By running or loading software programs and / or units stored in the memory 1102, and calling data stored in the memory 1102, the processor 1101 executes various functions of the electronic device 1100 and processes data, thereby monitoring the electronic device 1100 as a whole. The processor 1101 can be a processor (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU), a network processor (Network Processor, NP), etc., and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application.

[0156] In the embodiment of the present application, the processor 1101 in the electronic device 1100 will load instructions corresponding to the processes of one or more application programs into the memory 1102 according to the following steps, and the processor 1101 will run the application programs stored in the memory 1102 to implement various functions, such as:

[0157] receiving a control trigger request, and determining, according to target control information of a target control corresponding to the control trigger request, shielding information of the target control on the screen;

[0158] A trigger operation is performed on the target control according to the shielding information.

[0159] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0160] Optional, such as Figure 4 As shown, the electronic device 1100 further includes: a touch screen 1103, a radio frequency circuit 1104, an audio circuit 1105, an input unit 1106, and a power supply 1107. The processor 1101 is electrically connected to the touch screen 1103, the radio frequency circuit 1104, the audio circuit 1105, the input unit 1106, and the power supply 1107, respectively. Those skilled in the art can understand that Figure 4 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0161] The touch display screen 1103 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 1103 may include a display panel and a touch panel. Among them, the display panel can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device, and these graphical user interfaces can be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. The touch panel can be used to collect the touch operation of the user on or near it (such as the user using any suitable object or accessory such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 1101, and can receive the command sent by the processor 1101 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 1101 to determine the type of touch event, and then the processor 1101 provides corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present invention, the touch panel and the display panel can be integrated into the touch display screen 1103 to realize the input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 1103 can also be used as a part of the input unit 1106 to realize the input function.

[0162] The radio frequency circuit 1104 may be used to send and receive radio frequency signals, so as to establish wireless communication with network medical devices or other electronic devices through wireless communication, and to send and receive signals between network medical devices or other electronic devices.

[0163] The audio circuit 1105 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 1105 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 1105 and converted into audio data, and then the audio data is output to the processor 1101 for processing, and then sent to another electronic device through the radio frequency circuit 1104, or the audio data is output to the memory 1102 for further processing. The audio circuit 1105 may also include an earplug jack to provide communication between an external headset and an electronic device.

[0164] The input unit 1106 may be used to receive input numbers, character information or user feature information (such as fingerprint, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0165] The power supply 1107 is used to supply power to various components of the electronic device 1100. Optionally, the power supply 1107 can be logically connected to the processor 1101 through a power management device, so that the power management device can manage charging, discharging, power consumption and other functions. The power supply 1107 can also include one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0166] although Figure 4 Not shown, the electronic device 1100 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0167] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0168] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0169] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of computer programs are stored. The computer program can be loaded by a processor to execute any one of the control triggering methods provided in the embodiments of the present application. The computer program can execute the following steps of the control triggering method:

[0170] receiving a control trigger request, and determining, according to target control information of a target control corresponding to the control trigger request, shielding information of the target control on the screen;

[0171] A trigger operation is performed on the target control according to the shielding information.

[0172] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0173] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0174] Since the computer-readable storage medium can implement the beneficially stored computer program that can be implemented by any control triggering method provided in the embodiments of the present application, and can execute any control triggering method provided in the embodiments of the present application, the effect is detailed in the previous embodiments and will not be repeated here.

[0175] Optionally, an embodiment of the present application further provides a vehicle, which includes any of the above electronic devices, electronic devices, computer-readable storage media, computer program products, or executes the above-mentioned control triggering method.

[0176] In the above-mentioned control triggering method, electronic device, electronic device, vehicle, computer-readable storage medium, computer program product, etc., the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-described electronic device, vehicle, computer-readable storage medium, computer program product and its corresponding units can refer to the description of the control triggering method in the above embodiment, and will not be repeated here.

[0177] The above is a detailed introduction to a control triggering method, an electronic device, an electronic device, a vehicle, a computer-readable storage medium, and a computer program product provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A control triggering method, characterized in that: The method is applied to an electronic device, wherein the electronic device is provided with a screen, and the method comprises: receiving a control trigger request, and determining, according to target control information of a target control corresponding to the control trigger request, shielding information of the target control on the screen; A trigger operation is performed on the target control according to the shielding information.

2. The control triggering method according to claim 1, characterized in that: The triggering operation on the target control according to the shielding information includes: Determining an operation position of the target control according to the occlusion information; A trigger operation is performed on the target control according to the operation position.

3. The control triggering method according to claim 2, characterized in that: The triggering operation on the target control according to the operation position includes: The control trigger request is simulated through the Android accessibility service to perform a trigger operation on the operation position.

4. The control triggering method according to claim 2, characterized in that: The occlusion information includes at least one of unoccluded, partially occluded, and fully occluded; The step of determining the operation position of the target control according to the shielding information includes: When the occlusion information indicates that the target control is not occluded, the operation position is the control center coordinates of the target control; In a case where the occlusion information indicates that the part is occluded, determining the operation position according to an unoccluded area of ​​the target control; When the occlusion information indicates that the entire control is occluded, the target control cannot be triggered.

5. The control triggering method according to claim 4, characterized in that: The determining the operation position according to the unobstructed area of ​​the target control includes: Determine the display center coordinates of the unblocked area, and set the display center coordinates as the operation position; The unobstructed area includes at least one of the right side being unobstructed, the left side being unobstructed, the upper side being unobstructed, and the lower side being unobstructed.

6. The control triggering method according to claim 5, characterized in that: The determining the display center coordinates of the unblocked area includes: The coordinates of the corner points of the control in the unblocked area are obtained, and the coordinates of the display center are calculated according to the coordinates of the corner points of the control.

7. The control triggering method according to claim 4, characterized in that: The determining, according to target control information of a target control corresponding to the control trigger request, shielding information of the target control on the screen comprises: Based on the target control information, determining node information of the target control on the screen; Determine the occlusion information according to the node information and the window information of the target window on the screen; The target window includes windows in the screen except the window to which the target control belongs.

8. The control triggering method according to claim 7, characterized in that: The node information includes the first level and / or control position of the window to which the target control belongs; the window information includes the second level and / or window position of the target window; and determining the shielding information according to the node information and the window information of the target window on the screen includes: If the first level is greater than the second level, the occlusion information of the target control is determined to be unoccluded.

9. The control triggering method according to claim 8, characterized in that: The method further comprises: If the first level is smaller than the second level, the shielding information is determined according to the control position and the window position.

10. The control triggering method according to claim 9, characterized in that: The control position includes the control center coordinates and multiple control corner point coordinates, and the window position includes multiple window corner point coordinates; The determining the shielding information according to the control position and the window position includes: If the center coordinates of the control are not within the range enclosed by the coordinates of the multiple window corner points, determining that the occlusion information is not occluded; If the coordinates of the corner points of the plurality of controls are all within the range enclosed by the coordinates of the corner points of the plurality of windows, then determining that the occlusion information is all occluded; If the coordinates of the multiple control corner points are partially within the range surrounded by the multiple window corner point coordinates, and the center coordinates of the control are within the range surrounded by the multiple window corner point coordinates, then it is determined that the occlusion information is that the portion is occluded.

11. The control triggering method according to claim 1, characterized in that: The method further comprises: The target control information is determined according to the control trigger request and the scene information of the screen.

12. The control triggering method according to claim 11, characterized in that: The scene information includes at least one of character information, text information, image information, and audio information corresponding to the plurality of controls in the screen; and determining the target control information according to the control trigger request and the scene information of the screen includes: Obtaining request information corresponding to the control trigger request through a cloud and / or local semantic algorithm module, and matching the request information with the scene information to obtain target control information corresponding to the request information; The request information includes at least one of character information, text information, image information and audio information.

13. The control triggering method according to claim 12, characterized in that: Before determining the target control information according to the control trigger request and the scene information of the screen, the method further includes: For each window in the screen, the attribute information of all controls in the window is obtained through the Android accessibility service, and the scene information is integrated; wherein the attribute information includes at least one of a control name, a control identifier, a control text, a control position, a control image, and a control node.

14. The control triggering method according to claim 1, characterized in that: The control trigger request includes at least one of a voice request, a document request and an image request.

15. The control triggering method according to any one of claims 1 to 14, characterized in that: The target control information includes at least one of a control name, a control identifier, a control text, a control position, a control image, and a control node.

16. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the control triggering method as claimed in any one of claims 1 to 15.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the control triggering method according to any one of claims 1 to 15.

18. A vehicle, characterized in that: The electronic device comprises the electronic device as claimed in claim 16, or executes the control triggering method as claimed in any one of claims 1 to 15.

Images