Display method and related equipment

Abstract

The invention provides a display method and related equipment. A user does not need to move a position and click an application, a tool or a menu. Particularly, when the method is applied to large-screen display equipment, the equipment using efficiency of a user is greatly improved, and the user experience can also be improved. The method comprises the following steps: detecting the position of a target object; based on the position of the target object, an associated display area corresponding to the target object is determined, and the associated display area is used for displaying a range of a first menu; and displaying the first menu in an associated display area corresponding to the target object.

Description

Technical Field

[0001] This application relates to the field of electronic technologies, and in particular, to a display method and related devices. Background Art

[0002] As the demand for large-screen display devices by users increases, for example, display devices with sizes of 75, 85, 98, 110, and 135 inches are used by more and more users. In a meeting scenario, during the use of a large-screen display device, as Figure 1 shown, the user may need to walk to a certain position to click on an application, tool, or menu displayed on the screen. Summary of the Invention

[0003] This application provides a display method and related devices, which can enable the user to click on an application, tool, or menu without moving the position. Especially when applied to a large-screen display device, it greatly improves the efficiency of the user using the device and can also enhance the user experience.

[0004] In a first aspect, an embodiment of this application provides a display method, which can be applied to a display device; the method includes:

[0005] Detect the position of a target object;

[0006] Based on the position of the target object, determine an associated display area corresponding to the target object, where the associated display area is a range for displaying a first menu;

[0007] Display the first menu within the associated display area corresponding to the target object.

[0008] In a possible implementation manner, in the display method provided by the embodiment of this application, the target object is a controlled window; based on the position of the target object, determining the associated display area corresponding to the target object includes:

[0009] Based on the position of the controlled window, determine the associated display area corresponding to the target object.

[0010] In a possible implementation manner, in the display method provided by the embodiment of this application, the determining the associated display area of the target object based on the position of the controlled window includes:

[0011] Based on the position of the target boundary of the controlled window, determine the associated display area of the target object.

[0012] In a possible implementation manner, in the display method provided by the embodiment of this application, the area size of the associated display area is pre-configured.

[0013] In a possible implementation manner, in the display method provided by the embodiments of the present application, the target boundary is a preset boundary of the controlled window; or,

[0014] The target boundary is a boundary that meets a preset condition among all the boundaries of the controlled window, where the preset condition is any one of the following conditions:

[0015] Among the distances between each boundary and the ground, the distance between the target boundary and the ground is the smallest;

[0016] Among the distances between each boundary and the position where the most recent user operation occurred, the distance between the target boundary and the position where the most recent user operation occurred is the smallest;

[0017] Among all the boundaries, there is no display object in the target range on the side of the target boundary away from the controlled window.

[0018] In a possible implementation manner, in the display method provided by the embodiments of the present application, determining the associated display area of the target object based on the position of the controlled window includes:

[0019] Determining the associated display area of the target object based on the positions of the vertices of the controlled window.

[0020] In a possible implementation manner, in the display method provided by the embodiments of the present application, the target object is a user; the position of the target object is determined based on an image including the user collected by an image acquisition device.

[0021] In a possible implementation manner, in the display method provided by the embodiments of the present application, the method further includes:

[0022] In response to detecting an update in the position of the target object, updating the associated display area corresponding to the target object based on the updated position of the target object, and displaying the first menu in the updated associated display area.

[0023] In a possible implementation manner, in the display method provided by the embodiments of the present application, the first menu includes at least one fixed function control, where the function of the fixed function control is pre-configured.

[0024] In a possible implementation manner, in the display method provided by the embodiments of the present application, the first menu includes at least one first control.

[0025] In a possible implementation manner, in the display method provided by the embodiments of the present application, the at least one first control is determined based on the display content corresponding to the target object.

[0026] In a possible implementation manner, in the display method provided by the embodiments of the present application, when different target objects are detected, the first controls included in the first menu are different.

[0027] In a possible implementation manner, in the display method provided by the embodiments of the present application, the target object is a controlled window, and the display content corresponding to the target object refers to the content in the controlled window.

[0028] In a possible implementation manner, in the display method provided by the embodiments of the present application, the target object is a user, and the display content corresponding to the target object refers to the content in the area used by the user.

[0029] In a possible implementation manner, in the display method provided by the embodiments of the present application, displaying the first menu corresponding to the target object in the associated display area corresponding to the target object includes:

[0030] Displaying a second menu in the associated display area corresponding to the target object, the second menu including at least one second control;

[0031] In response to an operation on a target second control, displaying the first menu in the associated display area corresponding to the target object.

[0032] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes one or more of the following:

[0033] The start control of an application, the window of an application;

[0034] One of the first controls corresponds to any one of the following:

[0035] The function of a start control of an application, a partial function of an application.

[0036] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes the start control of a target application; the at least one first control includes a target first control, and the target first control corresponds to the function of the start control of the target application;

[0037] The method further includes:

[0038] In response to an operation on the target first control, generating a simulated click event for the start control of the target application.

[0039] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes the window of the target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application;

[0040] In response to an operation on the target first control, a partial function of the target application is triggered.

[0041] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes the window of the target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application; a partial function of the target application corresponds to a simulated click position;

[0042] The triggering of the partial function of the target application in response to an operation on the target first control includes:

[0043] In response to an operation on the target first control, in the window of the target application, a simulated click event is generated based on the simulated click position to trigger the partial function of the target application.

[0044] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes the window of the target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application;

[0045] The triggering of the partial function of the target application in response to an operation on the target first control includes:

[0046] In response to an operation on the target first control, in the window of the target application, a plurality of simulated click events are generated in sequence to trigger the partial function of the target application.

[0047] In a possible implementation manner, in the display method provided by the embodiments of the present application, a partial function of the target application corresponds to a plurality of simulated click positions with a preset arrangement order;

[0048] The generating of a plurality of simulated click events in sequence in the window of the target application includes:

[0049] In the window of the target application, a plurality of simulated click events are generated according to the plurality of simulated click positions with a preset arrangement order.

[0050] In a possible implementation manner, in the display method provided by the embodiments of the present application, a part of the target application is a target function; the plurality of simulated click positions with a preset arrangement order corresponding to the target function are determined in the following manner:

[0051] In response to an operation on the target application, take a screenshot of the display interface;

[0052] Use a pre-trained recognition model to recognize the controls in the screenshot, and determine the positions of the recognized plurality of third controls as the plurality of simulated click positions with a preset arrangement order, where the plurality of third controls are used to trigger the target function in the target application.

[0053] In a possible implementation manner, in the display method provided by the embodiments of the present application, some functions of the application are determined based on the usage conditions of the functions in the application;

[0054] The usage conditions of the functions include any one of the following:

[0055] The usage frequencies of the functions in the current device, the usage frequencies of the functions provided by the application provider, and the usage frequencies of the functions in the application by the users using the current device.

[0056] In a possible implementation manner, in the display method provided by the embodiments of the present application, the at least one first control is determined based on the target user intention.

[0057] In a possible implementation manner, in the display method provided by the embodiments of the present application, the method further includes:

[0058] Based on the display content corresponding to the target object and the target user intention, determine the first control in the first menu.

[0059] In a possible implementation manner, in the display method provided by the embodiments of the present application, the method further includes:

[0060] Based on the display content corresponding to the target object, determine the target scenario;

[0061] Based on the target scenario and the target operation of the user on the display device, determine the target user intention.

[0062] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes one or more of the following:

[0063] The start control of the application, the window of the application;

[0064] The method further includes:

[0065] Match each function of each application in the display content corresponding to the target object with the target user intention, and the matching result includes the function of the start control of at least one application and / or part of the functions of at least one application;

[0066] Determine the first control in the first menu based on the functions in the matching result.

[0067] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object is provided by a screen mirroring sending end;

[0068] The method further includes:

[0069] Determine a first coordinate mapping relationship based on the first coordinate information and the second coordinate information, where the first coordinate information is the coordinate information of the display content when the screen mirroring sending end displays the display content at a target resolution, and the second coordinate information is the coordinate information of the display content when the display device displays the display content at the target resolution;

[0070] Determine a second coordinate mapping relationship based on the third coordinate information of the display content when the display device currently displays the display content and the second coordinate information.

[0071] In a possible implementation manner, in the display method provided by the embodiments of the present application, the at least one first control includes a target first control, and the target first control corresponds to the function of the start control of a target application;

[0072] The method further includes:

[0073] In response to an operation on the target first control, obtain the first coordinate of the start control of the target first application in the display interface of the display device, and based on the first coordinate mapping relationship and the second coordinate mapping relationship, determine the second coordinate corresponding to the first coordinate, and send the second coordinate to the screen mirroring sending end so that the screen mirroring sending end generates a simulated click event corresponding to the second coordinate.

[0074] In a possible implementation manner, in the display method provided by the embodiments of the present application, the at least one first control includes a target first control, and the target first control corresponds to part of the functions of a target application; part of the functions of the target first control correspond to multiple simulated click positions in the display interface of the display device, and the order corresponding to each simulated click position among the multiple simulated click positions is pre-configured;

[0075] The method further includes:

[0076] In response to an operation on the target first control, based on the first coordinate mapping relationship and the second coordinate mapping relationship, determine the third coordinates corresponding to each simulated click position, and provide the third coordinates corresponding to each simulated click position to the screen mirroring sending end and indicate the order corresponding to each simulated click position, so that the screen mirroring sending end generates simulated click events for the third coordinates corresponding to each simulated click position in sequence according to the order.

[0077] In a possible implementation manner, in the display method provided by the embodiments of the present application, the number of the detected target objects is multiple;

[0078] In the associated display area corresponding to the target object, display the first menu corresponding to the target object, including:

[0079] For any detected target object, in the associated display area corresponding to the any target object, display the first menu corresponding to the any target object.

[0080] In a possible implementation manner, in the display method provided by the embodiments of the present application, the method further includes:

[0081] Based on the position of the any target object, determine the controlled area corresponding to the any target object, and the first menu corresponding to the any target object is generated based on the content in the controlled area corresponding to the any target object.

[0082] In a second aspect, the present application provides a display method, which can be applied to a display device. The display method may include:

[0083] Detect the position of the target object;

[0084] Based on the position of the target object, determine the display content corresponding to the target object;

[0085] Based on the display content corresponding to the target object, determine the first menu corresponding to the target object;

[0086] Display the first menu.

[0087] In a possible implementation manner, in the display method provided by the embodiments of the present application, the target object is a controlled window. The position of the target object is the position of the controlled window. When the controlled window is switched, the switched controlled window is used as the target object.

[0088] In a possible implementation manner, in the display method provided by the embodiments of the present application, the target object is a user; the position of the target object is determined based on an image including the user collected by an image acquisition device.

[0089] In a possible implementation manner, in the display method provided by the embodiments of the present application, the method further includes:

[0090] In response to detecting that the position of the target object is updated, based on the updated position of the target object, update the display content corresponding to the target object, and update the first menu, and display the updated first menu.

[0091] In a possible implementation manner, in the display method provided by the embodiments of the present application, the first menu includes at least one fixed function control, wherein the function of the fixed function control is pre-configured.

[0092] In a possible implementation manner, in the display method provided by the embodiments of the present application, the first menu includes at least one first control.

[0093] In a possible implementation manner, in the display method provided by the embodiments of the present application, the at least one first control is determined based on the display content corresponding to the target object.

[0094] In a possible implementation manner, in the display method provided by the embodiments of the present application, when different target objects are detected, the first controls included in the first menu are different.

[0095] In a possible implementation manner, in the display method provided by the embodiments of the present application, the target object is a controlled window, and the display content corresponding to the target object refers to the content in the controlled window.

[0096] In a possible implementation manner, in the display method provided by the embodiments of the present application, the target object is a user, and the display content corresponding to the target object refers to the content in the area used by the user.

[0097] In a possible implementation manner, in the display method provided by the embodiments of the present application, the method further includes:

[0098] Display a second menu, where the second menu includes at least one second control;

[0099] In response to an operation on a target second control, display the first menu.

[0100] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes one or more of the following:

[0101] The start control of the application program, the window of the application program;

[0102] One of the first controls corresponds to any one of the following:

[0103] The function of a start control of the application, and a partial function of the application.

[0104] In a possible implementation, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes a start control of a target application; the at least one first control includes a target first control, and the target first control corresponds to the function of the start control of the target application;

[0105] The method further includes:

[0106] In response to an operation on the target first control, generate a simulated click event for the start control of the target application.

[0107] In a possible implementation, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes a window of a target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application;

[0108] In response to an operation on the target first control, trigger the partial function of the target application.

[0109] In a possible implementation, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes a window of a target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application; the partial function of the target application corresponds to a simulated click position;

[0110] The step of, in response to an operation on the target first control, triggering the partial function of the target application includes:

[0111] In response to an operation on the target first control, in the window of the target application, generate a simulated click event based on the simulated click position to trigger the partial function of the target application.

[0112] In a possible implementation, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes a window of a target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application;

[0113] The step of, in response to an operation on the target first control, triggering the partial function of the target application includes:

[0114] In response to an operation on the target first control, in the window of the target application, generate a plurality of simulated click events in sequence to trigger the partial function of the target application.

[0115] In a possible implementation, in the display method provided by the embodiments of the present application, for some functions of the target application, there are multiple simulated click positions with a preset arrangement order;

[0116] In the window of the target application, generating multiple simulated click events in sequence includes:

[0117] In the window of the target application, generating multiple simulated click events according to the multiple simulated click positions with a preset arrangement order.

[0118] In a possible implementation, in the display method provided by the embodiments of the present application, a part of the target application is a target function; the multiple simulated click positions with a preset arrangement order corresponding to the target function are determined in the following manner:

[0119] In response to an operation on the target application, taking a screenshot of the display interface;

[0120] Using a pre-trained recognition model to recognize the controls in the screenshot, and determining the positions of the recognized multiple third controls as the multiple simulated click positions with a preset arrangement order, where the multiple third controls are used to trigger the target function in the target application.

[0121] In a possible implementation, in the display method provided by the embodiments of the present application, some functions of the application are determined based on the usage conditions of each function in the application;

[0122] The usage conditions of each function include any one of the following:

[0123] The usage frequency of each function in the current device, the usage frequency of each function provided by the application vendor, and the usage frequency of each function in the application by the user using the current device.

[0124] In a possible implementation, in the display method provided by the embodiments of the present application, the at least one first control is determined based on the target user's intention.

[0125] In a possible implementation, in the display method provided by the embodiments of the present application, the method further includes:

[0126] Based on the display content corresponding to the target object and the target user's intention, determining the first control in the first menu.

[0127] In a possible implementation, in the display method provided by the embodiments of the present application, the method further includes:

[0128] Determine a target scenario based on the display content corresponding to the target object;

[0129] Determine the target user intention based on the target scenario and the target operation of the user on the display device.

[0130] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes one or more of the following:

[0131] A start control of an application program, a window of an application program;

[0132] The method further includes:

[0133] Match each function of each application program in the display content corresponding to the target object with the target user intention, and the matching result includes the function of the start control of at least one application program and / or part of the functions of at least one application program;

[0134] Determine the first control in the first menu based on the functions in the matching result.

[0135] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object is provided by a screen mirroring sending end;

[0136] The method further includes:

[0137] Determine a first coordinate mapping relationship based on first coordinate information and second coordinate information, where the first coordinate information is the coordinate information of the display content when the screen mirroring sending end displays the display content at a target resolution, and the second coordinate information is the coordinate information of the display content when the display device displays the display content at the target resolution;

[0138] Determine a second coordinate mapping relationship based on third coordinate information of the display content when the display device currently displays the display content and the second coordinate information.

[0139] In a possible implementation manner, in the display method provided by the embodiments of the present application, the at least one first control includes a target first control, and the target first control corresponds to the function of the start control of a target application program;

[0140] The method further includes:

[0141] In response to an operation on the target first control, obtain the first coordinates of the start control of the target first application in the display interface of the display device, and based on the first coordinate mapping relationship and the second coordinate mapping relationship, determine the second coordinates corresponding to the first coordinates, and send the second coordinates to the screen mirroring sending end, so that the screen mirroring sending end generates a simulated click event corresponding to the second coordinates.

[0142] In a possible implementation manner, in the display method provided by the embodiments of the present application, the at least one first control includes a target first control, and the target first control corresponds to a part of the functions of the target application; a part of the functions of the target first control corresponds to multiple simulated click positions in the display interface of the display device, and the order corresponding to each simulated click position among the multiple simulated click positions is pre-configured;

[0143] The method further includes:

[0144] In response to an operation on the target first control, based on the first coordinate mapping relationship and the second coordinate mapping relationship, determine the third coordinates corresponding to each simulated click position, and provide the third coordinates corresponding to each simulated click position to the screen mirroring sending end and indicate the order corresponding to each simulated click position, so that the screen mirroring sending end sequentially generates simulated click events of the third coordinates corresponding to each simulated click position according to the order

[0145] In a possible implementation manner, in the display method provided by the embodiments of the present application, the number of the detected target objects is multiple;

[0146] For any detected target object, based on the position corresponding to the any target object, display the first menu corresponding to the any target object.

[0147] In a possible implementation manner, based on the position corresponding to the any target object, determine the associated display area corresponding to the target object, and the first menu corresponding to the target object is displayed in the associated display area corresponding to the target object.

[0148] In a possible implementation manner, in the display method provided by the embodiments of the present application, the method further includes:

[0149] Based on the position of the any target object, determine the controlled area corresponding to the any target object, and the first menu corresponding to the any target object is generated based on the content in the controlled area corresponding to the any target object.

[0150] In a third aspect, the present application further provides a display device, including a display module and a processing module;

[0151] The display module is used to display content;

[0152] The processing module is used to execute the display method provided in the first aspect and any one of its embodiments.

[0153] In a possible implementation, the display device may include a display module and a processing module;

[0154] The display module is used to display content;

[0155] The processing module is used to:

[0156] Detect the position of the target object;

[0157] Based on the position of the target object, determine the associated display area corresponding to the target object, where the associated display area is the range for displaying the first menu;

[0158] Cause the display module to display the first menu within the associated display area corresponding to the target object.

[0159] In a possible implementation, in the display device provided in the embodiments of the present application, the target object is a controlled window; the processing module determines the associated display area corresponding to the target object based on the position of the target object, specifically:

[0160] Based on the position of the controlled window, determine the associated display area corresponding to the target object.

[0161] In a possible implementation, in the display device provided in the embodiments of the present application, the processing module determines the associated display area of the target object based on the position of the controlled window, specifically:

[0162] Based on the position of the target boundary of the controlled window, determine the associated display area of the target object.

[0163] In a possible implementation, in the display device provided in the embodiments of the present application, the area size of the associated display area is pre-configured.

[0164] In a possible implementation, in the display device provided in the embodiments of the present application,

[0165] The target boundary is a preset boundary of the controlled window; or,

[0166] The target boundary is a boundary that meets a preset condition among all the boundaries of the controlled window, where the preset condition is any one of the following conditions:

[0167] Among the distances between each boundary and the ground, the distance between the target boundary and the ground is the smallest;

[0168] Among the distances between each boundary and the location where the most recent user operation occurred, the distance between the target boundary and the location where the most recent user operation occurred is the smallest;

[0169] Among all the boundaries, there is no display object within the target range on the side of the target boundary away from the controlled window.

[0170] In a possible implementation manner, in the display device provided by the embodiments of the present application, determining the associated display area of the target object based on the position of the controlled window includes:

[0171] Determining the associated display area of the target object based on the positions of the vertices of the controlled window.

[0172] In a possible implementation manner, in the display device provided by the embodiments of the present application, the target object is a user; the position of the target object is determined based on an image containing the user collected by an image acquisition device.

[0173] In a possible implementation manner, in the display device provided by the embodiments of the present application, the processing module is further configured to:

[0174] In response to detecting an update in the position of the target object, update the associated display area corresponding to the target object based on the updated position of the target object, and display the first menu within the updated associated display area.

[0175] In a possible implementation manner, in the display device provided by the embodiments of the present application, the first menu includes at least one fixed function control, where the function of the fixed function control is pre-configured.

[0176] In a possible implementation manner, in the display device provided by the embodiments of the present application, the first menu includes at least one first control.

[0177] In a possible implementation manner, in the display device provided by the embodiments of the present application, the at least one first control is determined based on the display content corresponding to the target object.

[0178] In a possible implementation manner, in the display device provided by the embodiments of the present application, when different target objects are detected, the first controls included in the first menu are different.

[0179] In a possible implementation manner, in the display device provided by the embodiments of the present application, the target object is a controlled window, and the display content corresponding to the target object refers to the content in the controlled window.

[0180] In a possible implementation manner, in the display device provided by the embodiments of the present application, the target object is a user, and the display content corresponding to the target object refers to the content in the area used by the user.

[0181] In a possible implementation manner, in the display device provided by the embodiments of the present application, the processing module displays a first menu corresponding to the target object in the associated display area corresponding to the target object, specifically for:

[0182] Display a second menu in the associated display area corresponding to the target object, where the second menu includes at least one second control;

[0183] In response to an operation on a target second control, display the first menu in the associated display area corresponding to the target object.

[0184] In a possible implementation manner, in the display device provided by the embodiments of the present application, the display content corresponding to the target object includes one or more of the following:

[0185] A start control of an application, a window of the application;

[0186] The first control corresponds to any one of the following:

[0187] The function of a start control of an application, a partial function of an application.

[0188] In a possible implementation manner, in the display device provided by the embodiments of the present application, the partial function of the application is triggered based on a control in the application.

[0189] In a possible implementation manner, in the display device provided by the embodiments of the present application, the display content corresponding to the target object includes a start control of a target application; the at least one first control includes a target first control, and the target first control corresponds to the function of the start control of the target application;

[0190] The processing module is further configured to:

[0191] In response to an operation on the target first control, call an interface to start the target application.

[0192] In a possible implementation manner, in the display device provided by the embodiments of the present application, the display content corresponding to the target object includes a start control of a target application; the at least one first control includes a target first control, and the target first control corresponds to the function of the start control of the target application;

[0193] The processing module is further configured to:

[0194] In response to an operation on a target first control, a simulated click event for the start control of the target application is generated.

[0195] In a possible implementation manner, in the display device provided by the embodiments of the present application, the display content corresponding to the target object includes a window of the target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application;

[0196] The processing module is further configured to:

[0197] In response to an operation on a target first control, a partial function of the target application is triggered.

[0198] In a possible implementation manner, in the display device provided by the embodiments of the present application, the display content corresponding to the target object includes a window of the target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application; the partial function of the target application corresponds to a simulated click position.

[0199] When the processing module responds to an operation on a target first control and triggers a partial function of the target application, specifically:

[0200] In response to an operation on the target first control, in the window of the target application, a simulated click event is generated based on the simulated click position to trigger the partial function of the target application.

[0201] In a possible implementation manner, in the display device provided by the embodiments of the present application, the display content corresponding to the target object includes a window of the target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application;

[0202] When the processing module responds to an operation on a target first control and triggers a partial function of the target application, specifically:

[0203] In response to an operation on the target first control, in the window of the target application, a plurality of simulated click events are generated in sequence to trigger the partial function of the target application.

[0204] In a possible implementation manner, in the display device provided by the embodiments of the present application, the partial function of the target application corresponds to a plurality of simulated click positions with a preset arrangement order;

[0205] When the processing module generates a plurality of simulated click events in sequence in the window of the target application, specifically:

[0206] In the window of the target application, generate a plurality of simulated click events according to the plurality of simulated click positions with a preset arrangement order.

[0207] In a possible implementation manner, in the display device provided by the embodiments of the present application, a part of the target application is a target function; the plurality of simulated click positions with a preset arrangement order corresponding to the target function are determined in the following manner:

[0208] In response to an operation on the target application, capture a screenshot of the display interface;

[0209] Use a pre-trained recognition model to recognize the controls in the screenshot, and determine the positions of the recognized plurality of third controls as the plurality of simulated click positions with a preset arrangement order, where the plurality of third controls are used to trigger the target function in the target application.

[0210] In a possible implementation manner, in the display device provided by the embodiments of the present application, a part of the functions of the application is determined based on the usage conditions of the functions in the application;

[0211] The usage conditions of the functions include any one of the following:

[0212] The usage frequencies of the functions in the current device, the usage frequencies of the functions provided by the application provider, and the usage frequencies of the functions in the application used by the users using the current device.

[0213] In a possible implementation manner, in the display device provided by the embodiments of the present application, the at least one first control is determined based on the target user intention.

[0214] In a possible implementation manner, in the display device provided by the embodiments of the present application, the processing module is further configured to:

[0215] Determine the first control in the first menu based on the display content corresponding to the target object and the target user intention.

[0216] In a possible implementation manner, in the display device provided by the embodiments of the present application, the processing module is further configured to:

[0217] Determine a target scenario based on the display content corresponding to the target object;

[0218] Determine the target user intention based on the target scenario and the target operation of the user on the display device.

[0219] In a possible implementation manner, in the display device provided by the embodiments of the present application, the display content corresponding to the target object includes one or more of the following:

[0220] Startup controls of the application, window of the application;

[0221] The processing module is further configured to:

[0222] Match each function of each application in the display content corresponding to the target object with the target user intention, and the matching result includes the functions of the startup controls of at least one application, and / or, partial functions of at least one application;

[0223] Determine the first control in the first menu based on the functions in the matching result.

[0224] In a possible implementation manner, in the display device provided by the embodiments of the present application, the display content corresponding to the target object is provided by the screen mirroring sending end;

[0225] The processing module is further configured to:

[0226] Determine a first coordinate mapping relationship based on the first coordinate information and the second coordinate information, where the first coordinate information is the coordinate information of the display content when the screen mirroring sending end displays the display content at the target resolution, and the second coordinate information is the coordinate information of the display content when the display device displays the display content at the target resolution;

[0227] Determine a second coordinate mapping relationship based on the third coordinate information of the display content when the display device currently displays the display content and the second coordinate information.

[0228] In a possible implementation manner, in the display device provided by the embodiments of the present application, the at least one first control includes a target first control, and the target first control corresponds to the function of the startup control of the target application;

[0229] The processing module is further configured to:

[0230] In response to an operation on the target first control, obtain the first coordinate of the startup control of the target first application in the display interface of the display device, and determine the second coordinate corresponding to the first coordinate based on the first coordinate mapping relationship and the second coordinate mapping relationship, and send the second coordinate to the screen mirroring sending end so that the screen mirroring sending end generates a simulated click event corresponding to the second coordinate.

[0231] In a possible implementation manner, in the display device provided by the embodiments of the present application, the at least one first control includes a target first control, and the target first control corresponds to a part of the functions of a target application; a part of the functions of the target first control corresponds to multiple simulated click positions in the display interface of the display device, and the order corresponding to each simulated click position among the multiple simulated click positions is pre-configured;

[0232] The processing module is further configured to:

[0233] In response to an operation on the target first control, based on the first coordinate mapping relationship and the second coordinate mapping relationship, determine the third coordinates corresponding to each simulated click position, and provide the third coordinates corresponding to each simulated click position to the screen mirroring sending end and indicate the order corresponding to each simulated click position, so that the screen mirroring sending end generates simulated click events for the third coordinates corresponding to each simulated click position in sequence according to the order.

[0234] In a possible implementation manner, in the display device provided by the embodiments of the present application, the number of the target objects is detected to be multiple;

[0235] Display the first menu within the associated display area corresponding to the target object, including:

[0236] For any detected target object, display the first menu corresponding to the any detected target object within the associated display area corresponding to the any detected target object.

[0237] In a possible implementation manner, in the display device provided by the embodiments of the present application, the processing module is further configured to:

[0238] Based on the position of the any detected target object, determine the controlled area corresponding to the any detected target object, and the first menu corresponding to the any detected target object is generated based on the content in the controlled area corresponding to the any detected target object.

[0239] In a fourth aspect, the present application further provides a display device, including a display module and a processing module;

[0240] The display module is configured to display content;

[0241] The processing module is configured to execute the display method provided in the second aspect and any one of its implementation manners.

[0242] In a fifth aspect, the present application further provides a display system, and the system includes an electronic device and a display device;

[0243] The electronic device provides screen mirroring content to the display device;

[0244] The display device is used to execute the display method provided in the first aspect and any of its implementation manners.

[0245] In a sixth aspect, the present application further provides a display system, which includes an electronic device and a display device;

[0246] The electronic device provides screen mirroring content to the display device;

[0247] The display device is used to execute the display method provided in the second aspect and any of its implementation manners.

[0248] In a seventh aspect, the present application further provides a computer-readable storage medium, which includes a computer program. When the computer program runs on an electronic device, the computer program is used to cause the electronic device to execute the steps in the display method provided in the first aspect and any of its implementation manners.

[0249] In an eighth aspect, the present application further provides a computer-readable storage medium, which includes a computer program. When the computer program runs on an electronic device, the computer program is used to cause the electronic device to execute the steps in the display method provided in the second aspect and any of its implementation manners.

[0250] In a ninth aspect, there is also provided a computer program product, which includes a computer program. When the computer program runs on a computer, the computer is caused to execute the display method provided in the first aspect and any of its implementation manners.

[0251] In a tenth aspect, there is also provided a computer program product, which includes a computer program. When the computer program runs on a computer, the computer is caused to execute the display method provided in the second aspect and any of its implementation manners.

[0252] In an eleventh aspect, an embodiment of the present application further provides a chip, which is coupled to a memory in an electronic device and is used to call a computer program stored in the memory and execute the display method provided in the first aspect and any of its implementation manners.

[0253] In a twelfth aspect, an embodiment of the present application further provides a chip, which is coupled to a memory in an electronic device and is used to call a computer program stored in the memory and execute the display method provided in the second aspect and any of its implementation manners. Description of the Drawings

[0254] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0255] Figure 1 Schematic diagram of an application scenario provided by an embodiment of the present application;

[0256] Figure 2a Schematic flowchart of a display method provided by an embodiment of the present application;

[0257] Figure 2b Schematic diagram of the display interface of a display device provided by an embodiment of the present application;

[0258] Figure 2c Schematic diagram of the display interface of a display device provided by an embodiment of the present application;

[0259] Figure 2d Schematic diagram of the display interface of a display device provided by an embodiment of the present application;

[0260] Figure 3a Schematic diagram of the boundary of a window;

[0261] Figure 3b Schematic diagram of the boundary of a window and the position of the most recent user operation;

[0262] Figure 3c Schematic diagram of the target range corresponding to each boundary of a window;

[0263] Figure 3d Schematic diagram of the vertex of a window;

[0264] Figure 4 Schematic diagram of an application scenario provided by an embodiment of the present application;

[0265] Figure 5 Schematic diagram of the display interface of a display device provided by an embodiment of the present application;

[0266] Figure 6 Schematic flowchart of a display method provided by an embodiment of the present application;

[0267] Figure 7a-7c Schematic diagrams of the display interfaces of a display device provided by embodiments of the present application respectively;

[0268] Figure 7d 、 Figure 7e Schematic diagrams of the partitions of a display device provided by embodiments of the present application respectively;

[0269] Figure 7f Schematic diagram of the user location and the area used by the user provided by the embodiment of the present application;

[0270] Figure 8a-8e Schematic diagrams of the display interfaces of the display device provided by the embodiments of the present application respectively;

[0271] Figure 8f Schematic diagram of the relationship between option controls in the multi-level menu of the target application;

[0272] Figure 8g Schematic diagram of the relationship between option controls in the multi-level menu;

[0273] Figure 8h Schematic diagram of the display interface of the display device provided by the embodiment of the present application;

[0274] Figure 8i Schematic diagram of the triggering sequence relationship of the option control Ya in the third application;

[0275] Figure 9 Schematic diagram of the display interface of the display device provided by the embodiment of the present application;

[0276] Figure 10a Schematic diagram of the display interface of the display device provided by the embodiment of the present application in the screen mirroring scenario;

[0277] Figure 10b Schematic diagram of the display interface of the display device provided by the embodiment of the present application in the screen mirroring scenario;

[0278] Figure 11a Schematic diagram of the display interface of the display device provided by the embodiment of the present application;

[0279] Figure 11b Schematic diagram of the display interface of the display device provided by the embodiment of the present application;

[0280] Figure 12a-12f Schematic flowcharts of the display method provided by an embodiment of the present application respectively;

[0281] Figure 13 Schematic diagram of the structure of the display device provided by the embodiment of the present application;

[0282] Figure 14 Schematic diagram of the structure of the display device provided by the embodiment of the present application. Detailed implementation manners

[0283] To make the objectives, technical solutions and advantages of the present application more clearly understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the scope of protection of the present application. Without conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0284] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application may represent at least two, for example, it may be two, three or more, and the embodiments of the present application do not make limitations. "A and / or B" in the present application may represent three situations: A, B, and A and B.

[0285] Figure 2a A display method provided by an embodiment of the present application is shown, which can be applied to a display device. The display method may include the following steps:

[0286] S201, detecting the position of a target object.

[0287] S202, based on the position of the target object, determining an associated display area corresponding to the target object, where the associated display area is a range for displaying a first menu.

[0288] S203, displaying the first menu within the associated display area corresponding to the target object.

[0289] In the present application, please combine Figure 2b , and determine the associated display area corresponding to the target object through the position of the target object. The purpose of the associated display area is to determine the first menu in the associated display area. The user can operate the first menu in the associated display area without moving the position, which improves the efficiency of the user using the display device and the user experience.

[0290] In some application scenarios, multiple users can use the display device simultaneously, and the number of target objects detected by the display device can be multiple. In this case, the display device can determine the associated display area corresponding to each target object. The first menus corresponding to each target object may not be exactly the same. That A is not exactly the same as B may mean that there is a part of the content in A that is the same as a part of the content in B, and there is a part of the content in A that is different from a part of the content in B. The display device displays the first menu corresponding to the target object in the associated display area corresponding to each target object.

[0291] In a possible application scenario, the target object can be implemented as a controlled window. In the display interface of the display device, the window of any application and the window of any component can be displayed. The user can operate on any window. In this application, the application program can include objects of types such as applications and components.

[0292] The window being used by the user can be used as the controlled window. Alternatively, the window that the user last operated on can be used as the controlled window. Optionally, the display device can include a touch screen. The display device can determine the controlled window based on the user's touch position and the positions of the windows. For example, if there are multiple windows in the display interface and only one window covers the touch position, then this window is used as the controlled window. Another example is that if there are multiple windows in the display interface and two windows respectively cover the touch position, then the window on the top layer among these two windows is used as the controlled window.

[0293] Optionally, the user can interact with the display device through a control device. The user can send a first signal to the screen of the display device using the control device. The screen of the display device can detect the first signal and determine the user operation position based on the position of the detected first signal. The display device can determine the controlled window based on the user operation position and the positions of the windows. For example, if there are multiple windows in the display interface and only one window covers the user operation position, then this window is used as the controlled window. Another example is that if there are multiple windows in the display interface and two windows respectively cover the user operation position, then the window on the top layer among these two windows is used as the controlled window.

[0294] In a possible implementation manner, in the above step S202, when the display device determines the associated area corresponding to the target object based on the position of the target object, the display device can determine the associated display area corresponding to the target object based on the position of the controlled window. Please combine Figure 2c , the display device can determine the associated display area corresponding to the target object in combination with the position situation of the controlled window. The size of the associated display area can be pre-configured. The size of the associated display area can be implemented as the area size of the associated display area, and / or the contour size of the associated display area.

[0295] For example, a region approximately 20 px below the lower left vertex of the overall position of the controlled window is determined as the associated display region. Those skilled in the art can determine the specific position and size of the associated display region according to actual application needs. The display device may adopt, but is not limited to, the process of determining the associated display region of the target object provided in the following examples.

[0296] The display device can determine the associated display region of the target object based on the position of the target boundary of the controlled window.

[0297] In a possible design, the target object is the controlled window. When the controlled window is updated, the position of the controlled window also follows the update, that is, the position of the target object also follows the update. Thus, the associated display region of the target object follows the update, realizing the dynamic change of the display position of the first menu.

[0298] The display device detecting the position of the target object may include, but is not limited to, detecting the position of the latest controlled window. Suppose Figure 2c at the first moment t1, the controlled window is the window of application 1. The display device determines the associated display region of the target object according to the position of the window of application 1, and the display device displays the first menu in this associated display region.

[0299] The controlled window can be switched. When the user uses other windows, the switching of the controlled window will be triggered. Please refer to Figure 2c and Figure 2d , suppose at the second moment t2 (t2 is a moment after t1), the display device detects that the latest controlled window is the window of application 2. The display device updates the associated display region of the target object according to the window position of application 2, and the display device displays the first menu in this associated display region. It can be seen that in this design, the display position of the first menu changes following the position of the controlled window.

[0300] In some examples, please refer to Figure 3a , the target boundary may be a preset boundary of the controlled window. Generally, the contour of a window is generally rectangular. For example, the controlled window includes an upper boundary, a lower boundary, a left boundary, and a right boundary. The preset boundary may be the upper boundary, or the lower boundary, or the left boundary, or the right boundary. Figure 3a Illustrates the case where the preset boundary is the lower boundary.

[0301] In other examples, the target boundary may be a boundary that meets a preset condition among all the boundaries of the controlled window, where the preset condition is any one of the following conditions:

[0302] Condition 1: Among the distances between each boundary and the ground, the distance between the target boundary and the ground is the smallest;

[0303] Condition 2: Among the distances between each boundary and the position where the most recent user operation occurred, the distance between the target boundary and the position where the most recent user operation occurred is the smallest;

[0304] Condition 3: Among all the boundaries, there is no display object within the target range on the side of the target boundary away from the controlled window.

[0305] In this embodiment, the contour shape of the controlled window is not specifically limited. The contour of the controlled window may be composed of multiple boundaries. The position of a boundary may be implemented as the position of the preset end point of the boundary, or may be implemented as the center point of the boundary, or as the position of a point at a preset distance from the preset end point of the boundary.

[0306] In a possible design, the preset condition is Condition 1. For example, please refer to Figure 3a , the contour of the controlled window may be rectangular. Among all the boundaries of the controlled window, the distance between the lower boundary and the ground is the smallest, and the display device may use the lower boundary as the target boundary.

[0307] In a possible design, the preset condition is Condition 2. Please refer to Figure 3b , the display device may determine the distances between each boundary of the controlled window and the position UP where the most recent user operation occurred. Among them, the distance between boundary S2 and the position UP where the most recent user operation occurred is the smallest. The display device may determine boundary S2 as the target boundary.

[0308] In a possible design, the preset condition is Condition 3. The display device may determine whether there is a display object within the target range on the side of each boundary of the controlled window away from the controlled window. Please refer to Figure 3c , taking boundary S1 as an example, on the side of boundary S1 away from the controlled window, that is, above boundary S1. The target range S1-1 above boundary S1 may be implemented as within a distance of d pixels above boundary S1. Or rather, the target range S1-1 is above boundary S1, and the distance between each point in the target range S1-1 and boundary S1 is less than or equal to d pixels. Figure 3c The target ranges in Figure 3c are indicated by black dashed lines. The boundary S2 and the target range S2-2 on the side of boundary S2 away from the controlled window are shown in

[0309] For example, there is no display object in the target range S4-4. That is to say, there is no window that blocks other applications, no window that blocks other components, and no control is blocked in the target range S4-4. The display device may determine the boundary S4 as the target boundary.

[0310] For example, there is no display object in either the target range S4-4 or the target range S3-3. The display device may determine either the boundary S3 or the boundary S4 as the target boundary.

[0311] In some examples, when the display device determines the associated display area of the target object based on the position of the controlled window, the display device may determine the associated display area of the target object based on the position of the vertex of the controlled window.

[0312] Please combine Figure 3d , usually the outline of the window is rectangular. A rectangle has 4 vertices, which are respectively denoted as the upper left vertex, the lower left vertex, the upper right vertex, and the lower right vertex.

[0313] The display device may determine the associated display area of the target object based on the position of any vertex of the controlled window. The outline of the associated display area may also be configured as a rectangle. Optionally, the position of one vertex of the outline of the associated display area may be the same as the position of any vertex of the controlled window, and the associated display area does not block the controlled window.

[0314] Optionally, the display device may determine the associated display area of the target object based on the position of the preset vertex of the controlled window. For example, the preset vertex may be the upper left vertex, or the lower left vertex, or the upper right vertex, or the lower right vertex. Please combine Figure 3d again. For the convenience of introduction, a plane coordinate system is established with the lower left vertex of the display interface as the origin. Taking the lower left vertex D1 of the controlled window as the preset vertex as an example, the abscissa of each point in the associated display area of the target object is less than or equal to the abscissa of the lower left vertex D1 of the controlled window, and the ordinate of each point in the associated display area of the target object is less than or equal to the ordinate of the lower left vertex D1 of the controlled window.

[0315] For another example, when the preset vertex is the lower right vertex D2 of the controlled window, the abscissa of each point in the associated display area of the target object is greater than or equal to the abscissa of the lower right vertex D2 of the controlled window, and the ordinate of each point in the associated display area of the target object is less than or equal to the ordinate of the lower right vertex D2 of the controlled window.

[0316] For another example, taking the upper left vertex D3 of the controlled window as the preset vertex, the abscissa of each point in the associated display area of the target object is less than or equal to the abscissa of the upper left vertex D3 of the controlled window, and the ordinate of each point in the associated display area of the target object is greater than or equal to the ordinate of the upper left vertex D3 of the controlled window.

[0317] For another example, when the preset vertex is the upper right vertex D4 of the controlled window, the abscissa of each point in the associated display area of the target object is greater than or equal to the abscissa of the upper right vertex D4 of the controlled window, and the ordinate of each point in the associated display area of the target object is greater than or equal to the ordinate of the upper right vertex D4 of the controlled window.

[0318] In a possible application scenario, the target object is a user. The position of the target object is determined based on an image containing the user collected by an image acquisition device. Or rather, the display device can determine the position of the user based on the image containing the user. The user can also be referred to as a pedestrian.

[0319] The target object can be a user interacting with the display device. Optionally, the pedestrian collected by the image acquisition device can be used as the user interacting with the display device.

[0320] The display device can have an image acquisition function. The display device can include an image acquisition device, such as a camera and other acquisition devices. The number of acquisition devices can be one or more. The display device can have a pre-configured pedestrian detection model. The display device can pre-store a preset mapping relationship. Please refer to Figure 4 , when the user uses the display device, the distance between the user and the display device is usually recorded as df centimeters. Generally speaking, the value of df can be configured according to the actual application scenario. The preset mapping relationship is generally determined based on df. The preset mapping relationship is used to determine the position and height of the user in the actual space according to the position and height of the user in the image.

[0321] The display device can perform pedestrian detection based on the image collected by the image acquisition device. The display device can determine the actual height of the pedestrian and the relative position between the pedestrian and the display device based on the relative position and height of the detected pedestrian in the image and the preset mapping relationship.

[0322] The embodiments of the present application do not specifically limit how the pedestrian detection model detects pedestrians. It should be understood that any detection algorithm that can implement a detection model for detecting whether an image includes a pedestrian (or a person, or a user) can be used as the pedestrian detection model in the present application.

[0323] In some examples, the pedestrian detection model can be a detection model based on Convolutional Neural Networks (CNN) technology. In some examples, the pedestrian detection model can perform detection based on the global feature method. The pedestrian detection model can use various static features of the image, such as edge features, shape features, statistical features, or transform features, to describe pedestrians, such as Haar wavelet features, HOG features, Edgelet features, Shapelet features, and contour template features. In some examples, the pedestrian detection model can perform detection based on the human body part method. For example, the human body is divided into several components, and then each component is detected separately. Finally, the detection results are integrated according to certain constraint relationships to finally determine whether there are pedestrians. In some examples, the pedestrian detection model can perform detection based on the stereo vision method. For example, by analyzing the three-dimensional information of the target in the images collected by at least two acquisition devices, pedestrians can be identified.

[0324] The display device can determine the touchable area of the user based on the user's position and height in the actual space. For example, the display device can consider all the positions that the user's hand can touch as positions in the touchable area.

[0325] Generally speaking, the display device can be pre-configured with the calculation relationship between the touchable area and the user's position and height in the actual space. The display device can determine the touchable area based on this calculation relationship, considering the user's position and height in the actual space, and use it as the associated display area corresponding to the target object, that is, the associated display area corresponding to the user. Such a design allows the user to perform control operations through the first menu in the associated display area without having to move.

[0326] Based on the display method provided in any of the above embodiments, after step S203, the display device can further perform the following process: in response to detecting an update in the position of the target object, update the associated display area corresponding to the target object based on the updated position of the target object, and display the first menu within the updated associated display area. Such a design can make the position of the first menu dynamically change. The user's position or the user's operation will affect the position of the target object. By updating the associated display area of the target object, the display position of the first menu is dynamically changed, enabling the user to control the display device through the first menu without having to move.

[0327] The display device can detect the position of the target object in real time, periodically, or dynamically. For example, Figure 5As shown, taking the target object as a user as an example, if the position of the target object changes, then based on the current position of the target object, the associated display area corresponding to the target object is dynamically determined, and the first menu corresponding to the target object is displayed within the updated associated display area.

[0328] Based on the display method provided in any of the above embodiments, the first menu corresponding to the target object may include one or more first controls, and any first control may be configured with a function. The user can trigger the function of the control through the first control in the first menu.

[0329] In a possible design, in the operation of step S203, the display device can directly display the first menu corresponding to the target object within the associated display area corresponding to the target object.

[0330] In another possible design, in the operation of the foregoing step S203, please combine Figure 6 , the display device can perform the following process:

[0331] S601, within the associated display area corresponding to the target object, display a second menu, where the second menu includes at least one second control.

[0332] S602, in response to an operation on the target second control, display the first menu within the associated display area corresponding to the target object.

[0333] Please combine Figure 7a , the display device can display the second menu 701 within the associated display area 700 corresponding to the target object. The second menu 701 may include one or more second controls 702. The functions of each second control 702 may be pre-configured. Optionally, at least one second control may include, but is not limited to: a return control, a return to the system main interface control (also referred to as a home control), and a target second control.

[0334] The function of the return control can be to cancel the user's most recent operation. The function of the return to the system desktop control can be to directly return to the system desktop interface. The function of the target second control is to display the first menu.

[0335] Optionally, as Figure 7a shown, the display form of the second menu can be an expanded form. Or, as Figure 7b shown, the display form of the second menu can be a collapsed form, and the second menu may include an expand control 703. After the user triggers the expand control 703, the second menu can be triggered to expand, displaying all the second controls in the second menu. Or, as Figure 7cAs shown, the display form of the second menu can be a semi-folded form. In the semi-folded form, a part of the second controls and the expansion control 703 are displayed. The other part of the second controls is hidden. The user can trigger the displayed second controls. After the user triggers the expansion control 703, all the second controls in the second menu are displayed.

[0336] Based on the display method provided in any of the foregoing embodiments, the first menu may include at least one fixed function control. Among them, the at least one fixed function control may include, but is not limited to: a return control, a return to the system main interface control (which can also be referred to as a home control). In such a design, the user can trigger fixed functions through the fixed function controls in the first menu. The fixed function controls in the first menu do not change and are not affected by changes in the position of the target object, changes in the display content of the target object, changes in the user's intention, etc.

[0337] Based on the display method provided in any of the foregoing embodiments, the display device can determine at least one first control in the first menu based on the actual usage of the display device by the user. Optionally, the controls in the first menu may include at least one first control and may also include the foregoing at least one fixed function control.

[0338] In a possible design, at least one first control in the first menu is determined based on the display content corresponding to the target object. Optionally, when the display content corresponding to the target object changes, the first menu is updated based on the changed display content corresponding to the target object.

[0339] In some examples, the target object is the controlled window, and the display content corresponding to the target object refers to the content in the controlled window.

[0340] The window being used by the user can be used as the controlled window. Or, the window of the user's most recent operation is used as the controlled window. Optionally, the display device may include a touch screen. The display device can determine the controlled window based on the user's touch position and the positions of each window. For example, if there are multiple windows in the display interface and only one window covers the touch position, then this window is used as the controlled window. Another example is that if there are multiple windows in the display interface and two windows respectively cover the touch position, then the window located on the top layer among these two windows is used as the controlled window.

[0341] Optionally, the user can interact with the display device through a control device. The user can send a first signal to the screen of the display device using the control device. The screen of the display device can detect the first signal and determine the user operation position based on the position of the detected first signal. The display device can determine the controlled window based on the user operation position and the positions of the windows. For example, if the display interface includes multiple windows and only one window covers the user operation position, then this window is taken as the controlled window. Another example is that if the display interface includes multiple windows and two windows respectively cover the user operation position, then the window located on the topmost layer among these two windows is taken as the controlled window.

[0342] In some examples, the target object is the user, and the display content corresponding to the target object can refer to the content in the area used by the user. The area used by the user can include, but is not limited to, the window used by the user, or the screen partition used by the user, or the area pre-set by the user.

[0343] In some application scenarios, the display device can support a split-screen mode. The display device can divide the display screen into multiple sub-screens, and one sub-screen can be used as a screen partition.

[0344] Optionally, multiple screen partitions can be arranged horizontally. Please refer to Figure 7d , the number of multiple screen partitions is 3, and they are arranged in sequence horizontally. Or, the number of multiple screen partitions can be 2, 4, etc., and can be flexibly configured according to actual application requirements. In one possible design, the display device can detect the user's position and determine the screen partition closest to the user as the screen partition used by the user. In another possible design, the display device can determine the screen partition where the position of the most recent user operation is located as the screen partition used by the user.

[0345] Optionally, multiple screen partitions can be arranged in an array. Please refer to Figure 7e , the number of multiple screen partitions can be 4, namely the upper left partition, the upper right partition, the lower left partition, and the lower right partition. Or, the number of multiple screen partitions can be 6, 8, etc., and can be flexibly configured according to actual application requirements. In one possible design, the display device can detect the user's position and determine the screen partition closest to the user as the screen partition used by the user. In another possible design, the display device can determine the screen partition where the position of the most recent user operation is located as the screen partition used by the user.

[0346] In some examples, the area used by the user can be a preset-sized area allocated by the display device based on the user's position. The size of the area allocated by the display device is preset. The specific position of the area allocated to the user is determined based on the user's position. Please refer to Figure 7f, the display device can determine the area location corresponding to the current user location according to the pre-set mapping relationship between the area location and the user location. Optionally, after the user moves and the user location changes, the display device can also determine the location of the allocated area according to the latest user location.

[0347] Optionally, when a user uses the display device, the display content corresponding to the target object can be implemented as the content in the display interface of the display device.

[0348] Optionally, the display device can support multiple users. The display device can support multiple windows for multiple users, or support a split-screen mode. In the case where the display device detects that the number of target objects is multiple, the display content corresponding to one target object can refer to the content in the usage area corresponding to one user. For example, the display content corresponding to the first user refers to the content in the usage area of the first user, and the display content corresponding to the second user refers to the content in the usage area of the second user.

[0349] In a possible design, when the target objects detected by the display device are different, the first controls included in the first menu are different. For example, when the target object is a controlled window and the display device detects multiple target objects. Each controlled window has a corresponding first menu. The first menu corresponding to the controlled window is determined based on the display content in the controlled window. Since the display content in each controlled window is different, it can be seen that the first controls included in the first menus corresponding to each controlled window are different. Optionally, the fixed function controls included in the first menus corresponding to each controlled window are the same.

[0350] In a possible design, when the target objects detected by the display device are different, the first controls included in the first menu may be the same, or not completely the same.

[0351] For example, when the target object is a user and the display device detects multiple target objects, each user has a corresponding first menu. The first menu corresponding to the user is determined based on the content in the user's usage area. Since the usage areas of each user can be partially overlapped, the first controls included in the first menus corresponding to the users may be completely the same, or partially the same. Optionally, the fixed function controls included in the first menus corresponding to each user are the same.

[0352] In the display method provided in any one of the above embodiments, the display content corresponding to the target object includes one or more of the following:

[0353] The start control of the application program, the window of the application program;

[0354] One of the first controls corresponds to any one of the following:

[0355] The function of a startup control of the application, and a partial function of the application.

[0356] The display content corresponding to the target object includes at least one startup control of a first application, and / or at least one window of a second application.

[0357] A partial function of an application can be implemented as any function among all the functions of the application. Generally, during the use of an application, all the functions of the application can include a first type of function and a second type of function. For the functions in the first type of function, each function can be triggered by a single user operation in the window of the application. For the functions in the second type of function, each function is triggered by multiple user operations in the window of the application.

[0358] In a possible design, a partial function of an application can be triggered based on a control in the application. For an application, the application has multiple controls. There are some controls that are controls in the first-level menu. The controls in the first-level menu can refer to the controls that are default displayed in the window of the application. The aforementioned first type of function can be triggered by the controls in the first-level menu.

[0359] Among the multiple controls of the application, there can also be some controls that are not in the first-level menu. For example, the controls in the second-level menu, and the functions of the controls in the second-level menu depend on the controls in the first-level menu. It is necessary to trigger the display of the second-level menu through a user operation on the controls in the first-level menu, and then the functions of the controls in the second-level menu can be triggered through user operations on the controls in the displayed second-level menu. The aforementioned second type of function can be triggered by the controls that are not in the first-level menu.

[0360] In some examples, the display content corresponding to the target object can include at least one startup control of a first application. Please combine Figure 8a , the display content corresponding to the target object can include the startup control of application P1, the startup control of application P2, and the startup control of application P3. In this application, the function of a startup control of an application is to start the application. The user can start the application by triggering the operation of the startup control of the application. Optionally, the startup control can also be called a shortcut, an icon, or a launcher, etc.

[0361] Among at least one first control in the first menu, the function of a first control can be the same as the function of a startup control of a first application, or the function of a first control can be the same as a partial function among all the functions of a first application.

[0362] In some examples, the display content corresponding to the target object may include windows of at least one second application. Please combine with Figure 8b , the display content corresponding to the target object may include the window of application T1. Please combine with Figure 8c , the display content corresponding to the target object may include the window of application T2 and the window of application T1. The window of application T1 is on the top layer, and a part of the window of application T2 may be blocked by the window of application T2.

[0363] Among at least one first control in the first menu, the function of one first control may be the same as the function of the start control of a second application, or the function of one first control may be the same as a part of the functions of all the functions of a second application.

[0364] In some examples, the display content corresponding to the target object may include the start controls of at least one first application and the windows of at least one second application. For example, please combine with Figure 8d , the display content corresponding to the target object may include the start control of application P1, the start control of application P2, and the window of application T1.

[0365] Among at least one first control in the first menu, the function of one first control may be the same as the function of the start control of a first application, or the function of one first control may be the same as a part of the functions of all the functions of a first application, or the function of one first control may be the same as a part of the functions of all the functions of a second application, or the function of one first control may be the same as the function of the start control of a second application.

[0366] Please combine with Figure 8d and Figure 8e , the display content corresponding to the target object may include the start control of application P1, the start control of application P2, and the window of application T1. The first menu may include first controls D1, D2, D3, and D4. Application P1 may be a system settings component, and application T1 may be an application with a handwriting function. First control D1 may correspond to the function of the start control of application P1. The function of first control D2 may correspond to the network settings function in application P1. First control D3 may correspond to the eraser function in application T1, and first control D4 may correspond to the brush adjustment function in application T1.

[0367] In a possible implementation, there may be a first control in the first menu corresponding to the function of the start control of the first application. For ease of introduction, the user can perform a selection operation on the first control in the first menu, and the selected first control is denoted as the target first control.

[0368] The display content corresponding to the target object includes the start control of the target application; the at least one first control includes the target control. In some examples, the function that the target first control corresponds to the start control of the target application can be implemented as the target first control corresponding to the interface (such as API) for starting the target application. The display device can also respond to the operation on the target first control and call the interface for starting the target application, so as to achieve the control effect of opening the target application.

[0369] In some examples, the display device can pre-store at least one simulated click position corresponding to the first control. The display device can generate a corresponding simulated click event based on the simulated click position corresponding to the first control to achieve triggering the function corresponding to the first control. The function that the target first control corresponds to the start control of the target application can be implemented as the simulated click position corresponding to the target first control being the position of the start control of the target application.

[0370] The display device can respond to the operation on the target first control and generate a simulated click event for the simulated click position corresponding to the target first control, that is, generate a simulated click event for the start control of the target application, so as to achieve triggering the start control of the target application and starting the target application.

[0371] In a possible implementation, there may be a first control in the first menu corresponding to a partial function of an application, which can be implemented as the interface of a function in all functions of the application corresponding to the first control. For ease of introduction, the user can perform a selection operation on the first control in the first menu, and the selected first control is denoted as the target first control.

[0372] In some examples, the display content corresponding to the target object includes the window of the target application; the function that the target first control corresponds to the partial function of the target application can be implemented as the target function interface in the full-level function interface of the target application corresponding to the target first control. Optionally, the target application can be the aforementioned first application or the aforementioned second application, and the embodiments of the present application do not make specific limitations in this regard.

[0373] The display device can respond to the operation on the target first control and call the target function interface of the target application, so as to achieve triggering the target function of the target application.

[0374] In some examples, the display device may pre-store at least one simulated click position corresponding to the first control. The display device may generate a corresponding simulated click event based on the simulated click position corresponding to the first control to trigger the function corresponding to the first control.

[0375] Please refer to Figure 8f , after the target application is launched, the window of the target application may include a primary menu, which can be independently displayed at the top of the window or in the sidebar. The primary menu may include multiple option controls. A certain option control in the primary menu may trigger the display of a secondary menu, and the secondary menu may include multiple option controls. Similarly, according to actual needs, a certain option control in the secondary menu may trigger the display of a tertiary menu, and the tertiary menu may also include multiple option controls. It can be seen that if it is necessary to trigger the function of an option control in the secondary menu, it depends on triggering the display of the secondary menu through the option control in the primary menu.

[0376] Similarly, please refer to Figure 8g , the function of the option control Ax in the X-level menu depends on the triggering of the option control Ax-1 in the X-1 level menu, depends on the triggering of the option control Ax-2 in the X-2 level menu,..., and depends on the triggering of the option control A1 in the 1 level menu. It can be seen that in accordance with the preset hierarchical triggering order, sequentially triggering the option control A1 in the 1 level menu,..., the option control Ax-2 in the X-2 level menu, and the option control Ax-1 in the X-1 level menu can trigger the display of the X-level menu, and the function of the option control Ax can be triggered through the displayed X-level menu.

[0377] In the embodiments of the present application, the partial function of the target application corresponding to the target first control is denoted as the target function of the target application. The target function may be the function of any option control in any level menu of the target application.

[0378] In some examples, the display content corresponding to the target object includes the window of the target application; the target first control corresponds to a partial function of the target application. The at least one first control includes the target first control, and the target first control corresponds to the partial function of the target application; the partial function of the target application corresponds to a simulated click position.

[0379] When the display device responds to an operation on the target first control and triggers the partial function of the target application, it may, in response to the operation on the target first control, generate a simulated click event based on the simulated click position in the window of the target application to trigger the partial function of the target application.

[0380] In a possible scenario, the target first control corresponds to the target function of the target application, which is the function of the target option control in the 1-level menu of the target application. The simulated click position corresponding to the target first control can be implemented as the position of the target option control in the 1-level menu of the target application. The 1-level menu of the target application, that is, the first-level menu of the target application. Usually, the first-level menu of the target application can be independently displayed at the top or side of the window.

[0381] The display device can, in response to an operation on the target first control, generate a simulated click event at the simulated click position corresponding to the target first control in the window of the target application, that is, generate a simulated click event at the position of the target option control in the 1-level menu of the target application, which can trigger the target control in the 1-level menu of the target application and implement the function of calling the target control in the 1-level menu of the target application.

[0382] In some examples, the display content corresponding to the target object includes the window of the target application; the at least one first control includes the target first control, and the target first control corresponds to a part of the functions of the target application.

[0383] When the display device triggers a part of the functions of the target application in response to an operation on the target first control, it can, in response to the operation on the target first control, generate a plurality of simulated click events in sequence in the window of the target application to trigger the part of the functions of the target application.

[0384] In a possible design, the part of the functions of the target application corresponds to a plurality of simulated click positions with a preset arrangement order; when the display device generates a plurality of simulated click events in sequence in the window of the target application, it can generate a plurality of simulated click events in the window of the target application according to the plurality of simulated click positions with the preset arrangement order.

[0385] The fact that the target first control corresponds to a part of the functions of the target application can be recorded as the target function of the target application. The target function corresponds to a plurality of simulated click positions, and the order corresponding to each simulated click position among the plurality of simulated click positions is pre-configured. The display device can, in response to an operation on the target first control, then generate simulated click events corresponding to each simulated click position in sequence in the window of the target application.

[0386] In this embodiment, the target function of the target application corresponds to multiple simulated click positions. The target function can be the function of any option control in any level of menu other than the first-level menu in the target application. Assume that the target function is the function of the target option control in the X-level menu of the target application, where X is a positive integer greater than 1. The target function of the target application corresponds to multiple simulated click positions, including X simulated click positions. The X simulated click positions include the simulated click position of the target option control in the X-level menu, as well as the positions of the option controls in the (X - 1)-level menu, (X - 2)-level menu,..., and 1-level menu on which the target option control depends.

[0387] As an example, the order of the X simulated click positions can be implemented as the preset hierarchical trigger order corresponding to the target function. The display device can respond to the operation on the target first control, and based on the first simulated click position among the X simulated click positions (assumed to be the position of option control A1 in the 1-level menu), generate the first simulated click event, which can achieve triggering the display of the 2-level menu by clicking option control A1 in the 1-level menu. Then, the display device generates the second simulated click event based on the second simulated click position (assumed to be the position of option control A2 in the 2-level menu), which can achieve triggering the display of the 3-level menu by clicking option control A2 in the 2-level menu. Similarly, the display device generates the i-th simulated click event based on the i-th simulated click position (assumed to be the position of option control Ai in the i-level menu), which can achieve triggering the display of the (i + 1)-level menu by clicking option control Ai in the i-level menu. By executing according to such a rule, until the X-th simulated click event is generated based on the X-th simulated click position (assumed to be the position of option control Ax in the X-level menu), the function of option control Ax can be achieved by clicking option control Ax in the X-level menu.

[0388] In a possible design, a part of the target application is the target function; the multiple simulated click positions corresponding to the target function and having a preset arrangement order are determined in the following manner:

[0389] Respond to the operation on the target application and take a screenshot of the display interface;

[0390] Use a pre-trained recognition model to recognize the controls in the screenshot, and determine the positions of the recognized multiple third controls as the multiple simulated click positions having a preset arrangement order. The multiple third controls are used to trigger the target function in the target application.

[0391] The display device can pre-store at least one simulated click position corresponding to the first control. The display device can generate a corresponding simulated click event based on the simulated click position corresponding to the first control to trigger the function corresponding to the first control. Among them, in some examples, at least one simulated click position corresponding to any first control in the first menu is pre-configured. In other examples, at least one simulated click position corresponding to any first control in the first menu is pre-determined by the display device.

[0392] The following introduces the method for the display device to determine at least one simulated click position corresponding to the first control. For any application program (denoted as the third application program), during the user's use of the third application program, the display device can capture a screenshot of the display interface. The display device can capture a screenshot of the display interface in response to the user's operation on the third application program. For example, each time the user operates on the third application program, the display device will capture a screenshot of the display interface after the operation, so as to obtain the display interface screenshot corresponding to each operation. Please refer to Figure 8h , the display device includes a pre-trained recognition model. The display device can use the pre-trained recognition model to recognize the controls in each screenshot. Through the recognition of the screenshots, the positions of the option controls in each level of the menu of the third application program and the functions of the option controls in each level of the menu can be determined. Generally, the option controls in each level of the menu are configured with description information, and the description information can indicate the function of the option control. The description information is generally in text form. For example, the description information of the "File" option control is "File". Another example is that the description information of the "Brush" option control is "Brush".

[0393] The display device can obtain the trigger sequence relationship among all the option controls in the third application program through the positions of the option controls in each level of the menu and the functions of the option controls. As Figure 8i shown, the option control Ya in the 3-level menu depends on the option control Yb in the 2-level menu being triggered and the option control Yc in the 1-level menu being triggered. The display device can obtain the trigger sequence including the option control Ya in the 3-level menu, that is, the option control Yc in the 1-level menu, the option control Yb in the 2-level menu, and the option control Ya in the 3-level menu. The display device can obtain the trigger sequence including the option control Yb in the 2-level menu, that is, the option control Yc in the 1-level menu and the option control Yb in the 2-level menu.

[0394] Thus, the display device can obtain the triggering sequence relationship among the option controls in the third application except for those in the first-level menu. For the function Ta of the option control Ya in the third-level menu, the display device can determine multiple simulated click positions corresponding to the function Ta based on the triggering sequence relationship including the option control Ya in the third-level menu. Specifically, assume that the first triggering sequence relationship in any triggering sequence relationship including the option control Ya in the third-level menu is used. In the first triggering sequence relationship, the option control Ya in the third-level menu is the option control at the m-th serial number in the first triggering sequence relationship. Then, the multiple simulated click positions corresponding to the function Ta are determined as the positions of the first m option controls in the first triggering sequence relationship.

[0395] Similarly, for the multiple simulated click positions corresponding to the foregoing target function, the display device can capture a screenshot of the display interface in response to an operation on the target application. Using a pre-trained recognition model, identify the controls in the screenshot, and determine the positions of the identified third controls and the functions of the third controls. The display device can determine the triggering sequence relationship among multiple third controls in the target application, and the number of the triggering sequence relationships is one or more. Among them, in the target triggering sequence relationship including the third control corresponding to the target function, the third control at the i-th serial number is the third control corresponding to the target function. The display device can use the positions of the first i serial numbers of the third controls in the target triggering sequence relationship as the multiple simulated click positions corresponding to the target function, where i is an integer greater than 1.

[0396] Optionally, in the above embodiments, the display device can perform a simulated touch event through the InputManager based on the simulated click position corresponding to the target first control, that is, generate a simulated click event.

[0397] Based on any of the foregoing embodiments, the first menu may include at least one first control. The at least one first control is determined based on the display content corresponding to the target object. The display content corresponding to the target object includes one or more of the following: the start control of the application, the window of the application. One of the first controls corresponds to any one of the following: the function of the start control of one application, a partial function of one application.

[0398] Among them, part of the functions of the application program are determined based on the usage of each function in the application program; the usage of each function includes any one of the following: the usage frequency of each function in the current device, the usage frequency of each function provided by the application program provider, and the usage frequency of each function in the application program by the user using the current device. Optionally, part of the functions of the application program can also be determined based on the user's configuration. When a user uses a certain application program, the function of the first control in the first menu can correspond to the specified function of the certain application program configured by the user.

[0399] In some application scenarios, the display device can pre-store the usage of each function of any application program. For example, the display device can store it in a file and classify it according to the application package name of the application program in json format.

[0400] In some examples, the display content corresponding to the target object includes the window of the fourth application program. Some or all of the first controls in the first menu generated by the display device can correspond to some functions of the fourth application program. For example, some of the first controls in the first menu correspond to some functions of the fourth application program, and the functions of the fourth application program corresponding to each of the first controls in some of the first controls are determined based on the usage frequency of each function in the fourth application program in the current device. If the number of some of the first controls is s, the s first controls can respectively correspond to the s functions with the highest usage frequency of each function in the fourth application program in the current device.

[0401] Based on any of the foregoing embodiments, the first menu may include at least one first control. The function of the first control in the first menu can be configured by the user. The function of one first control can be configured as the function of the start control of any application program. Or, the function of one first control can be configured as part of the functions of any application program. Optionally, the display device can set the corresponding relationship between the first control and the interface for starting the corresponding application program, or set the corresponding relationship between the first control and the function interface of the corresponding application program according to the user's configuration of the first control.

[0402] Based on any of the foregoing embodiments, the first menu may include at least one first control. One of the first controls corresponds to any one of the following: the function of the start control of an application program, and part of the functions of an application program. In some application scenarios, the first control corresponds to the function of the start control of an application program, and the presentation style of the first control can be determined based on the presentation style of the start control of the application program. For example, the display device can take a screenshot of the presentation style of the start control of the application program to obtain a first image, and after scaling the first image according to a preset size to obtain a second image, the display device can configure the second image as the presentation style of the first control.

[0403] Similarly, in some application scenarios, the first control corresponds to a partial function of an application. For example, the first control corresponds to the function of an option control of an application. The presentation style of the first control can be determined based on the presentation style of the option control of the application. For example, the display device can take a screenshot of the option style of the startup control of the application to obtain a third image, and after scaling the third image according to a preset size setting, obtain a fourth image. The display device can configure the fourth image as the presentation style of the first control.

[0404] Based on any of the foregoing embodiments, the first menu may include at least one first control. The at least one first control is determined based on the display content corresponding to the target object. In some application scenarios, the number of first controls in the first menu can be pre-configured. If the number of first controls determined by the display device based on the display content corresponding to the target object is greater than the pre-configured number of first controls, the display device can sort the determined first controls based on a preset elimination strategy and delete the first controls with a lower ranking. For example, the display device can use the least recently used (LRU) algorithm as the preset elimination strategy. LRU is a memory data elimination strategy, and it is commonly used to eliminate the least recently used data when the memory is insufficient.

[0405] In another possible design, the display device can determine at least one first control in the first menu based on the actual usage of the display device by the user. The at least one first control is determined based on the target user's intention.

[0406] In some examples, the display device can be configured with a multimodal large model, or the display device can have the ability to call a multimodal large model. A multimodal large model generally refers to a model that can process multiple types of data, such as text, audio and video, pictures and other types of data. The multimodal large model can fuse multiple types of information, execute more complex scenarios and tasks, and make more accurate judgments about the current scenario and subsequent intentions and other operations. The embodiments of the present application do not specifically limit the generation method or training method of the multimodal large model. Those skilled in the art should understand that the effect of using the multimodal large model to execute tasks is usually related to the performance of the multimodal large model, and a certain performance of the multimodal large model can be improved through existing technical means.

[0407] The display device can use the multimodal large model to determine the target user's intention. The display device can input one or more types of data into the multimodal large model so that the multimodal large model outputs the target user's intention. The data input into the multimodal large model may include, but is not limited to, the following types of data:

[0408] Screenshot data of the display interface of the display device;

[0409] Data such as writing trajectory, speed, and pressure on the touch screen of the display device;

[0410] Information of the applications running on the display device;

[0411] Speech text of the audio collected by the audio acquisition module of the display device;

[0412] Gaze point information of the viewers watching the display device.

[0413] In some examples, the display device can infer the function that the user needs to use according to the target user intention. The display device can determine the inferred function as the function of at least one first control. Optionally, the display device can determine the target user intention in real time, dynamically update the inferred function, and dynamically update the first controls in the first menu. Such a design can reduce the operation complexity when the user selects a first control from the first menu.

[0414] In a possible design, the display device can also determine the target scene based on the display content corresponding to the target object. Optionally, the display device can determine the target scene according to the situation of the window of the application included in the display content corresponding to the target object. For example, if the display content corresponding to the target object includes the window of a conference application, the target scene can be determined as a conference scene. Another example is that if the display content corresponding to the target object includes the window of a handwriting application, the target scene can be determined as a speech scene.

[0415] The display device can determine the target user intention based on the target scene and the target operation of the user on the display device. The target operation of the user on the display device can include, but is not limited to, operations such as forming a trajectory on the touch screen of the display device and applying pressure to the touch screen. Based on the operation of forming a trajectory, speed data can be obtained. The display device can determine the target user intention based on the target operation of the user on the display device and in combination with the target scene.

[0416] In some examples, the display device can also determine the first controls in the first menu based on the display content corresponding to the target object and the target user intention.

[0417] In a possible design, the display device may determine one or more alternative controls based on the display content corresponding to the target object. The display content corresponding to the target object includes one or more of the following: the start control of the first application, the window of the second application. The function of an alternative control may correspond to a partial function of the first application, or the function of the start control of the first application, or a partial function of the second application. The display device may match the target user intention with the functions of each alternative control, and determine the alternative control that matches the target user intention as the first control in the first menu.

[0418] For example, please combine Figure 9 , the target user intention may be to adjust the writing handwriting. The display content corresponding to the target object includes the window of the writing application. The function of the alternative control determined by the display device may be the function of the option control in each level of menu in the writing application. The display device may match the target user intention of "adjusting the writing handwriting" with the functions of the option controls in each level of menu in the writing application. Functions related to adjusting the writing handwriting, such as the function of erasing handwriting, the function of adjusting the brush color, the function of adjusting the brush thickness, the function of adjusting the brush type, etc. The display device may configure the matched function as the function of the first control in the first menu. In such a design, the user can trigger the function of erasing handwriting, or the function of adjusting the brush color, or the function of adjusting the brush thickness, or the function of adjusting the brush type through the first control in the first menu.

[0419] In a possible design, the display content corresponding to the target object includes the window of the second application, and the number of the second applications is one or more. The display device may determine the window that the user is operating based on the display content corresponding to the target object, that is, the controlled window. The controlled window is the window of the fifth application. The display device matches each function of the fifth application with the target user intention. The display device may configure the matched function as the function of the first control in the first menu.

[0420] In a possible design, the display content corresponding to the target object includes one or more of the following: the start control of the application, the window of the application.

[0421] The display device may also match each function of each application in the display content corresponding to the target object with the target user intention, and the matching result includes the function of the start control of at least one application, and / or, a partial function of at least one application.

[0422] The display device may determine the first control in the first menu based on the functions in the matching result. The display device may configure the functions in the matching result as the functions of the first control in the first menu.

[0423] Based on the display method provided in any of the above embodiments, this display method can be applied in a screen mirroring scenario, and the display device can act as a screen mirroring receiver. The display content corresponding to the target object in the foregoing embodiments is provided by the screen mirroring sender.

[0424] Please combine Figure 10a , and the display device can determine the first control in the first menu based on the display content corresponding to the target object, that is, the display device determines the first control in the first menu based on the screen mirroring content provided by the screen mirroring sender. The display device can display the first menu on the display interface of the display device. It should be clear that the first menu is generated by the display device and is displayed on the display device. There is no first menu at the screen mirroring sender.

[0425] The display device can feedback the operations of the user on the display device to the screen mirroring sender to achieve remote control of the screen mirroring sender. Since there is often a problem of resolution mismatch between the screen mirroring sender and the display device, it is necessary to convert the coordinate data of the user operation point at the display device to accurately control the screen mirroring sender to perform corresponding operations.

[0426] In some examples, the display device can determine a first coordinate mapping relationship based on the first coordinate information and the second coordinate information, where the first coordinate information is the coordinate information of the display content when the screen mirroring sender displays the display content at the target resolution, and the second coordinate information is the coordinate information of the display content when the display device displays the display content at the target resolution. The display device can determine a second coordinate mapping relationship based on the third coordinate information of the display content when the display device currently displays the display content and the second coordinate information.

[0427] The screen mirroring sender can provide the display device with the coordinate information of the screen mirroring content when the screen mirroring sender displays the screen mirroring content at the target resolution, denoted as the first coordinate information. For example, at the screen mirroring sender, the coordinate information of the area vertices of the area covered by the screen mirroring content (or the coordinate information of the center point of the area, the coordinate information of the preset reference point of the area). Optionally, the target resolution can be the resolution of the content displayed by the screen mirroring sender.

[0428] The display device can pre-determine the coordinate information of the screen mirroring content when the screen mirroring content is displayed at the target resolution, denoted as the second coordinate information. For example, at the display device, the coordinate information of the area vertices of the area covered by the screen mirroring content (or the coordinate information of the center point of the area, the coordinate information of the preset reference point of the area).

[0429] In an actual application scenario, the display device displays the screen mirroring content at other resolutions, which is conveniently referred to as the actual resolution. The display device can be based on the third coordinate information when displaying the screen mirroring content according to the actual resolution, such as the coordinate information of the area vertices of the area covered by the screen mirroring content (or the coordinate information of the center point of the area, the coordinate information of the preset reference point of the area).

[0430] The display device can determine the second coordinate mapping relationship based on the third coordinate information and the second coordinate information. That is, the coordinate mapping relationship between the position of the screen mirroring content at the actual resolution and the position of the screen mirroring content in the target coordinate system.

[0431] Please combine Figure 10b With this, based on the first coordinate mapping relationship and the second coordinate mapping relationship, the display device can determine the coordinate Q' corresponding to the coordinate Q of the target point in the current display interface of the display device. The coordinate Q' is the coordinate of the target point when the screen mirroring content is displayed at the target resolution on the screen mirroring sending end.

[0432] In a possible design, based on the display method provided in the foregoing embodiment, at least one of the first controls includes a target first control, and the target first control corresponds to the function of the start control of the target application program.

[0433] The display device can also respond to an operation on the target first control, obtain the first coordinate of the start control of the target first application program in the display interface of the display device, and determine the second coordinate corresponding to the first coordinate based on the first coordinate mapping relationship and the second coordinate mapping relationship, and send the second coordinate to the screen mirroring sending end so that the screen mirroring sending end generates a simulated click event corresponding to the second coordinate.

[0434] Among them, the display device obtains the first coordinate of the start control of the target first application program in the display interface of the display device, that is, obtains the coordinate information of the start control of the target application program when the display device currently displays the target content.

[0435] The display device can determine the second coordinate corresponding to the first coordinate based on the first coordinate mapping relationship and the second coordinate mapping relationship. The second coordinate is the coordinate of the start control of the target application program when the display content corresponding to the target object is displayed at the target resolution on the screen mirroring sending end.

[0436] The display device sends the second coordinate to the screen mirroring sending end so that the screen mirroring sending end generates a simulated click event corresponding to the second coordinate, realizing virtual clicking on the position of the start control of the target application program in the display content corresponding to the target object displayed on the screen mirroring sending end. In such a design, the user can implement reverse control of the screen mirroring sending end through the first control in the first menu.

[0437] Optionally, if the actual display resolution of the screen mirroring sender is not the target resolution, similar to the display device processing process, the screen mirroring sender can determine a third coordinate mapping relationship based on the target resolution and the actual display resolution of the screen mirroring sender, and determine the position corresponding to the received second coordinate, that is, the screen mirroring sender determines the position corresponding to the second coordinate under its own actual display resolution.

[0438] In a possible design, based on the display method provided in the foregoing embodiment, the at least one first control includes a target first control, and the target first control corresponds to some functions of the target application; some functions of the target first control correspond to multiple simulated click positions in the display interface of the display device, and the order corresponding to each simulated click position among the multiple simulated click positions is pre-configured.

[0439] The display device can also respond to an operation on the target first control, determine the third coordinates corresponding to each simulated click position based on the first coordinate mapping relationship and the second coordinate mapping relationship, provide the third coordinates corresponding to each simulated click position to the screen mirroring sender, and indicate the order corresponding to each simulated click position, so that the screen mirroring sender can sequentially generate simulated click events corresponding to the third coordinates of each simulated click position according to the order.

[0440] Among them, for the multiple simulated click positions corresponding to the functions of the target first control, the display device can determine the coordinates corresponding to each simulated click position when the target object is displayed at the target resolution on the screen mirroring sender based on the first coordinate mapping relationship and the second coordinate mapping relationship.

[0441] Based on the display method provided in any of the foregoing embodiments, the display device can support multiple users to use synchronously, or the display device can support multiple windows to use synchronously. For example, the display device can support a multi-window mode or a split-screen mode.

[0442] During the operation of the display device to detect the position of the target object, multiple target objects can be detected. The display device can display the first menu corresponding to any detected target object within the associated display area corresponding to each target object.

[0443] As Figure 11a shown, in the multi-window usage mode of the display device, the first menu AA corresponding to the target object A is displayed in the associated display area of the target object A. The first menu BB corresponding to the target object B is displayed in the associated display area of the target object B.

[0444] As Figure 11bAs shown, the display device is in a split-screen usage mode, or a partition usage mode. In the associated display area of the target object A, the first menu AA corresponding to the target object A is displayed. In the associated display area of the target object B, the first menu BB corresponding to the target object B is displayed.

[0445] In a possible design, the display device can determine the controlled area corresponding to any one of the target objects based on the position of any one of the target objects, and the first menu corresponding to any one of the target objects is generated based on the content in the controlled area corresponding to any one of the target objects.

[0446] In some examples, the target object is a controlled window. The content in the controlled area corresponding to the target object is the content in the controlled window.

[0447] In some examples, the target object is a user. The content in the controlled area corresponding to the user can be implemented as the content in a partition used by a user, or can be implemented as the content in a window used by a user. In some application scenarios, multiple users may operate on the same partition or the same window. In this case, the content in the controlled areas corresponding to each user is the same. The associated display areas corresponding to each user are determined based on the positions of the users. If the positions of the users are different, the associated display areas corresponding to the users are also different. Generally speaking, the associated display area corresponding to a user is near the position of the user and is an area that the user can touch.

[0448] For the process of the display device generating the first menu corresponding to a target object based on the content in the controlled area corresponding to the target object, reference can be made to the relevant introduction in the foregoing embodiments, and details are not described herein again.

[0449] Based on the relevant introduction of the foregoing display method, an embodiment of the present application further provides a display method, which can be executed by a display device, as Figure 12a shown, the display method may include the following steps:

[0450] S1201a, detecting the position of the target object;

[0451] S1202a, determining the associated display area corresponding to the target object based on the position of the target object, where the associated display area is the range for displaying the first menu;

[0452] S1203a, determining the first control in the first menu.

[0453] Optionally, the display device can determine the first control in the first menu based on the display content corresponding to the target object. For the execution process of the display device, reference can be made to the relevant introduction in the foregoing embodiments, and details are not described herein again.

[0454] Alternatively, the display device may determine a first control in the first menu based on the target user's intention. The execution process of the display device may refer to the relevant introduction in the foregoing embodiments, which will not be elaborated here.

[0455] S1204a. Display a first menu within the associated display area corresponding to the target object.

[0456] In this embodiment, for the operations performed by the display device in steps S1201a, S1202a, and S1204a, reference may be made to the relevant introduction in the foregoing embodiments, which will not be elaborated here.

[0457] Based on the relevant introduction of the foregoing display method, an embodiment of the present application further provides a display method that can be executed by a display device. As Figure 12b shown, the display method may include the following steps:

[0458] S1201b. Detect the position of the target object;

[0459] S1202b. Determine the associated display area corresponding to the target object based on the position of the target object. The associated display area is the range for displaying the first menu;

[0460] S1203b. Determine a first control in the first menu. The function of the first control in the first menu is determined based on the display content corresponding to the target object.

[0461] Alternatively, the display device may determine a first control in the first menu based on the target user's intention. The execution process of the display device may refer to the relevant introduction in the foregoing embodiments, which will not be elaborated here.

[0462] S1204b. Display a first menu within the associated display area corresponding to the target object;

[0463] S1205b. In response to an operation on the target first control, call an interface to start the target application. The target first control corresponds to the start control of the target application.

[0464] In this embodiment, for the operations performed by the display device in steps S1201b - 1205b, reference may be made to the relevant introduction in the foregoing embodiments, which will not be elaborated here.

[0465] Based on the relevant introduction of the foregoing display method, an embodiment of the present application further provides a display method that can be executed by a display device. As Figure 12c shown, the display method may include the following steps:

[0466] S1201c. Detect the position of the target object;

[0467] S1202c. Determine an associated display area corresponding to the target object based on the position of the target object, where the associated display area is the range for displaying the first menu;

[0468] S1203c. Determine a first control in the first menu, where the function of the first control in the first menu is determined based on the display content corresponding to the target object.

[0469] Alternatively, the display device may determine the first control in the first menu based on the target user intent. The execution process of the display device may refer to the relevant introduction in the foregoing embodiments and will not be elaborated here.

[0470] S1204c. Display the first menu within the associated display area corresponding to the target object;

[0471] S1205c. In response to an operation on the target first control, generate a simulated click event on the position of the start control of the target application, where the target first control corresponds to the start control of the target application.

[0472] In this embodiment, for the operations performed by the display device in steps S1201c - 1205c, reference may be made to the relevant introduction in the foregoing embodiments and will not be elaborated here.

[0473] Based on the relevant introduction of the foregoing display method, an embodiment of the present application further provides a display method that may be executed by a display device. As Figure 12d shown, the display method may include the following steps:

[0474] S1201d. Detect the position of the target object;

[0475] S1202d. Determine an associated display area corresponding to the target object based on the position of the target object, where the associated display area is the range for displaying the first menu;

[0476] S1203d. Determine a first control in the first menu, where the function of the first control in the first menu is determined based on the display content corresponding to the target object.

[0477] Alternatively, the display device may determine the first control in the first menu based on the target user intent. The execution process of the display device may refer to the relevant introduction in the foregoing embodiments and will not be elaborated here.

[0478] S1204d. Display the first menu within the associated display area corresponding to the target object;

[0479] The target first control corresponds to the target function of the target application, and the target function corresponds to multiple simulated click positions. Among them, the order corresponding to each simulated click position among the multiple simulated click positions is pre-configured.

[0480] S1205d. In response to an operation on a target first control, sequentially generate simulated click events corresponding to a plurality of simulated click positions corresponding to a target function.

[0481] In this embodiment, for the operations performed by the display device in steps S1201d - 1205d, reference can be made to the relevant descriptions in the foregoing embodiments, which will not be elaborated here.

[0482] Based on the relevant descriptions of the foregoing display method, an embodiment of the present application further provides a display method, which can be executed by a display device. As Figure 12e shown, the display method may include the following steps:

[0483] S1201e. Detect whether the display device has an image acquisition device. If so, perform step S1202e next. If not, perform step S1203e next.

[0484] S1202e. Take the currently controlled window as the target object.

[0485] S1203e. Take a pedestrian as the target object, and call a pedestrian detection model to detect the image collected by the image acquisition device.

[0486] S1204e. Determine the associated display area corresponding to each target object

[0487] S1205e. Determine the first menu corresponding to each target object

[0488] S1206e. In the associated display area corresponding to each target object, display the first menu corresponding to the target object.

[0489] For the operations performed by the display device in this embodiment, reference can be made to the relevant descriptions in the foregoing embodiments, which will not be elaborated here.

[0490] An embodiment of the present application further provides a display method, which can be executed by a display device. As Figure 12f shown, the display method may include the following steps:

[0491] S1201f. Detect the position of the target object;

[0492] S1202f. Based on the position of the target object, determine the display content corresponding to the target object;

[0493] S1203f. Based on the display content corresponding to the target object, determine the first menu corresponding to the target object.

[0494] S1204f. Display the first menu.

[0495] In this embodiment, for the operations of the display device to execute steps S1201f - S1203f, reference can be made to the relevant descriptions in the foregoing embodiments, which will not be elaborated herein.

[0496] In step S1204f, the display device may display the first menu at any position on the display interface. The specific position where the display device displays the first menu may be determined based on a preset menu display policy.

[0497] Optionally, the preset menu display policy may include, but is not limited to, any of the following policies:

[0498] Policy 1: Display the first menu in the associated display area of the target object;

[0499] Policy 2: Display the first menu at a position below the display interface without obscuring any display object;

[0500] Policy 3: Display the first menu in the user usage area;

[0501] Policy 4: Follow the movement of the user and display the first menu at a position that can be touched by the user's hand;

[0502] Policy 5: Display the first menu at the position configured by the user;

[0503] Policy 6: Display the first menu at the system default position.

[0504] In a possible implementation manner, in the display method provided by the embodiments of the present application, the target object is the controlled window. The position of the target object is the position of the controlled window. When the controlled window switches, the switched - to controlled window is used as the target object.

[0505] In a possible implementation manner, in the display method provided by the embodiments of the present application, the target object is the user; the position of the target object is determined based on an image of the user collected by an image acquisition device.

[0506] In a possible implementation manner, in the display method provided by the embodiments of the present application, the method further includes:

[0507] In response to detecting an update in the position of the target object, based on the updated position of the target object, update the display content corresponding to the target object, and update the first menu to display the updated first menu.

[0508] In a possible implementation manner, in the display method provided by the embodiments of the present application, the first menu includes at least one fixed - function control, and the functions of each fixed - function control are pre - configured.

[0509] In a possible implementation manner, in the display method provided by the embodiments of the present application, the first menu includes at least one first control.

[0510] In a possible implementation manner, in the display method provided by the embodiments of the present application, the at least one first control is determined based on the display content corresponding to the target object.

[0511] In a possible implementation manner, in the display method provided by the embodiments of the present application, when different target objects are detected, the first controls included in the first menu are different.

[0512] In a possible implementation manner, in the display method provided by the embodiments of the present application, when the target object is a controlled window, the display content corresponding to the target object refers to the content in the controlled window.

[0513] In a possible implementation manner, in the display method provided by the embodiments of the present application, when the target object is a user, the display content corresponding to the target object refers to the content in the area used by the user.

[0514] In a possible implementation manner, in the display method provided by the embodiments of the present application, the method further includes:

[0515] Displaying a second menu, the second menu including at least one second control;

[0516] In response to an operation on a target second control, displaying the first menu.

[0517] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes one or more of the following:

[0518] The start control of an application, the window of an application;

[0519] One of the first controls corresponds to any one of the following:

[0520] The function of a start control of an application, a partial function of an application.

[0521] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes the start control of a target application; the at least one first control includes a target first control, and the target first control corresponds to the function of the start control of the target application;

[0522] The method further includes:

[0523] In response to an operation on the target first control, generating a simulated click event for the start control of the target application.

[0524] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes a window of a target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application;

[0525] In response to an operation on the target first control, a partial function of the target application is triggered.

[0526] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes a window of a target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application; the partial function of the target application corresponds to a simulated click position.

[0527] The step of triggering the partial function of the target application in response to an operation on the target first control includes:

[0528] In response to an operation on the target first control, in the window of the target application, a simulated click event is generated based on the simulated click position to trigger the partial function of the target application.

[0529] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes a window of a target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application;

[0530] The step of triggering the partial function of the target application in response to an operation on the target first control includes:

[0531] In response to an operation on the target first control, in the window of the target application, a plurality of simulated click events are generated in sequence to trigger the partial function of the target application.

[0532] In a possible implementation manner, in the display method provided by the embodiments of the present application, the partial function of the target application corresponds to a plurality of simulated click positions with a preset arrangement order;

[0533] The step of generating a plurality of simulated click events in sequence in the window of the target application includes:

[0534] In the window of the target application, a plurality of simulated click events are generated according to the plurality of simulated click positions with a preset arrangement order.

[0535] In a possible implementation manner, in the display method provided by the embodiments of the present application, a part of the target application is a target function; the multiple simulated click positions corresponding to the target function with a preset arrangement order are determined in the following manner:

[0536] In response to an operation on the target application, capture a screenshot of the display interface;

[0537] Use a pre-trained recognition model to recognize the controls in the screenshot, and determine the positions of the recognized multiple third controls as the multiple simulated click positions with a preset arrangement order, where the multiple third controls are used to trigger the target function in the target application.

[0538] In a possible implementation manner, in the display method provided by the embodiments of the present application, a part of the functions of the application is determined based on the usage conditions of the functions in the application;

[0539] The usage conditions of the functions include any one of the following:

[0540] The usage frequencies of the functions in the current device, the usage frequencies of the functions provided by the application vendor, and the usage frequencies of the functions in the application used by the user of the current device.

[0541] In a possible implementation manner, in the display method provided by the embodiments of the present application, the at least one first control is determined based on the target user intention.

[0542] In a possible implementation manner, in the display method provided by the embodiments of the present application, the method further includes:

[0543] Based on the display content corresponding to the target object and the target user intention, determine the first control in the first menu.

[0544] In a possible implementation manner, in the display method provided by the embodiments of the present application, the method further includes:

[0545] Based on the display content corresponding to the target object, determine the target scene;

[0546] Based on the target scene and the target operation of the user on the display device, determine the target user intention.

[0547] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object includes one or more of the following:

[0548] The start control of the application, the window of the application;

[0549] The method further includes:

[0550] Match each function of each application in the display content corresponding to the target object with the target user intention, and the matching result includes the function of the start control of at least one application and / or part of the functions of at least one application;

[0551] Determine the first control in the first menu based on the functions in the matching result.

[0552] In a possible implementation manner, in the display method provided by the embodiments of the present application, the display content corresponding to the target object is provided by the screen mirroring sending end;

[0553] The method further includes:

[0554] Determine a first coordinate mapping relationship based on the first coordinate information and the second coordinate information, where the first coordinate information is the coordinate information of the display content when the screen mirroring sending end displays the display content at the target resolution, and the second coordinate information is the coordinate information of the display content when the display device displays the display content at the target resolution;

[0555] Determine a second coordinate mapping relationship based on the third coordinate information of the display content when the display device currently displays the display content and the second coordinate information.

[0556] In a possible implementation manner, in the display method provided by the embodiments of the present application, the at least one first control includes a target first control, and the target first control corresponds to the function of the start control of the target application;

[0557] The method further includes:

[0558] In response to an operation on the target first control, obtain the first coordinate of the start control of the target first application in the display interface of the display device, and determine the second coordinate corresponding to the first coordinate based on the first coordinate mapping relationship and the second coordinate mapping relationship, and send the second coordinate to the screen mirroring sending end so that the screen mirroring sending end generates a simulated click event corresponding to the second coordinate.

[0559] In a possible implementation manner, in the display method provided by the embodiments of the present application, the at least one first control includes a target first control, and the target first control corresponds to part of the functions of the target application; part of the functions of the target first control correspond to multiple simulated click positions in the display interface of the display device, and the order corresponding to each simulated click position among the multiple simulated click positions is pre-configured;

[0560] The method further includes:

[0561] In response to an operation on the target first control, based on the first coordinate mapping relationship and the second coordinate mapping relationship, determine the third coordinates corresponding to each simulated click position, and provide the third coordinates corresponding to each simulated click position to the screen mirroring sending end and indicate the order corresponding to each simulated click position, so that the screen mirroring sending end generates simulated click events for the third coordinates corresponding to each simulated click position in sequence according to the order.

[0562] In a possible implementation manner, in the display method provided by the embodiments of the present application, the number of the detected target objects is multiple;

[0563] For any detected target object, based on the position corresponding to the any target object, display the first menu corresponding to the any target object.

[0564] In a possible implementation manner, based on the position corresponding to the any target object, determine the associated display area corresponding to the target object, and the first menu corresponding to the target object is displayed in the associated display area corresponding to the target object.

[0565] In a possible implementation manner, in the display method provided by the embodiments of the present application, the method further includes:

[0566] Based on the position of the any target object, determine the controlled area corresponding to the any target object, and the first menu corresponding to the any target object is generated based on the content in the controlled area corresponding to the any target object.

[0567] Based on the same inventive concept, the present application further provides a display device. Please refer to Figure 13 , the display device may include a display module and a processing module;

[0568] The display module is used to display content;

[0569] The processing module is used to execute the steps in the display method provided in any of the foregoing embodiments.

[0570] In a possible implementation manner, the processing module is used to:

[0571] Detect the position of the target object;

[0572] Based on the position of the target object, determine the associated display area corresponding to the target object, and the associated display area is the range for displaying the first menu;

[0573] Cause the display module to display the first menu within the associated display area corresponding to the target object.

[0574] In a possible implementation manner, in the display device provided by the embodiments of the present application, the target object is a controlled window; the processing module determines an associated display area corresponding to the target object based on the position of the target object, specifically for:

[0575] Determine an associated display area corresponding to the target object based on the position of the controlled window.

[0576] In a possible implementation manner, in the display device provided by the embodiments of the present application, the processing module determines an associated display area of the target object based on the position of the controlled window, specifically for:

[0577] Determine an associated display area of the target object based on the position of the target boundary of the controlled window.

[0578] In a possible implementation manner, in the display device provided by the embodiments of the present application, the area size of the associated display area is pre-configured.

[0579] In a possible implementation manner, in the display device provided by the embodiments of the present application,

[0580] The target boundary is a preset boundary of the controlled window; or,

[0581] The target boundary is a boundary that meets a preset condition among all the boundaries of the controlled window, where the preset condition is any one of the following conditions:

[0582] Among the distances between each boundary and the ground, the distance between the target boundary and the ground is the smallest;

[0583] Among the distances between each boundary and the position where the most recent user operation occurred, the distance between the target boundary and the position where the most recent user operation occurred is the smallest;

[0584] In all the boundaries, there is no display object within the target range on the side of the target boundary away from the controlled window.

[0585] In a possible implementation manner, in the display device provided by the embodiments of the present application, the determining the associated display area of the target object based on the position of the controlled window includes:

[0586] Determine an associated display area of the target object based on the positions of the vertices of the controlled window.

[0587] In a possible implementation manner, in the display device provided by the embodiments of the present application, the target object is a user; the position of the target object is determined based on an image including the user collected by an image acquisition device.

[0588] In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module is further configured to:

[0589] In response to detecting that the position of the target object is updated, based on the updated position of the target object, update the associated display area corresponding to the target object, and display the first menu in the updated associated display area.

[0590] In a possible implementation manner, in the display device provided by the embodiment of the present application, the first menu includes at least one fixed function control, where the function of the fixed function control is pre-configured.

[0591] In a possible implementation manner, in the display device provided by the embodiment of the present application, the first menu includes at least one first control.

[0592] In a possible implementation manner, in the display device provided by the embodiment of the present application, the at least one first control is determined based on the display content corresponding to the target object.

[0593] In a possible implementation manner, in the display device provided by the embodiment of the present application, when different target objects are detected, the first controls included in the first menu are different.

[0594] In a possible implementation manner, in the display device provided by the embodiment of the present application, the target object is a controlled window, and the display content corresponding to the target object refers to the content in the controlled window.

[0595] In a possible implementation manner, in the display device provided by the embodiment of the present application, the target object is a user, and the display content corresponding to the target object refers to the content in the area used by the user.

[0596] In a possible implementation manner, in the display device provided by the embodiment of the present application, when the processing module displays the first menu corresponding to the target object in the associated display area corresponding to the target object, it is specifically configured to:

[0597] Display a second menu in the associated display area corresponding to the target object, where the second menu includes at least one second control;

[0598] In response to an operation on a target second control, display the first menu in the associated display area corresponding to the target object.

[0599] In a possible implementation manner, in the display device provided by the embodiment of the present application, the display content corresponding to the target object includes one or more of the following:

[0600] The start control of the application program, the window of the application program;

[0601] The first control corresponds to any one of the following:

[0602] The function of the startup control of an application, or a partial function of an application.

[0603] In a possible implementation manner, in the display device provided by the embodiment of the present application, the partial function of the application is triggered based on the control in the application.

[0604] In a possible implementation manner, in the display device provided by the embodiment of the present application, the display content corresponding to the target object includes the startup control of the target application; the at least one first control includes a target first control, and the target first control corresponds to the function of the startup control of the target application;

[0605] The processing module is further configured to:

[0606] In response to an operation on the target first control, call the interface for starting the target application.

[0607] In a possible implementation manner, in the display device provided by the embodiment of the present application, the display content corresponding to the target object includes the startup control of the target application; the at least one first control includes a target first control, and the target first control corresponds to the function of the startup control of the target application;

[0608] The processing module is further configured to:

[0609] In response to an operation on the target first control, generate a simulated click event for the startup control of the target application.

[0610] In a possible implementation manner, in the display device provided by the embodiment of the present application, the display content corresponding to the target object includes the window of the target application; the at least one first control includes a target first control, and the target first control corresponds to the partial function of the target application;

[0611] The processing module is further configured to:

[0612] In response to an operation on the target first control, trigger the partial function of the target application.

[0613] In a possible implementation manner, in the display device provided by the embodiment of the present application, the display content corresponding to the target object includes the window of the target application; the at least one first control includes a target first control, and the target first control corresponds to the partial function of the target application; the partial function of the target application corresponds to a simulated click position;

[0614] The processing module triggers some functions of the target application in response to an operation on the target first control, specifically for:

[0615] In response to an operation on the target first control, in the window of the target application, a simulated click event is generated based on the simulated click position to trigger some functions of the target application.

[0616] In a possible implementation manner, in the display device provided in the embodiments of the present application, the display content corresponding to the target object includes the window of the target application; the at least one first control includes the target first control, and the target first control corresponds to some functions of the target application;

[0617] The processing module triggers some functions of the target application in response to an operation on the target first control, specifically for:

[0618] In response to an operation on the target first control, multiple simulated click events are generated in sequence in the window of the target application to trigger some functions of the target application.

[0619] In a possible implementation manner, in the display device provided in the embodiments of the present application, some functions of the target application correspond to multiple simulated click positions with a preset arrangement order;

[0620] The processing module generates multiple simulated click events in sequence in the window of the target application, specifically for:

[0621] In the window of the target application, multiple simulated click events are generated according to the multiple simulated click positions with a preset arrangement order.

[0622] In a possible implementation manner, in the display device provided in the embodiments of the present application, some functions of the target application are the target functions; the multiple simulated click positions with a preset arrangement order corresponding to the target functions are determined in the following manner:

[0623] In response to an operation on the target application, take a screenshot of the display interface;

[0624] Use a pre-trained recognition model to recognize the controls in the screenshot, and determine the positions of the recognized multiple third controls as the multiple simulated click positions with a preset arrangement order. The multiple third controls are used to trigger the target function in the target application.

[0625] In a possible implementation manner, in the display device provided in the embodiments of the present application, some functions of the application are determined based on the usage conditions of each function in the application;

[0626] The usage situations of the respective functions include any one of the following:

[0627] The usage frequencies of the respective functions in the current device, the usage frequencies of the respective functions provided by the application program provider, and the usage frequencies of the respective functions in the application program used by the user of the current device.

[0628] In a possible implementation manner, in the display device provided by the embodiment of the present application, the at least one first control is determined based on the target user intention.

[0629] In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module is further configured to:

[0630] Determine the first control in the first menu based on the display content corresponding to the target object and the target user intention.

[0631] In a possible implementation manner, in the display device provided by the embodiment of the present application, the processing module is further configured to:

[0632] Determine a target scenario based on the display content corresponding to the target object;

[0633] Determine the target user intention based on the target scenario and the target operation of the user on the display device.

[0634] In a possible implementation manner, in the display device provided by the embodiment of the present application, the display content corresponding to the target object includes one or more of the following:

[0635] The start control of the application program, the window of the application program;

[0636] The processing module is further configured to:

[0637] Match each function of each application program in the display content corresponding to the target object with the target user intention, and the matching result includes the function of the start control of at least one application program and / or part of the functions of at least one application program;

[0638] Determine the first control in the first menu based on the functions in the matching result.

[0639] In a possible implementation manner, in the display device provided by the embodiment of the present application, the display content corresponding to the target object is provided by a screen mirroring sending end;

[0640] The processing module is further configured to:

[0641] Based on the first coordinate information and the second coordinate information, determine a first coordinate mapping relationship, where the first coordinate information is the coordinate information of the display content when the screen mirroring sending end displays the display content at a target resolution, and the second coordinate information is the coordinate information of the display content when the display device displays the display content at the target resolution;

[0642] Based on the third coordinate information of the display content when the display device currently displays the display content and the second coordinate information, determine a second coordinate mapping relationship.

[0643] In a possible implementation manner, in the display device provided in the embodiments of the present application, the at least one first control includes a target first control, and the target first control corresponds to the function of the start control of the target application;

[0644] The processing module is further configured to:

[0645] In response to an operation on the target first control, obtain the first coordinate of the start control of the target first application in the display interface of the display device, and based on the first coordinate mapping relationship and the second coordinate mapping relationship, determine the second coordinate corresponding to the first coordinate, and send the second coordinate to the screen mirroring sending end, so that the screen mirroring sending end generates a simulated click event corresponding to the second coordinate.

[0646] In a possible implementation manner, in the display device provided in the embodiments of the present application, the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application; the partial function of the target first control corresponds to multiple simulated click positions in the display interface of the display device, and the order corresponding to each simulated click position among the multiple simulated click positions is pre-configured;

[0647] The processing module is further configured to:

[0648] In response to an operation on the target first control, based on the first coordinate mapping relationship and the second coordinate mapping relationship, determine the third coordinates corresponding to each simulated click position, and provide the third coordinates corresponding to each simulated click position to the screen mirroring sending end and indicate the order corresponding to each simulated click position, so that the screen mirroring sending end sequentially generates simulated click events for the third coordinates corresponding to each simulated click position according to the order

[0649] In a possible implementation manner, in the display device provided in the embodiments of the present application, the number of the target objects is detected to be multiple;

[0650] Display the first menu in the associated display area corresponding to the target object, including:

[0651] For any detected target object, within the associated display area corresponding to the any target object, display a first menu corresponding to the any target object.

[0652] In a possible implementation manner, in the display device provided in the embodiments of the present application, the processing module is further configured to:

[0653] Based on the position of the any target object, determine a controlled area corresponding to the any target object, and the first menu corresponding to the any target object is generated based on the content in the controlled area corresponding to the any target object.

[0654] On the other hand, the present application further provides a display system, which may include an electronic device and a display device; the electronic device may be used as a screen mirroring sending end to provide screen mirroring content to the display device. The display device may execute the steps in the display method provided in any one of the foregoing embodiments.

[0655] Based on the same concept, Figure 14 FIG. 15 is a schematic structural diagram of an electronic device 1800 provided in the embodiments of the present application. The electronic device 1800 may be the first device or the second device in the foregoing. As Figure 14 shown, the electronic device 1800 may include: one or more processors 1801; one or more memories 1802; a communication interface 1803, and one or more computer programs 1804. The foregoing components may be connected through one or more communication buses 1805. Wherein the one or more computer programs 1804 are stored in the foregoing memory 1802 and are configured to be executed by the one or more processors 1801. The one or more computer programs 1804 include instructions. For example, the foregoing instructions may be used to execute the relevant steps in the display method executed by the display device in the corresponding foregoing embodiments. The communication interface 1803 is used to support connection with other devices. For example, the communication interface may be a transceiver.

[0656] In the foregoing embodiments provided by the present application, the method provided in the embodiments of the present application is introduced from the perspective of the display device as the execution subject. In order to implement the various functions in the method provided in the embodiments of the present application, the display device may include a hardware structure and / or a software module, and implement the various functions in the form of a hardware structure, a software module, or a combination of a hardware structure and a software module. Whether a certain function among the foregoing functions is executed in the form of a hardware structure, a software module, or a combination of a hardware structure and a software module depends on the specific application and design constraint conditions of the technical solution.

[0657] In addition, an embodiment of the present application provides a computer-readable storage medium for storing a computer program. When the computer program runs on a computer, the computer is caused to execute the steps in any of the above display methods.

[0658] As used in the above embodiments, depending on the context, the term "when..." or "after..." can be interpreted to mean "if...", or "after...", or "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if detecting (the stated condition or event)" can be interpreted to mean "if determining...", or "in response to determining...", or "when detecting (the stated condition or event)", or "in response to detecting (the stated condition or event)". In addition, in the above embodiments, relational terms such as first and second are used to distinguish one entity from another entity, without limiting any actual relationship and order between these entities.

[0659] References in this specification to "one embodiment" or "some embodiments" or the like mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, the appearances of the phrases "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. in different places in this specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0660] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0661] Embodiments of the present application also provide a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)). Without conflict, the solutions of the above embodiments may be combined and used.

Claims

1. A display method, wherein, Applied to a display device, including: Detect the position of the target object; Based on the position of the target object, determine the associated display area corresponding to the target object, where the associated display area is the range for displaying the first menu; Within the associated display area corresponding to the target object, display The the first menu.

2. The method according to claim 1, wherein The target object is the controlled window; Based on the position of the target object, determining the associated display area corresponding to the target object includes: Based on the position of the controlled window, determine the associated display area corresponding to the target object.

3. The method according to claim 2, wherein The determining the associated display area of the target object based on the position of the controlled window includes: Based on the position of the target boundary of the controlled window, determine the associated display area of the target object.

4. The method according to claim 3, wherein The area size of the associated display area is pre-configured.

5. The method according to claim 3, wherein, The target boundary is the preset boundary of the controlled window; or, The target boundary is the boundary that meets the preset conditions among all the boundaries of the controlled window, where the preset conditions are any one of the following conditions: Among the distances between each boundary and the ground, the distance between the target boundary and the ground is the smallest; Among the distances between each boundary and the position where the most recent user operation occurred, the distance between the target boundary and the position where the most recent user operation occurred is the smallest; Among all the boundaries, there is no display object within the target range on the side of the target boundary away from the controlled window.

6. The method according to claim 2, wherein The determining the associated display area of the target object based on the position of the controlled window includes: Based on the position of the vertex of the controlled window, determine the associated display area of the target object.

7. The method according to claim 1, wherein The target object is the user; the position of the target object is determined based on the image of the user collected by the image acquisition device.

8. The method according to claim 1, wherein, The method further includes: In response to detecting an update in the position of the target object, based on the updated position of the target object, update the associated display area corresponding to the target object, and display the first menu within the updated associated display area.

9. The method according to claim 8, wherein, The first menu includes at least one fixed function control, where the function of the fixed function control is pre-configured.

10. The method according to any one of claims 1-9, wherein, The first menu includes at least one first control.

11. The method according to claim 10, wherein, The at least one first control is determined based on the display content corresponding to the target object.

12. The method according to claim 10, wherein When different target objects are detected, the first controls included in the first menu are different.

13. The method according to claim 12, wherein, The target object is the controlled window, and the display content corresponding to the target object refers to the content in the controlled window.

14. The method according to claim 12, wherein, The target object is the user, and the display content corresponding to the target object refers to the content in the area used by the user.

15. The method according to claim 10, wherein, The displaying the first menu corresponding to the target object within the associated display area corresponding to the target object includes: Display a second menu within the associated display area corresponding to the target object, where the second menu includes at least one second control; In response to an operation on the target second control, display the first menu within the associated display area corresponding to the target object.

16. The method according to claim 10, wherein, The display content corresponding to the target object includes one or more of the following: The start control of the application and the window of the application; The first control corresponds to any one of the following: The function of the start control of an application and a partial function of an application.

17. The method according to claim 16, wherein, The partial function of the application is triggered by a control in the application.

18. The method according to claim 16, wherein, The display content corresponding to the target object includes the start control of the target application; the at least one first control includes a target first control, and the target first control corresponds to the function of the start control of the target application; The method further includes: In response to an operation on the target first control, calling an interface to start the target application.

19. The method according to claim 16, wherein, The display content corresponding to the target object includes the start control of the target application; the at least one first control includes a target first control, and the target first control corresponds to the function of the start control of the target application; The method further includes: In response to an operation on the target first control, generating a simulated click event on the start control of the target application.

20. The method according to claim 16, wherein, The display content corresponding to the target object includes the window of the target application; The at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application; The method further includes: In response to an operation on the target first control, triggering the partial function of the target application.

21. The method according to claim 20, wherein, The display content corresponding to the target object includes the window of the target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application; The partial function of the target application corresponds to a simulated click position; The step of, in response to an operation on the target first control, triggering the partial function of the target application includes: In response to an operation on the target first control, generating a simulated click event in the window of the target application based on the simulated click position to trigger the partial function of the target application.

22. The method according to claim 20, wherein, The display content corresponding to the target object includes the window of the target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application; The step of, in response to an operation on the target first control, triggering the partial function of the target application includes: In response to an operation on the target first control, generating a plurality of simulated click events in sequence in the window of the target application to trigger the partial function of the target application.

23. The method according to claim 22, wherein, The partial function of the target application corresponds to a plurality of simulated click positions with a preset arrangement order; The step of generating a plurality of simulated click events in sequence in the window of the target application includes: In the window of the target application, generating a plurality of simulated click events according to the plurality of simulated click positions with a preset arrangement order.

24. The method according to claim 23, wherein, The partial function of the target application is the target function; the plurality of simulated click positions with a preset arrangement order corresponding to the target function are determined in the following manner: Take a screenshot of the display interface in response to an operation on the target application; Use a pre-trained recognition model to recognize the controls in the screenshot, and determine the positions of the recognized multiple third controls as the multiple simulated click positions with a preset arrangement order. The multiple third controls are used to trigger the target function in the target application.

25. The method according to claim 16, wherein, Some functions of the application are determined based on the usage of each function in the application; The usage of each function includes any one of the following: The usage frequency of each function in the current device, the usage frequency of each function provided by the application vendor, and the usage frequency of each function in the application by the user using the current device.

26. The method according to claim 10, wherein, The at least one first control is determined based on the target user's intention.

27. The method according to claim 26, wherein, The method further includes: Determine the first control in the first menu based on the display content corresponding to the target object and the target user's intention.

28. The method according to claim 27, wherein, The method further includes: Determine the target scenario based on the display content corresponding to the target object; Determine the target user's intention based on the target scenario and the target operation of the user on the display device.

29. The method according to claim 27 or 28, wherein, The display content corresponding to the target object includes one or more of the following: The start control of the application, the window of the application; The method further includes: Match each function of each application in the display content corresponding to the target object with the target user's intention. The matching result includes the function of the start control of at least one application and / or some functions of at least one application; Determine the first control in the first menu based on the functions in the matching result.

30. The method according to claim 16, wherein, The display content corresponding to the target object is provided by the screen mirroring sender; The method further includes: Determine the first coordinate mapping relationship based on the first coordinate information and the second coordinate information. The first coordinate information is the coordinate information of the display content when the screen mirroring sender displays the display content at the target resolution, and the second coordinate information is the coordinate information of the display content when the display device displays the display content at the target resolution; Determine the second coordinate mapping relationship based on the third coordinate information of the display content when the display device currently displays the display content and the second coordinate information.

31. The method according to claim 30, wherein, The at least one first control includes a target first control, and the target first control corresponds to the function of the start control of the target application; The method further includes: In response to an operation on the target first control, obtain the first coordinate of the start control of the target first application in the display interface of the display device, and determine the second coordinate corresponding to the first coordinate based on the first coordinate mapping relationship and the second coordinate mapping relationship, and send the second coordinate to the screen mirroring sender so that the screen mirroring sender generates a simulated click event corresponding to the second coordinate.

32. The method according to claim 30, wherein, The at least one first control includes a target first control, and the target first control corresponds to a partial function of a target application; a partial function of the target first control corresponds to multiple simulated click positions in a display interface of the display device, and the order corresponding to each simulated click position among the multiple simulated click positions is pre-configured; The method further includes: In response to an operation on the target first control, based on the first coordinate mapping relationship and the second coordinate mapping relationship, determine third coordinates corresponding to each simulated click position, provide the third coordinates corresponding to each simulated click position to the screen mirroring sending end and indicate the order corresponding to each simulated click position, so that the screen mirroring sending end generates simulated click events of the third coordinates corresponding to each simulated click position in sequence according to the order.

33. The method according to claim 1, wherein, The number of the target objects is detected to be multiple; In an associated display area corresponding to the target object, display the first menu, including: For any detected target object, in the associated display area corresponding to the any detected target object, display the first menu corresponding to the any detected target object.

34. The method according to claim 33, wherein, The method further includes: Based on the position of the any detected target object, determine a controlled area corresponding to the any detected target object, and the first menu corresponding to the any detected target object is generated based on the content in the controlled area corresponding to the any detected target object.

35. A display method, wherein, The method includes: Detect the position of a target object; Based on the position of the target object, determine the display content corresponding to the target object; Based on the display content corresponding to the target object, determine the first menu corresponding to the target object; Display the first menu.

36. A display device, wherein, It includes a display module and a processing module; The display module is used to display content; The processing module is used to: Detect the position of a target object; Based on the position of the target object, determine an associated display area corresponding to the target object, and the associated display area is a range for displaying the first menu; Cause the display module to display the first menu in the associated display area corresponding to the target object.

37. The apparatus according to claim 36, wherein, The target object is a controlled window; the processing module determines the associated display area corresponding to the target object based on the position of the target object, specifically: Based on the position of the controlled window, determine the associated display area corresponding to the target object.

38. The device according to claim 37, wherein, The processing module determines the associated display area of the target object based on the position of the controlled window, specifically: Based on the position of a target boundary of the controlled window, determine the associated display area of the target object.

39. The device according to claim 38, wherein, The area size of the associated display area is pre-configured.

40. The device according to claim 38, wherein, The target boundary is a preset boundary of the controlled window; or, The target boundary is a boundary that meets a preset condition among all boundaries of the controlled window, where the preset condition is any one of the following conditions: Among the distances between each boundary and the ground, the distance between the target boundary and the ground is the smallest; Among the distances between each boundary and the position where the most recent user operation occurred, the distance between the target boundary and the position where the most recent user operation occurred is the smallest; Among all the boundaries, there is no display object in the target range on the side of the target boundary away from the controlled window.

41. The apparatus according to claim 37, wherein, Determining the associated display area of the target object based on the position of the controlled window includes: Determining the associated display area of the target object based on the positions of the vertices of the controlled window.

42. The device according to claim 36, wherein The target object is the user; the position of the target object is determined based on an image containing the user captured by an image acquisition device.

43. The apparatus according to claim 36, wherein, The processing module is further configured to: In response to detecting an update in the position of the target object, update the associated display area corresponding to the target object based on the updated position of the target object, and display the first menu within the updated associated display area.

44. The apparatus according to claim 43, wherein, The first menu includes at least one fixed function control, where the function of the fixed function control is pre-configured.

45. The device according to any one of claims 36-44, wherein The first menu includes at least one first control.

46. The apparatus according to claim 45, wherein, The at least one first control is determined based on the display content corresponding to the target object.

47. The apparatus according to claim 45, wherein, When different target objects are detected, the first controls included in the first menu are different.

48. The device according to claim 47, wherein, The target object is the controlled window, and the display content corresponding to the target object refers to the content in the controlled window.

49. The device according to claim 47, wherein, The target object is the user, and the display content corresponding to the target object refers to the content in the area used by the user.

50. The apparatus according to claim 45, wherein, The processing module displays the first menu corresponding to the target object within the associated display area corresponding to the target object, specifically by: Displaying a second menu within the associated display area corresponding to the target object, the second menu including at least one second control; In response to an operation on a target second control, displaying the first menu within the associated display area corresponding to the target object.

51. The apparatus according to claim 45, wherein, The display content corresponding to the target object includes one or more of the following: A start control for an application, a window of an application; The first control corresponds to any one of the following: The function of a start control for an application, a partial function of an application.

52. The device according to claim 51, wherein, The partial function of the application is triggered by a control in the application.

53. The device according to claim 51, wherein, The display content corresponding to the target object includes a start control for a target application; the at least one first control includes a target first control, and the target first control corresponds to the function of the start control for the target application; The processing module is further configured to: In response to an operation on the target first control, call an interface to start the target application.

54. The device according to claim 51, wherein, The display content corresponding to the target object includes a start control for a target application; the at least one first control includes a target first control, and the target first control corresponds to the function of the start control for the target application; The processing module is further configured to: In response to an operation on the target first control, generate a simulated click event for the start control of the target application.

55. The device according to claim 51, wherein, The display content corresponding to the target object includes the window of the target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application; The processing module is further configured to: In response to an operation on the target first control, trigger the partial function of the target application.

56. The apparatus according to claim 55, wherein, The display content corresponding to the target object includes the window of the target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application; the partial function of the target application corresponds to a simulated click position; When the processing module responds to an operation on the target first control and triggers the partial function of the target application, it is specifically configured to: In response to an operation on the target first control, in the window of the target application, generate a simulated click event based on the simulated click position to trigger the partial function of the target application.

57. The device according to claim 55, wherein, The display content corresponding to the target object includes the window of the target application; the at least one first control includes a target first control, and the target first control corresponds to a partial function of the target application; When the processing module responds to an operation on the target first control and triggers the partial function of the target application, it is specifically configured to: In response to an operation on the target first control, in the window of the target application, generate a plurality of simulated click events in sequence to trigger the partial function of the target application.

58. The device according to claim 57, wherein, The partial function of the target application corresponds to a plurality of simulated click positions with a preset arrangement order; When the processing module generates a plurality of simulated click events in sequence in the window of the target application, it is specifically configured to: In the window of the target application, generate a plurality of simulated click events according to the plurality of simulated click positions with a preset arrangement order.

59. The device according to claim 58, wherein, The partial function of the target application is the target function; the plurality of simulated click positions with a preset arrangement order corresponding to the target function are determined in the following manner: In response to an operation on the target application, capture a screenshot of the display interface; Use a pre-trained recognition model to recognize the controls in the screenshot, and determine the positions of the recognized plurality of third controls as the plurality of simulated click positions with a preset arrangement order. The plurality of third controls are used to trigger the target function in the target application.

60. The device according to claim 51, wherein, The partial function of the application is determined based on the usage of each function in the application; The usage of each function includes any one of the following: The usage frequency of each function in the current device, the usage frequency of each function provided by the application vendor, the usage frequency of each function in the application by the user using the current device.

61. The device according to claim 45, wherein, The at least one first control is determined based on the target user intention.

62. The device according to claim 61, wherein, The processing module is further configured to: Based on the display content corresponding to the target object and the target user intention, determine the first control in the first menu.

63. The device according to claim 62, wherein, The processing module is further configured to: Based on the display content corresponding to the target object, determine the target scenario; Determine the target user intent based on the target scenario and the target operation of the user on the display device.

64. The device according to claim 62 or 63, wherein, The display content corresponding to the target object includes one or more of the following: The start control of the application program, the window of the application program; The processing module is further configured to: Match each function of each application program in the display content corresponding to the target object with the target user intent, and the matching result includes the function of the start control of at least one application program, and / or, part of the functions of at least one application program; Determine the first control in the first menu based on the functions in the matching result.

65. The device according to claim 51, wherein, The display content corresponding to the target object is provided by the screen mirroring sender. The processing module is further configured to: Determine a first coordinate mapping relationship based on the first coordinate information and the second coordinate information, where the first coordinate information is the coordinate information of the display content when the screen mirroring sender displays the display content at the target resolution, and the second coordinate information is the coordinate information of the display content when the display device displays the display content at the target resolution; Determine a second coordinate mapping relationship based on the third coordinate information of the display content when the display device currently displays the display content and the second coordinate information.

66. The apparatus according to claim 65, wherein, The at least one first control includes a target first control, and the target first control corresponds to the function of the start control of the target application program; The processing module is further configured to: In response to an operation on the target first control, obtain the first coordinate of the start control of the target first application program in the display interface of the display device, and determine the second coordinate corresponding to the first coordinate based on the first coordinate mapping relationship and the second coordinate mapping relationship, and send the second coordinate to the screen mirroring sender so that the screen mirroring sender generates a simulated click event corresponding to the second coordinate.

67. The device according to claim 65, wherein, The at least one first control includes a target first control, and the target first control corresponds to part of the functions of the target application program; part of the functions of the target first control correspond to multiple simulated click positions in the display interface of the display device, and the order corresponding to each simulated click position among the multiple simulated click positions is pre-configured; The processing module is further configured to: In response to an operation on the target first control, determine the third coordinates corresponding to each simulated click position based on the first coordinate mapping relationship and the second coordinate mapping relationship, provide the third coordinates corresponding to each simulated click position to the screen mirroring sender and indicate the order corresponding to each simulated click position, so that the screen mirroring sender sequentially generates simulated click events corresponding to the third coordinates of each simulated click position according to the order.

68. The apparatus according to claim 36, wherein, The number of the target objects is detected to be multiple; Display the first menu in the associated display area corresponding to the target object, including: For any detected target object, display the first menu corresponding to the any target object in the associated display area corresponding to the any target object.

69. The apparatus according to claim 68, wherein, The processing module is further configured to: Based on the position of any one of the target objects, determine the controlled area corresponding to the any one of the target objects, and the first menu corresponding to the any one of the target objects is generated based on the content in the controlled area corresponding to the any one of the target objects.

70. A display device, wherein, It includes a processing module and a display module: The display module is used to display content; The processing module is used to detect the position of the target object; Based on the position of the target object, determine the display content corresponding to the target object; Based on the display content corresponding to the target object, determine the first menu corresponding to the target object; It is the display module that displays the first menu.

71. A display system, wherein, The system includes an electronic device and a display device; The electronic device provides screen mirroring content to the display device; The display device is used to execute the display method as described in claim 1.

72. A display system, wherein the system includes an electronic device and a display device; The electronic device provides screen mirroring content to the display device; The display device is used to execute the display method as described in claim 35.

73. A computer-readable storage medium, wherein, It includes a computer program, and when the computer program runs on an electronic device, the computer program is used to cause the electronic device to execute the steps of the method described in claim 1.

74. A computer-readable storage medium, wherein, It includes a computer program, and when the computer program runs on an electronic device, the computer program is used to cause the electronic device to execute the steps of the method described in claim 35.

Images