Display method and display system

Abstract

The invention discloses a display method and a display system.The display method comprises the steps that in response to a target instruction, first equipment is determined; the target instruction is used for displaying a target window on a first device, and the content displayed by the target window is used for indicating the completion condition of a target task; if the first equipment does not meet the target condition, determining second equipment meeting the target condition; cross-device display can be performed between the first device and the second device, and the target condition characterizes that the target task can be executed; and displaying the target window on the second equipment.

Description

Technical Field

[0001] The present application relates to, but is not limited to, the field of computer technology, and in particular to a display method and a display system. Background Art

[0002] With the advancement of computer technology, using multiple devices to collaboratively complete tasks, such as information display, data processing, or interactive operations, has become a common device usage model. In scenarios where multiple devices work together, how to distribute display content across different devices and ensure the proper execution of tasks has become a key issue in improving user experience and system efficiency. Summary of the Invention

[0003] In view of this, the present application at least provides a display method and a display system.

[0004] The technical solution of this application is achieved as follows:

[0005] In one aspect, the present application provides a display method, the method comprising:

[0006] In response to the target instruction, a first device is determined; the target instruction is used to display a target window on the first device, and the content displayed in the target window is used to indicate the completion status of the target task;

[0007] If the first device does not meet the target condition, determining a second device that meets the target condition; the first device and the second device can be displayed across devices, and the target condition indicates that the target task can be performed;

[0008] Display the target window on the second device.

[0009] In some embodiments, before determining the first device in response to the target instruction, the method further includes:

[0010] Obtaining a first input; the first input is for the first device or the second device, and is used to trigger a target application to perform a target task;

[0011] A target instruction is generated when it is determined that a task execution window corresponding to the first input is displayed on the first device.

[0012] In some embodiments, the method further comprises:

[0013] In response to completion of the target task, displaying other windows corresponding to the target application except the target window on the first device;

[0014] Alternatively, while the second device is displaying the target window, other windows corresponding to the target application except the target window are displayed on the first device.

[0015] In some embodiments, the target task representation utilizes a target sensor to perform object recognition, and the method further includes at least one of the following:

[0016] In response to the first device not including a target sensor, determining that the first device does not meet the target condition;

[0017] In response to a target sensor in the first device being in a fault state or an occupied state, it is determined that the first device does not meet the target condition.

[0018] In some implementations, after determining the first device in response to the target instruction, the method further includes:

[0019] In response to the first device satisfying the target condition, the target window is displayed on the first device.

[0020] In some implementations, determining a second device that meets a target condition includes:

[0021] Determining at least one third device that meets the target condition; wherein each third device can be displayed across devices with the first device;

[0022] A third device including a target sensor and capable of being called is determined as the second device.

[0023] In some embodiments, the third device including the target sensor and capable of being called by the target sensor is a plurality of third devices;

[0024] The method includes at least one of the following:

[0025] Obtaining the operator's current facial orientation, and determining, as the second device, a fourth device among the plurality of third devices whose display screen orientation has the smallest angle with the current facial orientation; wherein the operator is the subject operating the plurality of devices for performing cross-device display;

[0026] determining a fifth device among the plurality of third devices that matches the operator's interaction preference within a specified time period as the second device; wherein the operator is an object operating the plurality of devices for performing cross-device display;

[0027] In response to the plurality of third devices including a sixth device for running the target application, the sixth device is determined as the second device.

[0028] In some embodiments, determining, as the second device, a fourth device having a display screen facing the smallest angle with the current facial orientation among the plurality of third devices includes: determining physical positions of the plurality of third devices using the first model, and determining the fourth device using the physical positions of the plurality of third devices;

[0029] Determining a fifth device among multiple third devices that matches the operator's interaction preference within a specified time period as the second device includes: using a second model to determine the operator's interaction preference based on the operator's interaction information within the specified time period, and using the interaction preference to determine the fifth device.

[0030] In some embodiments, the method further comprises:

[0031] In response to an operator adjusting a display parameter of the target window, adjusting the display parameter of the target window from a first parameter to a second parameter;

[0032] The first parameter represents the display parameter before the adjustment operation; the second parameter represents the display parameter determined by the second model based on the adjustment operation and the interaction preference.

[0033] On the other hand, the present application also provides a display system, comprising:

[0034] a first device and a second device;

[0035] a processor, the processor running on the first device or the second device;

[0036] The processor is configured to determine a first device in response to a target instruction; the target instruction is configured to display a target window on the first device, wherein content displayed in the target window is configured to indicate a completion status of a target task;

[0037] The second device is configured to display a target window if the first device does not meet the target condition.

[0038] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0040] Figure 1 A schematic diagram of a multi-display screen usage scenario;

[0041] Figure 2 A schematic diagram of the implementation flow of a display method provided in this application;

[0042] Figure 3 A schematic diagram of an implementation flow of an embodiment provided in this application;

[0043] Figure 4 A schematic diagram of the system architecture of a display system provided in this application;

[0044] Figure 5 A schematic diagram of the hardware entity of an electronic device provided in this application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions of this application are further elaborated in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0046] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0047] The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first / second / third" can be interchanged with a specific order or sequence where permitted so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing this application only and are not intended to limit this application.

[0049] like Figure 1 As shown, in order to improve work efficiency, the operator 110 adds an extended display screen 120 and an extended display screen 140 on the basis of the main display screen 130. Among them, the main display screen 130 has a camera 131, and the extended display screen 120 and the extended display screen 140 are not equipped with cameras.

[0050] When operator 110 opens a camera-based application (e.g., the facial recognition login function built into the Windows system, or an application based on gesture recognition), the system automatically displays the application window on the display where the application was last closed, for example, on extended display 120. However, since extended display 120 is not equipped with a camera, operator 110 needs to turn his head so that his face faces camera 131 on main display 130 to perform facial verification. However, the application window of the camera-based application remains on extended display 120, so operator 110, who is facing main display 130, cannot promptly learn whether facial recognition is successful.

[0051] In the above scenario, the position of the camera 131 and the application window of the camera-based application are located on display screens with different orientations, causing the operator 110 to adjust the face orientation unnaturally, thereby interrupting the operator's workflow and causing a poor user experience.

[0052] Based on this, the present application provides a display method, which can be performed by an electronic device, which can be various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), etc., and can also be implemented as a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0053] In some embodiments, the display method provided in this application can be implemented as an artificial intelligence (AI) service. In some embodiments, the AI ​​service is a service running in the background of the operating system of the electronic device. In some embodiments, the AI ​​server has a corresponding application interface so that the user can use the application interface to configure the various functions of the AI ​​service, for example, configure the startup time of the AI ​​service, the application type or application group managed by the AI ​​service, etc. In some embodiments, the AI ​​service is automatically started when the operating system of the electronic device is started.

[0054] Below, the technical solution in this application will be clearly and completely described in conjunction with the drawings in this application.

[0055] Figure 2 A flow chart of a display method implementation provided by this application is shown as follows: Figure 2 As shown, the method includes the following steps S21 to S23:

[0056] Step S21, in response to a target instruction, determining a first device; the target instruction is used to display a target window on the first device, and the content displayed in the target window is used to indicate the completion status of the target task.

[0057] Here, the target task refers to a task performed using a target application. The target application can be any application installed on an electronic device. In some embodiments, the target task can be a user authentication task performed using the target application, or a task performed using the target application to implement a specific function of the application, such as a video call function or a gesture control function.

[0058] The target window refers to a graphical interface window that displays relevant information during the execution of the target task. In some embodiments, the target window can display information such as the execution progress, operation prompts, and execution results of the target task.

[0059] A target instruction may refer to an instruction used to trigger a target application to perform a target task. In some embodiments, the target instruction may be determined based on the operator's operation information or automatically determined by the system. For example, in response to an operator clicking the "Enter a meeting" control in conferencing software, a target instruction may be generated to initiate a video conference based on the target instruction. For another example, in response to an operator using a payment function in financial software, the system may automatically generate a target instruction based on the operator's operation behavior to authenticate the current operator based on the target instruction.

[0060] The first device refers to a device determined in response to a target instruction. The target instruction is used to display a target window on the first device, meaning that based on the target instruction, the first device can be determined as the initial device for displaying the target window. In some embodiments, the first device can be the device currently being operated by the operator, or the device displaying the current application window of the target application.

[0061] In some embodiments, when the method provided in the present application is implemented as an AI service, an application type detector in the AI ​​service can be used to identify events that trigger the target application to perform the target task through the first input through the application programming interface (API) of the electronic device operating system or system event monitoring.

[0062] Step S22: If the first device does not meet the target condition, determine a second device that meets the target condition; the first device and the second device can display across devices, and the target condition indicates that the target task can be performed.

[0063] Here, the target condition refers to the ability to perform the target task.

[0064] In some embodiments, the target condition represents the complete process of executing the target task. The complete process of the target task refers to any process requiring operator interaction during the execution of the target task. For example, if the target task is facial verification, the complete process of executing the target task includes at least the process of capturing the operator's facial image using a camera and the process of displaying the facial verification status to the operator.

[0065] In some embodiments, the conditions for executing the target task may be determined based on the type of the target task. For example, if the target task requires the use of a specified component, the target condition represents the presence of the specified component capable of executing the target task. Thus, if the first device does not meet the target condition, it means that the first device cannot execute the target task, or the first device cannot complete the entire process of the target task, or the first device does not have the specified component capable of executing the target task.

[0066] In the case where the first device does not meet the target condition, the second device can be determined based on the target condition. Among them, the second device and the first device can display across devices. In some embodiments, the first device and the second device are both independent terminal devices, and the two can synchronize and share information, content or applications, thereby achieving a cross-device display effect. For example, the first device and the second device are both independent laptops, tablet computers, desktop computers, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices) and other types of terminal devices, and can also be various types of server devices. In some embodiments, the first device and the second device can be multiple extended display screens or main display screens in a system consisting of a host and multiple extended display screens, thereby achieving cross-device display of information, content or applications between the first device and the second device through the host.

[0067] In some embodiments, when the method provided in the present application is implemented as an AI service, the screen configuration identification module in the AI ​​service can be used to determine the device configuration information of each device used to perform cross-device display; then, based on the device configuration information of each device, the intelligent window positioning engine in the AI ​​service can be used to determine a second device that meets the target conditions from multiple devices.

[0068] In some embodiments, after determining the first device in response to the target instruction, the output target window is displayed on the first device, and it is determined whether the first device meets the target condition; in response to the first device not meeting the target condition, a second device meeting the target condition is searched, and the target window is migrated from the first device to the second device.

[0069] In some embodiments, after determining the first device in response to the target instruction, it is determined whether the first device meets the target condition; in response to the first device not meeting the target condition, a second device meeting the target condition is searched for, and an output target window is displayed on the second device, that is, before determining that the first device does not meet the target condition, the output target window is not displayed on the first device.

[0070] Step S23: display the target window on the second device.

[0071] Here, after determining the second device that meets the target condition, the second device is determined as the target device for displaying the target window, and the output target window is displayed on the second device.

[0072] In the display method provided by this application, a first device is determined in response to a target instruction. If the first device does not meet the target condition, a second device that meets the target condition is determined, and the target window is displayed on the second device; wherein the first device and the second device can be displayed across devices. In this way, this application implements window switching display based on the task execution capabilities of different devices. On the one hand, it can guide the operator to pay attention to the second device, provide accurate data for the execution of the target task, and ensure the efficient completion of the task; on the other hand, it allows the operator to adjust his or her own behavior or status based on the target window displayed on the second device, thereby improving the operator's experience of using multi-display devices.

[0073] In some embodiments, before determining the first device in response to the target instruction, that is, before the above step S21, the method further includes the following steps S24 to S25:

[0074] Step S24: obtaining a first input; the first input is directed to the first device or the second device, and is used to trigger a target application to execute the target task.

[0075] Here, the first input is an input used to trigger the generation of the target instruction. In some embodiments, the first input can be any type of input. For example, the first input can be a click operation, a slide operation, text input, voice input, or gesture input performed by an operator on an input / output device on a target application, thereby triggering the target task based on the first input.

[0076] The first input is directed to the first device or the second device, meaning that the first input is an input performed by an operator on content or information displayed on the first device or the second device. For example, the first input may be an input operation performed by an operator on an application window of a target application displayed on the first device or the second device; for another example, the first input may be an input operation performed by an operator on an application icon of a target application displayed on the first device or the second device; and so on.

[0077] In this way, the first input is determined in response to a user's click operation, sliding operation, text input, voice input, gesture input, etc. on the application window or application icon of the target application displayed in the first device or the second device.

[0078] Step S25: Determine that the task execution window corresponding to the first input is displayed on the first device, and generate the target instruction.

[0079] Here, the task execution window refers to the application window of the target application, wherein the task execution window can be the application window currently being displayed or the application window to be displayed.

[0080] In some embodiments, it can be determined based on any suitable method that the task execution window corresponding to the first input is displayed on the first device. In some embodiments, when the task execution window is an application window that is currently being displayed, it can be determined that the task execution window corresponding to the first input is displayed on the first device. In some embodiments, when the task execution window is an application window to be displayed, it can be determined based on the device where the target application was last closed that the task execution window is displayed on the first device, that is, when the device where the target application was last closed is the first device, it is determined that the task execution window is displayed on the first device.

[0081] In some embodiments, the method further includes the following step S26 or step S27:

[0082] Step S26: In response to the target task being completed, displaying other windows corresponding to the target application except the target window on the first device.

[0083] Here, after the target task is completed, the target window completes its corresponding task. Therefore, in the following embodiment, in response to the target task being completed, the target window is removed.

[0084] Other windows corresponding to the target application other than the target window may refer to windows corresponding to the target application for performing any task other than the target task. In some embodiments, the other windows may be used to display different user interfaces, which are interfaces generated based on the success and failure of the target task, respectively. For example, in a face verification scenario, the other windows may be an authorization user interface displayed in response to a successful face verification, or a temporary user interface displayed in response to a failed face verification.

[0085] Since the application window of the target application is displayed on the first device before the target task is executed, or the application window of the target application was displayed on the first device when it was last closed, after the target task is executed, other windows corresponding to the target application except the target window are displayed on the first device, so that the display position of the application window of the target application can be consistent and continuous.

[0086] Step S27: While the second device is displaying the target window, the first device displays other windows corresponding to the target application except the target window.

[0087] Here, other windows corresponding to the target application except the target window may refer to application windows that are not related to the execution result of the target task.

[0088] In some embodiments, other windows corresponding to the target application, other than the target window, may be windows corresponding to other tasks unrelated to the target task. For example, in an instant messaging scenario, when the target task is a video call task, the target window is the video call window, and the other windows may be a text interaction window, a contact window, or a personal control center window corresponding to the instant messaging application.

[0089] In some embodiments, other windows corresponding to the target application, other than the target window, may be windows that prompt the operator to perform the target task. For example, in a scenario where the target application is logged in via facial verification, the target task is the facial verification task and the target window is the facial verification window. The other window may be a window that displays prompt information prompting the operator to perform the facial verification process.

[0090] In the above-mentioned embodiment provided in the present application, the device used to display other windows besides the target window does not need to meet the above-mentioned target conditions. Therefore, the other windows can be displayed on the first device to reduce the situation where the application windows of the same application jump between multiple devices, thereby making the display positions of the application windows of the same application consistent and continuous.

[0091] In some embodiments, the target task representation utilizes a target sensor for object recognition.

[0092] Here, the object may refer to the operator of the electronic device. In some embodiments, the object recognition may be facial recognition of the operator, gesture recognition of the operator, or the like.

[0093] A target sensor refers to an information collection device used to assist in object recognition. In some embodiments, when object recognition is for facial recognition or gesture recognition of an operator, the target sensor may be an image sensor used to collect facial or gesture information of the operator. Examples include an RGB sensor, an infrared sensor, or a Time of Flight (TOF) sensor.

[0094] It can be seen that since the target task is a task of performing object recognition using a target sensor, the target condition corresponding to the target task can represent a target sensor capable of performing the target task.

[0095] Thus, the method provided by the present application further includes at least one of the following steps S221 to S222:

[0096] Step S221: In response to the first device not including the target sensor, determining that the first device does not meet the target condition.

[0097] The first device does not include a target sensor, that is, the first device does not have a target sensor installed therein. For example, the first device does not have an integrated target sensor or an external target sensor.

[0098] In the case that the first device does not include the target sensor, the first device does not have the ability to perform the target task, and thus it is determined that the first device does not meet the target condition.

[0099] Step S222: In response to the target sensor in the first device being in a fault state or an occupied state, determining that the first device does not meet the target condition.

[0100] The target sensor in the first device is in a faulty state, which means that when the target sensor is integrated in the first device or the first device is externally connected to the target sensor, the function of the target sensor corresponding to the first device fails, or the performance parameter is lower than the normal threshold, and maintenance is required.

[0101] The target sensor in the first device is in an occupied state, which means that when the target sensor is integrated in the first device or the first device is externally connected to the target sensor, the target sensor corresponding to the first device is currently occupied by other applications or other tasks, resulting in it being unable to be used to execute the target task.

[0102] In the above embodiments provided in the present application, when the target sensor in the first device is in a faulty state or an occupied state, the target sensor corresponding to the first device cannot be called to perform the target task, resulting in the first device not having the ability to perform the target task, thereby determining that the first device does not meet the target condition.

[0103] In some implementations, after determining the first device in response to the target instruction, that is, after the above step S21, the following step S28 is further included:

[0104] Step S28: In response to the first device satisfying the target condition, displaying the target window on the first device.

[0105] The first device meeting the target condition means that the first device has the ability to perform the target task. For example, the first device includes a target sensor, and the target sensor is in a state that can be called.

[0106] In this way, since the first device is the initial device for displaying the target window determined in response to the target instruction (for example, when the target instruction is obtained, the application window of the target application is displayed on the first device, or the target application is displayed on the first device when it was last closed, etc.), therefore, when the first device meets the target condition, the target window is preferentially displayed on the first device so that the display device corresponding to the target window is the same as the display device corresponding to the current window or historical window of the target application, thereby improving the consistency and continuity of the application window display position, while reducing the operator's visual jumping between different devices, thereby improving the user experience.

[0107] In some implementations, determining the second device that meets the target condition in step S22 includes the following steps S223 to S224:

[0108] Step S223: Determine at least one third device that meets the target condition; wherein each of the third devices can be displayed across devices with the first device.

[0109] Here, the third device can be any device with a display screen.

[0110] In some embodiments, each third device is a terminal device capable of operating independently, and multiple third devices can achieve cross-device display effects. For example, each third device can be a laptop, tablet computer, desktop computer, mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device), or various types of server devices.

[0111] In some embodiments, at least one third device and the first device can be multiple extended display screens or main display screens in a system consisting of a host and multiple extended display screens, thereby enabling cross-device display of information, content, or applications between the first device and the second device through the host. For example, a designated third device or first device is the main display screen, and other third devices or first devices are extended display screens connected to the host. In this way, in some embodiments, different extended display screens respectively achieve cross-device display effects by interacting with information of the host. In some embodiments, each extended display has a communication unit and a processor unit to achieve communication connections between different extended displays, thereby achieving cross-device display effects between the host and multiple extended displays.

[0112] Here, the at least one third device that meets the target condition refers to at least one third device equipped with a target sensor.

[0113] In some embodiments, when the relationship between the at least one third device and the first device is similar to that between a host and multiple extended displays, the host maintains hardware configuration information for the primary and extended displays; based on this hardware configuration information, the host can determine at least one third device that meets the target criteria. In some embodiments, when different extended displays are communicatively connected, the hardware configuration information of the different extended displays and the host can be synchronized to determine at least one third device that meets the target criteria.

[0114] In some embodiments, when each third device is a terminal device or server device capable of operating independently, after determining that the first device does not meet the target conditions, different third devices can synchronize hardware configuration information to determine at least one third device that meets the target conditions.

[0115] Step S224: Determine a third device that includes the target sensor and can be called as the second device.

[0116] Here, after determining at least one third device, a third device that is equipped with a target sensor and can be called (ie, is in an idle state) is determined as the second device.

[0117] In the above embodiment provided by the present application, after determining that the first device does not meet the target condition, a third device including a target sensor and capable of being called is determined from at least one device displayed across devices, and the third device is determined as the second device. In this way, by migrating the target window to the device with the target sensor, during the execution of the target task, the operator does not need to switch his or her line of sight between different devices (i.e., the device with the target sensor installed and the device for displaying the target window), but can adjust his or her own behavior or state based on the target window displayed in the second device, thereby improving the operator's experience of using a multi-display device.

[0118] In some embodiments, the third device including the target sensor and capable of being called by the target sensor is a plurality of third devices;

[0119] Thus, the method provided by the present application includes at least one of the following steps S225 to S227:

[0120] Step S225, obtain the current facial orientation of the operator, and determine the fourth device among the multiple third devices whose display screen orientation has the smallest angle with the current facial orientation as the second device; wherein the operator is the object operating the multiple devices used to execute the cross-device display.

[0121] Here, the operator refers to an object that operates multiple devices for executing cross-device display. In the case where the target task is to perform object recognition using a target sensor, the operator is the object to be recognized.

[0122] In some embodiments, the operator's current facial orientation can be obtained by any suitable method. In some embodiments, first, cameras installed in multiple devices are used to capture environmental images, and a three-dimensional spatial coordinate system is established using multi-camera calibration technology. Then, when the operator's head appears in the field of view of a camera, the system uses a face detection algorithm to locate facial key points (such as the tip of the nose and the center of the eyes) and combines this with a coordinate conversion formula to determine the operator's facial orientation information. In some embodiments, first, at least one camera is used to capture an image of the operator's face. Then, the operator's facial orientation is predicted by extracting the positional features of the eyes in the image.

[0123] The orientation of the display screen refers to the physical orientation of the display screen, that is, the orientation information of the display screen in physical space. In some embodiments, the orientation of the display screens of multiple devices can be determined using any suitable method. In some embodiments, the physical positions of the display screens of multiple devices can be obtained through the system API, and the specific position information of the display interface of the display screen of each device can be determined in combination with the edge detection algorithm. In some embodiments, a wireless distance sensor or an electromagnetic induction sensor can be installed on a designated device, and a transmitter can be installed on other devices. In this way, the relative positions of other devices can be determined based on the signals received by the sensor on the designated device.

[0124] In this way, after determining the operator's current facial orientation and the display screen orientations of multiple third devices, the angle between the current facial orientation and the display screen orientation of each third device can be determined, and the third device corresponding to the smallest angle among the multiple angles (i.e., the fourth device) can be determined as the second device.

[0125] In the above embodiment, the fourth device among the multiple third devices whose display screen has the smallest angle with the operator's current facial orientation is determined as the second device. This way, after the target window is displayed on the second device, the operator only needs to turn their head slightly to face the second device, minimizing the operator's movement and, in turn, reducing the impact on the operator's workflow.

[0126] Step S226: Determine the fifth device among the multiple third devices that matches the operator's interaction preference within the specified time period as the second device; wherein the operator is the object operating the multiple devices for executing the cross-device display.

[0127] Here, the operator refers to an object that operates multiple devices for executing cross-device display. In the case where the target task is to perform object recognition using a target sensor, the operator is the object to be recognized.

[0128] Interaction preferences refer to an operator's habits and preferences for operating multiple devices. In some embodiments, interaction preferences may include which of multiple display screens the operator prefers to operate. For example, the operator prefers to operate the left-hand display screen. In some embodiments, interaction preferences may include the operator's preferred application window display parameters. For example, the operator's preferred application window size, font size, brightness, and other display parameters.

[0129] The operator's interaction preferences within a specified time period are determined based on analysis of the operator's operating habits and interaction information within the specified time period. In some embodiments, a model (e.g., a preference analysis model) can be used to analyze the operator's interaction information within the specified time period to determine the operator's interaction preferences. In some embodiments, statistics can be collected on the operator's interaction information within the specified time period, and the operator's interaction preferences can be determined based on the statistical results.

[0130] In the above embodiment, the fifth device among the multiple third devices that matches the operator's interaction preferences within a specified time period is determined as the second device. In other words, the operator's preferred device is used as the second device, so that the target window is displayed on the operator's preferred device. This allows the operator's interaction with the target window to better align with their interaction habits during the execution of the target task, thereby improving the user experience.

[0131] Step S227 : In response to the plurality of third devices including a sixth device for running the target application, determining the sixth device as the second device.

[0132] The sixth device refers to a device used to run the target application, that is, the process or thread corresponding to the target application is executed on the sixth device.

[0133] In the above embodiment, the sixth device running the target application is determined as the second device. That is, the target window is displayed on the device running the target application, so that the device displaying the target window and the device running the target application are the same device. In this way, the operator has a unified understanding of the display position of the target window and the device running the target application.

[0134] In some embodiments, determining the fourth device among the plurality of third devices whose display screen orientation has the smallest angle with the current facial orientation as the second device in step S225 above can be implemented as the following step S2251:

[0135] Step S2251: Determine the physical locations of the plurality of third devices using the first model, and determine the fourth device using the physical locations of the plurality of third devices.

[0136] Here, the first model may be a pre-trained model for identifying the physical locations of multiple devices.

[0137] In some embodiments, the first model can be a model for executing an edge detection algorithm. Thus, first, the coordinate positions of multiple third devices are obtained from the system APIs of multiple devices executing cross-device display, or from the system APIs of a host device among the multiple devices executing cross-device display. Then, the images displayed by the multiple third devices are input into the first model, which uses an edge detection algorithm to detect straight lines and curves in the images, thereby identifying the specific coordinates of the display boundary of each third device. Finally, the coordinate positions of the multiple third devices provided by the system API and the specific coordinates of the display boundary calculated by the first model are combined to more accurately determine the display boundary and position of each third device.

[0138] In some embodiments, the first model may be a model dynamically updated using an incremental algorithm. The incremental algorithm may include an online random forest (ORF) algorithm or a streaming K-means algorithm. Thus, the incremental algorithm may dynamically update the first model to adapt to changes in the device configurations of multiple devices executing cross-device display (e.g., the addition or removal of devices).

[0139] In some embodiments, the first model may be a model used to execute a device enumeration algorithm. Thus, for multiple devices performing cross-device display, the first model may be used to determine whether each device is equipped with a target sensor, such as a RAG sensor, a TOF sensor, or an infrared sensor.

[0140] In this way, using the first model, the physical positions of multiple third devices can be determined, and then the angle between the operator's current facial orientation and each third device can be calculated, and the third device corresponding to the minimum angle among the multiple angles can be determined as the fourth device.

[0141] In the above embodiment, the physical positions of multiple third devices are determined using the first model, which can quickly determine the physical positions of multiple third devices when the number of devices performing cross-device display is large and the screen orientation is complex, thereby improving the efficiency of determining the second device.

[0142] In some implementations, determining the fifth device among the plurality of third devices that matches the operator's interaction preference within a specified time period as the second device, that is, the above step S226, can be implemented as the following step S2261:

[0143] Step S2261: Using the second model, based on the interaction information of the operator within a specified time period, determine the interaction preference of the operator, and use the interaction preference to determine the fifth device.

[0144] Here, the second model can be any pre-trained model used to determine the operator's interaction preferences. In some embodiments, the second model can be a collaborative filtering model (CF), a matrix factorization model (MF), a graph neural network model (GNN), or a masked prediction language model.

[0145] In this way, the operator's interaction information within a specified time period is input into the second model, and the second model is used to predict the operator's interaction preferences within the specified time period, and then a fifth device that matches the operator's interaction preferences is determined from multiple third devices.

[0146] In the above embodiment, the model can be used to quickly determine the operator's interaction preference within a specified time period, thereby improving the efficiency of determining the second device.

[0147] In some embodiments, displaying the target window on the second device, i.e., step S23, can be implemented by first calculating the resolution ratio between the resolution of the first device and the resolution of the second device to obtain a first ratio; then, based on the first size of the target window on the first device and the first ratio, calculating the second size of the target window on the second device; and finally, displaying the target window on the second device at the second size. In this way, when the resolutions of the first device and the second device are different, for example, when the resolution of the first device is higher than that of the second device, by calculating the first ratio and using the first ratio to calculate the second size, the display size of the target window on the second device can be made to match the display size on the first device, and the problem of the target window being displayed too large on the second device will not occur.

[0148] In some embodiments, the method provided herein further comprises the following step S29:

[0149] Step S29, in response to the operator adjusting the display parameters of the target window, adjusting the display parameters of the target window from a first parameter to a second parameter; wherein the first parameter represents the display parameter before the adjustment operation; and the second parameter represents the display parameter determined by the second model based on the adjustment operation and the interaction preference.

[0150] Here, after the target window is displayed on the second device, if the operator adjusts the display parameters of the target window, the adjustment operation is sent as feedback information to the second model, so that the second model adjusts the display parameters of the target window based on the feedback operation.

[0151] In some implementations, the display parameters adjusted by the operator may be display size parameters, brightness parameters, font size parameters, font style parameters, font color parameters, or window background parameters of the target window.

[0152] The first parameter refers to the display parameter before the operator's adjustment operation, that is, the initial display parameter of the target window displayed on the second device.

[0153] The second parameter is a display parameter determined by the second model based on the operator's adjustment operation and the operator's interaction preference within a specified time period. In some embodiments, the second model can determine the interaction preference corresponding to the display parameter type corresponding to the operator's adjustment operation, and use the display parameter matching the interaction preference as the second parameter. For example, when the operator adjusts the size of the target window, the second model can determine the operator's preferred window size based on the operator's interaction preference, and use the window size as the second parameter; for another example, when the operator adjusts the font size in the target window, the second model can determine the operator's preferred font size based on the operator's interaction preference, and use the font size as the second parameter.

[0154] In some embodiments, a feedback analysis module in the AI ​​service can be used to receive the operator's adjustment operation on the target window and input the adjustment operation into the second model to analyze the user's adjustment operation using the second model and determine the second parameter.

[0155] In some embodiments, the second model can be fine-tuned based on the operator's behavioral data or feedback information (e.g., information on adjusting the display parameters of the target window). For example, when the operator adjusts the size of the target window and the size is different from the second parameter determined by the second model, the size can be used as a data source to continuously improve the second model. For another example, when the operator uses a camera-based application, the operator's head turning behavior is detected, and based on the method provided in this application, the relevant window (i.e., the above-mentioned target window) is promptly migrated to a device with an available camera, then this event is used as reward feedback to strengthen the second model.

[0156] In the above embodiment, by receiving the user's adjustment information on the display parameters of the target window, using the second model to automatically determine the second parameters that match the operator's interaction preferences, and using the second parameters to update the display parameters of the target window, the operator can be helped to quickly adjust the target window to display parameters that are in line with the operating habits, thereby improving the operator's interactive experience.

[0157] Next, combine Figure 3, describes the implementation process of an embodiment provided by this application. Among them, the multiple devices used to perform cross-device display are a host and multiple extended display screens connected to the host, and the host includes a main display screen; the target task is a face recognition task; the target sensor is a camera. Figure 3 As shown, this embodiment includes the following steps S31 to S310:

[0158] Step S31, identifying the screen position, size, and whether the main display screen and the plurality of extended display screens have cameras; thereafter, executing step S32;

[0159] Step S32, obtaining the start information of the face recognition task; then, executing step S33;

[0160] Here, the task based on video recording is face recognition task.

[0161] Step S33: Display the target window corresponding to the face recognition task on the first extended display screen; then, execute step S34;

[0162] Step S34, determining whether the first extended display screen is equipped with a camera; if so, executing step S35; if not, executing step S36;

[0163] Step S35, keeping the target window on the first extended display screen;

[0164] Step S36, adjusting the target window to the second extended display screen; the second extended display screen is an extended display screen equipped with a camera; thereafter, executing step S37;

[0165] Step S37, adjusting the window size of the target window in the second extended display screen; thereafter, executing step S38;

[0166] Here, the window size of the target window in the second extended display screen is calculated based on the ratio of the resolution of the first extended display screen to the resolution of the second extended display screen and the size of the target window in the first extended display screen, so that the window sizes of the target window before and after the migration are the same or similar.

[0167] Step S38, receiving the operator's adjustment information for the target window in the second extended display screen; then, executing step S39;

[0168] Here, the operator's adjustment information for the target window may be adjustment information for the window size, font size, font style, etc. of the target window.

[0169] Step S39, based on the received adjustment information and the operator's interaction preference, using the model to determine the optimized display parameters; thereafter, executing step S310;

[0170] Step S310: updating the target window using the optimized display parameters.

[0171] As can be seen from the above, the method provided by this application can automatically adjust the target window corresponding to the camera-based task to the display screen equipped with an available camera, that is, realize the window switching display according to the task execution capabilities of different devices. In this way, on the one hand, it can guide the operator to pay attention to the switched device (that is, the second extended display screen), provide accurate data for the execution of the task, and ensure the efficient completion of the task; on the other hand, it allows the operator to adjust his own behavior or state based on the target window displayed in the switched device, thereby improving the operator's experience of using multi-display devices; on the other hand, based on user feedback, the display parameters of the application window are optimized and the model is improved, which can continuously improve the model's adaptability to the user's interaction preferences and further enhance the user's interaction experience.

[0172] Based on the above embodiments, the present application provides a display system. Figure 4 As shown, the display system 400 includes a first device 410 and a second device 420;

[0173] a processor, the processor running on the first device 410 or the second device 420;

[0174] The processor is configured to determine the first device 410 in response to the target instruction; the target instruction is configured to display a target window on the first device 410, wherein the content displayed in the target window is configured to indicate the completion status of the target task;

[0175] The second device 420 is configured to display the target window when the first device does not meet the target condition.

[0176] In some embodiments, the first device 410 and the second device 420 may be terminal devices capable of operating independently. For example, the first device 410 and the second device 420 may be various types of terminal devices, such as laptop computers, tablet computers, desktop computers, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and the like.

[0177] In some embodiments, the first device and the second device may be a plurality of extended display screens or a main display screen in a system consisting of a host and a plurality of extended display screens.

[0178] In some embodiments, the processor may be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).

[0179] In some embodiments, the processor is configured to:

[0180] Obtaining a first input; the first input is for the first device 410 or the second device 420, and is used to trigger a target application to perform the target task;

[0181] When it is determined that the task execution window corresponding to the first input is displayed on the first device 410 , the target instruction is generated.

[0182] In some embodiments, the processor is configured to:

[0183] In response to completion of the target task, displaying other windows corresponding to the target application except the target window on the first device 410;

[0184] Alternatively, while the second device 420 is displaying the target window, the first device 410 displays other windows corresponding to the target application except the target window.

[0185] In some embodiments, the target task representation utilizes a target sensor to perform object recognition, and the processor is further configured to perform at least one of the following:

[0186] In response to the first device 410 not including the target sensor, determining that the first device 410 does not meet the target condition;

[0187] In response to the target sensor in the first device 410 being in a fault state or an occupied state, it is determined that the first device 410 does not meet the target condition.

[0188] In some implementations, the processor is configured to display the target window on the first device in response to the first device satisfying the target condition.

[0189] In some embodiments, the processor is configured to:

[0190] Determining at least one third device that meets the target condition; wherein each of the third devices can be displayed across devices with the first device;

[0191] A third device that includes the target sensor and can be called is determined as the second device.

[0192] In some embodiments, the third device including the target sensor and capable of being called by the target sensor is a plurality of third devices;

[0193] The processor is configured to perform at least one of the following:

[0194] Obtaining a current facial orientation of the operator, and determining, as the second device, a fourth device among the plurality of third devices whose display screen orientation has the smallest angle with the current facial orientation; wherein the operator is the object operating the plurality of devices for executing the cross-device display;

[0195] determining a fifth device among the plurality of third devices that matches the operator's interaction preference within a specified time period as the second device; wherein the operator is an object operating the plurality of devices for executing the cross-device display;

[0196] In response to the plurality of third devices including a sixth device for running the target application, the sixth device is determined as the second device.

[0197] In some embodiments, the processor is configured to perform at least one of the following:

[0198] Determining physical locations of the plurality of third devices using the first model, and determining the fourth device using the physical locations of the plurality of third devices;

[0199] The second model is used to determine the interaction preference of the operator based on the interaction information of the operator within a specified period of time, and the fifth device is determined using the interaction preference.

[0200] In some embodiments, the processor is configured to adjust the display parameter of the target window from a first parameter to a second parameter in response to the operator adjusting the display parameter of the target window;

[0201] The first parameter represents the display parameter before the adjustment operation; the second parameter represents the display parameter determined by the second model based on the adjustment operation and the interaction preference.

[0202] The description of the above system embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. In some embodiments, the functions or devices included in the system provided in the embodiments of the present application can be used to perform the methods described in the above method embodiments. For technical details not disclosed in the system embodiments of the present application, please refer to the description of the method embodiments of the present application for understanding.

[0203] It should be noted that, in the embodiment of the present application, if the above-mentioned display method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software and firmware.

[0204] An embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0205] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above method. The computer-readable storage medium may be transient or non-transient.

[0206] An embodiment of the present application provides a computer program, including computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0207] An embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, implements some or all of the steps in the above method. The computer program product can be implemented specifically by hardware, software, or a combination thereof. In some embodiments, the computer program product is embodied as a computer storage medium. In other embodiments, the computer program product is embodied as a software product, such as a software development kit (SDK), etc.

[0208] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between the various embodiments, and their similarities or similarities can be referenced to each other. The descriptions of the above device, storage medium, computer program, and computer program product embodiments are similar to the descriptions of the above method embodiments and have similar beneficial effects as the method embodiments. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the description of the method embodiments of this application for understanding.

[0209] It should be noted that Figure 5 This is a hardware entity diagram of the electronic device in this application, such as Figure 5 As shown, the hardware entity of the electronic device 500 includes: a processor 501, a communication interface 502 and a memory 503, wherein:

[0210] The processor 501 generally controls the overall operations of the electronic device 500 .

[0211] The communication interface 502 enables the electronic device to communicate with other terminals or servers through a network.

[0212] The memory 503 is configured to store instructions and applications executable by the processor 501, and can also cache data to be processed or processed by the processor 501 and various modules in the electronic device 500 (for example, image data, audio data, voice communication data, and video communication data). This can be implemented using flash memory (FLASH) or random access memory (RAM). Data can be transmitted between the processor 501, the communication interface 502, and the memory 503 via a bus 504.

[0213] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned steps / processes does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0214] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0215] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0216] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0217] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0218] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0219] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0220] The above is only an implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A display method, comprising: In response to the target instruction, determining a first device; The target instruction is used to display a target window on the first device, and the content displayed in the target window is used to indicate the completion status of the target task; If the first device does not meet the target condition, determining a second device that meets the target condition; the first device and the second device can be displayed across devices, and the target condition indicates that the target task can be performed; The target window is displayed on the second device.

2. The method according to claim 1, before determining the first device in response to the target instruction, further comprising: Get the first input; The first input is for the first device or the second device, and is used to trigger a target application to execute the target task; The target instruction is generated when it is determined that a task execution window corresponding to the first input is displayed on the first device.

3. The method according to claim 2, further comprising: In response to completion of the target task, displaying other windows corresponding to the target application except the target window on the first device; Alternatively, while the second device is displaying the target window, other windows corresponding to the target application except the target window are displayed on the first device.

4. The method according to claim 2, wherein the target task representation utilizes a target sensor to perform object recognition, and the method further comprises at least one of the following: In response to the first device not including the target sensor, determining that the first device does not meet the target condition; In response to the target sensor in the first device being in a fault state or an occupied state, it is determined that the first device does not meet the target condition.

5. The method according to claim 1, after determining the first device in response to the target instruction, further comprising: In response to the first device satisfying the target condition, the target window is displayed on the first device.

6. The method according to claim 4, wherein determining the second device that meets the target condition comprises: Determining at least one third device that meets the target condition; wherein each of the third devices can be displayed across devices with the first device; A third device that includes the target sensor and can be called is determined as the second device.

7. The method according to claim 6, wherein: The third device including the target sensor and capable of being called is a plurality of third devices; The method comprises at least one of the following: Obtaining a current facial orientation of the operator, and determining, as the second device, a fourth device among the plurality of third devices whose display screen orientation has the smallest angle with the current facial orientation; wherein the operator is the object operating the plurality of devices for executing the cross-device display; determining a fifth device among the plurality of third devices that matches the operator's interaction preference within a specified time period as the second device; wherein the operator is an object operating the plurality of devices for executing the cross-device display; In response to the plurality of third devices including a sixth device for running the target application, the sixth device is determined as the second device.

8. The method according to claim 7, wherein: Determining, as the second device, a fourth device having a display screen orientation with the smallest angle between the orientation of the current face and the orientation of the second device, comprising: determining physical positions of the plurality of third devices using a first model, and determining the fourth device using the physical positions of the plurality of third devices; The method of determining the fifth device among the multiple third devices that matches the operator's interaction preference within a specified time period as the second device includes: using a second model to determine the operator's interaction preference based on the operator's interaction information within the specified time period, and using the interaction preference to determine the fifth device.

9. The method according to claim 8, further comprising: In response to the operator adjusting the display parameter of the target window, adjusting the display parameter of the target window from a first parameter to a second parameter; The first parameter represents the display parameter before the adjustment operation; the second parameter represents the display parameter determined by the second model based on the adjustment operation and the interaction preference.

10. A display system comprising: a first device and a second device; a processor, wherein the processor runs on the first device or the second device; The processor is configured to determine a first device in response to a target instruction; the target instruction is configured to display a target window on the first device, wherein content displayed in the target window is configured to indicate a completion status of a target task; The second device is configured to display the target window when the first device does not meet the target condition.

Images