User interface display system, user interface display method, computer device and storage medium

Abstract

A user interface display system, a user interface display method, a computer device, and a storage medium are provided and belong to the field of display technology. The user interface display system comprises an image acquisition module, a control module, and a display module; the image acquisition module is configured to capture a scene image; the control module is configured to detect whether a target object is present in the scene image and, if present, to track the target object and determine its current position in the scene; and to determine a current coordinate point of the current position, which is mapped to an interface coordinate system of the user interface, according to the target object's current position in the scene and a mapping relationship between a previously established scene coordinate system and the interface coordinate system.and generate update information from at least one interface component in the user interface according to the current coordinate point and predefined auxiliary parameter information; and the display module is configured to display the at least one interface component according to the update information.

Description

AREA OF TECHNOLOGY

[0001] The present disclosure relates to the field of display technology and in particular to a user interface display system, a user interface display method, a computer device and a storage medium. STATE OF THE ART

[0002] With the updating and optimization of a user interface (UI) in a display device, more and more UI interfaces are being combined with human-computer interaction to create a 3D UI interface to meet the immersive experience of a viewer.

[0003] However, a conventional 3D UI interface display method requires the development of a separate application (app) to create a 3D scene in order to enhance the viewer's visual panorama experience. A significant amount of hardware resources are needed to generate the 3D scene for this separate app, resulting in poor system compatibility when other apps are called and impacting the use of other software functions. SUMMARY

[0004] The present disclosure aims to solve at least one of the technical problems in the related technology and provides a user interface display system, a user interface display method, a computer device and a storage medium.

[0005] In a first aspect, a technical solution adopted to solve the technical problem of the present disclosure is a user interface display system comprising an image acquisition module, a control module, and a display module; wherein the image acquisition module is configured to capture a scene image; wherein the control module is configured to detect whether a target object is present in the scene image and to track the target object and, if the target object is present, to determine a current position of the target object in a scene; to determine a current coordinate point of the current position, which is mapped to an interface coordinate system of a user interface, according to the current position of the target object in the scene and a mapping relationship between a previously established scene coordinate system and the interface coordinate system;and generate update information from at least one interface component in the user interface according to the current coordinate point and predefined auxiliary parameter information; and wherein the display module is configured to display the interface components according to the update information.

[0006] In some embodiments, the control module comprises a detection unit, an information determination unit, and an update information generation unit; wherein the detection unit is configured to detect whether the target object is present in the scene image; wherein the information determination unit is configured to track the target object and, if the target object is present, to determine its current position in the scene; and to determine the current coordinate point of the current position, which is mapped to the interface coordinate system, according to the target object's current position in the scene and a mapping relationship between the previously established scene coordinate system and the interface coordinate system;and wherein the update information generation unit is configured to generate the update information of the interface components in the user interface according to the current coordinate point and the predefined auxiliary parameter information.

[0007] In some embodiments, the predefined auxiliary parameter information includes a predefined auxiliary location point on the interface coordinate system and a dimensional scaling factor for each interface component; wherein the target object comprises a person; wherein the detection unit is configured to perform facial recognition on the scene image and to determine whether the target object is present; the current position includes a target key point; the information determination unit is configured to track the target object and determine a target key point in the face according to a facial feature of the target object in a case where facial recognition determines that the target object is present;and, according to the target key point and a mapping relationship between the scene coordinate system and the interface coordinate system, to determine a current coordinate point that corresponds to the target key point; and the update information generation unit is configured to generate the update information of each interface component according to the current coordinate point, the auxiliary location point, and the dimension scaling factor of each interface component.

[0008] In some embodiments, the system control module further comprises a first preprocessing unit; and the first preprocessing unit is configured to determine an auxiliary location point on the interface coordinate system according to an initial coordinate point mapped to the interface coordinate system, an initial position of the tracked target object, and an initial position of each interface component in the user interface; and to determine the dimensional scaling factor of each interface component according to the auxiliary location point, the initial coordinate point, and the initial position of each interface component.

[0009] In some embodiments, the user interface comprises layers that multiplex the same interface coordinate system; the user interface comprising the layers includes a foreground layer and a background layer and at least one intermediate layer between the foreground layer and the background layer; and includes a plurality of interface components; and the system control module further includes an interface control unit; the interface control unit is configured to control the display module to display various interface components in the foreground layer, the intermediate layer, and the background layer.

[0010] In some embodiments, the update information generation unit is configured to receive a component selection instruction to select an interface component, replace a dimension scaling factor of the currently selected interface component with a target scaling factor, and generate the update information of each interface component according to the current coordinate point, the replaced target scaling factor, and the predefined auxiliary location point on the interface coordinate system.

[0011] In some embodiments, the update information generation unit is configured to receive a component selection instruction to select an interface component and obtain a predefined dimensional scaling factor for each interface component; to replace the dimensional scaling factor of the currently selected interface component with the dimensional scaling factor of the interface component in the foreground layer in a case where the currently selected interface component is not in the foreground layer of the user interface; and to generate the update information for each interface component according to the current coordinate point, the replaced dimensional scaling factor, and the predefined auxiliary location point on the interface coordinate system.

[0012] In some embodiments, the current position comprises a target key point and a depth reference point; wherein the information determination unit is configured to track the target object and determine a target key point in the face according to a facial feature of the target object in a case where facial recognition determines that the target object is present; to identify a shoulder feature of the target object, and to take a point of a mean position of two shoulders as a depth reference point according to the positions of the two shoulders in the scene coordinate system; and wherein the update information generation unit is configured to determine a component scaling factor from the depth reference point and the target key point;and to generate the update information of each interface component according to the component scaling factor, the current coordinate point, the predefined auxiliary location point on the interface coordinate system, and the dimension scaling factor of each interface component.

[0013] In some embodiments, the information determination unit is configured to identify a pupil feature in the face and take a point at an intermediate position of two pupils as the target key point according to the positions of the two pupils in the scene coordinate system, in a case where facial recognition determines that the target object is present.

[0014] In some embodiments, the user interface comprises at least one layer, including the background layer; the interface component comprises the background image; and the system control module further comprises an interface control unit configured to control the display module so that the background image is displayed in the background layer.

[0015] In some embodiments, the predefined auxiliary parameter information further includes predefined size information of a user interface viewport, a predefined magnification factor of the background image, and size information of the background image; the target object comprises a person; the detection unit is configured to perform facial recognition on the scene image and assess whether the target object is present; the current position includes a target key point; the information determination unit is configured to track the target object and determine a target key point in the face according to a facial feature of the target object in a case where facial recognition determines that the target object is present;and to determine a current coordinate point corresponding to the target key point, according to the target key point and a mapping relationship between the scene coordinate system and the interface coordinate system; and the update information generation unit is further configured to determine an auxiliary reference point at which the current coordinate point is centrosymmetric with respect to a geometric center of the viewport, according to the current coordinate point and the viewport size information; and to generate the background image update information according to the viewport size information, the background image magnification factor, the auxiliary reference point, and the background image size information.

[0016] In some embodiments, the control module further includes a second preprocessing unit; and the second preprocessing unit is configured to determine the size information of the background image according to the size information of the viewing window and the magnification factor of the background image.

[0017] In some embodiments, the interface component includes at least one of the following elements: a control, a status bar, a map bar, a background image.

[0018] In a second aspect, an embodiment of the present disclosure further provides a user interface display method comprising: capturing a scene image and detecting whether a target object is present in the scene image; tracking the target object and determining a current position of the target object in a scene if the target object is present; determining a current coordinate point of the current position, which is mapped to an interface coordinate system of a user interface, according to the current position of the target object in the scene and a mapping relationship between a previously established scene coordinate system and the interface coordinate system; generating update information from each interface component in the user interface according to the current coordinate point and predefined auxiliary parameter information;and displaying each interface component according to the update information.;

[0019] In a third aspect, an embodiment of the present disclosure further provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions that can be executed by the processor; the processor and the memory communicate with each other via the bus when the computer device is running; the machine-readable instructions, when executed by the processor, perform the steps of the user interface display procedure in the embodiment above.

[0020] In a fourth aspect, an embodiment of the present disclosure further provides a non-transitory computer-readable storage medium with a computer program stored thereon, wherein the computer program, when executed by a processor, performs the steps of the user interface display method in the embodiment above. BRIEF DESCRIPTION OF THE FIGURES Fig. Figure 1 is a schematic diagram of the structure of a user interface display system according to an embodiment of the present disclosure; Fig. Figure 2 is a schematic diagram showing the trajectory tracking according to one embodiment of the present disclosure; Fig. 3a is a schematic diagram showing a positional relationship between auxiliary location points and interface components on an interface coordinate system according to an embodiment of the present disclosure; Fig. 3b is a schematic diagram showing a positional relationship between a current coordinate point and interface components in an interface coordinate system according to an embodiment of the present disclosure; Fig. 4a is a schematic front view of a user interface according to an embodiment of the present disclosure; Fig. Figure 4b is a schematic diagram of a layered user interface according to an embodiment of the present disclosure; Fig. 5a is a schematic diagram of interface components without updating according to an embodiment of the present disclosure; Fig. 5b is a schematic diagram of the updated interface components of Fig. 5a; Fig. Figure 6 is a schematic diagram showing a comparison between two different distances from a target object to a user interface according to an embodiment of the present disclosure; Fig. Figure 7 is a schematic diagram showing a positional relationship between a background image and a vertex of a viewing window on an interface coordinate system according to an embodiment of the present disclosure; Fig. Figure 8 is a flowchart of a user interface display method according to an embodiment of the present disclosure; and Fig. Figure 9 is a schematic diagram of the structure of a computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE REVELATION

[0021] To clarify the objects, technical solutions, and advantages of the embodiments of this disclosure, the technical solutions in the embodiments of this disclosure are described clearly and completely with reference to the drawings in the embodiments of this disclosure. It is evident that the described embodiments represent only a part, not all, of the embodiments of this disclosure. In general, the components of the embodiments of this disclosure, as described and illustrated in the drawings, can be arranged and designed in a variety of different configurations.The detailed description of the embodiments of the present disclosure shown in the drawings is therefore not intended to limit the scope of the claimed present disclosure, but is merely representative of selected embodiments of the present disclosure. All other embodiments that can be derived by a person skilled in the art from the embodiments of the present disclosure without creative effort fall within the scope of protection of the present disclosure.

[0022] Unless otherwise defined, the technical or scientific terms used herein have the usual meanings as understood by a person skilled in the art in the field to which this disclosure belongs. The terms "first," "second," and the like, used in this disclosure, are not intended to indicate any order, quantity, or importance, but rather serve to distinguish one element from another. Furthermore, the terms "a," "a," "the," or the like, used herein, do not signify any limitation of quantity, but denote the presence of at least one element. The terms "comprising," "including," or similar, mean that the element or object preceding the term encompasses the element or object following the term and its equivalent, but do not exclude other elements or objects. The terms "connected," "coupled," or similar,This is not limited to physical or mechanical connections, but can also include electrical connections, regardless of whether they are direct or indirect. The terms "above," "below," "left," "right," and the like are used only to indicate relative positional relationships, and if the absolute position of a described object changes, the relative positional relationships can also change accordingly.

[0023] The reference to “several or some” in this revelation means two or more. The term “and / or” describes a connection between objects that are related to one another, including three-way relationships. For example, A and / or B can stand for: A alone, or A and B, or B alone. The sign “ / ” generally indicates that the object before the sign “ / ” and the object after the sign “ / ” are in an “or” relationship to each other.

[0024] In the related prior art, a conventional 3D UI interface rendering method requires developing an app to create a 3D scene to enhance the viewer's visual panorama experience. A significant amount of hardware resources are needed to generate the 3D scene for the standalone app, resulting in poor system compatibility when other apps are called and impacting the use of other software functions.

[0025] Based on this, the embodiment of the present disclosure provides a user interface display system comprising an image acquisition module, a control module, and a display module; wherein the image acquisition module is configured to capture a scene image; wherein the control module is configured to detect whether a target object is present in the scene image and to track the target object and, if the target object is present, to determine a current position of the target object in a scene; to determine a current coordinate point of the current position, which is mapped to an interface coordinate system of a user interface, according to the current position of the target object in the scene and a mapping relationship between a previously established scene coordinate system and the interface coordinate system;and generate update information from at least one interface component in the user interface according to the current coordinate point and predefined auxiliary parameter information; and wherein the display module is configured to display the interface components according to the update information.

[0026] In one embodiment of the present disclosure, the image acquisition module integrated into the system captures the scene image of the environment in which the system is located. Furthermore, the system control module detects whether the target object is present in the scene image. If present, the target object is tracked in real time to determine its current position in the scene. The current coordinate point, which is displayed on the user interface, is determined according to the current position. The update information for the interface components in the user interface is generated using the current coordinate point and predefined auxiliary parameter information.Movement and / or zoom positions of the interface components in the user interface are calculated using a spatial perspective principle; the display module can show the interface components according to update information, such as movement of the interface components with the target object, consistent scaling, etc. A relatively realistic 3D scene experience can be simulated in 2D display mode using the parallax principle, and the interactive 3D effect is simulated in 2D display mode. Compared to a case where application software is developed independently to realize a 3D scene, in the present embodiment the functional modules are unified within the user interface display system, thus saving hardware resources and achieving better compatibility of the user interface display with other applications.Furthermore, the relatively realistic 3D scene is simulated using the spatial perspective principle, thus simplifying the development process of the 3D scene software and reducing software maintenance and iteration costs.

[0027] The user interface display system in the embodiment of the present disclosure can be applied to a terminal system with a large screen. The "large screen" can be understood as a screen with a large screen size and a high resolution, such as an ultra-high-definition screen with a screen size of more than 40 inches and a resolution of 8K.

[0028] Functional modules of a user interface display system according to an embodiment of the present disclosure are described in detail below.

[0029] Fig. Figure 1 is a schematic diagram of the structure of a user interface display system according to an embodiment of the present disclosure. As in Fig. As shown in Figure 1, the user interface display system comprises an image acquisition module 101, a system control module 102 and a display module 103.

[0030] The image capture module 101 is configured to capture a scene image. The image capture module 101 can be a camera integrated into the user interface display system and is configured to capture a scene image of the environment in which the camera is located. The camera continuously captures the scene image, and the scene image is updated in real time. A dynamic target object, a still image, or similar elements can appear in the scene image for a certain period of time.

[0031] The target object in the embodiment of the present disclosure can be a preset object with specific features, such as a person or a specific selected object or the like.

[0032] The control module 102 can be a processor of the user interface display system and is configured to process the captured scene image to obtain user interface update information, i.e., update information for each interface component within the user interface. The control module 102 comprises a detection unit 121, an information determination unit 122, and an update information generation unit 123.

[0033] Detection Unit 121 is configured to detect whether a target object is present in the scene image. Specifically, Detection Unit 121 is configured to perform an image feature extraction on the scene image, process the extracted feature, and determine whether the target object is present. If the target object is present, its current position in the scene is determined; that is, a coordinate point of the target object in a scene coordinate system is established. The scene coordinate system can be a pre-defined two-dimensional coordinate system of the scene image captured by the camera. Alternatively, the scene coordinate system can be a spatial coordinate system of the environment in which the camera is located.Optionally, to simplify the algorithm and save hardware resources, the scene coordinate system in the embodiment of the present disclosure adopts a two-dimensional coordinate system which is easier to establish, and therefore the determined current position of the target object in the scene is a two-dimensional coordinate point in the scene coordinate system.

[0034] The information determination unit 122 is configured to track the target object and determine its current position within the scene when the target object is present. That is, if the presence of the target object is detected, it will be tracked. It can be assumed that the camera is configured to capture the scene image in real time and position the target object's two-dimensional coordinate point within the scene coordinate system in real time to enable tracking of the target trajectory.

[0035] Fig. Figure 2 is a schematic diagram illustrating trajectory tracking according to one embodiment of the present disclosure. As in Fig. As shown in Figure 2, a previous position H1 (hx1, hy1) lies within the range (0,0) to (cx, cy), where cx is a maximum value of the abscissa of the scene image in a scene coordinate system and cy is a maximum value of the ordinate of the scene image in the scene coordinate system. When an observer moves, a starting point is H1 (hx1, hy1), and an endpoint is H2 (hx2, hy2), i.e., the current position H2. At this point, a displacement occurs: hxm = hx2 - hx1, and hym = hy2 - hy1. The endpoint here is a position point in the tracking process and not a starting position point of the target object.

[0036] The information determination unit 122 is configured to determine a current coordinate point of the current position, which is mapped to an interface coordinate system of a user interface, according to the current position of the target object in the scene and a mapping relationship between a previously established scene coordinate system and the interface coordinate system. A memory unit in the system control module 102 stores the mapping relationship between the specified scene coordinate system and the interface coordinate system of the user interface.The information determination unit 122 is specifically configured to map a two-dimensional coordinate point of the current position in the scene coordinate system to the interface coordinate system according to the mapping relationship between the scene coordinate system and the interface coordinate system, after the current position of the target object in the scene has been determined, and to determine the current coordinate point that is mapped to the interface coordinate system.

[0037] Continuing from Fig. In Figure 2, a screen's viewing window has a maximum abscissa of xm and a maximum ordinate of ym. The information determination unit 122 is configured to determine an offset amount (vxm, vym) of the movement according to the displacement hxm = hx2-hx1 and hym = hy2-hy1 of the target object, and to determine the current coordinate point V(vx, vy) according to the offset amount (vxm, vym) and the previous coordinate point. Here, vxm / xm = hxm / cx, vym / ym = hym / cy.

[0038] The update information generation unit 123 is configured to generate update information for the interface components in the user interface based on the current coordinate point and predefined auxiliary parameter information. The current coordinate point can be understood as a reference point that sets the interface components in motion within the user interface. The auxiliary parameter information consists of predefined parameter information that serves to support the current coordinate point in updating the states of the interface components and is used to represent the association relationship between the current coordinate point and each interface component in the user interface.The details of the auxiliary parameter information may refer to the following description of a preprocessing unit and are therefore not explained further here. For example, the update information of the interface components may include motion information and / or scaling information of the interface components, where the motion information may include, for example, a motion trajectory or an end coordinate point, and the like; the scaling information may include, for example, a scaling ratio, a scaling reference point, and the like. The motion trajectory, the end coordinate point, the scaling ratio, and the scaling reference point are all pieces of information contained within the interface coordinate system.

[0039] The display module 103 is configured to show the interface components according to the update information. For example, the interface components are displayed with updated positions. The position update can be a movement and / or zoom of the interface components within the viewing window, or similar.

[0040] In the units above, real-time interaction between each interface component in the user interface and the target object is realized by constructing the association relationship between the current position of the target object and the current coordinate point of the user interface, and by using the auxiliary parameter information (the association relationship between the current coordinate point and each interface component in the user interface), the 3D interaction effect is simulated through the 2D display mode and the user's interaction experience is improved.

[0041] In some embodiments, the target object comprises a person. The detection unit 121 is specifically configured to identify at least some of the features of the human body and to assess whether the target object is present. In particular, some features can be selected that are capable of representing a higher degree of recognizability of the person, such as a facial feature (including a feature of five sense organs), a head feature, a torso feature, or the like.

[0042] Optionally, the detection unit 121 is specifically configured to perform facial recognition on the scene image and to determine whether the target object is present. Specifically, facial features in the scene image are identified, and it is determined whether an image feature matching the facial features is present. If the image feature matching the facial features is present, it is determined that the target object is present; otherwise, it is determined that the target object is not present.

[0043] The current position includes a target key point. The Information Determination Unit 122 is specifically configured to track the target object and determine a target key point on the face according to a facial feature of the target object in cases where facial recognition determines that the target object is present. For example, when the presence of the target object is determined, the two-dimensional coordinate point of the target object is positioned in real time within the scene coordinate system, thus achieving target trajectory tracking. Meanwhile, the target key point on the human face is determined according to the facial features of the target object, such as the features of the five senses, specific facial bones, or similar features.The target key point can be, for example, a two-dimensional coordinate point of the five detected sensory organs in the scene coordinate system, a two-dimensional coordinate point of a bone key point in the scene coordinate system, or something similar.

[0044] The information determination unit 122 is specifically configured to determine a current coordinate point corresponding to the target key point, according to the target key point and a mapping relationship between the scene coordinate system and the interface coordinate system. For example, according to the mapping relationship between the scene coordinate system and the interface coordinate system, the target key point in the scene coordinate system is mapped to the interface coordinate system, and the mapped current coordinate point is determined.

[0045] The predefined auxiliary parameter information includes a predefined auxiliary location point on the interface coordinate system and a dimensional scaling factor for each interface component. The update information generation unit 123 is configured to generate update information for the interface components according to the current coordinate point, the auxiliary location point, and the dimensional scaling factor of each interface component.

[0046] Here, the auxiliary location point is a two-dimensional coordinate point on the interface coordinate system and is used to assist in locating the position of each interface component in the user interface. For example, auxiliary location point A' is used to assist in locating the position of interface component 31, and auxiliary location point B' is used to assist in locating the position of interface component 31 and / or interface component 32. The dimensional scaling factor of each interface component can be determined, at a minimum, based on an initial position of each interface component in the user interface and the auxiliary location point. The dimensional scaling factor of each interface component is used to assist in locating the position of each interface component.

[0047] In some embodiments, the interface component includes at least one of the following elements: a control, a status bar, a map bar, and a background image. The control can be a software entry in a user interface and is typically represented in the user interface as a software icon; the status bar can be used to display the status of a currently open window or software in the user interface, for example, the status bar might display the status of multiple open folders or the status of other software applications, or it might be a bar for requesting information, such as a "screenshot" bar, a "data" bar, or similar; the map bar can be a window in the user interface for displaying image or text information. The background image is, for example, a wallpaper in the user interface, etc.

[0048] In some embodiments, the system control module 102 further comprises a first preprocessing unit; the first preprocessing unit is configured to determine an auxiliary location point on the interface coordinate system according to an initial coordinate point of an initial position of the tracked target object, which is mapped onto the interface coordinate system, and an initial position of each interface component in the user interface. The dimensional scaling factor of each interface component is determined as a function of the auxiliary location point, the initial coordinate point, and the initial position of each interface component. The initial position is a position of the target object in the scene coordinate system when the detection unit 121 first detects the target object. A position of the initial position that is mapped onto the interface coordinate system is the initial coordinate point.The initial position of each interface component that is initialized in the user interface is known.

[0049] Fig. Figure 3a is a schematic diagram showing a positional relationship between auxiliary location points and interface components in an interface coordinate system according to an embodiment of the present disclosure. As in Fig. As shown in Figure 3a, the coordinates of the vertices of a viewport in the user interface are known, namely A(0, 0), B(0, ym), C(xm, 0), and D(xm, ym). The viewport here is a screen window of the user interface that can be viewed by the user. The initial coordinate point is V0. An initial position of interface component 31 (e.g., the map bar) comprises the coordinates of four vertices, designated M1, M2, M3, and M4, respectively; an initial position of interface component 32 (e.g., the status bar) comprises the coordinates of four vertices, designated F1, F2, F3, and F4, respectively. An auxiliary location point A' is used to assist in locating interface component 31. Specifically, the auxiliary location point A' is the intersection of an extension line of a line connecting the initial coordinate point V0 and point M1 with the line segment AB.An auxiliary location point B' is used to assist in locating interface components 31 and 32. It is the intersection of an extension line of a line connecting the initial coordinate point V0 and point F1 with line segment AB, and is also the intersection of an extension line of a line connecting the initial coordinate point V0 and point M3 with line segment AB. An auxiliary location point C' is used to assist in locating interface component 31. Specifically, auxiliary location point C' is the intersection of an extension line of a line connecting the initial coordinate point V0 and point M2 with line segment CD.An auxiliary location point D' is used to assist in locating the interface components 31 and 32 and is the intersection of an extension line of a line connecting the initial coordinate point V0 and point F2 with the line segment CD, and is the intersection of an extension line of a line connecting the initial coordinate point V0 and point M4 with the line segment CD. Simultaneously, V0, F3, and B lie on a common line, and V0, F4, and D lie on a common line.

[0050] A dimensional scaling factor of interface component 31 is a ratio of A'M1 to A'V0, also C'M2 to C'V0, also BM3 to BV0, also D'M4 to D'V0, all denoted as m. Here, m = B'M3 / BV0 = A'M1 / A'V0 = C'M2 / C'V0 = D'M4 / D'V0. A dimensional scaling factor of interface component 32 is a ratio of B'F1 to B'V0, also D'F2 to D'V0, also BF3 to BV0, also DF4 to DV0, all denoted as f. Here, f = B'F1 / B'V0 = D'F2 / D'V0 = BF3 / BV0 = DF4 / DV0.

[0051] The update information generation unit 123 is specifically configured to generate update information for each interface component according to the current coordinate point, the auxiliary location point, and the dimension scaling factor of each interface component. For example, Fig. 3b a schematic diagram representing a positional relationship between a current coordinate point and interface components in an interface coordinate system according to an embodiment of the present disclosure. As in Fig. As shown in Figure 3b, if the current coordinate point V(vx, vy), the auxiliary location point A'(0, a), the auxiliary location point B'(0, b), the auxiliary location point C'(xm, a), the auxiliary location point D'(xm, b), the dimension scaling factor m of interface component 31, and the dimension scaling factor f of interface component 32 are known, update information for interface component 31 (i.e., updated M1, M2, M3, and M4) and update information for interface component 32 (i.e., updated F1, F2, F3, and F4) are determined. The updated M1, M2, M3, and M4 are as follows: M1=[m×vx,a+m / (vy−a)] M2=[xm−m×(xm−vx),a−m×(a−vy)] M3=[m×vx,b−m×(b−vy)] M4=[xm−m×(xm−vx),b−m×(b−vy)]

[0052] The updated F1, F2, F3 and F4 are as follows: F1=[f×vx,b−f×(b−vy)] F2=[xm−f×(xm−vx),b−f×(b−vy)] F3=[f×vx,ym−f×(ym−vy)] F4=[xm−f×(xm−vx),ym−f×(ym−vy)]

[0053] Here, a, b, f and m are constants that can be determined according to the design requirements, which are not particularly restricted in the present embodiment.

[0054] It should be noted that at any given time, A (0, 0), B (0, ym), C (xm, 0), D (xm, ym), A' (0, a), B' (0, b), C' (xm, a), and D' (xm, b) are fixed, V, M1, and A' lie on the same straight line, V, M2, and C' lie on the same straight line, V, M3, and B' lie on the same line, V, M4, and D' lie on the same line, V, F1, and B' lie on the same line, V, F2, and D' lie on the same line, V, F3, and B lie on the same line, and V, F4, and D lie on the same line. Therefore, the positions of the interface components are constantly updated with the continuous change of the current coordinate point V.

[0055] Each of the above-mentioned units utilizes the association relationship between the target object and the current coordinate point, as well as the association relationship between the current coordinate point and each interface component, so that each interface component can follow the target object as it moves, thereby realizing a dynamic interaction between the target object and each interface component.

[0056] In some embodiments Fig. 4a a schematic front view of a user interface according to an embodiment of the present disclosure; Fig. Figure 4b is a schematic diagram of a layered user interface according to an embodiment of the present disclosure. As in Fig. 4a and Fig. As shown in Figure 4b, the user interface comprises layers that use the same interface coordinate system; the user interface with the layers includes a foreground layer 401 and a background layer 403 and at least one intermediate layer 402 located between the foreground layer 401 and the background layer 403; a variety of interface components are included, such as a status bar 41, a map bar 42 and a background image 43, as shown.

[0057] The system control module 102 further includes an interface control unit; the interface control unit is configured to control the display module 103 to display various interface components in the foreground layer, the intermediate layer, and the background layer. As in Fig. As shown in Figure 4b, the interface control unit is configured, for example, to control the display module 103 to display the status bar 41 in the foreground layer, the map bar 42 in the intermediate layer, and the background image 43 in the background layer. In the embodiment of the present disclosure, the user interface is layered, which creates the vertical feel of the image through the parallax principle and can represent a relatively realistic 3D interface display effect for the user.

[0058] In some embodiments, to further enhance the user interaction experience, the user interface display system can move the interface component selected in the intermediate layer to the foreground layer for display. Specifically, the currently selected interface component is zoomed into the display, thus highlighting the component currently selected by the user.

[0059] One possible implementation is to configure the update information generation unit 123 specifically to receive a component selection instruction to select an interface component and to capture a predefined dimensional scaling factor of each interface component; to replace the dimensional scaling factor of the currently selected interface component with the dimensional scaling factor of the interface component in the foreground layer in a case where the currently selected interface component is not in the foreground layer of the user interface; and to generate the update information of each interface component according to the current coordinate point, the replaced dimensional scaling factor, and the predefined auxiliary location point on the interface coordinate system.

[0060] For example, Fig. 5a a schematic diagram of interface components without updating according to an embodiment of the present disclosure; Fig. 5b is a schematic diagram of the updated interface components of Fig. 5a. As in Fig. 5a and Fig. As shown in Figure 5b, the component selection instruction can be an instruction for an interface component located in the intermediate layer and selected by the user. For example, if map bar 42 is selected and the dimensional scaling factor m of map bar 42 is known, the dimensional scaling factor m of map bar 42 is replaced by the dimensional scaling factor f of status bar 41. Meanwhile, the update information for map bar 42 and status bar 41 is generated according to the current coordinate point V (vx, vy), the replaced dimensional scaling factor, and the predefined auxiliary location points A' (0, a), B' (0, B), C' (xm, a), and D' (xm, B) on the interface coordinate system. The updated values ​​M1, M2, M3, and M4 are as follows: M1=[f×vx,a+f / (vy−a)] M2=[xm−f×(xm−vx),a−f×(a−vy)] M3=[f×vx,b−f×(b−vy)] M4=[xm−f×(xm−vx),b−f×(b−vy)]

[0061] The updated F1, F2, F3 and F4 are as follows: F1=[m×vx,b−m×(b−vy)] F2=[xm−m×(xm−vx),b−m×(b−vy)] F3=[m×vx,ym−m×(ym−vy)] F4=[xm−m×(xm−vx),ym−m×(ym−vy)]

[0062] Here, a, b, f and m are constants and can be set according to the design requirements, which are not particularly limited in the embodiment of the present disclosure.

[0063] Another possible embodiment is that the update information generation unit 123 is specifically configured to receive a component selection instruction to select an interface component, to replace a dimension scaling factor of the currently selected interface component with a target scaling factor, and to generate the update information of each interface component according to the current coordinate point, the replaced target scaling factor, and the predefined auxiliary location point on the interface coordinate system.

[0064] The target scaling factor can be empirically determined data. For example, the target scaling factor is chosen to be r. The component selection instruction can be an instruction for any interface component selected by the user; for example, status bar 41 is selected, the dimensional scaling factor f of status bar 41 is known, and the dimensional scaling factor f of status bar 41 is replaced by the target scaling factor r. Simultaneously, the update information for status bar 41 is generated according to the current coordinate point V (vx, vy), the replaced dimensional scaling factor r, and the predefined auxiliary location points B' (0, b) and D' (xm, b) on the interface coordinate system. The updated F1, F2, F3, and F4 are as follows: F1=[r×vx,b−r×(b−vy)] F2=[xm−r×(xm−vx),b−r×(b−vy)] F3=[r×vx,ym−r×(ym−vy)] F4=[xm−r×(xm−vx),ym−r×(ym−vy)]

[0065] Here, a, b and r are constants and can be determined according to the design requirements, which are not particularly limited in the embodiment of the present disclosure.

[0066] It should be noted that the status bar 41, obtained by using the replaced dimension scaling factor r, appears larger compared to the original status bar 41, obtained by using the non-replaced dimension scaling factor f.

[0067] In some embodiments, the current position includes a target key point and a depth reference point. The information determination unit 122 is specifically configured to track the target object and determine a target key point in the face according to a facial feature of the target object in a case where facial recognition determines that the target object is present. A shoulder feature of the target object is identified, and a point at the midpoint of two shoulders is taken as a depth reference point according to the positions of the two shoulders in the scene coordinate system.

[0068] For example, if the presence of the target object is detected, its two-dimensional coordinate point is positioned in real time within the scene coordinate system, thus enabling tracking of the target trajectory. Meanwhile, the target key point is determined based on the target object's facial features, such as the characteristics of the five senses, specific facial bones, or similar features. The target key point could, for instance, be a two-dimensional coordinate point of the five detected senses within the scene coordinate system, a two-dimensional coordinate point of a bone key point within the scene coordinate system, or something similar. Based on the shoulder feature, such as...For a shoulder bone feature, two-dimensional coordinate points of bone key points of two shoulders of the human body are determined in the scene coordinate system, and a central point of the two-dimensional coordinate points of the two shoulders (i.e., a point in the middle of the two shoulders) is determined and used as the depth distance reference point.

[0069] The update information generation unit 123 is specifically configured to determine a component scaling factor from the depth distance reference point and the target key point; and to generate the update information of each interface component according to the component scaling factor, the current coordinate point, the predefined auxiliary location point on the interface coordinate system, and the dimension scaling factor of each interface component.

[0070] Fig. Figure 6 is a schematic diagram showing a comparison between two different distances from a target object to a user interface according to an embodiment of the present disclosure. As in Fig. As shown in Figure 6, the target key point can select a central point 61 of two pupils of the target object, and a length z0 of the line connecting a depth-distance reference point 62 and a central point 61 of two pupils of a target object can be used as an initial scaling factor z0. An initial scaling factor of the target object at an earlier time is z0_1, and an initial scaling factor of the target object at a later time is z0_2. A component scaling factor z1 can be determined by using a preset algorithm according to the initial scaling factor z0. The preset algorithm can be determined based on a mapping relationship between the scene coordinate system and the interface coordinate system and the setting of an adjustment ratio parameter of some hardware devices (e.g., a camera, etc.).That is, the component scaling factor z1 can be determined by transforming the initial scaling factor z0 through the mapping relationship between the scene coordinate system and the interface coordinate system, and then adjusting the matching ratio parameter, where a specific matching ratio parameter is set according to an actual application scene and a selected device parameter, which is not particularly restricted in the embodiment of the present disclosure.

[0071] For example, if the component scaling factor z1_1 of the target object at the previous time, the current coordinate point V (vx, vy), the auxiliary location point A' (0, a), the auxiliary location point B' (0, b), the auxiliary location point C' (xm, a), the auxiliary location point D' (xm, b), (xm, a), the auxiliary location point C' (xm, a), the auxiliary location point D' (xm, b), the dimension scaling factor m of interface component 31, and the dimension scaling factor f of interface component 32 are known, the update information of interface component 31 (i.e., updated M1, M2, M3, and M4) and the update information of interface component 32 (i.e., updated F1, F2, F3, and F4) are determined. The updated M1, M2, M3, and M4 are as follows: M1=[z1_1×m×vx,a+z1_1×m×(vy−a)] M2=[xm−z1_1×m×(xm−vx),a−z1_1×m×(a−vy)] M3=[z1_1×m×vx,b−z1_1×m×(b−vy)] M4=[xm−z1_1×m×(xm−vx),b−z1_1×m×(b−vy)]

[0072] The updated die F1, F2, F3 and F4 also appear as follows: F1 = [z1_1×f×vx,b−z1_1×f×(b−vy)]. F2 = [xm−z1_1×f×(xm−vx),b−z1_1×f×(b−vy)]. F3=[z1_1×f×vx,ym−z1_1×f×(ym−vy)]. F4=[xm−z1_1×f×(xm−vx),ym−z1_1×f×(ym−vy)]

[0073] For example, if the component scaling factor z1_2 of the target object at the previous time, the current coordinate point V (vx, vy), the auxiliary location point A' (0, a), the auxiliary location point B' (0, b), the auxiliary location point C' (xm, a), the auxiliary location point D' (xm, b), (xm, a), the auxiliary location point C' (xm, a), the auxiliary location point D' (xm, b), the dimension scaling factor m of interface component 31, and the dimension scaling factor f of interface component 32 are known, the update information of interface component 31 (i.e., updated M1, M2, M3, and M4) and the update information of interface component 32 (i.e., updated F1, F2, F3, and F4) are determined. The updated M1, M2, M3, and M4 are as follows: M1=[z1_2×m×vx,a+z1_2×m×(vy−a)] M2=[xm−z1_2×m×(xm−vx),a_z1_2×m×(a−vy)] M3=[z1_2×m×vx,b−z1_2×m×(b−vy)] M4=[xm−z1_2×m×(xm−vx),b−z1_2×m×(b−vy)]

[0074] The updated F1, F2, F3 and F4 are as follows: F1=[z1_2×f×vx, b−z1_2×f×(b−vy)] F2=[xm−z1_2×f×(xm−vx),b−z1_2×f×(b−vy)] F3=[z1_2×f×vx, ym−z1_2×f×(ym−vy)] F4=[xm−z1_2×f×(xm−vx),ym−z1_2×f×(ym−vy)]

[0075] Here, a, b, f and m are constants and can be set according to the design requirements, which are not particularly limited in the embodiment of the present disclosure.

[0076] In this embodiment, the target key point is found using the face features, the depth distance reference point is found using the shoulder features, and the line connecting the target key point and the depth distance reference point is used as the reference condition for scaling the interface component to realize the scaling of the interface component, so that the relatively real 3D interaction effect is simulated in 2D display mode, the hardware resources of the system are saved, and the compatibility of the UI interface display and other applications can be better realized.

[0077] In some embodiments, the information determination unit 122 is specifically configured to identify a pupil feature in the face when facial recognition determines the presence of the target object. It then selects a point at the midpoint of the two pupils as the target key point, based on the positions of the two pupils in the scene coordinate system. A central point of the two pupils is chosen as the target key point to more accurately position the target object. Furthermore, the pupil feature has a higher degree of differentiation, thus simplifying facial recognition and positioning.

[0078] In some embodiments, such as in the Fig. 4a and Fig. As shown in Figure 4b, the user interface comprises at least one layer, including the background layer 403; the interface component includes the background image 43. The system control module 102 further comprises an interface control unit configured to control the display module 103 to display the background image in the background layer.

[0079] In some embodiments Fig. 7 A schematic diagram showing a positional relationship between a background image and a vertex of a viewing window on an interface coordinate system according to an embodiment of the present disclosure. As in Fig. As shown in Figure 7, the number 71 indicates the background image. To enhance the spatial impression, the background image 71 changes along with the target object. To ensure that no edge of the background image 71 is visible in the viewing window 72 during movement, a suitable size for the background image 71 must be set, which must necessarily be larger than the size of the viewing window.

[0080] The system control module 102 further includes a second preprocessing unit; the second preprocessing unit is configured to determine size information of the background image 71 according to a predefined magnification factor of the background image 71 and size information of the viewing window 72.

[0081] As in Fig. As shown in Figure 7, the size information of the viewing window 72 includes coordinate information of each vertex of the viewing window 72, and the coordinates of the vertices of the viewing window 72 are A (0, 0), B (0, ym), C (xm, 0), and D (xm, ym). The background image 71 has a magnification factor p, which can be adjusted according to the actual design requirements and is not particularly limited in the embodiment of the present disclosure. p = L1 + L2. The size information of the background image 71 includes a width Wx and a height Wy, and the size of the background image 71 is enlarged proportionally to the size of the viewing window 72, such that Wx / xm = Wy / ym is satisfied. The background image 71 has a width Wx = xm + p and a height Wy = (ym + ym / xm × p).The size information of the background image 71 further includes coordinate information for each vertex of the background image 71, and the coordinates of the vertices of the background image 71 are W1, W2, W3, and W4. The coordinates of the vertices of the background image 71 change with the update of the current coordinate points. In the initial state, a geometric center of the background image 71 coincides with a geometric center of the viewport 72.

[0082] In some embodiments, the target object comprises a person. The detection unit 121 is specifically configured to perform facial recognition on the scene image and to assess whether the target object is present. In particular, several features can be selected that are capable of representing a higher degree of intelligibility of the person, such as a facial feature (including a feature from five sense organs), a head feature, a torso feature, or the like. Optionally, the detection unit 121 is configured to perform facial recognition on the scene image and assess whether the target object is present. Specifically, the facial features in the scene image are identified, and it is assessed whether an image feature that fulfills the facial features is present or not.If the image feature that fulfills the facial features is present, it is determined that the target object is present; otherwise, it is determined that the target object is not present.

[0083] The current position includes a target key point. The Information Determination Unit 122 is specifically configured to track the target object and determine a target key point on the face according to a facial feature of the target object in cases where facial recognition detects the presence of the target object. For example, when the presence of the target object is detected, the two-dimensional coordinate point of the target object is positioned in real time within the scene coordinate system, thus achieving target trajectory tracking. Meanwhile, the target key point on the human face is determined according to the facial features of the target object, such as the features of the five senses, specific facial bones, or similar features.The target key point can be, for example, a two-dimensional coordinate point of the detected five sensory organs in the scene coordinate system, a two-dimensional coordinate point of a bone key point in the scene coordinate system, or something similar.

[0084] The information determination unit 122 is specifically configured to determine a current coordinate point corresponding to the target key point, according to the target key point and a mapping relationship between the scene coordinate system and the interface coordinate system. For example, according to the mapping relationship between the scene coordinate system and the interface coordinate system, the target key point in the scene coordinate system is mapped to the interface coordinate system, and the mapped current coordinate point is determined.

[0085] The predefined auxiliary parameter information further includes the size information of the viewport of the predefined user interface, the magnification factor, and the size information of the predefined background image. The update information generation unit 123 is further configured to determine an auxiliary reference point at which the current coordinate point is centrosymmetric with respect to the geometric center of the viewport, according to the current coordinate point and the viewport size information; and to generate the background image update information according to the viewport size information, the background image magnification factor, the auxiliary reference point, and the background image size information.

[0086] As in Fig. As shown in Figure 7, the auxiliary reference point V_mirror varies with the tracked current coordinate point V(vx, vy). At any given time, the auxiliary reference point V_mirror and the current coordinate point V(vx, vy) are centrosymmetric with respect to the geometric center of the viewport. A positional coordinate of the auxiliary reference point V_mirror in the interface coordinate system is (xm-vx, ym-vy). At any given time, point V_mirror, point A, and point W1 lie on the same straight line; point V_mirror, point C, and point W2 lie on the same straight line; point V_mirror, point B, and point W3 lie on the same straight line; and point V_mirror, point D, and point W4 lie on the same straight line. Therefore, the position of the background image is constantly updated with the constant change of the current coordinate point V. In particular, one coordinate of point W1 of the background image is [-p / xm×(xm-vx), -p / xm×(ym-vy)].

[0087] In this embodiment, the size of the background image is increased, and the position of the background image is updated along with the position of the target object, thus improving the spatial perception of the user interface and enhancing the user's interactive experience.

[0088] The user interface display system was implemented as described above.

[0089] The embodiment of the present disclosure further provides a user interface display method, wherein a main body performing the method can be an end device integrated with the user interface display system. The user interface display method can be applied to a large-screen end device. The "large screen" can be understood as a screen with a large screen size and high resolution, such as an ultra-high-definition screen with a screen size of more than 40 inches and a resolution of 8K.

[0090] Fig. Figure 8 is a flowchart of a user interface display method according to an embodiment of the present disclosure. As in Fig. As shown in section 8, the procedure comprises steps S11 to S15, in particular:

[0091] S11, Capturing a scene image and detecting whether a target object is present in the scene image.

[0092] It should be noted that for a specific implementation process of step S11, reference can be made to the above description of the image acquisition module 101 and the detection unit 121 in the system control module 102 in the user interface display system, and repeated descriptions are omitted.

[0093] S12, Tracking the target object and determining a current position of the target object in a scene in a case where the target object is present.

[0094] It should be noted that for a specific implementation process of step S12, reference can be made to the above description of the information determination unit 122 in the system control module 102, and repeated descriptions are omitted.

[0095] S13, Determining a current coordinate point of the current position, which is mapped to an interface coordinate system of a user interface, according to the current position of the target object in the scene and a mapping relationship between a previously established scene coordinate system and the interface coordinate system.

[0096] It should be noted that for a specific implementation process of step S13, reference can be made to the above description of the information determination unit 122 in the system control module 102, and repeated descriptions are omitted.

[0097] S14, Generating update information of interface components in the user interface according to the current coordinate point and predefined auxiliary parameter information.

[0098] It should be noted that for a specific implementation process of step S14, reference can be made to the above description of the update information generation unit 123 in the system control module 102, and repeated descriptions are omitted.

[0099] S15, Display of interface components according to the update information.

[0100] It should be noted that for a specific implementation process of step S15, reference can be made to the above description of display module 103, and repeated descriptions are omitted.

[0101] In some embodiments, the predefined auxiliary parameter information includes a predefined auxiliary location point on the interface coordinate system and a dimension scaling factor for each interface component.

[0102] For step S11, the target object specifically includes a person. Facial recognition is performed on the scene image, and it is assessed whether the target object is present or not. It should be noted that the process can refer to the specific implementation of the detection unit 121, and repetitive descriptions are omitted.

[0103] In step S12, the current position includes, in particular, a target key point. The target object is tracked, and a target key point on the face is determined according to a facial feature of the target object in a case where facial recognition establishes that the target object is present. It should be noted that for the process, reference can be made to the specific implementation of the information determination unit 122, and repetitive descriptions are omitted.

[0104] For step S13, a current coordinate point corresponding to the target key point is determined according to the target key point and a mapping relationship between the scene coordinate system and the interface coordinate system. It should be noted that the specific implementation of the information determination unit 122 can be referenced for this process, and repeated descriptions are omitted.

[0105] For step S14, update information for the interface components is generated according to the current coordinate point, the auxiliary location point, and the respective dimension scaling factors of the interface components. It should be noted that the specific implementation of the update information generation unit 123 can be referenced for this process, and repeated descriptions are omitted.

[0106] In some embodiments, the user interface display method further includes determining an auxiliary location point on the interface coordinate system according to an initial coordinate point mapped to the interface coordinate system, an initial position of the tracked target object, and an initial position of each interface component in the user interface; and determining the dimensional scaling factor of each interface component according to the auxiliary location point, the initial coordinate point, and the initial position of each interface component. It should be noted that reference can be made to the specific implementation of the first preprocessing unit for the process, and repetitive descriptions are omitted.

[0107] In some embodiments, the user interface comprises layers that multiplex the same interface coordinate system; the user interface with the layers includes a foreground layer and a background layer and at least one intermediate layer located between the foreground layer and the background layer; a plurality of interface components is included.

[0108] In particular, the user interface display procedure further includes displaying various interface components in the foreground layer, the intermediate layer, and the background layer. It should be noted that the process can refer to the specific implementation of the interface control unit, and repetitive descriptions are omitted.

[0109] In some embodiments, step S14 specifically receives a component selection instruction for selecting an interface component, replaces the dimensional scaling factor of the currently selected interface component with a target scaling factor, and generates the update information for each interface component according to the current coordinate point, the replaced target scaling factor, and the auxiliary location point on the predefined interface coordinate system. It should be noted that reference can be made to the specific implementation of the update information generation unit 123 for this process, and repetitive descriptions are omitted.

[0110] In some embodiments, step S14 in particular receives a component selection instruction to select an interface component, and a predefined dimensional scaling factor of each interface component is captured; the dimensional scaling factor of the currently selected interface component is replaced by the dimensional scaling factor of the interface component that is positioned in the foreground layer if the currently selected interface component is not positioned in the foreground layer of the user interface; and the update information of each interface component is generated according to the current coordinate point, the replaced dimensional scaling factor, and the predefined auxiliary location point on the interface coordinate system.It should be noted that the process can refer to the specific implementation of the Update Information Generation Unit 123, and repeated descriptions are omitted.

[0111] In some embodiments, particularly for step S12, the current position includes a target key point and a depth reference point. The information determination unit 122 is specifically configured to track the target object and determine a target key point in the face according to a facial feature of the target object in a case where facial recognition determines that the target object is present; to identify a shoulder feature of the target object and to take a point at the midpoint of two shoulders as a depth reference point according to the positions of the two shoulders in the scene coordinate system. It should be noted that reference can be made to the specific implementation of the information determination unit 122 for the process, and repetitive descriptions are omitted.

[0112] For step S12, a component scaling factor is determined from the depth distance reference point and the target key point; and the update information for each interface component is generated according to the component scaling factor, the current coordinate point, the predefined auxiliary location point on the interface coordinate system, and the dimensional scaling factor of each interface component. It should be noted that reference can be made to the specific implementation of the update information generation unit 123 for this process, and repetitive descriptions are omitted.

[0113] In some embodiments, in step S12, particularly in a case where facial recognition determines that the target object is present, a pupil feature in the face is identified, and a point at an intermediate position between two pupils is taken as a target key point according to the positions of the two pupils in the scene coordinate system. It should be noted that reference can be made to the specific implementation of the information determination unit 122 for the process, and repeated descriptions are omitted.

[0114] In some embodiments, the user interface comprises at least one layer, including the background layer; the interface component includes the background image.

[0115] In particular, the user interface display procedure further includes the control of the display module 103 to display a background image in the background layer. It should be noted that reference can be made to the specific implementation of the interface control unit for this process, and repeated descriptions are omitted.

[0116] In some embodiments, the predefined auxiliary parameter information also includes size information of a viewport of the predefined user interface, a magnification factor, and size information of the predefined background image.

[0117] For step S11, the target object specifically includes a person; and facial recognition is performed on the scene image to assess whether the target object is present or not. It should be noted that reference can be made to the specific implementation of the detection unit 121 for this process, and repetitive descriptions are omitted.

[0118] In step S12, the current position includes a target key point. The target object is tracked, and if facial recognition determines that the target object is present, a target key point in the face is determined according to a facial feature of the target object. It should be noted that the process can refer to the specific implementation of Information Determination Unit 122, and repetitive descriptions are omitted.

[0119] For step S13, a current coordinate point corresponding to the target key point is determined according to the target key point and a mapping relationship between the scene coordinate system and the interface coordinate system. It should be noted that the specific implementation of the information determination unit 122 can be referenced for this process, and repeated descriptions are omitted.

[0120] For step S14, the procedure further includes: determining an auxiliary reference point, where the current coordinate point is centrosymmetric with respect to the geometric center of the viewport, according to the current coordinate point and the viewport size information; and generating the background image update information according to the viewport size information, the background image magnification factor, the auxiliary reference point, and the background image size information. It should be noted that reference can be made to the specific implementation of the Update Information Generation Unit 123 for this process, and repetitive descriptions are omitted.

[0121] In some embodiments, the user interface display method further includes determining the size information of the background image based on the size information of the viewing window and the magnification factor of the background image. It should be noted that reference can be made to the specific implementation of the second preprocessing unit for the process, and repeated descriptions are omitted.

[0122] Fig. Figure 9 is a schematic diagram of the structure of a computer device according to an embodiment of the present disclosure. As in Fig.Figure 9 shows that an embodiment of the present disclosure provides a computer device comprising: one or more processors 901, a memory 902, and one or more I / O interfaces 903. The memory 902 stores one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the user interface display method in the embodiments described above; one or more I / O interfaces 903 are connected between the processors 901 and the memory 902 and are configured to enable information interaction between the processors 901 and the memory 902.

[0123] Each processor 901 is a device with data processing capabilities, including, but not limited to, a central processing unit (CPU) or the like; the memory 902 is a device with data storage capabilities, including, but not limited to, random access memory (RAM, such as SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EPROM), and a flash memory; each I / O interface (read / write interface) 903 is placed between the processors 901 and the memory 902 and can implement information interaction between the processors 901 and the memory 902 and includes, but is not limited to, a data bus or the like.

[0124] In some embodiments, the processors 901, the memory 902 and the I / O interfaces 903 are connected to each other and to other components of the computer device via the bus 904.

[0125] According to one embodiment of the present disclosure, a non-transitory, computer-readable storage medium is further provided. A computer program is stored on the non-transitory, computer-readable storage medium, wherein the computer program, when executed by a processor, implements the steps in the user interface display method as in one of the embodiments above.

[0126] In particular, the processes described above with reference to the flowchart can be implemented as computer software programs according to the embodiments of this disclosure. For example, embodiments of this disclosure include a computer program product, including a computer program embodied on a machine-readable medium, and the computer program includes program code for carrying out the procedure shown in the flowchart. In this embodiment, the computer program can be downloaded and installed from a network via a communication component and / or installed from a removable medium. The functions defined in the system of this disclosure are performed when the computer program is executed by the central processing unit (CPU).

[0127] It should be noted that the non-transient computer-readable storage medium in the present disclosure can be a computer-readable signaling medium, a computer-readable storage medium, or any combination of both. The computer-readable storage medium can be, for example, but not exclusively, an electronic, magnetic, optical, electromagnetic, infrared, or semiconducting system, apparatus, or device, or any combination thereof.More specific examples of the computer-readable storage medium include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any combination thereof. Within the scope of this disclosure, the computer-readable storage medium can be any physical medium capable of containing or storing a program for use by or in conjunction with a command execution system, apparatus, or device.Within the scope of this disclosure, a computer-readable signaling medium can contain a disseminated data signal embodied in computer-readable program code, for example, in the baseband or as part of a carrier wave. Such a transmitted data signal can take any form, including, but not limited to, electromagnetic or optical signals, or a suitable combination thereof. With the exception of the computer-readable storage medium, the computer-readable signaling medium can be any non-transitory computer-readable storage medium capable of transmitting, disseminating, or transporting a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied in the non-transitory computer-readable storage medium can be transmitted by any suitable medium, including, but not limited to: wirelessly, by wire, by fiber optic cable, RF, etc., or any suitable combination thereof.

[0128] The flowchart and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of a system, a process, and a computer program product according to various embodiments of the present disclosure. In this context, each block in the flowchart or block diagrams can represent a module, one or more program segments, or a portion of code comprising one or more executable instructions for implementing specific logical functions. It should also be noted that in some alternative implementations, the functions specified in the blocks may occur out of the order shown in the drawings.For example, two blocks connected sequentially may in reality be executed essentially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by special hardware-based systems that perform the specified functions or operations, or by combinations of special hardware and computer instructions.

[0129] It should be understood that the above embodiments are merely exemplary embodiments adopted to illustrate the principles of the present disclosure, and the present disclosure is not limited to them. It will be clear to a person skilled in the art that various changes and modifications can be made without infringing the spirit and scope of the present disclosure, and such changes and modifications also fall within the scope of the present disclosure.

Claims

[1] User interface display system comprising an image acquisition module, a system control module and a display module; where the image capture module is configured to capture a scene image; The system control module is configured to detect whether a target object is present in the scene image, and to track the target object and determine its current position in the scene if it is present; to determine a current coordinate point of the current position, which is mapped to an interface coordinate system of a user interface, according to the target object's current position in the scene and a mapping relationship between a previously established scene coordinate system and the interface coordinate system; and to generate update information from at least one interface component in the user interface according to the current coordinate point and predefined auxiliary parameter information; and The display module is configured to display at least one interface component according to the update information. [2] User interface display system according to claim 1, wherein the system control module comprises a detection unit, an information determination unit and an update information generation unit; the detection unit is configured to detect whether the target object is present in the scene image; the information determination unit is configured to track the target object and determine the target object's current position in the scene, if the target object exists; and to determine the current coordinate point of the current position, which is mapped to the interface coordinate system, according to the target object's current position in the scene and the mapping relationship between the previously established scene coordinate system and the interface coordinate system; and The update information generation unit is configured to generate the update information for each of the at least one interface component in the user interface according to the current coordinate point and predefined auxiliary parameter information. [3] User interface display system according to claim 2, wherein the predefined auxiliary parameter information comprises a predefined auxiliary location point on the interface coordinate system and a dimension scaling factor of each of the at least one interface component; the target object comprises a person; the detection unit is specifically configured to perform facial recognition on the scene image and to assess whether the target object is present; the current position includes a target key point; the information determination unit is specifically configured to track the target object and, according to a facial feature of the target object, determine a target key point in the face if facial recognition determines the presence of the target object; and, according to the target key point and a mapping relationship between the scene coordinate system and the interface coordinate system, determine a current coordinate point corresponding to the target key point; and The update information generation unit is configured to generate the update information of each of the at least one interface component according to the current coordinate point, the auxiliary location point, and the dimension scaling factor of each of the at least one interface component. [4] User interface display system according to claim 3, wherein the system control module further comprises a first preprocessing unit; and the first preprocessing unit is configured to determine an auxiliary location point on the interface coordinate system according to an initial coordinate point mapped to the interface coordinate system, an initial position of the tracked target object and an initial position of each of the at least one interface component in the user interface; and to determine the dimensional scaling factor of each of the at least one interface component according to the auxiliary location point, the initial coordinate point and the initial position of each of the at least one interface component. [5] User interface display system according to claim 2, wherein the user interface comprises a plurality of layers that multiplex the same interface coordinate system; the user interface with the layers comprises a foreground layer and a background layer as well as at least one intermediate layer between the foreground layer and the background layer; and the at least one interface component comprises a plurality of interface components; and the system control module further comprises an interface control unit; wherein the interface control unit is configured to control the display module to display different interface components in the foreground layer, the intermediate layer and the background layer. [6] User interface display system according to claim 5, wherein the update information generation unit is specifically configured to receive a component selection instruction to select an interface component, replace a dimension scaling factor of the currently selected interface component with a target scaling factor, and generate the update information of each interface component according to the current coordinate point, the replaced target scaling factor, and the predefined auxiliary location point on the interface coordinate system. [7] User interface display system according to claim 5, wherein the update information generation unit is configured to receive a component selection instruction of the interface components and to obtain a predefined dimensional scaling factor of each interface component; to replace the dimensional scaling factor of the currently selected interface component with the dimensional scaling factor of the interface component in the foreground layer in a case where the currently selected interface component is not in the foreground layer of the user interface; and to generate the update information of each interface component according to the current coordinate point, the replaced dimensional scaling factor and the predefined auxiliary location point on the interface coordinate system. [8] User interface display system according to claim 2, wherein the current position comprises a target key point and a depth reference point; the information determination unit is specifically configured to, in a case where the presence of the target object is determined by facial recognition, track the target object and determine a target key point in a face according to a facial feature of the target object; identify a shoulder feature of the target object; and take a point of a mean position between two shoulders as a depth reference point according to the positions of the two shoulders in the scene coordinate system; and the update information generation unit is specifically configured to determine a component scaling factor based on the depth reference point and the target key point;and to generate the update information of each interface component according to the component scaling factor, the current coordinate point, the predefined auxiliary location point on the interface coordinate system, and the dimension scaling factor of each interface component. [9] User interface display system according to claim 3 or 8, wherein the information determination unit is specifically configured to identify a pupil feature in the face in a case where facial recognition determines that the target object is present and to take a point of a mean position between two pupils as the target key point according to the positions of the two pupils in the scene coordinate system. [10] User interface display system according to claim 2, wherein the user interface comprises at least one layer comprising a background layer; the interface component comprises a background image; and the system control module further comprises an interface control unit configured to control the display module to display the background image in the background layer. [11] User interface display system according to claim 10, wherein the predefined auxiliary parameter information further comprises predefined size information of a viewport of the user interface, a predefined magnification factor of the background image and size information of the background image; the target object comprises a person; the detection unit is specifically configured to perform facial recognition on the scene image and to assess whether the target object is present; the current position includes a target key point; the information determination unit is specifically configured to track the target object and determine a target key point in a face according to a facial feature of the target object in a case where facial recognition determines that the target object is present; and to determine a current coordinate point corresponding to the target key point, according to the target key point and a mapping relationship between the scene coordinate system and the interface coordinate system; and The update information generation unit is further configured to determine an auxiliary reference point at which the current coordinate point is centrosymmetric with respect to a geometric center of the viewport, according to the current coordinate point and the viewport size information; and to generate the background image update information according to the viewport size information, the background image magnification factor, the auxiliary reference point, and the background image size information. [12] User interface display system according to claim 11, wherein the system control module further comprises a second preprocessing unit; and the second preprocessing unit is configured to determine the size information of the background image according to the size information of the viewing window and the magnification factor of the background image. [13] User interface display system according to claim 1, wherein the interface component comprises at least one of the following elements: a control element, a status bar, a map bar, a background image. [14] User interface display methods, including: Capturing a scene and detecting whether a target object is present in the scene; Tracking the target object and determining its current position in a scene in a case where the target object is present; Determining a current coordinate point of the current position, which is mapped to an interface coordinate system of a user interface, according to the current position of the target object in the scene and a mapping relationship between a previously established scene coordinate system and the interface coordinate system; Generating update information for at least one interface component in the user interface according to the current coordinate point and predefined auxiliary parameter information; and Displaying at least one interface component according to the update information. [15] Computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions that can be executed by the processor; the processor and the memory communicate with each other via the bus when the computer device is running; and the machine-readable instructions, when executed by the processor, perform the user interface display method according to claim 14. [16] Non-transitory computer-readable storage medium with a computer program stored thereon, wherein the computer program, when executed by a processor, performs the user interface display method according to claim 14.

Images