User interface display system and method, and computer device and storage medium

Abstract

The present disclosure belongs to the technical field of display. Provided are a user interface display system and method, and a computer device and a storage medium. The user interface display system comprises an image collection module, a system control module and a display module. The image collection module is configured to collect a scene picture. The system control module is configured to: detect whether there is a target object in the scene picture, and when there is a target object, track the target object and determine the current position of the target object in the scene; according to the current position of the target object in the scene and a pre-established mapping relationship between a scene coordinate system and an interface coordinate system of a user interface, determine the current coordinate point, to which the current position is mapped, in the interface coordinate system; and generate update information of each interface component in the user interface according to the current coordinate point and predefined auxiliary parameter information. The display module is configured to display each interface component according to the update information.

Description

User interface display system, method, computer device and storage medium Technical Field

[0001] The present disclosure belongs to the field of display technology, and particularly relates to a user interface display system, method, computer device, and storage medium. Background Art

[0002] With the update and optimization of the user interface (UI) in display devices, more and more UI interfaces begin to combine human-computer interaction to realize 3D UI interfaces to meet the observer's immersive experience.

[0003] However, conventional 3D UI display methods require the development of independent applications (apps) to create 3D scenes and enhance the viewer's panoramic visual experience. Creating 3D scenes in independent apps requires significant hardware resources, resulting in low system compatibility when used by other apps, impacting the use of other software functions.

[0004] Summary of the Invention

[0005] The present disclosure aims to solve at least one of the technical problems existing in the prior art and provides a user interface display system, method, computer device and storage medium.

[0006] In a first aspect, the technical solution adopted to solve the technical problem of the present disclosure is a user interface display system, which includes an image acquisition module, a system control module, and a display module;

[0007] The image acquisition module is configured to acquire scene images;

[0008] The system control module is configured to detect whether a target object exists in the scene image, and if the target object exists, track the target object and determine the current position of the target object in the scene; determine a current coordinate point of the current position mapped to the interface coordinate system based on the current position of the target object in the scene and a pre-established mapping relationship between the scene coordinate system and the interface coordinate system of the user interface; and generate update information of each interface component in the user interface based on the current coordinate point and pre-defined auxiliary parameter information;

[0009] The display module is configured to display each of the interface components according to the update information.

[0010] In some embodiments, the system control module includes a detection unit, an information determination unit, and an update information generation unit;

[0011] The detection unit is configured to detect whether there is a target object in the scene image;

[0012] The information determination unit is configured to, in the presence of the target object, track the target object and determine a current position of the target object in the scene; determine a current coordinate point of the current position mapped to the interface coordinate system based on the current position of the target object in the scene and a pre-established mapping relationship between the scene coordinate system and the interface coordinate system;

[0013] The update information generating unit is configured to generate update information of each interface component in the user interface according to the current coordinate point and predefined auxiliary parameter information.

[0014] In some embodiments, the predefined auxiliary parameter information includes predefined auxiliary positioning points on the interface coordinate system and size ratio coefficients of each of the interface components;

[0015] The target object includes a person; the detection unit is specifically configured to perform face recognition on the scene image to determine whether there is a target object;

[0016] The current position includes a target key point; the information determination unit is specifically configured to track the target object and determine the target key point in the face based on the facial features of the target object when face recognition determines that the target object exists; and determine the current coordinate point corresponding to the target key point based on the mapping relationship between the target key point and the scene coordinate system and the interface coordinate system;

[0017] The update information generating unit is configured to generate update information of each of the interface components according to the current coordinate point, the auxiliary positioning point and the size ratio coefficient of each of the interface components.

[0018] In some embodiments, the system control module further includes a first pre-processing unit;

[0019] The first preprocessing unit is configured to determine an auxiliary positioning point on the interface coordinate system based on the initial position of the tracked target object mapped to the initial coordinate point of the interface coordinate system and the initial position of each interface component in the user interface; and determine the size ratio coefficient of each interface component based on the auxiliary positioning point, the initial coordinate point and the initial position of each interface component.

[0020] In some embodiments, the user interface includes multiple layers, and each layer reuses the same interface coordinate system; the multiple layers of the user interface include a foreground layer and a background layer, and at least one intermediate layer located between the foreground layer and the background layer; the interface components include multiple;

[0021] The system control module further includes an interface control unit; the interface control unit is configured to control the display module to display different interface components on the foreground layer, the middle layer, and the background layer.

[0022] In some embodiments, the update information generation unit is specifically configured to receive a component selection instruction of the interface component, replace the size scale coefficient of the currently selected interface component with the target scale coefficient, and generate update information for each of the interface components based on the current coordinate point, the replaced target scale coefficient and the pre-defined auxiliary positioning point on the interface coordinate system.

[0023] In some embodiments, the update information generation unit is specifically configured to receive a component selection instruction of the interface component and obtain a pre-defined size ratio coefficient of each of the interface components; when the currently selected interface component is not located in the foreground layer of the user interface, the size ratio coefficient of the currently selected interface component is replaced with the size ratio coefficient of the interface component located in the foreground layer; and update information of each of the interface components is generated according to the current coordinate point, the replaced size ratio coefficient and the pre-defined auxiliary positioning point on the interface coordinate system.

[0024] In some embodiments, the current position includes a target key point and a depth distance reference point; the information determination unit is specifically configured to, when face recognition determines that the target object exists, track the target object and determine the target key point in the face based on the facial features of the target object; identify the shoulder features of the target object, and use the middle position point between the two shoulders as the depth distance reference point based on the positions of the two shoulders in the scene coordinate system;

[0025] The update information generation unit is specifically configured to determine the component scaling factor based on the depth distance reference point and the target key point; and generate update information for each interface component based on the component scaling factor, the current coordinate point, the pre-defined auxiliary positioning point on the interface coordinate system, and the size ratio coefficient of each interface component.

[0026] In some embodiments, the information determination unit is specifically configured to identify pupil features in the face when face recognition determines the presence of the target object, and to use the middle position point of the two pupils as the target key point based on the positions of the two pupils in the scene coordinate system.

[0027] In some embodiments, the user interface includes at least one layer, the at least one layer of the user interface includes a background layer; the interface component includes a background image;

[0028] The system control module further includes an interface control unit; the interface control unit is configured to control the display module to display a background image on the background layer.

[0029] In some embodiments, the predefined auxiliary parameter information further includes predefined size information of the visual window of the user interface, a predefined magnification factor of the background image, and size information of the background image;

[0030] The target object includes a person; the detection unit is specifically configured to perform face recognition on the scene image to determine whether there is a target object;

[0031] The current position includes a target key point; the information determination unit is specifically configured to track the target object and determine the target key point in the face based on the facial features of the target object when face recognition determines that the target object exists; and determine the current coordinate point corresponding to the target key point based on the mapping relationship between the target key point and the scene coordinate system and the interface coordinate system;

[0032] The update information generation unit is further configured to determine an auxiliary reference point that is centrally symmetric about the geometric center of the visual window with respect to the current coordinate point and the size information of the visual window; and generate update information of the background image based on the size information of the visual window, the magnification factor of the background image, the auxiliary reference point and the size information of the background image.

[0033] In some embodiments, the system control module further includes a second pre-processing unit;

[0034] The second pre-processing unit is configured to determine the size information of the background image according to the size information of the visual window and the magnification factor of the background image.

[0035] In some embodiments, the interface component includes at least one of a control, a status bar, a card bar, and a background image.

[0036] In a second aspect, an embodiment of the present disclosure further provides a user interface display method, which includes:

[0037] Capturing a scene image and detecting whether a target object exists in the scene image;

[0038] In the case where the target object exists, tracking the target object and determining a current position of the target object in the scene;

[0039] Determining a current coordinate point of the target object mapped to the interface coordinate system based on the current position of the target object in the scene and a pre-established mapping relationship between the scene coordinate system and the interface coordinate system of the user interface;

[0040] Generate update information of each interface component in the user interface according to the current coordinate point and predefined auxiliary parameter information;

[0041] Each of the interface components is displayed according to the update information.

[0042] In a third aspect, an embodiment of the present disclosure further provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the user interface display method described in the above embodiment are performed.

[0043] In a fourth aspect, an embodiment of the present disclosure further provides a computer non-volatile readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the user interface display method described in the above embodiment are executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1 is a schematic structural diagram of a user interface display system provided by an embodiment of the present disclosure;

[0045] FIG2 is a schematic diagram of trajectory tracking provided by an embodiment of the present disclosure;

[0046] FIG3a is a schematic diagram showing the positional relationship between auxiliary positioning points and interface components on an interface coordinate system provided by an embodiment of the present disclosure;

[0047] FIG3 b is a schematic diagram of the positional relationship between the current coordinate point and various interface components on the interface coordinate system provided by an embodiment of the present disclosure;

[0048] FIG4 a is a front view of a user interface provided by an embodiment of the present disclosure;

[0049] FIG4 b is a hierarchical schematic diagram of a user interface provided by an embodiment of the present disclosure;

[0050] FIG5a is a schematic diagram of various interface components before updating provided by an embodiment of the present disclosure;

[0051] FIG5b is a schematic diagram of each interface component after being updated in FIG5a;

[0052] FIG6 is a schematic diagram comparing two different distances of a target object relative to a user interface provided by an embodiment of the present disclosure;

[0053] FIG7 is a schematic diagram of the positional relationship between the background image and the vertices of the visual window in the interface coordinate system provided by an embodiment of the present disclosure;

[0054] FIG8 is a flow chart of a user interface display method provided by an embodiment of the present disclosure;

[0055] FIG9 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0057] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The words "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "an" or "the" do not indicate a quantity limitation, but rather indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0058] In this disclosure, "multiple or several" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0059] Conventional 3D UI display methods require the development of independent apps to create 3D scenes and enhance the viewer's panoramic visual experience. Creating 3D scenes in independent apps requires significant hardware resources, resulting in low system compatibility when used by other apps, impacting the use of other software functions.

[0060] Based on this, an embodiment of the present disclosure provides a user interface display system, including an image acquisition module, a system control module, and a display module; wherein the image acquisition module is configured to acquire a scene picture; the system control module is configured to detect whether there is a target object in the scene picture, and if there is a target object, track the target object and determine the current position of the target object in the scene; according to the current position of the target object in the scene and the mapping relationship between the pre-established scene coordinate system and the interface coordinate system of the user interface, determine the current coordinate point of the current position mapped to the interface coordinate system; according to the current coordinate point and pre-defined auxiliary parameter information, generate update information of each interface component in the user interface; the display module is configured to display each interface component according to the updated information.

[0061] The disclosed embodiment captures a scene image of the system's surrounding environment through an integrated image acquisition module. The system control module then detects the presence of a target object within the scene image and, if present, tracks the target object in real time to determine its current position within the scene. Based on the current position, the current coordinate point mapped to the user interface is found. Update information for each interface component in the user interface is generated using the current coordinate point and predefined auxiliary parameter information. This means that the movement and / or scaling of each interface component in the user interface is calculated using the principle of spatial perspective. The display module then displays the interface components based on the updated information, such as moving the interface components with the target object or scaling them proportionally. This 2D display method utilizes the principle of parallax to simulate a relatively realistic 3D scene experience, and simulates 3D interactive effects through a 2D display method. Compared to developing separate application software to implement 3D scenes, the disclosed embodiment integrates various functional modules into the user interface display system, saving hardware resources and improving compatibility between the UI interface display and other applications. Furthermore, the use of spatial perspective to simulate a relatively realistic 3D scene simplifies the development process of 3D scene software and reduces software maintenance and iteration costs.

[0062] The user interface display system in the embodiments of the present disclosure can be applied to a large-screen terminal system, where "large screen" can be understood as a display screen with a larger screen size and higher resolution, such as an ultra-high-definition display screen larger than 40 inches and with an 8K resolution.

[0063] The following describes in detail the various functional modules of a user interface display system provided in an embodiment of the present disclosure. FIG1 is a structural diagram of a user interface display system provided in an embodiment of the present disclosure. As shown in FIG1 , the user interface display system includes an image acquisition module 101, a system control module 102, and a display module 103.

[0064] Image acquisition module 101 is configured to capture scene images. Image acquisition module 101 can be a camera integrated into the user interface display system, configured to capture the scene images of the environment in which the camera is located. The camera continuously captures the scene images, and the scene images are updated in real time. Over a period of time, the scene images may contain dynamic objects or static images.

[0065] The target object in the embodiment of the present disclosure may be a pre-set object with specific characteristics, such as a person, or a selected special object.

[0066] System control module 102 may be a processor of the user interface display system, configured to process the captured scene images to obtain user interface update information, i.e., update information for each interface component in the user interface. System control module 102 includes a detection unit 121, an information determination unit 122, and an update information generation unit 123.

[0067] Among them, the detection unit 121 is configured to detect whether there is a target object in the scene picture. Specifically, the detection unit 121 is specifically configured to extract image features of the scene picture, process the extracted features, and determine whether there is a target object. If there is a target object, the current position of the target object in the scene is determined, that is, the coordinate point of the target object in the scene coordinate system is determined. Here, the scene coordinate system can be a two-dimensional coordinate system of the scene picture captured by a pre-established camera. Of course, the scene coordinate system can also be a spatial coordinate system of the environment in which the camera is located. Optionally, in order to simplify the algorithm and save hardware resources, the scene coordinate system in the embodiment of the present disclosure adopts a two-dimensional coordinate system that is easier to construct. In this way, the current position of the target object in the scene is also a two-dimensional coordinate point in a scene coordinate system.

[0068] Information determination unit 122 is configured to track the target object and determine its current position within the scene, if the target object is present. In other words, upon determining the presence of the target object, the target object is tracked. It will be appreciated that the camera captures the scene image in real time and locates the target object's two-dimensional coordinates in the scene coordinate system in real time to achieve target trajectory tracking.

[0069] For example, FIG2 is a schematic diagram of trajectory tracking provided by an embodiment of the present disclosure. As shown in FIG2 , the previous position H1 (hx1, hy1), (hx1, hy1) is within (0,0) to (cx, cy), cx is the maximum horizontal coordinate value of the scene image in the scene coordinate system, and cy is the maximum vertical coordinate value of the scene image in the scene coordinate system. When the observer moves, the starting point H1 (hx1, hy1) and the end point H2 (hx2, hy2), that is, the current position H2, the displacement obtained at this time is hxm = hx2 - hx1, hym = hy2 - hy1. The end point here is the position point in the tracking process, not the starting position point of the target object.

[0070] The information determination unit 122 is configured to determine the current coordinate point of the target object's current position mapped to the interface coordinate system based on the current position of the target object in the scene and the pre-established mapping relationship between the scene coordinate system and the interface coordinate system of the user interface. Here, the storage unit in the system control module 102 stores the pre-established mapping relationship between the scene coordinate system and the interface coordinate system of the user interface. Specifically, the information determination unit 122 is configured to, after determining the current position of the target object in the scene, map the two-dimensional coordinate point of the current position in the scene coordinate system to the interface coordinate system based on the mapping relationship between the coordinate systems, and determine the current coordinate point mapped to the interface coordinate system.

[0071] Continuing with FIG2 , the maximum horizontal coordinate of the visual window of the screen is xm, and the maximum vertical coordinate is ym. The information determination unit 122 is configured to determine the displacement (vxm, vym) of the target object based on the displacement hxm = hx2 - hx1, hym = hy2 - hy1, and determine the current coordinate point V (vx, vy) based on the displacement (vxm, vym) and the previous coordinate point. Here, vxm / xm = hxm / cx, and vym / ym = hym / cy.

[0072] The update information generation unit 123 is configured to generate update information for each interface component in the user interface based on the current coordinate point and predefined auxiliary parameter information. The current coordinate point can be understood as a reference point, which drives the movement of each interface component in the user interface. Auxiliary parameter information is predefined parameter information used to assist the current coordinate point to achieve the status update of each interface component. The auxiliary parameter information is used to characterize the association relationship between the current coordinate point and each interface component in the user interface. For the specific content of the auxiliary parameter information, please refer to the description of the preprocessing unit below and will not be described in detail here. The update information of each interface component here can be, for example, the movement information and / or scaling information of each interface component, wherein the movement information can include, for example, the movement trajectory or the end coordinate point, etc.; the scaling information can include, for example, the scaling ratio, the scaling reference point, etc. The movement trajectory, the end coordinate point, the scaling ratio, and the scaling reference point involved here are all information under the interface coordinate system.

[0073] The display module 103 is configured to display each interface component according to the update information. For example, it displays each interface component after the position update. The position update can be the movement and / or scaling of the interface component within the visual window.

[0074] The above-mentioned units realize real-time interaction between the interface components in the user interface and the target object by constructing the association relationship between the current position of the target object and the current coordinate point of the user interface, and utilizing auxiliary parameter information (the association relationship between the current coordinate point and the interface components in the user interface), simulating the 3D interaction effect through 2D display, and improving the user's interactive experience.

[0075] In some embodiments, the target object includes a person. The detection unit 121 is specifically configured to identify at least some features of the human body to determine whether the target object exists. Specifically, some highly recognizable features that can represent a person can be selected, such as facial features (including facial features), head features, torso features, etc.

[0076] Optionally, the detection unit 121 is specifically configured to perform face recognition on the scene image to determine whether the target object exists. Specifically, the scene image is identified with facial features to determine whether there are screen features that meet the facial features. If so, it is determined that the target object exists; otherwise, it is determined that the target object does not exist.

[0077] The current position includes the target key points. The information determination unit 122 is specifically configured to track the target object and determine the target key points in the face based on the facial features of the target object when face recognition determines that the target object exists. For example, when it is determined that the target object exists, the two-dimensional coordinate points of the target object in the scene coordinate system are located in real time to achieve target trajectory tracking. At the same time, based on the facial features of the target object, such as facial features, specific facial bones, and other features, the target key points in the face are determined. For example, the target key points can be the two-dimensional coordinate points of the scene coordinate system where the detected facial features are located, the two-dimensional coordinate points of the skeletal key points in the scene coordinate system, etc.

[0078] The information determination unit 122 is specifically configured to determine the current coordinate point corresponding to the target key point based on the mapping relationship between the target key point and the scene coordinate system and the interface coordinate system. For example, based on the mapping relationship between the scene coordinate system and the interface coordinate system, the target key point in the scene coordinate system is mapped to the interface coordinate system to determine the mapped current coordinate point.

[0079] The predefined auxiliary parameter information includes the predefined auxiliary positioning point on the interface coordinate system and the size ratio coefficient of each interface component. The update information generation unit 123 is configured to generate update information of each interface component according to the current coordinate point, the auxiliary positioning point and the size ratio coefficient of each interface component.

[0080] Here, auxiliary positioning points are two-dimensional coordinate points on the interface coordinate system that are used to assist in locating the positions of various interface components in the user interface. For example, auxiliary positioning point A' is used to assist in locating the position of interface component 31, and auxiliary positioning point B' is used to assist in locating the positions of interface component 31 and / or interface component 32. The size ratio coefficient of each interface component can be determined based on at least the initial position of each interface component in the user interface and the auxiliary positioning points. The size ratio coefficient of each interface component is used to assist in locating the position of each interface component.

[0081] In some embodiments, the interface components include at least one of a control, a status bar, a card bar, and a background image. A control may be a software entry in a UI interface, typically presented in the form of a software icon; a status bar may be a UI interface used to display the status of currently open windows or software, such as the status of multiple open folders or other software applications, or a field for prompting information, such as a "screenshot" field or a "data" field; a card bar may be a window in a UI interface used to display images or text information. A background image may be, for example, a wallpaper in a UI interface.

[0082] In some embodiments, the system control module 102 further includes a first pre-processing unit; the first pre-processing unit is configured to determine an auxiliary positioning point on the interface coordinate system based on the initial position of the tracked target object mapped to the initial coordinate point of the interface coordinate system and the initial position of each interface component in the user interface. Based on the auxiliary positioning point, the initial coordinate point and the initial position of each interface component, the size ratio coefficient of each interface component is determined. Here, the initial position is the position of the target object in the scene coordinate system when the detection unit 121 first detects the target object. The position after the initial position is mapped to the interface coordinate system is the initial coordinate point. The initial position of each interface component initialized in the user interface is known.

[0083] Figure 3a is a schematic diagram of the positional relationship between each auxiliary positioning point and each interface component on the interface coordinate system provided by an embodiment of the present disclosure. As shown in Figure 3a, the vertex coordinates of the visual window in the user interface are known, namely A(0,0), B(0,ym), C(xm,0) and D(xm,ym). Here, the visual window is the screen window of the UI interface that the user can view. The initial coordinate point is V0. The initial position of the interface component 31 (such as a card bar) includes the coordinates of four vertices, and the four vertices of the interface component 31 are respectively recorded as M1, M2, M3 and M4; the initial position of the interface component 32 (such as a status bar) includes the coordinates of four vertices, and the four vertices of the interface component 32 are respectively recorded as F1, F2, F3 and F4. The auxiliary positioning point A' is used to assist in positioning the interface component 31. Specifically, the auxiliary positioning point A' is the intersection of the extension line of the line connecting the initial coordinate point V0 and the point M1 and the AB line segment. The auxiliary positioning point B' is used to assist in positioning the interface component 31 and the interface component 32. The auxiliary positioning point B' is the intersection of the extension line of the line connecting the initial coordinate point V0 and the point F1 and the AB line segment, and is also the intersection of the extension line of the line connecting the initial coordinate point V0 and the point M3 and the AB line segment. The auxiliary positioning point C' is used to assist in positioning the interface component 31. Specifically, the auxiliary positioning point C' is the intersection of the extension line of the line connecting the initial coordinate point V0 and the point M2 and the CD line segment. The auxiliary positioning point D' is used to assist in positioning the interface component 31 and the interface component 32. The auxiliary positioning point D' is the intersection of the extension line of the line connecting the initial coordinate point V0 and the point F2 and the CD line segment, and is also the intersection of the extension line of the line connecting the initial coordinate point V0 and the point M4 and the CD line segment. At the same time, V0, F3 and B are located on the same straight line, and V0, F4 and D are located on the same straight line.

[0084] The size ratio coefficient of the interface component 31 is the ratio of A'M1 to A'V0, which is also the ratio of C'M2 to C'V0, which is also the ratio of B'M3 to BV0, and which is also the ratio of D'M4 to D'V0, all denoted by m. Here, m = B'M3 / BV0 = A'M1 / A'V0 = C'M2 / C'V0 = D'M4 / D'V0. The size ratio coefficient of the interface component 32 is the ratio of B'F1 to B'V0, which is also the ratio of D'F2 to D'V0, which is also the ratio of BF3 to BV0, and which is also the ratio of DF4 to DV0, all denoted by f. Here, f = B'F1 / B'V0 = D'F2 / D'V0 = BF3 / BV0 = DF4 / DV0.

[0085] The update information generation unit 123 is specifically configured to generate update information for each interface component based on the current coordinate point, the auxiliary positioning point and the size ratio coefficient of each interface component. Exemplarily, FIG3b is a schematic diagram of the positional relationship between the current coordinate point and each interface component on the interface coordinate system provided by an embodiment of the present disclosure. As shown in FIG3b , the current coordinate point V(vx,vy), the auxiliary positioning point A'(0,a), the auxiliary positioning point B'(0,b), the auxiliary positioning point C'(xm,a), the auxiliary positioning point D'(xm,b), the size ratio coefficient m of the interface component 31, and the size ratio coefficient f of the interface component 32 are known to determine the update information of the interface component 31, i.e., the updated M1, M2, M3 and M4, and the update information of the interface component 32, i.e., the updated F1, F2, F3 and F4. Among them, the updated M1, M2, M3 and M4 are as follows:

[0086] M1=[m×vx,a+m / (vy-a)]

[0087] M2=[xm-m×(xm-vx),am×(a-vy)]

[0088] M3=[m×vx,bm×(b-vy)]

[0089] M4=[xm-m×(xm-vx),bm×(b-vy)]

[0090] The updated F1, F2, F3 and F4 are as follows:

[0091] F1=[f×vx,bf×(b-vy)]

[0092] F2=[xm-f×(xm-vx),bf×(b-vy)]

[0093] F3=[f×vx,ym-f×(ym-vy)]

[0094] F4=[xm-f×(xm-vx),ym-f×(ym-vy)]

[0095] Here, a, b, f, and m are constants and can be set according to design requirements. The embodiments of the present disclosure do not specifically limit them.

[0096] It should be noted that at any given moment, the coordinates of A(0,0), B(0,ym), C(xm,0), D(xm,ym), A'(0,a), B'(0,b), C'(xm,a), and D'(xm,b) remain fixed, and V, M1, and A' are all on the same straight line; V, M2, and C' are all on the same straight line; V, M3, and B' are all on the same straight line; V, M4, and D' are all on the same straight line; V, F1, and B' are all on the same straight line; V, F2, and D' are all on the same straight line; V, F3, and B are all on the same straight line; and V, F4, and D are all on the same straight line. Therefore, as the current coordinate point V changes, the positions of the various interface components are constantly updated.

[0097] The above units utilize the association between the target object and the current coordinate point, and the association between the current coordinate point and each component, so that each interface component can follow the movement of the target object, thereby realizing dynamic interaction between the target object and each interface component.

[0098] In some embodiments, Figure 4a is a front schematic diagram of the user interface provided by an embodiment of the present disclosure, and Figure 4b is a layered schematic diagram of the user interface provided by an embodiment of the present disclosure. As shown in Figures 4a and 4b, the user interface includes multiple layers, and each layer reuses the same interface coordinate system; the multi-layer user interface includes a foreground layer 401 and a background layer 403, and at least one intermediate layer 402 located between the foreground layer 401 and the background layer 403; the interface components include multiple, such as the status bar 41, the card bar 42, and the background image 43 shown in the figure.

[0099] The system control module 102 also includes an interface control unit; the interface control unit is configured to control the display module 103 to display different interface components in the foreground layer, the middle layer, and the background layer. For example, as shown in Figure 4b, the interface control unit is configured to control the display module 103 to display a status bar 41 in the foreground layer, a card bar 42 in the middle layer, and a background image 43 in the background layer. By arranging the user interface in layers and utilizing the principle of parallax to create a sense of depth, the disclosed embodiment can present a more realistic 3D interface display effect to the user.

[0100] In some embodiments, in order to further improve the user's interactive experience, when the user selects the interface component of the middle layer, the user interface display system can also adjust the interface component of the middle layer to the foreground layer for display, specifically by enlarging the currently selected interface component to highlight the interface component currently selected by the user.

[0101] One possible implementation is an update information generation unit 123, which is specifically configured to receive a component selection instruction of an interface component and obtain a predefined size ratio coefficient of each interface component; when the currently selected interface component is not located in the foreground layer of the user interface, the size ratio coefficient of the currently selected interface component is replaced with the size ratio coefficient of the interface component located in the foreground layer; and update information of each interface component is generated based on the current coordinate point, the replaced size ratio coefficient and the auxiliary positioning point on the predefined interface coordinate system.

[0102] For example, FIG5a is a schematic diagram of each interface component before the update provided by the embodiment of the present disclosure, and FIG5b is a schematic diagram of each interface component in FIG5a after the update. As shown in FIG5a and FIG5b, the component selection instruction can be an instruction for the user to select an interface component located in the middle layer, for example, selecting the card bar 42. Given the size ratio coefficient m of the card bar 42, the size ratio coefficient m of the card bar 42 is replaced with the size ratio coefficient f of the status bar 41. At the same time, based on the current coordinate point V (vx, vy), the replaced size ratio coefficient, and the pre-defined auxiliary positioning points A'(0, a), B'(0, b), C'(xm, a), and D'(xm, b) on the interface coordinate system, update information for the card bar 42 and update information for the status bar 41 are generated. Among them, the updated M1, M2, M3, and M4 are as follows:

[0103] M1=[f×vx,a+f×(vy-a)]

[0104] M2=[xm-f×(xm-vx),af×(a-vy)]

[0105] M3=[f×vx,bf×(b-vy)]

[0106] M4=[xm-f×(xm-vx),bf×(b-vy)]

[0107] The updated F1, F2, F3 and F4 are as follows:

[0108] F1=[m×vx,bm×(b-vy)]

[0109] F2=[xm-m×(xm-vx),bm×(b-vy)]

[0110] F3=[m×vx,ym-m×(ym-vy)]

[0111] F4=[xm-m×(xm-vx),ym-m×(ym-vy)]

[0112] Here, a, b, f, and m are constants and can be set according to design requirements. The embodiments of the present disclosure do not specifically limit them.

[0113] Another possible implementation is the update information generation unit 123, which is specifically configured to receive a component selection instruction of an interface component, replace the size scale coefficient of the currently selected interface component with the target scale coefficient, and generate update information for each interface component based on the current coordinate point, the replaced target scale coefficient and the auxiliary positioning point on the pre-defined interface coordinate system.

[0114] Here, the target scale factor can be data determined based on experience, such as selecting the target scale factor r. The component selection instruction can be an instruction for any interface component selected by the user, such as selecting the status bar 41. Given the size scale factor f of the status bar 41, the size scale factor f of the status bar 41 is replaced with the target scale factor r. At the same time, based on the current coordinate point V(vx,vy), the replaced size scale factor r, and the auxiliary positioning point B'(0,b) and auxiliary positioning point D'(xm,b) on the pre-defined interface coordinate system, the updated information of the status bar 41 is generated. Among them, the updated F1, F2, F3 and F4 are as follows:

[0115] F1=[r×vx,br×(b-vy)]

[0116] F2=[xm-r×(xm-vx),br×(b-vy)]

[0117] F3=[r×vx,ym-r×(ym-vy)]

[0118] F4=[xm-r×(xm-vx),ym-r×(ym-vy)]

[0119] Here, a, b, and r are constants and can be set according to design requirements. The embodiments of the present disclosure do not specifically limit them.

[0120] It should be noted that the status bar 41 obtained by using the size ratio coefficient r after replacement is displayed larger than the original status bar 41 obtained by using the size ratio coefficient f before replacement.

[0121] In some embodiments, the current position includes a target key point and a depth distance reference point. Information determination unit 122 is specifically configured to, when facial recognition determines the presence of a target object, track the target object and determine the target key point on the face based on the target object's facial features. It also identifies the target object's shoulder features and, based on the positions of the two shoulders in the scene coordinate system, uses the midpoint between the two shoulders as the depth distance reference point.

[0122] For example, when it is determined that there is a target object, the two-dimensional coordinate points of the target object in the scene coordinate system are located in real time to achieve target trajectory tracking. At the same time, based on the facial features of the target object, such as facial features, specific facial bones and other features, the target key points in the face are determined. The target key points can be, for example, the two-dimensional coordinate points of the detected facial features in the scene coordinate system, the two-dimensional coordinate points of the facial skeleton key points in the scene coordinate system, etc. Based on the shoulder features, such as the shoulder bone features, the two-dimensional coordinate points of the two shoulder bone key points of the human body in the scene coordinate system are determined, and the center point of the two shoulder two-dimensional coordinate points is determined (that is, the middle position point of the two shoulders), and the center point is used as the depth distance reference point.

[0123] The update information generation unit 123 is specifically configured to determine the component scaling factor based on the depth distance reference point and the target key point; and generate update information for each interface component based on the component scaling factor, the current coordinate point, the auxiliary positioning point on the predefined interface coordinate system, and the size ratio coefficient of each interface component.

[0124] FIG6 is a schematic diagram comparing two different distances of a target object relative to a user interface provided by an embodiment of the present disclosure. As shown in FIG6 , the target key point here can be selected as the center point 61 of the two pupils of the target object, and the length z0 of the line connecting the depth distance reference point 62 and the center point 61 of the two pupils of a target object can be used as the initial scaling factor z0. Among them, the initial scaling factor of the target object at the previous moment is z0_1, and the initial scaling factor at the next moment is z0_2. Based on the initial scaling factor z0, a component scaling factor z1 can be obtained using a preset algorithm. Here, the preset algorithm can be determined based on the mapping relationship between the scene coordinate system and the interface coordinate system, as well as the adjustment of the matching ratio parameters of some hardware devices (such as a camera, etc.). That is, the component scaling factor z1 can be the initial scaling factor z0 converted by the mapping relationship between the scene coordinate system and the interface coordinate system, and then determined by adjusting the matching ratio parameters. The specific matching ratio parameters are set according to the actual application scenario and the selected device parameters, and are not specifically limited in the embodiment of the present disclosure.

[0125] For example, given the component scaling factor z1_1 of the target object at the previous moment, the current coordinate point V(vx,vy), the auxiliary positioning point A'(0,a), the auxiliary positioning point B'(0,b), the auxiliary positioning point C'(xm,a), the auxiliary positioning point D'(xm,b), the size scaling factor m of the interface component 31, and the size scaling factor f of the interface component 32, the updated information of the interface component 31, i.e., the updated M1, M2, M3, and M4, and the updated information of the interface component 32, i.e., the updated F1, F2, F3, and F4, are determined. The updated M1, M2, M3, and M4 are as follows:

[0126] M1=[z1_1×m×vx,a+z1_1×m×(vy-a)]

[0127] M2=[xm-z1_1×m×(xm-vx),a-z1_1×m×(a-vy)]

[0128] M3=[z1_1×m×vx,b-z1_1×m×(b-vy)]

[0129] M4=[xm-z1_1×m×(xm-vx),b-z1_1×m×(b-vy)]

[0130] The updated F1, F2, F3 and F4 are as follows:

[0131] F1=[z1_1×f×vx,b-z1_1×f×(b-vy)]

[0132] F2=[xm-z1_1×f×(xm-vx),b-z1_1×f×(b-vy)]

[0133] F3=[z1_1×f×vx,ym-z1_1×f×(ym-vy)]

[0134] F4=[xm-z1_1×f×(xm-vx),ym-z1_1×f×(ym-vy)]

[0135] For example, given the component scaling factor z1_2 of the target object at the previous moment, the current coordinate point V(vx,vy), the auxiliary positioning point A'(0,a), the auxiliary positioning point B'(0,b), the auxiliary positioning point C'(xm,a), the auxiliary positioning point D'(xm,b), the size scaling factor m of the interface component 31, and the size scaling factor f of the interface component 32, the updated information of the interface component 31, i.e., the updated M1, M2, M3, and M4, and the updated information of the interface component 32, i.e., the updated F1, F2, F3, and F4, are determined. The updated M1, M2, M3, and M4 are as follows:

[0136] M1=[z1_2×m×vx,a+z1_2×m×(vy-a)]

[0137] M2=[xm-z1_2×m×(xm-vx),a-z1_2×m×(a-vy)]

[0138] M3=[z1_2×m×vx,b-z1_2×m×(b-vy)]

[0139] M4=[xm-z1_2×m×(xm-vx),b-z1_2×m×(b-vy)]

[0140] The updated F1, F2, F3 and F4 are as follows:

[0141] F1=[z1_2×f×vx,b-z1_2×f×(b-vy)]

[0142] F2=[xm-z1_2×f×(xm-vx),b-z1_2×f×(b-vy)]

[0143] F3=[z1_2×f×vx,ym-z1_2×f×(ym-vy)]

[0144] F4=[xm-z1_2×f×(xm-vx),ym-z1_2×f×(ym-vy)]

[0145] Here, a, b, f, and m are constants and can be set according to design requirements. The embodiments of the present disclosure do not specifically limit them.

[0146] This embodiment uses facial features to find target key points, uses shoulder features to find depth distance reference points, and uses the line between the target key points and the depth distance reference points as a reference condition for scaling the interface components to achieve scaling of the interface components, thereby simulating a more realistic 3D interaction effect through a 2D display method, thereby saving the system's hardware resources, and also better achieving compatibility between the UI interface display and other applications.

[0147] In some embodiments, information determination unit 122 is specifically configured to, when face recognition determines the presence of a target object, identify pupil features in the face and, based on the positions of the two pupils in the scene coordinate system, select the center point between the two pupils as the target key point. Here, selecting the center point between the two pupils as the target key point allows for more accurate location of the target object. Furthermore, pupil features have a high degree of recognition, facilitating face recognition and location.

[0148] In some embodiments, as shown in Figures 4a and 4b, the user interface includes at least one layer, the at least one user interface layer including a background layer 403; the interface components include a background image 43. The system control module 102 also includes an interface control unit; the interface control unit is configured to control the display module 103 to display the background image on the background layer.

[0149] In some embodiments, FIG7 is a schematic diagram illustrating the positional relationship between the vertices of a background image and a visual window in an interface coordinate system according to embodiments of the present disclosure. As shown in FIG7 , 71 represents the background image. To enhance the sense of space, background image 71 also changes with the target object. To ensure that the edges of background image 71 are not exposed within visual window 72 during movement, a suitable size of background image 71 is required, which should be larger than the size of the visual window.

[0150] The system control module 102 further includes a second pre-processing unit; the second pre-processing unit is configured to determine the size information of the background image 71 according to a pre-defined magnification factor of the background image 71 and the size information of the visual window 72 .

[0151] Continuing with FIG7 , the size information of the visual window 72 includes the coordinate information of each vertex of the visual window 72, namely, the vertex coordinates A(0,0), B(0,ym), C(xm,0) and D(xm,ym) of the visual window 72. The magnification factor of the background image 71 is p, where p can be set according to actual design requirements and is not specifically limited in the present embodiment. p=L1+L2. The size information of the background image 71 includes the width Wx and the height Wy. The size of the background image 71 is proportionally enlarged according to the size of the visual window 72, satisfying Wx / xm=Wy / ym. The width Wx of the background image 71 is xm+p, and the height Wy is (ym+ym / xm×p). The size information of the background image 71 also includes the coordinate information of each vertex of the background image 71, namely, the vertex coordinates W1, W2, W3 and W4 of the background image 71. The vertex coordinates of the background image 71 change as the current coordinate point is updated. In the initial state, the geometric center of the background image 71 coincides with the geometric center of the visual window 72.

[0152] In some embodiments, the target object includes a person. The detection unit 121 is specifically configured to perform face recognition on the scene screen to determine whether the target object exists. Specifically, some highly recognizable features that can represent a person can be selected, such as facial features (including facial features), head features, torso features, etc. Optionally, the detection unit 121 is specifically configured to perform face recognition on the scene screen to determine whether the target object exists. Specifically, the scene screen is identified for facial features to determine whether there are screen features that meet the facial features. If so, it is determined that the target object exists; otherwise, it is determined that the target object does not exist.

[0153] The current position includes the target key points. The information determination unit 122 is specifically configured to track the target object and determine the target key points in the face based on the facial features of the target object when face recognition determines that the target object exists. For example, when it is determined that the target object exists, the two-dimensional coordinate points of the target object in the scene coordinate system are located in real time to achieve target trajectory tracking. At the same time, based on the facial features of the target object, such as facial features, specific facial bones, and other features, the target key points in the face are determined. For example, the target key points can be the two-dimensional coordinate points of the scene coordinate system where the detected facial features are located, the two-dimensional coordinate points of the skeletal key points in the scene coordinate system, etc.

[0154] The information determination unit 122 is specifically configured to determine the current coordinate point corresponding to the target key point based on the mapping relationship between the target key point and the scene coordinate system and the interface coordinate system. For example, based on the mapping relationship between the scene coordinate system and the interface coordinate system, the target key point in the scene coordinate system is mapped to the interface coordinate system to determine the mapped current coordinate point.

[0155] The predefined auxiliary parameter information also includes predefined user interface visual window size information, a predefined background image magnification factor, and background image size information. Update information generation unit 123 is further configured to determine, based on the current coordinate point and the visual window size information, an auxiliary reference point that is centrosymmetric about the geometric center of the visual window; and generate background image update information based on the visual window size information, the background image magnification factor, the auxiliary reference point, and the background image size information.

[0156] Continuing with Figure 7, the auxiliary reference point V_mirror changes as the tracked current coordinate point V(vx,vy) changes. At any moment, the auxiliary reference point V_mirror and the current coordinate point V(vx,vy) are centrally symmetric about the geometric center of the visual window. The position coordinates of the auxiliary reference point V_mirror in the interface coordinate system are (xm-vx, ym-vy). At any moment, point V_mirror, point A, and point W1 are all on the same straight line, point V_mirror, point C, and point W2 are all on the same straight line, point V_mirror, point B, and point W3 are all on the same straight line, and point V_mirror, point D, and point W4 are all on the same straight line. Therefore, as the current coordinate point V continues to change, the position of the background image is constantly updated. Specifically, the coordinates of point W1 of the background image are [-p / xm×(xm-vx),-p / xm×(ym-vy)].

[0157] This embodiment increases the spatial sense of the UI interface by enlarging the size of the background image and allowing the position of the background image to be updated according to the position of the target object, thereby improving the user's interactive experience.

[0158] The above is the complete description of the user interface display system.

[0159] The disclosed embodiments also provide a user interface display method, which can be executed by a terminal device integrated with the aforementioned user interface display system. This user interface display method can be applied to large-screen terminals, where "large screen" refers to a display screen with a larger screen size and higher resolution, such as an ultra-high-definition display screen larger than 40 inches and with 8K resolution.

[0160] FIG8 is a flow chart of a user interface display method provided by an embodiment of the present disclosure. As shown in FIG8 , the method includes steps S11 to S15. Specifically,

[0161] S11: Capture a scene image and detect whether a target object exists in the scene image.

[0162] It should be noted that the specific execution process of step S11 can refer to the above description of the image acquisition module 101 and the detection unit 121 in the system control module 102 in the user interface display system, and the repeated parts will not be repeated.

[0163] S12: If a target object exists, track the target object and determine the current position of the target object in the scene.

[0164] It should be noted that the specific execution process of step S12 can refer to the above description of the information determination unit 122 in the system control module 102, and the repeated parts will not be repeated.

[0165] S13. Determine a current coordinate point of the target object mapped from the current position to the interface coordinate system according to the current position of the target object in the scene and a pre-established mapping relationship between the scene coordinate system and the interface coordinate system of the user interface.

[0166] It should be noted that the specific execution process of step S13 can refer to the above description of the information determination unit 122 in the system control module 102, and the repeated parts will not be repeated.

[0167] S14. Generate update information of each interface component in the user interface according to the current coordinate point and predefined auxiliary parameter information.

[0168] It should be noted that the specific execution process of step S14 can refer to the above description of the update information generating unit 123 in the system control module 102, and the repeated parts will not be repeated.

[0169] S15. Display each interface component according to the updated information.

[0170] It should be noted that the specific execution process of step S15 can refer to the above description of the display module 103, and the repeated parts will not be repeated.

[0171] In some embodiments, the predefined auxiliary parameter information includes auxiliary positioning points on a predefined interface coordinate system and size ratio coefficients of each interface component.

[0172] Regarding step S11, specifically, the target object includes a person. Face recognition is performed on the scene image to determine whether the target object exists. It should be noted that the processing process can refer to the specific implementation of the detection unit 121 above, and the repeated parts will not be repeated.

[0173] In step S12, specifically, the current position includes the target key points. If facial recognition determines the presence of the target object, the target object is tracked and the target key points in the face are determined based on the target object's facial features. It should be noted that the specific implementation of the information determination unit 122 described above can be found in this processing process, and the repeated parts will not be repeated here.

[0174] Regarding step S13, specifically, the current coordinate point corresponding to the target key point is determined based on the mapping relationship between the target key point and the scene coordinate system and the interface coordinate system. It should be noted that the processing process can refer to the specific implementation of 123 above, and the repeated parts will not be repeated.

[0175] Regarding step S14, specifically, based on the current coordinate point, the auxiliary positioning point and the size ratio coefficient of each interface component, the update information of each interface component is generated. It should be noted that the processing process can refer to the specific implementation of the update information generation unit 123 above, and the repeated parts are not repeated here.

[0176] In some embodiments, the user interface display method further includes determining an auxiliary positioning point in the interface coordinate system based on the initial position of the tracked target object mapped to an initial coordinate point of the interface coordinate system and the initial positions of each interface component in the user interface; and determining a size scale factor of each interface component based on the auxiliary positioning point, the initial coordinate point, and the initial positions of each interface component. It should be noted that the specific implementation of the first pre-processing unit described above can be referred to for this processing process, and the repeated parts will not be repeated here.

[0177] In some embodiments, the user interface includes multiple layers, and each layer reuses the same interface coordinate system; the multi-layer user interface includes a foreground layer and a background layer, and at least one intermediate layer located between the foreground layer and the background layer; the interface components include multiple.

[0178] Specifically, the user interface display method further includes displaying different interface components in the foreground layer, the middle layer and the background layer. It should be noted that the processing process can refer to the specific implementation of the above-mentioned interface control unit, and the repeated parts will not be repeated.

[0179] In some embodiments, step S14 specifically includes receiving a component selection instruction for an interface component, replacing the size scale factor of the currently selected interface component with the target scale factor, and generating update information for each interface component based on the current coordinate point, the replaced target scale factor, and the auxiliary positioning point on the predefined interface coordinate system. It should be noted that the specific implementation of the update information generation unit 123 described above can be referenced for this processing process, and the repeated parts will not be repeated here.

[0180] In some embodiments, step S14 specifically includes receiving a component selection instruction for an interface component, obtaining predefined size scale coefficients for each interface component, replacing the size scale coefficient of the currently selected interface component with the size scale coefficient of the interface component located in the foreground layer if the currently selected interface component is not located in the foreground layer of the user interface, and generating update information for each interface component based on the current coordinate point, the replaced size scale coefficient, and the auxiliary positioning point on the predefined interface coordinate system. It should be noted that this processing process can refer to the specific implementation of the update information generation unit 123 described above, and the repeated parts will not be repeated here.

[0181] In some embodiments, with respect to step S12, specifically, the current position includes a target key point and a depth distance reference point. Information determination unit 122 is specifically configured to, if facial recognition determines the presence of a target object, track the target object and determine the target key point on the face based on the target object's facial features; identify the target object's shoulder features, and, based on the positions of the two shoulders in the scene coordinate system, use the midpoint between the two shoulders as the depth distance reference point. It should be noted that this processing process can be referenced to the specific implementation of information determination unit 122 described above, and any repetitive details will not be repeated here.

[0182] Specifically, step S12 determines the component scaling factor based on the depth distance reference point and the target key point. Update information for each interface component is generated based on the component scaling factor, the current coordinate point, the auxiliary positioning point in the predefined interface coordinate system, and the size ratio coefficient of each interface component. It should be noted that the specific implementation of the update information generation unit 123 described above can be found in this processing process, and the repeated parts will not be repeated here.

[0183] In some embodiments, with respect to step S12, specifically, if face recognition determines the presence of the target object, pupil features in the face are identified, and based on the positions of the two pupils in the scene coordinate system, the midpoint between the two pupils is used as the target key point. It should be noted that the specific implementation of the information determination unit 122 described above can be referenced for this processing, and the repeated parts will not be repeated here.

[0184] In some embodiments, the user interface includes at least one layer, the at least one user interface layer includes a background layer; the interface component includes a background image;

[0185] Specifically, the user interface display method further includes controlling the display module 103 to display a background image on the background layer. It should be noted that the processing process can refer to the specific implementation of the above-mentioned interface control unit, and the repeated parts will not be repeated.

[0186] In some embodiments, the predefined auxiliary parameter information further includes predefined size information of a visual window of the user interface, a predefined magnification factor of the background image, and size information of the background image;

[0187] Regarding step S11, specifically, the target object includes a person; face recognition is performed on the scene image to determine whether the target object exists. It should be noted that the processing process can refer to the specific implementation of the detection unit 121 above, and the repeated parts will not be repeated.

[0188] In step S12, specifically, the current position includes the target key points. If facial recognition determines the presence of the target object, the target object is tracked and, based on the target object's facial features, the target key points within the face are determined. It should be noted that the detailed implementation of the information determination unit 122 described above can be found in this processing, and any repetitive details will not be repeated here.

[0189] Regarding step S13, specifically, the current coordinate point corresponding to the target key point is determined based on the mapping relationship between the target key point and the scene coordinate system and the interface coordinate system. It should be noted that the processing process can refer to the specific implementation of the information determination unit 122 above, and the repeated parts will not be repeated.

[0190] Step S14 also includes determining an auxiliary reference point that is centrosymmetric about the geometric center of the visual window based on the current coordinate point and the size information of the visual window; and generating background image update information based on the size information of the visual window, the magnification factor of the background image, the auxiliary reference point, and the size information of the background image. It should be noted that the specific implementation of the update information generation unit 123 described above can be referred to for this processing process, and the repeated parts will not be repeated here.

[0191] In some embodiments, the user interface display method further includes determining the size information of the background image based on the size information of the visual window and the magnification factor of the background image. It should be noted that the processing process can refer to the specific implementation of the second pre-processing unit described above, and the repeated parts will not be repeated.

[0192] Figure 9 is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. As shown in Figure 9, an embodiment of the present disclosure provides a computer device comprising: one or more processors 901, a memory 902, and one or more I / O interfaces 903. The memory 902 stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the user interface display method described in the above embodiment; one or more I / O interfaces 903 are connected between the processor 901 and the memory 902 and are configured to implement information exchange between the processor 901 and the memory 902.

[0193] Among them, the processor 901 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 902 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically such as SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EPROM), and flash memory (FLASH); the I / O interface (read-write interface) 903 is connected between the processor 901 and the memory 902, and can realize information exchange between the processor 901 and the memory 902, including but not limited to a data bus (Bus).

[0194] In some embodiments, the processor 901 , the memory 902 , and the I / O interface 903 are connected to each other via a bus 904 , and further connected to other components of the computing device.

[0195] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium is further provided. The non-transitory computer-readable storage medium stores a computer program, wherein when the program is executed by a processor, the steps of the user interface display method in any of the above embodiments are implemented.

[0196] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the system of the present disclosure are executed.

[0197] It should be noted that the computer non-transitory readable storage medium shown in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (Erasable EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any non-transitory computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code contained on a non-transitory computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0198] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the aforementioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two connected boxes can actually represent execution in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0199] It is understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present disclosure, and the present disclosure is not limited thereto. Those skilled in the art may make various modifications and improvements without departing from the spirit and substance of the present disclosure, and such modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A user interface display system, comprising an image acquisition module, a system control module, and a display module; The image acquisition module is configured to acquire scene images; The system control module is configured to detect whether a target object exists in the scene image, and if the target object exists, track the target object and determine the current position of the target object in the scene; Determining, based on the current position of the target object in the scene and a pre-established mapping relationship between the scene coordinate system and the interface coordinate system of the user interface, a current coordinate point mapped to the interface coordinate system; and generating update information for each interface component in the user interface based on the current coordinate point and pre-defined auxiliary parameter information; The display module is configured to display each of the interface components according to the update information.

2. The user interface display system according to claim 1, wherein: The system control module includes a detection unit, an information determination unit and an update information generation unit; The detection unit is configured to detect whether there is a target object in the scene image; The information determining unit is configured to track the target object and determine a current position of the target object in the scene when the target object exists; Determining a current coordinate point of the target object mapped to the interface coordinate system based on the current position of the target object in the scene and a pre-established mapping relationship between the scene coordinate system and the interface coordinate system; The update information generating unit is configured to generate update information of each interface component in the user interface according to the current coordinate point and predefined auxiliary parameter information.

3. The user interface display system according to claim 2, wherein: The predefined auxiliary parameter information includes predefined auxiliary positioning points on the interface coordinate system and size ratio coefficients of each of the interface components; The target object includes a person; the detection unit is specifically configured to perform face recognition on the scene image to determine whether there is a target object; The current position includes a target key point; The information determination unit is specifically configured to track the target object and determine target key points in the face based on the facial features of the target object when the face recognition determines that the target object exists; Determining a current coordinate point corresponding to the target key point according to a mapping relationship between the target key point and the scene coordinate system and the interface coordinate system; The update information generating unit is configured to generate update information of each of the interface components according to the current coordinate point, the auxiliary positioning point and the size ratio coefficient of each of the interface components.

4. The user interface display system according to claim 3, wherein: The system control module further includes a first pre-processing unit; The first preprocessing unit is configured to determine an auxiliary positioning point on the interface coordinate system based on the initial position of the tracked target object mapped to the initial coordinate point of the interface coordinate system and the initial position of each interface component in the user interface; and determine the size ratio coefficient of each interface component based on the auxiliary positioning point, the initial coordinate point and the initial position of each interface component.

5. The user interface display system according to claim 2, wherein: The user interface comprises multiple layers, and each layer reuses the same interface coordinate system; the multiple layers of the user interface comprise a foreground layer and a background layer, and at least one intermediate layer located between the foreground layer and the background layer; the interface components comprise multiple layers; The system control module also includes an interface control unit; The interface control unit is configured to control the display module to display different interface components on the foreground layer, the middle layer, and the background layer.

6. The user interface display system according to claim 5, wherein: The update information generation unit is specifically configured to receive a component selection instruction of the interface component, replace the size scale coefficient of the currently selected interface component with the target scale coefficient, and generate update information for each of the interface components based on the current coordinate point, the replaced target scale coefficient and the pre-defined auxiliary positioning point on the interface coordinate system.

7. The user interface display system according to claim 5, wherein: The update information generation unit is specifically configured to receive a component selection instruction of the interface component, obtain a predefined size ratio coefficient of each of the interface components; and when a currently selected interface component is not located in the foreground layer of the user interface, replace the size ratio coefficient of the currently selected interface component with the size ratio coefficient of the interface component located in the foreground layer; Update information of each of the interface components is generated according to the current coordinate point, the replaced size ratio coefficient, and the predefined auxiliary positioning point on the interface coordinate system.

8. The user interface display system according to claim 2, wherein: The current position includes a target key point and a depth distance reference point; the information determination unit is specifically configured to track the target object and determine the target key point in the face based on the facial features of the target object when the face recognition determines that the target object exists; Identify the shoulder features of the target object, and use the middle point between the two shoulders as a depth distance reference point based on the positions of the two shoulders in the scene coordinate system; The update information generation unit is specifically configured to determine the component scaling factor based on the depth distance reference point and the target key point; and generate update information for each interface component based on the component scaling factor, the current coordinate point, the pre-defined auxiliary positioning point on the interface coordinate system, and the size ratio coefficient of each interface component.

9. The user interface display system according to claim 3 or 8, wherein: The information determination unit is specifically configured to identify pupil features in the face when face recognition determines that the target object exists, and to use the middle position point of the two pupils as the target key point based on the positions of the two pupils in the scene coordinate system.

10. The user interface display system according to claim 2, wherein: The user interface comprises at least one layer, wherein the at least one layer of the user interface comprises a background layer; the interface component comprises a background image; The system control module further includes an interface control unit; the interface control unit is configured to control the display module to display a background image on the background layer.

11. The user interface display system according to claim 10, wherein: The predefined auxiliary parameter information further includes predefined size information of the visual window of the user interface, a predefined magnification factor of the background image, and size information of the background image; The target object includes a person; the detection unit is specifically configured to perform face recognition on the scene image to determine whether there is a target object; The current position includes a target key point; The information determination unit is specifically configured to track the target object and determine target key points in the face based on the facial features of the target object when the face recognition determines that the target object exists; Determining a current coordinate point corresponding to the target key point according to a mapping relationship between the target key point and the scene coordinate system and the interface coordinate system; The update information generating unit is further configured to determine, based on the current coordinate point and the size information of the visual window, an auxiliary reference point that is centrosymmetric about the geometric center of the visual window with respect to the current coordinate point; Update information of the background image is generated according to the size information of the visual window, the magnification factor of the background image, the auxiliary reference point, and the size information of the background image.

12. The user interface display system according to claim 11, wherein: The system control module further includes a second pre-processing unit; The second pre-processing unit is configured to determine the size information of the background image according to the size information of the visual window and the magnification factor of the background image.

13. The user interface display system according to claim 1, wherein: The interface component includes at least one of a control, a status bar, a card bar, and a background image.

14. A user interface display method, wherein: include: Capturing a scene image and detecting whether a target object exists in the scene image; In the case where the target object exists, tracking the target object and determining a current position of the target object in the scene; Determining a current coordinate point of the target object mapped to the interface coordinate system based on the current position of the target object in the scene and a pre-established mapping relationship between the scene coordinate system and the interface coordinate system of the user interface; Generate update information of each interface component in the user interface according to the current coordinate point and predefined auxiliary parameter information; Each of the interface components is displayed according to the update information.

15. A computer device, wherein: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the user interface display method as described in claim 14 are performed.

16. A computer non-transitory readable storage medium, wherein: The computer non-transitory readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the user interface display method according to claim 14 .