A method and apparatus for UI stylization based on extended reality

Abstract

The present specification relates to the technical field of image processing, and particularly relates to a UI stylization method and device based on extended reality, the method comprising: acquiring a scene in a user's field of view and a UI element to be stylized; generating a stylization description instruction for the UI element to be stylized according to the scene and the UI element to be stylized, the stylization description instruction being used to integrate the UI element to be stylized into a visual expression of an extended reality scene; determining each single target UI element in the stylization description instruction according to the stylization description instruction; and rearranging each single target UI element to generate a stylized UI component. According to the present scheme, the visual performance of the UI to be stylized is adjusted in real time according to the specific style of the XR scene, without interrupting the user's experience in the immersive scene. Through machine learning and deep learning technology, the model is trained to recognize and adapt to different scene styles, so that the UI can be naturally integrated into the user's immersive environment.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of image processing, in particular to a UI stylization method and device based on extended reality. BACKGROUND

[0002] When a user uses an extended reality (XR) device to enter an immersive scene (for example, the user plays an XR game), the user still needs to receive prompt information outside the scene in which the user is located, such as important information prompts, device power messages, and prompts of other applications. However, the prompt information outside the scene in which the user is located only has the original default style, which is somewhat jarring in the immersive scene in which the user is currently located, and strong prompts can even cause the user's experience to be interrupted. Therefore, it is necessary to stylize the prompt information outside the scene in which the user is located to the scene in which the user is currently located, to achieve stylization of the prompt information.

[0003] In a conventional UI stylization scheme, the SDK interface is relied on, and there are problems of separation of functionality and visual style and insufficient stylization. SUMMARY

[0004] To solve the problem of poor experience caused by mismatch between prompt information and immersive application style in the prior art, the embodiments of the present specification provide a UI stylization method and device based on extended reality, which comprises: acquiring an extended reality scene in a user's field of view and a UI element to be stylized; generating a stylization description instruction for the UI element to be stylized according to the extended reality scene and the UI element to be stylized, the stylization description instruction being used to integrate the UI element to be stylized into a visual representation of the extended reality scene; determining each single stylized target UI element in the stylization description instruction according to the stylization description instruction; and rearranging each single stylized target UI element to generate a stylized UI component.

[0005] According to an aspect of the embodiments of the present specification, generating a stylization description instruction for the UI element to be stylized according to the extended reality scene and the UI element to be stylized comprises: inputting the visual representation corresponding to the extended reality scene and the UI element to be stylized into a large language model to obtain a stylization description instruction output by the large language model; and wherein the large language model is obtained by pre-training.

[0006] According to an aspect of some embodiments of the present disclosure, determining each single stylized target UI element in the stylized description instruction according to the stylized description instruction comprises: inputting the stylized description instruction into a text-image model to obtain a single stylized target UI element output by the text-image model; and the text-image model is obtained by training an initial text-image model by using a stylized sample description instruction.

[0007] According to an aspect of some embodiments of the present disclosure, determining each single stylized target UI element in the stylized description instruction according to the stylized description instruction comprises: converting the stylized description instruction into corresponding code; and determining corresponding UI elements in the stylized description instruction according to the code to obtain a plurality of single stylized target UI elements.

[0008] According to an aspect of some embodiments of the present disclosure, the text-image model is obtained by training in the following manner: obtaining a training sample set, wherein the training sample set comprises: a sample text and a label UI element corresponding to the sample text; inputting the sample text into an initial text-image model to obtain a single predicted UI element output by the initial text-image model; constructing a loss function according to the label UI element and the predicted UI element, and iteratively updating parameters of the initial text-image model based on the loss function until the loss function converges to a preset threshold to obtain the text-image model.

[0009] According to an aspect of some embodiments of the present disclosure, after obtaining the text-image model and the single stylized target UI element output by the text-image model, the method further comprises: determining a block image-text loss function according to image features of each target UI element and text features corresponding to the target UI element; determining a visual feature extraction loss function according to visual features of a target image and visual features of a scene image in the visual expression, wherein the target image is obtained by combining all target UI elements; determining a scene consistency loss function according to features of the target image and scene descriptions of the scene image; and guiding at least one of a large language model and a text-image model to fine-tune according to at least one of the block image-text loss function, the visual feature extraction loss function and the scene consistency loss function.

[0010] According to an aspect of some embodiments of the present disclosure, the large language model further outputs an arrangement layout of the scene image and the UI element to be stylized, and the rearranging each single stylized target UI element to generate a stylized UI component comprises: rearranging each single stylized target UI element according to the arrangement layout of the scene image and the UI element to be stylized to generate a stylized UI component.

[0011] According to one aspect of the embodiments of this specification, the method further includes:

[0012] The total loss function is constructed using the following formula:

[0013] + + ;

[0014] in, , , They represent the weights of the block image and text loss function, the scene consistency loss function, and the normalized visual feature extraction loss function respectively; represents the block image and text loss function, represents the scene consistency loss function, Represents the normalized visual feature extraction loss function.

[0015] An embodiment of the present specification provides a UI stylization device based on augmented reality, the device comprising: an acquisition unit for acquiring an augmented reality scene and a UI element to be stylized in a user's field of view; a first generation unit for generating a stylized description instruction for the UI element to be stylized based on the augmented reality scene and the UI element to be stylized, the stylized description instruction being used to integrate the UI element to be stylized into the visual expression of the augmented reality scene; a determination unit for determining each individual target UI element in the stylized description instruction based on the stylized description instruction; and a second generation unit for rearranging each individual target UI element to generate a stylized UI component.

[0016] An embodiment of this specification also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the UI stylization method based on augmented reality is implemented.

[0017] The embodiments of this specification also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for UI stylization based on augmented reality is implemented.

[0018] This solution adjusts the UI's visual presentation in real time based on the specific style of the XR scene, without interrupting the user's immersive experience. By leveraging machine learning and deep learning technologies, the model is trained to recognize and adapt to different visual styles, allowing the UI to naturally blend into various immersive environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only constitute some embodiments of the present specification, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0020] Figure 1 A flowchart of a UI stylization method based on extended reality according to an embodiment of the present specification is shown.

[0021] Figure 2 A flowchart of a method for determining a single target UI element according to an embodiment of the present specification is shown.

[0022] Figure 3 A stylized UI effect diagram according to an embodiment of the present specification is shown.

[0023] Figure 4 A flowchart of a method for training a text image model according to an embodiment of the present specification is shown.

[0024] Figure 5 A flowchart of a method for constructing a loss function for model fine-tuning according to an embodiment of the present specification is shown.

[0025] Figure 6 A flowchart of a UI stylization method based on an extended reality scene according to an embodiment of the present specification is shown.

[0026] Figure 7 A structural diagram of a UI stylization device based on extended reality according to an embodiment of the present specification is shown.

[0027] Figure 8 A diagram of a UI to be stylized according to an embodiment of the present specification is shown.

[0028] Figure 9 A diagram of a scene image in which a user is playing a game according to an embodiment of the present specification is shown.

[0029] Figure 10 A structural diagram of a computer device according to an embodiment of the present specification is shown.

[0030] Explanation of drawing symbols:

[0031] 701, acquisition unit;

[0032] 702, first generation unit;

[0033] 703, determination unit;

[0034] 704, second generation unit;

[0035] 1002, computer device;

[0036] 1004, processor;

[0037] 1006, memory;

[0038] 1008, drive mechanism;

[0039] 1010, input / output module;

[0040] 1012, input device;

[0041] 1014, output device;

[0042] 1016, presentation device;

[0043] 1018, graphical user interface;

[0044] 1020, network interface;

[0045] 1022, communication link;

[0046] 1024, communication bus. DETAILED DESCRIPTION

[0047] In order to make the technical solutions in the specification better understood by the person skilled in the art, the technical solutions in the specification will be described clearly and completely in the following with reference to the drawings in the specification. Obviously, the described embodiments are only some of the embodiments of the specification, not all. Based on the embodiments in the specification, all other embodiments obtained by the person skilled in the art without making creative efforts are within the protection scope of the specification.

[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the specification and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be exchanged under appropriate circumstances, so that the embodiments of the specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product or equipment including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or equipment.

[0049] This specification provides method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many orderings and does not represent the only execution order. When a system or device product is actually executed, the method can be executed in the order shown in the embodiments or the drawings or in parallel.

[0050] It should be noted that the UI stylization method and device based on augmented reality in this specification can be used in the field of image processing and the field of natural language processing. This specification does not limit the application field of the UI stylization method and device based on augmented reality.

[0051] Figure 1 The flowchart of a UI stylization method based on augmented reality according to an embodiment of this specification is shown, which specifically includes the following steps:

[0052] Step 101: Obtain an augmented reality scene and UI elements to be stylized in the user's field of view.

[0053] In this specification, the scene in the user's field of view is obtained based on the application or software currently used and operated by the user using an extended reality device (such as an XR device). In this specification, the scene in the user's field of view can include a variety of different visual expressions, such as scene images, scene text, 3D objects, etc. Among them, this can be achieved by recording a video presented in the user's field of view when the user uses the XR device and intercepting one or more frames of the video as the scene image; or intercepting the scene image in the user's field of view when the user uses the XR device at a preset time point at that time; further, it is also possible to intercept the scene image in the user's field of view when the user uses the XR device at that time point at a random time point.

[0054] In the embodiments of this specification, the scene in the user's field of view is related to the application, software, or tool currently being used by the user on the XR device. Specifically, when the user uses the XR device to play a game, the scene in the user's field of view is an image related to the game scene; when the user uses the social software on the XR device to chat, the scene in the user's field of view is an image related to the chat dialog box; when the user uses the XR device to browse economic information on a news website, the scene in the user's field of view is an image related to the news report.

[0055] In the embodiments of the present specification, the to-be-styled UI element includes: a to-be-styled UI for prompting a user or having a notification function, a to-be-styled UI for performing an operation function, a to-be-styled UI having a specific function, and the like. Among them, the to-be-styled UI element has various UI elements in its content, and the to-be-styled UI element usually has a system default style. If the content of the to-be-styled UI with the system default style is directly pushed to the user currently in the immersive scene, the prompt from outside the scene (for example, from other applications) is inevitably abrupt, the content of the to-be-styled UI does not match the immersive application style, and in particular, the prompt of strong reminder can even cause the user's experience to be interrupted. Therefore, in the present specification, it is necessary to convert the to-be-styled UI element into a new styled prompt UI element similar in style to the scene in the user's field of view, so as to ensure that the new styled UI element is naturally integrated into the visual style of the scene in the user's field of view.

[0056] As Figure 8 The to-be-styled UI is shown in the upper left corner of the figure. The upper right corner has a purple close button (“X”). In the upper middle of the to-be-styled UI, there is a reminder message: “Don't forget to do some exercise today!”, next to which there is a weightlifting emoji. Below the reminder message, there are two interactive buttons: “Go do it!” (black background and white text, emphasizing immediate action); “Remind me later” (gray background and black text, providing an option for later reminders).

[0057] Step 102, generating a styled description instruction for the to-be-styled UI element according to the extended reality scene and the to-be-styled UI element, the styled description instruction being used to integrate the to-be-styled UI element into the visual expression of the extended reality scene.

[0058] In the present specification, the system identifies the scene style obtained in step 101 and the to-be-styled UI element through a large language model, generates a styled description instruction matching the scene style, and modifies the to-be-styled UI element using the styled description instruction. The styled instruction is used to integrate the to-be-styled UI element into the scene image. The styled description instruction includes: a requirement for designing a new UI for the visual expression in the extended reality scene in the user's current field of view, and the requirement needs to include: various UI elements (including: the shape, lines, colors, text, and brightness of the UI, etc.) and the layout form, position, etc. of each UI element in the scene image.

[0059] For example, when a user is playing an XR game with a science fiction theme using an XR device, receives a UI element to be stylized, this step needs to convert the style of the UI element to be stylized according to the science fiction theme of the XR game.

[0060] Specifically, the visual expression corresponding to the scene in step 101 and the UI element to be stylized are input into a large language model to obtain a stylized description instruction output by the large language model; wherein the large language model is obtained by pre-training. In this specification, the scene image in the visual expression corresponding to the scene and the UI to be stylized can be input into the large language model to obtain the stylized description instruction.

[0061] In this specification, the stylized description instruction includes stylizing each element of the UI respectively and generating detailed text descriptions. Wherein the stylized description instruction is carried out in the system, and the user is unaware.

[0062] As shown in Figure 8 The large language model identifies Figure 8 As shown in the schematic diagram of the UI element to be stylized, after the large language model identifies the UI element to be stylized, it converts and generates the following text content:

[0063] a. Title and close button:

[0064] In the top left corner of the UI to be stylized, there is a faded title "Exercise Reminder".

[0065] In the top right corner of the UI to be stylized, there is a purple close button ("X"), which the user can click to close the pop-up window.

[0066] b. Reminder information:

[0067] In the middle of the UI to be stylized, there is a reminder message: "Don't forget to do some exercise today!", Next to it is a weightlifting emoji, adding interest and visual appeal.

[0068] c. Interactive buttons:

[0069] The user can choose two buttons:

[0070] "Go do it!", The button is in a black background and uses white text, emphasizing immediate action.

[0071] "Remind me later", the button is in a gray background and uses black text, providing the option of reminding later.

[0072] The present specification generates a stylized description instruction through a large language model, generates a new stylized UI without being limited to a predefined structure, and ensures that the UI component naturally blends into the visual style of the user's scene in an immersive environment while retaining its core functionality and interactivity.

[0073] Figure 9 The figure shows a scene image of a user playing a game, which includes an extended reality scene in the user's field of view corresponding to step 101. After the large language model identifies the content and style of the immersive game scene and interprets the scene style, the generated text content is as follows:

[0074] The style of this VR scene has a strong futuristic and technological feel. Here are some features:

[0075] Color usage: mainly uses blue and purple neon lights to create a high-tech and mysterious atmosphere.

[0076] Geometric design: there are many circular and rectangular geometric patterns in the scene, giving a sense of precision and modernity.

[0077] Lighting effect: uses glowing rings and lines to enhance the three-dimensionality and technological feel of the scene.

[0078] Material and texture: the surface looks smooth and has a metallic texture, further emphasizing the futuristic design.

[0079] Overall, this scene gives a sense of being in a science fiction movie or a high-tech laboratory.

[0080] According to the above description, the content and style of the game scene obtained by the large language model are combined with the text content generated by the large language model after identifying the UI to be stylized, to generate a stylized description instruction for the game scene, as shown below:

[0081] Design a modern, high-tech user interface (UI) pop-up window, containing the following elements:

[0082] Title: Use a futuristic font to display "Sports Reminder" in a gradient blue color.

[0083] Close button: design a glowing purple "X" button that users can click to close the pop-up window.

[0084] Reminder information: a reminder message "Don't forget to do some sports today!" with a tech-style weightlifting emoji next to it to convey energy and vitality.

[0085] Interaction Buttons:

[0086] “Go For it! Go For it!”: Designed with glowing white text on a black background, the button has a subtle blue glow around the edge to emphasize immediate action.

[0087] “Remind me later”: Designed with a gray background and black text, the button has a faint halo around the edge to provide the option to remind me later.

[0088] Background: Use a dark, tech-themed background with abstract lines and shapes to enhance the futuristic feel. Ensure the overall design matches the style of a modern VR environment and use blue and purple neon lighting effects to create a high-tech and mysterious atmosphere.

[0089] Step 103: Determine, according to the stylized description instruction, each individual stylized target UI element in the stylized description instruction.

[0090] In this specification, the stylized description instruction may be a stylized description instruction in text form, or a corresponding code form.

[0091] In some embodiments of the present specification, if the stylized description instruction is in the form of natural language text, determining each individual target UI element in the stylized description instruction according to the stylized description instruction includes: inputting the stylized description image in the form of natural language text into a text image model (for example, a diffusion model), and obtaining a single target UI element output by the text image model corresponding to the stylized description; wherein the text image model is obtained by training an initial text image model through the stylized sample description instruction.

[0092] In some embodiments of this specification, if the stylized description instruction is in code form, by converting the stylized description instruction into code form, and determining a UI element among the UI elements to be stylized corresponding to the stylized description instruction according to the code, further, in the process of inputting the stylized description instruction into the text image model, the stylized description instruction can also be converted into code, and the code can be input into the text image model for prediction. For details, see Figure 2 describe.

[0093] In this specification, a text image model is a multimodal model capable of processing both images and text. It is used for matching images and text, and has functions such as text parsing, UI layout generation, and image rendering. In UI stylization scenarios, the text image model can be used for style transfer. Specifically, the text image model is used to match a known UI element to be stylized with a certain style. For example, a regular UI design can be compared with a "future" style or a "hand-drawn" style to optimize the image appearance of the UI design.

[0094] In the present specification, any one of the CLIP model, the ALIGN model can be combined with the Stable Diffusion model and any one of the GANs to construct a text image generation model for stylized UI. For the training process of the text image model, please refer to Figure 4 Description.

[0095] Step 104, rearranging each single stylized target UI element to generate a stylized UI component.

[0096] The plurality of single target UI elements obtained according to step 103 are not arranged in order. In this step, all single target UI elements can be rearranged and laid out according to the style of the extended reality scene in the user's field of view and the arrangement and layout of the target UI elements to be stylized to generate an overall stylized UI component, as shown in Figure 3 In an embodiment of the present specification, the single target UI elements are rearranged according to the preset arrangement and layout to generate a stylized UI component. The preset arrangement and layout includes a preset arrangement standard, a preset interface, or a template of a preset arrangement and layout. These preset arrangement and layouts can be automatically generated by a large language model. In some other embodiments of the present specification, the single target UI elements can also be rearranged according to the arrangement and layout of the UI elements of the historical scene similar to the extended reality scene in the user's field of view after being pre-processed. This step does not limit the basis for rearranging the target UI elements.

[0097] In this step, the stylized target UI elements are determined again according to the single stylized target UI elements, so that each stylized target UI element can be naturally integrated into the immersive environment of the user's game scene.

[0098] Figure 2 As shown in the method flowchart for determining a single target UI element, Figure 2 As shown in the method flowchart for determining a single target UI element,

[0099] Step 201, converting the stylized description instruction into corresponding code.

[0100] According to the style description instruction content, the text is described, and the modification code corresponding to the style of the UI element in the to-be-styled UI code is generated. Specifically, the styling description instruction is input into a model with instruction conversion function to obtain the code corresponding to the styling description instruction. Wherein, for each sentence of text in the styling description instruction content, the corresponding code is generated; or for each paragraph of text in the styling description instruction content, the corresponding code is generated. Wherein, the style of the UI element includes but is not limited to: different interactive state UI style, sound feedback, etc.

[0101] Step 202, according to the code, determine the corresponding UI element in the style description instruction, get the single target UI element after styling. In this step, the new styled UI element is rendered according to the modification code, and the single styled target UI element is generated. In this specification, the single target UI element includes but is not limited to: styled button, icon and text box, etc. The generated single target UI element conforms to the style attribute of the extended reality scene.

[0102] Figure 4 The figure shows a method flow chart for training a text image model according to an embodiment of the present specification, which specifically includes the following steps:

[0103] Step 401, obtain a training sample set, which includes: sample text and label UI element corresponding to the sample text.

[0104] In order to construct a text image model, the training sample set is prepared in advance. Wherein, the sample text in the training sample set can be code form data. Specifically, the code can be a complete code, or multiple sub-codes, which can be collected from open source projects, code repositories or online programming platforms. When the sample code is a complete code, the code corresponds to multiple UI elements, and when the sample code is multiple sub-codes, each sub-code corresponds to a UI element. Further, the training sample set also includes the label UI element corresponding to each sample code, which is used to indicate that the sample code describes a button, or the appearance attributes such as color, material and transparency of an icon.

[0105] Step 402, input the sample text into the initial text image model to obtain the single predicted UI element output by the initial text image model.

[0106] According to the foregoing step description, the initial text image can be CLIP model, ALIGN model, etc. Before inputting the sample text into the initial text image model, the sample text needs to be converted into an input format that the model can accept, such as word embedding or code vector. Further, the output layer of the initial text image model is designed to generate pixel values or feature vectors corresponding to UI elements.

[0107] According to the input sample text, the initial text-image model outputs a plurality of individual predicted UI elements. Before the initial text-image model is iterated and fine-tuned multiple times, the individual predicted UI elements output by the initial text-image model are quite different from the extended reality scene in terms of style and visual effect, and are not accurate enough. Therefore, the parameters and parameter weights in the initial text-image model need to be further adjusted.

[0108] At step 403, a loss function is constructed according to the label UI elements and the predicted UI elements. The parameters of the initial text-image model are iteratively updated based on the loss function until the loss function converges to a preset threshold, and a text-image model is constructed.

[0109] In this step, a suitable loss function is selected to evaluate the difference between the predicted UI elements generated by the model and the label UI elements. Specifically, a cross-entropy loss function can be used to calculate the loss value of the predicted UI elements and the label UI elements, and an optimizer is used to iteratively update the parameters and weights of the initial text-image model according to the loss value to minimize the loss function until the loss function converges to a preset threshold, and the text-image model is considered to be constructed.

[0110] Figure 5 The method flowchart shown is one embodiment of the present specification for constructing a loss function for model fine-tuning. In the present specification, after the text-image model is constructed, the models can continue to be fine-tuned to improve the proximity of the predicted UI elements of the model to the extended reality scene. The present method specifically includes the following steps:

[0111] At step 501, a block image-text loss function is determined according to the image features of each target UI element and the text features corresponding to the target UI element.

[0112] In the present specification, since a complete UI component is usually composed of discrete elements (such as buttons, backgrounds, etc.), by modifying the block image-text loss in the target UI element, different UI elements in the plurality of target UI elements are preferentially processed, thereby more finely controlling part of the UI elements (for example, highlighting only one button), and thus different styles can be applied to different parts of the UI elements.

[0113] In the present specification, the formula of the block image-text loss function is as follows:

[0114] ;

[0115] wherein, represents the block image-text loss function, represents a feature extractor of the text-image model, and the output of the feature extractor is a feature vector; an i-th target UI element represented by a text image model output, an image feature representing the i-th target UI element; a literal text or code text corresponding to the i-th target UI element, a text feature representing the text corresponding to the i-th target UI element, a weight representing the i-th target UI element. In the block image-text loss function, the image feature of the target UI element and the text feature corresponding to the i-th target UI element are extracted by the feature extractor of the text image model. The block image-text loss function extracts the image feature of the target UI element and the corresponding text description feature through the feature extractor, and measures the difference between the two. Cosine similarity for measuring the similarity between the image feature and the text feature, converts the similarity into a difference measure. The block image-text loss function reflects the difference between the image feature of the target UI element and the corresponding text description feature, and the weight for adjusting the contribution of different UI elements to the total loss.

[0116] Step 502, determining a visual feature extraction loss function according to the visual features of the target image and the visual features of the scene image in the visual representation.

[0117] In this step, the visual features (such as color, texture) in the scene image in the field visual representation are extracted by the feature extraction network or visual encoder and passed to the target image, and then the difference between the visual features in the scene image and the visual features of the target image is calculated to form the visual feature loss function. Wherein, the scene image is denoted as , and the target image is denoted as . The target image is an image composed of all single target UI elements after all the above steps.

[0118] The formula of the visual feature extraction loss function is as follows:

[0119] ( encoder ( ;

[0120] ;

[0121] wherein, denotes the visual feature extraction loss function, denotes a visual encoder function for extracting high-level visual features from an input image, which can represent any model capable of effectively extracting image features, such as convolutional neural network (CNN), Vision Transformer (ViT), etc. a visual feature representing an image composed of all single target UI elements, a visual feature representing an image composed of all single target UI elements; may represent a maximum value or a preset value observed during the training process. a maximum value or a preset value observed during the training process. represents a normalized visual feature extraction loss function. In this specification, the visual feature extraction loss function is intended to reflect the difference between the overall visual feature of the image composed of all single target UI elements and the overall visual feature of the scene image. After determining the visual feature extraction loss function value according to the visual feature of the UI prediction image and the visual feature of the scene image, it is determined whether the value of the loss function meets the preset threshold. If not, continue to fine-tune the text image model and the large language model.

[0122] Step 503, determining a scene consistency loss function according to the features of the target image and the scene description of the scene image.

[0123] Specifically, the formula of the scene consistency loss is as follows:

[0124] ), ( ;

[0125] wherein, represents a scene consistency loss function, represents a feature extractor, and the output is a feature vector; represents a text description of the scene image, represents a text feature extracted from the scene image, represents the entire target image, represents an image feature extracted from the target image. Cosine similarity ), ( is used to measure the similarity between the text feature and the image feature, converts the similarity into a difference measure.

[0126] In this step, the text feature extracted from the scene image is obtained, which reflects the text description related to the overall style or overall theme of the scene image, such as "soft forest theme" or "modern technology" and the like; the image feature extracted from the complete target image. In this specification, the scene consistency loss function is intended to reflect the difference between the image feature of the target image and the text feature of the scene image.

[0127] The scene consistency loss function can make the finally generated target image visually consistent with the original scene image after stylization, so that the stylized target image matches the scene description of the scene image.

[0128] In the embodiments of the present specification, the method further comprises:

[0129] The total loss function is constructed by using the following formula:

[0130] + + ;

[0131] wherein, , , respectively represent the weights of the block graph-text loss function, the scene consistency loss function, and the normalized visual feature extraction loss function; represents the block graph-text loss function, represents the scene consistency loss function, represents the normalized visual feature extraction loss function. The specific values of the three weights in the formula can be adjusted through experiments.

[0132] In the present specification, the scene stylization requirement may change, so a dynamic weight is introduced in the loss function to adjust the influence of each loss term in the loss function in real time. For example, when the scene visual feature is distinctive, the weight of the visual feature extraction loss can be increased, and when the scene style is simple, the weight of the content loss is increased to maintain the clarity of the UI.

[0133] The formula of the dynamic weight is as follows:

[0134] = =g ;

[0135] wherein, represents a function of adjusting the weight according to the scene complexity, =g represents a function of adjusting the weight according to the scene clarity.

[0136] Step 504, according to at least one of the block graph-text loss function, the scene consistency loss function and the normalized visual feature extraction loss function, guiding at least one of the large language model and the text image model to fine-tune.

[0137] In the present specification, the fine-tuning of the large language model by using the loss function comprises: fine-tuning or fine-tuning the large language model by using the loss function.

[0138] When the amount of data is large enough, all layers of the large language model are fine-tuned; when the amount of data is small, the top layer of the large language model is fine-tuned, reducing the consumption of computing resources, while still allowing the model to have sufficient flexibility to learn new tasks. In the embodiments of the present specification, the bottom layer of the large language model can also be fixed, and only the top layer of the large language model is fine-tuned, keeping the bottom layer feature representation learned by the large language model in the training phase unchanged, while enabling the top layer to better adapt to new tasks.

[0139] In some other embodiments of the present specification, the large language model can also be fine-tuned layer by layer from the bottom layer through the loss function until all layers of the large language model are fine-tuned, focusing on using continuous prompts (such as embedding vectors) to adjust the behavior of the large language model, rather than directly modifying the weights of the model.

[0140] In some embodiments of the present specification, fine-tuning of the text-image model includes adjusting the code input to the text-image model to obtain UI elements that better fit the style effects of the scene image, and then generating an image. Through a scene consistency loss function, the large language model and the text-image model are iteratively fine-tuned until the scene consistency loss function value converges to a preset threshold, thereby determining that the fine-tuning of the large language model and the text-image model is complete.

[0141] In some embodiments of the present specification, fine-tuning of the large language model using a visual feature extraction loss function includes freezing the parameters of the large language model and optimizing only a series of continuous task-specific vectors (i.e., prefixes) to achieve optimization tasks. Lightweight design avoids waste of storage and computing resources, while maintaining the performance of the large language model.

[0142] In the embodiments of the present specification, the large language model can be fine-tuned using a low-rank adapter method. For example, the low-rank adaptive (LoRA, Low-Rank Adaptation) method is used to freeze the weights of the large language model, and a trainable rank decomposition matrix is injected into each layer of the Transformer architecture, significantly reducing the number of trainable parameters in downstream tasks while maintaining model quality performance. The present specification can also use a dynamic low-rank adaptation (DyLoRA, Dynamic Low-Rank Adaptation) method to address the problems of LoRA blocks (such as fixed size and difficulty in rank optimization), introduce upper and lower projection matrices, and a mechanism for dynamically adjusting the rank. The training speed is faster, the performance almost does not decrease, and excellent performance is exhibited in a wider range of ranks. In the present specification, the fine-tuned large language model generates more intelligent and user habit-oriented stylized description instructions based on scene images and UI to be stylized.

[0143] For example, according toFigure 8 In the embodiment in the foregoing description, the stylized description instruction output by the large language model can be: "display a reminder information in the middle: 'Don't forget to do some sports today!', add a weightlifting emoji in a science and technology style to convey energy and vitality; and add an archery symbol in the game to fit the current game scene of the user". In the stylized description instruction, the addition of the weightlifting emoji and the archery symbol are intelligent results output by the large language model after fine-tuning.

[0144] In the embodiment of the present description, in addition to outputting the stylized description instruction for the scene image and the UI to be stylized, the large language model can also output the arrangement and layout of the scene image and the UI to be stylized through the learning of the model itself. The arrangement and layout of the extended reality scene and the UI elements to be stylized include: determining the area in which the UI to be stylized can be arranged in the scene image, the positions of the elements in the UI elements to be stylized in the UI to be stylized, the size of the scene image, and the like.

[0145] For example, it is determined that the close button ("X") in the UI to be stylized is located at the upper left corner or the upper right corner of the scene image, it is determined that the text in the UI to be stylized should be located at the blank of the scene image, or can be overlaid on the current scene image, the UI elements in the UI to be stylized should be located within the scene image area, but should not exceed the size of the scene image, the reminder information "Don't forget to do some sports today!" is located above the interactive button provided to the user, and the interactive button is set at the middle position or the area below the middle position of the system UI.

[0146] Therefore, after the arrangement and layout is determined, the plurality of single UI elements obtained in the foregoing are arranged according to the area of the scene image and the arrangement and layout order in the UI to be stylized, to obtain a complete stylized image, as shown in Figure 3 .

[0147] Figure 3 As shown in the figure, the upper left corner uses a future font to display "Sports Reminder" in a gradient blue color. The upper right corner sets a glowing purple "X" button, which can be clicked by the user to close the pop-up window. The middle position of the figure displays a reminder information: "Don't forget to do some sports today!", and a weightlifting emoji in a science and technology style beside it to convey energy and vitality. The lower third of the page sets interactive buttons: "Go For it!" and "Remind me". Figure 3Use dark, tech-style background, with abstract lines and shapes to enhance the futuristic feel. The overall design matches the modern VR scene style, using blue and purple neon light effects to create a high-tech and mysterious atmosphere.

[0148] In the present specification, before obtaining the scene in the user's field of view, the UI element to be stylized, and generating the stylized instruction, the method further comprises:

[0149] According to the scene in the user's field of view, it is judged whether the scene is a multi-task scene. In this step, by intercepting the image in the user's field of view, it is determined whether the scene image existing in the user's field of view involves multiple scenes. Specifically, the scene image in the user's field of view can be continuously collected within a certain time period, and it is judged whether multiple different style scene images appear within the time period.

[0150] If yes, use the default notification UI style. In this step, when the scene is a multi-task scene, the UI to be stylized does not need to be stylized according to different task scenes.

[0151] If no, the scene image is stylized. In this step, when the scene is a single-task scene, the UI to be stylized needs to be stylized according to the style of the scene, so as to execute the steps shown in Figures 1 to 2 and Figures 4 to 5 .

[0152] Figure 6 The flowchart shown in the present specification is a process of UI stylization based on an extended reality scene.

[0153] Figure 6 In the present specification, first, the marine-style XR scene content in the user's field of view and the UI to be stylized (the UI to be stylized is represented by A in the first subgraph in the figure) are obtained. The XR scene in the user's field of view shows the visual expression of marine organisms, including the image of a jellyfish and the 3D image of a shark, as well as the visual expression elements such as light effects, color tones, textures, and the like in the immersive marine environment scene. The marine-style XR scene content and the UI to be stylized (A) are input into the large language model, and the text in the UI to be stylized (A) and the image description in the XR scene are output to generate a stylized description instruction.

[0154] Figure 6In the second subgraph, the stylized description instruction obtained from the first graph is further converted into corresponding code, and the code is input into the text-image model to obtain the single target UI element after stylization in the stylized description instruction determined by the text-image model. Then, according to the arrangement and layout of various visual expressions in the XR scene, the stylized target UI element is rearranged according to the corresponding arrangement and layout to obtain the arranged stylized UI component shown in the third subgraph. The visual properties of the plurality of single target UI elements are adjusted as a whole to obtain the stylized UI component shown in the fourth subgraph. Such a stylized UI component is deployed in the XR scene in an immersive manner, and does not interrupt the immersive experience of the user in the current scene.

[0155] As shown in Figure 7 Fig. 1 is a structural schematic diagram of a mixed reality-based UI stylization device according to an embodiment of the present specification. The basic structure of the mixed reality-based UI stylization device is described in the figure, and the functional units and modules therein can be implemented in a software manner, or can be implemented by using general-purpose chips or specific chips to implement the mixed reality-based UI stylization. The device specifically includes:

[0156] The acquisition unit 701 is configured to acquire a scene in a field of view of a user and a UI element to be stylized.

[0157] The first generation unit 702 is configured to generate a stylized description instruction for the UI element to be stylized according to the scene and the UI element to be stylized. The stylized description instruction is used to integrate the UI element to be stylized into the scene.

[0158] The determination unit 703 is configured to determine each single target UI element in the stylized description instruction according to the stylized description instruction.

[0159] The second generation unit 704 is configured to rearrange each single target UI element to generate a stylized UI component.

[0160] As shown in Figure 10As shown, it is a schematic diagram of a computer device provided in an embodiment of this specification. The UI stylization method based on augmented reality described in this application can be applied to the computer device. The computer device 1002 may include one or more processors 1004, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 1002 may also include any memory 1006, which is used to store any kind of information such as code, settings, data, etc. Non-limitingly, for example, the memory 1006 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 1002. In one case, when the processor 1004 executes an associated instruction stored in any memory or combination of memories, the computer device 1002 can perform any operation of the associated instruction. The computer device 1002 also includes one or more drive mechanisms 1008 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, and the like.

[0161] Computer device 1002 may also include an input / output module 1010 (I / O) for receiving various inputs (via input devices 1012) and providing various outputs (via output devices 1014). A specific output mechanism may include a presentation device 1016 and an associated graphical user interface (GUI) 1018. In other embodiments, input / output module 1010 (I / O), input devices 1012, and output devices 1014 may not be included, and the computer device 1002 may simply function as a computer device on a network. Computer device 1002 may also include one or more network interfaces 1020 for exchanging data with other devices via one or more communication links 1022. One or more communication buses 1024 couple the components described above.

[0162] The communication link 1022 may be implemented in any manner, for example, via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1022 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0163] Corresponding to Figures 1 to 5 The method in this specification also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the above method are executed.

[0164] The embodiments of the present specification also provide a computer readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the method as shown in Figures 1 to 5

[0165] It should be understood that the size of the sequence number of the above-mentioned processes in various embodiments of the present specification does not mean the order of execution, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present specification.

[0166] It should also be understood that in the embodiments of the present specification, the term "and / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships. For example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in the present specification generally represents an "or" relationship between the front and rear associated objects.

[0167] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present specification can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present specification.

[0168] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0169] In several embodiments provided in the present specification, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, and the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displays or discussions can be indirect coupling or communication connection through some interfaces, devices or units, and can also be electrical, mechanical or other forms of connection.

[0170] ​The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0171] In addition, the functional units in the various embodiments of this specification may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0173] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, based on the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting this specification.

Claims

1. A UI stylization method based on augmented reality, characterized in that: The method comprises: Acquire an augmented reality scene and a UI element to be stylized in a user's field of view, wherein the augmented reality scene is acquired by using an augmented reality device; generating, according to the augmented reality scene and the UI element to be stylized, a stylized description instruction for the UI element to be stylized, wherein the stylized description instruction is used to integrate the UI element to be stylized into the visual expression of the augmented reality scene; Determining, according to the stylized description instruction, each individual stylized target UI element in the stylized description instruction, comprising: inputting the stylized description instruction into a text image model to obtain a single stylized target UI element output by the text image model; Determine a block image-text loss function based on the image features of each target UI element and the text features corresponding to the target UI element; Determining a visual feature extraction loss function based on visual features of a target image and visual features of a scene image in the visual representation, wherein the target image is obtained by combining all target UI elements; Determine the scene consistency loss function based on the image features of the target image and the text features of the scene image; guiding fine-tuning of the text image model according to at least one loss function among the block image-text loss function, the visual feature extraction loss function, and the scene consistency loss function; Rearrange the individual stylized target UI elements to generate stylized UI components.

2. The method according to claim 1, characterized in that Generating a stylized description instruction for the UI element to be stylized according to the augmented reality scene and the UI element to be stylized includes: Inputting the visual expression corresponding to the augmented reality scene and the UI elements to be stylized into the large language model to obtain the stylized description instructions output by the large language model; The large language model is obtained through pre-training.

3. The method according to claim 1, characterized in that Determining, according to the stylized description instruction, each individual stylized target UI element in the stylized description instruction includes: The text image model is obtained by training an initial text image model through stylized sample description instructions.

4. The method according to claim 3, characterized in that Determining, according to the stylized description instruction, each individual stylized target UI element in the stylized description instruction further includes: Converting the stylized description instructions into corresponding codes; The corresponding UI element in the stylized description instruction is determined according to the code to obtain a stylized single target UI element.

5. The method according to claim 3, characterized in that The text image model is trained in the following way: Obtaining a training sample set, the training sample set including: sample text and label UI elements corresponding to the sample text; Inputting the sample text into an initial text image model to obtain a single predicted UI element output by the initial text image model; A loss function is constructed according to the label UI element and the predicted UI element, and the parameters of the initial text image model are iteratively updated based on the loss function until the loss function converges to a preset threshold, thereby constructing a text image model.

6. The method according to claim 5, characterized in that After the text image model is constructed and the text image model outputs a single stylized target UI element, the method further includes: The large language model is guided to be fine-tuned according to at least one loss function among the block image and text loss function, the visual feature extraction loss function, and the scene consistency loss function.

7. The method according to claim 2, characterized in that The large language model further outputs an arrangement layout for the augmented reality scene and the UI elements to be stylized, including: rearranging each individual stylized target UI element according to the augmented reality scene and the arrangement layout of the UI elements to be stylized to generate a stylized UI component.

8. The method according to claim 1, characterized in that Rearrange each target UI element to generate a stylized UI component including: Rearrange each individual target UI element according to the preset arrangement layout to generate a stylized UI component.

9. The method according to claim 6, characterized in that The method further comprises: The total loss function is constructed using the following formula: + + ； in, , , They represent the weights of the block image and text loss function, the scene consistency loss function, and the normalized visual feature extraction loss function respectively; represents the block image and text loss function, represents the scene consistency loss function, Represents the normalized visual feature extraction loss function.

10. A UI stylization device based on augmented reality, characterized in that: The device comprises: an acquisition unit, configured to acquire an augmented reality scene and a UI element to be stylized in a user's field of view, wherein the augmented reality scene is acquired by using an augmented reality device; a first generating unit, configured to generate, based on the augmented reality scene and the UI element to be stylized, a stylized description instruction for the UI element to be stylized, wherein the stylized description instruction is used to integrate the UI element to be stylized into a visual expression of the augmented reality scene; a determining unit, configured to determine, based on the stylized description instruction, each individual stylized target UI element in the stylized description instruction, comprising: inputting the stylized description instruction into a text image model to obtain a single stylized target UI element output by the text image model; Determine a block image-text loss function based on the image features of each target UI element and the text features corresponding to the target UI element; Determining a visual feature extraction loss function based on visual features of a target image and visual features of a scene image in the visual representation, wherein the target image is obtained by combining all target UI elements; Determine the scene consistency loss function based on the image features of the target image and the text features of the scene image; guiding fine-tuning of the text image model according to at least one loss function among the block image-text loss function, the visual feature extraction loss function, and the scene consistency loss function; The second generating unit is configured to rearrange the individual stylized target UI elements to generate a stylized UI component.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images