User interface generation method and device and storage medium

Abstract

The invention discloses a user interface generation method and device and a storage medium. The method comprises the steps of obtaining environment information and body information of a user; determining a target projection area according to the three-dimensional space information of the environment where the user is located, the three-dimensional space information of the body of the user and the operation intention of the user; determining a layout cue word according to the target projection area and the user operation intention, and determining layout information of the user interface by using a first diffusion model according to the target projection area and the layout cue word; and determining a user interface cue word according to the user operation intention, and generating the user interface according to the target projection area, the layout information of the user interface and the user interface cue word by using a second diffusion model. By means of the scheme, the user interface is flexibly generated according to the user environment, then the generated user interface is projected to the target projection area and provided for the user to achieve the user operation intention, and the requirements of the user are greatly met.

Description

Technical Field

[0001] The present application relates to the technical field of Internet of Things, and in particular to a method for generating a user interface, a device for generating a user interface, a computer-readable storage medium and a computer program product. Background Art

[0002] The projector projects on a reserved area such as a white wall or a screen without a display, and can achieve a better projection effect. If the user needs to operate the projected image, he needs to use external devices such as a remote control or keyboard. With the development of technology, users can interact directly with the projected image without the help of external electronic devices such as a remote control or keyboard, achieving the effect of touch control, that is: the projector projects the user interface, and the user directly operates on the projected image to control it. However, if the projection area is not ideal, such as if there is contamination, obstruction or irregularity, the projection effect will be affected. In actual applications, it may be difficult to find a suitable area as the projection area in the user's area, and the projector cannot flexibly generate a high-quality user interface according to the on-site situation, and cannot meet user needs. Summary of the invention

[0003] In view of the above-mentioned prior art, an embodiment of the present invention discloses a method for generating a user interface, which can overcome the defect of poor projection effect caused by an undesirable projection area and achieve the purpose of flexibly generating a high-quality user interface according to actual conditions.

[0004] In view of this, an embodiment of the present application proposes a method for generating a user interface, the method comprising:

[0005] Acquire the user's environmental information and user's body information, wherein the environmental information includes the three-dimensional spatial information of the user's environment, and the user's body information includes the three-dimensional spatial information of the user's own body, and determine the target projection area according to the three-dimensional spatial information of the user's environment, the three-dimensional spatial information of the user's own body, and the acquired user operation intention;

[0006] Determining a layout prompt word according to the target projection area and the user operation intention, wherein the layout prompt word is prompt information describing the layout of the user interface; determining the layout information of the user interface according to the target projection area and the layout prompt word and using a first diffusion model;

[0007] A user interface prompt word is determined according to the user operation intention, wherein the user interface prompt word is prompt information describing the appearance of the user interface; and the user interface is generated according to the target projection area, the layout information of the user interface and the user interface prompt word and using a second diffusion model.

[0008] In view of the above-mentioned prior art, an embodiment of the present invention discloses a user interface generation device, which can overcome the defect of poor projection effect caused by an undesirable projection area and achieve the purpose of flexibly generating a high-quality user interface according to actual conditions.

[0009] In view of this, an embodiment of the present application proposes a device for generating a user interface, the device comprising:

[0010] a detection module, configured to obtain the user's environmental information and user body information, wherein the environmental information includes the three-dimensional spatial information of the user's environment, and the user's body information includes the three-dimensional spatial information of the user's own body, and determine the target projection area according to the three-dimensional spatial information of the user's environment, the three-dimensional spatial information of the user's own body, and the obtained user operation intention;

[0011] a layout generation module, configured to determine a layout prompt word according to the target projection area and the user operation intention, wherein the layout prompt word is prompt information describing the layout of the user interface; and determine the layout information of the user interface according to the target projection area and the layout prompt word and using a first diffusion model;

[0012] The interface generation module is used to determine a user interface prompt word according to the user operation intention, wherein the user interface prompt word is prompt information describing the appearance of the user interface; and generate the user interface according to the target projection area, the layout information of the user interface and the user interface prompt word and using a second diffusion model.

[0013] In view of the above-mentioned prior art, the embodiments of the present invention disclose a computer-readable storage medium, which can overcome the defect of poor projection effect caused by an undesirable projection area and achieve the purpose of flexibly generating a high-quality user interface according to actual conditions.

[0014] A computer-readable storage medium stores computer instructions, which, when executed by a processor, can implement the steps of any one of the methods for generating a user interface.

[0015] In view of the above-mentioned prior art, an embodiment of the present invention discloses a computer program product, which can overcome the defect of poor projection effect caused by an undesirable projection area and achieve the purpose of flexibly generating a high-quality user interface according to actual conditions.

[0016] A computer program product comprises computer instructions, wherein when the computer instructions are executed by a processor, the method for generating a user interface as described in any one of the above is implemented.

[0017] In summary, the embodiment of the present application obtains the user's environmental information, user body information and user intention to determine the target projection area. On this basis, two diffusion models are used to generate the layout information of the user interface and the final user interface respectively, so as to achieve the goal of flexibly generating high-quality user interfaces according to actual conditions in any scenario to meet the needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a flow chart of the first embodiment of the method of the present application.

[0020] Figure 2 It is a flow chart of the second embodiment of the method of the present application.

[0021] Figure 3 This is a scene diagram of the second embodiment of the method of the present application.

[0022] Figure 4 It is a flow chart of the third embodiment of the method of the present application.

[0023] Figure 5 This is a scene diagram of the third embodiment of the method of the present application.

[0024] Figure 6 It is a structural diagram of the first embodiment of the device for generating the user interface of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0027] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0028] The projector can be a fixed projector or a movable projector. A fixed projector is fixed in a certain place for projection operation, while a movable projector can be carried with you and projected at a suitable angle in any suitable location. The projector can be connected to various smart devices such as computers, tablets, mobile phones, smart switches, etc. In practical applications, when the projection area is not ideal (such as being blocked by obstacles, stained areas, or irregular areas, etc.), the existing technology usually corrects the projection effect by reducing the size of the projection screen, otherwise it often leads to poor visual effects due to the imperfect projection area.

[0029] As described above, the prior art has difficulty finding a suitable area when the projection area is not ideal, and cannot flexibly achieve high-quality projection effects according to actual conditions. In view of the defects of the prior art, the embodiment of the present application proposes a method for generating a user interface, which can be built into a projector or deployed separately, and can flexibly generate a high-quality user interface according to the actual conditions of the projection area to meet user needs.

[0030] Figure 1 This is a flow chart of the first embodiment of the method of this application. Figure 1 As shown, the method includes:

[0031] Step 101: Obtain the user's environmental information and user body information, where the environmental information includes the three-dimensional spatial information of the user's environment, and the user body information includes the three-dimensional spatial information of the user's own body.

[0032] In practical applications, a collection device, such as a three-dimensional camera, can be installed in the user's spatial environment to collect the user's environmental information and user body information. The collected data can be three-dimensional data or point cloud data. The collection device can be deployed in the projector or deployed separately. The environmental information is the three-dimensional spatial information of the user's environment, which can include but is not limited to describing the length, width, height and other dimensions of the spatial environment, describing the length, width, height and other dimensions of the object, and can further detect and analyze the object name, object material, object hardness and other attribute information. The user's own body information is the characteristic information describing the user's body parts occupying the three-dimensional space, which can include but is not limited to human height and human proportions. Bone detection and bone point detection can also be performed and the length of the bones can be analyzed to determine the size of each body part. In practical applications, when detecting and analyzing the user's environmental information and user's body information, existing technologies can be used to achieve it, such as image segmentation and depth estimation methods, which will not be repeated here.

[0033] In addition, this step can also generate a two-dimensional space environment image. The two-dimensional space environment image can be generated based on the collected three-dimensional data mapping, or can be generated directly through a two-dimensional camera, which is not limited here.

[0034] Step 102: determining a target projection area according to the three-dimensional spatial information of the user's environment, the user's own three-dimensional spatial information, and the acquired user operation intention.

[0035] After collecting the three-dimensional spatial information of the user's environment and the user's own three-dimensional spatial information, it is possible to determine which areas can be used for projection and which areas can be touched by the user's body. In the embodiment of the present application, in order to facilitate the user to operate the projection screen, the areas that can be used for projection can be screened one by one with the user as the center. The screened areas are all areas that the user can touch with a part of the body (such as a hand). Of course, there may be multiple projection areas that meet the screening conditions, and this step can select one from them as the target projection area according to the user's operation intention. The user operation intention described here refers to the user's active expression of intention, which can be obtained by detecting any one or more of the user's voice, user gestures, user eyes, and user expressions, or by a pre-set method. For example, if a user raises his right hand, the area closest to the user's right hand can be used as the target projection area. Or, if a user is left-handed and raises his left hand, the area closest to the user's left hand can be used as the target projection area. Or, if a user raises his right leg, the area closest to the user's right leg can also be used as the target projection area. Or, if a user's voice expression is projected directly in front of the face, the area directly in front of the face can be used as the target projection area. In summary, this step can determine the target projection area based on the 3D spatial information of the user's environment and the user's own 3D spatial information and in accordance with the user's operation intention. The target projection area is the area used for projection after the user interface is subsequently generated.

[0036] Step 103: Determine layout prompt words according to the target projection area and the user's operation intention. The layout prompt words are prompt information describing the layout of the user interface.

[0037] In practice, the target projection areas used by different users may be different, the target projection areas used by the same user at different times may be different, the elements of the user interface may be different in different situations, and so on. The target projection area may be a regular area or an irregular area, a large area or a small area, and the elements of the user interface may be many or few. In order to generate a more reasonable and beautiful user interface, this embodiment needs to consider the layout of the user interface. In this step, when planning the layout of the user interface, not only the target projection area but also the user's operation intention is considered. As described above, the user's operation intention can be obtained by detecting any one or more of the combinations of user voice, user gesture, user eye expression, or by a pre-set method. For example, if the user wants to arrange the elements of the user interface in a matrix manner, the user's operation intention can be determined by a pre-set method, or the voice indicates "uncrowded arrangement", then the algorithm may obtain a layout prompt word such as "number of rows = 4, number of columns = 3" through calculation. Of course, this is just a simple example, and it is not limited to this in actual application, as long as prompt information describing the layout of the user interface can be generated.

[0038] Step 104: Determine the layout information of the user interface according to the target projection area and the layout prompt words and by using the first diffusion model.

[0039] After determining the target projection area and layout prompt words, this step uses a diffusion model to complete the layout. The diffusion model is a probability-based generative model that learns the distribution of data by simulating the data generation process and can generate new data samples from the learned distribution. The core principles of the diffusion model include the forward diffusion process and the reverse diffusion process, which together constitute the basis of the diffusion model. Forward diffusion process: This process simulates the data generation process by gradually adding noise to the data. This process is irreversible. By gradually adding noise, the original data is gradually "diffused" or "contaminated" and eventually becomes a Gaussian distributed noise data. Reverse diffusion process: Contrary to the forward process, the reverse diffusion process learns the ability to recover the original data from the noisy data. By training a neural network model, it can gradually predict and remove noise from the noisy data, and finally recover the original data samples.

[0040] In the embodiment of the present application, the first diffusion model can be trained in advance using a large number of samples, that is, using a large number of different projection areas, a large number of different layout prompt words, and a large number of different layout information, so that the first diffusion model learns how to generate a reasonable and beautiful layout according to the layout prompt words for different projection areas. The layout information described in this step can be an abstract layout structure or a specific layout picture.

[0041] Step 105: Determine a user interface prompt word according to the user operation intention, where the user interface prompt word is prompt information describing the appearance of the user interface.

[0042] As described above, the user's operation intention can be obtained by detecting any one or a combination of more than one of the following methods: user voice, user gestures, user eyes, and user expressions, or by a pre-set method. This step can also further determine the user interface prompt words based on the user's operation intention. For example, if the user expresses his or her voice or sets his or her preferred "blue 3D style" in advance, then a user interface prompt word such as "color = blue, shape = 3D" can be generated. Of course, this is just a simple example, and it is not limited to this in actual applications, as long as prompt information describing the appearance of the user interface can be generated.

[0043] Step 106: Generate a user interface according to the target projection area, the layout information of the user interface and the user interface prompt words and by using a second diffusion model.

[0044] After determining the layout information of the user interface and the user interface prompt words, this step also uses a diffusion model to complete the generation of the final user interface. In order to distinguish it from the diffusion model for determining the layout in step 104, the diffusion model in this step 106 is called a second diffusion model. In an embodiment of the present application, the second diffusion model can be trained in advance using a large number of samples, that is, using a large number of different projection areas, a large number of different user interface layout information, and a large number of different user interface prompt words for training, so that the second diffusion model learns how to target different projection areas and user interface layout information, and generates the final user interface according to the user interface prompt words. The final user interface generated here will conform to the shape of the target projection area, and will be displayed in a reasonable layout structure and in a style preferred by the user.

[0045] By using the solution of the first embodiment of the method of the present application, a user interface can be flexibly generated according to a suitable position in the environment where the user is located. After that, the generated user interface is projected to the target projection area and provided to the user to realize the user's operation intention, that is, the user can conveniently operate the user interface in the touchable target projection area without the need for the user to operate other electronic devices such as a remote control or keyboard, which greatly meets the needs of the user.

[0046] In another embodiment, since the user's environmental information and user body information are obtained, that is, the three-dimensional space information of the user's environment is obtained, and the three-dimensional space information occupied by each part of the user's body is also obtained, it is possible to determine which areas are user-touchable areas, and then select an area from them as the target projection area based on the obtained user operation intention. The user-touchable area refers to an area that can be touched by a certain part of the user's body, such as touching with a hand or foot; or, it can also be touched with an object, such as an elderly person touching with a cane in his hand. The target projection area can be a touchable projection area that is easy for the user to operate, and can be a regular area or an irregular area. The irregular area is an area surrounded by any irregular geometric figures, such as a triangle, trapezoid, semicircle, hexagon and other irregular geometric figures.

[0047] In practical applications, fixed projectors can be fixedly installed outdoors or indoors, while movable projectors can be moved from indoors to outdoors, and vice versa. If installed indoors, it is difficult to find a blank and regular area conveniently and flexibly indoors because the interior of an ordinary family is always blocked by furniture, home appliances or various other household devices. For example, before going out, the user wants to project the user interface of various electrical switches (air conditioners, televisions, lamps, etc.) indoors to the side of the gate, but there are sundries such as access control blocking the side of the gate, leaving only some irregular areas available. For another example, the user wants to project the user interface onto a wall in the children's room, but there are places on the wall that are graffiti-stained, leaving only some irregular areas available. This type of irregular area formed by actual conditions is different from the regular area specifically used for projection. If the user interface is directly projected into an irregular area, it is likely that the projection image will be difficult to recognize or inconvenient for the user to touch and operate due to blocking or staining.

[0048] As mentioned above, the diffusion model includes a forward diffusion process and a reverse diffusion process, which together form the basis of the diffusion model. The first diffusion model training process is introduced below.

[0049] Before the first diffusion model is trained, a large number of training samples need to be collected, such as a large number of regular and irregular areas, and a large number of layout prompt words. In the specific implementation, it contains a rich layout and user interface elements, allowing the model to learn the distribution and boundary characteristics of various regular and irregular areas. Subsequently, the conditional generation strategy is used to consider the spatial efficiency and aesthetics of the arrangement of various elements during the generation process, and the layout of each element is optimized, so that a reasonable layout is generated in the projection area of ​​various shapes.

[0050] In the forward diffusion process, the clear user interface layout information can be used as the original data, and noise can be gradually introduced into it, and finally it becomes pure noise, completing the diffusion process. This process can be expressed as:

[0051]

[0052] Among them, x t+1 is the data state at time step t, β t is the preset noise level parameter, Indicates the current state x t The retained part, ε t is the noise sampled from a standard normal distribution, is the added noise. As the time step t increases, the data gradually becomes more and more blurred and noisy from the original clear state, until it finally becomes almost completely random noise x at time step T T .

[0053] In the reverse diffusion process, the original data is gradually restored from the noisy data. The reverse diffusion process can learn to gradually remove the noise from the Gaussian noise, return to the mapping of the original data distribution, and restore the clean data state from a highly noisy state. The reverse diffusion process can be expressed as:

[0054]

[0055] where β t is the preset noise level parameter, which indicates the proportion of noise added to the data at time step t and controls the amount of noise introduced at each step. t It defines the proportion of data retained in the same time step, that is, the part of the data not covered by noise. Usually, α t =1-β t ε θ (x t ,t) is a parameterized network, which represents the current data state x t and time step t and the noise estimated by the neural network model. In the embodiment of the present application, ε θ (x t ,t) also includes layout prompt words, which are the encoded form of the text description, usually implemented by some form of text encoder (such as Transformer or BERT). t is a standard deviation adjusted over time, used to control the amount of random noise z added, where z is a random noise vector sampled from a standard normal distribution. The back diffusion process needs to predict which noise should be removed from the current noisy image based on the noisy image, time step, layout hint words, etc.

[0056] Using the forward diffusion process and the backward diffusion process described above, the first diffusion model will learn how to generate clear and unambiguous user interface layout information from pure noise.

[0057] After the first diffusion model is trained, only the reverse diffusion process is used in actual use. In each step, the first diffusion model uses the reverse diffusion model learned during training to infer the layout information of the denoised user interface. That is to say, the above step 104 is a step of determining the layout information of the user interface based on the target projection area and the layout prompt words and using the first diffusion model, specifically including: taking the three-dimensional spatial information and layout prompt words of the target projection area as inputs of the first diffusion model, and outputting the layout information of the user interface after calculation by the first diffusion model. The first diffusion model is a model trained in advance by the above method. Of course, in actual applications, the first diffusion model also requires other necessary input information to complete the calculation. The embodiment of the present application emphasizes the three-dimensional spatial information and layout prompt words of the target projection area, so as to generate the layout information of the user interface that conforms to the layout prompt words for the target projection area. The generated layout information can be an image representing the layout framework.

[0058] In another embodiment, step 103 may determine the layout prompt words according to the target projection area and the user's operation intention, specifically including:

[0059] Step 1031: Perform three-dimensional detection on the target projection area to determine the three-dimensional spatial parameter estimation of the object in the target projection area. The three-dimensional spatial parameter estimation is the spatial position information describing the object in the target projection area.

[0060] After determining the target projection area, the area can be 3D detected again to determine the 3D spatial parameter estimation of the object in the target projection area. For example, it can be determined whether there are abnormal conditions such as occlusions, concave and convex parts or stained parts in the area, and the 3D spatial parameter estimation of these abnormal conditions, i.e., size, dimensions, 3D coordinates and other parameters, can be calculated.

[0061] Step 1032: Determine the layout situation in the target projection area according to the three-dimensional space parameter estimation of the objects in the target projection area, and generate layout prompt words based on the layout situation.

[0062] After determining the three-dimensional spatial parameter estimate of the object in the target projection area, the layout of the user interface can be planned according to these abnormal situations. For example, when there is an obstruction, a concave-convex part or a damaged part in the middle of the target projection area, the elements of the user interface may not be laid out at these positions, but instead be laid out in a way that surrounds the middle part. For another example, when there is an obstruction, a concave-convex part or a damaged part on the right side of the target projection area, the original layout in the form of three columns and four rows can be changed to a layout in the form of two columns and six rows. In short, the embodiment of the present application can calculate a reasonable layout according to the actual situation in the target projection area and generate layout prompts.

[0063] After the layout is generated, only the framework of the user interface is determined. Specific elements need to be filled into the user interface, that is, a real user interface is generated. Similar to the first diffusion model, the embodiment of the present application also uses a diffusion model to generate the final user interface, that is, the second diffusion model. The second diffusion model training process is described below.

[0064] Before training the second diffusion model, a large number of training samples need to be collected, such as a large number of regular areas and irregular areas, a large amount of user interface layout information and a large number of user interface prompts. In specific implementation, it contains a wealth of user interface elements, such as control types and control positions, etc. This information can be represented as structured data such as JSON format, so that the model can learn the user interface features of various regular and irregular areas, and can also use data enhancement and other technologies to make the model have better generalization ability.

[0065] In the forward diffusion process, the clear user interface can be used as the original data, and noise is gradually introduced into it, and finally it becomes pure noise, completing the diffusion process. The process can be expressed as:

[0066]

[0067] Among them, y t+1 is the data state at time step t, b t is the preset noise level parameter, Indicates the current state y t The retained part, ε t is the noise sampled from a standard normal distribution, is the added noise. As the time step t increases, the data gradually becomes more and more blurred and noisy from the original clear state, until it finally becomes almost completely random noise y at time step T T .

[0068] In the reverse diffusion process, the original data is gradually restored from the noisy data. The reverse diffusion process can learn to gradually remove the noise from the Gaussian noise, return to the mapping of the original data distribution, and restore the clean data state from a highly noisy state. The reverse diffusion process can be expressed as:

[0069]

[0070] where b t is the preset noise level parameter, which indicates the proportion of noise added to the data at time step t and controls the amount of noise introduced at each step. t It defines the proportion of data retained in the same time step, that is, the part of the data not covered by noise. Usually, let a t =1-b t ε θ (y t , t) is a parameterized network, which represents the current data state y t and time step t and the noise estimated by the neural network model. In the embodiment of the present application, ε θ (y t , t) also includes user interface prompt words, which are the encoded form of the text description, usually implemented by some form of text encoder (such as Transformer or BERT). t is a standard deviation adjusted over time that controls the amount of random noise z added, where z is a random noise vector sampled from a standard normal distribution. The back diffusion process needs to predict which noise should be removed from the current noisy image based on the noisy image, time step, user interface prompt words, etc.

[0071] Using the forward diffusion process and the backward diffusion process described above, the second diffusion model will learn how to generate a clear and unambiguous user interface from pure noise.

[0072] After the second diffusion model is trained, only the reverse diffusion process is used in actual use. In each step, the second diffusion model uses the reverse diffusion model learned during training to infer the denoised user interface. That is to say, the above step 106 is a step of generating a user interface based on the target projection area, the layout information of the user interface and the user interface prompt words and using the second diffusion model, specifically including: using the three-dimensional spatial information of the target projection area, the layout information of the user interface and the user interface prompt words as the input of the second diffusion model, and outputting the user interface through the calculation of the second diffusion model, and the second diffusion model is a pre-trained model. Of course, in practical applications, the second diffusion model also requires other necessary input information to complete the calculation. The embodiment of the present application emphasizes the three-dimensional spatial information and the user interface prompt words of the target projection area, so as to generate a user interface that conforms to the user interface prompt words for the target projection area. In practical applications, the generated user interface can also be subjected to trapezoidal correction to make the projection picture effect better.

[0073] In some other embodiments, step 105 may determine a user interface prompt word according to the user's operation intention. The user interface prompt word is prompt information describing the appearance of the user interface, specifically including:

[0074] Step 1051: Determine a user interface style for the target projection area according to the user operation intention, where the user interface style is information describing the appearance style of the user interface.

[0075] Different users may prefer different interface styles. The interface style described here is information that describes the appearance style of the user interface, such as the color of the interface border, the color of the button, whether it is three-dimensional, whether it has a shadow, the size of the text, the font of the text, etc. The user's operation intention can be obtained by detecting any one or a combination of more than one of the user's voice, user gestures, user eyes, and user expressions, or by a pre-set method. For example, if it is determined that the user is a young woman, a pink series style user interface can be determined for her.

[0076] Step 1052: Determine a user interface prompt word in the target projection area according to the user interface style.

[0077] After the style of the user interface is determined, the user interface prompt words can be determined based on this. Taking the above steps as an example, if it is determined that the user is a young female, the user interface prompt words can be generated as "pink series style". In short, the embodiment of the present application can generate accurate and reasonable user interface prompt words according to the user's operation intention.

[0078] Method Embodiment 2 applies the present application scheme to an indoor teaching scenario. In this scenario, the projector can be a fixed projector, and the present application embodiment scheme is combined with the fixed projector. The projector can also be a movable projector, and the present application embodiment scheme is combined with the mobile projector. The projector is installed in the direction facing the blackboard in the room, and is equipped with a camera and a sensor. The projector projects the PPT of teaching-related content in front of the blackboard, and detects the classroom environment information and the teacher's body information in real time. Among them, the detected classroom environment information is the environment information of the above-mentioned user, and the teacher's body information is the above-mentioned user's body information. The lecturer walks back and forth in front of the podium and blackboard, hoping to generate a PPT operation button in the appropriate area when the PPT needs to be manipulated. In order to simplify the problem, the present application embodiment divides the operation of the PPT operation button into three types: "next page", "previous page" and "exit". In other words, the present application embodiment method can generate a PPT operation button in a suitable area for the teacher to operate in the scenario where the user walks back and forth to explain. The following is a detailed description of the present application embodiment method.

[0079] Figure 2 This is a flow chart of Embodiment 2 of the present application method. Figure 2 As shown, the method includes:

[0080] Step 201: Acquire the user's environmental information and user body information, where the environmental information includes the three-dimensional spatial information of the user's environment, and the user body information includes the three-dimensional spatial information of the user's own body.

[0081] Figure 3 This is a scene diagram of the second embodiment of the method of this application. Figure 3 As shown, the teacher usually explains beside the PPT on the podium, and often points to a certain content in the PPT page. The camera in the projector can detect the indoor blackboard, podium, and PPT and other environmental information, such as the size, dimensions, and three-dimensional coordinates of the indoor blackboard and podium, which are the content and which are the blank areas on the PPT page, and the location, shape, and size of the blank areas, etc. It can also detect the teacher's body information, such as the human body, human proportions, and the size of each body part.

[0082] Step 202: Obtain the user's operation intention.

[0083] The user's operation intention can be obtained by detecting any one or more combinations of user voice, user gestures, user eyes, and user expressions, or by a pre-set method. In this embodiment, since the projector is equipped with a camera and a sensor, the teacher's body information can be detected, and the teacher's voice, facial expression, gesture, posture, and other information can also be detected, and the operation intention can be determined from these information. For example, when the teacher finishes explaining the content of a certain PPT page and raises his right hand to turn the page, it can be judged that his operation intention is to click the next page. For another example, the teacher says "just mentioned", and raises his right hand to turn the page, it can be judged that his operation intention is to click the previous page. As for how to obtain the user's operation intention according to any one or more combinations of user voice, user gestures, user eyes, and user expressions, or by a pre-set method, it can be achieved by the prior art, and will not be described in detail here.

[0084] Step 203: Determine the projection area that can be touched by the user according to the three-dimensional spatial information of the user's environment and the three-dimensional spatial information of the user himself.

[0085] After collecting the three-dimensional spatial information of the user's environment and the user's own three-dimensional spatial information, it is possible to determine which areas the user can touch and can be used for projection. In order to facilitate the user to operate the projection screen, the areas that can be used for projection can be screened one by one with the teacher as the center. The screened areas are all areas that the user can touch with a part of the body (such as the hand), and there may be multiple projection areas that meet the screening conditions.

[0086] Step 204: selecting an area from the projection areas that can be touched by the user as the target projection area according to the acquired user operation intention.

[0087] Since there may be multiple projection areas that meet the screening criteria, it is necessary to select the best projection area from them. The method of selecting the best projection area can be determined based on the user's operation intention. For example, if the teacher stands on the right side of the PPT and raises his right hand, it can be judged that the teacher wants to use his right hand to operate the button. In this case, the user-touchable projection area closest to the teacher's right hand can be used as the target projection area.

[0088] Step 205: Perform three-dimensional detection on the target projection area to determine the three-dimensional spatial parameter estimation of the object in the target projection area. The three-dimensional spatial parameter estimation is the spatial position information describing the object in the target projection area.

[0089] After determining the target projection area, it is also necessary to determine the three-dimensional spatial parameter estimation of the objects in the target projection area in order to facilitate subsequent reasonable layout. In the embodiment of the present application, it is assumed that the area where the PPT is located is used as the target projection area. Since there is content displayed on the PPT page, if the PPT operation button is directly projected on the content, the picture effect is not good and difficult to recognize. Therefore, this step can be used to detect the target projection area to detect which positions are already covered by content and which positions are relatively blank. It should be noted that the three-dimensional detection performed in this step is to determine the three-dimensional spatial parameter estimation of objects in the target projection area, which can include coordinates in the three directions of X, Y, and Z. For the content in the PPT plane, the Z direction coordinate can be set to 0, and only the X and Y directions can be detected.

[0090] Step 206: Determine the layout situation in the target projection area according to the three-dimensional space parameter estimation of the objects in the target projection area, and generate layout prompt words based on the layout situation.

[0091] Assuming that the area on the upper right side of the PPT is taken as the target projection area, the specific situation in the target projection area can be determined through three-dimensional detection. Figure 3 It can be seen that the blank part on the upper right side has a small width on the left and right sides and a large width on the top and bottom sides, which is suitable for arranging the PPT operation buttons vertically, so the corresponding prompt words can be generated, such as "one column and three rows".

[0092] Step 207: The three-dimensional space information and layout prompt words of the target projection area are used as inputs of the first diffusion model, and the layout information of the user interface is output after calculation by the first diffusion model. The first diffusion model is a pre-trained model.

[0093] The first diffusion model here can be trained for the forward diffusion process and the reverse diffusion process according to the above method. This step directly uses the trained reverse diffusion process, takes the three-dimensional spatial information of the target projection area and the layout prompt words as input, and calculates the layout information of the user interface by the trained first diffusion model, that is, specifically generates a one-column and three-row operation button framework.

[0094] Step 208: Determine a user interface style for the target projection area according to the user operation intention, where the user interface style is information describing the appearance style of the user interface.

[0095] In this embodiment, it is assumed that the user's operation intention is to obtain the "blue simple style" through pre-setting, so in this step, the user interface style of the target projection area can be determined as the "blue simple style".

[0096] Step 209: Determine a user interface prompt word in the target projection area according to the user interface style.

[0097] In this step, the user interface prompt word is determined as "blue simple style" according to the user interface style of the target projection area.

[0098] Step 210: The three-dimensional spatial information of the target projection area, the layout information of the user interface and the user interface prompt words are used as inputs of the second diffusion model, and the user interface is output after calculation by the second diffusion model. The second diffusion model is a pre-trained model.

[0099] The second diffusion model here can also be trained for the forward diffusion process and the reverse diffusion process according to the above method. This step directly uses the trained reverse diffusion process, takes the three-dimensional spatial information of the target projection area, the layout information of the user interface and the user interface prompt words as input, and calculates the user interface by the trained second diffusion model. Assume that "next page" is represented by a right arrow, "previous page" is represented by a left arrow, and "exit" is represented by a cross, that is, a column and three rows of operation buttons are specifically generated, and the operation buttons are displayed as blue simple patterns according to the user's intention. In practical applications, other types of buttons can also be generated, such as a "drawing tool" button, so that teachers can perform drawing operations on PPT at any time.

[0100] Step 211: Project the user interface onto the target projection area and provide it to the user to realize the user's operation intention.

[0101] In actual applications, due to the installation angle of the projector, the generated user interface can also be corrected using the trapezoidal correction method before projection. Figure 3 As shown, after the user interface generated by the embodiment of the present application is projected to the target projection area, an operation button will be presented on the upper right side of the PPT. In this way, the teacher does not need to go to the computer or use the device to operate, but only needs to operate the PPT operation button projected near the right hand.

[0102] Figure 3 This is just a simple example and is not intended to limit the scope of protection of the embodiments of the present application. It is understandable that the method of the embodiments of the present application can also generate PPT operation buttons in other areas. For example, when the teacher walks to the left side of the PPT, a PPT operation button can be generated on the left side, or when the teacher walks to the side of the desk, the PPT operation button can be projected on the desk, and the layout and style of the projection are related to the actual target projection area and the user's intention, so as to achieve the purpose of flexible projection to meet user needs.

[0103] Method Example 3 applies the present application scheme to a home or office indoor scene. In this scenario, the projector can be installed in the living room, bedroom, kitchen, office area, etc., and has built-in cameras and sensors. The camera and sensor can also be deployed separately. The camera and sensor detect the home or office environment information and the user's body information in real time, and the detected home or office environment information is the user's environment information. The user walks around the home or office at will, and hopes to turn off the switches of certain indoor electrical appliances or adjust the volume of the audio system before going out. In order to simplify the problem, the present application embodiment sets the user interface to the switch button of the electrical appliance. However, in actual applications, the user interface is not limited to the switch button, but any interface that interacts with the user is acceptable, and is not limited to the embodiments of the present application. The following is a detailed description of the present application embodiment method.

[0104] Figure 4 This is a flow chart of Embodiment 3 of the present application method. Figure 4 As shown, the method includes:

[0105] Step 401: Obtain the user's environmental information and user body information, where the environmental information includes the three-dimensional spatial information of the user's environment, and the user body information includes the three-dimensional spatial information of the user's own body.

[0106] Users move around freely in the room and may appear in any location such as the living room, bedroom, kitchen or work area. In this case, a movable projector can be used, which is convenient for users to carry with them and place it in a suitable location in any room. In this way, even if the user moves around in different spatial areas indoors, the movable projector can be used to detect the user's environment and the user himself in real time. Of course, in actual applications, fixed projectors or movable projectors can also be used separately in different rooms, and there is no restriction here. The camera and sensor in the projector can detect the user's indoor environment and the user himself. Figure 5 As shown, it is assumed that the kitchen sink area where the user is located, or a wall display area in the office area, or the front door is detected. At this time, the camera in the projector can detect the area in front of the sink, the area in the display area, and the area near the door. Of course, the camera will also detect other areas around the user, which are not listed here one by one. For these areas, the size, dimensions, three-dimensional coordinates, etc. in front of the sink will be detected, the size, dimensions, three-dimensional coordinates, etc. of the display area will be detected, and the size, dimensions, three-dimensional coordinates, etc. of the area near the door will be detected. Of course, this step will also detect the user's body information, such as the human body, human proportions, and the size of each body part.

[0107] Step 402: Obtain the user's operation intention.

[0108] The user's operation intention can be obtained by detecting any one or more combinations of user voice, user gestures, user eyes, user expressions, or by a pre-set method. In this embodiment, when the user wants to go out and wants to turn off the indoor electrical equipment, he can raise his hand or say "turn off the switch", etc., which will determine that the user's operation intention is to operate the switch button. As for how to obtain the user's operation intention according to any one or more combinations of user voice, user gestures, user eyes, user expressions, or by a pre-set method, it can be achieved through existing technologies and will not be described in detail here.

[0109] Step 403: Determine the projection area that can be touched by the user according to the three-dimensional spatial information of the user's environment and the three-dimensional spatial information of the user himself.

[0110] After collecting the three-dimensional spatial information of the user's environment and the user's own three-dimensional spatial information, it is possible to determine which areas the user's body can touch and can be used for projection.

[0111] Step 404: selecting an area from the projection areas that can be touched by the user as the target projection area according to the acquired user operation intention.

[0112] Since there may be multiple projection areas that meet the screening conditions, it is necessary to select the best projection area from them. The method of selecting the best projection area can be determined according to the user's operation intention. For example, if the user stands at the door and raises his left hand, it can be judged that the user wants to operate the switch button with his left hand, and the projection area that is closest to the user's left hand and can be touched by the user is used as the target projection area.

[0113] Step 405: Perform three-dimensional detection on the target projection area to determine the three-dimensional spatial parameter estimation of the object in the target projection area. The three-dimensional spatial parameter estimation is the spatial position information describing the object in the target projection area.

[0114] After determining the target projection area, it is also necessary to determine the three-dimensional spatial parameter estimation of the objects in the target projection area in order to facilitate subsequent reasonable layout. In the embodiment of the present application, it is assumed that the area on the left side of the gate is used as the target projection area. Since there is a painting on the left side of the gate, if the switch button is directly projected on the content, the picture effect is not good and difficult to recognize. Therefore, this step can be used to detect the target projection area to detect which positions are already covered by content and which positions are relatively blank.

[0115] Step 406: Determine the layout situation in the target projection area according to the three-dimensional space parameter estimation of the objects in the target projection area, and generate layout prompt words based on the layout situation.

[0116] Assuming that the blank area on the left side of the gate is the target projection area, the specific situation in the target projection area can be determined through three-dimensional detection. Figure 5 It can be seen that the area on the left side of the gate is relatively spacious and regular, so the switch buttons can be arranged in a matrix manner, so corresponding prompt words can be generated, such as "two columns and four rows".

[0117] Step 407: The three-dimensional space information and layout prompt words of the target projection area are used as inputs of the first diffusion model, and the layout information of the user interface is output after calculation by the first diffusion model. The first diffusion model is a pre-trained model.

[0118] The first diffusion model here can be trained for the forward diffusion process and the reverse diffusion process according to the above method. This step directly uses the trained reverse diffusion process, takes the three-dimensional spatial information of the target projection area and the layout prompt words as input, and calculates the layout information of the user interface by the trained first diffusion model, that is, specifically generates a two-column and four-row operation button framework.

[0119] Step 408: Determine a user interface style for the target projection area according to the user operation intention, where the user interface style is information describing the appearance style of the user interface.

[0120] In this embodiment, it is assumed that the user's operation intention is to obtain the "color 3D style" through pre-settings. In this case, the user interface style of the target projection area can be determined as the "color 3D style" in this step.

[0121] Step 409: Determine a user interface prompt word in the target projection area according to the user interface style.

[0122] In this step, the user interface prompt word is determined as "color 3D style" according to the user interface style of the target projection area.

[0123] Step 410: The three-dimensional spatial information of the target projection area, the layout information of the user interface and the user interface prompt words are used as inputs of the second diffusion model, and the user interface is output after calculation by the second diffusion model. The second diffusion model is a pre-trained model.

[0124] The second diffusion model here can also be trained for the forward diffusion process and the reverse diffusion process according to the above method. This step directly uses the trained reverse diffusion process, takes the three-dimensional spatial information of the target projection area, the layout information of the user interface and the user interface prompt words as input, and calculates the user interface by the trained second diffusion model. Assuming that the button of each electrical device has a LOGO representing the electrical device, the LOGO of each electrical device can be placed on the button, or text can be used on each button, such as "TV", "stereo", "curtain", "light", etc. At this time, this step will generate two columns and four rows of switch buttons, and display the switch buttons as colorful three-dimensional style patterns according to the user's intention.

[0125] Step 411: Project the user interface to the target projection area and provide it to the user to realize the user's operation intention.

[0126] In practical applications, due to the installation angle of the projector, the generated user interface can also be corrected using a trapezoidal correction method before projection. Figure 5 This is a scene diagram of the third embodiment of the method of this application. Figure 5 As shown, after the user interface generated in the third embodiment of the present application is projected onto the target projection area, a switch button will be presented on the left side of the door. In this way, the user does not need to go to the bedroom or electrical equipment to operate, but only needs to operate the switch button projected near the left hand.

[0127] The above description is based on the example of projecting the switch button to the left side of the door before the user goes out. In actual applications, it can also be projected to other places. Figure 5 As shown, assuming that the user is located at the kitchen sink, the switch button can be projected to the area in front of the sink. If the area is an irregular pentagon, the layout of the switch button can be planned according to the characteristics of the area and a corresponding user interface can be generated. Similarly, assuming that the user is located near the display area and wants to adjust the audio volume, and the display area has a triangular area, the layout can be planned according to the characteristics of the area and a corresponding user interface can be generated, and the moving bar for adjusting the audio volume can be projected to the triangular area. In short, by applying the embodiment of the present application, a user interface can be flexibly generated in an area that is convenient for the user to touch, and provided to the user for operation, which greatly improves the convenience of user operation and improves the user experience.

[0128] The present application also provides a device for generating a user interface, which can be deployed in a projector or deployed separately. Figure 6 1 is a structural diagram of the first embodiment of the user interface generation device of the present application. Figure 6 As shown, the device includes: a detection module 601, a layout generation module 602, and an interface generation module 603. Among them:

[0129] The detection module 601 is used to obtain the user's environmental information and user body information, where the environmental information includes the three-dimensional spatial information of the user's environment, and the user's body information includes the three-dimensional spatial information of the user's own body, and determine the target projection area based on the three-dimensional spatial information of the user's environment, the three-dimensional spatial information of the user's own body and the obtained user operation intention.

[0130] In practical applications, the detection module 601 can be a camera and a sensor, etc., for collecting the user's environmental information, user body information and user operation intention. The environmental information is the three-dimensional space information of the user's environment, which can include but is not limited to describing the length, width, height and other dimensions of the spatial environment, describing the length, width, height and other dimensions of the object, and can further detect and analyze the object name, object material, object hardness and other attribute information. The user's own body information is the characteristic information describing the three-dimensional space occupied by each part of the user's body, which can include but is not limited to human height and human proportions, and can also perform bone detection and bone point detection and analyze the length of the bones, thereby determining the size of each body part. The user's operation intention refers to the user's active expression of meaning, which can be obtained by detecting any one or a combination of more than one of the user's voice, user gestures, user eyes, and user expressions, or by a pre-set method.

[0131] The layout generation module 602 is used to determine layout prompt words according to the target projection area and the user operation intention, where the layout prompt words are prompt information describing the user interface layout; and determine the layout information of the user interface according to the target projection area and the layout prompt words and using the first diffusion model.

[0132] In order to generate a more reasonable and beautiful user interface, the present embodiment needs to consider the layout of the user interface. When planning the layout of the user interface, not only the target projection area is considered, but also the user's operation intention is considered. After determining the target projection area and the layout prompt word, this step adopts a diffusion model to complete the layout. The diffusion model is a probability-based generation model that learns the distribution of data by simulating the generation process of data and can generate new data samples from the learned distribution. The core principles of the diffusion model include the forward diffusion process and the reverse diffusion process, which together constitute the basis of the diffusion model. The first diffusion model can be trained in advance using a large number of samples, that is, using a large number of different projection areas, a large number of different layout prompt words, and a large number of different layout information for training, so that the first diffusion model learns how to generate a reasonable and beautiful layout according to the layout prompt words for different projection areas.

[0133] The interface generation module 603 is used to determine the user interface prompt word according to the user operation intention, and the user interface prompt word is the prompt information describing the appearance of the user interface; and generate the user interface according to the target projection area, the layout information of the user interface and the user interface prompt word and using the second diffusion model.

[0134] In practical applications, the user interface prompt words can be determined according to the user's operation intention. After the layout information of the user interface and the user interface prompt words are determined, this step also uses a diffusion model to complete the generation of the final user interface. The second diffusion model can be trained in advance using a large number of samples, that is, using a large number of different projection areas, a large number of different user interface layout information, and a large number of different user interface prompt words for training, so that the second diffusion model learns how to target different projection areas and user interface layout information, and generate the final user interface according to the user interface prompt words.

[0135] By using the solution of the first embodiment of the device of the present application, a user interface can be flexibly generated according to a suitable position in the environment where the user is located. After that, the generated user interface is projected to the target projection area and provided to the user to realize the user's operation intention, that is, the user can conveniently operate the user interface in the touchable target projection area without the need for the user to operate other electronic devices such as a remote control or keyboard, which greatly meets the needs of the user.

[0136] The embodiment of the present application also provides a computer-readable medium, the computer-readable storage medium stores instructions, and the instructions can execute the steps in the method for generating a user interface as described above when executed by a processor. In practical applications, the computer-readable medium may be included in the device / device / system described in the above embodiment, or it may exist alone without being assembled into the device / device / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method for generating a user interface described in the above embodiments can be implemented. According to the embodiment disclosed in the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above, but it is not used to limit the scope of protection of the present application. In the embodiment disclosed in the present application, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, a device or a device or used in combination with it.

[0137] An embodiment of the present application further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, the method described in any of the above embodiments is implemented.

[0138] The flow chart and block diagram in the accompanying drawings of the present application show the possible architecture, function and operation of the system, method and computer program product according to the various embodiments disclosed in the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in the order of the standards in different figures. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0139] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope disclosed in the present application.

[0140] Specific embodiments are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas, and is not used to limit the present application. For those skilled in the art, changes can be made in the specific implementation methods and application scopes according to the ideas, spirits and principles of the present invention, and any modifications, equivalent substitutions, improvements, etc. made therein should be included in the scope of protection of this application.

Claims

1. A method for generating a user interface, It is characterized in that The method includes: Acquire the user's environmental information and user's body information, wherein the environmental information includes the three-dimensional spatial information of the user's environment, and the user's body information includes the three-dimensional spatial information of the user's own body, and determine the target projection area according to the three-dimensional spatial information of the user's environment, the three-dimensional spatial information of the user's own body, and the acquired user operation intention; Determining a layout prompt word according to the target projection area and the user operation intention, wherein the layout prompt word is prompt information describing the layout of the user interface; determining the layout information of the user interface according to the target projection area and the layout prompt word and using a first diffusion model; A user interface prompt word is determined according to the user operation intention, wherein the user interface prompt word is prompt information describing the appearance of the user interface; and the user interface is generated according to the target projection area, the layout information of the user interface and the user interface prompt word and using a second diffusion model.

2. The method according to claim 1, It is characterized in that After the step of generating the user interface according to the target projection area, the layout information of the user interface and the user interface prompt word and using a second diffusion model, the method further comprises: The user interface is projected onto the target projection area and provided to the user to realize the user's operation intention.

3. The method according to claim 1, It is characterized in that The step of determining the target projection area according to the three-dimensional spatial information of the user's environment, the three-dimensional spatial information of the user himself and the acquired user operation intention comprises: Determine a projection area that can be touched by the user according to the three-dimensional spatial information of the user's environment and the three-dimensional spatial information of the user himself; According to the acquired user operation intention, an area is selected from the projection area that can be touched by the user as the target projection area.

4. The method according to claim 3, It is characterized in that Before the step of selecting an area from the projection area touchable by the user as the target projection area according to the acquired user operation intention, the method further comprises: The method of obtaining the user operation intention includes: obtaining the user operation intention by detecting any one or a combination of more than one of the following methods: user voice, user gestures, user eyes, and user expressions.

5. The method according to claim 1, It is characterized in that The target projection area is a touchable projection area that is easy for the user to operate, and is a regular area or an irregular area. The irregular area is an area surrounded by any irregular geometric figures.

6. The method according to claim 1, It is characterized in that The step of determining the layout prompt word according to the target projection area and the user operation intention comprises: Performing three-dimensional detection on the target projection area to determine a three-dimensional spatial parameter estimate of an object in the target projection area, wherein the three-dimensional spatial parameter estimate is spatial position information describing the object in the target projection area; The layout situation is determined in the target projection area according to the three-dimensional space parameter estimation of the object in the target projection area, and the layout prompt word is generated according to the layout situation.

7. The method according to claim 1, It is characterized in that The step of determining the layout information of the user interface according to the target projection area and the layout prompt word and using the first diffusion model comprises: The three-dimensional space information of the target projection area and the layout prompt words are used as inputs of the first diffusion model, and the layout information of the user interface is output after calculation by the first diffusion model. The first diffusion model is a pre-trained model.

8. The method according to claim 1, It is characterized in that The step of determining the user interface prompt word according to the user operation intention comprises: Determine a user interface style for the target projection area according to the user operation intention, where the user interface style is information describing the appearance style of the user interface; The user interface prompt word is determined in the target projection area according to the user interface style.

9. The method according to claim 1, It is characterized in that The step of generating the user interface according to the target projection area, the layout information of the user interface and the user interface prompt word and using a second diffusion model comprises: The three-dimensional spatial information of the target projection area, the layout information of the user interface and the user interface prompt words are used as inputs of the second diffusion model, and the user interface is output after calculation by the second diffusion model. The second diffusion model is a pre-trained model.

10. A device for generating a user interface, It is characterized in that The device includes: a detection module, configured to obtain the user's environmental information and user body information, wherein the environmental information includes the three-dimensional spatial information of the user's environment, and the user's body information includes the three-dimensional spatial information of the user's own body, and determine the target projection area according to the three-dimensional spatial information of the user's environment, the three-dimensional spatial information of the user's own body, and the obtained user operation intention; a layout generation module, configured to determine a layout prompt word according to the target projection area and the user operation intention, wherein the layout prompt word is prompt information describing the layout of the user interface; and determine the layout information of the user interface according to the target projection area and the layout prompt word and using a first diffusion model; The interface generation module is used to determine a user interface prompt word according to the user operation intention, wherein the user interface prompt word is prompt information describing the appearance of the user interface; and generate the user interface according to the target projection area, the layout information of the user interface and the user interface prompt word and using a second diffusion model.

11. A computer-readable storage medium having computer instructions stored thereon, It is characterized in that When the instructions are executed by a processor, the steps of the method for generating a user interface according to any one of claims 1 to 9 can be implemented.

12. A computer program product comprising computer instructions, It is characterized in that When the computer instructions are executed by a processor, the method for generating a user interface as described in any one of claims 1 to 9 is implemented.

Images