Picture display method and device, electronic equipment, medium and program product

Abstract

The invention provides a picture display method and device, electronic equipment, a medium and a program product, and relates to the technical field of virtualized.The method comprises the steps that a fusion picture is displayed, and the fusion picture comprises a real environment picture, a first virtual object and a controller; changing a position of the controller in the fused picture in response to a first movement or a first gesture of the first device; when the first end of the controller is close to the first virtual object and prompt information is presented in the fusion picture, the first end is located on the first virtual object and / or in a display area corresponding to the first virtual object; and after the controller is bent, changing the positions of the controller and the virtual object in the fused picture. In the application, the controller dynamically changes according to the real-time position information of the virtual object in the interaction process, interaction between the first device and the object is realized, and the degree of participation and immersion of a user in the virtual world are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of virtual reality technology, and in particular to a screen display method, device, electronic device, medium, and program product. Background Art

[0002] In a virtual reality (VR) scene, you can interact with an object through a first device. The first device can be a virtual reality control handle for dragging and moving the object, and the interaction between the first device and the object is specifically achieved by the first device emitting a ray. The ray is rendered by the LineRender component. The principle of the LineRender component is to use an array of two or more points in 3D space to draw a straight line between each point. The straight ray rendered by the LineRender component is relatively stiff when dragging and moving the object, which will cause dizziness in the user's operation and is not conducive to the user's integration into the scene. Summary of the Invention

[0003] The embodiments of the present application provide a screen display method, apparatus, electronic device, medium, and program product to solve the problem of poor user experience in existing first device interaction.

[0004] In order to solve the above technical problems, this application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a screen display method, comprising:

[0006] Displaying a fused image, wherein the fused image includes a real environment image, a first virtual object, and a controller, wherein the controller includes a control line or a control rod, and the control line or the control rod is straight;

[0007] In response to a first movement or a first gesture of the first device, changing a position of the controller in the fused screen; wherein the position of the first virtual object in the fused screen remains unchanged;

[0008] When the first end of the controller is close to the first virtual object and prompt information is displayed in the fused screen, in response to a first trigger or a second gesture of the first device, the controller is bent, and the first end is located on the first virtual object and / or within a display area corresponding to the first virtual object, wherein the display area includes any pixel within the display area;

[0009] When the controller is bent, in response to the second movement or third gesture of the first device, the positions of the controller and the first virtual object in the fused screen are changed, and the length and / or bending degree of the controller changes with the movement and / or gesture of the first device, and / or the size of the first virtual object changes with the movement and / or gesture of the first device; wherein, during the process of changing the position of the controller, the first end is located on the first virtual object and / or within the display area corresponding to the first virtual object.

[0010] Optionally, the real environment picture is a picture obtained by capturing the surrounding environment, and the real environment picture changes with changes in the capturing perspective and / or changes in the surrounding environment.

[0011] Optionally, the fused image includes multiple virtual objects, and the multiple virtual objects include the first virtual object.

[0012] Optionally, the multiple virtual objects further include a second virtual object, and the second virtual object is associated with the first virtual object and / or moves with the first virtual object.

[0013] Optionally, the fused image is displayed in layers, wherein each layer corresponds to one of the real environment image, the first virtual object and the controller.

[0014] Optionally, the second end of the controller is a mapping point of the movement and / or gesture of the first device on the fused screen, and the controller appears smooth.

[0015] Optionally, the controller includes a plurality of monomers connected in sequence, and the key point position information of each monomer is determined based on the position information of the first end and the second end of the controller, and the length and / or bending degree of the controller is presented through the key point position information and / or the monomer's own shape; the key point position information includes: the position of the monomer and the orientation of the monomer.

[0016] Optionally, when the position of the controller changes, the controller uses the first monomer located at the second end as the first moving monomer, and drives the subsequent monomers to move in sequence, forming a dynamic curve that transmits toward the end point.

[0017] Optionally, the length and / or curvature of the controller is also related to the movement degree of the first device and / or the properties of the first virtual object; wherein the movement degree of the first device includes at least one of the following: speed, position and distance; the properties of the first virtual object include at least one of the following: mass, size, material.

[0018] In a second aspect, an embodiment of the present application provides a screen display device, comprising:

[0019] A display module is used to display a fused image, wherein the fused image includes a real environment image, a first virtual object, and a controller, wherein the controller includes a control line or a control rod, and the control line or the control rod is straight;

[0020] A first processing module, configured to change a position of the controller in the fused image in response to a first movement or a first gesture of the first device; wherein the position of the first virtual object in the fused image remains unchanged;

[0021] a second processing module, configured to, when the first end of the controller is close to the first virtual object and prompt information is presented in the fused image, in response to a first trigger or a second gesture of the first device, cause the controller to be bent, with the first end being located on the first virtual object and / or within a display area corresponding to the first virtual object, wherein the display area includes any pixel within the display area;

[0022] The third processing module is used to change the positions of the controller and the first virtual object in the fusion screen in response to the second movement or third gesture of the first device when the controller is bent, and the length and / or bending degree of the controller changes with the movement and / or gesture of the first device, and / or the size of the first virtual object changes with the movement and / or gesture of the first device; wherein, during the process of changing the position of the controller, the first end is located on the first virtual object and / or within the display area corresponding to the first virtual object.

[0023] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the screen display method as described in any one of the first aspects.

[0024] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps in the screen display method as described in any one of the first aspects are implemented.

[0025] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in the screen display method as described in any one of the first aspects.

[0026] In the present application, a fused screen is displayed, the fused screen including a real environment screen, a first virtual object, and a controller, the controller including a control line or a control rod, the control line or control rod being straight; in response to a first movement or a first gesture of a first device, the position of the controller in the fused screen is changed; wherein the position of the first virtual object in the fused screen remains unchanged; when the first end of the controller is close to the first virtual object and prompt information is presented in the fused screen, in response to a first trigger or a second gesture of the first device, the controller is bent, and the first end is located on the first virtual object and / or in a display area corresponding to the first virtual object, wherein the display area includes any pixel within the display area; after the controller is bent, in response to a second movement or a third gesture of the first device, the positions of the controller and the first virtual object in the fused screen are changed, and the length and / or degree of bending of the controller changes with the movement and / or gesture of the first device, and / or the size of the first virtual object changes with the movement and / or gesture of the first device; wherein, during the process of changing the position of the controller, the first end is located on the first virtual object and / or in the display area corresponding to the first virtual object. During the interaction process, the controller dynamically changes according to the real-time position information of the virtual object, thereby realizing the interaction between the first device and the object, improving the user's participation and immersion in the virtual world, and solving the problem of poor user experience of existing first device interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0028] Figure 1 This is a flow chart of a screen display method provided by an embodiment of the present application;

[0029] Figure 2 is a ray diagram of a screen display method provided in an embodiment of the present application;

[0030] Figure 3 This is a schematic diagram of a controller control method for displaying a picture provided by an embodiment of the present application;

[0031] Figure 4 This is a schematic diagram of a controller control method of another screen display method provided in an embodiment of the present application;

[0032] Figure 5 This is a schematic diagram of key point settings of another screen display method provided by an embodiment of the present application;

[0033] Figure 6 This is a structural diagram of a screen display device provided in an embodiment of the present application;

[0034] Figure 7 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0036] Please refer to Figure 1 , an embodiment of the present application provides a screen display method, comprising:

[0037] Step 11: Displaying a fused image, wherein the fused image includes the real environment image, the first virtual object, and a controller, wherein the controller includes a control line or a control rod, and the control line or the control rod is straight;

[0038] In the examples of this application, please refer to Figure 2 In virtual reality (VR) applications, a first device is usually used as the main device for users to interact with a virtual world. The first device interacts with the virtual world by forming a virtual controller. The virtual world is presented in the form of a fused screen. The first device interacts with virtual objects in the virtual environment by pointing to and selecting them through the controller. The controller includes a control line or a joystick. The control line or joystick is straight and is usually visually represented as a straight line or light beam directly emitted from the first device. It can help users accurately point to and operate virtual objects in three-dimensional space, so that users can see their pointing direction, which helps to improve the intuitiveness of the interaction and the user's spatial perception.

[0039] In the embodiment of the present application, optionally, the real environment picture is a picture obtained by capturing the surrounding environment, and the real environment picture changes with changes in the capturing perspective and / or changes in the surrounding environment.

[0040] In an embodiment of the present application, the real environment picture changes with the change of the acquisition perspective and / or the change of the surrounding environment, that is, the acquisition of the real environment picture can be performed based on the user's movements sensed by the sensor worn on the user's head. The acquisition of the real environment can enable users to experience more realistic scenes, enhance immersion and participation, and improve the fun and participation of the experience.

[0041] In the embodiment of the present application, optionally, the fused image includes multiple virtual objects, and the multiple virtual objects include the first virtual object.

[0042] In the embodiment of the present application, optionally, the multiple virtual objects further include a second virtual object, and the second virtual object is associated with the first virtual object and / or moves with the first virtual object.

[0043] In an embodiment of the present application, the fused screen includes multiple virtual objects, which can be virtual display screens and arbitrary virtual elements. The user can select from the multiple virtual objects, wherein at least some of the virtual objects are controllable, and the user can interact with the controller by selecting the first virtual object as the target object to complete subsequent control operations; the second virtual object is associated with the first virtual object and / or moves with the first virtual object to enhance immersion and authenticity.

[0044] In an embodiment of the present application, the fused image is optionally displayed in layers, wherein each layer corresponds to one of the real environment image, the first virtual object, and the controller.

[0045] In the embodiment of the present application, different types of information are displayed in layers, which can avoid information confusion, make it easier for users to understand and operate, and enable the controller to serve as a visual guide to help users better understand how to interact with virtual objects.

[0046] In an embodiment of the present application, the corresponding virtual object in the fused screen is determined based on the projected position of the controller emitted by the first device in the fused screen, and the mass of the virtual object and / or the real-time position information of the virtual object are obtained, so that the degree of bending of the controller is dynamically calculated based on the mass of the virtual object and / or the real-time position information of the virtual object, thereby achieving the effect of flexible rays, reducing the discomfort of user operation, and enhancing the user's participation and immersion in the virtual world.

[0047] Step 12: In response to a first movement or a first gesture of the first device, changing the position of the controller in the fused image; wherein the position of the first virtual object in the fused image remains unchanged;

[0048] In an embodiment of the present application, in response to a first movement or a first gesture of the first device, wherein the first gesture may include but is not limited to operations such as raising, lowering, and moving the user's hand, the position of the controller in the fused screen is changed, wherein the controller is composed of multiple key points, and the key points may be capsules or cylinders, that is, a ray is divided into several equal parts to obtain multiple small line segments as the controller. In order to ensure a smooth curve and avoid more complicated calculations, 101 key points can be selected to constitute the controller; specifically, a 3D object of a cylinder or a capsule can be created according to actual needs using the MeshRender component to correspond to the multiple small line segments. The more the number, the smoother the bending degree of the line segments formed by these 3D objects, so that the bending degree of the controller can be dynamically calculated and simulated according to the quality of the virtual object and / or the real-time position information of the virtual object, so as to achieve the effect of flexible rays, reduce the discomfort of user operations, and enhance the user's participation and immersion in the virtual world.

[0049] Step 13: When the first end of the controller is close to the first virtual object and prompt information is displayed in the fused image, in response to the first trigger or the second gesture of the first device, the controller is bent, and the first end is located on the first virtual object and / or within the display area corresponding to the first virtual object, wherein the display area includes any pixel within the display area;

[0050] In an embodiment of the present application, the first virtual object can be a virtual display screen and any virtual element, and the first end can be located at any position of the virtual object. Taking the virtual display screen as an example, it can be within the virtual display screen or the border. The first trigger or second gesture can be but is not limited to selection and manipulation, such as: picking, moving and rotating operations.

[0051] Step 14: When the controller is bent, in response to the second movement or third gesture of the first device, the positions of the controller and the first virtual object in the fusion screen are changed, and the length and / or bending degree of the controller changes with the movement and / or gesture of the first device, and / or the size of the first virtual object changes with the movement and / or gesture of the first device; wherein, during the process of changing the position of the controller, the first end is located on the first virtual object and / or in the display area corresponding to the first virtual object.

[0052] In the embodiment of the present application, the shape of the controller changes dynamically according to the state of the first virtual object in the fusion screen. Figure 3 and Figure 4In response to the second movement or third gesture of the first device, the bending degree of the controller dynamically changes according to the mass of the virtual object and / or the real-time position information of the virtual object, thereby realizing interaction between the first device and the object through a flexible ray, reducing the user's discomfort in operation and enhancing the user's participation and immersion in the virtual world.

[0053] In an embodiment of the present application, a fused screen is displayed, the fused screen including a real environment screen, a first virtual object, and a controller, the controller including a control line or a control rod, the control line or control rod being straight; in response to a first movement or a first gesture of a first device, the position of the controller in the fused screen is changed; wherein the position of the first virtual object in the fused screen remains unchanged; when a first end of the controller is close to the first virtual object and prompt information is presented in the fused screen, in response to a first trigger or a second gesture of the first device, the controller is bent, and the first end is located on the first virtual object and / or within a display area corresponding to the first virtual object, wherein the display area includes any pixel within the display area; after the controller is bent, in response to a second movement or a third gesture of the first device, the positions of the controller and the first virtual object in the fused screen are changed, and the length and / or degree of bending of the controller changes with the movement and / or gesture of the first device, and / or the size of the first virtual object changes with the movement and / or gesture of the first device; wherein, during the process of changing the position of the controller, the first end is located on the first virtual object and / or within the display area corresponding to the first virtual object. During the interaction process, the controller dynamically changes according to the real-time position information of the virtual object, thereby realizing the interaction between the first device and the object, improving the user's participation and immersion in the virtual world, and solving the problem of poor user experience of existing first device interaction.

[0054] In the embodiment of the present application, optionally, the second end of the controller is a mapping point of the movement and / or gesture of the first device on the fused screen, and the controller appears smooth.

[0055] In the embodiment of the present application, the controller is composed of multiple key points, which can be capsules or cylinders. That is, a ray is divided into several equal parts to obtain multiple small line segments as the controller. Please refer to Figure 5If there is only one key point, it is similar to the state of a straight line; if there are two key points, the effect is similar to the state of a broken line; if there are multiple key points, the more key points there are, the smoother the curve is, but too many key points will make the calculation more complicated. Therefore, in order to ensure the smoothness of the curve and avoid more complicated calculations, 101 key points can be selected to form the controller, and the initial position information of the key points constituting the controller can be defined and stored in an array. Without affecting the performance, a smoother flexible ray experience effect can be obtained. In addition, the distance at which the controller and the object can interact is also set with an interaction threshold. For example, the interaction threshold is set to 20m, that is, interaction is possible within 20m, and interaction is not possible beyond the range.

[0056] Specifically, according to the position information of the first device, the initial position information of each key point, and the information of the virtual object, the position information from the first device to each key point and the position information from the virtual object to each key point are obtained;

[0057] The position information of each key point is dynamically updated and calculated in each frame of the first device's position change, and is synchronously saved in an array. The real-time position information of each key point is dynamically updated following the position change of the first device; the bending degree of the controller is dynamically changed according to the mass of the virtual object and / or the real-time position information of the virtual object, and the interaction between the first device and the object is realized through a flexible ray, which reduces the user's discomfort in operation and enhances the user's participation and immersion in the virtual world.

[0058] In an embodiment of the present application, optionally, the controller includes a plurality of monomers connected in sequence, and the key point position information of each monomer is determined based on the position information of the first end and the second end of the controller, and the length and / or bending degree of the controller is presented through the key point position information and / or the monomer's own form; the key point position information includes: the position of the monomer and the orientation of the monomer.

[0059] In an embodiment of the present application, in each frame of the first device position change, the starting position information of the first device and the position information of the virtual object, as well as the ending position information of the first device and the position information of the virtual object, and the time transformation are recorded; and the positions of the first device position information and the position information of the virtual object are synchronously updated; the position information of each key point in the controller is calculated, and based on the position ratio and curvature radius of each key point in the controller, the position transformation from the first device position to the key point and the position transformation from the target object to the impact point are obtained, thereby obtaining the position information of each key point in the controller that is dynamically updated following the first device;

[0060] Specifically include:

[0061] In the case of a space curve, the radius of curvature R is the length of the curvature vector, and in the case of a plane curve, the radius of curvature R must be taken as an absolute value; Wherein, s is the arc length of a fixed point on the curve, α is the tangent angle, and K is the curvature; if the radius R of the circle is larger, the central angle θ corresponding to the unit arc length s is smaller, which means that the farther the distance is, the smaller the curvature of the unit arc length is, that is, the curvature radius R is inversely proportional to the distance, and the position and orientation of the key point are dynamically changed according to the mass of the virtual object and / or the real-time position information of the virtual object, which means that when the virtual object is controlled, when the virtual object is closer to the user, the curvature of the controller is greater, that is, the curvature radius of the controller is inversely proportional to the distance of the virtual object from the user, as shown in FIG. Figure 3 and Figure 4 As shown; assuming that the distance parameter C between the launch position of the handle and the virtual object is constant, then

[0062] Regarding the definition of the mass of the object, in the Unity3D tool, the behavior of the user and the virtual object is physically controlled through the rigid body component, that is, the rigid body can accept forces and torques, so that the object moves in a realistic way. Based on the control of the rigid body component physics engine, the control effect of the virtual space object close to the real object can be simulated. If the mass parameter m is set (the default setting is 1, the unit is kg), the larger the mass parameter of the virtual object is set, for example, based on the size of 1 cubic decimeter, the mass of the wooden block is 0.8kg, and the mass of the iron block is 7.8kg; the position of the key point and the orientation of the key point are dynamically changed according to the mass of the virtual object and / or the real-time position information of the virtual object. The greater the mass of the virtual object, the greater the degree of curvature per unit arc length, that is, the radius of curvature of the controller is proportional to the mass of the virtual object; Wherein, m is the quality parameter of the target object;

[0063] If the controller is expressed in Cartesian coordinates as y(x), the curvature radius of the controller is: If the controller is given by the function x(t) and y(t) parameters, then the curvature radius of the controller is:

[0064] If γ:R→R n It is R n The curve parameters are: ρ:R→R, the curvature radius at each point of the simulation curve is:

[0065] As a special case, if f(t) is an R to R function, then γ(t) = (t, f(t)),

[0066] Continue to calculate the orientation information of each key point in the controller, according to the cross product of the vectors of each two adjacent key points Processing is done according to different situations: if the dot product of two adjacent key points is greater than 0 and less than 1, it means that the two vectors have the same direction, that is, the same upward or downward direction; otherwise, the direction of the vector of the key point itself is taken.

[0067] In an embodiment of the present application, by combining the mass of the virtual object and / or the real-time position information of the virtual object, the shape of the controller dynamically changes according to the mass of the virtual object and / or the real-time position information of the virtual object during the process of dragging the object, thereby realizing interaction between the first device and the object through a flexible ray, reducing the user's discomfort in operation, and enhancing the user's participation and immersion in the virtual world.

[0068] In an embodiment of the present application, optionally, when the position of the controller changes, the controller uses the first monomer located at the second end as the first moving monomer, and drives the subsequent monomers to move in sequence, forming a dynamic curve that transmits toward the end point.

[0069] In an embodiment of the present application, the controller is first moved following the position change of the first device, and the controlled virtual object then moves synchronously to achieve a slow motion process. The slow motion effect can be specifically implemented by dynamically updating the position information and orientation calculation of the key points of the controller in the LaterUpdate tool. The slow motion effect can reduce the user's discomfort and dizziness during operation, and enhance the user's participation and immersion in the virtual world.

[0070] In an embodiment of the present application, optionally, the length and / or curvature of the controller is also related to the movement degree of the first device and / or the attributes of the first virtual object; wherein the movement degree of the first device includes at least one of the following: speed, position and distance; the attributes of the first virtual object include at least one of the following: mass, size, material.

[0071] In an embodiment of the present application, the degree of movement of the first device may include but is not limited to at least one of the following: speed, position and distance; the attributes of the first virtual object may include but are not limited to at least one of the following: mass, size, material; by combining the length and / or bending degree of the controller with the degree of movement of the first device and the attributes of the first virtual object, the intuitiveness, immersion and operation accuracy of the user experience are significantly improved, and the user's interactive experience is enhanced.

[0072] Please refer to Figure 6 , an embodiment of the present application provides a picture display device, comprising:

[0073] A display module 61 is used to display a fusion image, wherein the fusion image includes a real environment image, a virtual object, and a controller, and the controller is in a straight shape;

[0074] a first processing module 62 configured to change a position of the controller in the fused image in response to a first movement or a first gesture of the first device; wherein the position of the virtual object in the fused image remains unchanged;

[0075] A second processing module 63 is configured to, when the first end of the controller is close to the virtual object and prompt information is displayed in the fused image, in response to the first trigger or the second gesture of the first device, cause the controller to be bent, with the first end being located on the virtual object and / or within a display area corresponding to the virtual object, wherein the display area includes any pixel within the display area;

[0076] The third processing module 64 is used to change the positions of the controller and the virtual object in the fusion screen in response to the second movement or third gesture of the first device when the controller is bent, and the length and / or bending degree of the controller changes with the movement / gesture of the first device, and / or the size of the virtual object changes with the movement / gesture of the first device; wherein, during the process of changing the position of the controller, the first end is located on the virtual object and / or within the display area corresponding to the virtual object.

[0077] In the embodiment of the present application, optionally, the real environment picture is a picture obtained by capturing the surrounding environment, and the real environment picture changes with changes in the capturing perspective and / or changes in the surrounding environment.

[0078] In the embodiment of the present application, optionally, the fused image includes multiple virtual objects, and the multiple virtual objects include the first virtual object.

[0079] In the embodiment of the present application, optionally, the multiple virtual objects further include a second virtual object, and the second virtual object is associated with the first virtual object and / or moves with the first virtual object.

[0080] In an embodiment of the present application, the fused image is optionally displayed in layers, wherein each layer corresponds to one of a real environment image, a virtual object, and a controller.

[0081] In the embodiment of the present application, optionally, the second end of the controller is a mapping point of the movement / gesture of the first device on the fused screen, and the controller appears smooth.

[0082] In an embodiment of the present application, optionally, the controller includes a plurality of monomers connected in sequence, and the key point position information of each monomer is determined based on the position information of the first end and the second end of the controller, and the length and / or bending degree of the controller is presented through the key point position information and / or the monomer's own form; the key point position information includes: the position of the monomer and the orientation of the monomer.

[0083] In an embodiment of the present application, optionally, when the position of the controller changes, the controller uses the first monomer located at the second end as the first moving monomer, and drives the subsequent monomers to move in sequence, forming a dynamic curve that transmits toward the end point.

[0084] In an embodiment of the present application, optionally, the length and / or curvature of the controller is also related to the movement degree of the first device and / or the attributes of the first virtual object.

[0085] The screen display device provided in the embodiment of the present application can achieve Figure 1 The various processes implemented by the method embodiment achieve the same technical effect and are not described here again to avoid repetition.

[0086] The present application embodiment provides an electronic device 70, see Figure 7 As shown, Figure 7 This is a principle block diagram of an electronic device 70 according to an embodiment of the present application, which includes a processor 71, a memory 72, and a program or instruction stored in the memory 72 and executable on the processor 71. When the program or instruction is executed by the processor, the steps in any screen display method of the present application are implemented.

[0087] An embodiment of the present application provides a readable storage medium, which stores programs or instructions. When the programs or instructions are executed by a processor, the various processes of the embodiments of the screen display method such as any of the above-mentioned items are implemented, and the same technical effects can be achieved. To avoid repetition, they will not be repeated here.

[0088] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and to avoid repetition, they will not be described here.

[0089] Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0090] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0091] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a service classification device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0093] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A screen display method, characterized in that: The method comprises: Displaying a fused image, wherein the fused image includes a real environment image, a first virtual object, and a controller, wherein the controller includes a control line or a control rod, and the control line or the control rod is straight; In response to a first movement or a first gesture of the first device, changing a position of the controller in the fused screen; wherein the position of the first virtual object in the fused screen remains unchanged; When the first end of the controller is close to the first virtual object and prompt information is displayed in the fused screen, in response to a first trigger or a second gesture of the first device, the controller is bent, and the first end is located on the first virtual object and / or within a display area corresponding to the first virtual object, wherein the display area includes any pixel within the display area; When the controller is bent, in response to the second movement or third gesture of the first device, the positions of the controller and the first virtual object in the fused screen are changed, and the length and / or bending degree of the controller changes with the movement and / or gesture of the first device, and / or the size of the first virtual object changes with the movement and / or gesture of the first device; wherein, during the process of changing the position of the controller, the first end is located on the first virtual object and / or within the display area corresponding to the first virtual object.

2. The screen display method according to claim 1, wherein: The real environment picture is a picture obtained by capturing the surrounding environment, and the real environment picture changes with changes in the capturing perspective and / or changes in the surrounding environment.

3. The screen display method according to claim 1, wherein: The fused image includes multiple virtual objects, and the multiple virtual objects include the first virtual object.

4. The screen display method according to claim 3, wherein: The plurality of virtual objects further includes a second virtual object that is associated with the first virtual object and / or moves with the first virtual object.

5. The screen display method according to claim 1, wherein: The fused image is displayed in layers, wherein each layer corresponds to one of the real environment image, the first virtual object, and the controller.

6. The screen display method according to claim 1, wherein: The second end of the controller is a mapping point of the movement and / or gesture of the first device on the fused screen, and the controller appears smooth.

7. The screen display method according to claim 6, wherein: The controller includes a plurality of cells connected in sequence, and key point position information of each cell is determined based on position information of a first end and a second end of the controller, and the length and / or curvature of the controller are presented by the key point position information and / or the shape of the cell itself; The key point position information includes: the position of the monomer and the orientation of the monomer.

8. The screen display method according to claim 7, wherein: When the position of the controller changes, the controller takes the first monomer located at the second end as the first moving monomer and drives the subsequent monomers to move in sequence, forming a dynamic curve that transmits toward the end point.

9. The screen display method according to claim 1, wherein: The length and / or curvature of the controller is also related to the movement degree of the first device and / or the properties of the first virtual object; wherein the movement degree of the first device includes at least one of the following: speed, position and distance; the properties of the first virtual object include at least one of the following: mass, size, material.

10. A screen display device, characterized in that: include: A display module is used to display a fused image, wherein the fused image includes a real environment image, a first virtual object, and a controller, wherein the controller includes a control line or a control rod, and the control line or the control rod is straight; A first processing module, configured to change a position of the controller in the fused image in response to a first movement or a first gesture of the first device; wherein the position of the first virtual object in the fused image remains unchanged; a second processing module, configured to, when the first end of the controller is close to the first virtual object and prompt information is presented in the fused image, in response to a first trigger or a second gesture of the first device, cause the controller to be bent, with the first end being located on the first virtual object and / or within a display area corresponding to the first virtual object, wherein the display area includes any pixel within the display area; The third processing module is used to change the positions of the controller and the first virtual object in the fusion screen in response to the second movement or third gesture of the first device when the controller is bent, and the length and / or bending degree of the controller changes with the movement and / or gesture of the first device, and / or the size of the first virtual object changes with the movement and / or gesture of the first device; wherein, during the process of changing the position of the controller, the first end is located on the first virtual object and / or within the display area corresponding to the first virtual object.

11. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps in the screen display method according to any one of claims 1 to 9.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps in the screen display method according to any one of claims 1 to 9 are implemented.

13. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps in the screen display method according to any one of claims 1 to 9.

Images