Electronic equipment control method and device, electronic equipment and storage medium

Abstract

The invention provides a control method and device of electronic equipment, the electronic equipment and a storage medium. The electronic equipment comprises a display generation component and one or more input equipment, wherein the input equipment comprises eye tracking equipment and gesture detection equipment; the method comprises the following steps: displaying a physical environment in which a user is located through a display generation component; in response to a preset processing flow, generating at least one second object on the first object in the physical environment; detecting that the gaze of the user points to a target second object in the second object through eye tracking equipment, and generating a visual indication object; detecting a hand gesture of the user via the gesture detection device, and controlling the target second object based on the hand gesture and the visual indication object; and when the fitting degree of the target second object and the first object is adjusted based on the hand gesture to meet user requirements, determining attribute information of the target second object. The interactive experience of the user is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a control method and device for an electronic device, an electronic device, and a storage medium. Background Art

[0002] Extended reality includes virtual reality, mixed reality, augmented reality, etc. In the mixed reality world, objects in the real world and their corresponding virtual models can be displayed. Before displaying the virtual model, calibration is usually required. Summary of the invention

[0003] The present disclosure provides a control method, an apparatus, an electronic device and a storage medium for a mixed reality device.

[0004] The present disclosure adopts the following technical solutions.

[0005] In some embodiments, the present disclosure provides a control method of an electronic device, the electronic device comprising: a display generation component and one or more input devices, the input devices comprising an eye tracking device and a gesture detection device; the method comprising:

[0006] Displaying the user's physical environment via a display generation component;

[0007] In response to a preset processing flow, generating at least one second object on the first object in the physical environment;

[0008] detecting, via the eye tracking device, that the user's gaze is directed toward a target second object among the second objects, and generating a visual indication object;

[0009] detecting a hand gesture of the user via the gesture detection device, and controlling the target second object based on the hand gesture and the visual indication object;

[0010] When the fit between the target second object and the first object is adjusted based on the hand gesture to meet the user's needs, the attribute information of the target second object is determined.

[0011] In some embodiments, the present disclosure provides an electronic device, including: a display generation component and one or more input devices, the input devices including an eye tracking device and a gesture detection device;

[0012] The display generation component is used to display the physical environment where the user is located;

[0013] The display generation component is further configured to generate at least one second object on the first object in the physical environment in response to a preset processing flow;

[0014] The eye tracking device is used to detect the gaze direction of the user;

[0015] The display generation component is further used for: detecting, via the eye tracking device, that the user's gaze is directed toward a target second object among the second objects, and generating a visual indication object;

[0016] The gesture detection device is used to detect the hand gesture of the user;

[0017] The display generation component is also used to: control the target second object based on the hand gesture of the user detected by the gesture detection device and the visual indication object;

[0018] The display generation component is further used to: determine the attribute information of the target second object when the fit between the target second object and the first object is adjusted based on the hand gesture to meet the user's needs.

[0019] In some embodiments, the present disclosure provides an electronic device, comprising: at least one memory and at least one processor;

[0020] The memory is used to store program codes, and the processor is used to call the program codes stored in the memory to execute the above method.

[0021] In some embodiments, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and when the program code is executed by a processor, the processor is prompted to perform the above method.

[0022] The control method of the electronic device provided by the embodiment of the present disclosure selects the target second object by the user's gaze, and uses hand gestures and visual indication objects to control the target second object. The entire process does not require the use of tools such as handles, which provides a better user experience and can be used to calibrate the first object. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar figures represent the same or similar elements. It should be understood that the drawings are schematic and that the components and elements are not necessarily drawn to scale.

[0024] Figure 1 is a schematic diagram of using a mixed reality device according to an embodiment of the present disclosure.

[0025] Figure 2 It is a schematic diagram of executing the calibration process of an embodiment of the present disclosure.

[0026] Figure 3 is a flowchart of a method for controlling an electronic device according to an embodiment of the present disclosure.

[0027] Figure 4 It is a schematic diagram of the integration of objects in the mixed reality world according to an embodiment of the present disclosure.

[0028] Figures 5 to 13 It is a schematic diagram of the execution process of the control method of the electronic device in the embodiment of the present disclosure.

[0029] Fig.14 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0031] It should be understood that the various steps described in the method embodiments of the present disclosure can be performed in sequence and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0033] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0034] It should be noted that the modification of “one” mentioned in the present disclosure is illustrative rather than restrictive, and those skilled in the art should understand that it should be understood as “one or more” unless otherwise clearly indicated in the context.

[0035] It should be noted that the steps of obtaining the user's personal data mentioned in the present disclosure are performed with the user's authorization. For example: in response to receiving the user's active request, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require the acquisition and use of the user's personal data. Thus, the user can autonomously choose whether to provide personal information to the electronic device, application, server or storage medium and other software or hardware that performs the operation of the technical solution of the present disclosure according to the prompt message. As an optional but non-limiting implementation method, in response to receiving the user's active request, the method of sending the prompt message to the user can be, for example, a pop-up window, and the prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal data to the electronic device. It can be understood that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation method of the present disclosure. Other methods that meet relevant laws and regulations can also be applied to the implementation method of the present disclosure. It can be understood that the data involved in the technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws and regulations and relevant provisions.

[0036] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0037] The solution provided by the embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0038] In one or more embodiments of the present disclosure, mixed reality technology can combine reality and virtuality through computers to provide users with a mixed reality space that allows human-computer interaction. In the mixed reality space, users can use mixed reality devices such as helmet-mounted displays (Head Mount Display, HMD) to conduct social interaction, entertainment, learning, work, telecommuting, and create UGC (User Generated Content).

[0039] refer to Figure 1 Users can enter the mixed reality space through mixed reality devices such as head-mounted glasses, and control their own virtual characters (Avatar) in the mixed reality space to engage in social interaction, entertainment, learning, remote work, etc. with virtual characters controlled by other users.

[0040] In the related art, in the mixed reality space, such as Figure 1 As shown, the user can implement relevant interactive operations through one or more input devices that communicate with the electronic device, and the input device includes a controller (such as a handle) connected to the device.

[0041] The mixed reality devices described in the embodiments of the present disclosure may include but are not limited to the following types:

[0042] The computer-side mixed reality device uses the computer to perform relevant calculations and data output of the mixed reality function, and the external computer-side mixed reality device uses the data output by the computer to achieve the mixed reality effect.

[0043] The mobile expansion device supports the setting of a mobile terminal (such as a smart phone) in various ways (such as a head-mounted display with a dedicated card slot). Through a wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations of the mixed reality function and outputs data to the mobile mixed reality device, such as watching a mixed reality video through the mobile terminal's APP.

[0044] The all-in-one mixed reality device has a processor for performing related calculations for mixed reality functions, and thus has independent mixed reality input and output functions. It does not need to be connected to a computer or mobile terminal and has a high degree of freedom in use.

[0045] Of course, the form of mixed reality devices is not limited to this, and can be further miniaturized or enlarged as needed.

[0046] In a mixed reality world (a world generated using mixed reality technology), virtual models corresponding to objects in the real world (such as objects) can be displayed. When displaying virtual models, it is necessary to calibrate the objects in the real world. Calibration usually includes determining the position and size of the objects in the real world, and also assigning some physical properties to the objects (such as occlusion properties, collision properties), etc., so that a virtual model with the same size and overlapping position as the object can be displayed in the mixed reality world. The mixed reality world is usually displayed using a mixed reality device. In related technologies, mixed reality devices usually have paired handles. When performing calibration operations through the handles, the operation convenience may not be sufficient.

[0047] Visual Tracking:

[0048] In some embodiments of the present disclosure, an eye tracking device in an electronic device (e.g., a head-mounted electronic device) performs visual tracking on the user's eyes, and the eye tracking device can be used to obtain the user's visual information, such as the user's line of sight, gaze point, etc. In one embodiment, the eye tracking device includes at least one eye tracking camera (e.g., an infrared (IR) or near infrared (NIR) camera), and an illumination source (e.g., an IR or NIR light source, such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye tracking camera may be directed toward the user's eyes to receive the IR or NIR light reflected directly from the eyes by the light source, or alternatively may be directed toward a "hot" mirror located between the user's eyes and the display panel, which reflects the IR or NIR light from the eyes to the eye tracking camera while allowing visible light to pass through. The gaze tracking device optionally captures images of the user's eyes (e.g., as a video stream captured at 60 to 120 frames per second (fps)), analyzes these images to generate gaze tracking information, and transmits the gaze tracking information to the electronic device, so that some human-computer interaction functions can be completed based on the user's gaze information, such as realizing content navigation based on gaze information. In some embodiments, the user's two eyes are tracked separately by corresponding eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by corresponding eye tracking cameras and illumination sources.

[0049] Hand tracking:

[0050] In some embodiments of the present disclosure, a gesture detection device in an electronic device detects the user's hand information. The user's hand information, such as the user's hand gestures, can be obtained through the gesture detection device. The hand gesture detection device is part of the electronic device (for example, embedded in a head-mounted device or attached to a head-mounted device). In some embodiments, the gesture detection device includes an image sensor (for example, one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information including at least the hand of a human user. The image sensor captures hand images with sufficient resolution so that fingers and their corresponding positions can be distinguished. In some embodiments, the gesture detection device captures and processes a time series of depth maps containing the user's hand when the user moves his hand (for example, the entire hand or one or more fingers). The software running on the image sensor and processor of the electronic device processes the 3D mapping data to extract image block descriptors of the hand in these depth maps. The software can match these descriptors with image block descriptors stored in a database based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D position of the user's hand joints and finger tips. The software can also analyze the trajectory of the hand and / or finger over multiple frames in the sequence to identify gestures. The pose estimation function described in the present disclosure can be alternated with the motion tracking function, so that only one pose estimation based on the image block is performed every two (or more) frames, and the tracking is used to find the changes in pose that occur on the remaining frames, and provide pose, motion and gesture information to the application running on the electronic device. The program can, for example, move and modify the image presented on the display generation component of the electronic device in response to the pose and / or gesture information, or perform other functions, such as executing control instructions corresponding to gestures.

[0051] Calibration process:

[0052] In some embodiments of the present disclosure, for example, a forward-facing camera (RGB camera or / and depth camera) on a head-mounted electronic device captures a graphic of the physical environment, which is then merged in real time with a computer-generated image and displayed on a display generation component, thereby giving a feeling that the human eye can directly see the surrounding real world through the electronic device. This function can be called a perspective function. Under the perspective function of the electronic device, objects in the physical environment can be calibrated (see Figure 2 ), calibration mainly refers to the user depicting the outline of objects in the physical environment, such as a room, in a certain way to help electronic devices understand the physical environment so that they have the occlusion and collision relationships of the physical environment.

[0053] like Figure 3 As shown, Figure 3The electronic device includes: a display generation component and one or more input devices, the display generation component may be a head mounted display, the input device includes an eye tracking device and a gesture detection device, and the eye tracking device may detect the use of the electronic device; the method includes:

[0054] S11. Display the user's physical environment via a display generation component.

[0055] S12: In response to a preset processing flow, generate at least one second object on the first object in the physical environment.

[0056] S13. Detect, via the eye tracking device, that the user's gaze is directed toward a target second object among the second objects, and generate a visual indication object.

[0057] S14. Detect the user's hand gesture via the gesture detection device, and control the target second object based on the hand gesture and the visual indication object.

[0058] S15. When the fit between the target second object and the first object is adjusted based on the hand gesture to meet user needs, determine attribute information of the target second object.

[0059] In some embodiments, the electronic device of the method proposed in the present disclosure may be a mixed reality device, and the user may be a user wearing and using the electronic device. The user may be using the mixed reality device, and after the real physical environment in which the user is located is displayed in the electronic device for the current user, a preset processing flow (such as a calibration flow) may be executed. The first object is a real object in a real physical environment, such as an object in a real physical environment, specifically a wall, a ground, or a furniture, etc. The first object is also displayed in a mixed reality world, and the mixed reality world is the world displayed by the display generation component, including a real physical environment and virtual objects. The eye tracking device on the electronic device may have the function of detecting the user's eyes, so that the user's line of sight can be determined. The target second object in the second object is determined according to the user's line of sight. If the second object is one, the second object may be the target second object. For example, the second object that the user's line of sight looks at is the target second object. The second object is an object related to the first object in the mixed reality world, such as a virtual model or a part of a virtual model corresponding to the first object, and the second object may be a purely virtual object.

[0060] In some embodiments, a visual indication object can be used to show the location of the user's line of sight, which can be a cursor. The user can control the target second object by performing gesture operations (i.e., hand gestures) in combination with the visual indication object, such as adjusting the position, size, etc. In some embodiments of the present disclosure, the user's line of sight can be used to provide position navigation, and hand gestures can provide further operations, and the two are combined to complete the control. The target second object has a fit with the first object, such as the similarity between the two, and the user can perform a specific gesture operation to indicate that the fit has met the user's needs. At this time, the attribute information of the target second object will be determined, and the attribute information of the target second object can be fixed at this time.

[0061] In some embodiments of the present disclosure, a target second object is selected by the user's gaze, and the second object is controlled using hand gestures and visual indication objects. The entire process does not require the use of external control tools such as a handle. This method can be used to calibrate the first object. Compared with the method of using a handle, the user experience is better and the convenience of interaction is expanded.

[0062] In some embodiments of the present disclosure, the eye tracking device is used to detect that the user's gaze is directed to a target second object among the second objects, including: the second object has a corresponding score, and when the user's sight passes over or stays on the second object, the score of the second object that the sight passes over or stays on is changed, or the scores of other second objects except the second object that the sight passes over or stays on are changed; and the target second object is determined based on the score of the second object.

[0063] In some embodiments, since the user's line of sight may be somewhat erratic, in order to better determine the target second object that the user's line of sight is looking at, such as Figure 4 As shown, a preset score, such as 100 points, can be set for a second object (such as an object) in the mixed reality world. When the user's line of sight passes over or stays on the second object in the mixed reality world, the score of the second object will remain unchanged or increase. When the user's line of sight does not pass over or stay on the second object in the mixed reality world, the score of the second object that is not passed over or stayed on will decrease. Finally, the second object with a high score can be used as the target second object. The target second object is the second object that the user's line of sight passes over more times or stays on for a longer time, which indicates that the user is looking at the second object.

[0064] In some embodiments of the present disclosure, detecting, via the eye tracking device, that the user's gaze is directed toward a target second object among the second objects includes: in response to receiving a first voice message indicating a selection intention, taking the second object that the user's line of sight is looking at when the first voice message is received as the target second object.

[0065] In some embodiments, when the user's line of sight is looking at a second object, the current user can send a first voice message, for example, the current user can say "mark it", "select it", etc. The first voice message does not need to say the name of the second object. The first voice message indicates the user's selection intention. At this time, the second object in the mixed reality world that the user's line of sight is looking at is also used as the target second object. The second object can be a purely virtual object, not a real object in the physical environment. The attribute information of the target second object includes one or more of the following: the size of the target second object, the position of the target second object, the shape of the target second object, the posture of the target second object, and the semantics of the target second object. The semantics of the target second object can be the type of the target second object, such as wall, ground, furniture, etc.

[0066] In some embodiments of the present disclosure, detecting the hand gesture of the user via the gesture detection device and controlling the target second object based on the hand gesture and the visual indication object includes: detecting the hand gesture of the user via the gesture detection device, and adjusting the attribute information of the target second object in response to an adjustment instruction corresponding to the hand gesture in combination with the visual indication object.

[0067] In some embodiments, one or more of the attribute information of the target second object can be adjusted, for example, one or more of the size, position, shape, posture, and semantics of the target second object can be adjusted so that the position of the target second object can be the same size, position, shape, posture, and / or the same type of object as its corresponding first object. In this way, in the mixed reality world, the electronic device can understand the first object through the target second object, and when processing the first object, it can be processed based on the attributes of the target second object. For example, if you want to achieve a collision effect between a virtual object and a first object, you can use the target second object to calculate the interaction effect with the virtual object, and achieve interaction between the physical environment and the virtual object.

[0068] In some embodiments, the method further includes: in response to the first type of gesture operation, determining whether the fit between the target second object and the first object meets the user's needs. In some embodiments, the user can perform the first type of gesture operation to indicate that the current second object and the first object meet their needs.

[0069] In some embodiments of the present disclosure, when the first object is the ground, there may be only one second object, which is used as the target second object after being pointed at by the user, and the second object is a plane indication object corresponding to the ground, and the plane indication object can be any virtual object that can represent the plane, such as a virtual ground model, a virtual frame, a virtual plane coordinate, etc. The user's hand gesture is detected via the gesture detection device, and the target second object is controlled based on the hand gesture and the visual indication object, including: in response to the visual indication object being located at the plane indication object and the hand gesture being the second type of gesture operation, the plane indication object is lifted or lowered.

[0070] In some embodiments of the present disclosure, when the first object is the ground, after entering the calibration process (preset processing process), a virtual ground model (plane indication object) corresponding to the ground is generated and displayed; the user looks at the virtual ground model and determines that the virtual ground model is the target second object. Figures 5 to 7 As shown, after entering the process of calibrating the ground, the ground position can be calculated by the ground monitoring algorithm and a virtual ground model can be generated. The virtual ground model can be an infinite or finite plane, which can be parallel to the ground. The user's line of sight looks at the virtual ground model, and a cursor is displayed on the virtual ground model as a visual indicator object, indicating that the user selects the virtual ground model as the target second object. Figure 6 As shown, when the visual indication object is located on the virtual ground model, the second type of gesture operation can be performed. The second type of gesture operation is a specific type of gesture operation, for example, it can be raising or lowering the arm. After the second type of gesture operation is recognized, the virtual ground model will be raised or lowered (for example, the virtual ground model will be raised or lowered at a fixed speed). In this way, the current user can adjust the position of the virtual ground model through the second type of gesture operation to achieve ground calibration. If the first type of gesture operation is detected later (see Figure 7 ), then record the height position and other attribute information of the current virtual ground model to complete the ground calibration.

[0071] In some embodiments of the present disclosure, when the first object is a wall, the second object is a corner line model of each corner line of the wall; detecting, via the eye tracking device, that the user's gaze is directed to a target second object in the second object, includes: in response to the user's line of sight looking at the target corner line model, determining that the target corner line model is the target second object; detecting, via the gesture detection device, a hand gesture of the user, and controlling the target second object based on the hand gesture and the visual indication object, includes: in response to the visual indication object being located in the target corner line model and the hand gesture being a third-category gesture operation, calibrating the target corner line model; when the fit between the target second object and the first object is adjusted to meet user needs based on the hand gesture, determining the attribute information of the target second object includes: after the calibrated corner line model forms a virtual closed surface, determining the attribute information of the calibrated corner line model and using the virtual closed surface as a virtual wall model for calibrating the wall.

[0072] In some embodiments, after completing the preset processing flow for calibrating the ground, the preset processing flow for calibrating the wall may be automatically entered. In this case, the first object is the wall. Figure 8 As shown, the current user's line of sight is looking at a corner line model on the wall, and the corner line model being looked at can be highlighted as the target second object, and then the user can perform a third type of gesture operation to confirm the calibration of the target corner line (such as Fig. 9 As shown), then repeat the above steps to calibrate other corner line models of the wall. After the calibrated corner line models form a closed surface, it means that all corner line models of the wall have been calibrated. The closed surface formed by these corner line models will be used as the virtual wall model of the wall. The virtual wall model can overlap with the wall. Then, Fig.10 As shown, the first type of gesture operation is performed to complete the calibration of the wall.

[0073] In some embodiments of the present disclosure, when the first object is a piece of furniture, the second object is a virtual shape model corresponding to the piece of furniture; detecting, via the eye tracking device, that the user's gaze is directed to a target second object in the second object, including: determining, based on the user's line of sight, that the target shape model that the line of sight is looking at is the target second object; when the fit between the target second object and the first object is adjusted to meet user needs based on the hand gesture, determining attribute information of the target second object, including: determining the attribute information of the target shape model in response to a fourth type of gesture operation.

[0074] In some embodiments, after completing the preset process of calibrating the wall, the preset process of calibrating the furniture is automatically entered. Fig.11 As shown, the shape model will be displayed when calibrating furniture. There can be multiple shape models, such as Fig.11 The hexahedron and quadrilateral in the image are shown in the figure. The shape model that the user's line of sight is looking at is the target shape model. If the user performs the fourth type of gesture operation (the fourth type of gesture operation can be the same as the third type of gesture operation), such as pressing the thumb, the attribute information of the target shape model can be selected and determined. The target shape model will be used as the virtual furniture model. The virtual furniture model is a model corresponding to the furniture in the mixed reality world. The result of calibrating the furniture can be to generate a virtual furniture model with the same size, shape and position as the furniture.

[0075] In some embodiments of the present disclosure, the user's hand gesture is detected via the gesture detection device, and the target second object is controlled based on the hand gesture and the visual indication object, including: determining a third object based on the visual indication object, and displaying a virtual target shape model at the position of the third object; and in response to a fifth type of gesture operation, positioning the target shape model at the position where the target shape model is currently located.

[0076] In some embodiments, after selecting the target shape model, the current user can display the target shape model at the location of the third object by looking at the ground, wall, ceiling, etc., and then position the target shape model at its current location, such as the location of the third object, by performing the fifth type of gesture operation, such as making a fist. For example, the target shape model can be a hexahedron or a quadrilateral, which can have a default size. The default size of the hexahedron can be 1.2×0.7×1.5m, and the default size of the quadrilateral can be 1×1m.

[0077] In some embodiments of the present disclosure, the gesture detection device detects the hand gesture of the user, and controls the target second object based on the hand gesture and the visual indication object, further comprising: determining a target edge in the target shape model according to the visual indication object, and adjusting the size of the target edge in response to the sixth type of gesture operation performed by the user. In some embodiments, the target edge can be any edge in the target shape model, such as Fig.12 As shown, the size of the target edge can be enlarged or reduced by, for example, two-finger zooming (the sixth gesture operation). In some embodiments, the maximum threshold and minimum threshold of the edge are pre-set, and the size of the target edge cannot be set to be greater than the maximum threshold or less than the minimum threshold. The maximum threshold may be 3m, and the minimum threshold may be 0.1m, because furniture in a real environment will not be too large or too small.

[0078] In some embodiments, the hand gesture of the user is detected via the gesture detection device, and the target second object is controlled based on the hand gesture and the visual indication object, further comprising: rotating the target shape model in response to a seventh type of gesture operation performed by the user when the visual indication object is located on the target shape model. In some embodiments, the seventh type of gesture operation can be a two-handed grabbing gesture plus a clockwise or counterclockwise rotation, so that the target shape model also rotates clockwise or counterclockwise.

[0079] In some embodiments, the gesture detection device detects the user's hand gesture, and the target second object is controlled based on the hand gesture and the visual indication object, and further includes: in response to the eighth type of gesture operation performed by the user when the visual indication object is located on the target shape model, moving the target shape model. In some embodiments, the eighth type of gesture operation can be, for example, grabbing with both hands and moving up, down, left, right, front, and back, so that the virtual furniture model also moves up, down, left, right, front, and back accordingly. When the eighth type of gesture operation moves the virtual furniture model, the speed of movement can be less than the speed of moving the virtual furniture model by sight, so that it is convenient to fine-tune through the eighth type of gesture operation.

[0080] In some embodiments, such as Fig.13 As shown, the preset processing flow of calibrating furniture can be completed by executing the first type of gesture operation, and the entire calibration process can be completed at this time.

[0081] In some embodiments of the present disclosure, eye tracking and gesture interaction are added, and the cursor and other visual indicators are controlled by the line of sight to select the target second object and move the position, and gestures are used for real-time operation. The interaction mode is expanded through multi-modal interaction, which can reduce the dependence on the handle. It can be used to calibrate the room in the mixed reality scene.

[0082] In some embodiments of the present disclosure, an electronic device is also provided, which may be a mixed reality device, such as a head-mounted mixed reality device, including: a display generation component and one or more input devices, wherein the input device includes an eye tracking device and a gesture detection device;

[0083] The display generation component is used to display the physical environment where the user is located;

[0084] The display generation component is further configured to generate at least one second object on the first object in the physical environment in response to a preset processing flow;

[0085] The eye tracking device is used to detect the gaze direction of the user;

[0086] The display generation component is further used for: detecting, via the eye tracking device, that the user's gaze is directed toward a target second object among the second objects, and generating a visual indication object;

[0087] The gesture detection device is used to detect the hand gesture of the user;

[0088] The display generation component is also used to: control the target second object based on the hand gesture of the user detected by the gesture detection device and the visual indication object;

[0089] The display generation component is further used to: determine the attribute information of the target second object when the fit between the target second object and the first object is adjusted based on the hand gesture to meet the user's needs.

[0090] In some embodiments, detecting, via the eye tracking device, that the user's gaze is directed to a target second object among the second objects comprises:

[0091] The second object has a corresponding score, and when the user's sight passes over or stays on the second object, the score of the second object passed over or stayed by the sight is changed, or the scores of other second objects except the second object passed over or stayed by the sight are changed; the target second object is determined according to the score of the second object;

[0092] or,

[0093] In response to receiving a first voice message indicating a selection intention, a second object that the user is looking at when the first voice message is received is used as the target second object.

[0094] In some embodiments,

[0095] The second object is a virtual object; and / or,

[0096] The attribute information of the target second object includes one or more of the following: the size of the target second object, the shape of the target second object, the position of the target second object, the posture of the target second object, and the semantics of the target second object; and / or,

[0097] In some embodiments, the gesture detection device is further configured to determine, in response to the first type of gesture operation, whether the fit between the target second object and the first object meets user requirements.

[0098] In some embodiments, controlling the target second object based on the hand gesture and the visual indication object includes: adjusting the attribute information of the target second object by detecting the hand gesture of the user via the gesture detection device and responding to the adjustment instruction corresponding to the hand gesture by the visual indication object.

[0099] In some embodiments, when the first object is the ground, the second object is a virtual plane indicating object corresponding to the ground;

[0100] The user's hand gesture is detected via the gesture detection device, and the target second object is controlled based on the hand gesture and the visual indication object, including: in response to the visual indication object being located on the plane indication object and the hand gesture being a second type of gesture operation, lifting or lowering the plane indication object.

[0101] In some embodiments, when the first object is a wall, the second object is a corner line model of each corner line of the wall;

[0102] Detecting, via the eye tracking device, that the user's gaze is directed to a target second object in the second objects, comprising: in response to the user's sight looking at a target wall corner line model, determining that the target wall corner line model is the target second object;

[0103] Detecting the user's hand gesture via the gesture detection device, and controlling the target second object based on the hand gesture and the visual indication object, including: in response to the visual indication object being located at the target corner line model and the hand gesture being a third type of gesture operation, calibrating the target corner line model;

[0104] When the fit between the target second object and the first object is adjusted based on the hand gesture to meet user needs, determining the attribute information of the target second object includes: after the calibrated corner line model forms a virtual closed surface, determining the attribute information of the calibrated corner line model and using the virtual closed surface as a virtual wall model for calibrating the wall.

[0105] In some embodiments, when the first object is furniture, the second object is a virtual shape model corresponding to the furniture;

[0106] Detecting, via the eye tracking device, that the user's gaze is directed to a target second object in the second object, comprising: determining, according to the user's line of sight, a target shape model that the line of sight is looking at as the target second object;

[0107] When the fit between the target second object and the first object is adjusted based on the hand gesture to meet user needs, the attribute information of the target second object is determined, including: in response to a fourth type of gesture operation, determining the attribute information of the target shape model.

[0108] In some embodiments, detecting the user's hand gesture via the gesture detection device, and controlling the target second object based on the hand gesture and the visual indication object includes:

[0109] Determine a third object according to the visual indication object, and display a virtual target shape model at the position of the third object;

[0110] In response to the fifth type of gesture operation, the target shape model is positioned at the current position of the target shape model.

[0111] In some embodiments, detecting the user's hand gesture via the gesture detection device, and controlling the target second object based on the hand gesture and the visual indication object, further includes one or more of the following:

[0112] determining a target edge in the target shape model according to the visual indication object, and adjusting a size of the target edge in response to a sixth type of gesture operation performed by the user;

[0113] In response to a seventh type of gesture operation performed by the user when the visual indication object is located on the target shape model, rotating the target shape model;

[0114] In response to an eighth type of gesture operation performed by the user when the visual indication object is located on the target shape model, moving the target shape model.

[0115] For the embodiments of the device, since they basically correspond to the method embodiments, the relevant parts refer to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the modules described as separation modules may or may not be separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present embodiment. Those of ordinary skill in the art can understand and implement without paying creative work.

[0116] The method and device of the present disclosure are described above based on the embodiments and application examples. In addition, the present disclosure also provides an electronic device and a computer-readable storage medium, which are described below.

[0117] Reference below Fig.14, which shows a schematic diagram of the structure of an electronic device (such as a terminal device or a server) 800 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptops, PADs (tablet computers), and head-mounted VR devices. The electronic device described below is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0118] The electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0119] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although the electronic device 800 with various devices is shown in the figure, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have alternatively.

[0120] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0121] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0122] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0123] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0124] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method of the present disclosure.

[0125] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0126] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0128] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0129] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] According to one or more embodiments of the present disclosure, a control method of an electronic device is provided, wherein the electronic device comprises: a display generation component and one or more input devices, wherein the input devices comprise an eye tracking device and a gesture detection device; the method comprises:

[0131] Displaying the user's physical environment via a display generation component;

[0132] In response to a preset processing flow, generating at least one second object on the first object in the physical environment;

[0133] detecting, via the eye tracking device, that the user's gaze is directed toward a target second object among the second objects, and generating a visual indication object;

[0134] detecting a hand gesture of the user via the gesture detection device, and controlling the target second object based on the hand gesture and the visual indication object;

[0135] When the fit between the target second object and the first object is adjusted based on the hand gesture to meet the user's needs, the attribute information of the target second object is determined.

[0136] According to one or more embodiments of the present disclosure, a method for controlling an electronic device is provided, wherein detecting, via an eye tracking device, that a user's gaze is directed to a target second object among the second objects, comprises:

[0137] The second object has a corresponding score, and when the user's sight passes over or stays on the second object, the score of the second object passed over or stayed by the sight is changed, or the scores of other second objects except the second object passed over or stayed by the sight are changed; the target second object is determined according to the score of the second object;

[0138] or,

[0139] In response to receiving a first voice message indicating a selection intention, a second object that the user is looking at when the first voice message is received is used as the target second object.

[0140] According to one or more embodiments of the present disclosure, a method for controlling an electronic device is provided, wherein the second object is a virtual object; and / or,

[0141] The attribute information of the target second object includes one or more of the following: the size of the target second object, the shape of the target second object, the position of the target second object, the posture of the target second object, and the semantics of the target second object; and / or,

[0142] In response to the first type of gesture operation, it is determined that the fit between the target second object and the first object meets user requirements.

[0143] According to one or more embodiments of the present disclosure, a method for controlling an electronic device is provided, wherein detecting a hand gesture of the user via the gesture detection device, and controlling the target second object based on the hand gesture and the visual indication object comprises:

[0144] The hand gesture of the user is detected via the gesture detection device, and the attribute information of the target second object is adjusted in response to the adjustment instruction corresponding to the hand gesture in combination with the visual indication object.

[0145] According to one or more embodiments of the present disclosure, a method for controlling an electronic device is provided, wherein when the first object is a ground surface, the second object is a virtual plane indicating object corresponding to the ground surface;

[0146] The user's hand gesture is detected via the gesture detection device, and the target second object is controlled based on the hand gesture and the visual indication object, including: in response to the visual indication object being located on the plane indication object and the hand gesture being a second type of gesture operation, lifting or lowering the plane indication object.

[0147] According to one or more embodiments of the present disclosure, a control method of an electronic device is provided, wherein when the first object is a wall, the second object is a corner line model of each corner line of the wall;

[0148] Detecting, via the eye tracking device, that the user's gaze is directed to a target second object in the second objects, comprising: in response to the user's sight looking at a target wall corner line model, determining that the target wall corner line model is the target second object;

[0149] Detecting the user's hand gesture via the gesture detection device, and controlling the target second object based on the hand gesture and the visual indication object, including: in response to the visual indication object being located at the target corner line model and the hand gesture being a third type of gesture operation, calibrating the target corner line model;

[0150] When the fit between the target second object and the first object is adjusted based on the hand gesture to meet user needs, determining the attribute information of the target second object includes: after the calibrated corner line model forms a virtual closed surface, determining the attribute information of the calibrated corner line model and using the virtual closed surface as a virtual wall model for calibrating the wall.

[0151] According to one or more embodiments of the present disclosure, a method for controlling an electronic device is provided, wherein when the first object is a piece of furniture, the second object is a virtual shape model corresponding to the piece of furniture;

[0152] Detecting, via the eye tracking device, that the user's gaze is directed to a target second object in the second object, comprising: determining, according to the user's line of sight, a target shape model that the line of sight is looking at as the target second object;

[0153] When the fit between the target second object and the first object is adjusted based on the hand gesture to meet user needs, the attribute information of the target second object is determined, including: in response to a fourth type of gesture operation, determining the attribute information of the target shape model.

[0154] According to one or more embodiments of the present disclosure, a control method of an electronic device is provided, wherein a hand gesture of the user is detected via the gesture detection device, and the target second object is controlled based on the hand gesture and the visual indication object, including:

[0155] Determine a third object according to the visual indication object, and display a virtual target shape model at the position of the third object;

[0156] In response to the fifth type of gesture operation, the target shape model is positioned at the current position of the target shape model.

[0157] According to one or more embodiments of the present disclosure, a method for controlling an electronic device is provided, wherein a hand gesture of the user is detected via the gesture detection device, and the target second object is controlled based on the hand gesture and the visual indication object, and further comprising one or more of the following:

[0158] determining a target edge in the target shape model according to the visual indication object, and adjusting a size of the target edge in response to a sixth type of gesture operation performed by the user;

[0159] In response to a seventh type of gesture operation performed by the user when the visual indication object is located on the target shape model, rotating the target shape model;

[0160] In response to an eighth type of gesture operation performed by the user when the visual indication object is located on the target shape model, moving the target shape model.

[0161] According to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: a display generation component and one or more input devices, wherein the input devices include an eye tracking device and a gesture detection device;

[0162] The display generation component is used to display the physical environment where the user is located;

[0163] The display generation component is further configured to generate at least one second object on the first object in the physical environment in response to a preset processing flow;

[0164] The eye tracking device is used to detect the gaze direction of the user;

[0165] The display generation component is further used for: detecting, via the eye tracking device, that the user's gaze is directed toward a target second object among the second objects, and generating a visual indication object;

[0166] The gesture detection device is used to detect the hand gesture of the user;

[0167] The display generation component is also used to: control the target second object based on the hand gesture of the user detected by the gesture detection device and the visual indication object;

[0168] The display generation component is further used to: determine the attribute information of the target second object when the fit between the target second object and the first object is adjusted based on the hand gesture to meet the user's needs.

[0169] According to one or more embodiments of the present disclosure, there is provided an electronic device, including: at least one memory and at least one processor;

[0170] The at least one memory is used to store program codes, and the at least one processor is used to call the program codes stored in the at least one memory to execute any one of the above methods.

[0171] According to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store program code, and when the program code is executed by a processor, the processor is prompted to execute the above method.

[0172] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0173] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0174] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A control method for an electronic device, characterized in that: The electronic device comprises: a display generation component and one or more input devices, wherein the input devices comprise an eye tracking device and a gesture detection device; the method comprises: Displaying the user's physical environment via a display generation component; In response to a preset processing flow, generating at least one second object on the first object in the physical environment; detecting, via the eye tracking device, that the user's gaze is directed toward a target second object among the second objects, and generating a visual indication object; detecting a hand gesture of the user via the gesture detection device, and controlling the target second object based on the hand gesture and the visual indication object; When the fit between the target second object and the first object is adjusted based on the hand gesture to meet the user's needs, the attribute information of the target second object is determined.

2. The method according to claim 1, characterized in that Detecting, via the eye tracking device, that the user's gaze is directed to a target second object among the second objects, comprising: The second object has a corresponding score, and when the user's sight passes over or stays on the second object, the score of the second object passed over or stayed by the sight is changed, or the scores of other second objects except the second object passed over or stayed by the sight are changed; the target second object is determined according to the score of the second object; or, In response to receiving a first voice message indicating a selection intention, a second object that the user is looking at when the first voice message is received is used as the target second object.

3. The method according to claim 1, characterized in that The second object is a virtual object; and / or, The attribute information of the target second object includes one or more of the following: the size of the target second object, the shape of the target second object, the position of the target second object, the posture of the target second object, and the semantics of the target second object; and / or, In response to the first type of gesture operation, it is determined that the fit between the target second object and the first object meets user requirements.

4. The method according to claim 1, characterized in that: Detecting the user's hand gesture via the gesture detection device, and controlling the target second object based on the hand gesture and the visual indication object includes: The hand gesture of the user is detected via the gesture detection device, and the attribute information of the target second object is adjusted in response to the adjustment instruction corresponding to the hand gesture in combination with the visual indication object.

5. The method according to claim 1, characterized in that When the first object is the ground, the second object is a virtual plane indicating object corresponding to the ground; The user's hand gesture is detected via the gesture detection device, and the target second object is controlled based on the hand gesture and the visual indication object, including: in response to the visual indication object being located on the plane indication object and the hand gesture being a second type of gesture operation, lifting or lowering the plane indication object.

6. The method according to claim 1, characterized in that When the first object is a wall, the second object is a corner line model of each corner line of the wall; Detecting, via the eye tracking device, that the user's gaze is directed to a target second object in the second objects, comprising: in response to the user's sight looking at a target wall corner line model, determining that the target wall corner line model is the target second object; Detecting the user's hand gesture via the gesture detection device, and controlling the target second object based on the hand gesture and the visual indication object, including: in response to the visual indication object being located at the target corner line model and the hand gesture being a third type of gesture operation, calibrating the target corner line model; When the fit between the target second object and the first object is adjusted based on the hand gesture to meet user needs, determining the attribute information of the target second object includes: after the calibrated corner line model forms a virtual closed surface, determining the attribute information of the calibrated corner line model and using the virtual closed surface as a virtual wall model for calibrating the wall.

7. The method according to claim 1, characterized in that When the first object is furniture, the second object is a virtual shape model corresponding to the furniture; Detecting, via the eye tracking device, that the user's gaze is directed to a target second object in the second object, comprising: determining, according to the user's line of sight, a target shape model that the line of sight is looking at as the target second object; When the fit between the target second object and the first object is adjusted based on the hand gesture to meet user needs, the attribute information of the target second object is determined, including: in response to a fourth type of gesture operation, determining the attribute information of the target shape model.

8. The method according to claim 7, characterized in that Detecting the user's hand gesture via the gesture detection device, and controlling the target second object based on the hand gesture and the visual indication object, comprising: Determine a third object according to the visual indication object, and display a virtual target shape model at the position of the third object; In response to the fifth type of gesture operation, the target shape model is positioned at the current position of the target shape model.

9. The method according to claim 8, characterized in that Detecting the user's hand gesture via the gesture detection device, and controlling the target second object based on the hand gesture and the visual indication object, further includes one or more of the following: determining a target edge in the target shape model according to the visual indication object, and adjusting a size of the target edge in response to a sixth type of gesture operation performed by the user; In response to a seventh type of gesture operation performed by the user when the visual indication object is located on the target shape model, rotating the target shape model; In response to an eighth type of gesture operation performed by the user when the visual indication object is located on the target shape model, moving the target shape model.

10. An electronic device, characterized in that: include: a display generation component and one or more input devices, the input devices including an eye tracking device and a gesture detection device; The display generation component is used to display the physical environment where the user is located; The display generation component is further configured to generate at least one second object on the first object in the physical environment in response to a preset processing flow; The eye tracking device is used to detect the gaze direction of the user; The display generation component is further used for: detecting, via the eye tracking device, that the user's gaze is directed toward a target second object among the second objects, and generating a visual indication object; The gesture detection device is used to detect the hand gesture of the user; The display generation component is also used to: control the target second object based on the hand gesture of the user detected by the gesture detection device and the visual indication object; The display generation component is further used to: determine the attribute information of the target second object when the fit between the target second object and the first object is adjusted based on the hand gesture to meet the user's needs.

11. An electronic device, comprising: at least one memory and at least one processor; The at least one memory is used to store program code, and the at least one processor is used to call the program code stored in the at least one memory to execute the method according to any one of claims 1 to 9. 12 . A computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, wherein when the program code is executed by a processor, the processor is prompted to execute the method according to claim 1 .

Images