Method, apparatus, device, and storage medium for augmented reality interaction

Abstract

A method for interacting with a virtual object in an augmented reality (AR) scene is provided, wherein an indication of a target object and one or more components of the target object are presented in an AR scene presented on a user device. In response to a change in the user device's orientation, rendering of the one or more components in the AR scene is updated. When the updated one or more components visually match the indication, the target object as a whole is presented in the AR scene. According to this method, a user can flexibly control one or more components (406-410, 506, 508, 510, 512, 514) in an AR scene (150, 400, 500) by changing the orientation of the device, so that the components visually match and finally assemble into a target object, thereby improving the user's interaction experience and interest in the AR scene (150, 400, 500).

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims priority to a Chinese invention patent application filed on September 20, 2022, bearing application number CN202211141379.9 and entitled "Method, apparatus, device, and storage medium for augmented reality interaction," the entire contents of which are incorporated herein by reference.

[0002] FIELD OF THE INVENTION Exemplary embodiments of the present invention relate generally to augmented reality (AR), and more particularly to methods, apparatus, devices, and computer-readable storage media for interacting in an AR scene. [Background technology]

[0003] Augmented reality (AR) technology integrates virtual information with the real world. AR devices can present virtual objects in an AR scene by overlaying them on a screen in the real world. In this way, the image displayed in the user's field of view includes both the real-world screen and the virtual objects. Using an AR scene, users can simultaneously view virtual objects and the real world, creating an immersive experience. AR has a wide range of applications in various fields, including games, simulations, and emulation. Various interaction techniques allow users to interact with virtual objects or objects in an AR scene. The method, efficiency, and effectiveness of these interaction techniques directly affect the user's experience using an AR device. Summary of the Invention

[0004] According to a first aspect of the present invention, there is provided a method for augmented reality (AR) interaction, the method comprising: ,versus Elephant instruction information and Againstpresenting one or more parts of the elephant; updating the rendering of the one or more parts in the AR scene based on a change in the user device's pose; and visually matching the updated one or more parts with the instruction information in response to the updated one or more parts being presented in the AR scene as a whole. Pair of and presenting the elephant.

[0005] According to a second aspect of the present invention, there is provided an apparatus for augmented reality (AR) interaction, the apparatus comprising: ,versus Elephant instruction information and Against a presentation control module for presenting one or more parts of the elephant; a rendering update module for updating the rendering of the one or more parts in the AR scene based on a change in the posture of the user device; and a rendering update module for updating the rendering of the one or more parts in the AR scene as a whole in response to the updated one or more parts visually matching the instruction information. Pair of and an object interaction module for presenting the elephant.

[0006] In a third aspect of the present invention, there is provided an electronic device comprising at least one processing unit and at least one memory coupled to the at least one processing unit and adapted to store instructions executed by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform a method according to the first aspect of the present invention.

[0007] In a fourth aspect of the present invention, there is provided a computer-readable storage medium having stored thereon a computer program that, when executed by a processor, causes a method according to the first aspect of the present invention to be performed.

[0008] It should be understood that the contents described in the summary of the present invention are not intended to limit the main or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will be readily apparent from the following description. [Brief explanation of the drawings]

[0009] The above-mentioned and other features, advantages, and aspects of each embodiment of the present invention will become more apparent from the following detailed description taken in conjunction with the drawings, in which like or similar reference numerals designate like or similar elements, and in which: [Figure 1] 1 shows a schematic diagram of an exemplary environment in which embodiments of the present invention may be implemented; [Figure 2] A schematic diagram illustrating the principle of "close is big, far is small" is shown. [Figure 3] 1 illustrates a flowchart of a method for AR interaction according to some embodiments of the present invention. [Figure 4A] 1A-1C show some exemplary schematic diagrams for presenting pointing information of an object in an AR scene according to some embodiments of the present invention; [Figure 4B] 1A-1C show some exemplary schematic diagrams for presenting pointing information of an object in an AR scene according to some embodiments of the present invention; [Figure 4C] 10A-10C show updated schematic diagrams of renderings of one or more components that change depending on the user device pose according to some embodiments of the present invention. [Figure 4D] 10A-10C show updated schematic diagrams of renderings of one or more components that change depending on the user device pose according to some embodiments of the present invention. [Figure 4E] 10A-10C show schematic diagrams of when one or more components visually match an indication of an object, according to some embodiments of the present invention. [Figure 4F] 1 shows a schematic diagram of the overall presentation and manipulation control of an object according to some embodiments of the present invention. [Figure 5A]10 shows another exemplary schematic diagram of manipulating an object component to conform to an object contour according to some embodiments of the present invention. [Figure 5B] 10 shows another exemplary schematic diagram of manipulating an object component to conform to an object contour according to some embodiments of the present invention. [Figure 5C] 10 shows another exemplary schematic diagram of manipulating an object component to conform to an object contour according to some embodiments of the present invention. [Figure 6] 1 illustrates a block diagram of an apparatus for augmented reality according to some embodiments of the present invention. [Figure 7] 1 shows a block diagram of a device capable of implementing several embodiments of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, the embodiments of the present invention will be described in more detail with reference to the drawings. Although the drawings show several embodiments of the present invention, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, but rather, these embodiments are provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes and are not used to limit the protection scope of the present invention.

[0011] In describing embodiments of the present invention, the term "comprising" and similar terms are intended to be open-ended inclusions, including, but not limited to, the term "based on" should be understood as "based at least in part on." The terms "one embodiment" or "the embodiment" should be understood as "at least one embodiment." The term "some embodiments" should be understood as "at least some embodiments." Other explicit and implicit definitions may be included below.

[0012] The term "in response to" refers to the occurrence of a corresponding event or the fulfillment of a condition. It will be understood that the timing of the execution of a subsequent action performed in response to the event or condition is not necessarily strongly correlated with the timing of the event occurring or the condition being fulfilled. In some cases, the subsequent action may be executed simultaneously with the event occurring or the condition being fulfilled, or in other cases, the subsequent action may be executed some time after the event occurring or the condition being fulfilled.

[0013] It is understood that data related to the present technical solution (including but not limited to the data itself, the acquisition of data, or the use of data) should comply with applicable laws and regulations and related specified requirements.

[0014] It is understood that before using the technical solutions disclosed in each embodiment of the present invention, the type, scope of use, and usage scenario of personal information related to the present invention should be notified to users in an appropriate manner in accordance with relevant laws and regulations, and consent from users should be obtained.

[0015] For example, in response to receiving an unsolicited request from a user, presentation information may be sent to the user to explicitly indicate to the user that the requested operation requires the acquisition and use of the user's personal information, allowing the user to independently choose whether or not to provide the personal information to software or hardware, such as an electronic device, application, server, or storage medium, that performs the operation of the technical solution of the present invention, based on the presentation information.

[0016] In an optional, non-limiting implementation, the method for sending the user the offered information in response to receiving the user's unsolicited request may be, for example, a method using a pop-up window in which the offered information can be displayed in text form, and the pop-up window may further include a selection control for the user to select "agree" or "disagree" to providing personal information to the electronic device.

[0017] It is understood that the notification and user authorization process described above is merely a general outline and is not intended to limit the implementation of the present invention, and that other methods that comply with relevant laws and regulations may also be applied to the implementation of the present invention.

[0018] An embodiment of the present invention will now be described in conjunction with the accompanying drawings. As will be understood from the following description, the embodiment of the present invention provides a solution for a user to interact with a virtual object in an AR scene. In the AR scene, a user is simultaneously presented with instruction information about a target virtual object and one or more components of the virtual object. Initially, these components are visually dispersed, i.e., not assembled into a target virtual object. The user can change the pose (e.g., orientation, tilt, distance, etc.) of the user device to change the field of view (FoV) of the AR virtual camera of the AR scene. The rendering effect of these components in the AR scene is updated accordingly. When the user device reaches a certain predetermined pose, the one or more components rendered in the AR scene visually coincide with the target virtual object. At this time, an "assembly" operation can be triggered to render the dispersed components into a whole target virtual object. The user can then interactively manipulate and control the whole target virtual object. This method provides a user with an interesting interaction process and improves the user experience of the AR environment.

[0019] 1 shows a schematic diagram of an exemplary environment 100 in which embodiments of the present invention can be implemented. In the exemplary environment 100, an AR scene 150 is presented to a user 130 at or by a user device 110. The AR scene 150 may be presented on a screen of the user device 110. The AR scene 150 may include a real-world screen 154 and virtual objects 1531 and 1532 superimposed on the screen 154.

[0020] On screen 154, objects 1541 and 1542 are representations of real objects in the real world in AR scene 150, e.g., images or other forms of representation of the real objects. For ease of discussion only, objects 1541 and 1542 are also referred to herein as 3D objects. As the orientation of user device 110 changes, the field of view of the AR camera used for the AR scene changes accordingly. Accordingly, screen 154 is changed based on the change in orientation of user device 110. Accordingly, the rendering of the 3D objects presented on screen 154 is updated, and the visual effects of these 3D objects are changed accordingly. Similarly, the rendering of virtual objects 1531 and 1532 is updated, and the visual effects of these virtual objects are changed accordingly.

[0021] In some implementations, the user device 110 may include a positioning component that acquires the pose of the user device 110 and determines the positions of objects and the virtual camera in the AR scene in real time based on the pose, thereby enabling rendering updates in response to changes in the pose of the user device 110.

[0022] It should be understood that AR scene 150 is illustrative only and is not intended to limit the scope of the present invention. AR scene 150 may include more or fewer virtual objects superimposed on screen 154, and may include other elements, such as user interface (UI) elements.

[0023] User device 110 may be any type of mobile, fixed, or handheld terminal, including a mobile phone, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, media computer, multimedia tablet, gaming device, wearable device, personal communication system (PCS) device, personal navigation device, assistant (PDA), audio / video player, digital camera / camcorder, positioning device, television receiver, radio receiver, e-book device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. In some embodiments, user device 110 may also support any type of interface to a user (such as "wearable" circuitry).

[0024] The user device 110 may include an AR engine 120. The AR engine 120 is used to drive the presentation of the AR scene 150. In some embodiments, the AR engine 120 may be an AR game engine, and accordingly, the AR scene 150 may be an AR game scene. In some embodiments, the AR engine 120 may be part of a content sharing application (also referred to as a "social application"). The social application may provide services related to multimedia content consumption to the user 130, such as allowing the user to publish, view, comment, forward, and create multimedia works. Accordingly, the AR scene 150 may be an AR content creation scene. For example, in certain embodiments, the AR scene 150 may be part of a special effect provided by the social application. The positioning component described above may be part of the AR engine 120.

[0025] It should be understood that the structure and functionality of environment 100 is described for illustrative purposes only and is not intended to limit the scope of the invention in any way. User device 110 may include any suitable structure and functionality implemented for interaction with an AR scene.

[0026] According to the principle of perspective, it is understandable that the visual effect of 3D objects presented in an AR scene will change as the user's or virtual camera's field of view changes. Taking the example of the human eye, the lens in the eye corresponds to a convex lens, and the retina corresponds to the image plane. The human eye sees objects by forming an image of the object on the retina. The angle of expansion from the center of the pupil to the object is equal to the visual angle. Therefore, the size of the visual angle determines the size of the object's image on the retina. When an object is farther from the eye, the visual angle is smaller, so the perceived size is smaller. When the same object is closer to the eye, the visual angle is larger, and the perceived visual effect is greater. The size of the object does not change, but the visual effect changes due to the different visual angles. This is the principle of "closer is bigger, farther is smaller." Cameras also capture and form images according to this principle.

[0027] Using this principle, it is possible to realize visual effects that do not occur in the real world. For example, Figure 2 shows a scene 200 captured by a camera. In scene 200, the image of the hand 202 appears larger because the hand is closer to the viewpoint, and the image of the sun 204 appears smaller because the sun is farther from the viewpoint. This allows the effect of "picking the sun with your hands" to be realized.

[0028] Similarly, different viewpoints and / or viewing angles can result in numerous variations in the visual effect of the same object being rendered in an AR scene. According to embodiments of the present invention, these visual and imaging principles are fully utilized in supporting user interaction with 3D objects in an AR scene to provide a set of interaction mechanisms that enable a good user experience.

[0029] 3 illustrates a flowchart of a method 200 for AR interaction according to some embodiments of the present invention. In some embodiments, the method 200 may be implemented in, for example, the user device 110 shown in FIG. 1, for example, by the AR engine 120 or other suitable module / apparatus.

[0030] In box 302, in the AR scene presented on the user device 110, ,versus Elephant instruction information and Against Presents one or more parts of an elephant.

[0031] FIG. 4A shows an AR scene 400 presented on the user device 110. Against 1 shows a schematic diagram of an AR scene 400 presenting an elephant's pointing information and its component parts. As shown in the figure, the AR scene 400 includes a real-world portion 402 captured by the camera of the user device 110 and a virtual object. In this example, the real-world portion 402 is a bookshelf, which is used as the background portion of the scene.

[0032] The virtual object is ,versus In the example shown in FIG. ,versus The elephant (or "target virtual object") is a snowman. AR scene 400 contains ,versus Shown are multiple parts of the elephant 406, 408, and 410. More specifically, part 406 is the snowman's hat, part 408 is the snowman's head, and part 410 is the snowman's body. In certain embodiments, the most dominant or largest part may be the snowman's head, body, or body. ,versus Note that only one component of the elephant can be presented.

[0033] AR Scene 400 ,versus In the example shown in FIG. 4A, the instruction information for the elephant snowman is ,versus1 shows an outline 404 of an elephant "snowman." The outline 404 is presented as an outline. In certain embodiments, the outline 404 may be presented in the AR scene 400 at a predetermined size and / or position. That is, even if the user changes the orientation of the user device 110, thereby triggering a change in the field of view of the AR scene 400, the position and size of the outline 404 on the user interface of the user device 110 may remain unchanged. Alternatively, in some other embodiments, the outline 404 may also change position and / or size as the scene changes. Simply viewing the disassembled components may not provide a clear idea of ​​the structure the user should assemble. Countermeasures It is difficult to identify the overall shape of the elephant. Against By displaying the outline of the elephant, the goal of the patchwork of pieces can be clearly understood, which further improves the user experience.

[0034] The outline is ,versus Note that the elephant indicia is only one example. Other suitable forms of indicia are possible. For example, in the example shown in FIG. ,versus The elephant instruction information has been changed from an outline to an icon 412, but the other presented content is consistent with the example of Figure 4A. The appearance of icon 412 is that of a snowman, which currently Pair of Show the user that the elephant is a snowman. Use text, animation, etc. in addition to or instead of the icon. Against The content of the present invention is not limited in this respect.

[0035] Returning to FIG. 3, in box 304, based on the change in the posture of the user device 110, ,versusThe rendering of one or more components in the AR scene of the elephant is updated. Specifically, in operation, the user can change various pose parameters of the user device 110, such as the orientation, angle, and position. The change in pose of the user device 110 results in a change in the field of view of the AR camera used for the AR scene. The visual effect of the 3D objects in the scene changes accordingly in the rendering results presented to the user via the user interface. For example, the positioning component can obtain the pose of the user device 110 in real time, thereby changing the position of the AR camera and the coordinates of one or more components. The rendering of the one or more components also changes accordingly.

[0036] 4C and 4D each show a schematic diagram of updating renderings of one or more components depending on the pose of the user device 110, according to some embodiments of the present invention. As can be seen in the schematic diagram shown in FIG. 4C, when the user changes the orientation of the user device 110 in the direction indicated by the arrow 414, the rendering of the 3D objects in the AR scene 400 is updated accordingly. As the field of view of the camera of the user device 110 changes, the scene included in the background portion (i.e., the image of the real-world scene) changes compared to FIGS. 4A-4B.

[0037] at the same time ,versus The rendering of the elephant's components 406-408 also updates as the AR camera field of view changes. Specifically, component 406 becomes visually smaller and component 410 becomes visually larger, following the "closer is bigger, farther is smaller" principle. Furthermore, as the viewing angle of the AR camera changes, the visible portion of each component may change accordingly. That is, previously invisible portions may become visible, and previously visible portions may become invisible.

[0038] Furthermore, based on visual and imaging principles, a change in the field of view of an AR camera can change the visual relative positions of multiple objects whose actual relative positions remain unchanged. According to these principles, a change in the orientation of the user device 110 can also change the visual relative positions of each component in the AR scene due to changes in the perspective relationship. Specifically, in the illustrated example, the distances between components 406-410 appear to change compared to the state shown in FIGS. 4A and 4B. Note that this is merely a visual change brought about by the rendering update; their actual relative positions in the image remain unchanged.

[0039] FIG. 4D shows the result of the user further changing the orientation of the user device 110 along direction 416. ,versus 4D shows an update to the rendering of the image components 406-410. Specifically, in the example shown in FIG. 4D, the user may notice that the updated rendered components 406-410 appear to be on the right side in the AR scene 400 compared to FIG. 4C, and therefore Against The user device 110 is panned to the left so that it is closer to the elephant's indicia (specifically, the outline 404 in this example).

[0040] In summary, in operation, the field of view of the AR camera is determined based on the orientation of the user device 110, and the AR scene 400 is displayed based on the field of view. Against 4C and 4D , outline 404 does not further change its appearance in response to changes in the orientation of user device 110. That is, outline 404 maintains a fixed position and size in the user interface. Alternatively, in other embodiments, outline 404 may appropriately change its position and size in response to changes in the orientation of user device 110, although the degree of change may differ from that of portions 406-410.

[0041] When the user changes the device position, AgainstAdjusting the presentation of one or more parts 406-410 of the elephant may result in the updated part or parts 406-410 appearing visually different in a particular state. Against It will be understood that the information on the elephant matches each other. For example, in the state shown in FIG. 4E, the overall outline of the visually constructed components 406-410 is ,versus It matches the elephant's outline 404, i.e., they match each other in shape and size within a certain tolerance.

[0042] Various suitable algorithms can detect such visual matches. For example, in some implementations, the AR engine 120 ,versus Associated with elephants Second The camera positions can be pre-calculated and stored. The camera positions of the AR cameras associated with the AR scene are Second When close to or aligned with the camera position ,versus The elephant's components are considered to fit precisely within the area defined by the visual contour. In operation, as the user device 110 changes its pose, the AR engine 120 adjusts the AR camera accordingly. 1st Determine the camera position 1st Camera position Second Compare with the camera position.

[0043] The camera position can be indicated in various suitable ways, such as camera orientation, height, distance, and direction. 1st Camera position and the desired Second If the difference between the camera position and the contour is less than a predetermined threshold, it can be determined that one or more components visually match the contour. 1st Camera position and Second If the position and rotation angle direction of the camera position are within a certain error range, the two camera positions are considered to be close, and therefore it can be determined that one or more components visually match the contour.

[0044] It will be appreciated that in other embodiments where the indication is not a contour, the above camera position matching method is equally applicable. Second As the user manipulates the user device 110 to reach the camera position, the overall visual effect of these components is believed to achieve a match with the displayed icon or text.

[0045] However, such a predetermined Second It should be understood that camera position matching is not required and is merely exemplary. For example, in some alternative embodiments, after updating the rendering of the components, the maximum outer bounding box of those components is calculated and the bounding box and A visual match can be detected by matching contours as elephant indicators. Other matching methods are possible, and the invention is not limited in this respect.

[0046] FIG. 4E is a visual representation of components 406-410 according to some embodiments of the present invention. Against 4 shows a schematic diagram of the elephant's pointing information and how they match with each other. In this example, it can be seen that the components 406-410 as a whole fit precisely within the visual contour 404.

[0047] Returning to FIG. 3, in response to the above-described match, in box 306, Pair of For example, referring to FIG. 4F, in AR scene 400, a complete, overall Pair of An elephant 418, or snowman, is rendered. At this point, the snowman's components are no longer separate, independent entities. Instead, they are presented and rendered as a single subject or virtual object in the AR scene 400.

[0048] For example, as the user repositions the user device 110, the representation of the assembled snowman is updated accordingly, but the "exploded view" effect of FIGS. 4C and 4D, in which the components are separated from one another, is no longer presented. ,versus Visual representation of the elephant's components Against After the elephant's pointing information and the elephant's pointing information are matched, the AR engine 120 no longer updates the rendering of the components 406-410 according to the change in the posture of the user device 110, but instead renders them as a whole. Pair of It just renders and updates the elephant 418.

[0049] In some implementations, if a positioning component is used, the positioning component can be disabled or turned off in response to the above-described match. This means that user changes to the user device 110's pose (e.g., user shaking) do not affect the position of the AR camera. Rather, a controlled calibration of the camera position can be used to adjust the camera position of the AR camera. Second Align with the camera position. ,versus The elephant is rendered as a single whole.

[0050] In such an embodiment, the camera position of the AR camera is Second When approaching the camera position ,versus The elephant can be presented as a single whole. This method can avoid the user having to spend a long time to achieve the target assembly goal due to slight shaking of the user device (e.g., shaking of the user's hand), thereby improving the user experience.

[0051] Alternatively or additionally, in some embodiments, one or more components may be Against Once "assembled" into the elephant, the user ,versusThe elephant 418 can be manipulated as a whole, such as by rotating, scaling, dragging, etc. For example, after completing the administrative calibration described above, the AR component can be activated, returning manipulation control to the user. The user can then manipulate the elephant 418 as a whole. Pair of You can interact with the elephants.

[0052] In certain embodiments, the updated component is once visually Against If the elephant's pointing information matches, the AR engine 120 further calculates the contour information in the AR scene 400. Pair of The drawing of the elephant's indicia can be stopped. For example, in Figure 4F, outline 404 is no longer displayed.

[0053] Several embodiments of the present invention have been described above with reference to several example scenes. According to the embodiments of the present invention, a user can assemble one or more independent components of an AR scene into a single complete scene by adjusting the orientation of their user device. Pair of The entire interaction process is challenging and interesting, and can be widely applied to various fields, such as AR special effects for social applications, cognitive behavioral training and support, simulations, and games, to improve the user's interaction experience.

[0054] 5A, 5B, and 5C illustrate operations according to some embodiments of the present invention. Works Manipulating the elephant's components Against 1 shows another exemplary schematic diagram of matching elephant contours. In the illustrated example, the AR scene 500 also has a background portion 502, which is an image of a real-world scene captured by the camera of the user device 110, in this example the background portion.

[0055] As shown in FIG. 5A, in addition to the background portion 502, first, in the AR scene 500, Against the outline of the elephant "hamburger" 504, and , the pairThe user was presented with multiple elephant parts 506, 508, 510, 512, 514. More specifically, in this example, these parts are the bread slices, patty, cheese slices, and vegetables that make up a hamburger.

[0056] 5B, the user adjusts the orientation of the user device 110 so that the components 506, 508, 510, 512, and 514 visually fit exactly within the contour 504, and the difference between their boundaries is within a predetermined tolerance. As mentioned above, in some implementations, this can occur, for example, when the camera position of the AR camera is adjusted to a predetermined Second This may be detected by detecting whether the camera position has been reached.

[0057] As shown in Figure 5C, once it is determined that components 506, 508, 510, 512, 514 visually match contour 504, the positioning component is disabled to transfer the user's operational control authority. ,versus The elephant "hamburger" 516 is then subject to rendering updates and / or responsive user manipulation as a whole, and each of the constituent parts 506, 508, 510, 512, 514 can no longer be independently manipulated and controlled, and at the same time, the outline 504 ceases to be displayed.

[0058] 6 shows a block diagram of an apparatus 600 for AR interaction. As shown in the figure, the apparatus 600 is configured to ,versus Elephant direction information and front Counterpart The apparatus 600 further comprises a rendering update module 620 for updating a rendering of the one or more components in the AR scene based on a change in the user device's posture. The apparatus 600 further comprises a rendering update module 620 for updating a rendering of the one or more components in the AR scene in response to the updated one or more components visually matching the instruction information. Counterpart It further comprises an object interaction module 630 for presenting the elephant.

[0059] In some embodiments, the presentation control module 610 may present a front view of the AR scene at a predetermined size and / or position. Counterpart It presents the outline of an elephant.

[0060] In some embodiments, the presentation control module 610 further stops presenting the contour in the AR scene in response to the updated one or more components visually matching the instruction information.

[0061] In some embodiments, the target interaction module 630 further includes: Counterpart The elephant is presented as a whole, then Counterpart and, in response to receiving an input for the image, performing a priori processing based on the input. Counterpart Operate and control the elephant as a whole.

[0062] In some embodiments, the apparatus 600 determines the orientation of the AR camera used in the AR scene based on the orientation of the user device. 1st determining a camera position; 1st Camera position forward Counterpart A given elephant is associated with Second comparing with the camera position and 1st Camera position and Second and a visual matching module (not shown) for determining whether the one or more components visually match the instruction information by determining that the one or more components visually match the instruction information in response to a difference from the camera position being less than a predetermined threshold.

[0063] In some embodiments, the rendering update module 620 determines a field of view of an AR camera used for the AR scene based on the pose of the user device, and renders the one or more components in the AR scene based on the field of view of the AR camera.

[0064] In some embodiments, the rendering update module 620 further stops updating the rendering of the one or more components with changes in the user device's posture in response to the updated one or more components visually matching the instruction information.

[0065] The units included in apparatus 600 can be implemented in various forms, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to, or as an alternative to, machine-executable instructions, some or all units of apparatus 600 may be implemented, at least in part, by one or more hardware logic components. By way of example and not limitation, typical types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.

[0066] 7 illustrates a block diagram of a computing device 700 capable of implementing one or more embodiments of the present invention. It should be understood that the computing device 700 illustrated in FIG. 7 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The computing device 700 illustrated in FIG. 7 may be used to implement the user device 110 of FIG. 1.

[0067] 7, computing device 700 is a form of general-purpose computing device. Components of computing device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a real or virtual processor and may perform various processes based on programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel, thereby increasing the parallel processing capabilities of computing device 700.

[0068] Computing device 700 typically includes a number of computer storage media. Such media may be any obtainable media accessible by computing device 700, including, but not limited to, volatile and nonvolatile media, removable and non-removable media. Memory 720 may be volatile memory (e.g., registers, high-speed cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 may be removable or non-removable media and may include machine-readable media, such as a flash memory drive, a magnetic disk, or any other media, that may be used to store information and / or data (e.g., training data for training) and that may be accessible within computing device 700.

[0069] The computing device 700 may further include other removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 7, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a path (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules configured to perform various methods or operations of various embodiments of the present invention.

[0070] The communications unit 740 implements communications with other computing devices over a communications medium. Additionally, the functionality of the components of computing device 700 may be implemented as a single computing cluster or multiple computing machines, which can communicate over a communications connection. Thus, computing device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.

[0071] The input device(s) 750 may be one or more input devices such as a mouse, keyboard, trackball, etc. The output device(s) 760 may be one or more output devices such as a display, speakers, printer, etc. The computing device 700 may further communicate with one or more external devices (not shown) such as a storage device, a display device, etc. via the communications unit 740, as needed, to communicate with one or more devices that enable a user to interact with the computing device 700, or any device (e.g., a netbook card, a modem, etc.) that enables the computing device 700 to communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0072] According to an exemplary implementation of the present invention, a computer-readable storage medium having stored thereon one or more computer instructions is provided, the one or more computer instructions being executed by a processor to implement the above-described method.

[0073] Aspects of the present invention have been described herein with reference to flowchart and / or block diagrams of methods, apparatus (systems), and computer program products implemented by the present invention. It will be understood that each box in the flowchart and / or block diagrams, and combinations of boxes in the flowchart and / or block diagrams, can all be implemented by computer-readable program instructions.

[0074] These computer-readable program instructions may be provided to a processing unit of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to generate a machine such that, when the instructions are executed by the processing unit of the computer or other programmable data processing apparatus, it generates an apparatus for implementing the functions / acts specified in one or more boxes in the flowcharts and / or block diagrams. These computer-readable program instructions may be stored on a computer-readable storage medium such that the instructions cause the computer, programmable data processing apparatus, and / or other device to operate in a particular manner such that the computer-readable medium on which the instructions are stored constitutes an article of manufacture containing instructions that implement each aspect of the functions / acts specified in one or more boxes in the flowcharts and / or block diagrams.

[0075] Loading the computer-readable program instructions into a computer, other programmable data processing apparatus, or other device causes the computer, other programmable data processing apparatus, or other device to perform a series of operational steps to produce a computer-implemented process, such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more boxes in the flowcharts and / or block diagrams.

[0076] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of various possible systems, methods, and computer program products according to the present invention. In this regard, each box in the flowcharts or block diagrams may represent a module, program segment, or portion of instructions, which includes one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions depicted in the boxes may occur in a different order than depicted in the figures. For example, two consecutive boxes may actually be executed substantially in parallel, or may be executed in the reverse order, depending on the functionality involved. It should also be noted that each box in the block diagrams and / or flowcharts, and combinations of boxes in the block diagrams and / or flowcharts, may be implemented by a special-purpose hardware-based system that performs the specified function(s) or operation(s), or by a combination of special-purpose hardware and computer instructions.

[0077] Although various implementations of the present invention have been described above, the above descriptions are illustrative, not exhaustive, and are not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is intended to best interpret the principles, practical applications, or improvements to commercially available technologies of the implementations, or to enable those skilled in the art to understand the implementations disclosed herein.

Claims

1. presenting an indication of a target object and one or more components of the target object in an AR scene presented on a user device; updating a rendering of the one or more components of the AR scene based on changes in the pose of the user device; and and presenting the target object as a whole in the AR scene in response to the updated one or more components visually matching the instructional information. A method for augmented reality (AR) interaction.

2. and presenting the indication information includes presenting an outline of the target object at a predetermined size and / or position in the AR scene. The method of claim 1.

3. and ceasing presentation of the contour in the AR scene in response to the updated one or more components visually matching the indication. The method of claim 2.

4. and, in response to receiving an input for the target object after the target object is presented as a whole, manipulating and controlling the target object as a whole based on the input.

3. The method according to claim 1 or 2.

5. determining a current camera position of an AR camera used in the AR scene based on the pose of the user device; comparing the current camera position with a predetermined target camera position associated with the target object; determining whether the one or more components visually match the indication information in response to a difference between the current camera position and the target camera position being less than a predetermined threshold; 3. The method according to claim 1 or 2.

6. Updating the rendering of the one or more components in the AR scene comprises: determining a field of view of an AR camera to be used for the AR scene based on a pose of the user device; and rendering the one or more components in the AR scene based on the field of view of the AR camera.

3. The method according to claim 1 or 2.

7. and, in response to the updated one or more components visually matching the indication, ceasing to update the rendering of the one or more components with changes in the user device's attitude.

3. The method according to claim 1 or 2.

8. a presentation control module arranged to present an indication of a target object and one or more components of the target object in an AR scene presented on the user device; a rendering update module configured to update a rendering of the one or more components in the AR scene based on a change in a posture of the user device; an object interaction module configured to present the target object as a whole in the AR scene in response to the updated one or more components visually matching the instructional information; A device for augmented reality (AR) interaction.

9. the presentation control module is configured to present an outline of the target object at a predetermined size and / or position in the AR scene.

9. The apparatus of claim 8.

10. the presentation control module is further configured to stop presenting the contour in the AR scene in response to the updated one or more components visually matching the indication information.

10. The apparatus of claim 9.

11. the object interaction module is further configured to, in response to receiving an input for the target object after the target object is presented as a whole, manipulate and control the target object as a whole based on the input.

10. The device according to claim 8 or 9.

12. determining a current camera position of an AR camera used for the AR scene based on the pose of the user device; comparing the current camera position with a predetermined target camera position associated with the target object; and determining that the one or more components visually match the indication in response to a difference between the current camera position and the target camera position being less than a predetermined threshold.

10. The device according to claim 8 or 9.

13. The rendering update module: determining a field of view of an AR camera to be used for the AR scene based on the pose of the user device; positioned to render the one or more components in the AR scene based on the field of view of the AR camera.

10. The device according to claim 8 or 9.

14. the rendering update module is further configured to stop updating the rendering of the one or more components with a change in attitude of the user device in response to the updated one or more components visually matching the indication.

10. The device according to claim 8 or 9.

15. An electronic device, at least one processing unit; and at least one memory coupled to said at least one processing unit and adapted to store instructions to be executed by said at least one processing unit, said instructions, when executed by said at least one processing unit, causing said electronic device to perform the method of any one of claims 1 to 7. electronic equipment.

16. A computer program is stored in the memory, and when the program is executed by a processor, the method according to any one of claims 1 to 7 is realized. A computer-readable storage medium.

Images