Augmented reality interaction method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides an augmented reality interaction method and device, electronic equipment and storage medium, the method comprising: acquiring a real scene image; based on the real scene image and a pre-constructed virtual scene, an AR picture is displayed in an augmented reality AR device, the AR picture contains a target object; a plurality of virtual objects to be eliminated are suspended in the virtual scene, and the virtual objects move towards a preset direction; in response to the movement of the AR device, in the case that a preset relationship is met between the target object and any virtual object of the plurality of virtual objects, the virtual object meeting the preset relationship is removed from the AR picture. The embodiment of the present disclosure combines augmented reality with a virtual scene, and through the movement of the AR device, the virtual object can be removed in the AR picture, so that the user's interaction experience can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of Augmented Reality (AR) technology, and in particular, to an augmented reality interaction method, an augmented reality interaction device, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] With the emergence of various types of electronic games, more and more users experience games through mobile devices to enrich leisure and entertainment activities.

[0003] However, the traditional human-computer interaction mode of electronic games is usually that the user triggers on the screen of the mobile device to control the game object, which leads to a single interactive experience in the game. SUMMARY

[0004] The embodiments of the present disclosure at least provide an augmented reality interaction method, an augmented reality interaction device, an electronic device, and a computer readable storage medium, which are beneficial to improve the interactive experience of users.

[0005] The embodiments of the present disclosure provide an augmented reality interaction method, comprising:

[0006] obtaining a real scene image;

[0007] displaying an AR picture in an augmented reality (AR) device based on the real scene image and a pre-constructed virtual scene, wherein the AR picture contains a target object, and a plurality of virtual objects to be eliminated are suspended in the virtual scene, and the virtual objects move towards a preset direction;

[0008] in response to movement of the AR device, removing a virtual object that meets a preset relationship from the AR picture in a case where the target object and any virtual object of the plurality of virtual objects meet the preset relationship.

[0009] In the embodiments of the present disclosure, the real scene image is combined with the virtual scene, and the AR picture containing the target object and the plurality of virtual objects is displayed in the AR device. The user can change the position of the target object by moving the AR device, and in a case where the target object and any virtual object of the plurality of virtual objects meet the preset relationship, the virtual object in the AR picture will be eliminated. That is, the atmosphere of combining virtual and real is added to the elimination process, which can improve the interest of the elimination process. The position of the target object is changed by moving the AR device, which has the atmosphere of somatosensory interaction, so that the user can experience the game in situ, thereby improving the interactive experience of users.

[0010] In an optional implementation, the preset direction is a direction towards the AR device.

[0011] In the embodiments of the present disclosure, the virtual object moves towards the direction of the AR device, which can present a real effect that the virtual object "flies" to the user, and is beneficial to improve the game experience of the user.

[0012] In an optional implementation, the removing the virtual object satisfying the preset relationship from the AR picture comprises:

[0013] In response to the movement of the AR device, it is determined whether the shape of the target object matches the shape of a target virtual object in the plurality of virtual objects, wherein the target virtual object is a virtual object closest to the AR device;

[0014] In a case where the shape of the target object matches the shape of the target virtual object and the target object is in a moving direction of the target virtual object, the target virtual object is removed from the AR picture.

[0015] In the embodiments of the present disclosure, in a case where the shape of the target object matches the shape of the target virtual object and the target object is in the moving direction of the target virtual object, the target virtual object is removed from the AR picture, that is, in the process that the target virtual object "flies" to the user, the user needs to "catch" the target virtual object with the target object of the same shape, so that the target virtual object is eliminated, and the interactivity and the sense of immersion of the interactive experience of the user are improved.

[0016] In an optional implementation, the method further comprises:

[0017] In a case where the shape of the target object does not match the shape of the target virtual object, in response to a shape adjustment operation on the target object, the shape of the target object is adjusted to be the same as the shape of the target virtual object.

[0018] In a case where the target object after the shape adjustment is in the moving direction of the target virtual object, the target virtual object is removed from the AR picture.

[0019] In the embodiments of the present disclosure, if the shape of the target object does not match the shape of the target virtual object, the shape of the target object can be adjusted to be the same as the shape of the target virtual object, that is, the user can change the shape of the target object in real time, and the target virtual object is eliminated, so that the interactivity and the participation of the interaction are increased, and the game interest of the user is improved.

[0020] In an optional implementation, the virtual scene further comprises a three-dimensional virtual map matching the target real scene, and positions of the plurality of virtual objects are fixed relative to the three-dimensional virtual map; and the AR picture is displayed in the AR device based on the real scene image and the pre-constructed virtual scene, comprising:

[0021] determining whether a current real scene where the AR device is located is the target real scene based on the real scene image and the three-dimensional virtual map;

[0022] in a case where the current real scene is the target real scene, displaying the AR picture in the AR device based on the real scene image and position information of the plurality of virtual objects relative to the three-dimensional virtual map.

[0023] In the embodiments of the present disclosure, in a case where the current real scene is the target real scene, the AR picture is displayed based on the real scene image and the position information of the plurality of virtual objects relative to the three-dimensional virtual map, that is, the virtual scene and the real scene are fixedly corresponding, and the virtual scene can be loaded only when the user reaches the pre-set real scene, so that the pertinence of the interactive scene is improved and the loading efficiency of the game is improved.

[0024] In an optional implementation, the AR picture is displayed in the AR device based on the real scene image and the pre-constructed virtual scene, comprising:

[0025] obtaining pose information of the AR device in a current real scene where the AR device is located;

[0026] displaying the AR picture in the AR device based on the pose information, the real scene image and the virtual scene.

[0027] In the embodiments of the present disclosure, the AR picture can be displayed according to the pose information of the AR device in the current real scene where the AR device is located, the real scene image and the virtual scene, that is, the virtual scene and the real scene are not fixedly corresponding, and the virtual scene can be loaded in any real scene, so that the user can experience the game in any real scene, and the applicability of the game interaction is improved.

[0028] In an optional implementation, the AR picture is displayed in the AR device based on the pose information, the real scene image and the virtual scene, comprising:

[0029] determining a conversion relationship between the virtual coordinate system and a world coordinate system of the current real scene based on the pose information;

[0030] determine position information of the plurality of virtual objects in the world coordinate system based on the position information of the plurality of virtual objects in the virtual coordinate system and the conversion relationship;

[0031] display the AR picture in the AR device based on the position information of the plurality of virtual objects in the world coordinate system and the real scene image.

[0032] In the embodiments of the present disclosure, the position information of the plurality of virtual objects in the world coordinate system is determined based on the conversion relationship between the virtual coordinate system of the virtual scene and the world coordinate system of the current real scene, and then the distribution of the virtual objects in the current real scene is determined. In this way, the fusion of the virtual objects and the real scene can be realized, and the positions of the virtual objects relative to the real scene are relatively fixed.

[0033] In an optional implementation, after the virtual object satisfying the preset relationship is removed from the AR picture, the method further includes:

[0034] displaying a new target object in the AR picture, the new target object being used to eliminate the remaining virtual objects, and the shape of the new target object being randomly generated.

[0035] In the embodiments of the present disclosure, the new target object with a random shape is displayed in the AR picture to eliminate the remaining virtual objects, that is, the user can continue to move the AR device, so that the new target object continues to eliminate the remaining virtual objects. In this way, the integrity of the experience game can be improved. In addition, since the new target object is randomly generated, the challenge of the game can be improved while the interest of the game is ensured, thereby improving the enthusiasm of the user in participating in the game.

[0036] The embodiments of the present disclosure also provide an augmented reality interaction device, and the device includes:

[0037] an acquisition module configured to acquire a real scene image;

[0038] a display module configured to display an AR picture in an augmented reality (AR) device based on the real scene image and a pre-constructed virtual scene, the AR picture including a target object; the virtual scene having a plurality of virtual objects to be eliminated suspended therein, the virtual objects moving towards a preset direction.

[0039] a removal module configured to remove, in response to movement of the AR device, a virtual object satisfying a preset relationship from the AR picture in a case where the preset relationship is met between the target object and any virtual object of the plurality of virtual objects.

[0040] In an optional implementation, the preset direction is a direction towards the AR device.

[0041] In an optional implementation, the removing module is specifically configured to:

[0042] In response to the AR device moving, determine whether the shape of the target object matches the shape of a target virtual object in the plurality of virtual objects, wherein the target virtual object is a virtual object closest to the AR device in distance;

[0043] In a case where the shape of the target object matches the shape of the target virtual object and the target object is in a moving direction of the target virtual object, remove the target virtual object from the AR picture.

[0044] In an optional implementation, the removing module is specifically configured to:

[0045] In a case where the shape of the target object does not match the shape of the target virtual object, in response to a shape adjustment operation for the target object, adjust the shape of the target object to be the same as the shape of the target virtual object;

[0046] In a case where the target object after shape adjustment is in the moving direction of the target virtual object, remove the target virtual object from the AR picture.

[0047] In an optional implementation, the virtual scene further includes a three-dimensional virtual map matching a target real scene, and positions of the plurality of virtual objects are fixed relative to the three-dimensional virtual map; and the displaying module is specifically configured to:

[0048] Based on the real scene image and the three-dimensional virtual map, determine whether a current real scene where the AR device is located is the target real scene;

[0049] In a case where the current real scene is the target real scene, based on the real scene image and position information of the plurality of virtual objects relative to the three-dimensional virtual map, display the AR picture in the AR device.

[0050] In an optional implementation, the displaying module is specifically configured to:

[0051] Obtain pose information of the AR device in a current real scene where the AR device is located;

[0052] Based on the pose information, the real scene image, and the virtual scene, display the AR picture in the AR device.

[0053] In an optional implementation, the displaying module is specifically configured to:

[0054] determine a conversion relationship between the virtual coordinate system and a world coordinate system of the current real scene based on the pose information;

[0055] determine position information of the plurality of virtual objects in the world coordinate system based on the position information of the plurality of virtual objects in the virtual coordinate system and the conversion relationship;

[0056] display the AR picture in the AR device based on the position information of the plurality of virtual objects in the world coordinate system and the real scene image.

[0057] In an optional implementation, the display module is further configured to:

[0058] display a new target object in the AR picture, the new target object is used to eliminate the remaining virtual objects, and a shape of the new target object is randomly generated.

[0059] The embodiments of the present disclosure further provide an electronic device, including a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the above-mentioned augmented reality interaction method.

[0060] The embodiments of the present disclosure further provide a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the above-mentioned augmented reality interaction method.

[0061] The effects of the above-mentioned augmented reality interaction device, electronic device and computer readable storage medium are described in the above-mentioned augmented reality interaction method, which will not be repeated here.

[0062] In order to make the above-mentioned purposes, features and advantages of the present disclosure more obvious and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are described in detail as follows. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments, the drawings herein are incorporated into the description and form a part of the description, the drawings show the embodiments consistent with the present disclosure, and are used to illustrate the technical solutions of the present disclosure together with the description. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0064] Figure 1A schematic diagram of an execution subject of an augmented reality interaction method provided by an embodiment of the present disclosure is shown;

[0065] Figure 2 A flowchart of an augmented reality interaction method provided by an embodiment of the present disclosure is shown;

[0066] Figure 3 A schematic diagram of an AR picture provided by an embodiment of the present disclosure is shown;

[0067] Figure 4 A schematic diagram of an AR scene picture provided by an embodiment of the present disclosure is shown;

[0068] Figure 5 A flowchart of a method for removing a virtual object satisfying a preset relationship from an AR picture provided by an embodiment of the present disclosure is shown;

[0069] Figure 6 Another schematic diagram of an AR scene picture provided by an embodiment of the present disclosure is shown;

[0070] Figure 7 A schematic diagram of an AR picture before adjustment of a target object provided by an embodiment of the present disclosure is shown;

[0071] Figure 8 A schematic diagram of an AR picture after adjustment of a target object provided by an embodiment of the present disclosure is shown;

[0072] Figure 9 A flowchart of a method for displaying an AR picture provided by an embodiment of the present disclosure is shown;

[0073] Figure 10 Another flowchart of a method for displaying an AR picture provided by an embodiment of the present disclosure is shown;

[0074] Figure 11 A flowchart of a method for displaying an AR picture based on pose information, a real scene image and a virtual scene provided by an embodiment of the present disclosure is shown;

[0075] Figure 12 A structural schematic diagram of an augmented reality interaction device provided by an embodiment of the present disclosure is shown;

[0076] Figure 13 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0077] To make the purposes, technical solutions, and advantages of the embodiments of the present disclosure clearer, the following will be combined with the accompanying drawings of the embodiments of the present disclosure to make a clear and complete description of the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. The components of the embodiments of the present disclosure generally described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0078] It should be noted that similar reference numerals and letters refer to similar items throughout the accompanying drawings, and therefore, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings.

[0079] The term “and / or” herein only describes an association relationship, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the term “at least one” herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0080] It is found through research that at present, the experience mode for eliminating the casual game is usually a way for users to realize human-computer interaction on the screen of a mobile terminal, for example, a user can eliminate the objects to be matched by clicking the objects to be matched in the screen. However, this kind of interaction mode leads to a relatively single interaction experience of the user in the game.

[0081] With the development of Augmented Reality (AR) technology, the interactive form of AR technology is widely used, therefore, how to use the AR interactive form to improve the interactive feeling and the sense of immersion of the user relative to the game environment to realize an immersive game experience is a hot research direction in the current game field.

[0082] Among them, AR technology is a technology of fusing virtual information with a real scene. Due to the advantages of strong sense of presence, high entertainment, and strong interactivity of AR technology, AR technology is widely applied in the game field. AR technology can break through the space, time, and other objective restrictions, so that the user can experience the experience that cannot be experienced in the real world.

[0083] Based on the above research, the embodiment of the disclosure provides an augmented reality interaction method, which comprises the following steps: acquiring a real scene image; displaying an AR picture in an augmented reality AR device based on the real scene image and a pre-constructed virtual scene, wherein the AR picture contains a target object; a plurality of virtual objects to be eliminated are suspended in the virtual scene, and the virtual objects move towards a preset direction; and in response to movement of the AR device, removing a virtual object that meets a preset relationship from the AR picture in a case where the target object and any virtual object of the plurality of virtual objects meet the preset relationship.

[0084] In the embodiment of the disclosure, the real scene image is combined with the virtual scene, and the AR picture containing the target object and the plurality of virtual objects is displayed in the AR device. The user can change the position of the target object by moving the AR device, and the virtual object in the AR picture is eliminated in a case where the target object and any virtual object of the pluralityity of virtual objects meet the preset relationship. That is, the atmosphere of combining the virtual and the real is added to the elimination process, which can improve the interest of the elimination process. The change of the position of the target object is realized by moving the AR device, which has the atmosphere of somatosensory interaction, so that the user can experience the in-situ game experience, thereby facilitating the improvement of the interaction experience of the user.

[0085] Please refer to Figure 1 The execution subject of the augmented reality interaction method provided by the embodiment of the disclosure is shown in the schematic diagram. Figure 1 As shown in the schematic diagram, the execution subject of the method is an electronic device 100, and the electronic device 100 comprises a terminal and a server 10, wherein the terminal can be Figure 1 a smartphone 20, a tablet computer 30 or a notebook computer 40 shown in the schematic diagram, and can also be Figure 1 a smart speaker or a smart watch and the like not shown in the schematic diagram, and is not limited.

[0086] In an optional implementation, the terminal can further comprise an AR device, a VR (Virtual Reality) device, an MR (Mixed Reality) device and the like. For example, the AR device can be a smartphone or a tablet computer with AR function, or can be AR glasses, which are not limited here.

[0087] It should be noted that in some embodiments, the server 10 can communicate with the smartphone 20, the tablet computer 30 and the notebook computer 40 respectively through a network 50. The network 50 can comprise various connection types, such as wired, wireless communication links or optical fiber cables and the like.

[0088] Optionally, the augmented reality interaction method can be implemented by a processor invoking computer readable instructions stored in a memory.

[0089] Referring to Figure 2 , Figure 2 A flowchart of an augmented reality interaction method provided by an embodiment of the present disclosure. As shown in Figure 2 , the augmented reality interaction method provided by an embodiment of the present disclosure includes the following S101-S103:

[0090] S101, acquiring a real scene image.

[0091] The real scene image can be acquired in real time by a camera integrated in a terminal as shown in Figure 1 The real scene image can reflect environmental information of an environment in which the terminal is currently located, such as buildings, roads, pedestrians, landscapes, vehicles, etc.

[0092] In other embodiments, the real scene image can also be acquired by an external camera of the terminal, which is not limited here.

[0093] S102, based on the real scene image and a pre-constructed virtual scene, displaying an AR picture in an augmented reality AR device, the AR picture containing a target object; a plurality of virtual objects to be eliminated are suspended in the virtual scene, and the virtual objects move towards a preset direction.

[0094] The target object is a pattern displayed in the AR picture, which can be a rectangular, circular or other shaped pattern, which is not limited here. In the present disclosure, the shape of the target object can change in response to a triggering operation of a user.

[0095] It can be understood that, since the virtual scene is pre-constructed, the positions of the plurality of virtual objects in the virtual scene (e.g., the position of each virtual object and the relative positions between the plurality of virtual objects) are also pre-constructed. Optionally, the virtual scene can be built by a game development engine such as Unity or UE4, which is not limited here. The shapes of the virtual objects can be the same or different, which is not limited here.

[0096] The preset direction is a direction towards the AR device, so that the virtual objects appear to "fly" towards the AR device based on the experience angle of the user, which is beneficial to improve the "immersive" game experience of the user.

[0097] For example, referring to Figure 3This is a schematic diagram of an AR screen provided in an embodiment of this disclosure. Figure 3 As shown, AR screen A includes a real-world scene image A1, a target object A2, and a virtual object A3. The virtual object A3 moves toward the screen of the AR device, for example, it can move along the line connecting the center point of the virtual object A3 and the center point of the AR device screen.

[0098] For example, please see Figure 4 This is a schematic diagram of an AR scene provided in an embodiment of this disclosure, such as... Figure 4 As shown, the scene includes AR screen 02 displayed by AR device 01. AR screen 02 includes target object 021, real scene 03, and multiple virtual objects (e.g., virtual objects 04, 05, 06, 07, 08, and 09). Since virtual objects 04-09 are distributed in different positions in the real scene, AR device 01 can capture part of the real scene content in real scene 03 through the camera device at the current position. At this time, AR screen 02 does not include virtual objects.

[0099] It should be noted that, as Figure 4 As shown, since the position of virtual objects in the real scene is fixed, and since the range of the real scene image captured by the AR device in a stationary state is limited, when a user is playing the game at their current location, the virtual objects in the virtual scene may not be visible to the user in the current AR screen due to their different distribution positions. As the user moves or rotates the AR device, the virtual objects may appear to change in the current AR screen. Figure 4 Move the position in the middle to Figure 6 During the process of moving to the current location, virtual objects along the movement path will appear successively in the AR screen (such as...). Figure 6 (as shown in the image).

[0100] S103, in response to the movement of the AR device, if the target object and any one of the plurality of virtual objects meet a preset relationship, the virtual object that meets the preset relationship is removed from the AR screen.

[0101] The movement of the AR device can include translating the AR device in multiple directions and rotating the AR device. That is, the user can hold the AR device and move it slightly in a space. The AR device can use its built-in gyroscope and six degrees of freedom (6DOF) tracking technology to detect its own attitude and orientation in space.

[0102] In some embodiments, the preset relationship between the target object and any virtual object in the plurality of virtual objects refers to that the shape of the target object matches the shape of a target virtual object in the plurality of virtual objects, i.e., the shape of the target object is the same as the shape of the target virtual object, and the target object is in the moving direction of the target virtual object.

[0103] It should be noted that the target virtual object is the virtual object closest to the AR device.

[0104] For example, if the target virtual object is only partially displayed in the AR picture or does not exist in the AR picture, the user can pan the AR device in any direction or rotate the AR device, so that the target virtual object can be displayed in the AR picture, and the target virtual object flies to the user. The user moves the AR device to catch the target virtual object, thereby completing a game experience.

[0105] Optionally, while removing the virtual object satisfying the preset relationship from the AR picture, corresponding evaluation information can also be displayed in the AR picture, where the evaluation information can be a score corresponding to the elimination of the target virtual object, such as a score obtained if the user can catch the virtual object using the target object of the same shape, or a score deducted if the user cannot catch or the shapes do not match, or evaluation information for the elimination result, such as good, perfect, nice, etc. evaluation information displayed in the AR picture if the user can catch the virtual object using the target object of the same shape, which is not limited herein.

[0106] In the embodiments of the present disclosure, the real scene image is combined with the virtual scene, and the AR picture is displayed in the AR device. In response to the movement of the AR device, the position of the target object can be changed to eliminate the virtual object in the AR picture, i.e., the atmosphere of combining virtual and real is added to the elimination process, which can improve the interest of the elimination process. And because the AR device can move, it has a body-sensing interactive atmosphere, so that the user can experience an immersive game experience, thereby facilitating the improvement of the user's interactive experience.

[0107] In some embodiments, for step S103, in response to the movement of the AR device, the virtual object satisfying the preset relationship between the target object and any virtual object in the plurality of virtual objects is removed from the AR picture, please refer to Figure 5 may include the following S1031-S1034:

[0108] S1031, in response to the AR device moving, judging whether the shape of the target object matches the shape of a target virtual object in the plurality of virtual objects, if yes, executing step S1032, if no, executing step S1033.

[0109] S1032, in the case that the target object is in the moving direction of the target virtual object, removing the target virtual object from the AR picture.

[0110] It can be understood that if the shape of the target object matches the shape of the target virtual object and the target object is in the moving direction of the target virtual object, the target virtual object can be removed from the AR picture.

[0111] Exemplarily, please refer to Figure 4 and Figure 6 , Figure 6 is a schematic diagram of another AR scene picture provided by an embodiment of the present disclosure. As shown in Figure 4 , the pattern of the target object 021 in the AR picture 02 displayed by the AR device 01 is similar to the pattern of an inverse "7", and the shape of the target virtual object 05 displayed by the AR picture matches the shape of the target object 021. The user moves the AR device in the space, i.e., the AR device moves from the position shown in Figure 4 to the position shown in Figure 6 , so that the target object 021 coincides with the target virtual object 05, and thus the target virtual object 05 can be removed from the AR picture.

[0112] S1033, in response to a shape adjustment operation for the target object, adjusting the shape of the target object to be the same as the shape of the target virtual object.

[0113] S1034, in the case that the target object after the shape adjustment is in the moving direction of the target virtual object, removing the target virtual object from the AR picture.

[0114] It can be understood that in the case that the shape of the target object does not match the shape of the target virtual object, in order to remove the target virtual object from the AR picture, the shape of the target object can be adjusted until it is the same as the shape of the target virtual object, and then in the case that the shape of the target object is the same as the shape of the target virtual object and the target object after the shape adjustment is in the moving direction of the target virtual object, the target virtual object can be removed.

[0115] Exemplarily, please refer to Figure 7 and Figure 8 , wherein Figure 7 is a schematic diagram of an AR picture before a target object is adjusted provided by an embodiment of the present disclosure, Figure 8A schematic diagram of the AR picture adjusted for the target object. As shown in Figure 7 The real scene image 61 currently displayed in the AR picture 60, the target object 62 is a trapezoidal pattern, and the target virtual object 63 is a diamond pattern. Since the shape of the target object 62 does not match the shape of the target virtual object 63, the shape of the target object 62 can be adjusted in response to a shape adjustment operation of the target object 62 by the user (e.g., a click operation of the target object 62 by the user), until the shape of the target object 62 is adjusted to the shape of the target object as shown in Figure 8 The shape of the target object 62 is adjusted to the shape of the target object as shown in

[0116] In some embodiments, since there are multiple virtual objects, after the virtual object satisfying the preset relationship is removed from the AR picture, a new target object is displayed in the AR picture, and the shape of the new target object is randomly generated. Then, the remaining virtual objects can be eliminated based on the new target object. That is, the user continues to move the AR device so that the new target object continues to eliminate the remaining virtual objects. In this way, the integrity of the game experience can be improved. In addition, since the new target object is randomly generated, the challenge of the game can be improved while ensuring the interest of the game, thereby improving the enthusiasm of the user to participate in the game.

[0117] In some embodiments, for step S102, when the AR picture is displayed in the AR device based on the real scene image and the pre-constructed virtual scene, the AR picture can be displayed in the following two ways.

[0118] The first embodiment is that the pre-constructed virtual scene and the real scene are fixedly corresponding. That is, the pre-constructed virtual scene is loaded and placed in the pre-set real scene by an image capturing device such as a camera to realize the fixed correspondence between the virtual scene and the real scene. In this case, scanning other real scenes cannot identify and load the virtual scene, that is, the game experience cannot be realized.

[0119] Another embodiment is that the pre-constructed virtual scene and the real scene are not fixedly corresponding. That is, without pre-real scene collection, the current real scene environment can be scanned in real time by the AR device, and the virtual scene can be loaded and positioned in space for game experience.

[0120] The following will be described in detail for the two embodiments. For the first embodiment, when displaying an AR picture in the AR device based on the real scene image and the pre-constructed virtual scene, please refer to Figure 9 , which can include the following S102A-S102B:

[0121] S102A, determining whether the current real scene where the AR device is located is the target real scene based on the real scene image and the three-dimensional virtual map.

[0122] The three-dimensional virtual map is a map pre-constructed in the virtual scene, the three-dimensional virtual map matches the target real scene, the target real scene is pre-acquired, and the positions of the plurality of virtual objects in the virtual scene relative to the three-dimensional virtual map are fixed.

[0123] Since the target real scene is pre-acquired, the content of the real scene where the AR device is located can be compared with the content of the target real scene to determine whether the current real scene where the AR device is located is the target real scene.

[0124] S102B, in the case where the current real scene is the target real scene, displaying the AR picture in the AR device based on the real scene image and the position information of the plurality of virtual objects relative to the three-dimensional virtual map.

[0125] It can be understood that in the case where the current real scene is the target real scene, the AR picture is displayed based on the real scene image and the position information of the plurality of virtual objects relative to the three-dimensional virtual map, that is, in this case, the virtual scene is loaded and the AR picture is displayed.

[0126] Optionally, the three-dimensional virtual map can be constructed in the following manner:

[0127] (a) obtaining a plurality of real scene images.

[0128] Exemplarily, the real scene of the target real scene (e.g. a park) can be pre-acquired by an image acquisition device for multi-angle image acquisition, obtaining a plurality of target real scene images.

[0129] (b) constructing an initial three-dimensional scene virtual model representing the real scene based on the plurality of real scene sample images.

[0130] Specifically, constructing a three-dimensional scene virtual model can include the following steps:

[0131] (1) extracting a plurality of feature points from each real scene sample image obtained;

[0132] (2) generating an initial three-dimensional scene virtual model based on the extracted plurality of feature points and a pre-stored three-dimensional sample graph matching the real scene; wherein the three-dimensional sample graph is a pre-stored three-dimensional graph representing the topographic features of the real scene.

[0133] Specifically, the feature points extracted for each real scene sample image can be points that can represent the key information of the real scene sample image, such as for a real scene sample image containing a target object (such as a tree, the ground, etc.), the feature points here can represent the feature points of the contour information of the target object.

[0134] Exemplarily, the pre-stored three-dimensional sample graph of the real scene here can include a three-dimensional graph that is set in advance and can represent the topographic features of the real scene with size annotations, such as a Computer Aided Design (CAD) three-dimensional graph representing the topographic features of the real scene.

[0135] For the real scene, when the extracted feature points are sufficient, the feature point cloud formed by the feature points can form a three-dimensional model representing the real scene. The feature points in the feature point cloud are unitless, and the three-dimensional model formed by the feature point cloud is also unitless. Then, after aligning the feature point cloud with the three-dimensional graph with scale annotations and capable of representing the topographic features of the real scene, the initial three-dimensional scene virtual model corresponding to the real scene is obtained.

[0136] (c) aligning the calibration feature points on the constructed initial three-dimensional scene virtual model with the calibration feature points corresponding to the real scene to generate a three-dimensional virtual scene map.

[0137] The generated initial three-dimensional model may have distortion phenomena, which can then be adjusted by the corresponding two-dimensional map of the real scene to obtain a three-dimensional scene virtual model with higher accuracy.

[0138] For step (c), when aligning the calibration feature points on the constructed initial three-dimensional scene model with the calibration feature points corresponding to the real scene to generate a three-dimensional virtual scene map, it includes:

[0139] (c1) extracting calibration feature points for representing a plurality of spatial position points of the real scene in the initial three-dimensional scene model corresponding to the real scene;

[0140] (c2) determining the real coordinate data of the calibration feature points in the real two-dimensional map corresponding to the real scene, and adjusting the coordinate data of each feature point in the initial three-dimensional scene model based on the real coordinate data corresponding to each calibration feature point.

[0141] Exemplarily, some feature points representing the spatial position points of the edges and corners of the target object can be selected as the calibration feature points here, and then based on the real coordinate data corresponding to the calibration feature points and the coordinate data of the calibration feature points in the initial three-dimensional scene virtual model, the coordinate data adjustment amount is determined, and then the coordinate data of each feature point in the initial three-dimensional model is corrected based on the coordinate data adjustment amount, that is, a three-dimensional scene virtual model with higher accuracy can be obtained.

[0142] In some possible implementations, when judging whether the current real scene where the AR device is located is the target real scene, the following can be included:

[0143] (I) Extracting feature points contained in the real scene image and extracting feature points of each real scene sample image when the three-dimensional virtual map is pre-constructed.

[0144] (II) Based on the feature points corresponding to the real scene image and the feature points corresponding to each real scene sample image when the three-dimensional virtual map is pre-constructed, judging whether the current real scene where the AR device is located is the target real scene.

[0145] Exemplarily, after obtaining the real scene image shot by the AR device, the target real scene sample image with the highest similarity to the real scene image can be found through the feature points in the real scene image and the feature points of each real scene sample image when the three-dimensional virtual map is pre-constructed. For example, the similarity values of the real scene image and each real scene sample image can be determined based on the feature information of the feature points of the real scene image and the feature information of the feature points of each real scene sample image, and the real scene sample image with the highest similarity value and exceeding a similarity threshold is taken as the target real scene sample image here.

[0146] After the target real scene sample image is determined, whether the current real scene where the AR device is located is the target real scene can be judged.

[0147] For another implementation, when the AR picture is displayed in the AR device based on the real scene image and the pre-constructed virtual scene, please refer to Figure 10 The following S1021-S1022 can be included:

[0148] S1021, obtaining the pose information of the AR device in the current real scene where the AR device is located.

[0149] The pose information includes the current position information and the current attitude information of the AR device in the world coordinate system corresponding to the current real scene, and the current attitude information can include the current orientation of the AR device, which can be obtained by the built-in gyroscope.

[0150] S1022, display the AR picture in the AR device based on the pose information, the real scene image, and the virtual scene.

[0151] Since the virtual scene in the embodiment is not fixedly corresponding to the real scene, that is, the virtual scene can be loaded in any real scene and the AR picture is displayed in the AR device. In this way, the user can experience the game in any real scene, thereby improving the applicability of game interaction.

[0152] Specifically, for step S1022, please refer to Figure 11 When displaying the AR picture in the AR device based on the pose information, the real scene image, and the virtual scene, the following S10221-S10223 can be included:

[0153] S10221, determine the conversion relationship between the virtual coordinate system and the world coordinate system of the current real scene based on the pose information.

[0154] When determining the conversion relationship between the virtual coordinate system and the world coordinate system of the current real scene, the reference position in the virtual coordinate system of the virtual scene can be first aligned with the position of the AR device in the current real scene.

[0155] The specific position information of the reference position in the virtual coordinate system of the virtual scene can be set according to actual conditions, for example, any reference point can be used as the reference position. For example, the origin of the virtual coordinate system can be used as the reference position, and other points in the virtual coordinate system can also be used as the reference position.

[0156] It can be understood that when the reference position in the virtual coordinate system of the virtual scene is aligned with the position of the AR device in the current real scene, the conversion relationship between the virtual coordinate system and the world coordinate system of the real scene is determined. That is, the AR device corresponds to a virtual position information in the virtual coordinate system and a real position information in the real scene. Thus, the conversion relationship between the virtual coordinate system and the world coordinate system can be determined based on the pose information.

[0157] S10222, determine the position information of the plurality of virtual objects in the world coordinate system based on the position information of the plurality of virtual objects in the virtual coordinate system and the conversion relationship.

[0158] S10223, display the AR picture in the AR device based on the position information of the plurality of virtual objects in the world coordinate system and the real scene image.

[0159] Exemplarily, after the conversion relationship is determined, the position of each virtual object in the world coordinate system can be determined according to the position information of the plurality of virtual objects in the virtual coordinate system, and thus the position of each virtual object in the real scene image can be determined, and then the AR picture can be displayed in the AR device. That is, through the conversion relationship between the virtual coordinate system and the world coordinate system, the virtual scene can be loaded in any real scene, that is, the AR picture does not need to be displayed based on the preset real scene, and thus the application range of the virtual scene loading can be improved, so that the user can eliminate the virtual object in any real scene, and thus the user experience is improved.

[0160] The interaction method in the embodiment of the present disclosure will be described below with a specific game scene as an example.

[0161] The user holds the AR device and starts the camera to obtain the real scene image, and loads the virtual scene to fuse the real scene image with the virtual scene, so that the user can see the fused AR picture in the screen of the AR device.

[0162] Then, if there is no virtual object in the current AR picture, the user can move back and forth in the space or rotate the AR device to have different poses of the AR device, so as to adjust the user's view angle, so that the user can observe the to-be-eliminated virtual objects (block matrix) floating in the air in the AR picture, and these block matrixes will randomly fly to the player, at the same time, the target object (pattern) will appear in the AR picture, corresponding to the different shapes of the to-be-eliminated virtual objects in the space, when the virtual objects of different shapes fly, the player needs to switch the pattern matching the virtual object to "catch" the virtual object (for example, the target object can be switched by clicking the target object), when the pattern shape of the target object matches the shape of the virtual object, a score can be obtained, and when the virtual object is missed or matched with the wrong pattern, the score will be reduced, and as many scores as possible are obtained as the winning condition.

[0163] Based on the same inventive concept, the present embodiment also provides an augmented reality interaction device corresponding to the augmented reality interaction method. Since the principle of the device in the embodiment of the present disclosure solves the problem similar to the augmented reality interaction method of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described here.

[0164] Please refer to Figure 12 The structure of an augmented reality interaction device provided in the embodiment of the present disclosure is shown in the figure, and the augmented reality interaction device 1200 includes an acquisition module 1210, a display module 1220 and a removal module 1230; wherein,

[0165] The acquisition module 1210 is configured to acquire a real scene image.

[0166] The display module 1220 is configured to display an AR picture in an augmented reality (AR) device based on the real scene image and a pre-constructed virtual scene, the AR picture containing a target object, and a plurality of virtual objects in the virtual scene floating and moving towards a preset direction.

[0167] The removal module 1230 is configured to, in response to movement of the AR device, remove a virtual object from the AR picture when the target object and the virtual object meet a preset relationship.

[0168] In an optional implementation, the preset direction is a direction towards the AR device.

[0169] In an optional implementation, the removal module 1230 is specifically configured to:

[0170] In response to the movement of the AR device, determine whether a shape of the target object matches a shape of a target virtual object in the plurality of virtual objects, the target virtual object being a virtual object closest to the AR device.

[0171] In a case where the shape of the target object matches the shape of the target virtual object and the target object is in a moving direction of the target virtual object, remove the target virtual object from the AR picture.

[0172] In an optional implementation, the removal module 1230 is specifically further configured to:

[0173] In a case where the shape of the target object does not match the shape of the target virtual object, in response to a shape adjustment operation on the target object, adjust the shape of the target object to be the same as the shape of the target virtual object.

[0174] In a case where the target object after the shape adjustment is in the moving direction of the target virtual object, remove the target virtual object from the AR picture.

[0175] In an optional implementation, the virtual scene further includes a three-dimensional virtual map matching a target real scene, and positions of the plurality of virtual objects relative to the three-dimensional virtual map are fixed.

[0176] The display module 1220 is specifically configured to:

[0177] In a case where the current real scene is the target real scene, the AR device displays the AR picture based on the real scene image and the position information of the virtual objects relative to the three-dimensional virtual map.

[0178] In an optional implementation, the display module 1220 is specifically configured to:

[0179] obtain pose information of the AR device in a current real scene;

[0180] display the AR picture in the AR device based on the pose information, the real scene image, and the virtual scene.

[0181] In an optional implementation, the display module 1220 is specifically configured to:

[0182] determine a conversion relationship between the virtual coordinate system and a world coordinate system of the current real scene based on the pose information;

[0183] determine position information of the virtual objects in the world coordinate system based on the position information of the virtual objects in the virtual coordinate system and the conversion relationship;

[0184] display the AR picture in the AR device based on the position information of the virtual objects in the world coordinate system and the real scene image.

[0185] In an optional implementation, the display module 1220 is further configured to:

[0186] display a new target object in the AR picture, the new target object is used to eliminate the remaining virtual objects, and a shape of the new target object is randomly generated.

[0187] The descriptions of the processing procedures of the modules in the apparatus and the interaction procedures between the modules can refer to the related descriptions in the above method embodiments, and will not be repeated here.

[0188] Based on the same technical concept, the embodiments of the present disclosure further provide an electronic device. Referring to FIG. 13, Figure 13 As shown in FIG. 13, the electronic device 1300 provided by the embodiments of the present disclosure includes a processor 1301, a memory 1302, and a bus 1303. The memory 1302 is configured to store execution instructions, including an internal memory 13021 and an external memory 13022; the internal memory 13021 is also called an internal storage, and is used to temporarily store operation data in the processor 1301 and exchange data with the external memory 13022 such as a hard disk, and the processor 1301 exchanges data with the external memory 13022 through the internal memory 13021.

[0189] In the embodiments of the present application, the memory 1302 is specifically used for storing application program codes for executing the scheme of the present application, and is controlled to execute by the processor 1301. That is, when the electronic device 1300 is running, the processor 1301 and the memory 1302 communicate through the bus 1303, so that the processor 1301 executes the application program codes stored in the memory 1302, and further executes the method in any of the foregoing embodiments.

[0190] The processor 1301 can be an integrated circuit chip having a processing capability of signals. The processor mentioned above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; or can be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general processor can be a microprocessor or the processor can also be any conventional processor or the like.

[0191] The memory 1302 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0192] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 1300. In some other embodiments of the present application, the electronic device 1300 can include more or fewer components than those shown, or combine some components, or split some components, or different arrangement of components. The components shown can be implemented in hardware, software or a combination of software and hardware.

[0193] The embodiments of the present disclosure also provide a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is run by a processor, the steps of the augmented reality interaction method described in the method embodiments are executed. The storage medium can be a volatile or non-volatile computer readable storage medium.

[0194] The embodiment of the present disclosure further provides a computer program product carrying program codes, the program codes comprising instructions for executing steps of the augmented reality interaction method described in the above method embodiments, which can be referred to the above method embodiments and will not be repeated here.

[0195] The computer program product can be implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0196] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system and terminal can refer to the corresponding process in the above method embodiments, which will not be repeated here. In the several embodiments provided by the present disclosure, it should be understood that the disclosed system, terminal and method can be implemented by other ways. The terminal embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be through some communication interface, or indirect coupling or communication connection between units, which can be electrical, mechanical or other forms.

[0197] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the present embodiment.

[0198] In addition, each functional unit in each embodiment of the present disclosure can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0199] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or say the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for making an electronic device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0200] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present disclosure, used to illustrate the technical solutions of the present disclosure, and not to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can make modifications or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present disclosure, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. An augmented reality interaction method, characterized by, The method comprises: acquiring a real scene image; displaying an augmented reality (AR) picture in an AR device based on the real scene image and a pre-constructed virtual scene, the AR picture containing a target object; the virtual scene having a plurality of virtual objects to be eliminated floating therein, the virtual objects moving towards a preset direction; in response to movement of the AR device, removing a virtual object from the AR picture in a case where the target object and the virtual object meet a preset relationship, the position of the target object changing with the movement of the AR device; wherein the virtual object closest to the AR device is a target virtual object, and the preset relationship is that the target object and the target virtual object are of the same shape and coincide with each other.

2. The method of claim 1, wherein, The preset direction is a direction towards the AR device.

3. The method of claim 2, wherein, The method further comprises: in a case where the shape of the target object does not match the shape of the target virtual object, adjusting the shape of the target object to be the same as the shape of the target virtual object in response to a shape adjustment operation for the target object; in a case where the target object after the shape adjustment and the target virtual object coincide with each other, removing the target virtual object from the AR picture.

4. The method of claim 1, wherein, The virtual scene further comprises a three-dimensional virtual map matching a target real scene, the positions of the plurality of virtual objects being fixed relative to the three-dimensional virtual map; the displaying of the AR picture in the AR device based on the real scene image and the pre-constructed virtual scene comprises: judging whether a current real scene in which the AR device is located is the target real scene based on the real scene image and the three-dimensional virtual map; in a case where the current real scene is the target real scene, displaying the AR picture in the AR device based on the real scene image and the position information of the plurality of virtual objects relative to the three-dimensional virtual map.

5. The method of claim 1, wherein, The displaying of the AR picture in the AR device based on the real scene image and the pre-constructed virtual scene comprises: acquiring pose information of the AR device in a current real scene in which the AR device is located; displaying the AR picture in the AR device based on the pose information, the real scene image and the virtual scene.

6. The method of claim 5, wherein, The displaying of the AR picture in the AR device based on the pose information, the real scene image and the virtual scene comprises: determining a conversion relationship between a virtual coordinate system and a world coordinate system of the current real scene based on the pose information; determining position information of the plurality of virtual objects in the world coordinate system based on the position information of the plurality of virtual objects in the virtual coordinate system and the conversion relationship; displaying the AR picture in the AR device based on the position information of the plurality of virtual objects in the world coordinate system and the real scene image.

7. The method of claim 1, wherein, After the removing of the virtual object meeting the preset relationship from the AR picture, the method further comprises: The new target object is used to eliminate the remaining virtual objects, and a shape of the new target object is randomly generated.

8. An augmented reality interactive device, characterized by, The method comprises the steps of: An acquisition module is configured to acquire a real scene image. An exhibition module is configured to exhibit an AR picture in an augmented reality (AR) device based on the real scene image and a pre-constructed virtual scene, wherein the AR picture contains a target object, and a plurality of virtual objects to be eliminated are suspended in the virtual scene and move towards a preset direction. A removal module is configured to remove a virtual object from the AR picture when a preset relationship is met between the target object and any virtual object of the plurality of virtual objects in response to movement of the AR device, wherein a position of the target object changes with movement of the AR device, the virtual object closest to the AR device is a target virtual object, and the preset relationship is that the target object and the target virtual object are of the same shape and coincide with each other.

9. An electronic device, comprising: The method comprises the steps of: A processor, a memory, and a bus, wherein the memory stores machine readable instructions executable by the processor, the processor and the memory communicate through the bus when the electronic device is running, and the machine readable instructions are executed by the processor to perform the augmented reality interaction method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the augmented reality interaction method of any one of claims 1 to 7.

Images