Interaction method and device based on augmented reality, electronic device and storage medium

Abstract

The present disclosure relates to an interactive method and device based on augmented reality, an electronic device and a storage medium. The method comprises: collecting an environment image; obtaining a visual positioning result of the user terminal according to the environment image; obtaining interactive information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the visual positioning result; and displaying the interactive information in a real scene image displayed on a display interface of the user terminal.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to an augmented reality-based interaction method and device, an electronic device, and a storage medium. Background Art

[0002] Augmented Reality (AR) is a technology that cleverly integrates virtual information with the real world. It can apply computer-generated text, images, three-dimensional models, music, videos and other virtual information to the real world through a variety of technical means, and combine and interact with real-world scenes to enhance the real world. As augmented reality technology becomes more and more popular, its applications are also diversified, enabling a more interactive and intelligent experience. Summary of the invention

[0003] The present invention provides an interactive technology solution based on augmented reality.

[0004] According to one aspect of the present disclosure, an augmented reality-based interaction method is provided, which is applied to a user terminal, and the method includes:

[0005] Collect environmental images;

[0006] Obtaining a visual positioning result of the user terminal according to the environment image;

[0007] Acquire interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the visual positioning result;

[0008] The interactive information is displayed in the real scene image displayed on the display interface of the user terminal.

[0009] By collecting environmental images, a visual positioning result of the user terminal is obtained according to the environmental images, and interaction information is obtained according to one or both of the current time and the environmental information of the environment where the user terminal is located, as well as the visual positioning result. The interaction information is displayed in a real-scene image displayed on a display interface of the user terminal. Thus, the interaction information can be automatically transformed in combination with the current time and / or the environmental information of the environment where the user terminal is located, together with the visual positioning result of the user terminal, so as to provide users with a more diverse and scenario-based interactive experience based on augmented reality.

[0010] In a possible implementation manner, obtaining a visual positioning result of the user terminal according to the environment image includes:

[0011] Generate a visual positioning request according to the environment image;

[0012] Sending the visual positioning request to the server, so that the server performs visual positioning according to the environment image and a preset point cloud map to obtain a visual positioning result of the user terminal;

[0013] Receive the visual positioning result of the user terminal returned by the server.

[0014] In this implementation, a visual positioning request is generated based on the environmental image, and the visual positioning request is sent to the server, so that the server performs visual positioning based on the environmental image and a preset point cloud map to obtain the visual positioning result of the user terminal, and receives the visual positioning result of the user terminal returned by the server. This can reduce the computational complexity of the user terminal, save the power consumption of the user terminal, and obtain accurate visual positioning results of the user terminal.

[0015] In a possible implementation manner, the environmental information includes at least one of temperature information, light information, and sound information.

[0016] In this implementation, interaction information is obtained by combining at least one of temperature information, lighting information and sound information with the visual positioning result of the user terminal, or interaction information is obtained by combining at least one of temperature information, lighting information and sound information with the current time and the visual positioning result of the user terminal, thereby obtaining interaction information that better matches the environmental information of the environment where the user terminal is located, thereby providing a more natural and rich interaction experience.

[0017] In a possible implementation manner, acquiring the interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the visual positioning result, includes:

[0018] In response to the current time belonging to a preset festival day, interaction information is acquired according to the preset festival day and the visual positioning result.

[0019] In this implementation, in response to the current time belonging to a preset festival day, interaction information is obtained according to the preset festival day and the visual positioning result, thereby being able to provide a richer AR scenario experience on the preset festival day.

[0020] In a possible implementation manner, acquiring the interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the visual positioning result, includes:

[0021] Generate an interactive information acquisition request according to one or both of the current time and the environmental information of the environment where the user terminal is located, and the visual positioning result;

[0022] Sending the interaction information acquisition request to the server, so that the server determines the interaction information corresponding to one or both of the current time and the environmental information of the environment where the user terminal is located and the visual positioning result;

[0023] Receive the interaction information returned by the server.

[0024] In this implementation, an interaction information acquisition request is generated based on the current time and one or both of the environmental information of the environment where the user terminal is located, and the visual positioning result, and the interaction information acquisition request is sent to the server, so that the server determines the interaction information corresponding to the current time and one or both of the environmental information of the environment where the user terminal is located, and the visual positioning result, and receives the interaction information returned by the server, thereby reducing the calculation amount of the user terminal, saving the power consumption of the user terminal, and obtaining accurate interaction information.

[0025] In a possible implementation manner, the interaction information includes operable interaction information and / or inoperable interaction information.

[0026] In this implementation, by displaying the operable interactive information in the real-scene image displayed on the display interface of the user terminal, interactivity can be improved and the user experience of AR interaction can be improved; by displaying the inoperable interactive information in the real-scene image displayed on the display interface of the user terminal, a richer scenario-based AR interaction experience can be provided.

[0027] In a possible implementation manner, displaying the interaction information in a real scene image displayed on a display interface of the user terminal includes:

[0028] In the real-scene image displayed on the display interface of the user terminal, an interactive function option bar is displayed;

[0029] generating interactive function options according to the interactive information;

[0030] The interactive function options are displayed in the interactive function option bar.

[0031] In this implementation, an interactive function option bar is displayed in the real-scene image displayed on the display interface of the user terminal, interactive function options are generated based on the interactive information, and the interactive function options are displayed in the interactive function option bar. In this way, in response to the user triggering any one of the interactive function options in the interactive function option bar, the AR interactive scene corresponding to the interactive function option is opened.

[0032] In a possible implementation manner, acquiring the interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the visual positioning result, includes:

[0033] Determining the moving direction information of the user terminal according to the visual positioning result;

[0034] The interaction information is acquired according to one or both of the current time and the environment information of the environment where the user terminal is located, and the moving direction information of the user terminal.

[0035] In this implementation, the moving direction information of the user terminal is determined based on the visual positioning result, and the interaction information is obtained based on the current time and one or both of the environmental information of the environment where the user terminal is located, as well as the moving direction information of the user terminal. Thus, as the user moves, the user's intention can be predicted in combination with the current time and one or both of the environmental information of the environment where the user terminal is located, thereby providing the user with more personalized AR interaction information that is more practical for the user.

[0036] In a possible implementation manner, acquiring the interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the moving direction information of the user terminal includes:

[0037] In response to the moving direction information including leaving the parking lot, information of a target geographical area outside the parking lot is acquired as the interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located.

[0038] In this implementation, in response to the movement direction information including leaving the parking lot, information of the target geographic area outside the parking lot is obtained as the interaction information based on one or both of the current time and the environmental information of the environment where the user terminal is located, thereby helping the user to obtain the information of the target geographic area outside the parking lot in advance, thereby helping to save the user's time.

[0039] In a possible implementation manner, the information of the target geographical area includes one or both of weather information of the target geographical area and traffic condition information of the target geographical area.

[0040] In this implementation, by responding to the movement direction information including leaving the parking lot, weather information and / or traffic condition information of the target geographical area outside the parking lot is obtained, and the weather information and / or traffic condition information of the target geographical area is displayed in the real-life image displayed on the display interface of the user terminal. This can help the user obtain the weather information and / or traffic condition information of the target geographical area outside the parking lot in advance, thereby helping to save the user's time.

[0041] In a possible implementation manner, acquiring the interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the moving direction information of the user terminal includes:

[0042] In response to the moving direction information including entering a shopping mall, information of a preset type of store in the shopping mall is obtained as the interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located.

[0043] In this implementation, in response to the movement direction information including entering a shopping mall, information of preset types of stores in the shopping mall is obtained as the interaction information based on the current time and one or both of the environmental information of the environment where the user terminal is located. Thus, as the user moves, the user's intention can be predicted in combination with the current time and one or both of the environmental information of the environment where the user terminal is located, thereby providing the user with more personalized AR interaction information that is more practical for the user.

[0044] According to one aspect of the present disclosure, an augmented reality-based interaction device is provided, which is applied to a user terminal and includes:

[0045] A collection module, used for collecting environmental images;

[0046] A visual positioning module, used to obtain a visual positioning result of the user terminal according to the environment image;

[0047] An acquisition module, configured to acquire interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the visual positioning result;

[0048] A display module is used to display the interactive information in the real scene image displayed on the display interface of the user terminal.

[0049] In a possible implementation, the visual positioning module is used to:

[0050] Generate a visual positioning request according to the environment image;

[0051] Sending the visual positioning request to the server, so that the server performs visual positioning according to the environment image and a preset point cloud map to obtain a visual positioning result of the user terminal;

[0052] Receive the visual positioning result of the user terminal returned by the server.

[0053] In a possible implementation manner, the environmental information includes at least one of temperature information, light information, and sound information.

[0054] In a possible implementation, the acquisition module is used to:

[0055] In response to the current time belonging to a preset festival day, interaction information is acquired according to the preset festival day and the visual positioning result.

[0056] In a possible implementation, the acquisition module is used to:

[0057] Generate an interactive information acquisition request according to one or both of the current time and the environmental information of the environment where the user terminal is located, and the visual positioning result;

[0058] Sending the interaction information acquisition request to the server, so that the server determines the interaction information corresponding to one or both of the current time and the environmental information of the environment where the user terminal is located and the visual positioning result;

[0059] Receive the interaction information returned by the server.

[0060] In a possible implementation manner, the interaction information includes operable interaction information and / or inoperable interaction information.

[0061] In a possible implementation, the display module is used to:

[0062] In the real-scene image displayed on the display interface of the user terminal, an interactive function option bar is displayed;

[0063] generating interactive function options according to the interactive information;

[0064] The interactive function options are displayed in the interactive function option bar.

[0065] In a possible implementation, the acquisition module is used to:

[0066] Determining the moving direction information of the user terminal according to the visual positioning result;

[0067] The interaction information is acquired according to one or both of the current time and the environment information of the environment where the user terminal is located, and the moving direction information of the user terminal.

[0068] In a possible implementation, the acquisition module is used to:

[0069] In response to the moving direction information including leaving the parking lot, information of a target geographical area outside the parking lot is acquired as the interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located.

[0070] In a possible implementation manner, the information of the target geographical area includes one or both of weather information of the target geographical area and traffic condition information of the target geographical area.

[0071] In a possible implementation, the acquisition module is used to:

[0072] In response to the moving direction information including entering a shopping mall, information of a preset type of store in the shopping mall is obtained as the interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located.

[0073] According to one aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.

[0074] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0075] In an embodiment of the present disclosure, an environmental image is captured, and a visual positioning result of the user terminal is obtained based on the environmental image. Interaction information is obtained based on one or both of the current time and the environmental information of the environment in which the user terminal is located, as well as the visual positioning result. The interaction information is displayed in a real-scene image displayed on a display interface of the user terminal. Thus, the interaction information can be automatically transformed in combination with the current time and / or the environmental information of the environment in which the user terminal is located, together with the visual positioning result of the user terminal, thereby providing users with a more diverse and scenario-based interactive experience based on augmented reality.

[0076] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0077] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0079] Figure 1 A flow chart of an augmented reality-based interaction method provided by an embodiment of the present disclosure is shown.

[0080] Figure 2 A schematic diagram showing a point cloud map.

[0081] Figure 3 A block diagram of an augmented reality-based interaction device provided by an embodiment of the present disclosure is shown.

[0082] Figure 4 A block diagram of an electronic device 800 provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0083] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0084] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0085] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0086] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0087] In an embodiment of the present disclosure, an environmental image is captured, and a visual positioning result of the user terminal is obtained based on the environmental image. Interaction information is obtained based on one or both of the current time and the environmental information of the environment in which the user terminal is located, as well as the visual positioning result. The interaction information is displayed in a real-scene image displayed on a display interface of the user terminal. Thus, the interaction information can be automatically transformed in combination with the current time and / or the environmental information of the environment in which the user terminal is located, together with the visual positioning result of the user terminal, thereby providing users with a more diverse and scenario-based interactive experience based on augmented reality.

[0088] The following is a detailed description of the augmented reality-based interaction method provided by the embodiments of the present disclosure in conjunction with the accompanying drawings. Figure 1 A flow chart of an augmented reality-based interaction method provided by an embodiment of the present disclosure is shown. In one possible implementation, the augmented reality-based interaction method may be executed by a user terminal or other processing device. The user terminal may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device or a wearable device, etc. In some possible implementations, the augmented reality-based interaction method may be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the interactive method based on augmented reality includes steps S11 to S14.

[0089] In step S11, an environment image is collected.

[0090] In step S12, a visual positioning result of the user terminal is obtained according to the environment image.

[0091] In step S13, the interaction information is acquired according to one or both of the current time and the environmental information of the environment where the user terminal is located, and the visual positioning result.

[0092] In step S14, the interaction information is displayed in the real scene image displayed on the display interface of the user terminal.

[0093] In the disclosed embodiment, the environmental image may represent an image obtained by photographing the surrounding environment of the user terminal. For example, a user may hold up a user terminal (such as a mobile phone), and the user terminal may acquire an image of the environment facing the image acquisition module through an image acquisition module (such as a camera module) to obtain the environmental image. In the disclosed embodiment, the number of the environmental images may be one or more, and the environmental image may also be a short video including multiple frames of images, which is not limited here. In the disclosed embodiment, visual positioning may be performed based on the environmental image and a preset point cloud map to obtain a visual positioning result of the user terminal. For example, if the augmented reality-based interaction method is applied in a shopping mall, the preset point cloud map may include the point cloud map of the shopping mall; if the augmented reality-based interaction method is applied in a parking lot, the preset point cloud map may include the point cloud map of the parking lot; if the augmented reality-based interaction method is applied on a pedestrian street, the preset point cloud map may include the point cloud map of the pedestrian street; if the augmented reality-based interaction method is applied in an amusement park, the preset point cloud map may include the point cloud map of the amusement park; if the augmented reality-based interaction method is applied at a scenic spot, the preset point cloud map may include the point cloud map of the scenic spot; if the augmented reality-based interaction method is applied in a square, the preset point cloud map may include the point cloud map of the square; and so on.

[0094] In the embodiment of the present disclosure, the visual positioning result of the user terminal may include the coordinate information and posture information of the user terminal. The posture information of the user terminal may include the orientation, pitch angle, etc. of the user terminal. According to the visual positioning result of the user terminal, the coordinates and orientation of the user when using the user terminal to shoot the environmental image can be determined, so that the positioning information of the user can be obtained. In the embodiment of the present disclosure, the visual positioning result of the user terminal is obtained by performing visual positioning according to the environmental image and the preset point cloud map, thereby obtaining the visual positioning result of the user terminal only through the visual information around the user terminal without arranging any active equipment in advance, that is, the coordinate information and posture information of the user terminal can be obtained only through the visual information around the user terminal, which can reduce the positioning cost and obtain accurate user positioning information. By implementing the interactive method based on augmented reality based on the preset point cloud map, the accuracy of the interactive control based on augmented reality can be improved.

[0095] In a possible implementation, obtaining the visual positioning result of the user terminal according to the environment image includes: generating a visual positioning request according to the environment image; sending the visual positioning request to a server, so that the server performs visual positioning according to the environment image and a preset point cloud map to obtain the visual positioning result of the user terminal; and receiving the visual positioning result of the user terminal returned by the server. In this implementation, the user terminal may generate a visual positioning request according to the environment image, and send the visual positioning request to the server, wherein the visual positioning request may include the environment image. Thus, by sending the visual positioning request to the server through the user terminal, the server can obtain the environment image. After receiving the visual positioning request, the server may respond to the visual positioning request, perform visual positioning according to the environment image and the preset point cloud map, and obtain the visual positioning result of the user terminal. For example, the server may pre-store the preset point cloud map, extract feature points of the environment image, match the feature points of the environment image with the preset point cloud map, and obtain the visual positioning result of the user terminal. After obtaining the visual positioning result of the user terminal, the server may return the visual positioning result of the user terminal to the user terminal. The user terminal may receive the visual positioning result of the user terminal returned by the server. In this implementation, a visual positioning request is generated according to the environmental image, and the visual positioning request is sent to the server, so that the server performs visual positioning according to the environmental image and a preset point cloud map to obtain the visual positioning result of the user terminal, and receives the visual positioning result of the user terminal returned by the server, thereby reducing the amount of calculation of the user terminal, saving the power consumption of the user terminal, and obtaining accurate visual positioning results of the user terminal.

[0096] As an example of this implementation, if the server fails to determine a visual positioning result that matches the environmental image, it may return positioning failure information to the user terminal. After receiving the positioning failure information, the user terminal may prompt the user to collect the environmental image again. For example, the user is prompted to change the orientation of the user terminal, adjust the pitch angle of the user terminal, move the position of the user terminal, etc., so as to collect environmental images from different perspectives and different positions, and after collecting the environmental image, a visual positioning request is sent to the server again, thereby increasing the success probability of visual positioning.

[0097] In another possible implementation, the user terminal may obtain the preset point cloud map in advance, and perform visual positioning according to the environment image and the preset point cloud map to obtain the visual positioning result of the user terminal. For example, the user terminal may extract feature points of the environment image, match the feature points of the environment image with the preset point cloud map, and obtain the visual positioning result of the user terminal. In this implementation, visual positioning can still be achieved when the user terminal is offline.

[0098] Figure 2 A schematic diagram of a point cloud map is shown. In one possible implementation, a large number of photos and / or videos of a certain place can be collected to restore the sparse feature point cloud of the place and obtain a point cloud map of the place. In one possible implementation, a point cloud map can be generated by SFM (Structure From Motion). For example, a large number of photos or videos of a shopping mall (for example, a collection of photos of the shopping mall at different times, different angles, and different locations) can be obtained, and based on the large number of photos and / or videos of the shopping mall, the SFM technology is used to generate a point cloud map of the shopping mall. For another example, a large number of photos or videos of a pedestrian street (for example, a collection of photos of the pedestrian street at different times, different angles, and different locations) can be obtained, and based on the large number of photos and / or videos of the pedestrian street, the SFM technology is used to generate a point cloud map of the pedestrian street.

[0099] In the embodiment of the present disclosure, the environmental information of the environment where the user terminal is located may represent any information that can reflect the characteristics of the environment where the user terminal is located. In one possible implementation, the interaction information may be obtained according to the current time and the visual positioning result of the user terminal. In another possible implementation, the interaction information may be obtained according to the environmental information of the environment where the user terminal is located and the visual positioning result of the user terminal. In another possible implementation, the interaction information may be obtained according to the current time, the environmental information of the environment where the user terminal is located, and the visual positioning result of the user terminal. In the embodiment of the present disclosure, at least one item of interaction information may be obtained according to the current time and one or two items of the environmental information of the environment where the user terminal is located, and the visual positioning result. By obtaining the interaction information according to the current time and one or two items of the environmental information of the environment where the user terminal is located, and the visual positioning result, the interaction information matching the visual positioning result of the user terminal can be obtained, that is, the interaction information supported by the current position of the user can be obtained, and the obtained interaction information matches the current time and / or the environmental information of the environment where the user terminal is located. In an embodiment of the present disclosure, an environmental image is captured, and a visual positioning result of the user terminal is obtained based on the environmental image. Interaction information is obtained based on one or both of the current time and the environmental information of the environment in which the user terminal is located, as well as the visual positioning result. The interaction information is displayed in a real-scene image displayed on a display interface of the user terminal. Thus, the interaction information can be automatically transformed in combination with the current time and / or the environmental information of the environment in which the user terminal is located, together with the visual positioning result of the user terminal, thereby providing users with a more diverse and scenario-based interactive experience based on augmented reality.

[0100] In one possible implementation, the environmental information includes at least one of temperature information, lighting information, and sound information. For example, in the case where the interaction information includes an AR interactive game, a more natural and rich interaction experience can be provided by combining the environmental information of the environment in which the user terminal is located. In this implementation, the interaction information is obtained by combining at least one of the temperature information, lighting information, and sound information with the visual positioning result of the user terminal, or the interaction information is obtained by combining at least one of the temperature information, lighting information, and sound information with the current time and the visual positioning result of the user terminal, thereby obtaining interaction information that better matches the environmental information of the environment in which the user terminal is located, thereby providing a more natural and rich interaction experience.

[0101] In a possible implementation, the acquiring of interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the visual positioning result, includes: in response to the current time belonging to a preset festival day, acquiring interaction information according to the preset festival day and the visual positioning result. In this implementation, the preset festival day may include at least one of a preset festival, a preset holiday, a preset activity day, etc. For example, the preset festival may include at least one of New Year's Day, Spring Festival, Lantern Festival, Children's Day, Dragon Boat Festival, Mid-Autumn Festival, National Day, Double Eleven, Christmas, etc. The preset activity day may be an activity day preset by a merchant. For example, the preset activity day may include a preset store anniversary day, a preset theme activity day, etc. In this implementation, the current time may include the current date. For example, if the current time includes the current date "October 1, 2020", and the preset festival day includes National Day, it can be determined that the current time belongs to the preset festival day. In this implementation, when the current time belongs to the preset holiday, the obtained interaction information can match the preset holiday and the visual positioning result of the user terminal. For example, on Double Eleven Day, the interaction information corresponding to the designated area of ​​shopping mall A includes AR red envelope rain. For another example, on Christmas Day, the interaction information corresponding to the designated area of ​​shopping mall B includes an AR image of Santa Claus. In this implementation, in response to the current time belonging to the preset holiday, the interaction information is obtained according to the preset holiday and the visual positioning result, thereby providing a richer AR scenario experience on the preset holiday.

[0102] In a possible implementation, the acquiring of interaction information based on the current time and one or both of the environmental information of the environment where the user terminal is located, and the visual positioning result, includes: generating an interaction information acquisition request based on the current time and one or both of the environmental information of the environment where the user terminal is located, and the visual positioning result; sending the interaction information acquisition request to a server, so that the server determines the interaction information corresponding to the current time and one or both of the environmental information of the environment where the user terminal is located, and the visual positioning result; and receiving the interaction information returned by the server.

[0103] As an example of this implementation, the user terminal may generate an interaction information acquisition request based on the current time and the visual positioning result of the user terminal, and send the interaction information acquisition request to the server, wherein the interaction information acquisition request includes the current time and the visual positioning result of the user terminal. In response to the interaction information acquisition request, the server may determine the interaction information corresponding to the current time and the visual positioning result of the user terminal, and return the interaction information to the user terminal.

[0104] As another example of this implementation, the user terminal may generate an interaction information acquisition request based on the environmental information of the environment where the user terminal is located and the visual positioning result of the user terminal, and send the interaction information acquisition request to the server, wherein the interaction information acquisition request includes the environmental information of the environment where the user terminal is located and the visual positioning result of the user terminal. In response to the interaction information acquisition request, the server may determine the interaction information corresponding to the environmental information of the environment where the user terminal is located and the visual positioning result of the user terminal, and return the interaction information to the user terminal.

[0105] As another example of this implementation, the user terminal may generate an interaction information acquisition request based on the current time, environmental information of the environment where the user terminal is located, and the visual positioning result of the user terminal, and send the interaction information acquisition request to the server, wherein the interaction information acquisition request includes the current time, environmental information of the environment where the user terminal is located, and the visual positioning result of the user terminal. The server may determine the interaction information corresponding to the current time, environmental information of the environment where the user terminal is located, and the visual positioning result of the user terminal in response to the interaction information acquisition request, and return the interaction information to the user terminal.

[0106] In this implementation, an interaction information acquisition request is generated based on the current time and one or both of the environmental information of the environment where the user terminal is located, and the visual positioning result, and the interaction information acquisition request is sent to the server, so that the server determines the interaction information corresponding to the current time and one or both of the environmental information of the environment where the user terminal is located, and the visual positioning result, and receives the interaction information returned by the server, thereby reducing the calculation amount of the user terminal, saving the power consumption of the user terminal, and obtaining accurate interaction information.

[0107] In another possible implementation, the acquiring of interaction information based on the current time and one or both of the environmental information of the environment where the user terminal is located, and the visual positioning result, includes: determining the interaction information based on the current time and one or both of the environmental information of the environment where the user terminal is located, and the visual positioning result. In this implementation, the user terminal may pre-store the correspondence between the interaction information and the time, the environmental information of the environment where the user terminal is located, and the visual positioning result, and may pre-store the interaction information, thereby locally determining the interaction information based on the current time and one or both of the environmental information of the environment where the user terminal is located, and the visual positioning result. In this implementation, the acquisition of interaction information can still be achieved when the user terminal is offline.

[0108] In a possible implementation, the interactive information includes operable interactive information. For example, the operable interactive information may include interactive information of indoor navigation, interactive game, real-time bullet screen, message wall, stranger social interaction, etc. In this implementation, by displaying the operable interactive information in the real scene image displayed on the display interface of the user terminal, the interactivity can be improved, thereby improving the user experience of AR interaction.

[0109] In a possible implementation, the interaction information includes inoperable interaction information. In this implementation, the inoperable interaction information may represent interaction information for display. For example, the inoperable interaction information may also be referred to as display-type interaction information. For example, the inoperable interaction information may include fireworks, mascots, advertisements, video displays, voice broadcasts, and the like. In this implementation, by displaying the inoperable interaction information in the real-life image displayed on the display interface of the user terminal, a richer scenario-based AR interaction experience can be provided.

[0110] In a possible implementation, displaying the interactive information in the real-scene image displayed on the display interface of the user terminal includes: displaying an interactive function option bar in the real-scene image displayed on the display interface of the user terminal; generating interactive function options according to the interactive information; and displaying the interactive function options in the interactive function option bar. In this implementation, in response to obtaining the interactive information, an interactive function option bar can be displayed in the real-scene image displayed on the display interface of the user terminal. The interactive function option bar can be used to display at least one interactive function option. In this implementation, according to the interactive information obtained, an interactive function option corresponding to the interactive information can be generated. The number of interactive function options can be the same as the number of items of the interactive information obtained. For example, in response to obtaining M items of interactive information, M interactive function options corresponding to the M items of interactive information can be generated, where M is an integer greater than or equal to 1. For example, the interactive information includes weather information of the target geographical area outside the parking lot and traffic information of the target geographical area, and the interactive function options may include a weather button corresponding to the weather information of the target geographical area and a road condition button corresponding to the traffic information of the target geographical area. In this implementation, by displaying an interactive function option bar in the real-life image displayed on the display interface of the user terminal, generating interactive function options according to the interactive information, and displaying the interactive function options in the interactive function option bar, the AR interactive scene corresponding to the interactive function option can be opened in response to the user triggering any one of the interactive function options in the interactive function option bar. For example, the interactive function option can be determined to be triggered in response to the user clicking on any one of the interactive function options in the interactive function option bar. In this implementation, the interactive function options in the interactive function option bar can be automatically updated as at least one of the visual positioning result of the user terminal, the current time, and the environmental information of the environment where the user terminal is located changes, so as to provide users with a more diverse and more scenario-based interactive experience based on augmented reality. As an example of this implementation, in response to the interaction information currently acquired being different from the interaction information acquired last time, the interaction function option bar may be updated according to the interaction information currently acquired.

[0111] In a possible implementation, the obtaining of interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located, and the visual positioning result, includes: determining the moving direction information of the user terminal according to the visual positioning result; obtaining the interaction information according to the current time and one or both of the environmental information of the environment where the user terminal is located, and the moving direction information of the user terminal. In this implementation, the moving direction information of the user terminal can be determined according to the visual positioning results of the user terminal at multiple time points. The moving direction information of the user terminal can be any information that can represent the moving direction of the user terminal. For example, the moving direction information of the user terminal can be from north to south, from west to east, entering a parking lot, leaving a parking lot, entering a shopping mall, leaving a shopping mall, and so on. In this implementation, the moving direction information of the user terminal is determined based on the visual positioning result, and the interaction information is obtained based on the current time and one or both of the environmental information of the environment where the user terminal is located, as well as the moving direction information of the user terminal. Thus, as the user moves, the user's intention can be predicted in combination with the current time and one or both of the environmental information of the environment where the user terminal is located, thereby providing the user with more personalized AR interaction information that is more practical for the user.

[0112] As an example of this implementation, the acquisition of interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located, and the moving direction information of the user terminal, includes: in response to the moving direction information including leaving the parking lot, acquiring information of the target geographical area outside the parking lot as the interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located. In this example, the target geographical area outside the parking lot may represent a geographical area outside the parking lot and related to the road section and / or the place that the user may reach after leaving the parking lot. The information of the target geographical area outside the parking lot may represent any information related to the target geographical area outside the parking lot. In one example, the target geographical area may include a geographical area around at least one exit of the parking lot. In another example, the target geographical area may include a geographical area between the parking lot and the current destination of the user, wherein the current destination represents the destination of the user after leaving the parking lot. In another example, the target geographical area may include the current destination of the user. For example, if the current time belongs to the commuting period or the peak period, and the moving direction information includes leaving the parking lot, the traffic situation information of the target geographical area may be acquired as the interaction information. For another example, if the current time is in the morning (e.g., 6:00-10:00), and the moving direction information includes leaving the parking lot, the weather information of the target geographical area can be obtained as the interaction information. In this example, in response to the moving direction information including leaving the parking lot, the information of the target geographical area outside the parking lot is obtained as the interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located, thereby helping the user to obtain the information of the target geographical area outside the parking lot in advance, thereby helping to save the user's time.

[0113] In one example, the information of the target geographical area includes one or both of the weather information of the target geographical area and the traffic information of the target geographical area. In this example, the traffic information of the target geographical area outside the parking lot may represent the information of the traffic conditions of the target geographical area outside the parking lot. That is, the traffic information of the target geographical area outside the parking space may reflect the traffic conditions of the target geographical area outside the parking lot. The traffic information may be displayed in the form of a map, text information, picture, audio and video, etc. For example, a map may include a 2D (2 Dimensions, two-dimensional) map and / or a 3D (3 Dimensions, three-dimensional) map. For example, the traffic information may include at least one of the following: congestion information, construction information, and accident information. For example, the target geographical area may include a geographical area around at least one exit of the parking lot. By displaying the traffic information of the geographical area around at least one exit of the parking lot (for example, including congestion information), the confusion of the user who does not know which exit is the best choice when driving out of the parking lot (for example, a parking lot of a commercial center or other parking lot) can be effectively solved. For another example, the target geographical area may include the current destination of the user, and by displaying the weather information of the current destination of the user, the user can be helped to understand the weather conditions of the destination in advance. In the above example, in response to the movement direction information including leaving the parking lot, the weather information and / or traffic information of the target geographical area outside the parking lot is obtained, and the weather information and / or traffic information of the target geographical area is displayed in the real-life image displayed on the display interface of the user terminal, thereby helping the user to obtain the weather information and / or traffic information of the target geographical area outside the parking lot in advance, thereby helping to save the user's time.

[0114] As another example of this implementation, the acquiring of interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located, and the moving direction information of the user terminal, includes: in response to the moving direction information including entering a shopping mall, acquiring information of a preset type of store in the shopping mall as the interaction information according to one or both of the environmental information of the current time and the environment where the user terminal is located. For example, if the current time is noon (e.g., 11:00-14:00), the moving direction information includes entering a shopping mall, then the information of the restaurant in the shopping mall can be acquired as the interaction information. In this example, the preset type of store can include a restaurant. For another example, if the current time is afternoon (e.g., 14:00-16:00), the moving direction information includes entering a shopping mall, then the information of the dessert shop in the shopping mall can be acquired as the interaction information. In this example, the preset type of store can include a dessert shop. In the given example, in response to the moving direction information including entering a shopping mall, information on preset types of stores in the shopping mall is obtained as the interaction information based on the current time and one or both of the environmental information of the environment where the user terminal is located. As a result, the user's intention can be predicted as the user moves, combined with the current time and one or both of the environmental information of the environment where the user terminal is located, thereby providing the user with more personalized AR interaction information that is more practical for the user.

[0115] The embodiments of the present disclosure can be applied to application scenarios such as AR guided tours, AR cultural tourism, and AR interactive games. The following describes the interactive method based on augmented reality provided by the embodiments of the present disclosure through several specific application scenarios.

[0116] Application scenario 1: A user walks along the river bank and raises a mobile phone (user terminal) to collect an environmental image. The mobile phone generates a visual positioning request based on the environmental image and sends the visual positioning request to the server. The server performs visual positioning based on the environmental image and the point cloud map around the river bank to obtain the visual positioning result of the mobile phone, and returns the visual positioning result of the mobile phone to the mobile phone. The mobile phone generates an interactive information acquisition request based on the current time and the visual positioning result of the mobile phone, and sends the interactive information acquisition request to the server, wherein the interactive information acquisition request includes the current time and the visual positioning result of the mobile phone, and the current time belongs to night. In response to the interactive information acquisition request, the server determines that the interactive information corresponding to the current time and the visual positioning result of the mobile phone includes AR fireworks show information, and returns the AR fireworks show information to the mobile phone. The mobile phone receives the AR fireworks show information returned by the server, and generates a fireworks button (i.e., an interactive function option corresponding to the AR fireworks show information) based on the AR fireworks show information. The mobile phone displays an interactive function option bar in the real scene image displayed on the display interface, and displays the fireworks button in the interactive function option bar. In response to the fireworks button being clicked, the mobile phone displays the visual effects of the AR fireworks show in the real-scene image displayed on the display interface according to the AR fireworks show information.

[0117] Application scenario 2: The user walks to the central square of the shopping mall and raises the mobile phone to collect environmental images. The mobile phone generates a visual positioning request based on the environmental image and sends the visual positioning request to the server. The server performs visual positioning based on the environmental image and the point cloud map of the shopping mall to obtain the visual positioning result of the mobile phone, and returns the visual positioning result of the mobile phone to the mobile phone. The mobile phone generates an interactive information acquisition request based on the current time and the visual positioning result of the mobile phone, and sends the interactive information acquisition request to the server, wherein the interactive information acquisition request includes the current time and the visual positioning result of the mobile phone, and the current time includes the current date "November 11, 2020". In response to the interactive information acquisition request, the server determines that the interactive information corresponding to the current time and the visual positioning result of the mobile phone includes AR red envelope rain information, and returns the AR red envelope rain information to the mobile phone. The mobile phone receives the AR red envelope rain information returned by the server, and generates an interactive game button (i.e., an interactive function option corresponding to the AR red envelope rain information) based on the AR red envelope rain information. The mobile phone displays an interactive function option bar in the real-life image displayed on the display interface, and displays the interactive game button in the interactive function option bar. In response to the interactive game button being clicked, the mobile phone displays AR red envelope rain in the real-life image displayed on the display interface according to the AR red envelope rain information. For example, AR red envelope rain can be displayed every half an hour. At the same time, the mobile phone can display a dart button in the real-life image displayed on the display interface according to the AR red envelope rain information. In response to the dart button being clicked, the mobile phone can display the visual effect of a dart hitting a red envelope in the real-life image displayed on the display interface, and can distribute coupons, red envelopes, etc. to the user's account in response to the dart hitting the red envelope.

[0118] Application scenario three: The user is in an amusement park and raises his mobile phone to collect an environmental image. The mobile phone generates a visual positioning request based on the environmental image and sends the visual positioning request to the server. The server performs visual positioning based on the environmental image and the point cloud map of the amusement park to obtain the visual positioning result of the mobile phone and returns the visual positioning result of the mobile phone to the mobile phone. The mobile phone generates an interactive information acquisition request based on the current time and the visual positioning result of the mobile phone and sends the interactive information acquisition request to the server, wherein the interactive information acquisition request includes the current time and the visual positioning result of the mobile phone, and the current time includes the current date "June 1, 2020". In response to the interactive information acquisition request, the server determines that the current time belongs to Children's Day and the user's current location belongs to a specific area of ​​the amusement park, thereby determining that the interactive information corresponding to the current time and the visual positioning result of the mobile phone includes AR mascot information, and returns the AR mascot information to the mobile phone. The mobile phone receives the AR mascot information returned by the server and generates a mascot button (i.e., an interactive function option corresponding to the AR mascot information) based on the AR mascot information. The mobile phone displays an interactive function option bar in the real-life image displayed on the display interface, and displays the mascot button in the interactive function option bar. In response to the mascot button being clicked, the mobile phone can display an AR mascot in the real-life image displayed on the display interface according to the AR mascot information. Alternatively, in response to the mascot button being clicked, the mobile phone can display an AR mascot in the real-life image displayed on the display interface according to the AR mascot information, and the AR mascot can provide activity guidance.

[0119] Application scenario 4: The user walks into a pedestrian street and raises the mobile phone to collect the environment image. The mobile phone generates a visual positioning request based on the environment image and sends the visual positioning request to the server. The server performs visual positioning based on the environment image and the point cloud map of the shopping mall to obtain the visual positioning result of the mobile phone, and returns the visual positioning result of the mobile phone to the mobile phone. The mobile phone generates an interactive information acquisition request based on the current time and the visual positioning result of the mobile phone, and sends the interactive information acquisition request to the server, wherein the interactive information acquisition request includes the current time and the visual positioning result of the mobile phone. In response to the interactive information acquisition request, the server determines that the current time belongs to the Lantern Festival and the user's current location belongs to a specific area of ​​the pedestrian street, thereby determining that the interactive information corresponding to the current time and the visual positioning result of the mobile phone includes AR Lantern Festival corridor information, and returns the AR Lantern Festival corridor information to the mobile phone. The mobile phone receives the AR Lantern Festival corridor information returned by the server, and generates a Lantern Festival button (i.e., an interactive function option corresponding to the AR Lantern Festival corridor information) based on the AR Lantern Festival corridor information. The mobile phone displays an interactive function option bar in the real scene image displayed on the display interface, and displays the Lantern Festival button in the interactive function option bar. In response to the Lantern Festival button being clicked, the mobile phone can display an AR Lantern Festival corridor in the real scene image displayed on the display interface according to the AR Lantern Festival corridor information. The AR Lantern Festival corridor can include visual effects such as lanterns, and can also include an interactive game of guessing lantern riddles.

[0120] Application scenario five: In the morning, the user is in the car, and the user's mobile phone is fixed on the car mobile phone holder. The mobile phone collects environmental images, generates a visual positioning request based on the environmental images, and sends the visual positioning request to the server. The server performs visual positioning based on the environmental images and the point cloud map of the parking lot, obtains the visual positioning result of the mobile phone, and returns the visual positioning result of the mobile phone to the mobile phone. The mobile phone can determine the moving direction information of the mobile phone based on the visual positioning results of the mobile phone at multiple time points. Alternatively, the server can determine the moving direction information of the mobile phone based on the visual positioning results of the mobile phone at multiple time points. The moving direction information of the mobile phone includes leaving the parking lot. The mobile phone can generate an interactive information acquisition request based on the current time and the moving direction information of the mobile phone, and send the interactive information acquisition request to the server, wherein the interactive information acquisition request includes the current time and the moving direction information of the mobile phone, and the current time belongs to the morning. In response to the interactive information acquisition request, the server determines that the interactive information corresponding to the current time and the moving direction information of the mobile phone includes AR information of the target geographical area outside the parking lot, and returns the AR information of the target geographical area to the mobile phone. The mobile phone receives the AR information of the target geographical area returned by the server, and generates a weather button and a traffic button (i.e., the interactive function options corresponding to the AR information of the target geographical area) according to the AR information of the target geographical area. The mobile phone displays an interactive function option bar in the real-life image displayed on the display interface, and displays the weather button and the traffic button in the interactive function option bar. In response to the weather button being clicked, the mobile phone can broadcast the weather information of the target geographical area through the AR anchor voice in the real-life image displayed on the display interface according to the AR information of the target geographical area. Alternatively, in response to the traffic condition button being clicked, the mobile phone can display the traffic condition information of the target geographical area in the real-life image displayed on the display interface according to the AR information of the target geographical area.

[0121] Application scenario six: A user walks into a shopping mall at noon and raises a mobile phone to collect an environmental image. The mobile phone generates a visual positioning request based on the environmental image and sends the visual positioning request to the server. The server performs visual positioning based on the environmental image and the point cloud map of the shopping mall, obtains the visual positioning result of the mobile phone, and returns the visual positioning result of the mobile phone to the mobile phone. The visual positioning result of the mobile phone is obtained and returned to the mobile phone. The mobile phone can determine the moving direction information of the mobile phone based on the visual positioning results of the mobile phone at multiple time points. Alternatively, the server can determine the moving direction information of the mobile phone based on the visual positioning results of the mobile phone at multiple time points. The moving direction information of the mobile phone includes entering the shopping mall. The mobile phone can generate an interactive information acquisition request based on the current time and the moving direction information of the mobile phone, and send the interactive information acquisition request to the server, wherein the interactive information acquisition request includes the current time and the moving direction information of the mobile phone, and the current time is noon. In response to the interactive information acquisition request, the server determines that the interactive information corresponding to the current time and the moving direction information of the mobile phone includes the AR information of the restaurant in the mall, and returns the AR information of the restaurant in the mall to the mobile phone. The mobile phone receives the AR information of the restaurant in the mall returned by the server, and generates a restaurant button (i.e., an interactive function option corresponding to the AR information of the restaurant in the mall) according to the AR information of the restaurant in the mall. The mobile phone displays an interactive function option bar in the real-life image displayed on the display interface, and displays the restaurant button in the interactive function option bar. In response to the restaurant button being clicked, the mobile phone can display the AR information of the restaurant in the mall in the real-life image displayed on the display interface according to the AR information of the restaurant in the mall. The mobile phone can also display AR navigation information from the user's current location to the restaurant in response to the icon of any restaurant being triggered.

[0122] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0123] In addition, the present disclosure also provides an augmented reality-based interaction device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any augmented reality-based interaction method provided by the present disclosure. The corresponding technical solutions and technical effects can be found in the corresponding records in the method part and will not be repeated here.

[0124] Figure 3FIG. 1 is a block diagram of an augmented reality-based interactive device provided by an embodiment of the present disclosure. The augmented reality-based interactive device is applied to a user terminal. Figure 3 As shown, the interactive device based on augmented reality includes:

[0125] A collection module 31, used for collecting environmental images;

[0126] A visual positioning module 32, used to obtain a visual positioning result of the user terminal according to the environment image;

[0127] An acquisition module 33, configured to acquire interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the visual positioning result;

[0128] The display module 34 is used to display the interaction information in the real scene image displayed on the display interface of the user terminal.

[0129] In a possible implementation, the visual positioning module 32 is used to:

[0130] Generate a visual positioning request according to the environment image;

[0131] Sending the visual positioning request to the server, so that the server performs visual positioning according to the environment image and a preset point cloud map to obtain a visual positioning result of the user terminal;

[0132] Receive the visual positioning result of the user terminal returned by the server.

[0133] In a possible implementation manner, the environmental information includes at least one of temperature information, light information, and sound information.

[0134] In a possible implementation, the acquisition module 33 is used to:

[0135] In response to the current time belonging to a preset festival day, interaction information is acquired according to the preset festival day and the visual positioning result.

[0136] In a possible implementation, the acquisition module 33 is used to:

[0137] Generate an interactive information acquisition request according to one or both of the current time and the environmental information of the environment where the user terminal is located, and the visual positioning result;

[0138] Sending the interaction information acquisition request to the server, so that the server determines the interaction information corresponding to one or both of the current time and the environmental information of the environment where the user terminal is located and the visual positioning result;

[0139] Receive the interaction information returned by the server.

[0140] In a possible implementation manner, the interaction information includes operable interaction information and / or inoperable interaction information.

[0141] In a possible implementation, the display module 34 is used to:

[0142] In the real-scene image displayed on the display interface of the user terminal, an interactive function option bar is displayed;

[0143] generating interactive function options according to the interactive information;

[0144] The interactive function options are displayed in the interactive function option bar.

[0145] In a possible implementation, the acquisition module 33 is used to:

[0146] Determining the moving direction information of the user terminal according to the visual positioning result;

[0147] The interaction information is acquired according to one or both of the current time and the environment information of the environment where the user terminal is located, and the moving direction information of the user terminal.

[0148] In a possible implementation, the acquisition module 33 is used to:

[0149] In response to the moving direction information including leaving the parking lot, information of a target geographical area outside the parking lot is acquired as the interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located.

[0150] In a possible implementation manner, the information of the target geographical area includes one or both of weather information of the target geographical area and traffic condition information of the target geographical area.

[0151] In a possible implementation, the acquisition module 33 is used to:

[0152] In response to the moving direction information including entering a shopping mall, information of a preset type of store in the shopping mall is obtained as the interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located.

[0153] In an embodiment of the present disclosure, an environmental image is captured, and a visual positioning result of the user terminal is obtained based on the environmental image. Interaction information is obtained based on one or both of the current time and the environmental information of the environment in which the user terminal is located, as well as the visual positioning result. The interaction information is displayed in a real-scene image displayed on a display interface of the user terminal. Thus, the interaction information can be automatically transformed in combination with the current time and / or the environmental information of the environment in which the user terminal is located, together with the visual positioning result of the user terminal, thereby providing users with a more diverse and scenario-based interactive experience based on augmented reality.

[0154] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. Its specific implementation and technical effects can refer to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0155] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.

[0156] The embodiment of the present disclosure further provides a computer program, comprising a computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device executes the above method.

[0157] The embodiments of the present disclosure also provide a computer program product for storing computer-readable instructions, which, when executed, enable a computer to perform the operations of the augmented reality-based interaction method provided in any of the above embodiments.

[0158] An embodiment of the present disclosure also provides an electronic device, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.

[0159] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0160] The present disclosure relates to the field of augmented reality, which obtains image information of target objects in a real environment, and then uses various vision-related algorithms to detect or identify the relevant features, states and attributes of the target objects, so as to obtain an AR effect that combines virtuality and reality and matches the specific application. Exemplarily, the target object may involve faces, limbs, gestures, actions, etc. related to the human body, or markers and markers related to objects, or sand tables, display areas or display items related to venues or places. Vision-related algorithms may involve visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, and position or depth detection of objects. Specific applications may not only involve interactive scenes such as guided tours, navigation, explanations, reconstructions, virtual effect overlay displays, etc. related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, body beautification, special effects display, virtual model display and other interactive scenes.

[0161] The convolutional neural network can be used to detect or identify the relevant features, states and attributes of the target object. The above convolutional neural network is a network model obtained by model training based on a deep learning framework.

[0162] Figure 4 A block diagram of an electronic device 800 provided by an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0163] Reference Figure 4 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0164] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0165] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0166] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0167] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0168] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0169] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0170] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0171] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (Wi-Fi), a second generation mobile communication technology (2G), a third generation mobile communication technology (3G), a fourth generation mobile communication technology (4G) / long term evolution (LTE) of universal mobile communication technology, a fifth generation mobile communication technology (5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0172] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0173] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0174] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0175] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0176] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0177] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0178] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0179] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0180] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0181] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0182] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0183] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An interactive method based on augmented reality, characterized in that: Applied to a user terminal, the method comprises: Collect environmental images; Obtaining a visual positioning result of the user terminal according to the environment image; Acquire interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the visual positioning result; Displaying the interaction information in a real-scene image displayed on a display interface of the user terminal; The acquiring of the interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located, and the visual positioning result, includes: determining the moving direction information of the user terminal according to the visual positioning results of the user terminal at multiple time points; acquiring the interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located, and the moving direction information of the user terminal; The obtaining of interaction information based on the current time, one or both of the environmental information of the environment in which the user terminal is located, and the moving direction information of the user terminal includes: in response to the moving direction information including entering a shopping mall, obtaining information of a preset type of store in the shopping mall as the interaction information based on the current time.

2. The method according to claim 1, characterized in that The obtaining, according to the environment image, a visual positioning result of the user terminal includes: Generate a visual positioning request according to the environment image; Sending the visual positioning request to the server, so that the server performs visual positioning according to the environment image and a preset point cloud map to obtain a visual positioning result of the user terminal; Receive the visual positioning result of the user terminal returned by the server.

3. The method according to claim 1, characterized in that The environmental information includes at least one of temperature information, light information and sound information.

4. The method according to any one of claims 1 to 3, characterized in that The acquiring the interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the visual positioning result, includes: In response to the current time belonging to a preset festival day, interaction information is acquired according to the preset festival day and the visual positioning result.

5. The method according to any one of claims 1 to 3, characterized in that: The acquiring the interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the visual positioning result, includes: Generate an interactive information acquisition request according to one or both of the current time and the environmental information of the environment where the user terminal is located, and the visual positioning result; Sending the interaction information acquisition request to the server, so that the server determines the interaction information corresponding to one or both of the current time and the environmental information of the environment where the user terminal is located and the visual positioning result; Receive the interaction information returned by the server.

6. The method according to any one of claims 1 to 3, characterized in that: The interaction information includes operable interaction information and / or inoperable interaction information.

7. The method according to any one of claims 1 to 3, characterized in that The displaying of the interactive information in the real scene image displayed on the display interface of the user terminal includes: In the real-scene image displayed on the display interface of the user terminal, an interactive function option bar is displayed; generating interactive function options according to the interactive information; The interactive function options are displayed in the interactive function option bar.

8. The method according to any one of claims 1 to 3, characterized in that The acquiring the interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the moving direction information of the user terminal, includes: In response to the moving direction information including leaving the parking lot, information of a target geographical area outside the parking lot is acquired as the interaction information according to one or both of the current time and the environmental information of the environment where the user terminal is located.

9. The method according to claim 8, characterized in that The information of the target geographical area includes one or both of weather information of the target geographical area and traffic condition information of the target geographical area.

10. An interactive device based on augmented reality, characterized in that: Applied to user terminals, including: A collection module, used for collecting environmental images; A visual positioning module, used to obtain a visual positioning result of the user terminal according to the environment image; An acquisition module, configured to acquire interaction information according to one or both of the current time and the environment information of the environment where the user terminal is located, and the visual positioning result; A display module, used to display the interactive information in a real scene image displayed on a display interface of the user terminal; Among them, the acquisition module is specifically used to: determine the moving direction information of the user terminal based on the visual positioning results of the user terminal at multiple time points; in response to the moving direction information including entering a shopping mall, obtain the information of a preset type of store in the shopping mall according to the current time as the interaction information.

11. An electronic device, characterized in that: include: one or more processors; a memory for storing executable instructions; The one or more processors are configured to call executable instructions stored in the memory to execute the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images