Information display method, device and electronic equipment

Abstract

Embodiments of the present application disclose information display methods and apparatuses and electronic devices, including: in response to a request for displaying a target object in an augmented reality (AR) mode, loading a three-dimensional model corresponding to the target object, and mapping the three-dimensional model to an AR space for display, the target object being a handheld device type object with a display screen; detecting an interaction operation performed by a user on the three-dimensional model in the AR space, the interaction operation being used to simulate an operation related to starting a target function in the target object; and displaying interface content corresponding to the target function in a region of the three-dimensional model where the display screen is located according to the detected interaction operation. Through the embodiments of the present application, a user can be provided with an experience closer to that in an offline purchase or trial process of a real product, which is conducive to helping the user make a purchase decision.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular to information display methods, devices and electronic devices. Background Technology

[0002] In product information service systems, images, videos, and live streams are common methods for describing products. Users can obtain information about the product's features through these descriptions, thus helping them make purchasing decisions. In recent years, solutions have emerged that use 3D models to display product information. This involves reconstructing the product in 3D, presenting a dynamic 3D effect to the user on the product information service system's client side, and enabling user interaction with the product. For example, users can trigger the product to rotate by swiping the screen, allowing them to view the product's appearance from multiple perspectives.

[0003] However, even when products are displayed using 3D models, the information available to users remains relatively limited. Therefore, how to provide users with more detailed information about specific products and create an experience closer to that of purchasing goods offline has become a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] This application provides information display methods, devices, and electronic devices that can provide users with an experience closer to that of purchasing or trying out physical goods offline, thus helping users make purchasing decisions.

[0005] This application provides the following solution:

[0006] An information display method, comprising:

[0007] In response to a request to display a target object via augmented reality (AR), a 3D model corresponding to the target object is loaded, and the 3D model is mapped into AR space for display; the target object is a handheld device with a display screen.

[0008] The interactive operations performed by the user on the 3D model through the AR space are detected, and the interactive operations are used to simulate the operations related to activating the target function in the target object;

[0009] Based on the detected interactive operations, the interface content corresponding to the target function is displayed in the area where the display screen is located in the 3D model.

[0010] The detection of user interaction with the 3D model through the AR space includes:

[0011] Detecting the user's gestures performed on the display screen of the target object within the AR space;

[0012] The area where the display screen in the 3D model is located displays the interface content corresponding to the target function, including:

[0013] In the area where the display screen is located in the three-dimensional model, the interface content corresponding to the target function activated by the gesture operation is displayed.

[0014] The target functions include one of the following: photo taking function, flashlight function, intelligent voice assistant function, or music control function.

[0015] This also includes:

[0016] During the process of displaying the interface content corresponding to the target function through the display screen area in the three-dimensional model, materials related to the target function are loaded and displayed or played through the AR space.

[0017] The method further includes:

[0018] The interface of the AR space provides prompts regarding how to perform the interactive operations.

[0019] The step of mapping the 3D model into AR space for display includes:

[0020] Identify the pose information of hand images from a stream of captured real-world images;

[0021] Based on the pose information of the identified hand image, the 3D model is mapped to the location of the hand image in AR space for display.

[0022] The step of mapping the 3D model to the location of the hand image in AR space based on the pose information of the recognized hand image for display includes:

[0023] When the hand image is identified as being in a half-clenched position with the palm facing upwards, the 3D model is mapped face up onto the location of the hand image in the AR space.

[0024] This also includes:

[0025] When the hand image changes from a half-clenched posture to a clenched fist posture, and continues to change to a target posture that meets the flipping conditions, the 3D model is flipped so that the back is facing up, and mapped into the AR space for display.

[0026] Wherein, the three-dimensional model has the same size as the target object, and the method further includes:

[0027] The size information of the three-dimensional model is displayed in the AR space.

[0028] This also includes:

[0029] If the area of ​​the 3D model is larger than the area of ​​the hand image, then a cue message is provided at the edge of the 3D model in the AR space.

[0030] This also includes:

[0031] The AR space interface provides an object list area, which is used to display at least one other similar object related to the target object, and its corresponding AR display entry point;

[0032] After receiving an AR display request for other similar objects through the AR display entry point, the 3D model displayed in the AR space is replaced with the 3D model corresponding to the other similar objects.

[0033] An object information display device, comprising:

[0034] The AR display unit is used to respond to a request to display a target object through augmented reality (AR), load the 3D model corresponding to the target object, and map the 3D model into AR space for display; the target object is a handheld device with a display screen.

[0035] An interactive behavior detection unit is used to detect the interactive operations performed by the user on the three-dimensional model through the AR space. The interactive operations are used to simulate the operations related to activating the target function in the target object.

[0036] The interface content display unit is used to display the interface content corresponding to the target function in the area of ​​the display screen in the three-dimensional model according to the detected interactive operation.

[0037] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the preceding methods.

[0038] An electronic device, comprising:

[0039] One or more processors; and

[0040] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the preceding descriptions.

[0041] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0042] Through the embodiments of this application, for handheld devices with displays, during AR display, the interactive operations performed by the user on the 3D model through the AR space can be detected. These interactive operations can be used to simulate operations defined in the target object related to activating a target function. Subsequently, based on the detected interactive operations, the interface content corresponding to the target function can be displayed in the area where the display screen is located in the 3D model. This allows users to experience or test the functions of specific handheld devices while browsing them via AR, thus obtaining an experience closer to that of purchasing or trying out a physical product offline. In product display scenarios, this helps users make purchasing decisions.

[0043] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of the system architecture provided in the embodiments of this application;

[0046] Figure 2 This is a flowchart of the method provided in the embodiments of this application;

[0047] Figure 3 This is a schematic diagram of the interface provided in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of the device provided in the embodiments of this application;

[0049] Figure 5 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0051] First, it's important to note that to enhance the realism of product and other target object information, one approach is to combine 3D product reconstruction technology with AR (Augmented Reality) technology for product display. In this solution, the product is pre-reconstructed in 3D. When displayed on the client side, a real-time image stream from the real world is captured using the terminal device's camera. Planes such as "tables" and "floors" within the image stream are located, and the product's 3D model is projected onto the positions of these planes in the real-world image stream, thus presenting the product as if it were actually "placed" in a real-world environment.

[0052] The above-mentioned solution of displaying product information using AR technology allows users to more intuitively judge whether a specific product is suitable for their real-world environment. For example, whether a sofa or other home furnishing product is suitable for placing in their living room, etc., thereby helping users make better purchasing decisions.

[0053] However, in current technologies, when displaying product information using AR, it's usually just a matter of mechanically placing the product's 3D model into the AR space. While the viewing angle can be changed by swiping the screen or rotating the phone to achieve a 360-degree view of the product's appearance, and this is generally sufficient to help users make purchasing decisions for items like sofas and coffee tables, this simple and mechanical AR placement is often inadequate for other product categories.

[0054] For example, when purchasing handheld devices such as smartphones offline, users may not only need to check the appearance of the product, but also test or experience its various functions. This includes checking the screen resolution and trying out features like the smart voice assistant and music control. Additionally, they may need to experience the phone in their hand to determine if the size suits them, and so on.

[0055] Therefore, in this embodiment of the application, for handheld devices such as mobile phones, during the AR-based display process, various interactive operations can be performed on the specific 3D model, such as lighting up the screen and triggering specific function options. Correspondingly, the display screen portion of the 3D model displayed in the AR space can show the interface content triggered by the specific interactive operations, etc. In other words, users can experience or test the functions of handheld devices such as mobile phones through the AR space. In addition, optionally, a function can also be provided to "hands-on" experience the 3D model of the specific product, that is, to allow users to experience the handling effect of the mobile phone or other product they want to purchase based on the AR space.

[0056] From a system architecture perspective, see Figure 1 The solutions provided in this application can involve both a client and a server in a product information service system. The server primarily provides a specific 3D model of the product. In this application embodiment, the 3D model of the product can be generated by 3D reconstruction based on the product's actual dimensions. To enable users to experience the various functions of the product, it can also provide materials, including images, audio, and video, for display or playback in AR space. The client primarily supports AR display. To enhance the realism of the display effect, a high-performance AR rendering engine can be used to support the rendering of the 3D model in AR space. It can also perform scaling operations. Furthermore, triggered by user interaction, it can map specific images and other materials onto the display area of ​​the specific 3D model in AR space for display, and so on.

[0057] The specific implementation schemes provided in the embodiments of this application will be described in detail below.

[0058] First, this application provides a method for displaying product information, see [link to relevant documentation]. Figure 2 The method may include:

[0059] S201: In response to a request to display a target object via augmented reality (AR), load the 3D model corresponding to the target object and map the 3D model into AR space for display; the target object is a handheld device with a display screen.

[0060] In this embodiment, the specific target object can be a commodity, or it can be other items, such as exhibits in a museum, etc. In the scenario of commodity display, the specific target commodity can be a handheld device with a display screen, such as a mobile communication device like a mobile phone. Specifically, an entry point for displaying the commodity via AR can be provided on pages such as the commodity details page. For example, the access point can be provided in the main image display area of ​​the commodity details page, etc. Thus, users can initiate a request to display the commodity via AR through this access point while accessing the commodity details page. Of course, in specific implementations, the above request can also be initiated in other ways. For example, commodities with configured 3D models can be aggregated into the same theme page, and an entry point for displaying each commodity via AR can be provided on the theme page. In this way, users can directly initiate a request to display a specific target commodity via AR through this theme page, etc. In this embodiment, the interaction methods in the specific AR display process are mainly described using commodities as an example.

[0061] Upon receiving a specific AR display request, an AR space can be created, and a 3D model of the target product can be loaded. This 3D model is then rendered into the AR space for display. Specifically, when creating the AR space, components such as the camera on the terminal device can be activated first to capture images of the real world. Optionally, the user can be prompted to extend their hand in front of the camera, and prompts can be provided regarding hand placement, for example... Figure 3 As shown in (A), the user can be prompted to have their palm facing upwards, in a semi-clenched position, etc. In this way, the client can detect hand images in the captured real-world image stream, and after detecting the hand image, it can also determine information such as the hand's pose (i.e., position and posture).

[0062] Specifically, in determining the hand pose, a real-time 3D reconstruction of the hand can be performed based on the hand image identified from the real-world image stream to obtain the coordinates of multiple 3D hand key points. The hand pose can then be determined based on these key point coordinates. Regarding the recognition of the real hand image in the real-world image, existing algorithm models can be used. Furthermore, after recognizing the real hand image, a pre-trained deep learning algorithm model can be used to perform 3D mesh reconstruction of the real hand image. The algorithm models for hand image recognition and 3D mesh reconstruction of the hand are not the focus of this application's embodiments and therefore will not be detailed here.

[0063] After recognizing the pose information of a specific hand image, the 3D model of the specific product can be mapped to the location of the hand image in AR space for display. Because the hand pose is recognized, the effect of a hand grasping a specific 3D model can be presented in AR space. Furthermore, when the hand moves or changes its posture, the display position and posture of the 3D model in AR space can also change accordingly.

[0064] In practical implementation, considering that when users hold physical devices like phones in the real world, parts of the device may be obscured by their fingertips, simply displaying the 3D model at the location of the hand image in AR space might result in the 3D model appearing to float above the palm, making the user experience unrealistic. Therefore, to enhance the realism of the effect presented in AR space, a standard pose 3D hand model (referred to as the basic hand model for ease of description) can be pre-created for the target product's 3D model. By default, this basic hand model can hold the target product's 3D model in a standard pose. In other words, the target product's 3D model can be held by a basic hand model, and the relative positional relationship between the target product's 3D model and the basic hand model can remain fixed. Thus, the product's 3D model can move with the movement of the basic hand model. Of course, this basic hand model does not need to be displayed; therefore, it can be invisible to the user.

[0065] Given the existence of a base hand model, the estimation result of the user's actual hand pose can include: the rotation matrix and / or translation vector of the actual hand pose relative to the base hand model. That is, during real-time 3D reconstruction of the user's actual hand image, the specific 3D space can be aligned with the 3D space used when establishing the base hand model. Thus, after recognizing the hand image from the current AR space's real-world image stream, the pose information of that hand image can be expressed using the rotation matrix and / or translation vector relative to the base hand model. Subsequently, by applying the estimated rotation matrix and / or translation vector to the base hand model, the 3D model of the target product can be projected onto the real-world image, and effects such as fingers covering the 3D model and the 3D model rotating with the hand can be displayed.

[0066] In this embodiment, the 3D model of the specific product can be created at a 1:1 scale with the actual product. This, combined with hand image recognition and positioning, allows users to intuitively feel the size of the product and determine if it suits them. Alternatively, the dimensions of the 3D model, including length, width, and thickness, can be marked in AR space. For example, the specific display effect can be as follows: Figure 3 As shown in (B).

[0067] Additionally, it can be determined whether the area of ​​a specific product's 3D model exceeds the user's palm area, allowing for prompting the user. For example, it could indicate that the product is too large and may not be suitable for the current user. There are various ways to provide this prompt; for instance, adding color or patterns to the edges of the 3D model to indicate that the current device exceeds the user's palm area, and so on.

[0068] In practical implementation, the 3D model of the product can be displayed in different states in AR space based on changes in the user's hand posture. For example, when the hand image is identified as a half-clenched posture with the palm facing upwards, the 3D model can be mapped face-up (i.e., facing the user's current device camera) to the location of the hand image in AR space. Of course, during implementation, the realism of the display effect can be enhanced based on the aforementioned basic hand model.

[0069] Additionally, since users may also need to view the color and texture of the back of their mobile phones or other devices, this embodiment also supports users flipping the 3D model to its back for display. Specifically, since the 3D model can follow hand movements, theoretically, if the user's palm was facing up and the 3D model in the AR space was being displayed face up, the user can simply flip their hand 180 degrees to have the back of their hand facing up, thus changing the display so the 3D model is displayed face down. However, in practice, since the user needs to hold their mobile phone or other terminal device in one hand and hold the other hand in front of the camera for experiencing the AR space, the action of flipping 180 degrees with one hand may be somewhat stiff. Furthermore, after flipping, because the hand is on top and the 3D model is below, a large portion of the 3D model will be obscured by the hand image.

[0070] Therefore, in this embodiment, a more convenient way to view the back of a 3D model can be provided. Specifically, if the user was previously in a half-clenched position with their palm facing up, and the 3D model was displayed face up at the location of the hand, if the user wants to view the back of the 3D model, they can first change to a fist-clenched position. At this time, the 3D model can be temporarily not displayed. Then, the user can make a predetermined gesture, such as the back of the hand facing the camera, or making a victory sign, etc. At this time, the back of the 3D model can be displayed in the AR space. That is to say, when the posture of the hand image changes from the half-clenched posture to the fist-clenched posture, and continues to change to a target posture that meets the flipping conditions, the 3D model can be flipped to a back-facing state and mapped into the AR space for display. In this way, it is more convenient for users to perform interactive operations, and at the same time, they can view the entire back of the 3D model more completely.

[0071] S202: Detect the interactive operations performed by the user on the 3D model through the AR space, the interactive operations being used to simulate operations related to activating the target function in the target object.

[0072] S203: Based on the detected interactive operation, display the interface content corresponding to the target function in the area where the display screen is located in the three-dimensional model.

[0073] Based on displaying a 3D model of a specific product using AR space, the interactive operations performed by the user on the 3D model through AR space can be detected. Furthermore, based on the detected interactive operations, the corresponding interface content can be displayed in the area where the display screen is located within the 3D model. In other words, when using handheld devices, users primarily utilize the various functions offered by the product through its display screen. Therefore, when selecting a specific product, in addition to determining whether the size and appearance meet requirements, users may need to power on the device and turn on the display screen to experience the specific functions. For this purpose, in this embodiment, AR space can also be used to simulate this experience for the user.

[0074] The specific interactive operations can be varied. For example, the power button area on the 3D model can simulate pressing the power button, which would turn on the 3D model's display screen and play a startup animation, etc. Alternatively, if the product supports waking up the display screen by double-tapping, the user can simulate this double-tapping operation on the 3D model's display screen area, which would also turn on the display screen, and so on.

[0075] In addition, specific interactive operations can also be gesture operations performed on the display screen in the AR space, so as to display the interface content triggered by the gesture operation in the area where the display screen is located in the 3D model. For example, specific gesture operations can be used to switch the display screen to a lit state, or they can also include gesture operations to activate the target function of the target product.

[0076] For example, some mobile phone products support activating certain functions by swiping on the display screen when the screen is off. Therefore, the solution in this application embodiment allows users to experience such gestures and the activated functions through AR space. Specific function options may include camera, flashlight, smart voice assistant, or music control, etc. Different function options can be activated by different gestures. For example, drawing an "O" gesture can activate the camera, drawing a "V" gesture can activate the flashlight, and so on. After detecting such a gesture in AR space, the corresponding target function option can be determined based on the specific gesture type, and then the corresponding interface content after the target function option is activated can be displayed on the display area of ​​the 3D model.

[0077] In order to further enhance the user experience, while displaying the interface content after the target function is activated through the display screen area in the 3D model, materials related to the target function can also be loaded and displayed or played through the AR space.

[0078] For example, if the target function is to take a photo, a photo-taking interface can be displayed in the area where the screen of the 3D model is located. This interface can also include a photo-taking control, which the user can further simulate clicking. At this point, a specific photo can be displayed in the area where the screen of the 3D model is located. This photo can be a pre-acquired photo related to the current product; after the user simulates clicking the photo control, the photo is loaded and mapped into the current AR space. Furthermore, the pose of the photo can be adjusted according to the pose of the current 3D model. For example, if the current 3D model is tilted at a certain angle, the photo can also be tilted at a corresponding angle to enhance the realism of the display.

[0079] Alternatively, after the user simulates clicking the camera control, specific image content can be extracted from the real-world image stream captured by the current terminal device. This image content is then processed based on the parameters (e.g., pixels) of the camera component corresponding to the target product to generate a target image. This target image is then mapped onto the display area within the 3D model for display. In other words, this method better simulates the process of taking a photo using a 3D model in AR space, thereby enhancing the user experience.

[0080] If the flashlight function is specifically activated by a gesture, the interface content after the flashlight function is activated can be displayed in the display area of ​​the 3D model. In addition, light source materials can be added to the AR space to simulate the effect of light emitted through the camera component of the 3D model.

[0081] If the smart voice assistant function is specifically activated by gesture, the interface content after the smart voice assistant function is activated can be displayed in the display area of ​​the screen in the 3D model. In addition, the corresponding voice dialogue material can also be played.

[0082] If the music control function is specifically activated by the gesture, the interface content triggered after the music control function is activated can be displayed in the area of ​​the display screen in the 3D model. At the same time, the corresponding music can also be played, and so on.

[0083] The interface content displayed on the screen area in the AR space after various functions are activated can be achieved by loading separately saved image materials, or it can be implemented directly in the 3D model.

[0084] It's important to note that when displaying a 3D model using AR, one hand (e.g., right hand) might be used to hold the terminal device, while the other hand (e.g., left hand) is used for interaction in front of the camera. In this state, if a user performs various interactive operations on the 3D model's display screen with only one hand while maintaining a specific hand posture, it can be inconvenient. For example, maintaining a semi-clenched position with the palm facing upwards might require using only the thumb for interaction, which has limited range of motion, making it difficult to reach some buttons or perform certain gestures. To address this, in an optional implementation, the user can separate the hands-on experience from the functional experience into two stages. For example, the user can first extend their left hand in front of the camera to handle the 3D model of the product. Then, if further interaction is needed, the 3D model can be placed on a plane within the real-world image captured in the AR space. For example, if a user is experiencing AR and there happens to be a table nearby, the table can be included in the image capture range. Then, the user can place the 3D model on the table by moving their left hand to the tabletop in the AR space, and then use their left hand to perform various interactive operations on the 3D model.

[0085] It should also be noted that the interactive operations performed by users on specific 3D models through AR space are primarily used to simulate the interactive operations defined in the target product for activating target functions. That is, the specific interaction method used to activate a particular function depends on the definition of the activation method for that function within the specific product. However, different products may have different activation methods for the same function. Therefore, to facilitate user operation, prompts regarding the execution methods of various interactive operations can be provided on the AR space interface. In this way, while viewing specific 3D models through AR space, users can perform specific interactive operations based on the prompts on the interface to experience the specific functions of the product.

[0086] In addition, in this embodiment of the application, an AR display entry point for other similar products related to the current product can be provided in the interface of the specific AR space. Thus, after receiving an AR display request for other similar products through this AR display entry point, the current 3D model displayed in the AR space can be replaced with the corresponding 3D model of the other similar product.

[0087] For example, such as Figure 3As shown at point 31 in (B), during the process of displaying a 3D model of a mobile phone product using AR, a list of other mobile phone products related to that product can be displayed in the AR space interface. These other mobile phone products can be determined in various ways. For example, they could be other mobile phone models from the same store as the currently displayed product, or they could be recommended across stores, such as mobile phone products from other stores with the same price range as the currently displayed product, and so on. In practice, there can be various specific product recommendation strategies. Of course, the recommended products must be associated with a 3D model; that is, only products with pre-existing 3D models will be included in this AR space recommendation process.

[0088] After providing the aforementioned product recommendation list through the AR space interface, users can easily initiate AR display requests for other similar products. For example, in Figure 3 In the interface shown in (B), if the user has already completed the AR browsing of the current mobile phone product, they can directly click on other mobile phone products in the list to map the 3D model of the other mobile phone products into the current AR space, without having to exit the current interface, enter the details page of another mobile phone product, and re-initiate the AR display request, etc. Therefore, the interaction efficiency can be improved.

[0089] In addition, when replacing the 3D model in the AR space by clicking on the product recommendation list in the AR space interface, it is also possible to compare the 3D models of different products based on the real environment image in the same AR space, so that users can make a choice among multiple different products.

[0090] In summary, through the embodiments of this application, for handheld device objects with displays, during the display process via AR, the interactive operations performed by the user on the 3D model through the AR space can be detected. These interactive operations can be used to simulate operations defined in the target object related to activating the target function. Subsequently, based on the detected interactive operations, the interface content corresponding to the target function can be displayed in the area where the display screen is located in the 3D model. This allows users to experience or test the functions of a specific handheld device object while browsing it via AR, thus obtaining an experience closer to that of purchasing or trying out a physical object offline. In product display scenarios, this helps users make purchasing decisions.

[0091] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0092] Corresponding to the foregoing method embodiments, this application also provides an information display device, see below. Figure 4 The device may include:

[0093] AR display unit 401 is used to respond to a request to display a target object through augmented reality (AR) method, load the three-dimensional model corresponding to the target object, and map the three-dimensional model into AR space for display; the target object is a handheld device object with a display screen.

[0094] The interactive behavior detection unit 402 is used to detect the interactive operations performed by the user on the three-dimensional model through the AR space. The interactive operations are used to simulate the operations related to activating the target function in the target object.

[0095] The interface content display unit 403 is used to display the interface content corresponding to the target function in the area of ​​the display screen in the three-dimensional model according to the detected interactive operation.

[0096] The interactive operations performed by the user on the 3D model through the AR space include: gesture operations performed based on the display screen in the AR space;

[0097] At this time, the interface content display unit can be specifically used for:

[0098] In the area where the display screen is located in the three-dimensional model, the interface content corresponding to the target function activated by the gesture operation is displayed.

[0099] The target functions may include photo taking, flashlight, smart voice assistant, or music control, etc.

[0100] In a specific implementation, the device may further include:

[0101] The material loading unit is used to load materials related to the target function during the process of displaying the interface content corresponding to the target function through the display screen area in the three-dimensional model, and display or play them through the AR space.

[0102] An interaction method prompting unit is used to provide prompting information about the execution method of the interaction operation in the interface of the AR space.

[0103] The AR display unit can be used for:

[0104] Identify the pose information of hand images from a stream of captured real-world images;

[0105] Based on the pose information of the identified hand image, the 3D model is mapped to the location of the hand image in AR space for display.

[0106] Specifically, the AR display unit can be used for:

[0107] When the hand image is identified as being in a half-clenched position with the palm facing upwards, the 3D model is mapped face up onto the location of the hand image in the AR space.

[0108] Alternatively, the AR display unit can also be used for:

[0109] When the hand image changes from a half-clenched posture to a clenched fist posture, and continues to change to a target posture that meets the flipping conditions, the 3D model is flipped so that the back is facing up, and mapped into the AR space for display.

[0110] Specifically, the three-dimensional model may have the same size as the target object, and the device may further include:

[0111] A size display unit is used to display the size information of the three-dimensional model in the AR space.

[0112] Additionally, the device may also include:

[0113] A size prompting unit is used to provide prompt information on the edge portion of the 3D model in the AR space if the area of ​​the 3D model is larger than the area of ​​the hand image.

[0114] Additionally, the device may also include:

[0115] An object list providing unit is used to provide an object list area in the interface of the AR space, the object list area being used to display at least one other similar object related to the target object, and its corresponding AR display entry;

[0116] The model replacement unit is used to replace the 3D model displayed in the AR space with the 3D model corresponding to the other similar object after receiving an AR display request for the other similar object through the AR display entry.

[0117] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0118] And an electronic device, comprising:

[0119] One or more processors; and

[0120] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0121] in, Figure 5 The architecture of an electronic device is illustrated by example. For instance, device 500 could be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, aircraft, etc.

[0122] Reference Figure 5 The device 500 may include one or more of the following components: processing component 502, memory 504, power supply component 506, multimedia component 508, audio component 510, input / output (I / O) interface 512, sensor component 514, and communication component 516.

[0123] Processing component 502 typically controls the overall operation of device 500, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the methods provided in this disclosure. Furthermore, processing component 502 may include one or more modules to facilitate interaction between processing component 502 and other components. For example, processing component 502 may include a multimedia module to facilitate interaction between multimedia component 508 and processing component 502.

[0124] Memory 504 is configured to store various types of data to support the operation of device 500. Examples of this data include instructions for any application or method operating on device 500, contact data, phonebook data, messages, pictures, videos, etc. Memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0125] Power supply component 506 provides power to various components of device 500. Power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 500.

[0126] Multimedia component 508 includes a screen that provides an output interface between device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When device 500 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0127] Audio component 510 is configured to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) configured to receive external audio signals when device 500 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 also includes a speaker for outputting audio signals.

[0128] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0129] Sensor assembly 514 includes one or more sensors for providing state assessments of various aspects of device 500. For example, sensor assembly 514 may detect the on / off state of device 500, the relative positioning of components such as the display and keypad of device 500, changes in the position of device 500 or a component of device 500, the presence or absence of user contact with device 500, the orientation or acceleration / deceleration of device 500, and temperature changes of device 500. Sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 514 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0130] Communication component 516 is configured to facilitate wired or wireless communication between device 500 and other devices. Device 500 can access wireless networks based on communication standards, such as WiFi, or mobile communication networks such as 2G, 3G, 4G / LTE, and 5G. In one exemplary embodiment, communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 516 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0131] In an exemplary embodiment, device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0132] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, which can be executed by a processor 520 of device 500 to perform the method provided by the present disclosure. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0133] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0134] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0135] The foregoing has provided a detailed description of the product information display method, apparatus, and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are merely for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An information display method, characterized in that, include: In response to a request to display a target object via augmented reality (AR), a 3D model corresponding to the target object is loaded, and the pose information of the hand image is identified from the acquired real-world image stream. Based on the pose information of the hand image, the 3D model is mapped to the location of the hand image in the AR space for display. The target object is a handheld device with a display screen. The three-dimensional model has the same dimensions as the target object; The system detects user interactions with the 3D model through the AR space. These interactions simulate operations related to activating a target function on the target object. The interactions include gestures performed on the display screen of the target object in the AR space, with different gestures triggering different function options. After detecting that the interaction is the gesture operation, the target function option corresponding to the gesture operation is determined, and the interface content corresponding to the target function option after it is invoked is displayed in the display screen area of ​​the 3D model.

2. The method according to claim 1, characterized in that, The detection of user interactions with the 3D model via the AR space includes: The gesture operations performed by the user on the display screen of the target object in the AR space are detected.

3. The method according to claim 1, characterized in that, The target functions include one of the following: photo function, flashlight function, intelligent voice assistant function, or music control function.

4. The method according to claim 1, characterized in that, Also includes: During the process of displaying the interface content corresponding to the target function option after it is activated through the display screen area in the three-dimensional model, materials related to the target function are loaded and displayed or played through the AR space.

5. The method according to claim 1, characterized in that, The method further includes: The interface of the AR space provides prompts regarding how to perform the interactive operations.

6. The method according to claim 1, characterized in that, The step of mapping the 3D model to the location of the hand image in AR space based on the pose information of the recognized hand image for display includes: When the hand image is identified as being in a half-clenched position with the palm facing upwards, the 3D model is mapped face up onto the location of the hand image in the AR space.

7. The method according to claim 6, characterized in that, Also includes: When the hand image changes from a half-clenched posture to a clenched fist posture, and continues to change to a target posture that meets the flipping conditions, the 3D model is flipped so that the back is facing up, and mapped into the AR space for display.

8. The method according to claim 1, characterized in that, Also includes: If the area of ​​the 3D model is larger than the area of ​​the hand image, then a cue message is provided at the edge of the 3D model in the AR space.

9. The method according to any one of claims 1 to 5, characterized in that, Also includes: The AR space interface provides an object list area, which is used to display at least one other similar object related to the target object, and its corresponding AR display entry point; After receiving an AR display request for other similar objects through the AR display entry point, the 3D model displayed in the AR space is replaced with the 3D model corresponding to the other similar objects.

10. An information display device, characterized in that, include: The AR display unit is used to respond to a request to display a target object through augmented reality (AR) method, load the three-dimensional model corresponding to the target object, and identify the pose information of the hand image from the acquired real environment image stream, so as to map the three-dimensional model to the position of the hand image in the AR space for display based on the pose information of the hand image. The target object is a handheld device with a display screen. The three-dimensional model has the same dimensions as the target object; An interactive behavior detection unit is used to detect interactive operations performed by the user on the 3D model through the AR space. The interactive operations are used to simulate operations related to activating target functions in the target object. The interactive operations include gesture operations performed on the display screen of the target object in the AR space, and different gesture operations correspond to activating different function options. The interface content display unit is used to determine the target function option corresponding to the gesture operation after the detected interaction operation is the gesture operation, and to display the interface content corresponding to the target function option after it is invoked in the display screen area of ​​the three-dimensional model.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 9.

12. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 9.

Images