Commodity display method and electronic device

Abstract

This application discloses a product display method and electronic device. The method includes: in response to a request to display a target product using augmented reality (AR), acquiring a real-world environment image and loading a 3D model corresponding to the target product; the target product includes handheld items; detecting a user's hand image from the acquired real-world environment image; and mapping the 3D model of the target product to the location of the hand image in the real-world environment image for display. This application allows users to have an interactive experience of "hands-on" trying out products offline, thereby helping them make purchasing decisions.

Description

Technical Field

[0001] This application relates to the field of product display technology, and in particular to product display methods and electronic devices. Background Technology

[0002] In product information service systems, images, videos, and live streams are common methods for describing products. Users can obtain information about the product's features through these descriptions, thus helping them make purchasing decisions. In recent years, solutions have emerged that use 3D models to display product information. This involves reconstructing the product in 3D, presenting a dynamic 3D effect to the user on the product information service system's client side, and enabling user interaction with the product. For example, users can trigger the product to rotate by swiping the screen, allowing them to view the product's appearance from multiple perspectives.

[0003] However, even when products are displayed using 3D models, the information available to users remains relatively limited. Therefore, how to provide users with more detailed information about specific products and create an experience closer to that of purchasing goods offline has become a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] This application provides a product display method and electronic device that enables users to have an interactive experience of "hands-on" trying out products offline, thereby helping users make purchasing decisions.

[0005] This application provides the following solution:

[0006] A product display method, comprising:

[0007] In response to a request to display a target product using augmented reality (AR), an image of the real-world environment is captured and a 3D model corresponding to the target product is loaded; the target product includes handheld items.

[0008] Detecting the user's hand image from the collected real-world environment images;

[0009] The 3D model of the target product is mapped to the location of the hand image in the real-world environment image for display.

[0010] This also includes:

[0011] Detect the user's hand placement posture;

[0012] The step of mapping the 3D model of the target product onto the location of the hand image in the real-world environment image for display includes:

[0013] If the hand placement posture meets the conditions, the 3D model of the target product is mapped to the location of the hand image in the real environment image for display.

[0014] This also includes:

[0015] If the hand pose in the hand image does not meet the requirements, a prompt message regarding the placement of the hand is provided.

[0016] The dimensions of the three-dimensional model are the same as the actual dimensions of the target product;

[0017] The method further includes:

[0018] The dimensional information of the three-dimensional model is displayed.

[0019] This also includes:

[0020] After detecting an interactive operation performed by the user on the 3D model through hand gestures from the real-world environment image, dynamic content about the 3D model triggered by the interactive operation is displayed.

[0021] The display of dynamic content about the 3D model triggered by the interactive operation includes:

[0022] When the hand image is detected to have shifted or flipped, the 3D model is displayed following the shift or flip of the hand image.

[0023] The handheld item includes a handheld device with a display screen;

[0024] The interactive operation is used to simulate operations related to activating the target function in the handheld device;

[0025] The display of dynamic content about the 3D model triggered by the interactive operation includes:

[0026] The interface content corresponding to the target function is displayed in the area where the display screen is located in the three-dimensional model.

[0027] This also includes:

[0028] The hand image is reconstructed in real time using a 3D mesh to obtain the coordinates of multiple 3D key points of the hand.

[0029] Determine the depth information of the hand based on the coordinates of the three-dimensional key points of the hand;

[0030] Based on the coordinates of the three-dimensional hand key points and the depth information, the display position and display size of the three-dimensional model of the target product in the real-world environment image are determined.

[0031] This also includes:

[0032] During the 3D mesh reconstruction of the hand image, camera parameter information is also acquired;

[0033] Based on the coordinates of the three-dimensional hand key points, depth information, and camera parameter information, hand pose estimation is performed;

[0034] The display posture of the target product's 3D model when projected onto the real-world environment image is determined based on the hand posture estimation results.

[0035] The target product's 3D model is bound to a standard pose 3D hand model. By default, the 3D hand model holds the target product's 3D model in the standard pose, and the standard pose 3D hand model is invisible.

[0036] The hand pose estimation result includes: the rotation matrix and / or translation vector of the true hand pose relative to the standard pose;

[0037] The step of determining the display posture of the target product's 3D model when projected onto the real-world environment image based on hand posture estimation results includes:

[0038] By applying the estimated rotation matrix and / or translation vector to a 3D hand model bound to the 3D model of the target product, the display posture of the 3D model of the target product when projected onto the real-world environment image is determined.

[0039] A product display device, comprising:

[0040] The request receiving unit is configured to, in response to a request to display a target product via augmented reality (AR), acquire an image of the real-world environment and load a 3D model corresponding to the target product; the target product includes handheld items.

[0041] A hand image detection unit is used to detect the user's hand image from the acquired real-world environment image;

[0042] The AR display unit is used to map the 3D model of the target product onto the location of the hand image in the real environment image for display.

[0043] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the preceding methods.

[0044] An electronic device, comprising:

[0045] One or more processors; and

[0046] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of any of the preceding methods.

[0047] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0048] This application provides an entry point for displaying handheld items via AR. After a user initiates an AR display request through this entry point, an image of the real-world environment can be captured and a 3D model of the target product can be loaded. Simultaneously, the user's hand image can be detected from the captured real-world image, and the 3D model of the target product can be mapped to the location of the hand image in the real-world image for display. This allows users to have an interactive experience of "hands-on" trying out the product offline, thereby assisting them in making purchasing decisions.

[0049] The specific 3D model can be the same size as the actual product. By estimating the depth of field of the hand image, the display size of the 3D model on the screen can also be determined, thus reflecting the relative size relationship between the 3D model and the hand image.

[0050] Furthermore, the 3D model can move in AR space along with the hand image, presenting a "nearer is larger, farther is smaller" display effect, and can also be flipped. Moreover, a standard-pose 3D hand model can be created for the target product's 3D model; that is, by default, the target product's 3D model can be held in a standard pose by a transparent 3D hand model. During hand pose estimation, the rotation matrix and / or translation vector of the real-world hand's pose relative to this labeled pose can be determined. By applying the estimated rotation matrix and / or translation vector to the 3D hand model bound to the target product's 3D model, the display pose of the target product's 3D model projected onto the real-world environment image can be determined. This allows for a better presentation of the target product's 3D model being held by a real hand, and even effects such as the product's 3D model being partially obscured by fingers, and more easily achieves the effect of the 3D model moving with the hand, providing users with a more realistic experience.

[0051] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of the system architecture provided in the embodiments of this application;

[0054] Figure 2 This is a flowchart of the method provided in the embodiments of this application;

[0055] Figure 3 This is a schematic diagram of the AR display effect provided in the embodiments of this application;

[0056] Figure 4 This is a schematic diagram of the device provided in the embodiments of this application;

[0057] Figure 5 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0059] First, it's important to note that to enhance the realism of product and other target object information, one approach is to combine 3D product reconstruction technology with AR (Augmented Reality) technology for product display. In this solution, the product is pre-reconstructed in 3D. When displayed on the client side, a real-time image stream from the real world is captured using the terminal device's camera. Planes such as "tables" and "floors" within the image stream are located, and the product's 3D model is projected onto the positions of these planes in the real-world image stream, thus presenting the product as if it were actually "placed" in a real-world environment.

[0060] The above-mentioned solution of displaying product information using AR technology allows users to more intuitively judge whether a specific product is suitable for their real-world environment. For example, whether a sofa or other home furnishing product is suitable for placing in their living room, etc., thereby helping users make better purchasing decisions.

[0061] However, in current technologies, when displaying product information using AR, it's usually just a matter of mechanically placing the product's 3D model into the AR space. While the viewing angle can be changed by swiping the screen or rotating the phone to achieve a 360-degree view of the product's appearance, and this is generally sufficient to help users make purchasing decisions for items like sofas and coffee tables, this simple and mechanical AR placement is often inadequate for other product categories.

[0062] For example, when purchasing handheld devices like smartphones offline, users may not only need to see the product's appearance but also experience how it feels to hold it, including whether the size suits their hand. However, displaying a product model on a flat surface using AR technology cannot achieve the same effect.

[0063] Therefore, in this embodiment, an implementation scheme is provided to offer users a "hands-on" experience based on AR. Specifically, a 3D model can be pre-generated for the specific product. In a preferred implementation, the 3D model can be created at a 1:1 scale based on the actual size of the product. For handheld items with 3D models, an "AR" entry point can be provided on pages such as the product details page. After the user initiates a relevant request through this entry point, the 3D model can be loaded, and the terminal device's camera can be activated to capture images of the real environment. The user can also be prompted to extend their hand in front of the camera, and the user's hand image can be recognized from the real environment image. Then, the 3D model of the specific product can be mapped to the location of the hand image for display. In this way, users can obtain a "hands-on" experience of trying out specific products in an offline scenario through AR. In addition, in the specific implementation, users can also interact with the 3D model through hand movements or gestures. For example, during the translation or flipping of the hand, the 3D image of the product also translates or flips accordingly, allowing users to change the angle to view the trial effect, and also view the back of the product, etc.

[0064] Furthermore, to make the "hands-on" experience more realistic, after recognizing the hand image from the acquired real-world environment image, real-time 3D mesh reconstruction can be performed on the hand image to obtain the coordinates of multiple 3D hand key points (e.g., typically 778, including key points such as some joints of the hand). Based on this information, the display position, posture, etc., of the product's 3D model when projected onto the real-world environment image can be determined, thus presenting a 3D model of the target product held by a real hand, and the 3D model of the target product can follow the movement of the real hand.

[0065] From a system architecture perspective, see Figure 1 This application embodiment can involve the server and client sides of a product information service system. The server side is mainly used to pre-reconstruct the product in 3D and save the corresponding 3D model. Additionally, in an optional implementation, a relevant algorithm model for real-time 3D reconstruction of hand images can be trained and distributed to the terminal device on the client side. On the client side, an entry point for displaying products via AR can be provided to the user. When the user enters through this entry point, the 3D model of the product can be loaded. Simultaneously, the camera component of the terminal device is activated to capture images of the real environment and identify the user's hand image. Then, the 3D model is displayed at the location of the hand image in the real environment image. Optionally, the aforementioned algorithm model can also be used to perform real-time 3D mesh reconstruction of the real hand image to obtain multiple 3D hand key point coordinates. Based on these 3D hand key point coordinates, the 3D model of the product is then projected onto the real environment image.

[0066] The specific implementation schemes provided in the embodiments of this application will be described in detail below.

[0067] First, from the perspective of the aforementioned client, this application provides a product display method, see [link to relevant documentation]. Figure 2 The method may include:

[0068] S201: In response to a request to display a target product using augmented reality (AR), acquire an image of the real-world environment and load a 3D model corresponding to the target product; the target product includes handheld items.

[0069] In this embodiment, the specific target product can be a handheld item, such as a mobile communication device like a mobile phone. Specifically, an entry point for displaying the product via AR can be provided on pages such as the product details page. For example, this entry point can be provided in the main image display area of ​​the product details page. This allows users to request AR display of the product while accessing the product details page. Alternatively, other methods can be used. For instance, products with configured 3D models can be aggregated into a single theme page, providing an entry point for AR display of each product. Users can then directly request AR display of a specific target product through this theme page. This embodiment primarily uses a product as an example to describe the interactive methods during the AR display process.

[0070] Upon receiving a specific AR display request, an AR space can be created, and a 3D model of the target product can be loaded for rendering and display within the AR space. Specifically, when creating the AR space, components such as the camera on the terminal device can be activated to capture images of the real world. Optionally, the user can be prompted to extend their hand in front of the camera, and prompts can be provided regarding hand placement, such as palm facing upwards in a semi-clenched position.

[0071] S202: Detect the user's hand image from the acquired real-world environment image.

[0072] After acquiring images of the real environment, the client can detect hand images from the acquired image stream. The detection of hand images can be achieved using pre-generated algorithms. For example, subject recognition can be performed from the acquired image stream based on the features of the hand images, etc., which will not be detailed here.

[0073] S203: Map the 3D model of the target product to the location of the hand image in the real-world environment image and display it.

[0074] After identifying the specific location of the hand image from the real-world image, the 3D model of the target product can be mapped to the location of the hand image in the real-world image for display, allowing the user to experience "hands-on" use of the product's 3D model. Furthermore, since the size of the 3D model can be the same as the actual size of the target product, the size information of the 3D model can also be displayed in the implementation. For example, as... Figure 3 As shown in (A), the width and height of the 3D model can be indicated. When the 3D model is rotated at a certain angle for display, its thickness can also be indicated, such as... Figure 3 As shown in (B). This size information can be displayed directly by default, or it can be displayed after an interactive operation is performed, etc. For example, an interactive button can be displayed on the screen of the current terminal device. The user can use one hand holding the terminal device (in the scenario of this application embodiment, the user holds their mobile phone or other terminal device in one hand and places the other hand in front of the camera to experience the virtual 3D model) to click the specific interactive button on the interface to trigger the display of specific size information, etc.

[0075] In practical implementation, upon detecting a hand image, the user's hand placement posture can also be detected to determine the hand's pose (i.e., not only the hand's position but also its posture). Based on this posture, the posture of the 3D model (including rotation and translation angles) can be determined. It should be noted that, in practical implementation, the client can also determine whether the hand placement posture meets certain conditions. If it does, the 3D model of the target product can be mapped to the location of the hand image in the real-world environment image for display, so as to show the effect of the user holding the 3D model through AR. For example, a suitable hand placement posture could be a palm-up, semi-clenched position, etc. If the hand posture in the hand image does not meet the conditions, prompts regarding the placement posture can be provided. These prompts can be displayed through graphics, text, etc. on the interface.

[0076] In practical applications, the size of a hand displayed on a screen varies depending on its distance from the lens. Therefore, to more realistically represent the relative size of the hand and the 3D model, it is best to obtain the depth information of the hand image in image space. This allows for the projection of the 3D model based on the depth information, enabling the 3D model to exhibit a "nearer is larger, farther is smaller" effect, thereby further enhancing the realism of the AR display. However, in the scenario described in this application embodiment, users typically browse product information using ordinary mobile phones or other terminal devices. The cameras on these devices are mostly 2D cameras, lacking the ability to directly acquire depth information. Therefore, to obtain depth information, 3D reconstruction based on the hand image can also be used for recognition.

[0077] In other words, based on hand images identified from real-world image streams, real-time 3D reconstruction of the hand can be performed to obtain the coordinates of multiple 3D hand key points. The hand's position can then be determined based on these key point coordinates, including the depth information of the hand image. After 3D reconstruction of the hand image, multiple 3D hand key points can be identified, each corresponding to different depth information. In this embodiment, the depth of key points such as the center of the palm can be used to represent the depth of the hand image, and so on. Specifically, a pre-trained deep learning algorithm model can be used to perform 3D mesh reconstruction of the hand image. The algorithm model used for 3D mesh reconstruction of the hand is not the focus of this embodiment and will not be detailed here.

[0078] After identifying the location information of a specific hand image, the 3D model of a specific product can be mapped to the location of the hand image in AR space for display. Since the depth information of the hand is identified, the depth information of the 3D model can be determined, and the display size of the 3D model under that depth condition can be displayed. Furthermore, since the hand posture can also be identified, the posture of the 3D model (e.g., rotation or tilt angle) can be determined based on this posture, thus presenting the effect of a hand holding a specific 3D model in AR space. In addition, since the 3D reconstruction of the hand can be performed in real time, the display position, display size, and posture of the 3D model in AR space can also change accordingly when the hand moves or changes posture. Of course, the change in display size here refers to the fact that the size of the 3D model displayed on the screen can change with the movement of the hand, appearing larger when closer and smaller when farther away. However, regardless of the change in display size, the actual size of the 3D model and its relative size to the human hand remain unchanged. For example, when a person's hand moves away from the camera, the size of the hand displayed on the screen decreases. Correspondingly, the 3D model of the product also moves away from the camera and its size displayed on the screen also decreases. However, the relative size of the 3D model and the person's hand remains unchanged, and vice versa.

[0079] In order to quantitatively identify hand posture, during the process of 3D mesh reconstruction of the real hand image, camera parameter information of the terminal device can also be obtained. In this way, hand posture can be estimated based on the coordinates of the 3D hand key points, the depth information of the hand image, and the camera parameter information. Then, the display posture of the target product when the 3D model is projected into the real environment image can be determined based on the hand posture estimation result.

[0080] In practical implementation, to better achieve the goal of determining the posture of the 3D model of the product through estimated hand posture, and to enhance realism (e.g., to show the effect of the fingertips partially obscuring the 3D model while it is being held), in an optional implementation, a standard posture 3D hand model can be pre-created for the target product's 3D model. By default, this 3D hand model can hold the target product's 3D model in a standard posture. That is, the target product's 3D model can be held by a standard posture 3D hand model, and the target product's 3D model can move with the movement of this 3D hand model. Of course, this standard posture 3D hand model does not need to be displayed; therefore, it can be invisible.

[0081] In the case of a standard-pose 3D hand model, the estimation result of the actual hand pose can include: the rotation matrix and / or translation vector of the actual hand pose relative to the standard pose. Thus, by applying the estimated rotation matrix and / or translation vector to the 3D hand model bound to the 3D model of the target product, the display pose of the 3D model of the target product when projected onto the real-world environment image can be determined.

[0082] Specifically, when estimating hand pose based on the coordinates of the three-dimensional hand key points, hand depth information, and camera parameter information, the projection of the three-dimensional hand key points in the two-dimensional image space can be obtained first using the conversion formula from camera coordinates to two-dimensional image space. Then, based on the coordinates of the three-dimensional hand model in the world space coordinate system and the projection of the three-dimensional hand key points in the two-dimensional image space, the rotation matrix and / or translation vector of the actual hand pose relative to the standard pose can be determined.

[0083] In this way, the global pose of the hand can be obtained based on the positions of various 3D hand keypoints in the hand mesh. Using this information, the AR effect of fingers and palm holding a product together can be better achieved. It can also reflect the occlusion of the product by the fingers during the hand's grasping process, and better ensure that the product moves with the hand. Through hand depth estimation, the size of the product's 3D model that should appear on the screen can be more accurately determined, unaffected by the estimated hand size. That is, when displaying a product in AR using hands of different sizes, if the hand depth of field is the same, the size of the 3D product model displayed on the screen will be consistent, thus enhancing the realism.

[0084] In specific implementation, besides demonstrating the effect of a user holding a 3D model with their hand, the user can also interact with the 3D model through hand gestures or movements. Specifically, the client can detect user interactions with the 3D model through hand gestures from the real-world environment image and display dynamic content about the 3D model triggered by the interaction. For example, when the hand image is detected to shift or flip, the 3D model can be displayed to follow the shift or flip of the hand image. Figure 3As shown in (A) and 3(B), the 3D model follows the hand's rotation before and after the hand is rotated. Alternatively, if the specific handheld item is a handheld device with a display screen (e.g., a mobile phone), the interactive operation can also be used to simulate operations related to activating the target function on the target object. In this case, the interface content corresponding to the target function is displayed in the area where the display screen is located in the 3D model. For example, the power button area on the 3D model can be used to simulate pressing the power button. At this time, the display screen of the 3D model can be switched to a lit state, and a boot animation can be played, etc. Alternatively, if the specific product supports waking up the display screen by double-tapping, the user can also simulate performing this double-tapping operation in the display screen area of ​​the 3D model. At this time, the display screen can also be switched to a lit state, etc. In addition, the handheld device's camera function, flashlight function, intelligent voice assistant function, or music control function can also be experienced.

[0085] In summary, this application provides an entry point for displaying handheld items via AR. After a user initiates an AR display request through this entry point, an image of the real-world environment can be captured and a 3D model of the target product can be loaded. Simultaneously, the user's hand image can be detected from the captured real-world image, and the 3D model of the target product can be mapped to the location of the hand image in the real-world image for display. This allows users to experience "hands-on" trying out the product offline, thus aiding their purchasing decisions.

[0086] The specific 3D model can be the same size as the actual product. By estimating the depth of field of the hand image, the display size of the 3D model on the screen can also be determined, thus reflecting the relative size relationship between the 3D model and the hand image.

[0087] Furthermore, the 3D model can move in AR space along with the hand image, presenting a "nearer is larger, farther is smaller" display effect, and can also be flipped. Moreover, a standard-pose 3D hand model can be created for the target product's 3D model; that is, by default, the target product's 3D model can be held in a standard pose by a transparent 3D hand model. During hand pose estimation, the rotation matrix and / or translation vector of the real-world hand's pose relative to this labeled pose can be determined. By applying the estimated rotation matrix and / or translation vector to the 3D hand model bound to the target product's 3D model, the display pose of the target product's 3D model projected onto the real-world environment image can be determined. This allows for a better presentation of the target product's 3D model being held by a real hand, and even effects such as the product's 3D model being partially obscured by fingers, and more easily achieves the effect of the 3D model moving with the hand, providing users with a more realistic experience.

[0088] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0089] Corresponding to the foregoing method embodiments, this application also provides a product display device, see below. Figure 4 The device may include:

[0090] The request receiving unit 401 is used to respond to a request to display a target product through augmented reality (AR) by acquiring an image of the real environment and loading a 3D model corresponding to the target product; the target product includes handheld items.

[0091] The hand image detection unit 402 is used to detect the user's hand image from the acquired real-world environment image;

[0092] AR display unit 403 is used to map the three-dimensional model of the target product onto the location of the hand image in the real environment image for display.

[0093] In a specific implementation, the device may further include:

[0094] The posture detection unit is used to detect the user's hand placement posture;

[0095] The AR display unit can specifically be used for:

[0096] When the hand placement posture meets the conditions, the 3D model of the target product is mapped to the location of the hand image in the real environment image for display, so as to show the effect of the user holding the 3D model with their hand through the AR method.

[0097] Additionally, the device may also include:

[0098] The prompting unit is used to provide prompting information about the placement posture if the hand posture in the hand image does not meet the conditions.

[0099] The dimensions of the three-dimensional model are the same as the actual dimensions of the target product;

[0100] The device may further include:

[0101] The size information display unit is used to display the size information of the three-dimensional model.

[0102] Additionally, the device may also include:

[0103] The interactive unit is used to detect, after detecting an interactive operation performed by the user on the 3D model through hand gestures from the real-world environment image, display dynamic content about the 3D model triggered by the interactive operation.

[0104] Specifically, the interactive unit can be used for:

[0105] When the hand image is detected to have shifted or flipped, the 3D model is displayed in accordance with the shift or flip of the hand image.

[0106] Alternatively, the handheld item may include a handheld device with a display screen;

[0107] The interactive operation is used to simulate the operations related to activating the target function in the target object;

[0108] At this time, the interactive unit can be specifically used for:

[0109] The interface content corresponding to the target function is displayed in the area where the display screen is located in the three-dimensional model.

[0110] Additionally, the device may also include:

[0111] A hand 3D reconstruction unit is used to perform real-time 3D mesh reconstruction on the hand image to obtain the coordinates of multiple 3D hand key points;

[0112] A hand depth estimation unit is used to determine the depth information of the hand based on the coordinates of the three-dimensional hand key points.

[0113] The display position and size determination unit is used to determine the display position and size of the three-dimensional model of the target product in the real environment image based on the coordinates of the three-dimensional hand key points and the depth information.

[0114] Additionally, the device may also include:

[0115] The camera parameter acquisition unit is used to acquire camera parameter information during the process of 3D Mesh reconstruction of the hand image;

[0116] The hand pose estimation unit is used to estimate the hand pose based on the coordinates of the three-dimensional hand key points, depth information, and camera parameter information.

[0117] The display posture determination unit is used to determine the display posture of the target product's 3D model when projected onto the real-world environment image based on the hand posture estimation results.

[0118] In addition, the 3D model of the target product is bound to the 3D hand model in the standard pose. By default, the 3D hand model holds the 3D model of the target product in the standard pose, and the 3D hand model in the standard pose is invisible.

[0119] The hand pose estimation result includes: the rotation matrix and / or translation vector of the true hand pose relative to the standard pose;

[0120] The display posture determination unit can be specifically used to: determine the display posture of the three-dimensional model of the target product when it is projected onto the real environment image by applying the estimated rotation matrix and / or translation vector to the three-dimensional hand model bound to the three-dimensional model of the target product.

[0121] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0122] And an electronic device, comprising:

[0123] One or more processors; and

[0124] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0125] in, Figure 5The architecture of an electronic device is illustrated by example. For instance, device 500 could be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, aircraft, etc.

[0126] Reference Figure 5 The device 500 may include one or more of the following components: processing component 502, memory 504, power supply component 506, multimedia component 508, audio component 510, input / output (I / O) interface 512, sensor component 514, and communication component 516.

[0127] Processing component 502 typically controls the overall operation of device 500, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 502 may include one or more processors 520 to execute instructions to perform all or part of the steps of the methods provided in this disclosure. Furthermore, processing component 502 may include one or more modules to facilitate interaction between processing component 502 and other components. For example, processing component 502 may include a multimedia module to facilitate interaction between multimedia component 508 and processing component 502.

[0128] Memory 504 is configured to store various types of data to support the operation of device 500. Examples of this data include instructions for any application or method operating on device 500, contact data, phonebook data, messages, pictures, videos, etc. Memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0129] Power supply component 506 provides power to various components of device 500. Power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 500.

[0130] Multimedia component 508 includes a screen that provides an output interface between device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 508 includes a front-facing camera and / or a rear-facing camera. When device 500 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0131] Audio component 510 is configured to output and / or input audio signals. For example, audio component 510 includes a microphone (MIC) configured to receive external audio signals when device 500 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 504 or transmitted via communication component 516. In some embodiments, audio component 510 also includes a speaker for outputting audio signals.

[0132] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0133] Sensor assembly 514 includes one or more sensors for providing state assessments of various aspects of device 500. For example, sensor assembly 514 may detect the on / off state of device 500, the relative positioning of components such as the display and keypad of device 500, changes in the position of device 500 or a component of device 500, the presence or absence of user contact with device 500, the orientation or acceleration / deceleration of device 500, and temperature changes of device 500. Sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 514 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 514 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0134] Communication component 516 is configured to facilitate wired or wireless communication between device 500 and other devices. Device 500 can access wireless networks based on communication standards, such as WiFi, or mobile communication networks such as 2G, 3G, 4G / LTE, and 5G. In one exemplary embodiment, communication component 516 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 516 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0135] In an exemplary embodiment, device 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0136] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 504 including instructions, which can be executed by a processor 520 of device 500 to perform the method provided by the present disclosure. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0137] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0138] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0139] The product display method and electronic device provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for displaying goods, characterized in that, include: In response to a request to display a target product using augmented reality (AR), an image of the real-world environment is captured and a 3D model of the target product is loaded. The target goods include handheld items; the handheld items include handheld devices with displays. The system detects user hand images from acquired real-world environment images, as well as user interactions with the target product's 3D model via hand gestures. It detects the user's hand placement and reconstructs the hand images in 3D to obtain multiple 3D hand keypoint coordinates. Based on these coordinates, it determines the hand's position information, including depth information at the locations of the keypoints in the hand image. The interactive operations simulate actions related to activating the target function on the handheld device. The 3D model of the target product is mapped to the key points of the hand image in the real environment image and displayed thereon. Dynamic content about the 3D model of the target product, triggered by the interactive operation, is displayed in the display area of ​​the 3D model of the target product. The dynamic content includes the interface content corresponding to the target function.

2. The method according to claim 1, characterized in that, The step of mapping the 3D model of the target product onto the key points of the hand image in the real-world environment image for display includes: If the hand placement posture meets the conditions, the 3D model of the target product is mapped to the location of the key points of the hand image in the real environment image for display.

3. The method according to claim 2, characterized in that, Also includes: If the hand pose in the hand image does not meet the requirements, a prompt message regarding the placement of the hand is provided.

4. The method according to claim 1, characterized in that, The dimensions of the 3D model of the target product are the same as the actual dimensions of the target product; The method further includes: The dimensional information of the three-dimensional model of the target product is displayed.

5. The method according to claim 1, characterized in that, The dynamic content displaying the 3D model of the target product triggered by the interactive operation includes: When the hand image is detected to have shifted or flipped, the 3D model of the target product is displayed following the shift or flip of the hand image.

6. The method according to any one of claims 1 to 5, characterized in that, The three-dimensional reconstruction of the hand image includes: real-time three-dimensional mesh reconstruction of the hand image; The method further includes; Based on the coordinates of the multiple 3D hand key points and the depth information, the display position and display size of the 3D model of the target product in the real-world environment image are determined.

7. The method according to claim 6, characterized in that, Also includes: During the real-time 3D mesh reconstruction of the hand image, camera parameter information is also acquired. Based on the coordinates of the three-dimensional hand key points, depth information, and camera parameter information, hand pose estimation is performed; The display posture of the target product's 3D model when projected onto the real-world environment image is determined based on the hand posture estimation results.

8. The method according to claim 7, characterized in that, The 3D model of the target product is bound to the 3D hand model in a standard pose. By default, the 3D hand model holds the 3D model of the target product in the standard pose, and the 3D hand model in the standard pose is invisible. The hand pose estimation result includes: the rotation matrix and / or translation vector of the true hand pose relative to the standard pose; The step of determining the display posture of the target product's 3D model when projected onto the real-world environment image based on hand posture estimation results includes: By applying the estimated rotation matrix and / or translation vector to a 3D hand model bound to the 3D model of the target product, the display posture of the 3D model of the target product when projected onto the real-world environment image is determined.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 8.

10. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 8.

Images