Image display method, apparatus, device, and medium

Abstract

The present disclosure relates to an image display method, device, equipment and medium. The image display method comprises: acquiring a real-time image of a target body part; and displaying a composite image in real time, wherein the composite image is an image obtained by superimposing a target three-dimensional image on the target body part in the real-time image, the target three-dimensional image is obtained by rendering a wearable component three-dimensional model according to a real-time posture and a real-time unoccluded area of the target body part, and the real-time posture and the real-time unoccluded area are determined according to the real-time image. According to the embodiment of the present disclosure, the fusion of the added three-dimensional decoration effect and the original image can be improved, and the user experience is improved.

Description

Technical Field

[0001] This disclosure relates to the field of multimedia technology, and in particular to an image display method, apparatus, device and medium. Background Technology

[0002] With the rapid development of computer and mobile communication technologies, various image capture platforms based on electronic devices have become widely used, greatly enriching people's daily lives. More and more users are happy to take pictures or videos on these platforms to obtain satisfactory results.

[0003] To enhance the fun of image capture, decorative effects can be added to each frame in real-time as the user takes the picture. However, while existing decorative effects are somewhat interesting, they often blend poorly with the original image, frequently resulting in continuity errors and degrading the user experience. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides an image display method, apparatus, device and medium.

[0005] In a first aspect, this disclosure provides an image display method, including:

[0006] Acquire real-time images of the target body part;

[0007] The composite image is displayed in real time. The composite image is obtained by superimposing the target 3D image on the target body part in the real-time image. The target 3D image is obtained by rendering the 3D model of the wearable component based on the real-time pose of the target body part and the real-time unoccluded area. The real-time pose and the real-time unoccluded area are determined based on the real-time image.

[0008] Secondly, this disclosure provides an image display device, comprising:

[0009] The acquisition unit is configured to acquire real-time images of the target body part;

[0010] The display unit is configured to display a synthetic image in real time. The synthetic image is an image obtained by superimposing a target 3D image onto the target body part in the real-time image. The target 3D image is obtained by rendering the 3D model of the wearable component based on the real-time pose of the target body part and the real-time unoccluded area. The real-time pose and the real-time unoccluded area are determined based on the real-time image.

[0011] Thirdly, this disclosure provides an image display device, including:

[0012] processor;

[0013] Memory, used to store executable instructions;

[0014] The processor is used to read executable instructions from memory and execute the executable instructions to implement the image display method described in the first aspect.

[0015] Fourthly, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the image display method described in the first aspect.

[0016] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0017] The image display method, apparatus, device, and medium of this disclosure can, after acquiring a real-time image of a target body part, display in real-time a composite image obtained by superimposing a target 3D image onto the target body part in the real-time image. The target 3D image is obtained by rendering a 3D model of a wearable component based on the real-time pose and unobstructed area of ​​the target body part. The real-time pose and unobstructed area are directly determined based on the real-time image, thereby achieving the purpose of automatically adding a 3D decorative effect with wearable components to the real-time image. Since the pose and occlusion of the body part wearing the decorative effect are considered during the addition process, the integration of the added 3D decorative effect with the original image can be improved, avoiding visual glitches and enhancing the user experience. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0019] Figure 1 This is a schematic flowchart illustrating an image display method provided in an embodiment of the present disclosure;

[0020] Figure 2 A schematic diagram of a composite image provided in an embodiment of this disclosure;

[0021] Figure 3 A schematic diagram of another composite image provided in an embodiment of this disclosure;

[0022] Figure 4 This is a schematic diagram of a renderable image region provided in an embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of another renderable image region provided in an embodiment of this disclosure;

[0024] Figure 6 A schematic diagram of yet another renderable image region provided in an embodiment of this disclosure;

[0025] Figure 7 A schematic diagram of an obstructed area provided in an embodiment of this disclosure;

[0026] Figure 8 A schematic diagram of yet another renderable image region provided in an embodiment of this disclosure;

[0027] Figure 9 A schematic diagram of a preset occlusion model provided in an embodiment of this disclosure;

[0028] Figure 10 A schematic diagram of yet another composite image provided in an embodiment of this disclosure;

[0029] Figure 11 A schematic diagram of another composite image provided in an embodiment of this disclosure;

[0030] Figure 12 This is a schematic diagram of the structure of an image display device provided in an embodiment of the present disclosure;

[0031] Figure 13 This is a schematic diagram of the structure of an image display device provided in an embodiment of the present disclosure. Detailed Implementation

[0032] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0033] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0034] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0035] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0036] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0037] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0038] Augmented Reality (AR) technology is a technology that can calculate the position and angle of a camera in real time and add corresponding images, videos or 3D models. This technology can combine the virtual world and the real world on the screen and allow for interaction.

[0039] With the development of computer technology, AR technology is being applied more and more widely, and is gradually being incorporated into the shooting functions of image capture platforms. To increase the fun of image capture, AR technology can be used to add decorative effects to parts of the user's body in real time across each frame of the image. For example, adding a helmet effect to the user's head, or a watch effect to the user's wrist.

[0040] However, the applicant found that while the existing decorative effects were somewhat interesting, they still had the following problems:

[0041] Decorative effects cover the entire body part, and if there are other obstructions on that body part, the decorative effect will also cover those obstructions. For example, when adding a helmet effect to a user's head, if the head area where the helmet effect is applied is obscured by the user's hands, the helmet effect will also cover the user's hands. This issue results in poor integration of the decorative effect with the original image, frequently causing continuity errors and making it difficult for users to maintain immersion when using decorative effects, thus reducing the user experience.

[0042] In addition, when adding decorative effects to users' fingers, the occlusion relationship of fingers is more complex and difficult to simulate than that of the head, wrist, etc., making the decorative effects blend worse with the original image. For example, when users try on AR rings, it is easier to see flaws such as poor ring fit or rings embedded in nearby fingers.

[0043] To address the aforementioned problems, embodiments of this disclosure provide an image display method, apparatus, device, and medium that takes into account the posture and occlusion of body parts wearing the decorations when adding decorative effects.

[0044] The following is a reference. Figure 1 The image display method provided in the embodiments of this disclosure will be described.

[0045] In this embodiment of the disclosure, the image display method can be executed by an electronic device. The electronic device may include devices with communication functions such as mobile phones, tablets, desktop computers, laptops, in-vehicle terminals, wearable electronic devices, all-in-one machines, and smart home devices, or it may be a device simulated by a virtual machine or simulator.

[0046] Figure 1 A flowchart illustrating an image display method provided in an embodiment of this disclosure is shown.

[0047] like Figure 1 As shown, the image display method includes the following steps.

[0048] S110. Acquire real-time images of the target body part.

[0049] In this embodiment of the disclosure, when a user wants to add decorative effects to a target body part in a real-time image, the electronic device can acquire a real-time image of the target body part.

[0050] In some embodiments, electronic devices may capture images via a camera to obtain real-time images of the target body part.

[0051] In other embodiments, the electronic device may receive images sent by other devices to obtain real-time images of the target body part.

[0052] In some other embodiments, the electronic device can read images selected by the user in a local image to obtain real-time images of the target body part.

[0053] After the electronic device acquires a real-time image of the target body part, decorative effects can be added to the target body part in the real-time image.

[0054] The target body part can be any body part of the human body that is pre-defined according to actual application needs, and there are no restrictions here. For example, the target body part can include body parts of the first granularity, such as any one of the head, torso, upper limbs, hands, lower limbs, and feet. As another example, the target body part can also include body parts of the second granularity, which has a higher degree of precision than the first granularity, such as any one of the ears, neck, wrists, fingers, and ankles.

[0055] Furthermore, the number of target body parts can be any number preset according to actual application needs, without any restrictions. For example, the number of target body parts can be 1, 2, 3, etc.

[0056] S120. Real-time display of the composite image. The composite image is an image obtained by superimposing the target 3D image on the target body part in the real-time image. The target 3D image is obtained by rendering the 3D model of the wearable component based on the real-time posture of the target body part and the real-time unoccluded area. The real-time posture and the real-time unoccluded area are determined based on the real-time image.

[0057] In this embodiment of the disclosure, after the electronic device acquires a real-time image of a target body part, it can add decorative effects to the target body part in the real-time image in real time, and display in real time a composite image obtained by superimposing the three-dimensional decorative effect onto the target body part in the real-time image.

[0058] In this embodiment of the disclosure, the electronic device can determine the real-time posture of a target body part based on real-time images.

[0059] Optionally, the real-time pose can be the real-time three-dimensional pose of the target body part. The real-time pose can include the real-time rotational pose of the target body part.

[0060] Furthermore, the real-time rotational posture of the target body part can include the real-time three-dimensional rotational posture of each joint in the target body part.

[0061] In this embodiment of the disclosure, the electronic device can determine the real-time unobstructed area of ​​a target body part based on real-time images.

[0062] Optionally, the real-time unobstructed area may include the area of ​​the target body part that is not obscured by any obstruction. The obstruction may include at least one of the following: a non-body part object, a non-target body structure other than the target body structure to which the target body part belongs, and a non-target body part of the same type as the target body part. These can be preset according to actual application requirements.

[0063] Non-body part objects can include at least one of the following: the image background in the real-time image and objects other than the human body that occlude any part of the target body part.

[0064] The target body structure can be any body structure belonging to a target body part that is predefined according to actual application requirements, and there are no restrictions on this. For example, when the target body part is a second-level body part, the target body structure can be a first-level body part, such as any one of the following: head, torso, upper limbs, hands, lower limbs, and feet.

[0065] Furthermore, the target body part can be the target body structure itself or a part of the target body structure, without any limitation.

[0066] Non-target body structures can be any other body structure besides the target body structure as defined by the granularity of the target body structure, and there are no restrictions here.

[0067] For example, if the target body structure is the hand, then the non-target body structures can be the head, torso, upper limbs, lower limbs, and feet, etc.

[0068] Non-target body parts can be any body parts other than the target body parts as defined by the granularity of the target body parts, and there are no restrictions here.

[0069] For example, if the target body part is a finger, then the non-target body part can be another finger.

[0070] Therefore, in this embodiment of the present disclosure, the electronic device can render a target 3D image based on the real-time posture of the target body part and the real-time unobstructed area of ​​the wearable component 3D model, and then superimpose the target 3D image onto the target body part in the real-time image to obtain a composite image.

[0071] Furthermore, the electronic device can render the 3D model of the wearable component based on the real-time unobstructed area of ​​the target body part according to the real-time posture of the target body part, and obtain a 3D image of the target.

[0072] Furthermore, electronic devices can overlay the target's 3D image onto the wearable component position of the target body part in a real-time image to obtain a synthetic image.

[0073] The wearable component can be placed at any pre-defined location on the target body part, depending on the actual application requirements; there are no restrictions on this.

[0074] In some embodiments of this disclosure, the target body part may include the target finger.

[0075] The target finger can be a pre-defined finger based on actual application requirements, and there are no restrictions on its selection. For example, the target finger can be at least one of the thumb, index finger, middle finger, ring finger, and little finger.

[0076] Accordingly, the 3D model of the wearable component can be a 3D model of the component to be worn on the target finger, that is, the 3D model of the wearable component is the 3D model corresponding to the wearable component to be worn on the target finger.

[0077] In these embodiments, the real-time unobstructed area may include the area of ​​the target finger in the real-time image that is not obscured by any occluder.

[0078] Alternatively, to account for all possible occlusions of the target finger, the occluder can be configured to include non-body parts, body structures other than the hand to which the target finger belongs, and fingers other than the target finger.

[0079] Non-body part objects can include at least one of the following: image background and objects other than the human body that obscure any part of the target finger.

[0080] The target finger is a body part at the second granularity, while the corresponding target body structure is a body part at the first granularity. Therefore, the target body structure to which the target finger belongs is the hand to which the target finger belongs. At this point, the non-target hand body structure can be any other body structure other than the hand to which the target finger belongs, based on the first granularity.

[0081] The non-target body part that is of the same body part type as the target finger can be any finger other than the target finger.

[0082] In some embodiments, the wearable component can be a component for fitting snugly on a target finger, such as a ring.

[0083] In these embodiments, optionally, the electronic device can render the portion of the wearable component 3D model located within the real-time unobstructed area of ​​the target finger according to the real-time posture of the target finger, such as the real-time rotation posture, to obtain a target 3D image, and then superimpose the target 3D image onto the wearable component wearing position of the target finger in the real-time image to obtain a composite image.

[0084] Figure 2 A schematic diagram of a composite image provided by an embodiment of this disclosure is shown.

[0085] like Figure 2 As shown, the synthesized image can be an image including the ring finger 201, and the wearable component can be a ring 205. The entire boundary of the hand to which the ring finger 201 belongs is connected to the image background 202. A portion of the area within the ring finger 201 is occluded by the little finger 203 and the middle finger 204. Therefore, the image background 202, the little finger 203, and the middle finger 204 can all be used as occlusions of the ring finger 201. When rendering the 3D model of the ring, the portion of the 3D model of the ring located within the unoccluded area of ​​the ring finger 201 that is not occluded by the aforementioned occlusions can be rendered according to the real-time pose of the ring finger 201, resulting in the 3D ring 205. The ring 205 is then overlaid and displayed on the ring wearing position of the ring finger 201.

[0086] In other embodiments, the wearable component may also be a component for at least partially non-fitted wear on the target finger, such as a nail extension.

[0087] In these embodiments, optionally, the wearable component 3D model includes a first model portion corresponding to the fitting portion that fits with the target finger and a second model portion corresponding to the non-fitting portion that does not fit with the target finger. The electronic device can render the portion of the first model portion in the wearable component 3D model that is located within the real-time unobstructed area of ​​the target finger and the portion of the second model portion that is located within the real-time unobstructed background area of ​​the real-time image according to the real-time posture of the target finger, such as the real-time rotation posture, to obtain a target 3D image. Then, the target 3D image is superimposed on the wearable component wearing position of the target finger in the real-time image to obtain a composite image.

[0088] The real-time unoccluded background area can include the image background of the real-time image and the image area in the real-time image corresponding to the body structure that is not connected to the non-target body structure in the occluder.

[0089] Therefore, in these embodiments, the posture and occlusion of the fingers can be taken into account when adding finger decoration effects, and the finger decoration effects can be added to the unoccluded areas of the fingers.

[0090] In other embodiments of this disclosure, the target body part may include the target head.

[0091] Accordingly, the 3D model of the wearable component can be a 3D model of the component to be worn on the target head, that is, the 3D model of the wearable component is the 3D model corresponding to the wearable component to be worn on the target head.

[0092] In these embodiments, the real-time unobstructed area may include the area in the real-time image where the target head is not obscured by any occlusion.

[0093] Optionally, to account for all possible occlusions of the target head, the occluders can be set to include non-body parts and non-target head body structures other than the target head.

[0094] Non-body part objects can include at least one of the image background in the real-time image and objects other than the human body that obscure any part of the target's head.

[0095] Non-target head body structures can be any body structure other than the target head, based on the first granularity division.

[0096] In some embodiments, the wearable component can be a component for fitting snugly on the target head, such as a headband.

[0097] In these embodiments, optionally, the electronic device can render the portion of the wearable component's 3D model located within the real-time unobstructed area of ​​the target head according to the target head's real-time posture, such as real-time rotation posture, to obtain a target 3D image, and then superimpose the target 3D image onto the wearable component's position on the target head in the real-time image to obtain a composite image.

[0098] In other embodiments, the wearable component may also be a component for wearing entirely non-fittedly on the target head, such as a helmet.

[0099] In these embodiments, optionally, the electronic device can render the portion of the wearable component's 3D model located within the real-time unobstructed area of ​​the target head and the portion located within the real-time unobstructed background area of ​​the real-time image according to the real-time posture of the target head, such as the real-time rotation posture, to obtain a target 3D image. Then, the target 3D image is superimposed on the wearable component's position on the target head in the real-time image to obtain a composite image.

[0100] The real-time unoccluded background area can include the image background of the real-time image and the image area in the real-time image corresponding to the body structure that is not connected to the non-target body structure in the occluder.

[0101] Figure 3 A schematic diagram of another composite image provided by an embodiment of this disclosure is shown.

[0102] like Figure 3 As shown, the synthesized image can be an image including the target head 301, and the wearing component can be a helmet 306. A portion of the target head 301 is occluded by a hand 302, which is connected to the upper limb 303. Therefore, when rendering the helmet's 3D model, based on the real-time pose of the target head 301, the portion of the helmet's 3D model within the unoccluded area of ​​the target head 301 (not occluded by the hand 302), the portion of the helmet's 3D model within the image background 304, and the portion of the helmet's 3D model within the body 305 are rendered to obtain a 3D helmet 306. The helmet 306 is then overlaid on the helmet-wearing position of the target head 301. The image regions corresponding to the image background 304 and the body 305 form the unoccluded background region.

[0103] Therefore, in these embodiments, the head posture and occlusion can be taken into account when adding head decoration effects, and head decoration effects can be added to the unoccluded areas of the head.

[0104] In this embodiment of the disclosure, after acquiring a real-time image of a target body part, a composite image obtained by overlaying a target 3D image onto the target body part in the real-time image can be displayed in real time. The target 3D image is obtained by rendering a 3D model of a wearable component based on the real-time pose and the real-time unoccluded area of ​​the target body part. The real-time pose and the real-time unoccluded area are directly determined based on the real-time image, thereby achieving the purpose of automatically adding a 3D decorative effect with wearable components to the real-time image. Since the pose and occlusion of the body part wearing the decorative effect are taken into account during the process of adding the 3D decorative effect, the integration of the added 3D decorative effect with the original image can be improved, avoiding any visual glitches and thus enhancing the user experience.

[0105] In another embodiment of this disclosure, in order for the electronic device to reliably display the synthetic image, the target 3D image can also be rendered based on the real-time pose and the real-time unoccluded area to obtain the part of the wearable component 3D model that is not occluded by the preset body part model, which can be used to simulate the target body part.

[0106] Optionally, in Figure 1 Before step S120 as shown, the image display method may further include:

[0107] Based on the real-time posture, determine the first depth information of the wearable component's 3D model and the second depth information of the preset body part model;

[0108] Based on the real-time unobstructed area, determine the part of the wearable component's 3D model to be rendered;

[0109] Based on the first depth information and the second depth information, the part of the part to be rendered that has a depth less than the preset body part model is rendered to obtain the target 3D image.

[0110] In this embodiment of the disclosure, the electronic device can acquire the real-time posture of the target body part, and determine the first depth information of the wearable component 3D model and the second depth information of the preset body part model based on the real-time posture. At the same time, based on the real-time unoccluded area, the part of the wearable component 3D model to be rendered is determined, and then based on the first depth information and the second depth information, the part of the part to be rendered with a depth less than the preset body part model is rendered to obtain the target 3D image.

[0111] The real-time attitude can include the real-time rotational attitude of the target body part, and the real-time rotational attitude of the target body part can include the real-time three-dimensional rotational attitude of each joint in the target body part.

[0112] Furthermore, real-time posture can be represented by the real-time three-dimensional rotational posture information of each joint in the target body part. This real-time three-dimensional rotational posture information can include Euler angles or rotation matrices, etc., without limitation. For example, the three-dimensional posture representation of a human hand represents the three-dimensional rotational information of each joint of the fingers, expressed using Euler angles (i.e., the rotation angles of a finger joint around the three axes in three-dimensional space) or rotation matrices.

[0113] Optionally, before determining the depth information of the three-dimensional model of the wearable component based on the real-time pose, the image display method may further include using a pre-trained three-dimensional pose detection model to perform pose detection on the target body part in the real-time image to obtain the real-time pose of the target body part.

[0114] Therefore, in this embodiment of the present disclosure, the electronic device can first perform pose detection on the target body part in the real-time image to obtain the real-time pose of the target body part, and then determine the first depth information of the three-dimensional model of the wearable component and the second depth information of the preset body part model based on the real-time pose of the target body part.

[0115] In some embodiments, the electronic device can synchronously rotate the 3D model of the wearable component and the preset body part model according to the real-time posture of the target body part, so that the model posture of the 3D model of the wearable component and the preset body part model is consistent with the real-time posture of the target body part, and then extract the first depth information of the 3D model of the wearable component and the second depth information of the preset body part model under the model posture.

[0116] In other embodiments, the electronic device may first use a pre-trained feature point detection model to detect feature points of the target body structure to which the target body part belongs in the real-time image, obtain each feature point of the target body structure, and then, based on the real-time pose of the target body part and each feature point of the target body part, synchronously scale, rotate and translate the three-dimensional model of the wearable component and the preset body part model, so that the model pose of the three-dimensional model of the wearable component and the preset body part model are consistent with the real-time pose, real-time size and real-time position of the target body part, thereby extracting the first depth information of the three-dimensional model of the wearable component and the second depth information of the preset body part model.

[0117] In one example, taking the 3D model of the wearable component as a ring model and the target body part as the ring finger, the specific method for scaling, rotating, and translating the 3D model of the wearable component can be as follows:

[0118] The electronic device can generate a 3D rotation matrix M of a ring 3D model based on the real-time posture of the hand to which the ring finger belongs. ring ∈R 3×3The 3D rotation matrix M of the ring 3D model ring ∈R 3×3 It can be determined by the three-dimensional rotation matrix M of the wrist joint. wrist Multiply by the three-dimensional rotation matrix M of the ring finger joint on the left. finger Obtain, i.e., M ring =M wrist M finger .

[0119] By setting the camera intrinsic parameters of the electronic device, such as the field of view (FOV), we can estimate the size of the hand in the real-time image by calculating the length of feature point pairs. Based on this size, we can deduce the depth of the 3D position of the ring model. Then, using the depth of the ring model and the pixel coordinates of the ring finger keypoint, we can calculate the approximate 3D position of the ring model in the camera's coordinate system, i.e., the translation vector V of the ring model. ring ∈R 3×1 .

[0120] Then, the three-dimensional rotation matrix M can be... ring ∈R 3×3 With translation vector V ring ∈R 3×1 By piecing together the matrices, we obtain the 3D rotation and translation matrix of the ring's 3D model. Right now

[0121] Therefore, the three-dimensional rotation and translation matrix of the ring's three-dimensional model can be used. Rotate and translate each pixel in the ring 3D model to obtain a ring 3D model that is consistent with the real-time pose, real-time size and real-time position of the target body part.

[0122] In some embodiments of this disclosure, the real-time unobstructed area may include the area of ​​the target body part that is not obstructed by an obstruction, and the obstruction may include a non-target body structure other than the target body structure to which the target body part belongs.

[0123] In these embodiments, the specific method by which the electronic device determines the part to be rendered of the three-dimensional model of the wearable component based on the real-time unoccluded area may include: firstly, performing image segmentation on the real-time image for the body structure to obtain a target body structure image, a non-target body structure image, and a background image; determining the real-time unoccluded area of ​​the target body part in the target body structure image; and thus, the electronic device can determine the part to be rendered of the three-dimensional model of the wearable component based on the target body structure image.

[0124] When the wearable component is a component designed to fit snugly on a target body part, after obtaining an image of the target body structure, the electronic device can use the portion of the 3D model of the wearable component located in the real-time unoccluded area corresponding to the target body structure image as the part to be rendered.

[0125] When the wearable component is a component for wearing at least partially non-fittedly on a target body part, the 3D model of the wearable component may include a first model part corresponding to the fitted part of the target body part and a second model part corresponding to the non-fitted part that is not fitted to the target body part. After obtaining the target body structure image, the non-target body structure image and the background image, the electronic device may take the part of the first model part that is located in the real-time unobstructed area of ​​the target finger and the part of the second model part that is located in the real-time unobstructed background area of ​​the real-time image as the part to be rendered.

[0126] The real-time unoccluded background area can include the area corresponding to the background image and the area corresponding to the non-target body structure image that is not connected to the non-target body structure in the occluder.

[0127] When the wearable component is designed to be worn entirely non-fitted on the target's head, after obtaining the target body structure image, the non-target body structure image, and the background image, the electronic device can use the portion of the wearable component's 3D model located within the real-time unoccluded area of ​​the target's finger and the portion located within the real-time unoccluded background area of ​​the real-time image as the part to be rendered.

[0128] The real-time unoccluded background area can include the area corresponding to the background image and the area corresponding to the non-target body structure image that is not connected to the non-target body structure in the occluder.

[0129] Figure 4 A schematic diagram of a renderable image region provided in an embodiment of this disclosure is shown.

[0130] like Figure 4 As shown, the target head 401 is occluded by the hand 402. Therefore, the area of ​​the target head 401 not occluded by the hand 402 is the real-time unoccluded area 403 (the area of ​​the target head 401 excluding the shadow portion). The hand 402 is connected to the upper limb 404; therefore, the real-time unoccluded background area can include the areas corresponding to the image background 405 and the body 406. Thus, when the wearable component is a helmet, since the helmet is worn on the target head 401 in a partially fitted manner, the real-time unoccluded area 403 and the real-time unoccluded background area can form a renderable image area, and the part of the helmet 3D model to be rendered can include the portion of the helmet 3D model located within the renderable image area.

[0131] Therefore, electronic devices can simulate the occlusion of 3D decorative effects by non-target body structures, even when the occlusion includes non-target body structures other than the target body structure to which the target body part belongs, thereby improving the integration of the added 3D decorative effects with the original image.

[0132] In other embodiments of this disclosure, the real-time unobstructed area may include the area of ​​the target body part that is not obstructed by an obstruction, and the obstruction may include at least one of a non-body part object and a non-target body structure other than the target body structure to which the target body part belongs.

[0133] Accordingly, based on the real-time unobstructed area, the part of the wearable component's 3D model to be rendered can specifically include:

[0134] The real-time image is segmented according to the target body structure to which the target body part belongs, to obtain the target body structure image;

[0135] In the target body structure image, determine the unoccluded area in real time;

[0136] The part to be rendered is determined based on the real-time unobstructed area.

[0137] In these embodiments, the electronic device can perform image segmentation on the real-time image for the target body structure to which the target body part belongs, obtain the target body structure image, and take the area of ​​the target body part located in the target body structure image as the real-time unoccluded area, and then determine the part to be rendered of the three-dimensional model of the wearable component based on the real-time unoccluded area.

[0138] When the wearable component is a component designed to fit snugly on a target body part, after obtaining an image of the target body structure, the electronic device can use the portion of the 3D model of the wearable component located in the real-time unoccluded area corresponding to the target body structure image as the part to be rendered.

[0139] Figure 5 A schematic diagram of another renderable image region provided by an embodiment of this disclosure is shown.

[0140] like Figure 5As shown, neither the ring finger 501 nor the hand 502 is obscured by other body structures or fingers. Therefore, the entire area of ​​the ring finger 501 on the hand 502 is the real-time unobscured area. When the wearable component is a ring, since the ring is designed to fit snugly on the ring finger 501, the real-time unobscured area can form a renderable image area. The part of the ring's 3D model to be rendered can include the portion of the ring's 3D model located within the renderable image area. The background area of ​​the wearable component outside the hand 502 is not rendered; this background area, as a non-hand area, can be obtained through hand image segmentation.

[0141] In embodiments of this disclosure, when the wearable component is a component other than one for fitting snugly on the target body part, the method for determining the portion to be rendered is similar to the method in embodiments where the occlusion includes a non-target body structure, and will not be described in detail here.

[0142] Therefore, electronic devices can simulate the occlusion of 3D decorative effects by non-body parts and non-target body structures when the occluders include at least one of non-body parts and non-target body structures, thereby improving the integration of the added 3D decorative effects with the original image.

[0143] In some embodiments of this disclosure, the obstruction may include, in addition to non-target body structures other than non-body parts and the target body structure to which the target body part belongs, non-target body parts of the same type as the target body part, such as... Figure 2 As shown.

[0144] Accordingly, determining the real-time unoccluded region in the target body structure image can specifically include:

[0145] The feature points of the target body structure are obtained by performing feature point detection on the target body structure image.

[0146] Based on feature points, determine the area of ​​non-target body parts that occlude the target body parts.

[0147] Based on the occlusion area, determine the real-time unoccluded area in the target body structure image.

[0148] In these embodiments, the electronic device can first use a pre-trained feature point detection model to detect feature points in the target body structure image in the real-time image to obtain each feature point of the target body structure. Then, based on the feature points of the non-target body parts and the target body parts, the occlusion area of ​​the non-target body parts on the target body parts is determined. Then, based on the occlusion area, the real-time unoccluded area in the target body structure image is determined to determine the part to be rendered.

[0149] Specifically, the electronic device can first determine the first and second feature points closest to the wearing position of the wearable component among the feature points corresponding to the target body part. Then, among all feature points corresponding to non-target body parts, it can determine the third and fourth feature points closest to the first feature point, and the fifth and sixth feature points closest to the second feature point. Next, it can calculate the first intermediate point between the first and third feature points, the second intermediate point between the first and fourth feature points, the third intermediate point between the second and fifth feature points, and the fourth intermediate point between the second and sixth feature points. Furthermore, the first, second, third, and fourth intermediate points are divided into two groups, each group including two intermediate points corresponding to the same non-target body part.

[0150] Therefore, the electronic device can generate a parallelogram-shaped occlusion area corresponding to the non-target body part to which each set of intermediate points belongs, based on the line segments formed by connecting each set of intermediate points.

[0151] For example, an electronic device can use the line segment formed by connecting each set of intermediate points as the hypotenuse, and generate a parallelogram-shaped occlusion area based on the long side of a preset length.

[0152] After generating the occlusion area, the electronic device can use the occlusion area to cover the target body structure image, and then use the image area in the target body structure image that is not covered by the occlusion area as the real-time unoccluded area.

[0153] Therefore, electronic devices can simulate the occlusion of 3D decorative effects by non-body parts, non-target body structures, and non-target body parts when the occluders include at least one of non-body parts, non-target body structures, and non-target body parts, thereby improving the integration of the added 3D decorative effects with the original image.

[0154] Figure 6 A schematic diagram of yet another renderable image region provided in an embodiment of this disclosure is shown. Figure 7 A schematic diagram of an occlusion area provided in an embodiment of this disclosure is shown. Figure 8 A schematic diagram of another renderable image region provided by an embodiment of the present disclosure is shown.

[0155] like Figure 6 As shown, the hand 602 to which the ring finger 601 belongs is not obscured by other body structures. Therefore, the real-time unobscured area of ​​the ring finger 601 can be determined in the hand 602.

[0156] Since the ring finger 601 overlaps with the middle finger 603 and the little finger 604, it is necessary to further determine the area of ​​the ring finger 601 that is not covered by the little finger 604 and the middle finger 603 in order to finally determine the real-time uncovered area of ​​the ring finger 601.

[0157] like Figure 7 As shown, the first feature point 605 and the second feature point 606 of the ring finger 601 are the feature points closest to the wearing position of the wearable component. The two feature points closest to the first feature point 605 include the third feature point 607 of the middle finger 603 and the fourth feature point 608 of the little finger 604. The two feature points closest to the second feature point 606 include the fifth feature point 609 of the middle finger 603 and the sixth feature point 610 of the little finger 604. The midpoint between the first feature point 605 and the third feature point 607 is the first midpoint 611, the midpoint between the first feature point 605 and the fourth feature point 608 is the second midpoint 612, the midpoint between the second feature point 606 and the fifth feature point 609 is the third midpoint 613, and the midpoint between the second feature point 606 and the sixth feature point 610 is the fourth midpoint 614. Using the line segment connecting the first midpoint 611 and the third midpoint 613 as the hypotenuse, a first occlusion region 615 of parallelogram shape corresponding to the middle finger 603 is generated according to the long side of the preset length. Using the line segment connecting the second midpoint 612 and the fourth midpoint 614 as the hypotenuse, a second occlusion region 616 of parallelogram shape corresponding to the little finger 604 is generated according to the long side of the preset length.

[0158] like Figure 8 As shown, the first occlusion area 615 and the second occlusion area 616 are superimposed on the hand 602, and the area of ​​the ring finger 601 that is not covered by the first occlusion area 615 and the second occlusion area 616 is taken as the real-time unoccluded area.

[0159] When the wearable component is a ring, since the ring is worn on the ring finger 601 in a completely close fit, the real-time unobstructed area can form a renderable image area, and the part of the ring 3D model to be rendered can include the part of the ring 3D model located within the renderable image area.

[0160] Therefore, in this embodiment of the present disclosure, when the fingers are closed, the occlusion of the three-dimensional decoration effect worn on the ring finger can be simulated by other fingers, thereby improving the integration of the added three-dimensional decoration effect with the original image and preventing the three-dimensional decoration effect from being embedded in the fingers adjacent to the ring finger.

[0161] In this embodiment of the disclosure, the 3D model of the ornament may include a 3D model of the wearable component and a preset body part model. The electronic device can render the portion of the 3D model of the wearable component that is not occluded by the preset body part model based on the real-time posture and the real-time unoccluded area to obtain the target 3D image.

[0162] The preset body part model can be a model pre-defined to simulate the target body part according to actual application needs, and there are no restrictions on this. For example, if the target body part is the head, the preset body part model can be a preset standard head model. As another example, if the target body part is a finger, the preset body part model can be a cylinder or a cuboid, etc.

[0163] Specifically, the 3D model of the wearable component can be worn on a preset body part model in the manner of wearing it on the target body part. The preset body part model can be scaled, rotated, and translated synchronously with the 3D model of the wearable component based on the real-time pose and feature points of the target body part to determine the first depth information of the 3D model of the wearable component and the second depth information of the preset body part model.

[0164] The first depth information may include the first depth at each pixel of the 3D model of the wearable component, and the second depth information may include the second depth at each pixel of the preset body part model.

[0165] The electronic device can compare a first depth at each pixel of the 3D model of the wearable component with a second depth at that pixel, and determine whether the pixel is located within the part to be rendered. If the first depth is less than the second depth and the pixel is located within the part to be rendered, then the pixel of the 3D model of the wearable component is rendered.

[0166] Figure 9 A schematic diagram of a preset occlusion model provided in an embodiment of this disclosure is shown.

[0167] like Figure 9As shown, the wearable component's 3D model can be a ring 3D model 901, and the preset body part model can be a cylinder 902. The cylinder 902 is used to simulate the ring finger, and the ring 3D model 901 can be fitted onto the cylinder 902. The electronic device can first synchronously scale, rotate, and translate the ring 3D model 901 and the cylinder 902 based on the real-time posture and feature points of the finger used to wear the ring. Then, it obtains the first depth of each pixel of the ring 3D model 901 and the second depth of each pixel of the cylinder 902. If the first depth at the same pixel is less than the second depth, it means that the ring 3D model 901 is closer to the image surface at that pixel than the cylinder 902. Therefore, the ring 3D model 901 can be rendered at that pixel. Otherwise, if the first depth at the same pixel is greater than the second depth, the ring 3D model 901 will not be rendered at that pixel.

[0168] Therefore, electronic devices can use preset body part models to render the 3D model of wearable components, simulate the occlusion of the target body part on the 3D decoration effect, and improve the integration of the added 3D decoration effect with the original image.

[0169] In yet another embodiment of this disclosure, the real-time image may be an image of the target body structure to which the target body part belongs. To further avoid continuity errors, the electronic device may also identify the target body part in real time from the real-time image, and display the composite image only when the target body part is identified.

[0170] Specifically, after acquiring a real-time image, the electronic device can first identify whether the target body part is displayed in the target body structure in the real-time image, that is, whether the target body part is displayed in the real-time image. If the target body part is identified as being displayed in the real-time image, the composite image can be displayed; otherwise, the real-time image is displayed.

[0171] Since the real-time images acquired by electronic devices may change at different times, electronic devices need to identify in real time whether the acquired real-time images show target body parts, and then determine the image to be displayed based on the identification results, so as to further avoid the appearance of continuity errors.

[0172] In another embodiment of this disclosure, the target 3D image can be obtained by rendering the functional component 3D model based on the target relative position and real-time pose, and by rendering the wearable component 3D model based on the target relative position, real-time pose, and real-time unoccluded area. The target relative position can be the relative position of the functional component 3D model and the wearable component 3D model in the real-time pose.

[0173] Among them, the 3D model of the functional component can be the 3D model corresponding to the functional component.

[0174] In some embodiments, the functional component may be a decorative component, such as a diamond, a bow, etc.

[0175] In other embodiments, the functional component may also be a component with a function, such as a searchlight, an antenna, etc.

[0176] Specifically, functional components and wearable components can form a complete three-dimensional decorative effect.

[0177] In this embodiment of the disclosure, the electronic device can render the three-dimensional model of the functional component based on the target's relative position and real-time posture, and render the three-dimensional model of the wearable component based on the target's relative position, real-time posture, and real-time unobstructed area to obtain a target three-dimensional image. Then, the target three-dimensional image is superimposed on the target body part in the real-time image to obtain a composite image.

[0178] Furthermore, the 3D model of the decoration can also include a 3D model of wearable components and a 3D model of functional components, which can be arranged according to a preset relative position. The electronic device can synchronously rotate the 3D models of wearable components and functional components according to the real-time posture of the target body part, so that the posture of the 3D models of wearable components and functional components is consistent with the real-time posture of the target body part. It can also obtain the target relative position of the 3D models of functional components and wearable components in the posture consistent with the real-time posture of the target body part. Then, based on the target relative position and real-time posture, the part of the 3D model of functional components that is not occluded by the 3D model of wearable components is rendered, and based on the target relative position, real-time posture, and real-time unoccluded area, the part of the 3D model of wearable components that is not occluded by the 3D model of functional components is rendered to obtain a target 3D image. Then, the target 3D image is superimposed on the target body part in the real-time image to obtain a composite image.

[0179] In some embodiments of this disclosure, Figure 1 Prior to S120, the image display method may further include:

[0180] Based on the real-time attitude, determine the yaw angle of the upper surface of the 3D model of the functional component;

[0181] When the yaw angle of the upper surface is within the first preset angle range, the 3D model of the wearable component and the 3D model of the functional component are rendered to obtain the target 3D image;

[0182] When the yaw angle of the upper surface is within the second preset angle range, the 3D model of the wearable component is rendered to obtain the target 3D image.

[0183] In this embodiment of the disclosure, the electronic device can determine the model posture of the three-dimensional model of the functional component based on the real-time attitude, and then determine the yaw angle of the upper surface of the three-dimensional model of the functional component based on the model posture of the three-dimensional model of the functional component, and determine the preset angle range to which the upper surface yaw angle belongs. If the upper surface yaw angle belongs to the first preset angle range, the three-dimensional model of the wearable component and the three-dimensional model of the functional component are rendered to obtain a target three-dimensional image including the wearable component and the functional component; otherwise, only the three-dimensional model of the wearable component is rendered to obtain a target three-dimensional image including only the wearable component.

[0184] Specifically, the electronic device can synchronously scale, rotate, and translate the 3D models of the functional components and the wearable components according to the real-time attitude, and determine the upper surface yaw angle of the upper surface of the 3D model of the functional components after scaling, rotating, and translating the 3D models of the functional components and the wearable components.

[0185] The upper surface can be the surface of a 3D model of a functional component pre-defined according to actual application requirements, and is not limited here. The first preset angle range can be the angle range pre-defined according to actual application requirements that allows the upper surface to face the direction visible to the user, and is not limited here. The second preset angle range can be the angle range pre-defined according to actual application requirements that allows the upper surface to face the direction away from the user, and is not limited here.

[0186] For example, the first preset angle range can be the angle range of [0°, 100°] along the clockwise direction and the counterclockwise direction, and the second preset angle range can be the angle range other than the first preset angle range.

[0187] Figure 10 A schematic diagram of yet another composite image provided by an embodiment of this disclosure is shown.

[0188] like Figure 10 As shown, the synthesized image can include an image of the ring finger 1001, the wearable component can be a ring 1002, and the functional component can be a diamond 1003. Specifically, by determining the yaw angle of the diamond's upper surface within a first preset range based on the posture of the ring finger 1001, it can be determined that the upper surface of the diamond's 3D model faces the user. Therefore, the diamond's 3D model can be rendered based on the real-time posture of the finger, and the ring's 3D model can be rendered by combining the real-time posture of the finger and the real-time unobstructed area, resulting in the ring 1002 and diamond 1003 in the target 3D image. This target 3D image is then overlaid on the ring wearing position of the ring finger 1001, allowing the user to simultaneously view both the ring effect and the diamond effect.

[0189] Figure 11 A schematic diagram of another composite image provided by an embodiment of the present disclosure is shown.

[0190] like Figure 11 As shown, the synthesized image can be an image including the ring finger 1101, the wearable component can be a ring 1102, and the functional component can be a diamond. Specifically, by determining the yaw angle of the diamond's upper surface within a second preset range based on the posture of the ring finger 1101, it can be determined that the upper surface of the diamond's 3D model faces away from the user. Therefore, only the 3D model of the wearable component can be rendered to obtain the 3D ring 1102, which is then superimposed on the ring finger 1101 at the wearing position, allowing the user to only see the ring effect.

[0191] Optionally, the electronic device can also render the 3D model of the functional component based on the real-time posture of the target body part and the real-time unobstructed background area to obtain the functional component in the target 3D image.

[0192] Specifically, when the yaw angle of the upper surface is within a preset angle range, the electronic device can render the part of the 3D model of the functional component that is in the real-time unobstructed background area to obtain the functional component in the target 3D image, which will not be elaborated here.

[0193] Therefore, electronic devices can further simulate the occlusion of functional components of 3D decorative effects by occluders, thereby improving the integration of the added 3D decorative effects with the original image.

[0194] In summary, the image display method provided in this disclosure can simulate the occlusion of decorative effects at the pixel level through various means, thereby simulating more refined occlusion relationships. When displaying decorative effects, it can greatly improve the realism of any object occluding the decorative effect in the image, improve the integration of the added three-dimensional decorative effect with the original image, avoid screen glitches, enhance the user's immersion, and thus improve the user experience.

[0195] This disclosure also provides an image display device capable of implementing the above-described image display method, as described below. Figure 12 The image display device provided in the embodiments of this disclosure will be described.

[0196] In this embodiment of the disclosure, the image display device can be an electronic device. The electronic device may include mobile phones, tablet computers, desktop computers, laptop computers, in-vehicle terminals, wearable electronic devices, all-in-one computers, smart home devices, and other devices with communication functions, or it may be a device simulated by a virtual machine or simulator.

[0197] Figure 12 A schematic diagram of the structure of an image display device provided in an embodiment of this disclosure is shown.

[0198] like Figure 12 As shown, the image display device 1200 may include an acquisition unit 1210 and a display unit 1220.

[0199] The acquisition unit 1210 can be configured to acquire real-time images of the target body part.

[0200] The display unit 1220 can be configured to display a synthetic image in real time. The synthetic image is an image obtained by superimposing a target 3D image onto a target body part in a real-time image. The target 3D image is obtained by rendering a 3D model of the wearable component based on the real-time pose of the target body part and the real-time unoccluded area. The real-time pose and the real-time unoccluded area are determined based on the real-time image.

[0201] In this embodiment of the disclosure, after acquiring a real-time image of a target body part, a composite image obtained by overlaying a target 3D image onto the target body part in the real-time image can be displayed in real time. The target 3D image is obtained by rendering a 3D model of a wearable component based on the real-time pose and the real-time unoccluded area of ​​the target body part. The real-time pose and the real-time unoccluded area are directly determined based on the real-time image, thereby achieving the purpose of automatically adding a 3D decorative effect with wearable components to the real-time image. Since the pose and occlusion of the body part wearing the decorative effect are taken into account during the process of adding the 3D decorative effect, the integration of the added 3D decorative effect with the original image can be improved, avoiding any visual glitches and thus enhancing the user experience.

[0202] In some embodiments of this disclosure, real-time attitude may include the real-time rotational attitude of a target body part.

[0203] In some embodiments of this disclosure, the real-time unobstructed area may include the area of ​​the target body part that is not obstructed by an obstruction. The obstruction may include at least one of the following: a non-body part object, a non-target body structure other than the target body structure to which the target body part belongs, and a non-target body part of the same type as the target body part.

[0204] In some embodiments of this disclosure, the target body part may include the target finger, the wearable component 3D model may be a 3D model of a component for wearing on the target finger, the real-time unobstructed area may include the area of ​​the target finger in the real-time image that is not obstructed by an obstruction, and the obstruction may include non-body part objects, body structures other than the hand to which the target finger belongs, and fingers other than the target finger.

[0205] In some embodiments of this disclosure, the target 3D image can be rendered based on the real-time pose and the real-time unoccluded area to obtain the part of the wearable component 3D model that is not occluded by the preset body part model, which can be used to simulate the target body part.

[0206] In some embodiments of this disclosure, the image display device 1200 may further include a first processing unit, a second processing unit, and a first rendering unit.

[0207] The first processing unit can be configured to determine the first depth information of the three-dimensional model of the wearable component and the second depth information of the preset body part model based on the real-time posture.

[0208] The second processing unit can be configured to determine the part of the wearable component's 3D model to be rendered based on the real-time unobstructed area.

[0209] The first rendering unit can be configured to render the part of the part to be rendered whose depth is less than the preset body part model based on the first depth information and the second depth information, so as to obtain the target three-dimensional image.

[0210] In some embodiments of this disclosure, the real-time unobstructed area may include the area of ​​the target body part that is not obstructed by an obstruction, and the obstruction may include at least one of a non-body part object and a non-target body structure other than the target body structure to which the target body part belongs.

[0211] Accordingly, the second processing unit may include a first processing subunit, a second processing subunit, and a third processing subunit.

[0212] The first processing subunit can be configured to perform image segmentation on the real-time image based on the target body structure to which the target body part belongs, to obtain the target body structure image.

[0213] The second processing subunit can be configured to determine the unoccluded area in the target body structure image in real time.

[0214] This third processing subunit can be configured to determine the part to be rendered based on the real-time unoccluded area.

[0215] In some embodiments of this disclosure, the obstruction may also include a non-target body part of the same type as the target body part.

[0216] Accordingly, the second processing subunit can be further configured as follows:

[0217] Feature point detection is performed on the target body structure image to obtain the feature points of the target body structure; based on the feature points, the occlusion area of ​​the target body part by non-target body parts is determined; based on the occlusion area, the real-time unoccluded area in the target body structure image is determined.

[0218] In some embodiments of this disclosure, the target 3D image can be obtained by rendering the 3D model of the functional component based on the target relative position and real-time pose, and by rendering the 3D model of the wearable component based on the target relative position, real-time pose, and real-time unoccluded area. The target relative position can be the relative position of the 3D model of the functional component and the 3D model of the wearable component under the stated pose.

[0219] In some embodiments of this disclosure, the image display device 1200 may further include a third processing unit, a second rendering unit, and a third rendering unit.

[0220] The third processing unit can be configured to determine the yaw angle of the upper surface of the 3D model of the functional component based on the real-time attitude.

[0221] The second rendering unit can be configured to render the 3D model of the wearable component and the 3D model of the functional component when the yaw angle of the upper surface is within the first preset angle range, so as to obtain the target 3D image.

[0222] The third rendering unit can be configured to render the 3D model of the wearable component to obtain the target 3D image when the yaw angle of the upper surface is within the second preset angle range.

[0223] It should be noted that, Figure 12 The image display device 1200 shown can perform... Figures 1 to 11 The various steps in the method embodiment shown are implemented. Figures 1 to 11 The processes and effects in the method embodiments shown are not described in detail here.

[0224] This disclosure also provides an image display device, which may include a processor and a memory, the memory being used to store executable instructions. The processor can be used to read the executable instructions from the memory and execute the executable instructions to implement the image display method described above.

[0225] Figure 13 A schematic diagram of the structure of an image display device provided in an embodiment of this disclosure is shown. Refer to the following for details. Figure 13 It shows a schematic diagram of a structure suitable for implementing the image display device 1300 in the embodiments of this disclosure.

[0226] The image display device 1300 in this embodiment can be an electronic device. This electronic device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc.

[0227] It should be noted that, Figure 13 The image display device 1300 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0228] like Figure 13 As shown, the image display device 1300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage device 1308 into a random access memory (RAM) 1303. The RAM 1303 also stores various programs and data required for the operation of the image display device 1300. The processing unit 1301, ROM 1302, and RAM 1303 are interconnected via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0229] Typically, the following devices can be connected to I / O interface 1305: input devices 1306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1309. Communication device 1309 allows image display device 1300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 13 An image display device 1300 with various devices is shown; however, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or included alternatively.

[0230] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the image display method described above.

[0231] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs.

[0232] This disclosure also provides a computer program product, which may include a computer program that, when executed by a processor, causes the processor to implement the image display method described above.

[0233] For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1309, or installed from storage device 1308, or installed from ROM 1302. When the computer program is executed by processing device 1301, it performs the functions defined in the image display method of embodiments of this disclosure.

[0234] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0235] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP, and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0236] The aforementioned computer-readable medium may be included in the aforementioned image display device; or it may exist independently and not assembled into the image display device.

[0237] The aforementioned computer-readable medium carries one or more programs that, when executed by the image display device, cause the image display device to perform:

[0238] Acquire real-time images of the target body parts; display the composite image in real-time. The composite image is obtained by overlaying the target 3D image onto the target body parts in the real-time image. The target 3D image is obtained by rendering the 3D model of the wearable component based on the real-time pose of the target body parts and the real-time unoccluded area. The real-time pose and the real-time unoccluded area are determined based on the real-time image.

[0239] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0240] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0241] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0242] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0243] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0244] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0245] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0246] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image display method, characterized in that, include: Acquire real-time images of the target body part; A composite image is displayed in real time. The composite image is an image obtained by superimposing a target 3D image on the target body part in the real-time image. The target 3D image is obtained by rendering a 3D model of the wearable component based on the real-time pose and the real-time unoccluded area of ​​the target body part. The real-time pose and the real-time unoccluded area are determined based on the real-time image. The occluders of the target body part include non-target body parts of the same type as the target body part. In the target body structure image, determining the real-time unobstructed area includes: Determine the first and second feature points of the target body part that are closest to the wearing position of the wearable component; Based on the first feature point and the feature point closest to the first feature point among the non-target body parts, an intermediate point is determined, and based on the second feature point and the feature point closest to the second feature point among the non-target body parts, an intermediate point is determined. The intermediate points belonging to the non-target body parts on the same side are connected to obtain a line segment formed by connecting the intermediate points. An occlusion region is generated based on the line segment, and the occlusion region is used to cover the target body structure image. The image area in the target body structure image that is not covered by the occlusion region is taken as the real-time unoccluded area.

2. The method according to claim 1, characterized in that, The real-time unobstructed area also includes the area where the target body part is not obscured by any obstruction, and the obstruction includes at least one of non-body part objects and non-target body structures other than the target body structure to which the target body part belongs.

3. The method according to claim 2, characterized in that, The target body part includes the target finger, and the wearable component 3D model is a 3D model of a component for wearing on the target finger.

4. The method according to claim 1, characterized in that, The target 3D image is obtained by rendering the part of the wearable component 3D model that is not occluded by the preset body part model based on the real-time pose and the real-time unoccluded area. The preset body part model is used to simulate the target body part.

5. The method according to claim 4, characterized in that, Prior to the real-time display of the synthesized image, the method further includes: Based on the real-time posture, determine the first depth information of the wearable component's 3D model and the second depth information of the preset body part model; Based on the real-time unobstructed area, determine the part of the wearable component's 3D model to be rendered; Based on the first depth information and the second depth information, the portion of the part to be rendered that has a depth less than the preset body part model is rendered to obtain the target 3D image.

6. The method according to claim 5, characterized in that, The real-time unobstructed area includes the area of ​​the target body part that is not obstructed by any obstruction, and the obstruction includes at least one of non-body part objects and non-target body structures other than the target body structure to which the target body part belongs; The step of determining the part to be rendered of the 3D model of the wearable component based on the real-time unobstructed area includes: The real-time image is segmented according to the target body structure to which the target body part belongs, to obtain a target body structure image; In the target body structure image, determine the real-time unobstructed area; The part to be rendered is determined based on the real-time unobstructed area.

7. The method according to claim 1, characterized in that, The target 3D image is obtained by rendering the functional component 3D model based on the target relative position and the real-time posture, and by rendering the wearable component 3D model based on the target relative position, the real-time posture, and the real-time unoccluded area. The target relative position is the relative position of the functional component 3D model and the wearable component 3D model under the real-time posture.

8. The method according to claim 7, characterized in that, Prior to the real-time display of the synthesized image, the method further includes: Based on the real-time attitude, determine the yaw angle of the upper surface of the three-dimensional model of the functional component; When the yaw angle of the upper surface is within a first preset angle range, the 3D model of the wearable component and the 3D model of the functional component are rendered to obtain the target 3D image. When the yaw angle of the upper surface is within the second preset angle range, the three-dimensional model of the wearable component is rendered to obtain the target three-dimensional image.

9. The method according to claim 1, characterized in that, The real-time attitude includes the real-time rotational attitude of the target body part.

10. An image display device, characterized in that, include: The acquisition unit is configured to acquire real-time images of the target body part; The display unit is configured to display a synthetic image in real time. The synthetic image is an image obtained by superimposing a target 3D image onto the target body part in the real-time image. The target 3D image is obtained by rendering a 3D model of the wearable component based on the real-time pose and the real-time unoccluded area of ​​the target body part. The real-time pose and the real-time unoccluded area are determined based on the real-time image. The occluders of the target body part include non-target body parts of the same type as the target body part. In the target body structure image, determining the real-time unobstructed area includes: Determine the first and second feature points of the target body part that are closest to the wearing position of the wearable component; Based on the first feature point and the feature point closest to the first feature point among the non-target body parts, an intermediate point is determined, and based on the second feature point and the feature point closest to the second feature point among the non-target body parts, an intermediate point is determined. The intermediate points belonging to the non-target body parts on the same side are connected to obtain a line segment formed by connecting the intermediate points. An occlusion region is generated based on the line segment, and the occlusion region is used to cover the target body structure image. The image area in the target body structure image that is not covered by the occlusion region is taken as the real-time unoccluded area.

11. An image display device, characterized in that, include: processor; Memory, used to store executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the image display method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the image display method according to any one of claims 1-9.

Images