Display method, device and first electronic device
By acquiring the first-view image of the XR glasses and the position modeling information of the target object, an image including the target object is generated and displayed, which solves the problem that XR glasses cannot present occluded objects and realizes the localization of occluded objects.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- VIVO MOBILE COMM CO LTD
- Filing Date
- 2022-12-27
- Publication Date
- 2026-07-21
AI Technical Summary
XR glasses cannot display the location of obscured objects, making it impossible for users to know the specific location of the obscured object within their current field of view.
By acquiring the first-view image from the first camera, the location information and modeling information of the target object, a second image including the target object is generated and displayed.
It helps users locate occluded target objects, improving users' understanding of real-world physical scenes.
Smart Images
Figure CN115840552B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image display, and specifically relates to a display method, device and first electronic device. Background Technology
[0002] Extended Reality (XR) refers to an interactive environment created by combining a real physical world scene with a virtual application scene through computer technology and electronic devices. When using electronic devices, the devices can combine the real physical world scene as the background and the virtual application scene content as the foreground to present to the user.
[0003] However, taking electronic devices as XR glasses as an example, when users use XR glasses, the XR glasses can only present objects in the real physical world scene captured by the camera in real time. As a result, some obscured objects cannot be presented by the XR glasses, causing users to be unable to know the position of the obscured object within the current field of view. Summary of the Invention
[0004] The purpose of this application is to provide a display method, apparatus, and first electronic device that can solve the problem that electronic devices cannot help users locate obscured objects.
[0005] In a first aspect, embodiments of this application provide a display method, the method comprising: acquiring a first image corresponding to a first viewpoint of a first camera; acquiring position information and modeling information of a target object, wherein the target object is occluded by a first object within the first viewpoint, and the modeling information is used to establish an object model of the target object; and generating and displaying a second image based on the position information and modeling information of the target object, and the first image, wherein the second image includes the target object.
[0006] Secondly, embodiments of this application provide a display device, which includes an acquisition module and a processing module. The acquisition module is configured to acquire a first image corresponding to a first viewpoint of a first camera; and to acquire position information and modeling information of a target object, wherein the target object is occluded by a first object within the first viewpoint, and the modeling information is used to establish an object model of the target object. The processing module is configured to generate and display a second image based on the position information and modeling information of the target object, and the first image, wherein the second image includes the target object.
[0007] Thirdly, embodiments of this application provide a first electronic device, which includes a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.
[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.
[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.
[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.
[0011] In this embodiment, the first electronic device can acquire a first image corresponding to a first viewpoint of the first camera; and acquire the location information and modeling information of a target object, which is occluded by a first object within the first viewpoint. Based on the location information and modeling information of the target object, and the first image, a second image is generated and displayed, the second image including the target object. In this solution, since the first electronic device can acquire the first image corresponding to the first viewpoint of the first camera, and the location information and modeling information of the target object occluded by the first object within the first viewpoint, the first electronic device can generate and display a second image including the target object based on the acquired location information and modeling information of the target object, and the first image. Thus, the first electronic device can help the user locate the occluded target object within the first viewpoint. Attached Figure Description
[0012] Figure 1 This is a schematic diagram of a display method provided in an embodiment of this application;
[0013] Figure 2 This is one of the schematic diagrams illustrating an example of displaying a target object provided in this application embodiment;
[0014] Figure 3 This is a second schematic diagram illustrating an example of displaying a target object provided in this application embodiment;
[0015] Figure 4 This is a schematic diagram of the structure of a display device provided in an embodiment of this application;
[0016] Figure 5 This is one of the hardware structure diagrams of a first electronic device provided in the embodiments of this application;
[0017] Figure 6 This is a second schematic diagram of the hardware structure of a first electronic device provided in an embodiment of this application. Detailed Implementation
[0018] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0019] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0020] The display method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.
[0021] The display method in this application embodiment can be applied to scenarios where electronic devices display obscured objects.
[0022] Currently, electronic devices (e.g., XR glasses) sometimes blend the content of the application scene with the real physical world scene before displaying it. This is common in the boundary systems of virtual reality (VR) devices or the visual experience of augmented reality (AR). Specifically, taking XR glasses as an example, when blending application scene content with the real physical world scene, XR glasses typically use the real physical world scene as the background and the application scene content as the foreground to achieve a realistic human visual experience. However, in this method, XR glasses can only present objects in the real physical world scene captured by the camera in real time. Some occluded objects cannot be displayed to the user, thus preventing the user from knowing the position of the occluded object within their current field of view.
[0023] In this embodiment of the application, when the target object (i.e. the occluded object) is outside the first viewpoint range of the first camera, the XR glasses can acquire the first image corresponding to the first viewpoint of the first camera, as well as the position information and modeling information of the target object occluded by the first object in the first viewpoint. Based on the acquired position information and modeling information of the target object, and the first image, a second image including the target object is generated and displayed. In this way, the XR glasses can help the user locate the occluded target object in the first viewpoint.
[0024] The display method provided in this application is executed by a display device. This display device can be a first electronic device or a functional control module within the first electronic device; this application does not limit this. The technical solution provided in this application will be described below using a first electronic device as an example.
[0025] This application provides a display method. Figure 1 A flowchart illustrating a display method provided in an embodiment of this application is shown. Figure 1 As shown, the display method provided in this application embodiment may include the following steps 201 to 203.
[0026] Step 201: The first electronic device acquires the first image corresponding to the first viewpoint of the first camera.
[0027] Optionally, in this embodiment of the application, the first camera is the camera of the first electronic device.
[0028] Optionally, in this embodiment of the application, the first image can be an image captured by the first electronic device from a first viewpoint, i.e., a two-dimensional image.
[0029] Optionally, in this embodiment of the application, the first image may be a three-dimensional image constructed by the first electronic device based on a two-dimensional image captured by the first camera within a first viewpoint.
[0030] For example, the first electronic device can convert a two-dimensional image captured by the first camera from a first perspective into a three-dimensional image using Structure From Motion (SFM) technology.
[0031] Step 202: The first electronic device acquires the location information and modeling information of the target object.
[0032] In this embodiment of the application, the target object is occluded by a first object within the first viewpoint, and the modeling information is used to establish the object model of the target object.
[0033] Optionally, in the embodiments of this application, the target object may include at least one of the following: other users, plants, animals, buildings, electronic products, tables and chairs, etc.
[0034] For example, the first object mentioned above may include at least one of the following: other users, plants, animals, buildings, electronic products, tables and chairs, etc.
[0035] It should be noted that when the target object is obscured by the first object within the first viewpoint, it can be understood that the target object is outside the first viewpoint of the first camera, or that the target object is not included in the display scene of the first electronic device, or that the target object is not included in the image within the first viewpoint captured by the first electronic device through the first camera. In other words, the user cannot observe the target object through the display screen of the first electronic device.
[0036] Optionally, in this embodiment of the application, the first electronic device can acquire an image captured by the second camera, and then obtain the location information of the target object based on the image and the location information of the second camera.
[0037] For example, the second camera may be any of the following: a camera of the target electronic device connected to the first electronic device, a webcam, a local camera, etc.
[0038] For example, the distance between the second camera and the first electronic device is less than or equal to a preset threshold.
[0039] For example, the target electronic device mentioned above is a mobile electronic device (such as a mobile phone, tablet computer, laptop computer, PDA, etc.), or it can be a non-mobile electronic device (such as a personal computer, television, etc.).
[0040] For example, when a user uses the first electronic device, the user can trigger the first electronic device to establish a connection with the second camera or the target electronic device corresponding to the second camera, so as to obtain the scene in the second view of the second camera in real time, that is, to obtain the image captured by the second camera based on the second view in real time.
[0041] For example, the target object is located in the second view of the second camera, and it is understood that the image captured by the second camera includes the target object.
[0042] For example, after the first electronic device obtains an image of the target object captured by the second camera, it can analyze the image to obtain the position information of the target object relative to the second camera. Then, the first electronic device can calculate the position information of the target object relative to the first electronic device based on the position information of the second camera and the position information of the target object relative to the second camera.
[0043] Optionally, in this embodiment of the application, the first electronic device may obtain modeling information of the target object based on the image captured by the second camera.
[0044] For example, the modeling information of the target object mentioned above includes the data of the target object rendering, including color data, material data, etc.
[0045] For example, after the first electronic device obtains an image of the target object captured by the second camera, it can use image modeling technology to process the image to obtain modeling information of the target object.
[0046] For example, the first electronic device can directly obtain pre-stored modeling information of the target object.
[0047] Optionally, in this embodiment of the application, the first electronic device may obtain at least one of the location information of the target object and the modeling information of the target object based on the map data of the target map.
[0048] For example, the target map described above contains target objects.
[0049] For example, the map data for the target map mentioned above can be a pre-prepared map, either locally or online.
[0050] data.
[0051] For example, the first electronic device can obtain the name of the target object, and then query the map data of the target map based on the name of the target object to obtain the location information of the target object.
[0052] For example, the first electronic device can query the map data of the target map according to the name of the target object to obtain some image information associated with the target object. Then, the first electronic device can use image modeling technology to obtain the modeling information of the target object based on this image information.
[0053] For example, the first electronic device can directly obtain the modeling information of the target object from the map data of the target map based on the name of the target object, and the modeling information of the target object is pre-stored in the map data.
[0054] Step 203: The first electronic device, based on the location information and modeling information of the target object, and the first...
[0055] The image is used to generate and display a second image.
[0056] In this embodiment of the application, the second image includes the target object.
[0057] Optionally, in this embodiment of the application, the first electronic device may process the first image based on the location information and modeling information of the target object to generate and display the second image.
[0058] For example, the second image described above can be understood as a perspective image of the target object, which is
[0059] It is obtained by performing perspective projection on the object model of the target object onto the first object.
[0060] Optionally, in this embodiment of the application, after the first electronic device generates and displays the second image, some resources of the first electronic device can be released, such as memory, central processing unit (CPU) usage, etc.
[0061] Optionally, in this embodiment of the application, after the first electronic device generates and displays the second image, if the window size of the second image completely covers the foreground application of the first electronic device, the first electronic device may pause the running of the foreground application.
[0062] For example, the window size for displaying the second image can be the system default size, and the user can adjust the window size.
[0063] Optionally, in the embodiments of this application, the first electronic device generating and displaying a second image including the target object can be understood as entering a perspective mode to display the occluded target object from a perspective.
[0064] For example, a user can input information into the first electronic device to trigger it to exit perspective mode, i.e., cancel the display of the second image.
[0065] For example, the above input can be any of the following: voice input, key input, gesture input, etc., which can be determined according to actual usage needs, and the embodiments of the present invention do not limit it.
[0066] For example, users can trigger the first electronic device to exit perspective mode by using voice input such as "exit perspective mode" or "cancel location of XX item".
[0067] For example, users can input physical buttons on commonly used objects such as mobile phones, headphones, and watches associated with the first electronic device, or perform specific gesture inputs to trigger the first electronic device to exit the perspective mode.
[0068] For example, when the first electronic device detects that the target object is no longer obscured by other objects, the first electronic device can actively exit the perspective mode.
[0069] For example, after the first electronic device exits perspective mode, it can reload the resources.
[0070] For example, after the first electronic device exits perspective mode, if the foreground application is in a paused state, the first electronic device can restart the foreground application.
[0071] This application provides a display method in which a first electronic device can acquire a first image corresponding to a first viewpoint of a first camera; and acquire the position information and modeling information of a target object, wherein the target object is occluded by a first object within the first viewpoint. Based on the position information and modeling information of the target object, and the first image, a second image is generated and displayed, wherein the second image includes the target object. In this solution, since the first electronic device can acquire the first image corresponding to the first viewpoint of the first camera, and the position information and modeling information of the target object occluded by the first object within the first viewpoint, the first electronic device can generate and display a second image including the target object based on the acquired position information and modeling information of the target object, and the first image. Thus, the first electronic device can help the user locate the occluded target object within the first viewpoint.
[0072] Optionally, in this embodiment of the application, the first camera is the camera of the first electronic device, and the target object is the second electronic device. The above step 202 can be implemented by the following step 202a.
[0073] Step 202a: When the second electronic device receives the information, the first electronic device acquires the location information and modeling information of the second electronic device.
[0074] Optionally, in the embodiments of this application, the second electronic device can be a mobile electronic device or a non-mobile electronic device.
[0075] Optionally, in this embodiment of the application, when the user uses the first electronic device, the user can trigger the first electronic device to establish a connection with the second electronic device (e.g., a wired connection or a wireless connection) so that when the second electronic device receives information, the user can determine that the second electronic device has received the information, thereby obtaining the location information and modeling information of the second electronic device.
[0076] For example, when the first electronic device establishes a connection with the second electronic device, the first electronic device can directly obtain the location information of the second electronic device (i.e., the target object).
[0077] Optionally, in this embodiment of the application, the above information may be voice calls, video calls, notifications, etc.
[0078] In this embodiment of the application, when the second electronic device receives information, the first electronic device can obtain the location information and modeling information of the second electronic device, and then generate and display the second image based on the location information and modeling information of the second electronic device and the first image. In this way, it can quickly help the user locate the second electronic device to view the received information.
[0079] Optionally, in the embodiments of this application, step 202 above can be specifically implemented by step 202b below.
[0080] Step 202b: If the acquired user voice information includes the target object, the first electronic device acquires the location information and modeling information of the target object.
[0081] Optionally, in this embodiment of the application, when a user needs to locate certain obscured items, i.e. target objects, the user can make voice input to the first electronic device to trigger the first electronic device to obtain the location information and modeling information of the target objects, thereby generating and displaying a second image including the target objects based on the obtained location information and modeling information of the target objects and the first image.
[0082] For example, when a user needs to locate a target object, such as an item named "XX", the user can use voice input such as "Help me find the XX item", "Locate the XX item", or "Where is the XX item?" to trigger the first electronic device to obtain the location information and modeling information of the target object.
[0083] In this embodiment of the application, when the user voice information recognized by the first electronic device includes the target object, the first electronic device can obtain the location information and modeling information of the target object, and then generate and display the second image based on the location information and modeling information of the target object and the first image. In this way, it can quickly help the user locate the target object.
[0084] Optionally, in this embodiment of the application, when the user needs to locate the target object, the first electronic device can perform key input or gesture input, etc., which can be determined according to the actual use needs, and the embodiment of the present invention does not limit it.
[0085] For example, users can input physical buttons on commonly used objects such as mobile phones, headphones, and watches associated with the first electronic device, or perform specific gesture inputs, to trigger the first electronic device to obtain the location information and modeling information of the target object.
[0086] Optionally, in the embodiments of this application, step 203 can be implemented by step 203a as described below.
[0087] Step 203a: If it is found that the target object poses a security risk to the user, the first electronic device generates and displays a second image based on the location information and modeling information of the target object, as well as the first image.
[0088] Optionally, in this embodiment of the application, when it is determined that the target object is outside the first view of the first camera, that is, when the target object is obscured, the first electronic device can determine that the target object poses a safety hazard to the user.
[0089] For example, if the image captured in real time by the first electronic device does not include the target object and the target object is within the second view of the second camera, the first electronic device can determine that the target object is occluded.
[0090] It is understandable that when the first electronic device detects that the target object is within the second view of the second camera, but the target object is not included in the image captured by the first camera within the first view, the first electronic device can determine that the target object is outside the first view of the first camera. That is, the first electronic device is about to approach the target object, but the target object is blocked by other objects. Thus, the first electronic device can determine that the target object poses a safety hazard to the user.
[0091] For example, such as Figure 2 As shown in (A), the current environment includes object 11 and tree 12, and the first camera 13 can capture the corresponding first-view image 14; as Figure 2 As shown in (B) in the image, the object 11 is included in the image 14. If the first electronic device detects that the second viewpoint corresponding to the second camera includes the tree 12, the first electronic device can determine that the first electronic device is about to approach the tree 12. However, the tree 12 is blocked by other objects, so the first electronic device can determine that the target object poses a safety hazard to the user.
[0092] In this embodiment of the application, when a target object is identified as posing a security risk to the user, the first electronic device can generate and display a second image based on the acquired location information and modeling information of the target object, as well as the first image, thereby quickly helping the user locate the target object and avoid security risks.
[0093] Optionally, in this embodiment of the application, when the target object is a second electronic device, when the second electronic device establishes a connection with the first electronic device and receives information, the first electronic device can obtain the location information of the second electronic device to detect whether the second electronic device is outside the first view of the first camera based on the location information of the second electronic device. When it is determined that the second electronic device is outside the first view of the first camera, a second image including the second electronic device is generated and displayed based on the obtained location information and modeling information of the second electronic device, as well as the first image.
[0094] For example, the first electronic device can combine the location information of the second electronic device with the image captured by the first camera within a first field of view to detect whether the second electronic device is outside the first field of view of the first camera.
[0095] For example, if the first electronic device detects that the second electronic device is not included in the image captured by the first camera within the first field of view, and at the same time, the first electronic device determines that the second electronic device is within the preset range of the first electronic device based on the location information of the second electronic device, then the first electronic device can determine that the second electronic device is outside the first field of view of the first camera.
[0096] Optionally, in the embodiments of this application, step 203 can be implemented by steps 203b and 203c as described below.
[0097] Step 203b: The first electronic device adds an image of the target object to the first image based on the location information and modeling information of the target object to generate a third image.
[0098] Optionally, in this embodiment of the application, when the first image is a three-dimensional image constructed by the first electronic device based on a two-dimensional image captured by the first camera within a first perspective, the first electronic device may add a three-dimensional image of the target object to the first image based on the location information and modeling information of the target object to generate a third image.
[0099] It can be understood that the aforementioned third image includes a three-dimensional image of the target object, and the third image is a three-dimensional image.
[0100] For example, the third image mentioned above is a three-dimensional image corresponding to the first perspective.
[0101] For example, after the first electronic device obtains the first image, the first electronic device can determine the depth information of the target object in the first image based on the position information of the target object; then, the first electronic device can obtain a three-dimensional image of the target object based on the modeling information and depth information of the target object; finally, the first electronic device can obtain the placement direction of the target object in the real physical world scene based on the target object in the second view of the second camera, so that the first electronic device can add the three-dimensional image of the target object to the first image according to the direction of the target object in the real physical world scene to obtain the third image.
[0102] For example, assuming the target object is a "car", when the car is not in the first view of the first camera but in the second view of the second camera, the first electronic device can determine that the car is obscured by other objects in the first view; and when the right side of the car is displayed in the second view of the second camera, it can be determined that the car is placed in the real physical world scene with its right side facing the second camera. Therefore, the first electronic device can add the 3D image of the car to the first image with its right side facing the second camera, that is, according to the placement direction of the vehicle in the real physical world scene, add the 3D image of the vehicle to the first image to generate a third image.
[0103] It is understandable that after generating the third image, the first electronic device can obtain the relative position of the target object in the scene within the first viewpoint based on the third image.
[0104] Step 203c: The first electronic device processes the third image based on a first viewpoint to generate and display a second image.
[0105] Optionally, in this embodiment of the application, the first electronic device can capture images within a first viewpoint using a first camera, and then process the third image and the images within the first viewpoint captured by the first camera based on the first viewpoint to generate and display a second image including the target object.
[0106] For example, the first electronic device may perform perspective projection on the third image and the image captured by the first camera within the first perspective, based on a first perspective, to generate and display a second image including the target object.
[0107] For example, the second image described above can be understood as a perspective image of the target object, which is obtained by perspective projection of the object model of the target object onto the first object.
[0108] Optionally, in this embodiment of the application, the first electronic device may project the shape of the target object in the third image onto the first image based on a first viewpoint, so as to generate and display a second image including the target object.
[0109] For example, the first electronic device may perform image rendering processing on the third image and the first image based on a first viewpoint to generate and display a second image including the target object.
[0110] For example, a first electronic device may synthesize a third image and a first image using an Open (Graphics Library, GL) based on a first viewpoint to generate and display a second image including a target object.
[0111] For example, combined with Figure 2 The first electronic device can construct a first image corresponding to the first viewpoint based on image 14. Then, based on the position and modeling information of tree 12, the electronic device can add a 3D image of tree 12 to the first image to generate a third image. Thus, the first electronic device can obtain the relative position of tree 12 in the scene within the first viewpoint based on the third image. Furthermore, the first electronic device can perform perspective projection on the third image and the first image based on the first viewpoint to obtain a perspective image 15 of tree 12 within the first viewpoint, i.e., the second image, as shown below. Figure 3 As shown, tree 12 is displayed in perspective on object 11, making it easy for users to quickly locate tree 12.
[0112] Figure 4 A schematic diagram of a possible structure of the display device involved in an embodiment of this application is shown. For example... Figure 4 As shown, the display device 70 may include an acquisition module 71 and a processing module 72.
[0113] The acquisition module 71 is used to acquire a first image corresponding to the first viewpoint of the first camera; and to acquire the position information and modeling information of the target object, which is occluded by a first object within the first viewpoint. The modeling information is used to establish an object model of the target object. The processing module 72 is used to generate and display a second image based on the position information and modeling information of the target object, as well as the first image. The second image includes the target object.
[0114] This application provides a display device. Since the display device can acquire a first image corresponding to a first viewpoint of a first camera, as well as the position information and modeling information of a target object occluded by a first object within the first viewpoint, the display device can generate and display a second image including the target object based on the acquired position information and modeling information of the target object and the first image. In this way, the display device can help the user locate the occluded target object within the first viewpoint.
[0115] In one possible implementation, the first camera is the camera of the first electronic device, the target object is the second electronic device, and the acquisition module 71 is specifically used to acquire the location information and modeling information of the second electronic device when the second electronic device receives information.
[0116] In one possible implementation, the acquisition module 71 is specifically used to acquire the location information and modeling information of the target object when the acquired user voice information includes the target object.
[0117] In one possible implementation, the processing module 72 is specifically used to generate and display a second image based on the location information and modeling information of the target object, as well as the first image, when it is identified that the target object poses a security risk to the user.
[0118] In one possible implementation, the processing module 72 is specifically used to add an image of the target object to the first image based on the location information and modeling information of the target object to generate a third image; and to process the third image based on the first viewpoint to generate and display a second image.
[0119] The display device in this application embodiment can be a first electronic device or a component within the first electronic device, such as an integrated circuit or a chip. The first electronic device can be a terminal or other devices besides a terminal. For example, the first electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), AR / VR device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the scope of the device.
[0120] The display device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.
[0121] The display device provided in this application embodiment can implement the various processes implemented in the above method embodiments, and will not be described again here to avoid repetition.
[0122] Optionally, such as Figure 5 As shown, this application embodiment also provides an electronic device 900, including a processor 901 and a memory 902. The memory 902 stores a program or instructions that can run on the processor 901. When the program or instructions are executed by the processor 901, they implement the various steps of the above method embodiments and can achieve the same technical effect. To avoid repetition, they will not be described again here.
[0123] It should be noted that the first electronic device in the embodiments of this application includes the mobile electronic device and the non-mobile electronic device described above.
[0124] Figure 6 A schematic diagram of the hardware structure of a first electronic device according to an embodiment of this application.
[0125] The first electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.
[0126] Those skilled in the art will understand that the first electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The structure of the first electronic device shown does not constitute a limitation on the first electronic device. The first electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.
[0127] The processor 110 is used to acquire a first image corresponding to a first viewpoint of a first camera; and to acquire the position information and modeling information of a target object, wherein the target object is occluded by a first object within the first viewpoint, and the modeling information is used to establish an object model of the target object; and to generate and display a second image based on the position information and modeling information of the target object, and the first image, wherein the second image includes the target object.
[0128] This application provides a first electronic device. Since the first electronic device can acquire a first image corresponding to a first viewpoint of a first camera, as well as the position information and modeling information of a target object occluded by a first object within the first viewpoint, the first electronic device can generate and display a second image including the target object based on the acquired position information and modeling information of the target object and the first image. In this way, the first electronic device can help the user locate the occluded target object within the first viewpoint.
[0129] Optionally, the first camera is the camera of the first electronic device, the target object is the second electronic device, and the processor 110 is specifically used to obtain the location information and modeling information of the second electronic device when the second electronic device receives information.
[0130] Optionally, the processor 110 is specifically used to acquire the location information and modeling information of the target object when the acquired user voice information includes the target object.
[0131] Optionally, the processor 110 is specifically used to generate and display a second image based on the location information and modeling information of the target object, as well as the first image, when it is identified that the target object poses a security risk to the user.
[0132] Optionally, the processor 110 is specifically used to add an image of the target object to the first image based on the location information and modeling information of the target object to generate a third image; and to process the third image based on the first viewpoint to generate and display a second image.
[0133] The first electronic device provided in this application embodiment can implement all the processes implemented in the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.
[0134] For details on the beneficial effects of the various implementation methods in this embodiment, please refer to the beneficial effects of the corresponding implementation methods in the above method embodiments. To avoid repetition, these will not be repeated here.
[0135] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.
[0136] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback function, image playback function, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. Non-volatile memory...
[0137] The memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), or dynamic random access memory (DRAM).
[0138] DRAM, Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 109 in this embodiment includes, but is not limited to, these and any other suitable types of memory.
[0139] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface and applications, etc., and the modem processor mainly handles wireless communication signals, such as a baseband processor.
[0140] It is understandable that the aforementioned modem processor may not be integrated into the processor 110.
[0141] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.
[0142] The processor is the processor in the first electronic device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.
[0143] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.
[0144] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0145] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above method embodiments and achieve the same technical effects. To avoid repetition, it will not be described again here.
[0146] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0148] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A display method, characterized in that, The method includes: Acquire the first image corresponding to the first viewpoint of the first camera; The location information and modeling information of the target object are obtained. The target object is occluded by a first object within the first viewpoint. The modeling information is used to establish an object model of the target object. Based on the location information and modeling information of the target object, and the first image, a second image is generated and displayed. The second image includes the target object and is a perspective image of the target object. The acquisition of the target object's location information and modeling information includes: If the acquired user voice information includes the target object, the location information and modeling information of the target object are acquired.
2. The method according to claim 1, characterized in that, The step of generating and displaying a second image based on the location information and modeling information of the target object, and the first image, includes: If the target object is identified as posing a security risk to the user, a second image is generated and displayed based on the target object's location information and modeling information, as well as the first image.
3. The method according to claim 1 or 2, characterized in that, The step of generating and displaying a second image based on the location information and modeling information of the target object, and the first image, includes: Based on the location information and modeling information of the target object, an image of the target object is added to the first image to generate a third image; Based on the first perspective, the third image is processed to generate and display a second image.
4. A display device, characterized in that, The display device includes: an acquisition module and a processing module; The acquisition module is used to acquire a first image corresponding to a first viewpoint of a first camera; and to acquire the position information and modeling information of a target object, wherein the target object is occluded by a first object within the first viewpoint, and the modeling information is used to establish an object model of the target object; The processing module is used to generate and display a second image based on the location information and modeling information of the target object and the first image, wherein the second image includes the target object and is a perspective image of the target object; The acquisition module is specifically used to acquire the location information and modeling information of the target object when the acquired user voice information includes the target object.
5. The apparatus according to claim 4, characterized in that, The processing module is specifically used to generate and display a second image based on the location information and modeling information of the target object and the first image when it is identified that the target object poses a security risk to the user.
6. The apparatus according to claim 4 or 5, characterized in that, The processing module is specifically used to add an image of the target object to the first image based on the location information and modeling information of the target object to generate a third image; and to process the third image based on the first viewpoint to generate and display a second image.
7. A first electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the display method as described in any one of claims 1-3.