A virtual display processing method, device, equipment and medium

Abstract

The application discloses a virtual display processing method, a virtual display method and related devices, and the method comprises the steps of obtaining a space model of a target environment and space data corresponding to different positions in the space model; obtaining a target display position of a preset virtual object in the space model; obtaining target space data corresponding to the preset virtual object based on the space data corresponding to the target display position, wherein the target space data is used to determine the display position of the preset virtual object on a user terminal when the user is in the target environment; and through the above method, the application can configure the target space data for the preset virtual object, so as to display the preset virtual object at the position corresponding to the target space data in the subsequent process, and realize the reproduction of the virtual object at the fixed position.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular to a method, apparatus, device, and medium for processing virtual displays. Background Technology

[0002] Augmented Reality (AR) technology is a technique that cleverly integrates virtual information with the real world. It simulates computer-generated text, images, spatial feature point models, music, videos, and other virtual information, displaying both virtual information and real-time images of the real world simultaneously on the user's device. The two types of information complement each other, thus "enhancing" the real world. For example, users can see real-time images of the real world and virtual AR objects placed within them on their devices.

[0003] During the long-term research and development process, the applicant of this application discovered that in the prior art, users can choose to place AR objects in real-time images of the real world using AR applications, but cannot locate the AR objects. Once the currently used AR application is exited, and then the application is used again, the previously placed AR objects cannot be seen at that location, and the AR objects cannot be reproduced. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a virtual display processing method, apparatus, device, and medium.

[0005] To solve the above-mentioned technical problems, one technical solution adopted in this application is: to provide a virtual display processing method, the method comprising: acquiring a spatial model of a target environment and spatial data corresponding to different positions in the spatial model; obtaining a target display position of a preset virtual object in the spatial model; and obtaining target spatial data corresponding to the preset virtual object based on the spatial data corresponding to the target display position, wherein the target spatial data is used to determine the display position of the preset virtual object on the user terminal when the user is located in the target environment.

[0006] Therefore, by determining the display position of the preset virtual object in the spatial model of the target environment, and then determining the target spatial data corresponding to the preset virtual object in the target environment, it is possible to configure the target spatial data for the preset virtual object, that is, to configure the corresponding placement position of the preset virtual object in the target environment, so that the preset virtual object can be displayed in the position corresponding to the target spatial data, thereby realizing the reproduction of the virtual object in a fixed position.

[0007] The process of acquiring the spatial model of the target environment and the spatial data corresponding to different locations in the spatial model includes: constructing the spatial model of the target environment using a first construction method, constructing the map data of the target environment using a second construction method, wherein the map data contains spatial data corresponding to different locations in the spatial model, and synchronizing the constructed spatial model and map data.

[0008] Therefore, it is possible to obtain the spatial model and map data of the target environment, which can be used for the subsequent placement and configuration of preset virtual objects in the target spatial data. Furthermore, the spatial model and map data are synchronized and aligned, thus allowing the determination of the correspondence between the spatial data in the spatial model and the spatial data in the map, thereby enabling the configuration of preset virtual objects at their corresponding positions in the target environment.

[0009] The construction of the spatial model and the map data are carried out simultaneously.

[0010] Therefore, a spatial model and map data are constructed to ensure that the spatial model and map data are synchronized and aligned.

[0011] The first construction method is to use the Mesh construction method, and the second construction method is to use the ARWorldMap construction method.

[0012] Therefore, by using the Mesh construction method and the ARWorldMap construction method, it is possible to construct spatial models and map data.

[0013] The method of constructing a spatial model of the target environment using a first construction method includes: collecting data from the target environment using a first acquisition device to obtain first acquisition data; obtaining first spatial feature points and their depths in the target environment based on the first acquisition data; constructing a spatial model using the first spatial feature points and their depths; and / or constructing map data of the target environment using a second construction method, including: collecting data from the target environment using a second acquisition device to obtain second acquisition data; and determining second spatial feature points and their corresponding spatial data in the target environment based on the second acquisition data.

[0014] Therefore, based on the collected data of the target environment, spatial models and map data can be constructed to obtain spatial models and map data that reflect the target environment.

[0015] The first acquisition device and the second acquisition device are either the same acquisition device or two acquisition devices. In the case where the first acquisition device and the second acquisition device are two acquisition devices, the first acquisition device and the second acquisition device are installed on the same device.

[0016] Therefore, two acquisition devices using the same acquisition device or the same equipment can simultaneously acquire data about the target environment, thus aligning the spatial model and map data.

[0017] The first acquired data is at least one of image data and radar data, and the second acquired data is image data.

[0018] Therefore, by using image data or radar data, information about the target environment can be collected, and then spatial models and map data reflecting the target environment can be constructed.

[0019] The process of obtaining the target display position of the preset virtual object in the spatial model includes: displaying the spatial model; responding to the user's placement operation, displaying the preset virtual object at a specified position on the spatial model; the placement operation is used to instruct the preset virtual object to be placed at the specified position, and the specified position is obtained as the target display position.

[0020] Therefore, the user's placement operation can determine the specified position of the preset virtual object in the spatial model, thereby determining the position of the preset virtual object in the target environment. Thus, the position of the preset virtual object in the target environment can be flexibly configured according to the user's operation.

[0021] The method further includes saving the spatial data into the map data of the target environment. After obtaining the target spatial data of the preset virtual object based on the spatial data corresponding to the target display location, the method also includes adding the target spatial data of the preset virtual object into the map data.

[0022] Therefore, by adding the target space data of the preset virtual object to the map data, the preset virtual object can be added to the map data, that is, the position of the virtual object is saved in the map data. This can be used to locate and determine the position of the virtual object in the subsequent virtual display process, thereby realizing the reproduction of the virtual object in a fixed position.

[0023] The method, after obtaining the target spatial data corresponding to the preset virtual object based on the spatial data corresponding to the target display location, includes: saving the target spatial data corresponding to the preset virtual object; in response to a user's preset trigger operation, acquiring the saved target spatial data to determine the target spatial position of the preset virtual object in the target environment; in response to detecting the currently captured target spatial position; and displaying the preset virtual object in the currently captured image.

[0024] Therefore, by saving the target space data of the preset virtual object, the target space position of the preset virtual object in the target environment can be determined. When the target space position is captured, the preset virtual object is displayed in the current shooting screen. That is, the preset virtual object is intelligently displayed in the target space position in the target environment, so the preset virtual object can be reproduced in a fixed position.

[0025] The process of saving target space data corresponding to a preset virtual object includes adding the target space data of the preset virtual object to the map data. The process of obtaining the saved target space data includes obtaining the target space data from the map data, wherein the target space data represents the target space location of the preset virtual object in the target environment. The process of detecting the currently captured target space location includes using the map data and the currently captured image to perform positioning and obtain the current positioning data. The process of detecting the currently captured target space location is based on the current positioning data.

[0026] Therefore, by performing positioning, the target spatial data of the preset virtual object can be obtained from the map data, thereby determining the target spatial location, which can be used to realize the virtual display of the preset virtual object at the target spatial location.

[0027] The process of detecting the current captured target spatial location based on the current positioning data includes: determining whether there is at least one set of matching point pairs based on the current positioning data, wherein the matching point pairs include a first feature point in the current captured image and a second feature point in the target environment located at the target spatial location; and determining the current captured target spatial location in response to the existence of at least one set of matching point pairs.

[0028] Therefore, by utilizing the matching of feature points, it is possible to determine whether the target spatial location has been captured, thereby determining whether a preset virtual object needs to be displayed, and thus accurately displaying the preset virtual object.

[0029] The current positioning data includes the current pose; displaying the preset virtual object on the current display screen includes: determining the display parameters of the preset virtual object based on the current pose and the target spatial position, wherein the display parameters include at least one of the following: the display position in the current display screen and the display form of the preset virtual object; and displaying the preset virtual object on the current display screen according to the display parameters.

[0030] Therefore, by using the current pose and the target spatial position of the preset virtual object, it is possible to determine how to display the preset virtual object, thereby accurately displaying the preset virtual object.

[0031] The display form includes at least one of size and orientation.

[0032] Therefore, the preset virtual object can be accurately displayed by using at least one of the dimensions and orientation.

[0033] To solve the above-mentioned technical problems, another technical solution adopted in this application is: providing a virtual display processing device, the device comprising: a first acquisition module, a second acquisition module, and a third acquisition module, wherein the first acquisition module is used to acquire a spatial model of a target environment and spatial data corresponding to different positions in the spatial model; the second acquisition module is used to obtain the target display position of a preset virtual object in the spatial model; and the third acquisition module is used to obtain target spatial data corresponding to the preset virtual object based on the spatial data corresponding to the target display position, wherein the target spatial data is used to determine the display position of the preset virtual object on the user terminal when the user is located in the target environment.

[0034] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide an electronic device, which includes a processor and a memory, wherein the memory is used to store program data, and the processor is used to execute the program data to implement any of the above-mentioned virtual display processing methods or virtual display methods.

[0035] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium for storing program data, which can be executed to implement any of the above-mentioned virtual display processing methods or virtual display methods.

[0036] In the above scheme, a target spatial location can be pre-configured for a preset virtual object, so that the preset virtual object can be intelligently displayed at the target spatial location, thereby realizing the reproduction of the virtual object at a fixed location. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating an embodiment of the virtual display processing method of this application;

[0038] Figure 2 This is a flowchart illustrating another embodiment of the virtual display processing method of this application;

[0039] Figure 3 This is a flowchart illustrating another embodiment of step S210 of this application;

[0040] Figure 4 This is a flowchart illustrating another embodiment of step S220 of this application;

[0041] Figure 5 This is a flowchart illustrating another embodiment of step S240 of this application;

[0042] Figure 6 This is a flowchart illustrating another embodiment of the virtual display processing method in this application;

[0043] Figure 7This is a flowchart illustrating another embodiment of step S660 of this application;

[0044] Figure 8 This is a flowchart illustrating another embodiment of step S670 of this application;

[0045] Figure 9 This is a schematic diagram of the framework of an embodiment of the virtual display processing device of this application;

[0046] Figure 10 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0047] Figure 11 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0048] To make the purpose, technical solution and effects of this application clearer and more explicit, the following describes this application in further detail with reference to the accompanying drawings and embodiments.

[0049] In this article, the term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "more" in this article means two or more objects.

[0050] It is understood that the methods of this application may include the methods provided by any of the following method embodiments and any combination of the following method embodiments that do not conflict.

[0051] It is understood that the virtual display processing method in this application can be executed by a processing device, which can be any electronic device with processing capabilities, such as a mobile phone, tablet computer, computer, etc.

[0052] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the virtual display processing method of this application, the method including:

[0053] Step S110: Obtain the spatial model of the target environment and the spatial data corresponding to different locations in the spatial model.

[0054] The target environment is the environment in which the user needs to add preset virtual objects, and the preset virtual objects are the virtual objects that the user needs to view.

[0055] For example, some users have a need to view virtual objects in specific environments, such as keeping a virtual pet in their room or seeing virtual objects for checking in at landmarks. To achieve this, the preset virtual object that the user wants to view must first be fixedly placed in the target environment. The processing device can achieve this by executing the virtual object positioning steps in the virtual display processing method of this application.

[0056] It should be noted that the user in the virtual display processing method of this application can include virtual display application developers and virtual display application users. Generally speaking, the user involved in the virtual object positioning steps in the virtual display processing method can be the developer, while the user involved in the virtual display steps can be the user. Specifically, the processing device responds to the user's operation by executing the virtual object positioning steps to fix the preset virtual object to be viewed in the target environment. Then, it executes the virtual display steps to display the preset virtual object at the target spatial location in the target environment for the user to view.

[0057] Spatial data, or spatial location information, is used to determine the spatial location information of different locations in the spatial model within the target environment.

[0058] Step S120: Obtain the target display position of the preset virtual object in the spatial model.

[0059] It should be noted that the spatial model is a visual model built based on the target environment, while the spatial data corresponding to different locations in the spatial model is not visible. The spatial model can be used by users to determine the target display position of preset virtual objects in the visual model. The target display position can correspond to the target spatial position of the preset virtual object in the target environment. The processing device can obtain the target display position of the preset virtual object determined by the user in the spatial model for subsequent determination of the target spatial position.

[0060] Step S130: Based on the spatial data corresponding to the target display location, obtain the target spatial data corresponding to the preset virtual object.

[0061] It should be noted that after obtaining the spatial data corresponding to different positions in the spatial model and the target display position, the spatial data corresponding to the target display position can be determined as the target spatial data. This target spatial data can be used to determine the target spatial position of the preset virtual object in the target environment during the virtual display process.

[0062] It should be noted that the virtual display processing method of this application includes virtual display processing steps and virtual display related steps. The virtual display processing steps are executed by this processing device, and the virtual display related steps can be executed by this processing device or by a user terminal connected to this processing device.

[0063] In the above scheme, a target spatial location can be configured for a preset virtual object, so that the preset virtual object can be intelligently displayed at the target spatial location, thereby realizing the reproduction of the virtual object at a fixed location.

[0064] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the virtual display processing method of this application, the method including:

[0065] It should be noted that obtaining the spatial model of the target environment and the spatial data corresponding to different locations within the spatial model can be achieved by acquiring spatial models and map data constructed by other devices, wherein the map data includes spatial data corresponding to different locations within the spatial model; or it can be achieved by constructing the spatial model and map data of the target environment using the processing device, thereby obtaining the spatial model of the target environment and the spatial data corresponding to different locations within the spatial model. This embodiment uses the latter as an example, and step S110 can be implemented through steps S210 and S220, executed by the processing device. If the spatial model and map data of the target environment are constructed by other devices, then steps S210 and S220 are executed by those other devices, and then the processing device obtains the spatial model and map data constructed by those other devices.

[0066] The constructed spatial model and map data are both designed for the target environment and are synchronized. The map data contains spatial data corresponding to different locations in the spatial model.

[0067] Step S210: Using the first construction method, construct the spatial model of the target environment.

[0068] Furthermore, the first construction method can be to use the Mesh construction method, and the spatial model of the visual target environment obtained by this construction method can reflect the general outline of the objects contained in the target environment.

[0069] Step S220: Using the second construction method, construct the map data of the target environment.

[0070] Furthermore, the second construction method can be to use ARWorldMap to construct the map of the target environment, that is, to obtain map data. This map is not visible to the user and can be used to determine the spatial data of different locations in the target environment.

[0071] To align the spatial model and map data, the construction of the spatial model and the construction of the map data are carried out simultaneously, that is, steps S210 and S220 are executed at the same time.

[0072] Please refer to the following: Figure 2 and Figure 3 , Figure 3 This is a flowchart illustrating another embodiment of step S210 of this application. Step S210 includes:

[0073] The construction of the spatial model and map data of the target environment is based on the collection of data from the target environment, and then the construction is carried out based on the collected data. The processing equipment may include a data acquisition device for collecting data from the target environment. The data acquisition devices used for different construction methods may be the same or different. If the two construction methods use different data acquisition devices, then the two data acquisition devices are set on the same processing equipment.

[0074] Step S311: Use the first acquisition device to collect data from the target environment to obtain the first acquisition data.

[0075] In step S210, the acquisition device used is a first acquisition device, which can be a camera component or a radar. The form of the first acquisition data corresponds to the type of the first acquisition device used, and the first acquisition data is at least one of image data and radar data.

[0076] Specifically, the user activates the first acquisition device in the target environment and scans the target environment, thereby enabling the processing device to collect data about the target environment using the first acquisition device. During the user's scanning process, the processing device can use the first acquisition device to collect information about the external environment, i.e., the target environment, thus obtaining the first acquisition data.

[0077] Step S312: Based on the first collected data, obtain the first spatial feature point and the depth of the first spatial feature point in the target environment.

[0078] The first spatial feature point is a number of feature points selected from the target environment. The first spatial feature point in the target environment can be extracted and the corresponding depth can be determined using radar data. The depth refers to the distance between the first spatial feature point and the processing device. Alternatively, the first spatial feature point in the target environment can be extracted and the corresponding depth can be determined by analyzing image data.

[0079] Step S313: Construct a spatial model using the first spatial feature points and their depth.

[0080] After obtaining the first spatial feature points and their corresponding depths in the target environment, the processing device can construct a spatial model of the target environment based on the above information. Through the first spatial feature points and their corresponding depths, the spatial model can roughly reflect the outlines of objects in the target environment.

[0081] Please refer to the following: Figure 2 and Figure 4 , Figure 4 This is a flowchart illustrating another embodiment of step S220 of this application. Step S220 includes:

[0082] Step S421: Use the second acquisition device to collect data from the target environment to obtain the second acquisition data.

[0083] The second acquisition device is a shooting component. Specifically, the user can turn on the second acquisition device in the target environment and scan the target environment, thereby realizing the acquisition of the target environment using the second acquisition device of the device. The two steps of acquiring the target environment using the first acquisition device and acquiring the target environment using the second acquisition device are performed simultaneously, thereby ensuring that the constructed spatial model and map data are aligned.

[0084] It should be noted that users can move and rotate within the target environment, enabling the processing device to collect data from multiple locations and angles, thereby acquiring sufficient data to construct more accurate spatial models and map data.

[0085] Step S422: Determine the second spatial feature point in the target environment and the spatial data corresponding to the second spatial feature point based on the second collected data.

[0086] The second acquired data is image data, and the second spatial feature points are a number of feature points selected from the target environment. The processing device can extract the second spatial feature points from the acquired image data. The environmental texture in the target environment can be used as the second spatial feature points. For example, the corner of a table or the wood texture on the table can be used as the second spatial feature points.

[0087] It should be noted that map data actually stores information about secondary spatial feature points in the target environment. Therefore, the brightness and texture details of the target environment will affect the acquisition of the target environment and thus the construction of map data. If the brightness is too low or the texture details are lacking, such as acquiring data from a white wall, it may affect the construction of map data.

[0088] The spatial data corresponding to the second spatial feature point reflects its spatial location within the target environment. Specifically, the second spatial feature point and its corresponding spatial data are used as map data. This map data reflects the distribution of the second spatial feature point in the target environment and can be used to determine the position and orientation (i.e., pose) of preset virtual objects and processing devices within the target environment. Since the map data and the spatial model are aligned, the map data also includes the spatial data corresponding to different locations within the spatial model.

[0089] Step S230: Synchronize and align the constructed spatial model and map data.

[0090] Both the spatial model and the map data are constructed based on the target environment, and they can be constructed simultaneously. By aligning them synchronously, the spatial model and the map data can be mapped. Based on the map data, the spatial data corresponding to different locations in the spatial model, i.e., spatial location information, can be determined.

[0091] Step S240: Obtain the target display position of the preset virtual object in the spatial model.

[0092] It should be noted that the processing device can run a development tool, which can be provided to the user for operation, thereby implementing steps S240-S260. The user can use this development tool to develop virtual display applications, that is, AR applications, which are the main body for subsequently executing the virtual display method. For example, the development tool may include the Unity real-time content development platform, etc.

[0093] In some embodiments, a space model may contain several identical or different virtual objects, which can be determined according to the user's actual needs, and no specific restrictions are imposed here.

[0094] Please refer to the following: Figure 2 and Figure 5 , Figure 5 This is a flowchart illustrating another embodiment of step S240 of this application. Step S240 includes:

[0095] Step S541: Display the spatial model.

[0096] Specifically, the acquired spatial model and corresponding map data are imported into the development tool. The processing device can display the spatial model in the interface of the development tool for users to view, and determine the placement of preset virtual objects.

[0097] Since the spatial model can reflect the general outline of objects in the target environment, by placing preset virtual objects in the spatial model, the processing device can determine the target spatial location of the preset virtual objects in the target environment.

[0098] Step S542: In response to the user's placement operation, display the preset virtual object at the specified position on the spatial model.

[0099] It should be noted that the processing device can pre-store several pre-built virtual objects, and can display these virtual objects in the development tool interface for users to select and place. Users can select a preset virtual object and drag it to a specific location in the spatial model; this location is the user-defined location.

[0100] The processing device can respond to the above placement operation and display the preset virtual object at a specified position on the spatial model, so that the user can see the effect of placing the preset virtual object at the specified position.

[0101] Step S543: Obtain the specified location as the target display location.

[0102] The target display position of the spatial model corresponds to the target spatial position of the preset virtual object in the target environment.

[0103] Step S250: Based on the spatial data corresponding to the target display location, obtain the target spatial data corresponding to the preset virtual object.

[0104] After determining the spatial data corresponding to the target display location, the spatial data corresponding to the target display location can be obtained based on the spatial model and the map data aligned with it. This spatial data is used as the target spatial data corresponding to the preset virtual object. The target spatial location of the preset virtual object in the target environment can be determined by the target spatial data corresponding to the preset virtual object.

[0105] In some embodiments, the processing device can modify the target spatial data of the preset virtual object in response to the user's operation of moving the position of the preset virtual object in the spatial model. In the subsequent display of the preset virtual object, the modified target spatial data shall be used as the standard, thereby realizing flexible configuration of the position of the preset virtual object in the target environment according to the user's operation, convenient adjustment of the position of the preset virtual object, and improving development efficiency.

[0106] In some embodiments, the processing device can modify the target space data of the preset virtual object in response to the user's modification operation, such as adding, reducing, and replacing it. In the subsequent display of the preset virtual object, the modified target space data is used as the standard, thereby enabling convenient modification of the preset virtual object and improving development efficiency.

[0107] Step S260: Add the target space data of the preset virtual object to the map data.

[0108] It should be noted that if the processing device has the map data obtained in step S260 pre-stored, the user can scan the environment in which the user is located through the virtual display application installed on the processing device to determine whether the user is in the target environment. If the user is in the target environment, the aforementioned map data can be used to display a preset virtual object at the target location when the target location is captured.

[0109] In some embodiments, the above steps can be repeated to obtain multiple map data, and the multiple map data can be pre-installed in a virtual display application for the device to obtain and install the virtual display application, so that the user can see the corresponding preset virtual objects in different environments through the processing device.

[0110] In the above scheme, a target spatial location can be configured for a preset virtual object, and the target spatial data of the preset virtual object can be added to the map data. That is, the location of the virtual object is saved in the map data, so that the map data can be used to locate and determine the location of the virtual object in the subsequent virtual display process, thereby realizing the reproduction of the virtual object in a fixed location.

[0111] Please see Figure 6 , Figure 6 This is a flowchart illustrating another embodiment of the virtual display processing method in this application, the method including:

[0112] Step S610: Obtain the spatial model of the target environment and the spatial data corresponding to different locations in the spatial model.

[0113] Step S620: Obtain the target display position of the preset virtual object in the spatial model.

[0114] Step S630: Based on the spatial data corresponding to the target display location, obtain the target spatial data corresponding to the preset virtual object.

[0115] It is understood that the relevant descriptions of steps S610-S630 can refer to the aforementioned content regarding steps S110-S130, and will not be repeated here.

[0116] Step S640: Save the target space data of the preset virtual object.

[0117] Specifically, saving the target spatial data of a preset virtual object may include adding the target spatial data of the preset virtual object to the map data.

[0118] It should be noted that, through the above steps S610-S640, a preset virtual object can be placed in the target environment, or steps S610-S640 can be repeated to place virtual objects in multiple environments. For ease of explanation and understanding, this application will use the example of placing a preset virtual object only in the target environment and then having the user perform subsequent operations in the target environment.

[0119] Step S650: In response to the user's preset trigger operation, obtain the saved target space data to determine the target space location of the preset virtual object in the target environment.

[0120] It is understood that a virtual display application can run in the processing device. This virtual display application can be obtained through steps S610-S640, and the data of this application can include the aforementioned map data, that is, the map data of the target environment. The map data of the target environment can be obtained by the processing device performing the aforementioned steps S610-S640. This map data can include the target space data of a preset virtual object, which represents the target space location of the preset virtual object in the target environment.

[0121] The preset trigger operation can be a preset operation used to trigger the execution of relevant steps in the virtual display, such as opening the virtual display application.

[0122] Specifically, the processing device can acquire and install the virtual display application, thereby acquiring map data of the target environment. In response to the user's preset trigger operation, it can obtain target spatial data from the map data and use the target spatial data to determine the target spatial location of the preset virtual object in the target environment.

[0123] Step S660: Detect whether the target spatial location has been captured.

[0124] It should be noted that the processing device may not have captured the target spatial location at the moment, in which case the device will not display the preset virtual object. The preset virtual object will only be displayed when the target spatial location is captured; therefore, it is necessary to first check whether the processing device has captured the target spatial location.

[0125] Please refer to the following: Figure 6 and Figure 7 , Figure 7 This is a flowchart illustrating another embodiment of step S660 of this application. Step S660 includes:

[0126] Step S761: Use map data and the current captured image to determine the current location data.

[0127] It should be noted that this positioning process can also be called repositioning. Users can use a virtual display application running on the processing device to scan their current environment, thus obtaining the current captured image. Based on the captured image, spatial feature point information in the current environment can be determined. Since the processing device already has pre-stored map data, that is, map data of the target environment, which includes all spatial feature point information in the target environment, the processing device can use the spatial feature points in the current environment determined based on the captured image to match all spatial feature points in the target environment in the map data, thereby determining whether the current environment is consistent with the target environment. If the spatial feature points in the current environment match several spatial feature points in the target environment, then it can be determined that the current environment is consistent with the target environment, and it can be determined that the user is currently in the target environment. It is not necessary to re-collect and rebuild the map of the current environment; the current positioning data can be directly obtained using the map data.

[0128] If the spatial feature points in the current environment do not correspond to spatial feature points in the map data, then relocation can be considered a failure. It's important to note that even if the user is in the target environment, the location of the processing device during relocation may differ from its location during map construction. Therefore, for the same spatial feature point, the angle from which information about that feature point is collected during map construction may differ from the angle from which it is collected during relocation. If the difference is too large, even the same spatial feature point may fail to match. Therefore, collecting spatial feature point information from multiple locations and angles during map construction can improve the success rate of subsequent relocation. Furthermore, ensuring that the collection pose during relocation is consistent with the collection pose during map construction can further enhance the success rate of relocation.

[0129] Step S762: Detect whether the target spatial location has been captured based on the current positioning data.

[0130] It should be noted that after completing the positioning, it is possible to determine the current location in the target environment, which also allows us to determine the map data of the target environment and the target spatial data included therein.

[0131] The current positioning data obtained includes the current pose of the processing device, that is, its current position and orientation within the target environment. The processing device can determine whether the target spatial location has been captured by using the relationship between this pose and the target spatial location.

[0132] Alternatively, the processing device can determine whether the target spatial location has been captured by comparing the feature points in the current captured image with the feature points at the target spatial location. Specifically, the processing device can determine whether there is at least one set of matching point pairs based on the current captured image and the current positioning data. The matching point pair includes a first feature point in the current captured image and a second feature point in the target environment located at the target spatial location. In response to the existence of at least one set of matching point pairs, the processing device determines that the target spatial location has been captured.

[0133] Step S670: In response to detecting the current captured target spatial location, display a preset virtual object in the current captured image.

[0134] It is understandable that for the same preset virtual object, the same preset virtual object captured by the processing device at different locations will be different. For example, the size or orientation may be different. This is determined by the target spatial location and orientation of the preset virtual object and the pose of the processing device.

[0135] Please refer to the following: Figure 6 and Figure 8 , Figure 8 This is a flowchart illustrating another embodiment of step S670 of this application. Step S670 includes:

[0136] Step S871: Determine the display parameters of the preset virtual object based on the current pose and the target space position.

[0137] Understandably, during the repositioning process, the current pose of the processing device can be determined as the current positioning data.

[0138] Specifically, the display parameters of the preset virtual object can be determined based on the positional relationship between the current pose of the processing device and the target spatial position of the preset virtual object in the current environment. The display parameters may include at least one of the following: the display position in the current display screen and the display form of the preset virtual object, whereby the display form may include at least one of size and orientation.

[0139] Step S872: Display the preset virtual object in the current display screen according to the display parameters.

[0140] Specifically, the preset virtual object is displayed at a predetermined display position according to a predetermined size and orientation.

[0141] Steps S650-S670 can be considered as steps related to virtual display. These steps can be executed by the processing device or by the user terminal.

[0142] In the above scheme, by pre-storing the target spatial location of the preset virtual object in the target environment, when the processing device captures the target spatial location, the preset virtual object is displayed in the current shooting screen. That is, the preset virtual object is intelligently displayed at the target spatial location in the target environment, so the preset virtual object can be reproduced at a fixed position.

[0143] Please see Figure 9 , Figure 9 This is a schematic diagram of the framework of an embodiment of the virtual display processing device of this application.

[0144] In this embodiment, the virtual display processing device 90 includes a first acquisition module 91, a second acquisition module 92, and a third acquisition module 93.

[0145] The first acquisition module 91 is used to acquire the spatial model of the target environment and the spatial data corresponding to different positions in the spatial model. The second acquisition module 92 is used to obtain the target display position of the preset virtual object in the spatial model. The third acquisition module 93 is used to obtain the target spatial data corresponding to the preset virtual object based on the spatial data corresponding to the target display position. The target spatial data is used to determine the display position of the preset virtual object on the user terminal when the user is located in the target environment.

[0146] Specifically, the acquisition of the spatial model of the target environment and the spatial data corresponding to different locations in the spatial model includes: constructing a spatial model of the target environment using a first construction method, and constructing map data of the target environment using a second construction method, wherein the map data includes spatial data corresponding to different locations in the spatial model, and the constructed spatial model and map data are synchronously aligned; or, acquiring a spatial model and map data constructed by other devices.

[0147] The construction of the spatial model and the construction of the map data are carried out simultaneously. The first construction method is to use the Mesh construction method, and the second construction method is to use the ARWorldMap method.

[0148] Specifically, the spatial model of the target environment constructed using the first construction method includes: collecting data from the target environment using a first acquisition device to obtain first acquisition data; obtaining first spatial feature points and the depth of the first spatial feature points in the target environment based on the first acquisition data; and constructing a spatial model using the first spatial feature points and the depth of the first spatial feature points.

[0149] Specifically, the map data of the target environment constructed using the second construction method includes: collecting data from the target environment using a second acquisition device to obtain second acquisition data; and determining second spatial feature points and corresponding spatial data in the target environment based on the second acquisition data.

[0150] Wherein, the first acquisition device and the second acquisition device are the same acquisition device or two acquisition devices. When the first acquisition device and the second acquisition device are two acquisition devices, the first acquisition device and the second acquisition device are installed on the same device; the first acquisition data is at least one of image data and radar data, and the second acquisition data is image data.

[0151] Specifically, obtaining the target display position of the preset virtual object in the spatial model includes: displaying the spatial model; placing the preset virtual object on the displayed spatial model in response to the user's placement operation; and obtaining the current display position of the preset virtual object on the spatial model as the target display position.

[0152] The virtual display processing device 90 also includes an addition module for adding target spatial data of preset virtual objects to map data.

[0153] The virtual display processing device 90 further includes a storage module, a fourth acquisition module, and a display module. The storage module is used to store the target space data corresponding to the preset virtual object. The fourth acquisition module is used to acquire the stored target space data in response to the user's preset trigger operation to determine the target space position of the preset virtual object in the target environment. The display module is used to display the preset virtual object in the current shooting screen.

[0154] Specifically, the above-mentioned method of saving the target space data corresponding to the preset virtual object includes adding the target space data of the preset virtual object to the map data; the above-mentioned method of obtaining the saved target space data includes obtaining the target space data from the map data, wherein the target space data represents the target space location of the preset virtual object in the target environment; and the above-mentioned method of detecting the currently captured target space location includes using the map data and the currently captured image to perform positioning to obtain the current positioning data; and detecting the currently captured target space location based on the current positioning data.

[0155] Specifically, the above-mentioned detection of the current captured target spatial location based on the current positioning data includes determining whether there is at least one set of matching point pairs based on the current positioning data, wherein the matching point pairs include a first feature point in the current captured image and a second feature point in the target environment located at the target spatial location; and determining the current captured target spatial location in response to the existence of at least one set of matching point pairs.

[0156] The aforementioned current positioning data includes the current pose. The aforementioned display of the preset virtual object on the current display screen specifically includes determining the display parameters of the preset virtual object based on the current pose and the target spatial position. The display parameters include at least one of the following: the display position in the current display screen and the display form of the preset virtual object; and displaying the preset virtual object on the current display screen according to the display parameters.

[0157] Please see Figure 10 , Figure 10 This is a schematic diagram of the framework of an embodiment of the electronic device of this application.

[0158] In this embodiment, the electronic device 100 can be the processing device or user terminal described in the above embodiments. The electronic device 100 includes a memory 101 and a processor 102, wherein the memory 101 is coupled to the processor 102. Specifically, the various components of the electronic device 100 can be coupled together via a bus, or the processor 102 of the electronic device 100 can be connected to each other component one by one. The electronic device 100 can be any device with processing capabilities, such as a computer, tablet computer, mobile phone, etc.

[0159] The memory 101 is used to store program data executed by the processor 102 and data generated during the processing by the processor 102. Examples include map data, spatial models, and preset virtual objects. The memory 101 includes a non-volatile storage portion for storing the aforementioned program data.

[0160] Processor 102 controls the operation of electronic device 100. Processor 102 can also be referred to as CPU (Central Processing Unit). Processor 102 may be an integrated circuit chip with signal processing capabilities. Processor 102 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The general-purpose processor can be a microprocessor or any conventional processor. In addition, processor 102 can be implemented by multiple integrated circuit chips.

[0161] The processor 102 executes instructions by calling the program data stored in the memory 101 to implement any of the above-mentioned virtual display processing methods or virtual display methods.

[0162] Please see Figure 11 , Figure 11 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application.

[0163] In this embodiment, the computer-readable storage medium 110 stores processor-executable program data 111, which can be executed to implement any of the above-described virtual display processing methods or virtual display methods.

[0164] The computer-readable storage medium 110 can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or a medium that can store program data. Alternatively, it can be a server that stores the program data, which can send the stored program data to other devices for execution or run the stored program data itself.

[0165] In some embodiments, the computer-readable storage medium 110 may also be such as Figure 10 The memory shown.

[0166] This disclosure relates to the field of augmented reality (AR). It involves acquiring image information of target objects in a real-world environment and then using various visual algorithms to detect or identify the relevant features, states, and attributes of these objects, thereby achieving an AR effect that combines virtual and real elements to suit specific applications. For example, target objects may include human features such as faces, limbs, gestures, and movements; objects such as signs and markers; or venues such as sand tables, display areas, or displayed items. Visual algorithms may include visual localization, SLAM, 3D reconstruction, image registration, background segmentation, keypoint extraction and tracking of objects, and pose or depth detection. Specific applications can include interactive scenarios related to real-world scenes or objects, such as guided tours, navigation, explanations, reconstruction, and virtual effect overlay displays, as well as human-related special effects processing, such as makeup enhancement, body enhancement, special effects displays, and virtual model displays.

[0167] Convolutional neural networks (CNNs) can be used to detect or identify the relevant features, states, and attributes of target objects. The aforementioned CNNs are network models obtained through training using deep learning frameworks.

[0168] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0169] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0170] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for processing virtual displays, characterized in that, The method includes: Obtain a spatial model of the target environment and spatial data corresponding to different locations in the spatial model; Obtain the target display position of the preset virtual object in the spatial model; Based on the spatial data corresponding to the target display location, target spatial data corresponding to the preset virtual object is obtained. The target spatial data is used to determine the display location of the preset virtual object on the user terminal when the user is in the target environment again, so as to reproduce the preset virtual object at the display location. Add the target space data of the preset virtual object to the map data; The acquisition of the spatial model of the target environment and the spatial data corresponding to different locations in the spatial model includes: Using a first construction method, a spatial model of the target environment is constructed; using a second construction method, map data of the target environment is constructed. The map data includes spatial data corresponding to different locations in the spatial model. The construction of the spatial model and the construction of the map data are carried out simultaneously. The constructed spatial model and the map data are synchronized and aligned.

2. The method according to claim 1, characterized in that, The first construction method is to use the Mesh construction method, and the second construction method is to use the ARWorldMap construction method.

3. The method according to claim 1, characterized in that, The process of constructing a spatial model of the target environment using the first construction method includes: The target environment is collected using the first acquisition device to obtain the first acquisition data; Based on the first collected data, the first spatial feature point in the target environment and the depth of the first spatial feature point are obtained; The spatial model is constructed using the first spatial feature point and the depth of the first spatial feature point; And / or, The process of constructing map data of the target environment using the second construction method includes: The target environment is collected using a second acquisition device to obtain second acquisition data; Based on the second collected data, a second spatial feature point in the target environment and the spatial data corresponding to the second spatial feature point are determined.

4. The method according to claim 3, characterized in that, The first acquisition device and the second acquisition device are the same acquisition device or two acquisition devices. In the case that the first acquisition device and the second acquisition device are two acquisition devices, the first acquisition device and the second acquisition device are installed on the same device. And / or, the first acquired data is at least one of image data and radar data, and the second acquired data is image data.

5. The method according to any one of claims 1 to 4, characterized in that, Obtaining the target display position of the preset virtual object in the spatial model includes: Display the spatial model; In response to a user's placement operation, the preset virtual object is displayed at a specified location on the spatial model; the placement operation is used to instruct the preset virtual object to be placed at the specified location. The specified location is obtained as the target display location.

6. The method according to any one of claims 1 to 5, characterized in that, After obtaining the target spatial data corresponding to the preset virtual object based on the spatial data corresponding to the target display location, the method further includes: Save the target space data of the preset virtual object; In response to a user's preset trigger operation, the saved target space data is obtained to determine the target space location of the preset virtual object in the target environment; In response to detecting the target spatial location currently captured, the preset virtual object is displayed in the current captured image.

7. The method according to claim 6, characterized in that, The process of saving the target space data of the preset virtual object includes: Add the target space data of the preset virtual object to the map data; The process of acquiring and saving the target space data includes: The target spatial data is obtained from the map data, and the target spatial data represents the target spatial location of the preset virtual object in the target environment; The detection of the currently captured target spatial location includes: The current location data is obtained by using the map data and the currently captured image. Based on the current positioning data, the current spatial location of the target was detected.

8. The method according to claim 7, characterized in that, The step of detecting the current spatial location of the target based on the current positioning data includes: Based on the current positioning data, it is determined whether there is at least one pair of matching points, wherein the pair of matching points includes a first feature point in the current captured image and a second feature point in the target environment located at the target spatial location; In response to the existence of at least one set of matching point pairs, the current captured spatial location of the target is determined.

9. The method according to any one of claims 7 to 8, characterized in that, The current positioning data includes the current pose; Displaying the preset virtual object in the current shooting frame includes: Based on the current pose and the target spatial position, the display parameters of the preset virtual object are determined. The display parameters include at least one of the following: the display position in the current shooting frame and the display form of the preset virtual object. According to the display parameters, the preset virtual object is displayed in the current shooting screen.

10. The method according to claim 9, characterized in that, The display form includes at least one of size and orientation.

11. A virtual display processing device, characterized in that, include: The first acquisition module is used to acquire the spatial model of the target environment and the spatial data corresponding to different locations in the spatial model. The second acquisition module is used to obtain the target display position of the preset virtual object in the spatial model; The third acquisition module is used to obtain target spatial data corresponding to the preset virtual object based on the spatial data corresponding to the target display location. The target spatial data is used to determine the display location of the preset virtual object on the user terminal when the user is in the target environment again, so as to reproduce the preset virtual object at the display location; and to add the target spatial data of the preset virtual object to the map data. The acquisition of the spatial model of the target environment and the spatial data corresponding to different locations in the spatial model includes: Using a first construction method, a spatial model of the target environment is constructed; using a second construction method, map data of the target environment is constructed. The map data includes spatial data corresponding to different locations in the spatial model. The construction of the spatial model and the construction of the map data are carried out simultaneously. The constructed spatial model and the map data are synchronized and aligned.

12. An electronic device, characterized in that, The device includes a processor and a memory, the memory being used to store program data, and the processor being used to execute the program data to implement the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program data that can be executed to implement the method as described in any one of claims 1-10.

Images