Mixed reality implementation method and apparatus, edge cloud server, and storage medium

Abstract

The application provides a mixed reality implementation method and device, an edge cloud server and a storage medium, wherein the method obtains three-dimensional map information of a convergence area where the edge cloud server is located and user request information, and obtains virtual scene information corresponding to the convergence area and having a real space environment information feature from a central cloud server, so that mixed reality processing is performed based on the information, and mixed reality image and sound data are obtained and sent to a client for presentation. In the application, the client communicates with the edge cloud server and the central cloud server based on the edge cloud server, without renting a dedicated line, thereby reducing the cost. Moreover, the client and the edge cloud server support multiple connection modes such as mobile network, fixed network and dedicated line, and are suitable for a wide range of scenarios. In addition, the edge cloud server implements a mixed reality rendering scheme, reduces the hardware requirements for the mixed reality client, reduces the cost of the client and improves the wearing comfort of the user.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a mixed reality implementation method, apparatus, edge cloud server and storage medium. Background Technology

[0002] With the continuous development of technologies related to computer vision, technologies such as virtual reality and augmented reality are gradually entering people's lives, while mixed reality, as a more complete and widely applicable technology solution than virtual reality and augmented reality, has also made rapid progress.

[0003] Current mixed reality implementations typically rely on a central cloud server and mixed reality clients. The mixed reality client usually leases a dedicated line to the central cloud server. After receiving a user's request, it sends the request to the central cloud server via the leased line. The central cloud server then returns the corresponding data to the mixed reality client for rendering and presentation.

[0004] However, existing mixed reality implementation solutions require clients to lease dedicated lines to a central cloud server, resulting in high costs and limiting application scenarios. Furthermore, rendering solutions based on mixed reality clients place high demands on client hardware, further increasing costs and making mixed reality clients bulky and uncomfortable for users to wear. Summary of the Invention

[0005] This application provides a mixed reality implementation method, device, edge cloud server, and storage medium, which solves the problems of limited application scenarios and high cost of existing mixed reality implementations, and improves the comfort of users wearing the client.

[0006] In a first aspect, this application provides a mixed reality implementation method, which is applied to an edge cloud server, and the method includes the following steps:

[0007] After receiving the user request information sent by the mixed reality client, the system obtains the three-dimensional map information of the target aggregation area where the edge cloud server is located, and determines whether there is virtual scene information with real spatial environment information characteristics corresponding to the target aggregation area on the edge cloud server.

[0008] If there is no virtual scene information with real spatial environment information characteristics corresponding to the target aggregation area on the edge cloud server, then the virtual scene information with real spatial environment information characteristics is obtained from the central cloud server according to the identity identifier of the target aggregation area.

[0009] Based on the user request information, the 3D map information, and the virtual scene information with real spatial environment information characteristics, mixed reality processing is performed to obtain mixed reality image and sound data;

[0010] The mixed reality image and sound data is sent to a mixed reality client to be presented on the mixed reality client.

[0011] In one possible implementation, the user request information carries the user's location information;

[0012] The step of obtaining 3D map information of the target aggregation area where the edge cloud server is located after receiving user request information from the mixed reality client includes:

[0013] After receiving the user request information sent by the mixed reality client, the system determines whether the user is in the target aggregation area where the edge cloud server is located based on the user's location information.

[0014] If the user is in the target aggregation area where the edge cloud server is located, then obtain the three-dimensional map information of the target aggregation area where the edge cloud server is located.

[0015] In one possible implementation, obtaining the virtual scene information with real spatial environment information characteristics from the central cloud server based on the identity identifier of the target aggregation area includes:

[0016] Obtain the correspondence between the pre-stored identity identifiers of the aggregation area and the virtual scene information with real spatial environment information characteristics from the central cloud server;

[0017] Based on the correspondence, determine the virtual scene information with real spatial environment information characteristics corresponding to the identity identifier of the target aggregation area.

[0018] In one possible implementation, the mixed reality processing based on the user request information, the 3D map information, and the virtual scene information with real spatial environment information characteristics corresponding to the identity identifier of the target convergence area to obtain mixed reality image and sound data includes:

[0019] Obtain the pre-defined business logic for mixed reality business applications;

[0020] The user request information, the 3D map information, and the virtual scene information with real spatial environment information features are processed, rendered, and synthesized according to the predetermined business logic to generate synthesized mixed reality image and sound data.

[0021] In one possible implementation, the three-dimensional map information includes map information for outdoor navigation and / or map information for indoor navigation;

[0022] After sending the mixed reality image and sound data to the mixed reality client, the method further includes:

[0023] Receive navigation requests sent by the mixed reality client;

[0024] Based on the navigation request, the virtual scene information with real spatial environment information characteristics, the map information of the outdoor navigation and / or the map information of the indoor navigation are calculated accordingly to obtain mixed reality image and sound data containing the navigation route, and the mixed reality image and sound data containing the navigation route is sent to the mixed reality client.

[0025] In one possible implementation, the method further includes:

[0026] Obtain user permissions;

[0027] The process of performing mixed reality processing based on the user request information, the 3D map information, and the virtual scene information with real spatial environment information features to obtain mixed reality image and sound data includes:

[0028] Based on the user's permissions, the user request information, the 3D map information, and the virtual scene information with real spatial environment information characteristics, mixed reality processing is performed to obtain the mixed reality image and sound data.

[0029] Secondly, this application provides a mixed reality implementation device, which is applied to an edge cloud server, and the device includes:

[0030] The information acquisition module is used to acquire the three-dimensional map information of the target aggregation area where the edge cloud server is located after receiving the user request information sent by the mixed reality client, and to determine whether there is virtual scene information with real spatial environment information characteristics corresponding to the target aggregation area on the edge cloud server.

[0031] The scene acquisition module is used to obtain the virtual scene information with real spatial environment information characteristics from the central cloud server based on the identity identifier of the target aggregation area if there is no virtual scene information with real spatial environment information characteristics corresponding to the target aggregation area on the edge cloud server.

[0032] The mixed reality module is used to perform mixed reality processing based on the user request information, the three-dimensional map information, and the virtual scene information with real spatial environment information characteristics to obtain mixed reality image and sound data;

[0033] The data sending module is used to send the mixed reality image and sound data to the mixed reality client so as to present the mixed reality image and sound data on the mixed reality client.

[0034] In one possible implementation, the user request information carries the user's location information.

[0035] The information acquisition module is specifically used for:

[0036] After receiving the user request information sent by the mixed reality client, the system determines whether the user is in the target aggregation area where the edge cloud server is located based on the user's location information.

[0037] If the user is in the target aggregation area where the edge cloud server is located, then obtain the three-dimensional map information of the target aggregation area where the edge cloud server is located.

[0038] In one possible implementation, the scene acquisition module is specifically used for:

[0039] Obtain the correspondence between the pre-stored identity identifiers of the aggregation area and the virtual scene information with real spatial environment information characteristics from the central cloud server;

[0040] Based on the correspondence, determine the virtual scene information with real spatial environment information characteristics corresponding to the identity identifier of the target aggregation area.

[0041] In one possible implementation, the mixed reality module is specifically used for:

[0042] Obtain the pre-defined business logic for mixed reality business applications;

[0043] The user request information, the 3D map information, and the virtual scene information with real spatial environment information features are processed, rendered, and synthesized according to the predetermined business logic to generate synthesized mixed reality image and sound data.

[0044] In one possible implementation, the three-dimensional map information includes map information for outdoor navigation and / or map information for indoor navigation.

[0045] It also includes a navigation processing module, which is used to receive a navigation request sent by the mixed reality client after the data sending module sends the mixed reality image and sound data to the mixed reality client;

[0046] Based on the navigation request, the virtual scene information with real spatial environment information characteristics, the map information of the outdoor navigation and / or the map information of the indoor navigation are calculated accordingly to obtain mixed reality image and sound data containing the navigation route, and the mixed reality image and sound data containing the navigation route is sent to the mixed reality client.

[0047] In one possible implementation, a permission acquisition module is also included to acquire user permissions;

[0048] The mixed reality module is specifically used for:

[0049] Based on the user's permissions, the user request information, the 3D map information, and the virtual scene information with real spatial environment information characteristics, mixed reality processing is performed to obtain the mixed reality image and sound data.

[0050] Thirdly, this application provides an edge cloud server, comprising:

[0051] processor;

[0052] Memory; and

[0053] Computer programs;

[0054] The computer program is stored in the memory and configured to be executed by the processor, the computer program including instructions for performing the method as described in the first aspect.

[0055] Fourthly, this application provides a computer-readable storage medium storing a computer program that causes a server to perform the method described in the first aspect.

[0056] Fifthly, this application provides a computer program product including computer instructions, which are executed by a processor according to the method described in the first aspect.

[0057] This application provides a mixed reality implementation method, apparatus, edge cloud server, and storage medium. The method receives user requests from mixed reality clients via an edge cloud server, then obtains 3D map information of the target convergence area where the edge cloud server is located. If no virtual scene information with real spatial environment information characteristics corresponding to the target convergence area is available on the edge cloud server, the virtual scene information with real spatial environment information characteristics is obtained from a central cloud server. Based on the user request information, 3D map information, and virtual scene information, mixed reality processing is performed to obtain mixed reality image and sound data, which is then sent to the mixed reality client for presentation. In this application, the client communicates with the central cloud server via the edge cloud server, eliminating the need for leased lines and reducing costs. Furthermore, this application supports multiple connection methods between the client and the edge cloud server, including mobile networks, fixed networks, and leased lines, making it applicable to a wide range of scenarios. Additionally, the edge cloud server-based mixed reality rendering scheme in this application only requires the client to connect to the edge cloud server for information interaction, reducing hardware requirements for the mixed reality client, further lowering client costs, improving user comfort, and making it suitable for practical applications. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of the mixed reality implementation system architecture provided in the embodiments of this application;

[0060] Figure 2 A flowchart illustrating a mixed reality implementation method provided in this application embodiment;

[0061] Figure 3 A flowchart illustrating another mixed reality implementation method provided in this application embodiment;

[0062] Figure 4 This is a schematic diagram of the structure of a mixed reality implementation device provided in an embodiment of this application;

[0063] Figure 5 This is a schematic diagram of the structure of a mixed reality implementation device provided in an embodiment of this application;

[0064] Figure 6A This application provides a schematic diagram of the basic hardware architecture of an edge cloud server.

[0065] Figure 6B A schematic diagram of the basic hardware architecture of another edge cloud server provided in this application. Detailed Implementation

[0066] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0067] The terms “first,” “second,” “third,” and “fourth,” etc. (if present), in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0068] Mixed reality technology is a further development of virtual reality technology. This technology introduces and presents virtual objects or scene information into real environments or scenes, building an interactive feedback information loop between the real world, the virtual world and the user to enhance the realism of the user experience. It has the characteristics of realism, real-time interactivity and imagination.

[0069] Compared to augmented reality (AR), which overlays virtual elements onto the real world, mixed reality (MR) virtualizes real-world elements. This involves first capturing images with a camera, but these images are two-dimensional, flat, and lack depth. Therefore, the two-dimensional images are then used by a computer to create a three-dimensional virtual image and build a 3D model. Only after this virtualization can system-generated virtual objects, scenes, sounds, system prompts, and animations be embedded into the virtual 3D world constructed from the real world, achieving a seamless blend of reality and virtuality in the visual experience. Mixed reality technology allows users to maintain contact with both the real and virtual worlds simultaneously, adjusting their actions and interactions according to their needs and the context.

[0070] However, existing mixed reality systems are typically implemented based on a central cloud server and mixed reality clients. The client needs to lease a dedicated line to the central cloud server, which is costly and limits application scenarios. Moreover, existing mixed reality systems have high hardware requirements for the client, which not only further increases costs but also makes the mixed reality client relatively bulky and less comfortable for users to wear.

[0071] Therefore, this application proposes a mixed reality implementation method with an edge cloud server as the core, enabling the mixed reality client to communicate with the central cloud server based on the edge cloud server without the need for leased lines, thus reducing costs. Furthermore, this application embodiment supports multiple connection methods between the client and the edge cloud server, including mobile networks, fixed networks, and leased lines, making it applicable to a wide range of scenarios. In addition, the edge cloud server implementation of the mixed reality rendering scheme in this application embodiment only requires the client to connect to the edge cloud server for information interaction, reducing the hardware requirements for the mixed reality client, further lowering client costs, improving user comfort, and making it suitable for practical applications.

[0072] Optionally, the mixed reality implementation method provided in this application can be applied to Figure 1 The diagram shown illustrates the architecture of a mixed reality implementation system. Figure 1 As shown, the system may include a mixed reality client 101, an edge cloud server 102, and a central cloud server 103. The number of mixed reality clients and edge cloud servers can be determined according to actual needs. Figure 1 This example uses only a mixed reality client and an edge cloud server for illustration.

[0073] Here, the aggregation node in the aggregation area deploys a mobile core network forwarding plane (User Plane, UP) and a multi-service access gateway-user plane (MSG-U). All mobile and fixed network services are uniformly accessed by MSG-U at the aggregation node. The operator deploys edge cloud server 102 at the aggregation node, opening up the Infrastructure as a Service (IaaS) + Platform as a Service (PaaS) capabilities of edge cloud server 102 to third parties. Third parties deploy various mixed reality applications on edge cloud server 102, enabling multi-tenant sharing of cloud resources. Mobile and fixed network traffic is terminated and offloaded at the edge cloud, allowing access to third-party application content from the nearest location, thus improving user experience. Central cloud server 103 manages all computing nodes, while the computing nodes of edge cloud server 102 host lightweight edge cloud service applications for various upper-layer business applications.

[0074] The central cloud server 103 is deployed on the central cloud to permanently store all information and user data of the mixed reality application throughout its lifecycle. It can be divided into different sub-modules and numbered according to the different aggregation zones covered by the aggregation edge cloud. For example, if a large city has 30 aggregation zones, they can be numbered according to their identity, such as 1 to 30. The edge cloud server 102 of each aggregation zone obtains the virtual business scenario information of the corresponding aggregation zone from the central cloud server 103. At the same time, in order to avoid boundary effects, the edge cloud server 102 of each aggregation zone can include information of the boundary area, that is, the virtual business scenario information stored on the edge cloud servers 102 of adjacent aggregation zones overlaps appropriately.

[0075] The hardware of the mixed reality client 101 may include a camera (including a depth camera), a micro-projector, sensors, glasses, a display screen, and other devices. It can be used for acquiring information about real-world scenes, capturing user interaction information, collecting user geolocation information, and presenting synthesized mixed reality image and sound data. Specifically, the mixed reality client 101 can be mixed reality glasses.

[0076] In the specific implementation process, when a user enters a certain aggregation area, they can send a user request message to the edge cloud server 102 of that aggregation area through the mixed reality client 101. After receiving the request message, the edge cloud server 102 obtains the 3D map information of the aggregation area where it is located. Furthermore, the edge cloud server 102 can also obtain virtual scene information with real spatial environment information characteristics corresponding to the identity of the aggregation area from the central cloud server 103, based on the identity of the aggregation area. Then, based on the above information, mixed reality processing is performed to obtain mixed reality image and sound data, which is then sent to the mixed reality client 101 for presentation.

[0077] The aforementioned mixed reality client 101 communicates with the central cloud server 103 via an edge cloud server 102, eliminating the need for leased lines and reducing costs. Furthermore, the mixed reality client 101 supports multiple connection methods with the edge cloud server 102, including mobile networks, fixed networks, and leased lines, making it suitable for a wide range of scenarios. Additionally, the edge cloud server 102 implements the mixed reality rendering solution; the mixed reality client 101 only needs to connect to the edge cloud server 102 for information exchange, reducing hardware requirements for the mixed reality client, further lowering client costs, and improving user comfort.

[0078] It should be understood that the aforementioned edge cloud server 102 can be implemented by a processor reading instructions from memory and executing those instructions, or it can be implemented by a chip circuit.

[0079] The above system is only an example system. In specific implementation, it can be set up according to application requirements.

[0080] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the architecture of a mixed reality implementation system. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0081] Furthermore, the system architecture described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0082] The technical solutions of this application are described below using several embodiments as examples. The same or similar concepts or processes may not be repeated in some embodiments.

[0083] Figure 2 This application provides a flowchart illustrating a mixed reality implementation method. The execution entity in this embodiment can be... Figure 1 The edge cloud server 102 in the illustrated embodiment, such as Figure 2 As shown, the method may include:

[0084] S201: After receiving the user request information sent by the mixed reality client, obtain the 3D map information of the target aggregation area where the edge cloud server is located, and determine whether there is virtual scene information with real spatial environment information characteristics corresponding to the target aggregation area on the edge cloud server.

[0085] The aforementioned user request information may carry the user's location information, which may be the user's location coordinates.

[0086] After receiving a user request from a mixed reality client, the edge cloud server can determine whether the user is within the target convergence area where the edge cloud server is located, based on the user's location information. If the user is within the target convergence area, the server obtains a 3D map of that area.

[0087] Here, when the mixed reality client moves, the edge cloud server in the area where the mixed reality client is located provides services to it based on proximity. That is, when the mixed reality client is within aggregation area A of edge cloud server A, service is provided by A; and when the mixed reality client moves to aggregation area B of edge cloud server B, service is provided by B. Therefore, after receiving user request information from the mixed reality client, the aforementioned edge cloud server first determines whether the user is within the target aggregation area of ​​the aforementioned edge cloud server based on the location information carried in the user request information. If so, it executes the step of obtaining the 3D map information of the target aggregation area of ​​the edge cloud server; otherwise, the operation stops.

[0088] In this embodiment of the application, the above-mentioned mixed reality application can realize mixed reality by using a pool of mobile or fixed network IP addresses allocated to the user terminal, thus solving the problem of mixed reality terminal addressing. This can ensure that even when users move across aggregation areas, they can access the aggregation edge cloud server to which their geographical location belongs with low latency, thereby effectively solving the mobility problem.

[0089] In addition, after the aforementioned edge cloud server obtains the 3D map information of the target aggregation area where the edge cloud server is located, it can save this information. That is, it can digitize all the infrastructure in a aggregation area in terms of geographical location, convert it into data that can be stored in a computer system, and store this data as an infrastructure in the cloud of the corresponding aggregation edge cloud of that aggregation area.

[0090] Here, the aforementioned edge cloud server can provide the infrastructure as a public service, offering interfaces for mixed reality business developers to call upon to provide mixed reality services or perform secondary development, generating more detailed and industry-specific geodigital infrastructure. Mixed reality client users within this aggregation area can quickly access the aggregation edge cloud server through the fixed-mobile integrated bearer network. Moreover, by deploying distributed geodigital infrastructure on the edge clouds corresponding to all aggregation areas in a region, such as a city, comprehensive and continuous coverage of the city can be achieved, providing all mixed reality users within the city with geodigital infrastructure services that can be accessed and invoked at any time with low latency.

[0091] After obtaining the 3D map information of the target aggregation area where the edge cloud server is located, the edge cloud server first determines whether it has virtual scene information with real spatial environment information characteristics corresponding to the target aggregation area. If so, the edge cloud server can directly use its stored information to perform subsequent operations, which is simple and convenient. Otherwise, it obtains the virtual scene information with real spatial environment information characteristics from the central cloud server and performs subsequent operations based on the obtained information to ensure that subsequent operations proceed normally.

[0092] S202: If the aforementioned edge cloud server does not have virtual scene information with real spatial environment information characteristics corresponding to the aforementioned target aggregation area, then the aforementioned virtual scene information with real spatial environment information characteristics is obtained from the central cloud server based on the identity identifier of the aforementioned target aggregation area.

[0093] The aforementioned central cloud server can store all detailed content of mixed reality applications (such as panoramic images), including virtual scene information with real spatial environment characteristics corresponding to the identity identifiers of the convergence area (such as 3D model data constructed based on the spatial information characteristics of relevant buildings, shops, restaurants, transportation, and landmarks in the convergence area), providing mixed reality services to users. The central cloud server can also store user data, such as relevant application information belonging to the user, providing personalized information to the user.

[0094] As described above, taking a city as an example, the city may have multiple aggregation zones, such as 30. Each aggregation zone corresponds to different virtual scene information with real spatial environment information characteristics. Therefore, the central cloud server can pre-store the correspondence between the identity identifier of the aggregation zone and the virtual scene information with real spatial environment information characteristics. Here, the identity identifier of the aggregation zone can be the zone number, such as aggregation zone 1.

[0095] In this way, the aforementioned edge cloud server can obtain the correspondence between the identity identifier of the aforementioned aggregation area and the virtual scene information with real spatial environment information characteristics from the aforementioned central cloud server, and then determine the virtual scene information with real spatial environment information characteristics corresponding to the identity identifier of the aforementioned target aggregation area based on the correspondence.

[0096] In this embodiment, if the edge cloud server has storage capabilities, it can store content for subsequent mixed reality applications, such as virtual scene information with real spatial environment characteristics corresponding to the target aggregation area. If it does not have storage capabilities, the edge cloud server may not store all detailed content of the mixed reality application; it only needs to retrieve virtual scene information with real spatial environment characteristics corresponding to the identity identifier of the aggregation area where the edge cloud server is located from the central server when needed, thus reducing the performance requirements of the edge cloud server.

[0097] Furthermore, the aforementioned central server can also construct virtual scene information with real spatial environment information characteristics by scanning reality, enabling virtual operations. For example, in large-scale spatial digital modeling solutions, intelligent scanning robots developed based on machine learning and computer vision algorithms can scan thousands of square meters of indoor space per hour. After the scanning robot completes data collection, the central server can generate a model combining real-world data based on panoramic video optical flow stitching calculations and synchronous positioning and map construction calculations.

[0098] Furthermore, after the central server completes the modeling of the scene and objects, the mixed reality application provider can add interactive content according to business needs. For example, the mixed reality application provides a background editor, where the mixed reality application provider can add information such as images, videos, and 3D objects. The central server can then display this information in designated locations and can also add value-added functions such as hotspot templates and customized cool effects to meet various application needs.

[0099] S203: Based on the aforementioned user request information and 3D map information, as well as the aforementioned virtual scene information with real spatial environment information characteristics, perform mixed reality processing to obtain mixed reality image and sound data.

[0100] Here, after obtaining the aforementioned user request information, 3D map information, and virtual scene information with real spatial environment information characteristics, the aforementioned edge cloud server can obtain the predetermined business logic of mixed reality business applications. Thus, it can perform calculations, rendering, and synthesis on the aforementioned user request information, 3D map information, and virtual scene information with real spatial environment information characteristics according to the aforementioned predetermined business logic to generate synthesized mixed reality image and sound data.

[0101] In this embodiment, the mixed reality client does not need to configure local computing power and storage resources, nor does it need to download and save digital business information of various mixed reality application scenarios in advance within the terminal. Instead, this information is pre-stored on the edge cloud server. Based on the converged edge cloud architecture, low-latency communication support is provided for mixed reality services. The mixed reality client provides mixed reality services on demand through real-time information interaction via 5G, Wi-Fi, dedicated lines, and the converged edge cloud. This significantly reduces the hardware configuration of the mixed reality terminal (e.g., eliminating the terminal server configuration and reducing the terminal storage configuration), reducing investment and operating costs. Users only need to wear smart glasses to use mixed reality services.

[0102] S204: Send the above mixed reality image and sound data to the mixed reality client so that the mixed reality image and sound data can be presented on the mixed reality client.

[0103] Here, after the aforementioned mixed reality client presents the mixed reality image and sound data, users can quickly and accurately perform certain operations based on the information presented by the client, making it suitable for various applications. For example, when a customer arrives in front of a large shopping mall building, the aforementioned edge cloud server can present the user with mixed reality image and sound data of the building based on the user's location information carried in the user request information sent by the mixed reality client. This allows the user to quickly and accurately locate the target store, saving time and effort, and can even provide navigation services, achieving fast and convenient interaction.

[0104] In addition, this application embodiment fully leverages the unique advantages of edge cloud servers in cloud-network collaboration (access, latency, mobility management), cloud resources (computing power, resource sharing), and geography (location positioning, proximity to users). Specifically, it supports multiple access methods such as mobile networks, fixed networks, and leased lines to ubiquitous connectivity, end-to-end low latency and low jitter, and service scheduling characteristics based on the Internet Protocol (IP) address pool of fixed and mobile network clients to achieve flexible configuration of service addressing and proximity access. The large computing power and large storage of edge cloud servers relieve the constraints of client hardware configuration.

[0105] This embodiment of the application receives user requests from mixed reality clients via an edge cloud server. Then, it obtains 3D map information of the target convergence area where the edge cloud server is located. If the edge cloud server does not have virtual scene information with real spatial environment characteristics corresponding to the target convergence area, it obtains virtual scene information with real spatial environment characteristics from the central cloud server. Based on the user request information, 3D map information, and virtual scene information, mixed reality processing is performed to obtain mixed reality image and sound data, which is then sent to the mixed reality client for presentation. In this embodiment, the client communicates with the central cloud server via the edge cloud server, eliminating the need for leased lines and reducing costs. Furthermore, this embodiment supports multiple connection methods between the client and the edge cloud server, including mobile networks, fixed networks, and leased lines, making it applicable to a wide range of scenarios. Additionally, the edge cloud server implements the mixed reality rendering scheme in this embodiment, requiring only a connection to the edge cloud server for information interaction, reducing hardware requirements for the mixed reality client, further lowering client costs, improving user comfort, and making it suitable for practical applications.

[0106] Furthermore, in this embodiment, the aforementioned 3D map information may include map information for outdoor navigation and / or map information for indoor navigation. After sending the mixed reality image and sound data to the mixed reality client, the edge cloud server can also receive navigation requests from the mixed reality client. Based on these requests, it performs corresponding calculations to obtain mixed reality image and sound data containing navigation routes, and sends this data to the mixed reality client to meet the user's navigation needs. Moreover, the edge cloud server can also obtain user permissions. Based on user permissions, the aforementioned user request information, 3D map information, and the aforementioned virtual scene information and user data with real spatial environment characteristics, it performs mixed reality processing to obtain mixed reality image and sound data based on user permissions. This allows for providing different mixed reality data to users with different permissions, such as providing mixed reality data for different scenes. Users with higher permissions receive richer mixed reality data for their scenes. For example, the mixed reality data provided to VIP users may include product discount information, helping customers find affordable products more quickly and enabling merchants to promote and attract customers more efficiently, thus meeting practical application needs.

[0107] here, Figure 3 This is a flowchart illustrating another mixed reality implementation method proposed in an embodiment of this application. Figure 3 The embodiments of this application are shown to consider the above-mentioned user navigation and product purchase needs. For example... Figure 3 As shown, the method includes:

[0108] S301: After receiving the user request information sent by the mixed reality client, obtain the three-dimensional map information of the target aggregation area where the aforementioned edge cloud server is located, and determine whether there is virtual scene information with real spatial environment information characteristics corresponding to the target aggregation area on the edge cloud server.

[0109] S302: If there is no virtual scene information with real spatial environment information characteristics corresponding to the target aggregation area on the edge cloud server, then the virtual scene information with real spatial environment information characteristics corresponding to the target aggregation area is obtained from the central cloud server according to the identity identifier of the target aggregation area.

[0110] The steps S301-S302 are described in the above description of steps S201-S202, and will not be repeated here.

[0111] S303: Obtain user permissions.

[0112] The aforementioned user access permissions can include VIP users or regular users. In this way, the edge cloud server can present different mixed reality image and sound data to users with different permissions, meeting the application needs of various application scenarios.

[0113] S304: Based on the user's permissions, user request information, and 3D map information, as well as the virtual scene information and user data with real spatial environment information characteristics, perform mixed reality processing to obtain mixed reality image and sound data.

[0114] For example, the aforementioned edge cloud server can perform calculations, rendering, and synthesis based on the user's permissions, user request information, 3D map information, and virtual scene information with real spatial environment information characteristics, according to predetermined business logic, to generate synthesized mixed reality image and sound data.

[0115] After obtaining the mixed reality image and sound data, the aforementioned edge cloud server sends the mixed reality image and sound data to the mixed reality client for presentation. Different mixed reality data is provided to users with different permissions. For example, VIP users are provided with product discount information. When users see the panoramic map, the discount information of the store can be displayed. For customers, this can help them find cheap and affordable products more quickly. For merchants, this function can also help them promote and attract customers more efficiently.

[0116] S305: Send the above mixed reality image and sound data to the mixed reality client so that the mixed reality image and sound data can be presented on the mixed reality client.

[0117] Step S305 is described in the same way as step S204 above, and will not be repeated here.

[0118] S306: Receive the navigation request sent by the mixed reality client mentioned above.

[0119] S307: Based on the above navigation request, the above virtual scene information with real spatial environment information characteristics, as well as the above map information for outdoor navigation and / or map information for indoor navigation, are calculated accordingly to obtain mixed reality image and sound data containing the navigation route, and the mixed reality image and sound data containing the navigation route is sent to the above mixed reality client.

[0120] Here, the aforementioned 3D map information includes map information for outdoor navigation and / or indoor navigation. Using indoor navigation, taking a shopping mall as an example, the navigation request can carry store information, such as store names. The aforementioned edge cloud server can perform corresponding calculations based on the navigation request, the virtual scene information with real spatial environment characteristics, and the aforementioned map information for outdoor navigation and / or indoor navigation, to obtain mixed reality image and sound data containing the navigation route. This mixed reality image and sound data containing the navigation route is then sent to the mixed reality client to guide the user. For users who are not good at reading maps, this function can save them a lot of time and effort. Moreover, map navigation companies and indoor 3D scanning companies can collaborate to try to introduce the latter's technology into map navigation applications (APPs), allowing map navigation to be used indoors as well. Large shopping malls and supermarkets only need to scan the merchants and the premises and upload the data to the edge cloud server to effectively solve the problem of difficult navigation within the mall.

[0121] In this embodiment, the edge cloud server can also receive navigation requests sent by the mixed reality client. Based on these requests, it performs corresponding calculations to obtain mixed reality image and sound data containing navigation routes, and sends this data to the mixed reality client to meet the user's navigation needs. Furthermore, the edge cloud server provides different mixed reality data to users with different permissions, meeting practical application needs. Additionally, in this embodiment, the client communicates with the central cloud server via the edge cloud server, eliminating the need for leased lines and reducing costs. The client and edge cloud server support multiple connection methods, including mobile networks, fixed networks, and leased lines, making it suitable for a wide range of scenarios. The edge cloud server implements the mixed reality rendering scheme, and the client only needs to connect to the edge cloud server for information interaction, reducing the hardware requirements for the mixed reality client, further lowering client costs, improving user comfort, and making it suitable for practical applications.

[0122] Corresponding to the mixed reality implementation method in the above embodiments, Figure 4 This is a schematic diagram of a mixed reality implementation device provided in an embodiment of this application. For ease of explanation, only the parts relevant to the embodiments of this application are shown. Figure 4This is a schematic diagram of a mixed reality implementation device provided in an embodiment of this application. The mixed reality implementation device 40 includes: an information acquisition module 401, a scene acquisition module 402, a mixed reality module 403, and a data transmission module 404. The mixed reality implementation device here can be the aforementioned edge cloud server device itself, or a chip or integrated circuit that implements the functions of the aforementioned edge cloud server. It should be noted that the division of the information acquisition module, scene acquisition module, mixed reality module, and data transmission module is only a logical functional division; physically, they can be integrated or independent.

[0123] The information acquisition module 401 is used to acquire the three-dimensional map information of the target aggregation area where the edge cloud server is located after receiving the user request information sent by the mixed reality client, and to determine whether there is virtual scene information with real spatial environment information characteristics corresponding to the target aggregation area on the edge cloud server.

[0124] The scene acquisition module 402 is used to obtain virtual scene information with real spatial environment information characteristics corresponding to the target aggregation area from the central cloud server if there is no virtual scene information with real spatial environment information characteristics corresponding to the target aggregation area on the edge cloud server.

[0125] The mixed reality module 403 is used to perform mixed reality processing based on the user request information, the three-dimensional map information, and the virtual scene information with real spatial environment information characteristics to obtain mixed reality image and sound data.

[0126] The data sending module 404 is used to send the mixed reality image and sound data to the mixed reality client so as to present the mixed reality image and sound data on the mixed reality client.

[0127] In one possible implementation, the user request information carries the user's location information.

[0128] The information acquisition module 401 is specifically used for:

[0129] After receiving the user request information sent by the mixed reality client, the system determines whether the user is in the target aggregation area where the edge cloud server is located based on the user's location information.

[0130] If the user is in the target aggregation area where the edge cloud server is located, then obtain the three-dimensional map information of the target aggregation area where the edge cloud server is located.

[0131] In one possible implementation, the scene acquisition module 402 is specifically used for:

[0132] Obtain the correspondence between the pre-stored identity identifiers of the aggregation area and the virtual scene information with real spatial environment information characteristics from the central cloud server;

[0133] Based on the correspondence, determine the virtual scene information with real spatial environment information characteristics corresponding to the identity identifier of the target aggregation area.

[0134] In one possible implementation, the mixed reality module 403 is specifically used for:

[0135] Obtain the pre-defined business logic for mixed reality business applications;

[0136] The user request information, the 3D map information, and the virtual scene information with real spatial environment information features are processed, rendered, and synthesized according to the predetermined business logic to generate synthesized mixed reality image and sound data.

[0137] The apparatus provided in this application embodiment can be used to perform the above-described... Figure 2 The technical solutions of the method embodiments are similar in implementation principle and technical effect, and will not be described again here.

[0138] Figure 5 This is a schematic diagram of another mixed reality implementation device provided in an embodiment of this application. Figure 4 Based on this, the aforementioned mixed reality implementation device 40 also includes a navigation processing module 405 and an access control module 406.

[0139] The three-dimensional map information includes map information for outdoor navigation and / or map information for indoor navigation.

[0140] The navigation processing module 405 is used to receive a navigation request sent by the mixed reality client after the data sending module 404 sends the mixed reality image and sound data to the mixed reality client;

[0141] Based on the navigation request, the virtual scene information with real spatial environment information characteristics, the map information of the outdoor navigation and / or the map information of the indoor navigation are calculated accordingly to obtain mixed reality image and sound data containing the navigation route, and the mixed reality image and sound data containing the navigation route is sent to the mixed reality client.

[0142] In one possible implementation, the permission acquisition module 406 is used to acquire the user's permissions.

[0143] The mixed reality module 403 is specifically used for:

[0144] Based on the user's permissions, the user request information, the 3D map information, the virtual scene information with real spatial environment information characteristics, and user data, mixed reality processing is performed to obtain the mixed reality image and sound data.

[0145] The apparatus provided in this application embodiment can be used to perform the above-described... Figure 3 The technical solutions of the method embodiments are similar in implementation principle and technical effect, and will not be described again here.

[0146] Optionally, Figure 6A and 6B The following schematically illustrates one possible basic hardware architecture for the edge cloud server described in this application.

[0147] See Figure 6A and 6B The edge cloud server includes at least one processor 601 and a communication interface 603. Optionally, it may also include a memory 602 and a bus 604.

[0148] In the edge cloud server, the number of processors 601 can be one or more. Figure 6A and 6B Only one processor 601 is illustrated. Optionally, processor 601 can be a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). If the edge cloud server has multiple processors 601, the types of the multiple processors 601 can be different or the same. Optionally, the multiple processors 601 of the edge cloud server can also be integrated into a multi-core processor.

[0149] The memory 602 stores computer instructions and data; the memory 602 may store computer instructions and data required to implement the mixed reality implementation method provided in this application, for example, the memory 602 stores instructions for implementing the steps of the mixed reality implementation method. The memory 602 may be any one or any combination of the following storage media: non-volatile memory (e.g., read-only memory (ROM), solid-state drive (SSD), hard disk drive (HDD), optical disk), volatile memory.

[0150] The communication interface 603 can provide information input / output for the at least one processor. It may also include any one or any combination of the following devices: a network interface (e.g., an Ethernet interface), a wireless network card, or other devices with network access capabilities.

[0151] Optionally, the communication interface 603 can also be used for data communication between the edge cloud server and other computing devices or terminals.

[0152] Further optional, Figure 6A and 6B Bus 604 is represented by a thick line. Bus 604 connects processor 601 to memory 602 and communication interface 603. In this way, through bus 604, processor 601 can access memory 602 and can also use communication interface 603 to exchange data with other computing devices or terminals.

[0153] In this application, the edge cloud server executes computer instructions in memory 602, causing the edge cloud server to implement the above-described mixed reality implementation method provided in this application, or causing the edge cloud server to deploy the above-described mixed reality implementation device.

[0154] From the perspective of logical functional division, for example, such as Figure 6A As shown, the memory 602 may include an information acquisition module 401, a scene acquisition module 402, a mixed reality module 403, and a data transmission module 404. This inclusion refers only to the fact that the instructions stored in the memory, when executed, can respectively implement the functions of the information acquisition module, scene acquisition module, mixed reality module, and data transmission module, and is not limited to the physical structure.

[0155] One possible design, such as Figure 6B As shown, the memory 602 also includes a navigation processing module 405 and an access control module 406. Here, "including" only refers to the fact that when the instructions stored in the memory are executed, they can respectively implement the functions of the navigation processing module and the access control module, and is not limited to the physical structure.

[0156] In addition, the aforementioned edge cloud servers can, besides being able to, [do something like the above] Figure 6A and 6B Besides being implemented through software, it can also be implemented as a hardware module or as a circuit unit through hardware.

[0157] This application provides a computer-readable storage medium, the computer program product including computer instructions that instruct a computing device to execute the mixed reality implementation method provided in this application.

[0158] This application provides a computer program product, including computer instructions, which are executed by a processor using the mixed reality implementation method described above.

[0159] This application provides a chip including at least one processor and a communication interface, wherein the communication interface provides information input and / or output to the at least one processor. Furthermore, the chip may also include at least one memory for storing computer instructions. The at least one processor is used to invoke and execute the computer instructions to perform the mixed reality implementation method provided in this application.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0161] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A mixed reality implementation method characterized by, The method is applied to an edge cloud server, and comprises the following steps: After receiving user request information sent by a mixed reality client, three-dimensional map information of a target convergence area where the edge cloud server is located is acquired, and it is determined whether there is virtual scene information with real space environment information characteristics corresponding to the target convergence area on the edge cloud server, wherein the three-dimensional map information comprises map information of outdoor navigation and / or map information of indoor navigation; If there is no virtual scene information with real space environment information characteristics corresponding to the target convergence area on the edge cloud server, the virtual scene information with real space environment information characteristics is obtained from a central cloud server according to an identity of the target convergence area; Based on the user request information, the three-dimensional map information and the virtual scene information with real space environment information characteristics, mixed reality processing is performed to obtain mixed reality image sound data; The mixed reality image sound data is sent to the mixed reality client; A navigation request sent by the mixed reality client is received; According to the navigation request, the virtual scene information with real space environment information characteristics and the map information of outdoor navigation and / or the map information of indoor navigation, corresponding calculation is performed to obtain mixed reality image sound data containing a navigation route, and the mixed reality image sound data containing the navigation route is sent to the mixed reality client.

2. The method of claim 1, wherein, The user request information carries orientation information of a user; After receiving the user request information sent by the mixed reality client, the three-dimensional map information of the target convergence area where the edge cloud server is located is acquired, comprising the following steps: After receiving the user request information sent by the mixed reality client, it is determined whether the user is in the target convergence area where the edge cloud server is located according to the orientation information of the user; If the user is in the target convergence area where the edge cloud server is located, the three-dimensional map information of the target convergence area where the edge cloud server is located is acquired.

3. The method of claim 1, wherein, The virtual scene information with real space environment information characteristics corresponding to the identity of the target convergence area is obtained from the central cloud server, comprising the following steps: The corresponding relationship between the identity of a pre-stored convergence area and the virtual scene information with real space environment information characteristics is acquired from the central cloud server; According to the corresponding relationship, the virtual scene information with real space environment information characteristics corresponding to the identity of the target convergence area is determined.

4. The method according to any one of claims 1 to 3, characterized in that, Based on the user request information, the three-dimensional map information and the virtual scene information with real space environment information characteristics, mixed reality processing is performed to obtain mixed reality image sound data, comprising the following steps: Pre-determined business logic of a mixed reality business application is acquired; The user request information, the three-dimensional map information and the virtual scene information with real space environment information characteristics are operated, rendered and synthesized according to the pre-determined business logic to generate synthesized mixed reality image sound data.

5. The method according to any one of claims 1 to 3, characterized in that, The method further comprises the following steps: The rights of a user are acquired. The mixed reality processing is performed based on the user request information and the three-dimensional map information and the virtual scene information with the real space environment information feature, and mixed reality image sound data is obtained. The mixed reality processing is performed based on the user request information and the three-dimensional map information and the virtual scene information with the real space environment information feature, and the mixed reality image sound data is obtained. 6.A mixed reality implementation device, the device being applied to an edge cloud server, characterized in that, Comprise: An information acquisition module is configured to, after receiving user request information sent by a mixed reality client, acquire three-dimensional map information of a target aggregation area where an edge cloud server is located, and determine whether there is virtual scene information with a real space environment information feature corresponding to the target aggregation area on the edge cloud server, wherein the three-dimensional map information comprises map information of outdoor navigation and / or map information of indoor navigation; A scene acquisition module is configured to, if there is no virtual scene information with a real space environment information feature corresponding to the target aggregation area on the edge cloud server, acquire the virtual scene information with a real space environment information feature from a central cloud server according to an identity of the target aggregation area; A mixed reality module is configured to perform mixed reality processing based on the user request information and the three-dimensional map information and the virtual scene information with a real space environment information feature, and obtain mixed reality image sound data; A data sending module is configured to send the mixed reality image sound data to a mixed reality client, and receive a navigation request sent by the mixed reality client; According to the navigation request, the virtual scene information with a real space environment information feature, and the map information of outdoor navigation and / or the map information of indoor navigation, corresponding calculation is performed to obtain mixed reality image sound data containing a navigation route, and the mixed reality image sound data containing the navigation route is sent to the mixed reality client.

7. An edge cloud server, characterized by Comprise: A processor; A memory; And A computer program; The computer program is stored in the memory and is configured to be executed by the processor, and the computer program comprises instructions for executing the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program enables a server to execute the method of any one of claims 1-5.

9. A computer program product, characterised in that, The computer instructions are executed by the processor to execute the method of any one of claims 1-5.

Images