Cloud collaborative ar implementation method, apparatus, system, and communication device

Abstract

The present application relates to a cloud collaborative AR implementation method, an apparatus, a communication device, a storage medium, and a computer program product. The method comprises: collecting terminal location information and sending the terminal location information to a cloud, such that the cloud returns a target real scene digitized result to a terminal; on the basis of location and pose information of a local sensor, obtaining a corrected target real scene digitized result; and generating a virtual scene result on the basis of the terminal location information, and fusing the virtual scene result with the corrected target real scene digitized result to obtain a virtual-real fusion result. By using the method, an inherent real scene digitized result of a cloud can be reused, and on this basis, real-time correction processing is performed by means of a local sensor of a terminal, completing the call of local material of the terminal and virtual scene rendering, achieving virtual and real combined output; the real-scene digital processing pressure of the terminal is reduced, the service processing delay is also reduced, the convenience of cloud AR services is improved, and the stability of result output is ensured on the basis of existing real-scene digital resources.

Description

Cloud-based collaborative AR implementation method, device, system, and communication equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on December 26, 2023, with application number 2023118031198, and entitled “Cloud-based collaborative AR implementation method, device, system and communication equipment,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of cloud resource technology, and in particular to a cloud-based collaborative AR implementation method, apparatus, system, communication equipment, storage medium, and computer program product. Background Art

[0003] With the rapid development of AR (Augmented Reality) technology, the scope of application of AR is becoming more and more extensive. AR can use a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing to simulate computer-generated virtual information such as text, images, three-dimensional models, music, and videos, and apply them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0004] In related technologies, the surrounding real scenes are acquired based on the cameras of AR glasses, mobile phones and other terminals, and the virtual mirror overlay output based on the real scenes is realized based on the technical frameworks such as ARCore, ARkit, HUAWEI AR Engine that run locally on the terminals. This has strong logical processing requirements for the local AR terminals, especially in more complex actual scenarios, which require higher terminal processing capabilities. The terminal needs to perform more calculations, resulting in poor timeliness of the terminal's AR output results.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a cloud-collaborative AR implementation method, apparatus, system, communication equipment, storage medium, and computer program product, which can reduce the load pressure on the terminal and ensure the timeliness and quality of the AR results output by the terminal.

[0007] A cloud-based collaborative AR implementation method, the method comprising:

[0008] Collecting terminal location information and sending the terminal location information to the cloud, so that the management node in the cloud determines a target edge computing node that matches the terminal location information, and so that the target edge computing node determines a target real scene digitization result that matches the terminal location information, and returns the target real scene digitization result to the terminal;

[0009] Correcting the position and attitude of the target real scene digitized result based on the position and attitude information of the local sensor to obtain a corrected target real scene digitized result;

[0010] Based on the terminal position information, a virtual scene result is generated, the virtual scene result is fused with the corrected target real scene digitized result to obtain a virtual-real fusion result, and the virtual-real fusion result is output.

[0011] In one embodiment, generating a virtual scene result based on the terminal location information includes:

[0012] Extracting corresponding virtual scene materials based on the terminal location information, and determining real scene coordinate data of the corrected target real scene digitization result;

[0013] Based on the real scene coordinate data, the virtual scene material is rendered to obtain a virtual scene result.

[0014] In one embodiment, fusing the virtual scene result with the corrected target real scene digitized result to obtain a virtual-real fusion result includes:

[0015] The virtual scene result and the corrected target real scene digitized result are subjected to layer overlay processing to obtain a virtual-real fusion result.

[0016] In one embodiment, collecting the terminal location information and sending the terminal location information to the cloud includes:

[0017] Collect updated terminal location information, and when the updated terminal location information meets a preset difference condition, send the updated terminal location information to the cloud.

[0018] In one embodiment, the terminal location information includes positioning coordinates or a position relationship between the terminal and a base station.

[0019] In one embodiment, the position relationship between the terminal and the base station includes the base station distance and the base station angle.

[0020] A cloud-based collaborative AR implementation method, applied to a management node in the cloud, includes:

[0021] receiving terminal location information sent by the terminal;

[0022] Determine a target edge computing node that matches the terminal location information, and send the terminal location information to the target edge computing node.

[0023] In one embodiment, determining a target edge computing node that matches the terminal location information includes:

[0024] According to the proximity principle, a target edge computing node that matches the terminal location information is determined.

[0025] In one embodiment, the method further comprises:

[0026] Receive the request for obtaining the real scene digitization result sent by the edge computing node;

[0027] When a real scene digitization result matching the real scene digitization result acquisition request is found in the local storage space, the real scene digitization result is returned to the edge computing node.

[0028] A cloud-based collaborative AR implementation method, applied to edge computing nodes in the cloud, includes:

[0029] Receive terminal location information sent by the management node;

[0030] A target real scene digitization result matching the terminal location information is collected, and the target real scene digitization result is returned to the terminal.

[0031] In one embodiment, collecting a digitized result of a target real scene that matches the terminal location information includes:

[0032] When a real scene digitization result corresponding to the terminal location information is found in the local storage space, coordinate information and orientation information are determined based on the terminal location information;

[0033] The real scene digitization results are screened based on the coordinate information and the orientation information to obtain a target real scene digitization result.

[0034] In one embodiment, the method further comprises:

[0035] When no real scene digitization result corresponding to the terminal location information is found in the local storage space, a real scene digitization result acquisition request is generated and sent to the management node.

[0036] In one embodiment, the method further comprises:

[0037] When no real scene digitization result corresponding to the terminal location information is found in the local storage space, a real scene resource acquisition request is generated and sent to the management node, so that the management node returns the real scene resource data;

[0038] The real scene resource data is received, and the real scene coordinate data in the real scene resource data is reconstructed to obtain a target real scene digitization result.

[0039] A cloud-based collaborative AR implementation device, comprising:

[0040] A first acquisition module is configured to acquire terminal location information and send the terminal location information to a cloud, so that a management node in the cloud determines a target edge computing node that matches the terminal location information, and so that the target edge computing node determines a target real scene digitization result that matches the terminal location information and returns the target real scene digitization result to the terminal;

[0041] A correction module, configured to correct the position and attitude of the target real scene digitized result based on the position and attitude information of the local sensor to obtain a corrected target real scene digitized result;

[0042] The first generating module is used to generate a virtual scene result based on the terminal position information, fuse the virtual scene result with the corrected target real scene digitized result to obtain a virtual-real fusion result, and output the virtual-real fusion result.

[0043] A cloud-based collaborative AR implementation device, applied to a management node in the cloud, includes:

[0044] A first receiving module, configured to receive terminal location information sent by a terminal;

[0045] The first determining module is configured to determine a target edge computing node that matches the terminal location information and send the terminal location information to the target edge computing node.

[0046] A cloud-coordinated AR implementation device, applied to an edge computing node in the cloud, comprising:

[0047] A first receiving module, configured to receive terminal location information sent by the management node;

[0048] The second acquisition module is configured to acquire a target real scene digitization result that matches the terminal location information and return the target real scene digitization result to the terminal.

[0049] A cloud-coordinated AR implementation system includes a terminal and a cloud, wherein the cloud includes a management node and at least one edge computing node, wherein:

[0050] The terminal is used to collect terminal location information and send the terminal location information to the cloud;

[0051] The management node in the cloud is used to receive the terminal location information; determine a target edge computing node that matches the terminal location information, and send the terminal location information to the target edge computing node;

[0052] The target edge computing node is used to receive the terminal location information sent by the management node; collect the target real scene digitization result that matches the terminal location information, and return the target real scene digitization result to the terminal;

[0053] The terminal is also used to correct the position and posture of the target real scene digitization result based on the position and posture information of the local sensor to obtain the corrected target real scene digitization result; generate a virtual scene result based on the terminal position information, fuse the virtual scene result with the corrected target real scene digitization result to obtain a virtual-reality fusion result, and output the virtual-reality fusion result.

[0054] A communication device comprising: a transmitter and a processor;

[0055] The processor is configured to collect terminal location information and send the terminal location information to the cloud through the transmitter, so that the management node in the cloud determines a target edge computing node that matches the terminal location information, and the target edge computing node determines a target real scene digitization result that matches the terminal location information, and returns the target real scene digitization result to the terminal through the transmitter;

[0056] The processor is further configured to correct the position and attitude of the target real scene digitization result based on the position and attitude information of the local sensor to obtain a corrected target real scene digitization result;

[0057] The processor is further configured to generate a virtual scene result based on the terminal position information, fuse the virtual scene result with the corrected target real scene digitized result to obtain a virtual-real fusion result, and output the virtual-real fusion result.

[0058] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0059] Collecting terminal location information and sending the terminal location information to the cloud, so that the management node in the cloud determines a target edge computing node that matches the terminal location information, and so that the target edge computing node determines a target real scene digitization result that matches the terminal location information, and returns the target real scene digitization result to the terminal;

[0060] Correcting the position and attitude of the target real scene digitized result based on the position and attitude information of the local sensor to obtain a corrected target real scene digitized result;

[0061] Based on the terminal position information, a virtual scene result is generated, the virtual scene result is fused with the corrected target real scene digitized result to obtain a virtual-real fusion result, and the virtual-real fusion result is output.

[0062] A computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0063] Collecting terminal location information and sending the terminal location information to the cloud, so that the management node in the cloud determines a target edge computing node that matches the terminal location information, and so that the target edge computing node determines a target real scene digitization result that matches the terminal location information, and returns the target real scene digitization result to the terminal;

[0064] Correcting the position and attitude of the target real scene digitized result based on the position and attitude information of the local sensor to obtain a corrected target real scene digitized result;

[0065] Based on the terminal position information, a virtual scene result is generated, the virtual scene result is fused with the corrected target real scene digitized result to obtain a virtual-real fusion result, and the virtual-real fusion result is output.

[0066] The above-mentioned cloud-coordinated AR implementation method, device, communication device, storage medium and computer program product include: collecting terminal location information and sending the terminal location information to the cloud, so that the management node in the cloud determines the target edge computing node that matches the terminal location information, and so that the target edge computing node determines the target real scene digitization result that matches the terminal location information and returns the target real scene digitization result to the terminal; correcting the position and posture of the target real scene digitization result based on the position and posture information of the local sensor to obtain the corrected target real scene digitization result; generating a virtual scene result based on the terminal location information, fusing the virtual scene result with the corrected target real scene digitization result to obtain a virtual-real fusion result, and outputting the virtual-real fusion result. By adopting this method, the cloud-based real scene digitization result can be reused, and by combining the terminal's local sensor for real-time correction processing, the terminal's local material call and virtual scene rendering are completed to achieve virtual-real combined output, effectively eliminating operational bottlenecks, ensuring cloud carrying capacity, reducing the terminal's real scene digitization processing pressure, and also reducing the service processing delay, thereby improving the convenience of cloud AR services. Based on existing real scene digitization resources, the stability of AR result output is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] FIG1 is a diagram illustrating an application environment of a cloud-based collaborative AR implementation method according to an embodiment;

[0068] FIG2 is a flow chart of a method for implementing AR via cloud collaboration in one embodiment;

[0069] FIG3 is a schematic flow chart of steps for obtaining a virtual scene result in one embodiment;

[0070] FIG4 is a flow chart of a cloud-based collaborative AR implementation method according to another embodiment;

[0071] FIG5 is a schematic flow chart of steps for obtaining a digitized target scene in one embodiment;

[0072] FIG6 is a flow chart of a cloud-based collaborative AR implementation method according to another embodiment;

[0073] FIG7 is a schematic flow chart of steps for obtaining a digitized target scene in one embodiment;

[0074] FIG8 is a schematic flow chart of steps for obtaining a digitized target scene in one embodiment;

[0075] FIG9 is a signaling diagram of a cloud-based collaborative AR implementation method according to one embodiment;

[0076] FIG10 is a schematic diagram of a cloud-based collaborative AR implementation system in one embodiment;

[0077] FIG11 is a structural block diagram of a cloud-coordinated AR implementation device according to an embodiment;

[0078] FIG12 is a structural block diagram of a cloud-coordinated AR implementation device according to an embodiment;

[0079] FIG13 is a structural block diagram of a cloud-based collaborative AR implementation device according to an embodiment;

[0080] FIG14 is a diagram showing the internal structure of a communication device in one embodiment. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0082] Figure 1 is a schematic diagram of an application scenario for a cloud-based collaborative AR implementation method provided by an embodiment of the present application. As shown in Figure 1, the scenario includes a cloud 200 and multiple AR terminals 100, wherein the cloud 200 includes a management node 2001 and multiple edge computing nodes 2002; data is transmitted between the AR terminals 100 and the cloud 200 via the network, and data is also transmitted between the management node 2002 and each edge computing node 2002 via the network.

[0083] The AR terminal 100 may be a wireless terminal, which may be a device that provides voice and / or other service data connectivity to a user, or a handheld device with wireless connection capabilities, or other processing devices connected to a wireless modem. The wireless terminal may communicate with one or more core networks via a radio access network (RAN). The wireless terminal may be a mobile terminal, such as a mobile phone (or "cellular" phone) and a computer with a mobile terminal. For example, the wireless terminal may be a portable, pocket-sized, handheld, computer-built-in, or vehicle-mounted mobile device that exchanges voice and / or data with the radio access network. The wireless terminal may also be referred to as a system, subscriber unit, subscriber station, mobile station, mobile station, remote station, remote terminal, access terminal, user terminal, user agent, user device, or user equipment, without limitation herein.

[0084] AR technology is a bridge that builds communication between reality and virtuality. It can provide more intuitive, multi-dimensional information overlay and 3D image display based on real scenes, and has good development prospects in many fields such as games, education, and industrial manufacturing.

[0085] Related technologies are usually implemented on terminals such as AR glasses and mobile phones. They rely on terminal-side cameras to acquire the surrounding real scene. Based on technical frameworks such as ARCore, ARkit, and HUAWEI AR Engine running locally on the terminal, they implement real scene recognition, plane recognition, coordinate mapping, and map reconstruction. Finally, they achieve virtual image overlay output based on the real scene. This requires the terminal's relatively complex processing capabilities and has the following main problems:

[0086] High terminal capability requirements: AR terminals are required to have strong local logic processing, video recognition, graphics rendering, map reconstruction and other capabilities. In particular, for more complex actual scenarios, higher terminal processing capabilities are required, and the terminal side needs to have strong underlying chip support; terminal battery life is limited: AR terminals need to perform more complex local logic operations, graphics rendering and other processing, which directly leads to the need to consume more local terminal power, causing pressure on its overall battery life. At the same time, by increasing the battery capacity, the weight of the AR terminal will also increase, which brings challenges to the comfort of wearing or using it.

[0087] In addition to running AR technology on local terminals, traditional technologies also use cloud-based processing and a local display mode for AR terminals, known as the cloud AR mode. This technical implementation involves the AR terminal uploading the real-time scene to the cloud, which then performs logical operations and graphics rendering, completing related real-scene recognition, plane recognition, coordinate mapping, and map reconstruction. Ultimately, a virtual image is generated through the real-scene-related digital processing logic, plane recognition results, coordinate system, and map reconstruction results. This image is then sent to the terminal via audio and video streaming, and the relevant audio and video decoding and playback presentation on the terminal side ultimately implement the relevant cloud AR service. Cloud AR technology transfers computing pressure to the cloud, leading to higher cloud processing capacity requirements, insufficient terminal capability integration, and increased service latency.

[0088] 1. High cloud processing capability requirements: Cloud AR technology solutions require strong cloud computing, graphics rendering, video recognition, and other capabilities. They require high-end logic computing and graphics rendering chips and a large amount of storage space. In addition, to support multi-user operation, more complex virtualization or container operating mechanisms are required to build a more complex software and hardware environment. In particular, the process of generating virtual scenes on the cloud and using audio and video streaming processing places high demands on the cloud's own processing capabilities, pushing up the overall business deployment cost.

[0089] 2. Insufficient terminal capability integration: With the improvement of chip standards, their processing capabilities have also been rapidly enhanced. Therefore, the chips configured locally in AR terminals have strong processing capabilities. If a pure cloud processing mode is adopted, relying only on the terminal's real-scene acquisition, audio and video decoding, and layer overlay playback capabilities, the terminal's related capabilities cannot be released, resulting in a serious waste of the terminal's own processing capabilities.

[0090] 3. Increased service latency: Because the relevant processing processes such as map reconstruction based on real scenes, virtual image generation based on relevant map systems, and audio and video encoding processing need to be completed in the cloud, it requires a long processing latency. For users who require an immersive experience, the user experience quality will decrease, posing a challenge to the real-time service.

[0091] Based on the above-mentioned traditional technology, the embodiment of the present application provides a cloud-collaborative AR implementation method, which realizes the digital reuse of cloud resources and real scenes through the collaboration between the terminal and the cloud, that is, the integration of cloud computing capabilities and terminal computing capabilities is realized, so that the terminal and the cloud can achieve complementarity and optimization of computing resources; from the dimension of AR implementation process, by fully reusing existing resources, it is possible to avoid opening a mutually isolated operating environment for each AR terminal, reduce repeated processing processes, shorten the service processing process, fundamentally reduce the process of real-scene digital processing, shorten the overall processing delay, enhance the user's AR usage experience, and enhance the service carrying capacity of the cloud.

[0092] It should be noted that the beneficial effects or technical problems solved by the embodiments of the present application are not limited to this one, but may also include other implicit or related problems. For details, please refer to the description of the following embodiments.

[0093] Before introducing the specific embodiments of the present invention, the professional terms involved in this application are explained:

[0094] AR (Augmented Reality) is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. It simulates computer-generated virtual information such as text, images, three-dimensional models, music, and video, and applies it to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0095] Digital reuse of cloud resources and real scenes: Based on the existing digital processing results of real scenes in the cloud, the terminal obtains the corresponding digital processing results of real scenes from the cloud through location information. By reusing the existing digital processing results of real scenes, convenient and efficient cloud AR services are achieved.

[0096] Cloud-to-cloud integration: Integrates the processing capabilities of the cloud and terminals, including the cloud's inherent real-scene recognition coordinate system, plane and map construction results, and the terminal's local real-scene collection, location acquisition, and virtual-reality fusion processing capabilities, to achieve cloud-to-cloud collaborative processing.

[0097] Cloud-integrated AR implementation method: Compared with AR localization or pure cloud AR operation mode, the entire processing process from real scene acquisition, real scene recognition, plane recognition, coordinate mapping, map reconstruction, etc. must be completed before AR services can be obtained. In contrast, fast AR only needs to upload location information to the cloud. The cloud will directly send the logical map within the relevant range to the terminal. The terminal adjusts the map reconstruction range sent by the cloud according to the local real scene, thereby skipping the relevant real scene processing process and realizing the speed of service acquisition.

[0098] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0099] In one embodiment, as shown in FIG2 , a cloud-coordinated AR implementation method is provided. This method is described by taking the AR terminal in the cloud-coordinated AR implementation method in FIG1 as an example, and includes the following steps:

[0100] Step 202: collect terminal location information and send the terminal location information to the cloud, so that the management node in the cloud determines the target edge computing node that matches the terminal location information, and so that the target edge computing node determines the target real scene digitization result that matches the terminal location information, and returns the target real scene digitization result to the terminal.

[0101] Among them, the terminal location information can be identification information of the terminal's location, used to characterize the current location of the terminal, for example, it can be the terminal's positioning coordinate information, or the position relationship between the terminal and the communication base station; in one example, the terminal's positioning coordinate information can include GPS positioning coordinate information, or Beidou positioning coordinate information, etc., and the collected positioning coordinate information is used as the terminal location information; in another example, when the terminal is a terminal device using a cellular network, the terminal location information can be the corresponding position of the terminal and the base station near the terminal, for example, it can be the angle relationship with multiple base stations, and the current terminal location information is determined based on the base station location information; in another example, the terminal location information can be the current specific coordinates, altitude and other location information of the terminal determined by obtaining positioning information and combining the corresponding relationship between the terminal and the base station.

[0102] The cloud can include a management node and multiple edge computing nodes. The target edge computing node that matches the terminal's location information can be an edge computing node that meets a preset proximity principle with the terminal's location information. For example, it can be the edge computing node that is closest to the terminal among the multiple edge computing nodes included in the cloud. The real-scene digitization result can be the result of digitizing the actual surrounding environment, such as video data and image data. The target real-scene digitization result that matches the terminal's location information can be a real-scene digitization result that is consistent with the real scene that can be observed at the terminal's current location.

[0103] In one embodiment, when the terminal captures the real scene through the image acquisition device corresponding to the terminal, the terminal can also obtain the terminal location information of the terminal's current location and report the terminal location information of the terminal to the cloud. The terminal can use the terminal location information as the current location identifier of the terminal. After the management node in the cloud receives the reported terminal location information, it can parse and process the terminal location information to determine the target edge computing node that matches the terminal location information. For example, the edge computing node closest to the terminal corresponding to the terminal location information can be used as the target edge computing node. The management node can use the target edge computing node as the technical service node of the terminal, that is, serve the terminal through the target edge computing node. The target edge computing node can determine the target real scene digitization result that matches the terminal position in response to the instruction of the management node, and return the determined target real scene digitization result to the terminal.

[0104] In one embodiment, the image acquisition device corresponding to the terminal may be a built-in camera device of the terminal, or an external camera device of the terminal, etc. The terminal may realize real-time acquisition of video and pictures of the real scene where the terminal is located through the image acquisition device.

[0105] Step 204 : Correcting the position and attitude of the target real scene digitization result based on the position and attitude information of the local sensor to obtain a corrected target real scene digitization result.

[0106] Among them, the local sensor can be a sensor locally configured in the terminal, for example, it can include an azimuth sensor, an angle sensor, etc.; the position and posture information includes precise status information such as azimuth, angle, position change, and current position information, that is, position information and posture information, etc.; the position and posture information of the local sensor can be the position and posture information of the terminal collected by the local sensor, and the position and posture information includes position and posture, the position can be positioning coordinate information, and the posture can include information such as direction and tilt angle.

[0107] In one embodiment, the terminal can correct the received real-scene digitization result using the position and posture information collected by the local sensor to obtain a corrected target real-scene digitization result. In one example, the terminal can perform orientation correction, angle correction, and altitude correction on the target real-scene digitization result received from the target edge computing node in the cloud based on the terminal's orientation and tilt angle data collected by the local sensor to obtain a corrected target real-scene digitization result.

[0108] In one example, local sensors may include gyroscopes and gravimeters, and the terminal may obtain sensor information in real time through the local sensors. For example, the terminal may obtain relevant camera orientation, tilt angle, and other information of the terminal through the gyroscope and gravimeter sensors. If the terminal has a ranging function, the height of the current terminal above the ground may also be collected. The terminal determines the offset of the current terminal position information based on the above information and the front / back / left / right / up / down three-dimensional angle offset. If the coverage range of the target real-scene digitization result sent by the target edge computing node is consistent with the coverage range determined by the terminal based on the offset, the correction is determined to be completed, and the target real-scene digitization result sent by the target edge computing node is used as the corrected target real-scene digitization result. If the coverage range determined by the terminal based on the offset exceeds the coverage range of the target real-scene digitization result sent by the target edge computing node, the terminal needs to re-report to the target edge computing node to obtain the target real-scene digitization result that meets the coverage range determined based on the offset, and the obtained target real-scene digitization result that meets the coverage range is used as the corrected real-scene digitization result.

[0109] Step 206 : Generate a virtual scene result based on the terminal location information, fuse the virtual scene result with the corrected target real scene digitized result to obtain a virtual-real fusion result, and output the virtual-real fusion result.

[0110] In one embodiment, the virtual scene result may be a virtual scene generated based on the virtual scene material; the virtual-real fusion result may be AR scene data obtained by superimposing and fusing a layer of a real scene digitization result and a layer of a virtual scene result.

[0111] In one embodiment, the terminal can obtain virtual scene resources based on the terminal's location information, and generate a virtual scene result based on the obtained virtual scene resources. The terminal can fuse the virtual scene result with the corrected target real scene digitized result based on a graphic overlay mode to obtain a completed virtual-real fusion result. Based on this, the terminal can output the virtual-real fusion result.

[0112] In one example, the terminal can use the coordinate system of the corrected real-scene digitization result as the fusion basis, obtain virtual scene materials based on the terminal location information locally, render the virtual scene image according to the generation logic of the virtual scene result and the coordinate position corresponding to the real-scene digitization result, generate a virtual scene result, and fuse the virtual scene result with the corrected target real-scene digitization result in a layer overlay manner to obtain a virtual-reality fusion result.

[0113] In the above-mentioned cloud-coordinated AR implementation method, the terminal location information is collected and sent to the cloud, so that the management node in the cloud determines the target edge computing node that matches the terminal location information, and the target edge computing node determines the target real scene digitization result that matches the terminal location information, and returns the target real scene digitization result to the terminal; the position and posture of the target real scene digitization result is corrected based on the position and posture information of the local sensor to obtain the corrected target real scene digitization result; based on the terminal location information, a virtual scene result is generated, and the virtual scene result is fused with the corrected target real scene digitization result to obtain a virtual-real fusion result, and the virtual-real fusion result is output. By adopting this method, the real scene digitization result inherent in the cloud can be reused, and by combining the local sensor of the terminal for real-time correction processing, the call of the terminal local material and the virtual scene rendering are completed, and the virtual-real combined output is realized, which effectively removes the operation bottleneck, ensures the carrying capacity of the cloud, reduces the real scene digitization processing pressure of the terminal, and also reduces the service processing process delay, thereby improving the convenience of cloud AR services. Based on the existing real scene digitization resources, the stability of the AR result output is guaranteed.

[0114] In one embodiment, as shown in FIG3 , in step 206 , generating a virtual scene result based on the terminal location information may include:

[0115] Step 302: extracting corresponding virtual scene materials based on the terminal location information, and determining real scene coordinate data of the corrected target real scene digitization result;

[0116] In one embodiment, the virtual scene materials include virtual scene map materials, object modules, character images, action animations, special effect resources and other materials; the real scene coordinate data of the corrected target real scene digitization result is the coordinate system in the corrected target real scene digitization result.

[0117] In one embodiment, the terminal can retrieve virtual scene materials from a local database whose location information is consistent with the terminal's location information. In one embodiment, the terminal can retrieve virtual scene materials from a local database based on the corrected orientation, angle, and altitude. The terminal can also retrieve virtual resources, i.e., virtual scene materials, corresponding to a virtual scene that matches the corrected coordinate system in the digitized result of the real scene in the local database.

[0118] Step 304 : Based on the real scene coordinate data, the virtual scene material is rendered to obtain a virtual scene result.

[0119] In one embodiment, the terminal can perform graphics rendering processing on the acquired virtual scene materials according to the corresponding coordinate system in the digitized result of the corrected target real scene, that is, the acquired virtual scene map materials, object modules, character images, action animations, special effects and other resources stored locally in the terminal are rendered in the virtual scene according to the coordinate position corresponding to the real scene to generate a rendering result, which is the virtual scene result.

[0120] In this embodiment, the terminal can correct its position and posture based on local sensors, and call virtual scene resources based on the corrected results and render them locally on the terminal, making full use of locally stored virtual resources, reducing the pressure of real-time processing in the cloud, and reducing the overall service latency.

[0121] In one embodiment, in step S206, fusing the virtual scene result with the corrected target real scene digitized result to obtain a virtual-real fusion result may include:

[0122] The virtual scene result and the corrected target real scene digitized result are layered together to obtain the virtual-real fusion result.

[0123] In one embodiment, the terminal can fuse the rendered virtual scene result with the corrected target real scene digitized result in a layer overlay manner to obtain a virtual-real fusion result. In one example, the terminal can overlay the layer of the virtual scene result with the layer of the corrected target real scene digitized result to obtain an overlay result, and use the overlay result as the virtual-real fusion result; during the overlay process, the terminal can combine the correspondence between the virtual scene and the real scene, such as related occlusion, real scene plane mapping, etc., to ensure seamless connection between the virtual and real scenes, and output the virtual-real fusion result.

[0124] In this embodiment, by fusing the virtual scene result with the real scene digitization result locally in the terminal, the local computing resources of the terminal can be fully utilized, the service endurance level can be improved, and a better fusion of the virtual scene and the real scene can be ensured.

[0125] In one embodiment, in step S202, collecting the terminal location information and sending the terminal location information to the cloud may include:

[0126] Collect updated terminal location information, and when the updated terminal location information meets the preset difference condition, send the updated terminal location information to the cloud.

[0127] The content of the preset difference condition may be that the terminal location information after the update is inconsistent with the terminal location information before the update.

[0128] In one embodiment, the terminal can perform real-time detection of the terminal's location information and use the real-time detected terminal location information as updated terminal location information. If it is determined that the updated terminal location information meets a preset difference condition with the terminal location information before the update, it can be determined that the current terminal's location information has changed. The terminal can then report the updated terminal location information to a management node in the cloud. Upon receiving the updated terminal location information, the management node in the cloud can, based on the updated terminal location information, collect a real-scene digitization result that matches the updated terminal location information and send the collected real-scene digitization result to the terminal.

[0129] In this embodiment, by detecting the terminal position in real time, a real scene digitization result matching the terminal position information can be sent to the terminal in a timely manner, thereby ensuring the accuracy of the output virtual-reality fusion result.

[0130] In one embodiment, the terminal location information includes positioning coordinates or the location relationship between the terminal and the base station.

[0131] In one embodiment, the positioning coordinates of the terminal can be at least one of the collected GPS positioning coordinates, Beidou positioning, etc. The collected positioning information can be used as the current location of the terminal, that is, as the terminal location information; the terminal location information also includes the position relationship between the terminal and the base station. The base station can be multiple base stations with a distance less than a preset distance threshold. The position relationship between the terminal and the base station can include the distance information between the terminal and the base station, as well as the angle information between the terminal and the base station, etc.

[0132] In this embodiment, the comprehensiveness of the terminal location information is guaranteed to provide a stable data basis for subsequent virtual-reality fusion.

[0133] In one embodiment, the position relationship between the terminal and the base station includes the base station distance and the base station angle.

[0134] In one embodiment, the base station distance in the position relationship between the terminal and the base station may be the distance information between the terminal and multiple base stations near the terminal, or the angle information between the terminal and multiple base stations near the terminal. The multiple base stations near the terminal may be base stations whose distance from the terminal is less than a preset distance threshold. The specific value of the preset distance threshold may be determined by those skilled in the art based on the actual application scenario. This disclosure does not limit the specific value. For example, it may be within 1 km, within 5 km, etc.

[0135] In this embodiment, the comprehensiveness of the terminal location information is guaranteed to provide a stable data basis for subsequent virtual-reality fusion.

[0136] In one embodiment, step S204, correcting the position and attitude of the target real scene digitized result based on the position and attitude information of the local sensor to obtain the corrected target real scene digitized result, may include:

[0137] The terminal can determine the precise status information based on the position and posture information of the local sensor, for example, the precise status information such as orientation, angle, position change, and current position information; if the terminal determines that the coverage range of the target real-scene digitization result currently received and sent by the target edge computing node in the cloud does not meet the coverage range corresponding to the offset determined based on the precise status information, then the terminal determines that the target real-scene digitization result currently received does not meet the correction requirements. The terminal can use the precise status information as the position change result and report the position change result to the target edge computing node in the cloud. The target edge computing node can adjust the calling range of the real-scene digitization result based on the position change information, obtain the adjusted real-scene digitization result that matches the position change information, and send the adjusted real-scene digitization result to the terminal. After receiving the adjusted real-scene digitization result, the terminal can use the adjusted real-scene digitization result as the corrected target real-scene digitization result.

[0138] In this embodiment, the terminal can report the current position change result to the cloud in a timely manner based on the real-time position information, ensuring that the real scene digitization result sent from the cloud matches the current real-time position.

[0139] In one embodiment, as shown in FIG4 , a cloud-based collaborative AR implementation method is provided. The method is described by taking the application of the method to the management node in the cloud in FIG1 as an example, and includes the following steps:

[0140] Step 402: receiving terminal location information sent by the terminal;

[0141] In one embodiment, the management node may be a centralized control node in the cloud, configured to receive terminal location information reported by each terminal and allocate an edge computing node that matches the terminal location information reported by the terminal to each terminal.

[0142] In one embodiment, the terminal reports the terminal location information to a management node in the cloud. The management node may receive the terminal location information reported by the terminal via a communication network link between the terminal and the cloud.

[0143] Step 404: Determine a target edge computing node that matches the terminal location information, and send the terminal location information to the target edge computing node.

[0144] In one embodiment, the target edge computing node that matches the terminal location information may be a computing node determined based on the distance between each edge computing node and the terminal.

[0145] In one embodiment, the management node in the cloud can parse and process the terminal location information to determine the terminal as the positioning coordinates; and calculate the distance between each edge computing node contained in the cloud and the positioning coordinates of the terminal. Based on the distance between each edge computing node and the location of the terminal, the target edge computing node that matches the terminal location information is determined, and the terminal location information is forwarded to the target edge computing node.

[0146] In one example, the terminal can calculate the distance information between each edge computing node and the terminal's positioning coordinates, and use the edge computing node corresponding to the minimum distance as the target edge computing node that matches the terminal's location information. In another example, the terminal can calculate the distance information between each edge computing node and the terminal's positioning coordinates, determine the idle state of each edge computing node, and use the idle edge computing node with the minimum distance as the target edge computing node.

[0147] In this embodiment, by forwarding the terminal location information to the target edge computing node, the real-scene digitization results are obtained through the target edge computing node, further sharing the operating pressure of the cloud management node. The management node acts as an intermediary between the terminal and the target edge computing node to achieve distributed processing and ensure service processing delay.

[0148] In one embodiment, in step S404, determining a target edge computing node that matches the terminal location information may include:

[0149] Based on the proximity principle, determine the target edge computing node that matches the terminal location information.

[0150] In one embodiment, the content of the proximity principle may be the shortest distance to the positioning coordinates represented by the terminal location information.

[0151] In one embodiment, the terminal may calculate the distance information between each edge computing node and the terminal's positioning coordinates, and use the edge computing node corresponding to the minimum distance as the target edge computing node that matches the terminal's location information. In another example, the terminal may calculate the distance information between each edge computing node and the terminal's positioning coordinates, determine the idle state of each edge computing node, and use the idle edge computing node with the minimum distance as the target edge computing node.

[0152] In this embodiment, multiple edge computing nodes contained in the cloud are screened by distance, and the edge computing node closest to the terminal is used as the edge computing node for data interaction with the terminal, which can ensure the real-time performance of data interaction.

[0153] In one embodiment, as shown in FIG5 , the method further includes:

[0154] Step 502: receiving a request for obtaining a real scene digitization result sent by an edge computing node;

[0155] In one embodiment, the edge computing node may check locally stored resources based on the received terminal location information. If it is determined that no real-scene digitization results matching the terminal location information exist locally on the edge computing node, i.e., if it is determined that the node lacks the corresponding real-scene digitization results locally, a real-scene digitization result acquisition request may be generated and sent to a management node in the cloud. The real-scene digitization result acquisition request is used to cause the management node in the cloud to issue and update real-scene digitization resources matching the terminal location information received by the edge computing node.

[0156] Step 504: When a real scene digitization result matching the real scene digitization result acquisition request is found in the local storage space, the real scene digitization result is returned to the edge computing node.

[0157] In one embodiment, after receiving a request for obtaining a real-scene digitization result from another edge computing node, the management node may call the corresponding real-scene digitization resource in the local storage space and send the corresponding real-scene digitization resource to the edge computing node.

[0158] In one example, the management node can obtain the front, back, left, right, top, and bottom boundaries of the relevant real scene based on the terminal location information carried in the received request for obtaining the real scene digitization result, in a manner that meets one of the requirements of the real scene digitization distance reaching a certain threshold and having one of the requirements of obstruction such as buildings / natural landscapes, call the real scene digitization processing result, and send the collected real scene digitization result to the edge computing node.

[0159] In this embodiment, the cloud resource reuse mechanism is fully utilized to realize the sharing of real-scene digitization results, and the real-scene digitization processing results completed in advance are reused to multiple terminals, thereby realizing reuse and sharing for multiple users and multiple terminals, shortening service delays, and reducing the waste of computing resources caused by full processing.

[0160] In one embodiment, as shown in FIG6 , a cloud-based collaborative AR implementation method is provided. The method is described by applying it to an edge computing node in the cloud in FIG1 , and includes the following steps:

[0161] Step 602: receiving terminal location information sent by the management node;

[0162] In one embodiment, after receiving the terminal location information reported by the terminal, the management node can determine the distance between the positioning coordinates represented by the terminal location information and the target edge computing node that meets the proximity principle, and forward the terminal location information to the target edge computing node.

[0163] Step 604: Collect the target real scene digitization result that matches the terminal location information, and return the target real scene digitization result to the terminal.

[0164] In one embodiment, when the edge computing node is the target edge computing node determined by the management node, after receiving the terminal location information, the edge computing node can collect the target real scene digitization result that matches the terminal location information, and send the collected target real scene digitization result that matches the terminal location information to the terminal.

[0165] In this embodiment, the target edge computing node is used to determine the target real scene digitization result that matches the terminal location information, which can share the cloud computing pressure, reduce the real-time load on the cloud, and ensure the real-time output of the virtual-reality fusion result.

[0166] In one embodiment, as shown in FIG. 7 , in step S604 , collecting a digitized result of a target real scene that matches the terminal location information may include:

[0167] Step 702: When a real scene digitization result corresponding to the terminal location information is found in the local storage space, coordinate information and orientation information are determined based on the terminal location information.

[0168] Step 704 : Filter the real scene digitization results based on the coordinate information and the orientation information to obtain the target real scene digitization results.

[0169] In one embodiment, the edge computing node searches the local storage space based on the terminal location information. If a real-scene digitization result centered on the terminal location information is found, the edge computing node determines that a real-scene digitization result corresponding to the terminal location information has been found in the local storage space. Based on this, the edge computing node filters the locally stored real-scene digitization results based on the terminal location information and determines the real-scene digitization result with consistent orientation information and coordinate information as the target real-scene digitization result.

[0170] In one example, the edge computing node screening process can be as follows: the edge computing node needs to determine whether the existing resources in the local storage space can meet the terminal-side requirements for digitizing all boundaries of the real scene. This can be determined by using the location information of the real scene digitization distance and occlusion stored at the edge computing node. Only when the stored real scene digitization distance reaches a certain threshold or there is an occlusion requirement such as buildings / natural landscapes, can the resource be determined to meet the requirements and then determined as the target real scene digitization result.

[0171] In this embodiment, the real-scene digitization result resources stored locally in the edge computing node are filtered through coordinate information and orientation information to ensure the user's AR experience.

[0172] In one embodiment, the cloud-based collaborative AR implementation method further includes:

[0173] When no real scene digitization result corresponding to the terminal location information is found in the local storage space, a real scene digitization result acquisition request is generated and sent to the management node.

[0174] In one embodiment, the edge computing node may check locally stored resources based on the received terminal location information. If it is determined that no real-scene digitization results matching the terminal location information exist locally on the edge computing node, i.e., if it is determined that the node lacks the corresponding real-scene digitization results locally, a real-scene digitization result acquisition request may be generated and sent to a management node in the cloud. The real-scene digitization result acquisition request is used to cause the management node in the cloud to issue and update real-scene digitization resources matching the terminal location information received by the edge computing node.

[0175] In this embodiment, through data interaction between edge computing nodes and cloud management nodes, full reuse of cloud resources and sharing of real-scene digitization results are achieved. The real-scene digitization processing results completed in advance can be reused to multiple terminals, reducing the additional consumption of computing resources.

[0176] In one embodiment, as shown in FIG8 , the cloud-coordinated AR implementation method further includes:

[0177] Step 802: When no real scene digitization result corresponding to the terminal location information is found in the local storage space, a real scene resource acquisition request is generated and sent to the management node, so that the management node returns the real scene resource data.

[0178] Step 804: receiving the real scene resource data, reconstructing the real scene coordinate data in the real scene resource data, and obtaining a target real scene digitization result.

[0179] In one embodiment, the edge computing node can check locally stored resources based on the received terminal location information. If it is determined that the edge computing node does not have a real-scene digitization result matching the terminal location information locally, that is, if it is determined that the node lacks the corresponding real-scene digitization result locally, a real-scene digitization result acquisition request can be generated and sent to the management node in the cloud. The real-scene digitization result acquisition request is used to enable the management node in the cloud to issue and update the real-scene digitization resource that matches the terminal location information received by the edge computing node. The real-scene digitization resource can be real-scene resource data. After receiving the real-scene resource data, the edge computing node can reconstruct the real-scene coordinate data in the real-scene resource data based on the terminal location information to obtain the target real-scene digitization result.

[0180] In this embodiment, through data interaction between edge computing nodes and cloud management nodes, full reuse of cloud resources and sharing of real-scene digitization results are achieved. The real-scene digitization processing results completed in advance can be reused to multiple terminals, reducing the additional consumption of computing resources. The coordinates of the reused real-scene digitization results can also be reconstructed based on the terminal location information, further ensuring the accuracy of the output of the virtual-reality fusion results.

[0181] In one embodiment, FIG9 provides a signaling interaction flow chart of a cloud-based collaborative AR implementation method, which includes a terminal, edge computing nodes, and a management node (a centralized control node in the cloud). The cloud includes centralized control and several edge computing nodes. In this embodiment, multiple terminals may also be included. FIG9 shows only one terminal, but this is not intended to limit the present disclosure. As shown in FIG9, the method includes the following steps.

[0182] Step 1: Location information collection. Specifically, the terminal side first completes location information collection as its current location identifier.

[0183] Step 2: The terminal completes the real scene collection at the same time and reports the location information to the cloud-based centralized management and control module.

[0184] Step 3: Centralized cloud-based control analyzes and processes the location information reported by the terminal to find the edge computing node closest to it.

[0185] Step 4: Allocate edge computing nodes as the technical service nodes for the terminal according to the principle of proximity.

[0186] In step 5, each edge computing node obtains the terminal's location information and checks its own resources to determine whether the edge node has a local digitized version of the real scene related to the terminal's current location. If not, proceed to step 6. If so, proceed to step 9.

[0187] Step 6: When it is determined that the edge node lacks the corresponding real-scene digitization result, a request is made to issue and update the corresponding real-scene digitization resource in a centralized control mode.

[0188] Step 7: The centralized control module responds to edge computing node requests and calls the corresponding real-scene digital resources.

[0189] Step 8: The centralized management and control module sends the corresponding real-scene digital resources to the edge computing node.

[0190] In step 9, the edge computing node completes the range judgment of the corresponding real scene boundary based on the terminal's related location information.

[0191] Step 10: The edge computing node completes the corresponding real scene digitization result call based on the digitized range of the real scene boundary determined.

[0192] Step 11: The edge computing node sends the completed real-scene digitization result to the terminal side.

[0193] In step 12, the terminal obtains the result of the real scene digital processing sent by the edge computing node.

[0194] In step 13, the terminal obtains corresponding sensor information in real time during the above processing, including precise status information such as orientation, angle, position change, and current position information.

[0195] In step 14, the terminal performs a corresponding correction operation on the real-time digitization processing result based on its current real-time precision status information. If the current real-time digitization processing result meets the correction processing requirements, the process proceeds to step 18. If not, the process proceeds to step 15.

[0196] In step 15, the terminal reports the position change result to the corresponding edge computing node, so that the subsequent edge computing node can adapt to its position change and re-acquire the corresponding real scene digitization processing result.

[0197] In step 16, each edge computing node adjusts the calling range of the real scene digitization result according to the change of location information.

[0198] In step 17, each edge computing node sends the adjusted real-scene digitization results to the terminal side.

[0199] Step 18: According to the coordinate system corresponding to the digitized result of the corrected real scene, the corresponding resource call of the corresponding virtual scene is completed.

[0200] In step 19, the terminal side performs relevant virtual scene rendering processing based on the virtual scene resources that have been called.

[0201] In step 20, the terminal side completes the fusion output of the corresponding virtual scene and the current real scene based on the graphics overlay mode.

[0202] Among them, in step 1, the collection of terminal location information can be carried out by collecting specific coordinate positions or collecting position relationships related to base stations. Specific coordinate position collection: Collect at least one positioning information such as GPS, Beidou positioning, etc. as the current terminal location. Base station position relationship collection: For terminal devices using cellular networks, collect and obtain their corresponding positions with nearby related base stations, such as the angular relationship with multiple base stations, and combine the specific position information of the base stations to obtain the current terminal location information; it is also possible to obtain the specific coordinates, altitude and other position information of the current terminal location information relatively accurately by integrating the above two methods, by obtaining positioning information, and combining it with the corresponding relationship with the base station.

[0203] In step 2, real-time scene capture, video or photos of the surrounding scene can be captured in real time using the terminal's own camera or an external camera. This capture serves as the basis for the subsequent virtual-reality fusion overlay. The specific capture method is to enable real-time capture with the corresponding camera, without enabling local storage.

[0204] Steps 5 to 8 are the process of allocating edge computing nodes to interact and distribute resources with centralized control. The specific implementation method is as follows:

[0205] Each assigned edge computing node uses the acquired terminal location information to determine the surrounding range of the actual scene. First, it determines whether there is real-scene digital resource information with the location information as the center point. If there are related resources centered on the location information, it further determines whether the existing resources can meet the digital processing requirements within all boundaries of the real scene on the terminal side. This is mainly determined by the real-scene digital distance and occlusion conditions stored by the location information at the edge computing node. Only when the stored real-scene digital distance reaches a certain threshold or has one of the occlusion requirements such as buildings / natural landscapes, can it be determined that the resource meets the requirements; if it does not meet the requirements, it is necessary to request centralized control to issue relevant updated resources;

[0206] The centralized control module, through the requirements of the corresponding edge computing node, in accordance with the relevant location information, by satisfying one of the requirements of the real scene digitization distance reaching a certain threshold, and having buildings / natural landscapes and other obstructions, and by obtaining the front, back, left, right, top, and bottom boundaries of the relevant real scene, calls the relevant real scene digitization processing results and sends them to the edge computing node.

[0207] The corresponding edge computing node synchronizes the relevant resources sent down and stores them in a manner that integrates with its current real-world digital resources. If there is overlap with the existing resources of the edge computing node, the overlapping portion is processed by determining the correspondence between the sent data and the original data. If the sent data is more detailed, the original data is overwritten; if the original data is more detailed, the original data is retained.

[0208] In steps 9 and 10, each edge computing node analyzes and judges the real scene digitization processing process, which mainly includes position information analysis to obtain the range related to the real scene boundary covered by the current position information; and through this range, obtain the real scene digitization area related to the current position of the terminal, etc., and put it into the edge computing node memory or deliver it to the smart network card, etc., to provide efficiency of remote calling.

[0209] In steps 12 to 14, as a process of correcting the real-scene digitization processing result based on the current position of the terminal. First, it is necessary to obtain the corresponding real-scene digitization processing result through the edge node and use it as the basis for subsequent related corrections; the terminal obtains the relevant camera orientation, tilt angle and other information of the terminal by collecting the information of the corresponding sensors in real time, such as gyroscopes, gravimeters, etc. If the terminal side has a ranging function, the height of the current terminal and the ground is obtained; the relevant sensor information obtained by the terminal is used for the current coordinates of the corresponding position information obtained by the edge computing node through the front / left / right / up and down three-dimensional angle offset to determine the current terminal position information offset. When the real-scene digitization coverage range sent by the edge node meets the coverage range after the offset, the correction of the real-scene digitization result is completed directly. When it exceeds the real-scene digitization coverage range sent by the edge node, it is reported to the edge node to re-acquire the matching range.

[0210] In steps 18 to 20, the terminal mainly completes the call of local virtual materials and the output of virtual-real fusion. Based on the relevant coordinate system for completing the correction of the real-scene digitization results, and in accordance with the virtual scene generation logic, the terminal calls the relevant local virtual scene map materials, object modules, character images, action animations, special effects and other resources; the virtual scene image is rendered according to the coordinate position corresponding to the real scene, and the rendering result is output in real time by overlaying the layer with the real scene. During the overlay process, it is necessary to combine the correspondence between the virtual scene and the real scene, such as related occlusion and real scene plane mapping, to ensure seamless connection between virtual and real.

[0211] In an example, the system architecture diagram of the cloud-collaborative AR implementation system is shown in Figure 10, including a cloud and multiple terminals, which may include terminal 1, terminal 2,..., terminal n; the cloud includes a centralized control node and multiple edge computing nodes, such as edge computing node 1, edge computing node 2,..., edge computing node n.

[0212] Cloud: Includes a centralized control module for processing real-scene resources, multi-dimensional scheduling and management of edge node computing power, and various edge computing nodes that specifically perform location information analysis, call and distribute real-scene digitization results, and process location information changes.

[0213] Centralized management and control module, including terminal information docking, edge node calling, and resource scheduling and deployment.

[0214] Terminal information docking: By docking with the terminal, the terminal's current location information is obtained, laying the foundation for the subsequent allocation of the nearest service edge computing node.

[0215] Edge node call: Based on the current location information of the terminal, the matching edge computing node is searched and assigned according to the proximity principle.

[0216] Resource scheduling and deployment: By detecting the edge computing node of the corresponding service, it is determined whether it has the digital resources of the corresponding real scene, and the relevant real scene digital processing results are sent to the corresponding edge computing node as needed; at the same time, through the corresponding edge node request, the corresponding real scene digital processing results are sent or updated to the corresponding edge node.

[0217] Edge computing nodes: There are several edge computing nodes, which mainly provide functions such as terminal information acquisition, digital information storage, location information analysis, acquisition of real-scene digital processing results, digital processing result correction, digital result distribution, and location information change processing. Terminal information acquisition: The edge computing node obtains terminal-related location information, laying the foundation for the subsequent mapping of real-scene digital processing results. Digital information storage: It realizes docking with centralized control, obtains real-scene digital processing results, completes corresponding updates, and completes local storage of relevant real-scene digital processing results on the edge computing node. Location information analysis: Analyzes the current location information of the terminal obtained to determine the scope of the real scene coverage, including the current unobstructed visible boundary, and uses this boundary as the benchmark to complete the subsequent call of the real-scene digital processing results. Real-scene digital processing result acquisition: Query and call the corresponding real-scene digital processing results based on the observable real-scene boundary determined by the terminal location information.

[0218] Digital result delivery: The real-scene digitization results are delivered to the terminal so that the terminal can perform subsequent related operations. Position information change processing: When a new position information change is received from the terminal and the corresponding observable real-scene boundary is synchronously sent, the subsequent real-scene digitization processing results are adjusted.

[0219] Terminal: There are several corresponding terminals, which mainly provide functions including real-scene collection, location information reporting, real-time acquisition of sensor information, acquisition of real-scene digital processing results, correction of digital processing results, call of virtual scene materials, virtual scene rendering processing, virtual-reality fusion output and position change processing. Real-scene collection: The terminal uses a built-in camera or an external camera to realize real-time video or picture collection of the real scene. Location information reporting: While the terminal is collecting the real scene, its location information is reported to the cloud. Its location information includes positioning information or related information such as distance and angle with the base station. Real-time acquisition of sensor information: While the terminal is collecting the real scene and reporting the location information, the terminal’s camera’s specific direction, tilt angle, etc. are obtained locally through various built-in sensors in the terminal, which lays the foundation for the correction of accurate digital processing results. Real-scene digital processing result acquisition: The real-scene digital processing result is obtained through the corresponding edge computing node. The range is within the real-scene boundary that the terminal can observe as determined by the edge computing node.

[0220] Correction of digital processing results: According to the relevant direction, tilt angle and other data obtained by the local sensor of the terminal, the direction, angle and height of the digital processing results of the real scene are corrected.

[0221] Virtual scene material call: The terminal side obtains local virtual materials based on the corrected direction, angle, and altitude, so as to complete the subsequent virtual scene rendering.

[0222] Virtual scene rendering processing: The terminal side will call the virtual scene material and perform graphics rendering processing based on the coordinate system corresponding to the corrected real scene digital processing result.

[0223] Virtual-reality fusion output: While rendering the virtual scene, the layer overlay and fusion output with the real scene are completed based on the digital processing results of the real scene.

[0224] Position change processing: When the current real-scene range on the terminal side changes significantly and the original real-scene digitization processing results are difficult to cover its range, corresponding position change processing is required, and the new position information is uploaded to the cloud so that the cloud can adjust the subsequent real-scene digitization range.

[0225] The cloud-coordinated AR implementation method provided by the disclosed embodiment can reuse the existing real-scene digital processing results on the cloud, and combine with the terminal's local precise positioning and graphics rendering capabilities to achieve the integration of cloud capabilities, reducing the dual pressure of cloud rendering load and real-time processing of the terminal real scene; improving the cloud AR carrying capacity and reducing the real-time load on the cloud: adopting the mode of calling the cloud real-scene digital processing results, avoiding the original cloud AR having to rely on the cloud to complete the real-time digital processing, graphics rendering output and other processing processes of the real scene, and then performing related streaming and distribution. In this technical solution, the main load pressure on the cloud is the real-time digital processing of the real scene and the real-time rendering of the virtual scene, which directly affects the carrying capacity of the cloud AR and becomes the main obstacle to the deployment of cloud AR. Using this patent method, the cloud only needs to digitize the real scene once, and store its digitized results for future use, and no related graphics rendering processing is required, which can effectively improve the cloud carrying capacity.

[0226] The cloud-coordinated AR implementation method provided by the disclosed embodiment can also effectively utilize the terminal processing capabilities while improving the service endurance level: the AR terminal operates in local mode, and the real scene is processed in real time, such as plane recognition, map reconstruction, and then the digital processing is completed. At the same time, this process not only requires sufficient capacity support, but also consumes a lot of power. This patent saves this process by issuing it from the cloud, which not only lowers the threshold of AR terminals, but also improves the cruising capability of the overall service. Make full use of the cloud resource reuse mechanism and share the real scene digitization results: This patent uses the real scene digitization processing results completed in advance to achieve multiple users and multiple terminals. The reuse and sharing can shorten the service delay and reduce the waste of computing resources caused by full processing.

[0227] It should be understood that although the various steps in the flowcharts of Figures 1-9 are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in Figures 1-9 may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0228] In one embodiment, as shown in FIG11 , a cloud-coordinated AR implementation apparatus 1100 is provided, including:

[0229] The first acquisition module 1102 is configured to acquire terminal location information and send the terminal location information to the cloud, so that the management node in the cloud determines a target edge computing node that matches the terminal location information, and the target edge computing node determines a target real scene digitization result that matches the terminal location information and returns the target real scene digitization result to the terminal;

[0230] A correction module 1104 is configured to correct the position and attitude of the target real scene digitized result based on the position and attitude information of the local sensor to obtain a corrected target real scene digitized result;

[0231] The first generating module 1106 is configured to generate a virtual scene result based on the terminal location information, fuse the virtual scene result with the corrected target real scene digitized result to obtain a virtual-real fusion result, and output the virtual-real fusion result.

[0232] In one embodiment, as shown in FIG12 , a cloud-coordinated AR implementation apparatus 1200 is provided, which is applied to a management node in the cloud and includes:

[0233] The first receiving module 1202 is configured to receive terminal location information sent by the terminal;

[0234] The first determining module 1204 is configured to determine a target edge computing node that matches the terminal location information and send the terminal location information to the target edge computing node.

[0235] In one embodiment, as shown in FIG13 , a cloud-coordinated AR implementation apparatus 1300 is provided, which is applied to an edge computing node in the cloud and includes:

[0236] The second receiving module 1302 is configured to receive the terminal location information sent by the management node;

[0237] The second collection module 1304 is configured to collect a target real scene digitization result that matches the terminal location information and return the target real scene digitization result to the terminal.

[0238] In one embodiment, a cloud-coordinated AR implementation system is provided. The system includes a terminal and a cloud. The cloud includes a management node and at least one edge computing node, wherein:

[0239] The terminal is used to collect terminal location information and send it to the cloud;

[0240] The management node in the cloud is used to receive the terminal location information; determine the target edge computing node that matches the terminal location information, and send the terminal location information to the target edge computing node;

[0241] The target edge computing node is used to receive the terminal location information sent by the management node; collect the target real scene digitization result that matches the terminal location information, and return the target real scene digitization result to the terminal;

[0242] The terminal is also used to correct the position and posture of the target real scene digitization result based on the position and posture information of the local sensor to obtain the corrected target real scene digitization result; generate a virtual scene result based on the terminal position information, fuse the virtual scene result with the corrected target real scene digitization result to obtain a virtual-reality fusion result, and output the virtual-reality fusion result.

[0243] For the specific limitations of the cloud-cooperative AR implementation device, please refer to the limitations of the cloud-cooperative AR implementation method above, which will not be repeated here. The various modules in the above-mentioned cloud-cooperative AR implementation device can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0244] In one embodiment, a communication device is provided, see Figure 14. Figure 14 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. The terminal device 1400 shown in Figure 14 includes: at least one processor 1401, a memory 1402, at least one network interface 1404 and a user interface 1403. The various components in the terminal device 1400 are coupled together through a bus system 1405. It can be understood that the bus system 1405 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 1405 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 1405 in Figure 1. In addition, in an embodiment of the present invention, a transceiver 1406 is also included. The transceiver can be multiple components, that is, including a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium.

[0245] The user interface 1403 may include a display, a keyboard, or a pointing device (eg, a mouse, a trackball, a touchpad, or a touch screen).

[0246] It is understood that the memory 1402 in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 1402 of the systems and methods described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0247] In some embodiments, the memory 1402 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system 14021 and application programs 14022 .

[0248] The operating system 14021 includes various system programs, such as the framework layer, core library layer, and driver layer, for implementing various basic services and handling hardware-based tasks. Application programs 14022 include various application programs, such as a media player (MediaPlayer) and a browser (Browser), for implementing various application services. Programs implementing the methods of the embodiments of the present invention may be included in application programs 14022.

[0249] In the embodiment of the present invention, by calling the program or instruction stored in the memory 1402, specifically, the program or instruction stored in the application 14022, the processor is configured to collect terminal location information and send the terminal location information to the cloud through the transmitter, so that the management node in the cloud determines a target edge computing node that matches the terminal location information, and the target edge computing node determines a target real scene digitization result that matches the terminal location information, and returns the target real scene digitization result to the terminal through the transmitter;

[0250] The processor is further configured to correct the position and attitude of the target real scene digitization result based on the position and attitude information of the local sensor to obtain a corrected target real scene digitization result;

[0251] The processor is further configured to generate a virtual scene result based on the terminal position information, fuse the virtual scene result with the corrected target real scene digitized result to obtain a virtual-real fusion result, and output the virtual-real fusion result.

[0252] Some or all of the methods disclosed in the above embodiments of the present invention may also be applied to processor 1401, or implemented by processor 1401, or implemented by processor 1401 in conjunction with other components (e.g., a transceiver). Processor 1401 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method may be performed by hardware integrated logic circuits or software instructions in processor 1401. The above processor 1401 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present invention may be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1402, and processor 1401 reads information in memory 1402 and, in conjunction with its hardware, completes the steps of the above method.

[0253] It is understood that the embodiments described in the embodiments of the present invention can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or a combination thereof.

[0254] For software implementation, the techniques described in the embodiments of the present invention can be implemented through modules (e.g., procedures, functions, etc.) that perform the functions described in the embodiments of the present invention. The software code can be stored in a memory and executed by the processor 1401. The memory can be implemented in the processor 1401 or external to the processor 1401.

[0255] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0256] Collecting terminal location information and sending the terminal location information to the cloud, so that the management node in the cloud determines a target edge computing node that matches the terminal location information, and the target edge computing node determines a target real scene digitization result that matches the terminal location information, and returns the target real scene digitization result to the terminal; correcting the position and posture of the target real scene digitization result based on the position and posture information of the local sensor to obtain a corrected target real scene digitization result;

[0257] Based on the terminal location information, a virtual scene result is generated, the virtual scene result is fused with the corrected target real scene digitized result to obtain a virtual-real fusion result, and the virtual-real fusion result is output. The embodiment of the present application also provides a computer program product containing instructions, which, when executed on a computer, causes the computer to perform the following steps:

[0258] Collecting terminal location information and sending the terminal location information to the cloud, so that the management node in the cloud determines a target edge computing node that matches the terminal location information, and so that the target edge computing node determines a target real scene digitization result that matches the terminal location information, and returns the target real scene digitization result to the terminal;

[0259] Correcting the position and attitude of the target real scene digitized result based on the position and attitude information of the local sensor to obtain a corrected target real scene digitized result;

[0260] Based on the terminal location information, a virtual scene result is generated, the virtual scene result is fused with the corrected target real scene digitized result to obtain a virtual-real fusion result, and the virtual-real fusion result is output. It will be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, database, or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0261] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0262] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A cloud - collaborative AR implementation method, comprising: Collecting terminal location information and sending the terminal location information to the cloud, so that a management node in the cloud determines a target edge computing node that matches the terminal location information, and so that the target edge computing node determines a target real - scene digitalization result that matches the terminal location information, and returns the target real - scene digitalization result to the terminal; Correcting the position and attitude of the target real - scene digitalization result based on the position and attitude information of the local sensor to obtain a corrected target real - scene digitalization result; Generating a virtual scene result based on the terminal location information, fusing the virtual scene result with the corrected target real - scene digitalization result to obtain a virtual - real fusion result, and outputting the virtual - real fusion result.

2. The method according to claim 1, wherein, The generating a virtual scene result based on the terminal location information includes: Extracting corresponding virtual scene materials based on the terminal location information and determining the real - scene coordinate data of the corrected target real - scene digitalization result; Performing graphic rendering on the virtual scene materials based on the real - scene coordinate data to obtain a virtual scene result.

3. The method according to claim 1, wherein The fusing the virtual scene result with the corrected target real - scene digitalization result to obtain a virtual - real fusion result includes: Performing layer overlay processing on the virtual scene result and the corrected target real - scene digitalization result to obtain a virtual - real fusion result.

4. The method according to claim 1, wherein The collecting terminal location information and sending the terminal location information to the cloud includes: Collecting updated terminal location information, and when the updated terminal location information meets a preset difference condition, sending the updated terminal location information to the cloud.

5. The method according to claim 1, wherein The terminal location information includes positioning coordinates or the position relationship between the terminal and the base station.

6. The method according to claim 5, wherein The position relationship between the terminal and the base station includes the base - station distance and the base - station angle.

7. The method according to claim 1, wherein The correcting the position and attitude of the target real - scene digitalization result based on the position and attitude information of the local sensor to obtain a corrected target real - scene digitalization result includes: Determining the status information and the current position information of the terminal based on the position and attitude information of the local sensor; When it is determined that the coverage range of the target real - scene digitalization result returned by the target edge computing node does not meet the coverage range corresponding to the offset situation determined based on the status information, determining that the target real - scene digitalization result does not meet the correction requirements; Reporting the status information as a position change result to the target edge computing node, so as to adjust the calling range of the real - scene digitalization result based on the position change result, obtain an adjusted real - scene digitalization result that matches the position change result, and send the adjusted real - scene digitalization result to the terminal; Taking the adjusted real - scene digitalization result as the corrected target real - scene digitalization result.

8. A cloud - collaborative AR implementation method, applied to a management node in the cloud, the method comprising: Receiving terminal location information sent by the terminal; Determining a target edge computing node that matches the terminal location information, and sending the terminal location information to the target edge computing node.

9. The method according to claim 8, wherein, The determining of the target edge computing node matching the terminal location information includes: Determine the target edge computing node matching the terminal location information according to the principle of proximity.

10. The method according to claim 8 further includes: Receive a request for obtaining a real-scene digitalization result sent by an edge computing node; When a real-scene digitalization result matching the request for obtaining a real-scene digitalization result is queried in the local storage space, return the real-scene digitalization result to the edge computing node.

11. A method for realizing cloud collaborative AR, applied to an edge computing node in the cloud, the method includes: Receive the terminal location information sent by the management node; Collect a target real-scene digitalization result matching the terminal location information, and return the target real-scene digitalization result to the terminal.

12. The method according to claim 11, wherein The collecting of the target real-scene digitalization result matching the terminal location information includes: When a real-scene digitalization result corresponding to the terminal location information is queried in the local storage space, determine coordinate information and orientation information based on the terminal location information; Based on the coordinate information and the orientation information, screen the real-scene digitalization result to obtain a target real-scene digitalization result.

13. The method according to claim 12 further includes: When a real-scene digitalization result corresponding to the terminal location information is not queried in the local storage space, generate a request for obtaining a real-scene digitalization result, and send the request for obtaining a real-scene digitalization result to the management node.

14. The method according to claim 12 further includes: When a real-scene digitalization result corresponding to the terminal location information is not queried in the local storage space, generate a request for obtaining real-scene resources, and send the request for obtaining real-scene resources to the management node, so that the management node returns real-scene resource data; Receive the real-scene resource data, reconstruct the real-scene coordinate data in the real-scene resource data to obtain a target real-scene digitalization result.

15. A device for realizing cloud collaborative AR includes: A first collection module, configured to collect terminal location information and send the terminal location information to the cloud, so that a management node in the cloud determines a target edge computing node matching the terminal location information, and so that the target edge computing node determines a target real-scene digitalization result matching the terminal location information, and returns the target real-scene digitalization result to the terminal; A calibration module, configured to calibrate the position and pose of the target real-scene digitalization result based on the position and pose information of the local sensor to obtain a calibrated target real-scene digitalization result; A first generation module, configured to generate a virtual scene result based on the terminal location information, fuse the virtual scene result with the calibrated target real-scene digitalization result to obtain a virtual-real fusion result, and output the virtual-real fusion result.

16. A device for realizing cloud collaborative AR, applied to a management node in the cloud, the device includes: A first receiving module, configured to receive the terminal location information sent by the terminal; A first determination module, configured to determine a target edge computing node that matches the terminal location information, and send the terminal location information to the target edge computing node.

17. An AR implementation device for cloud collaboration, which is applied to an edge computing node in the cloud. The device includes: A second receiving module, configured to receive the terminal location information sent by the management node; A second acquisition module, configured to acquire a target real-scene digitalization result that matches the terminal location information, and return the target real-scene digitalization result to the terminal.

18. An AR implementation system for cloud collaboration, including a terminal and a cloud. The cloud includes a management node and at least one edge computing node, where: The terminal is configured to acquire terminal location information and send the terminal location information to the cloud; The management node in the cloud is configured to receive the terminal location information; determine a target edge computing node that matches the terminal location information, and send the terminal location information to the target edge computing node; The target edge computing node is configured to receive the terminal location information sent by the management node; acquire a target real-scene digitalization result that matches the terminal location information, and return the target real-scene digitalization result to the terminal; The terminal is further configured to correct the position and pose of the target real-scene digitalization result based on the position and pose information of the local sensor, to obtain a corrected target real-scene digitalization result; Generate a virtual scene result based on the terminal location information, fuse the virtual scene result with the corrected target real-scene digitalization result, to obtain a virtual-real fusion result, and output the virtual-real fusion result.

19. A communication device, comprising: A transmitter and a processor; The processor is configured to acquire terminal location information, and send the terminal location information to the cloud through the transmitter, so that the management node in the cloud determines a target edge computing node that matches the terminal location information, and so that the target edge computing node determines a target real-scene digitalization result that matches the terminal location information, and return the target real-scene digitalization result to the terminal through the transmitter; The processor is further configured to correct the position and pose of the target real-scene digitalization result based on the position and pose information of the local sensor, to obtain a corrected target real-scene digitalization result; The processor is further configured to generate a virtual scene result based on the terminal location information, fuse the virtual scene result with the corrected target real-scene digitalization result, to obtain a virtual-real fusion result, and output the virtual-real fusion result.

20. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 14.

21. A computer program product, comprising a computer program, wherein, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 14.

Images