Information annotation method, real scene navigation method and terminal based on WebGL

Abstract

The present application discloses an information annotation method, a real-scene navigation method and a terminal based on WebGL. The technical solutions provided by the embodiments of the present application first render the image data within the current field of view collected by the moving camera to the graphical user interface based on WebGL to obtain a real-scene space picture containing the viewing angle position corresponding to the camera; then select the first target object to be annotated in the real-scene space picture, and determine the target position coordinates of the first target object in the world coordinate system based on the parameter relationship between the first target object and the viewing angle position; after determining the annotation information of the first target object, establish a corresponding relationship between the target position coordinates and the annotation information, so as to display the first target object and its annotation information in the three-dimensional real-scene space corresponding to the target physical space, which solves the problem that users cannot flexibly obtain detailed information about the house when viewing houses online, and improves the user's online viewing experience.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to an information annotation method, a real-scene navigation method and a terminal based on WebGL. Background Art

[0002] With the development of virtual reality (VR) technology, it has been widely used in all aspects of life. For example, in the online house viewing scene, VR technology can be used to display the real house space corresponding to the real house online, so that users can understand the relevant information of the real house without leaving home.

[0003] However, in the current online house viewing scenario, users can only see the general structure of the interior of the house. For some detailed information, such as the size of each space such as the kitchen and bedroom, and the specifications, dimensions, and brands of the items on display, they rely on the introduction of the real estate agent who is familiar with the house. The method of obtaining information is not flexible enough and the user experience is not good. Summary of the invention

[0004] In order to solve or improve the problems existing in the prior art, each embodiment of the present application provides an information annotation method, a real-scene navigation method and a terminal based on WebGL.

[0005] In one embodiment of the present application, a WebGL-based information annotation method is provided, wherein a first graphical user interface is provided by a first electronic terminal equipped with a gyroscope and a camera, and the electronic terminal is located in a target physical space and is movable. The method comprises:

[0006] During the movement, the camera is used to collect first image data within the current field of view, and the first image data is rendered on the first graphical user interface based on WebGL to obtain a real-scene space picture within the current field of view, where the real-scene space picture includes a viewing angle position mapped from the camera to the real-scene picture space;

[0007] Selecting a first target object to be marked in the real-scene spatial picture, and calculating a first relative position relationship between the first target object and the viewing angle position, wherein the first target object is at least one of a spatial object appearing in the real-scene spatial picture, an object in the spatial object, and a position point in the spatial object;

[0008] Determine the target position coordinates of the first target object in the world coordinate system based on the first rotation angle information and the first relative position relationship; the first rotation angle information is the rotation angle information collected by the gyroscope when the camera collects the first image data;

[0009] Determine the annotation information of the first target object, and establish a corresponding relationship between the target position coordinates and the annotation information, so as to display the first target object and its annotation information in a three-dimensional real scene space corresponding to the target physical space.

[0010] In another embodiment of the present application, a real-view navigation method based on WebGL is provided, and a graphical user interface is provided by an electronic terminal equipped with a gyroscope and a camera. The method includes:

[0011] In response to the navigation trigger operation, the image data within the current field of view of the camera is rendered into the graphical user interface based on WebGL to obtain a real-scene navigation screen; wherein the current location of the electronic terminal is the starting location of the navigation;

[0012] Determine the target location of navigation, and obtain navigation path data from the starting location to the target location, the navigation path data including forward direction data and travel node data;

[0013] Determine the marked position in the real-life navigation screen according to the position coordinates corresponding to the travel direction data and the rotation angle information currently collected by the gyroscope;

[0014] Based on WebGL, a forward indication icon is set at the marked position in the real-scene navigation screen, and the node position corresponding to the travel node data in the real-scene navigation screen is marked to guide the user to carry the electronic terminal from the starting position to the target position.

[0015] In another embodiment of the present application, an electronic terminal is provided. The electronic terminal includes: a memory and a processor, wherein the memory is used to store a computer program; and the processor is coupled to the memory and is used to execute the computer program to perform the steps in the WebGL-based information annotation method.

[0016] In another embodiment of the present application, another electronic terminal is provided. The electronic terminal includes a memory and a processor, wherein the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to perform the steps in the WebGL-based real-scene navigation method.

[0017] In another embodiment of the present application, a computer-readable storage medium storing computer instructions is provided. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned WebGL-based information annotation method or execute the steps in the above-mentioned WebGL-based real-scene navigation method.

[0018] The technical solutions provided by the embodiments of the present application utilize an electronic terminal to move in a target physical space, utilize a camera to collect image data within a current field of view during the movement, and render the image data within the current field of view collected by the camera on a graphical user interface based on WebGL to obtain a real-life space picture within the current field of view of the camera; then, a first target object to be marked in the real-life space picture is selected, and based on a parameter relationship between the first target object and the corresponding viewing angle position of the camera in the real-life picture space, the target position coordinates of the first target object in the world coordinate system are determined; after determining the marking information of the first target object, a corresponding relationship between the target position coordinates and the marking information is established, so as to display the first target object and its marking information in a three-dimensional real-life space corresponding to the target physical space, thereby solving the problem that a user cannot flexibly obtain relevant object information when roaming in a three-dimensional real-life space, thereby improving the user experience when roaming in a three-dimensional real-life space, especially in an online house viewing scenario, the user can flexibly obtain detailed information about each house roamed to, thereby improving the user experience of online house viewing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1a A schematic diagram of a flow chart of a WebGL-based information annotation method provided in one embodiment of the present application;

[0021] Figure 1b A schematic diagram of the spherical coordinates of a first target object provided in an embodiment of the present application;

[0022] Figure 1c A schematic diagram of a flow chart of another information annotation method based on WebGL provided in an embodiment of the present application;

[0023] Figure 1d A flowchart of another WebGL-based information annotation method provided in an embodiment of the present application;

[0024] Figure 2 A flowchart of a real-scene navigation method based on WebGL provided in one embodiment of the present application;

[0025] Figure 3 A schematic diagram of the structure of an electronic terminal provided in one embodiment of the present application;

[0026] Figure 4A schematic diagram of the structure of another electronic terminal provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The present application provides the following embodiments to solve or partially solve the problems existing in the above-mentioned solutions. In order to enable people in the technical field to better understand the solution of the present application, the technical solution in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application.

[0028] In some processes described in the specification, claims and the above-mentioned figures of the present application, multiple operations appearing in a specific order are included, and these operations may not be executed or executed in parallel in the order in which they appear in this article. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "first" and "second" to different types. In addition, the embodiments described below are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0029] Figure 1a A flow chart of a WebGL-based information annotation method provided by an embodiment of the present application is shown. The WebGL-based information annotation method provided by an embodiment of the present application is applied to a first electronic terminal, on which at least a gyroscope and a camera are installed, and a first graphical user interface can be provided. The first electronic terminal is located in a target physical space that needs to be annotated with information and can move in the target physical space. Among them, the target physical space can be any physical space that can provide online services based on its three-dimensional real-life space and needs to perform information annotation in its three-dimensional real-life space, such as a physical house space or a physical vehicle space. Taking the physical house space as an example, an online house viewing service or an online decoration service can be provided based on its three-dimensional real-life space. In order to further improve the service quality of the online house viewing service or the online decoration service, it is necessary to perform information annotation on its three-dimensional real-life space.

[0030] like Figure 1a As shown, the WebGL-based information annotation method provided in the embodiment of the present application includes:

[0031] 101. When the first electronic terminal moves in the target physical space, the first electronic terminal uses its camera to collect first image data within the current field of view, and renders the first image data onto a first graphical user interface based on WebGL to obtain a real-space picture within the current field of view, where the real-space picture includes a viewing angle position mapped from the camera to the real-space picture;

[0032] 102. Select a first target object to be marked in the real-scene spatial picture, and calculate a first relative position relationship between the first target object and the viewing angle position, wherein the first target object is at least one of a spatial object appearing in the real-scene spatial picture, an object in the spatial object, and a position point in the spatial object;

[0033] 103. Determine the target position coordinates of the first target object in the world coordinate system based on the first rotation angle information and the first relative position relationship, wherein the first rotation angle information is rotation angle information collected by the gyroscope when the camera collects the first image data;

[0034] 104. Determine the annotation information of the first target object, and establish a corresponding relationship between the target position coordinates and the annotation information, so as to display the first target object and its annotation information in a three-dimensional real scene space corresponding to the target physical space.

[0035] In this embodiment, the user can carry the first electronic terminal and move in the target physical space. During the movement, the user can use the camera of the first electronic terminal to capture image data within the current field of view, and then render the collected image data to the first graphical user interface provided by the first electronic terminal based on the WebGL technology. Among them, WebGL (WebGraphics Library) is a 3D drawing protocol. This drawing technology standard allows JavaScript and OpenGL ES2.0 to be combined together. By adding a JavaScript binding to OpenGL ES 2.0, a tool for implementing Web page rendering based on the browser is provided, providing a method for rendering 3D graphics on a web page.

[0036] This embodiment does not limit the type of the first electronic terminal. Any electronic terminal that is equipped with a gyroscope and a camera and can provide a first graphical user interface is applicable to the embodiment of this application. For example, the first electronic terminal of this embodiment can be a smart phone, a smart wearable device, a desktop computer, a laptop computer, a tablet computer, and a virtual reality (VR) device. Among them, the VR device can include a computer, a VR head-mounted device, and a VR control device, etc. The user can display the graphical user interface through the display device of the VR head-mounted device, and can interact with the user through the VR control device, but is not limited to this.

[0037] In an embodiment of the present application, the first electronic terminal stores an application and is used to present a first graphical user interface. The application is conventionally downloaded and installed through the first electronic terminal and the application is run to present the first graphical user interface to the user. The first electronic terminal may provide the first graphical user interface to the user in a variety of ways, for example, it may be rendered and displayed on a display screen of the terminal, or provided to the user through a holographic projection. For example, the first electronic terminal may include a display screen and a processor, the display screen is used to present the first graphical user interface, the first graphical user interface includes an application screen, and the processor is used to run the application, generate the first graphical user interface, and control the display of the first graphical user interface on the display screen.

[0038] The first graphical user interface provided in this embodiment can be a web page, or any interface form that supports user interaction, such as an application interface. From the perspective of application development type, if the application program run by the first electronic terminal is developed based on a browser, the first electronic terminal can provide a web page to the user when running the application program, and the web page is the first graphical user interface of this embodiment; or, if the application program run by the first electronic terminal is an App developed based on a native system, the first electronic terminal can provide an application page to the user when running the application program, and the application page provided by the App is the first graphical user interface of this embodiment. From the perspective of the functions implemented by the application, the application program run by the first electronic terminal can be an application program specifically used for information annotation, or it can be an application program for displaying a real scene space, such as an online house viewing application program, an online car viewing application program, etc.

[0039] Taking the online house viewing application scenario as an example, the target physical space can be a real house for rent or sale. On the one hand, with the help of the online house viewing application, a three-dimensional real-life space corresponding to the house can be constructed and displayed to the user, so that the user can view the house online through the three-dimensional real-life space. On the other hand, with the help of the online house viewing application, some objects in the house can be annotated with information, so that when the three-dimensional real-life space corresponding to the house is displayed, the annotated information of these objects can be displayed synchronously, so that users can obtain relevant information of these objects in a timely, convenient and flexible manner, thereby improving the user's online house viewing experience. Specifically, taking online house viewing as an example, in order to facilitate users to understand the detailed information of each house space in a timely manner, the house agent uses the first electronic terminal to shoot each house space inside the house source and annotate the relevant information of each house space.

[0040] In this embodiment, when it is necessary to annotate information of a first target object in the target physical space, the camera of the first electronic terminal can be adjusted to face the first target object so that the first target object falls within the current field of view of the camera, and then the first electronic terminal is used to collect image data in the target physical space, and the collected image data is rendered on the first graphical user interface to obtain a real-life space picture of the current target physical space, which contains the first target object to be annotated; then, the first target object to be annotated is selected from the real-life space picture, and the first target object to be annotated in the real-life space picture is annotated, so that the user can understand the annotated information of the first target object in the target physical space, so as to achieve the purpose of quickly and conveniently understanding the information of the first target object in the target physical space. In this embodiment, the first target object can be at least one of a spatial object appearing in the real-life space picture, an object in a spatial object, and a position point in a spatial object, but is not limited thereto.

[0041] For example, taking the target physical space as a house space, the first target object can be various space objects in the house space, such as a bedroom, a living room, a kitchen, a dining room, and a bathroom; or, the first target object can also be an object in the space object, such as a piece of furniture or a home appliance, such as a bed in a bedroom, a sofa in a living room, a cabinet in a kitchen, a dining table in a dining room, and a bathtub in a bathroom; or, the first target object can also be a location point in the space object, such as a location point in a kitchen, a location point in a bedroom, or a location point on a balcony. Accordingly, if the first target object is a space object, the annotation information can be the type, size, orientation, and area of ​​the space object, etc., which are used to describe the attribute information of the space object; or, if the first target object is an object in the space object, the annotation information can be the category, name, size, placement, and material of the object, etc., which are used to describe the attribute information of the object; or, if the first target object is a location point in the space object, the annotation information can be the category, location coordinates, and information of the space object to which the location point belongs, etc., which are used to describe the attribute information of the location point.

[0042] Specifically, the camera permission of the first electronic terminal can be obtained through the front-end getUserMedia, the first image data within the current field of view can be collected by the camera, and the first image data can be rendered to the first graphical user interface based on WebGL to obtain the real-space picture within the current field of view of the camera. The real-space picture includes the viewing angle position of the camera mapped to the real-space picture. The viewing angle position refers to the position point in the real-space picture space that is adapted to the shooting angle of the camera, which is equivalent to a virtual camera in the real-space picture, and the position point where the current real-space picture is taken by the virtual camera. It should be noted that the virtual camera corresponding to the camera does not actually exist in the real-space picture. The role of the virtual camera is, on the one hand, to simulate the changes in the shooting angle and shooting position of the camera when rendering the real-space picture. For the convenience of description, in the embodiment of the present application, the virtual camera is referred to as a roaming camera. In addition to simulating the changes in the camera's shooting angle and shooting position when rendering the real-life space picture, the roaming camera can also simulate the perspective change information when the user roams and / or scans in the three-dimensional space picture after rendering the three-dimensional space picture. The perspective change information includes at least one of roaming point change information, roaming sight line direction change information, and roaming perspective angle change information.

[0043] Further, after obtaining the real scene space picture within the current field of view, jump to step 102, select the first target object to be marked in the real scene space picture, and calculate the first relative position relationship between the first target object and the viewing angle position. In step 102, selecting the first target object to be marked in the real scene space picture can be achieved in the following way.

[0044] Method 1: In response to the user's selection operation on the real scene space screen, determine the target position selected on the real scene space screen, and then determine the first target object according to the target position. When determining the first target object according to the target position, it can be implemented by at least one of the following methods:

[0045] In an optional embodiment, the first target object is a position point appearing in the real scene space picture, and the target position can be directly used as the first target object. Specifically, the target position points to a position point in the real scene space picture, and the position point is the first target object to be marked.

[0046] In another optional embodiment, the first target object is a spatial object contained in the real scene space picture, and the spatial object to which the target position belongs can be used as the first target object. Specifically, the target position points to a position point in the real scene space picture, and the position point belongs to a spatial object, and the spatial object is the first target object to be marked.

[0047] In another optional embodiment, the first target object is an object that appears in the real-life space picture, then the object in the real-life space picture that is within the specified range of the target position can be used as the first target object. Specifically, the target position points to a position point in the real-life space picture, and one or more objects that exist within the specified range of the position point are the first target objects to be marked. The embodiment of the present application does not limit the specified range, for example, it can be a circular area or a sector area with a target position as the center and a specified distance as the radius, and the specified distance can be measured in terms of the number of screen pixels, for example, it can be 30 or 50 pixels.

[0048] Method 2: Input the real-scene space picture or the first image data into the target detection model running on the first electronic terminal for target detection to obtain the first target object. The target detection model refers to a neural network model for detecting at least one object present in the image. In an optional embodiment, the real-scene space picture or the first image data is input into the target detection model running on the first electronic terminal for target detection to obtain the first target object. This can be achieved in the following way: input the real-scene space picture or the first image data into the target detection model, use the target detection frame to perform target detection on the real-scene space picture or the first image data, and give the probability of the existence of the object in the target detection frame; when the probability is greater than the set threshold, it is determined that there is an object in the target detection frame, and the object in the target detection frame is used as the first target object.

[0049] The target detection model in this embodiment is deployed on the first electronic terminal, and the detection of the first target object can be completed quickly on the terminal side, reducing the pressure on the server side and realizing cross-platform operation because it is on the terminal side. Optionally, the target detection model can be a YOLO-V4 model, but is not limited to this, and it can be run in a tensorflow.js environment. Among them, TensorFlow.js is an open source JavaScript library based on hardware acceleration for training and deploying machine learning models. The YOLO-V4 algorithm is based on the original YOLO target detection architecture, and adopts the best optimization strategy in the CNN field in recent years. It has different degrees of optimization in various aspects such as data processing, backbone network, network training, activation function, loss function, etc. The YOLO-V4 model has multiple processing layers, such as a data set training layer and a target detection layer. For example, the draknet53 layer is a backbone network in the detection network, which is actually a convolutional neural network model for classifying the first target object.

[0050] Optionally, in this embodiment, it can be assumed that the set threshold for the existence of an object in the target detection frame is 0.75, then the real scene space picture or the first image data is input into the YOLO-V4 model running on the first electronic terminal, and the target detection frame is used to perform target detection on the real scene space picture or the first image data. When the probability that the object exists in the target detection frame is greater than 0.75, it is determined that the object exists in the target detection frame, and the object in the target detection frame is taken as the first target object. It should be noted that in the above-mentioned method 2, the first target object to be marked is specifically an object existing in each spatial object detected by the target detection model. Further optionally, in addition to being able to detect the objects existing in each spatial object, the above-mentioned target detection model can also mark the category of the object. In the embodiment of the present application, the first target object with marking can be a specific category of objects existing in each spatial object, such as various furniture, various household appliances, various walls, or various doors and windows.

[0051] In this embodiment, after determining the first target object to be marked, the first relative position relationship between the first target object and the viewing angle position can be further calculated. Optionally, the first relative position relationship can be realized in the following manner: taking the viewing angle position as the origin, constructing at least one spherical coordinate system in the real scene space picture, and the radius of different spherical coordinate systems is different, and determining the target spherical coordinate system where the first target object is located. Specifically, since the first electronic terminal is constantly moving, the viewing angle and position of the camera are constantly changing, that is, the viewing angle position corresponding to the camera in the real scene space picture is also constantly moving, and each first target object appearing in the real scene space picture may be closer to the viewing angle position at times and farther from the viewing angle position at times. If only one spherical coordinate system is established, then when the first target object is farther from the viewing angle position and is not within the radius of the spherical coordinate system, the first target object in the rendered real scene picture may be blurred, so at least one spherical coordinate system with different radius is established, and the adaptive target spherical coordinate system is selected according to the distance of the first target object relative to the viewing angle position, thereby improving the clarity of the real scene picture. Furthermore, after determining the target coordinate system, a position vector from the viewing position to the first target object is established in the target spherical coordinate system, and based on the viewing position and the position coordinates of the first target object in the screen coordinate system, the zenith angle, azimuth angle and modulus corresponding to the position vector are calculated to represent the first relative position relationship.

[0052] In an optional embodiment, the position coordinates and height of the first target object can be given, and the angle between the projection line of the XOY plane of the spherical coordinate system and the +X axis can be determined by calculating the vector from the position of the view position to the position of the first target object. The zenith angle θ is determined by calculating the height difference between the position of the viewing angle and the position of the first target object. Figure 1bAs shown, assuming that the spherical coordinates of the first target object point P are Where r is the distance between the origin O and point P; θ is the angle between the directed line segment OP and the positive direction of the z-axis; is the angle rotated from the x-axis to OM in the counterclockwise direction from the positive z-axis, where M is the projection of point P on the xOy plane, r, θ, The value of is r∈[0,+∞),θ∈[0,π], Then the spherical coordinates The conversion relationship into rectangular coordinates (x, y, z) is: z=rcosθ, that is, the world coordinates of the first target object are

[0053] Further, after determining the first relative position relationship between the first target object and the viewing position, in step 103, based on the first rotation angle information and the first relative position relationship, the target position coordinates of the first target object in the world coordinate system are determined. Optionally, the determination of the target position coordinates can be achieved in the following manner: according to the position coordinates of the first electronic terminal when the camera collects the first image data and the first rotation angle information, the rotation vector of the camera is calculated as the first rotation vector of the viewing position; according to the first rotation vector of the viewing position and the first relative position relationship, the target position coordinates of the first target object in the world coordinate system are determined. Specifically, when the viewing angle position rotates, Euler angles or quaternions are generated according to gyroscope data; wherein the Euler angles or quaternions represent the rotational position of the rover camera; further, according to the conversion relationship between the Euler angles or quaternions and the rotation matrix, the viewing angle front vector and the vertical upward vector of the rover camera can be determined according to the Euler angles or quaternions, and these two vectors represent the rotation matrix of the rover camera; according to the viewing angle front vector and the vertical upward vector, combined with the corresponding relationship between the viewing angle front vector and the vertical upward vector and the rotation vector, that is, the front vector is the rotation matrix × [0,1,0] of the rover camera, and the vertical upward vector is the rotation matrix × [-1,0,0] of the rover camera, thereby obtaining the first rotation vector generated by the rover camera, and according to the first rotation vector of the rover camera and the first relative position relationship, determining the target position coordinates of the first target object in the world coordinate system.

[0054] Specifically, in an electronic terminal, a gyroscope can be bound to a deviceorientation event, that is, when the gyroscope outputs new gyroscope data, the deviceorientation event will be triggered; the data output by the deviceorientation event is the data obtained by comparing the gyroscope data with the world coordinate system, which can be represented by Euler angles, namely, alpha, beta and gamma, but is not limited to this, and can also be represented by quaternions. That is, the gyroscope can be called, and the rotation matrix of the roaming camera can be obtained through the three parameters b, c, and d by listening to the deviceorientation event (obtaining the three parameters b, c, and d), and then the forward vector and the vertical upward vector of the viewing angle (the viewing angle of the camera) can be determined. When the camera rotates, the forward vector Forward is the camera rotation vector × [0,1,0], and the vertical upward vector U is the camera rotation vector × [-1,0,0].

[0055] After determining the target position coordinates of the first target object in the world coordinate system, proceed to step 104. Specifically, determine the annotation information of the first target object, and based on the target position coordinates and annotation information of the first target object, establish a corresponding relationship between the target position coordinates and the annotation information, so as to display the first target object and its annotation information in the three-dimensional real scene space corresponding to the target physical space, thereby solving the problem that the user cannot flexibly obtain relevant object information when roaming in the three-dimensional real scene space, and improving the user's experience when roaming in the three-dimensional real scene space. It should be noted that in the embodiment of the present application, the implementation form of the annotation information is not limited, for example, it can be text, pictures or a combination of pictures and texts, and it can also be some AR interaction modules with interactive functions. Among them, pictures, texts or a combination of pictures and texts can be explanations, function introductions, attribute descriptions, etc. of the first target object. AR interaction module refers to an interface element that supports user interaction, such as links, controls, etc. Among them, through the AR interaction module, detailed information and / or related information of the first target object can also be displayed to the user, and this method allows the user to participate in the interaction, which is conducive to increasing the fun of VR display and increasing user stickiness.

[0056] In the above or following embodiments of the present application, further, Figure 1c As shown, the method of this embodiment also includes: step 105, constructing a three-dimensional real scene space corresponding to the target physical space according to the corresponding relationship between each real scene space picture, each first target object and the target position coordinates of each first target object and its annotation information obtained during the movement. It should be noted that the execution order of the three-dimensional real scene space construction process described in step 105 and the information annotation process described in steps 101-104 mentioned above is not limited. Figure 1cIn the figure, the example of first executing steps 101-104 and then executing step 105 is used for illustration, but it is not limited to this; for example, the two processes can also be executed in parallel, or the construction process of the three-dimensional real scene space described in step 105 is executed first, and then the information labeling process described in steps 101-104 is executed.

[0057] In an optional embodiment, the target position coordinates may be coordinates in a world coordinate system. When constructing a real scene space corresponding to the target physical space, the multiple first image data captured may be spliced ​​to obtain a real scene image of the target physical space, and the spliced ​​real scene image may be rendered to a first graphical user interface based on WebGL to obtain a three-dimensional real scene space corresponding to the target physical space, and then the corresponding relationship between the position coordinates of each first target object and the calculated first target object and its annotation information is associated and stored, and when the first target object is displayed, its corresponding annotation information is synchronously displayed in the three-dimensional real scene space, so that the user can quickly understand the detailed information of the target physical space, thereby improving the user experience. It should be noted that the three-dimensional real scene space constructed by the embodiment of the present application includes multiple roaming points, and the roaming points are set between the different real scene space pictures mentioned above, and the three-dimensional real scene space supports users to perform roaming operations between different roaming points.

[0058] Based on the above, after constructing the three-dimensional real space corresponding to the target physical space, the user can display the three-dimensional real space through his or her electronic device and roam in the three-dimensional real space to understand the target physical space online. Figure 1c As shown, the WebGL-based information annotation method also includes:

[0059] 106. Providing a second graphical user interface through a second electronic terminal, displaying a three-dimensional real scene space on the second graphical user interface, wherein the three-dimensional real scene space includes a plurality of roaming points;

[0060] 107. According to the target roaming point currently roamed, determine a first target object in the first real scene space picture corresponding to the target roaming point as the second target object;

[0061] 108. Display the first real space image and the second target object and its annotation information on the second graphical user interface according to the correspondence between the target position coordinates of the second target object and the annotation information.

[0062] In this embodiment, the second electronic terminal and the first electronic terminal may be the same electronic terminal or different electronic terminals, and this embodiment does not limit this. When the second electronic terminal and the first electronic terminal are the same electronic terminal, the electronic terminal is used to construct the three-dimensional real scene space and to display the three-dimensional real scene space to the user. When the second electronic terminal and the first electronic terminal are different electronic terminals, the first electronic terminal is used to construct the three-dimensional real scene space, and the second electronic terminal is used to display the three-dimensional real scene space to the user. Taking the online house viewing scene as an example, the real estate agent uses his own electronic terminal to shoot the house and generate the three-dimensional real scene space corresponding to the house, and publishes the three-dimensional real scene space online, and the user uses his own electronic terminal to view the house online.

[0063] In steps 107 and 108, the user can roam and / or scan in the three-dimensional real space. By tracking the user's perspective change information during the roaming and / or scanning process, the roaming point to which the user is currently roaming can be determined, which is recorded as the target roaming point, and the real space picture corresponding to the target roaming point, which is recorded as the first real space picture. Among them, the first real space picture is a real space picture corresponding to the target roaming point contained in the three-dimensional real space, and the real space picture contains a first target object that has been annotated with information. In this embodiment, for the convenience of description and distinction, the first target object in the first real space picture is recorded as the second target object. In this embodiment, the second target object refers to the target object that needs to display the annotated information. Furthermore, according to the correspondence between the position coordinates of the second target object in the first real space picture and the annotated information, the first real space picture and the second target object and its annotated information can be displayed on the second graphical user interface.

[0064] Further, step 108, based on the correspondence between the target position coordinates of the second target object and the annotation information, the first real space picture and the second target object and its annotation information are displayed on the second graphical user interface, which can be achieved in the following manner: according to the target position coordinates of the second target object, combined with the current position coordinates of the second electronic terminal and the second rotation angle information currently collected by the gyroscope of the second electronic terminal, the second relative position relationship between the second target object and the target roaming point (i.e., the roaming point corresponding to the current roaming perspective) is determined; based on the second relative position relationship, the first real space picture and the second target object and its annotation information are displayed on the second graphical user interface. In this embodiment, the determination of the second relative position relationship is the same as the determination method of the first relative position relationship described above, and will not be repeated here.

[0065] Furthermore, in an optional embodiment, the annotation information of the second target object is text information, picture information, audio and video information, or a combination of text and picture information. When the second target object is a spatial object in the target physical space, regardless of the implementation form of the annotation information, its content may include but is not limited to: the type, orientation, area size, internal layout information, internal home decoration information, and community environment information that can be viewed through the form of the spatial object, etc. In another optional embodiment, the annotation information of the second target object is an AR interaction module. Accordingly, the WebGL-based information annotation method also includes:

[0066] 109. In response to the triggering operation on the AR interaction module, display an AR interaction interface, where the AR interaction interface includes detailed information of the second target object and / or other related information.

[0067] The above content of the embodiment of the present application is to use the first electronic terminal to annotate the second target object, and to construct a three-dimensional real scene space, and to display the first target object and the annotated information in the three-dimensional real scene space during the display of the three-dimensional real scene space on the second electronic terminal. In an optional embodiment, the above-mentioned WebGL-based information annotation method can also be used for real scene navigation, such as Figure 1d As shown, another information annotation method based on WebGL includes:

[0068] 110. In response to the navigation trigger operation, render the second image data within the current field of view of the camera into the first graphical user interface based on WebGL to obtain a real-scene navigation screen; wherein the current location of the first electronic terminal is the starting location of the navigation;

[0069] 111. Determine the target location of navigation, and obtain navigation path data from the starting location to the target location, where the navigation path data includes forward direction data and travel node data;

[0070] 112. Determine the marked position in the real-view navigation picture according to the position coordinates corresponding to the traveling direction data and the third rotation angle information currently collected by the gyroscope;

[0071] 113. A forward indication icon is set at the marked position in the real-scene navigation picture based on WebGL, and the node position corresponding to the travel node data in the real-scene navigation picture is marked to guide the user to carry the first electronic terminal from the starting position to the target position.

[0072] In step 111, the forward direction data may be the direction of walking, such as east, south, west, north, or front, back, left, right, etc. For forward node data, the forward node data may be different depending on the navigation scenario. For example, when walking on the road, the forward node data may be a turn or an intersection. When walking in a house, the forward node data may be data from the living room to the bathroom, or a door or corridor in the room. And obtaining navigation path data from the starting position to the target position may be achieved in the following manner: sending the second image data and the target position to the server, so that the server can determine the current position of the first electronic terminal as the starting position according to the second image data and plan the navigation path data from the starting position to the target position; and receiving the navigation path data returned by the server.

[0073] In step 112, the traveling direction data actually also includes many target position points. The marked position in the real-life navigation screen is determined according to the position coordinates corresponding to the target position points and the third rotation angle information of the viewing position currently collected by the gyroscope at the target position points.

[0074] In step 113, the forward indicator icon can be an arrow or other sign indicating a direction, which is not limited in this embodiment. Based on WebGL, a forward indicator icon is set at the marked position in the real-life navigation screen, and the node position corresponding to the travel node data in the real-life navigation screen is marked, which can be used to inform the user of the specific location they are currently at, such as a certain intersection, T-junction or crossroads, etc., to guide the user to carry the first electronic terminal from the starting position to the target position. The real-life navigation on the Web provided in this embodiment can enhance the user's sense of immersion and has the advantage of being cross-platform and lightweight.

[0075] In an optional embodiment, the starting position and the target position are different positions in the target physical space. Accordingly, the first image data within the current field of view can be collected by using the camera. This can be achieved in the following manner: when the first electronic terminal moves to the target position, the first image data within the current field of view is collected by using the camera. For example, if the target position is position B, the first electronic terminal moves from position A to position B and takes a picture at position B. Alternatively, the starting position is a position outside the target physical space, the target position is the position where the target physical space is located, and the real-view navigation screen is used to guide the user from a position outside the target physical space to the target physical space. For example, if the target position is house A, and the real estate agent navigates from the agency to house A, then the starting position is the agency outside the position of house A; or, the starting position and the target position are different positions outside the target physical space, and the real-view navigation screen is used to guide the user from a position outside the target physical space to another position outside the target physical space. For example, if the target position is house A, and the agent wants to take the user to see house A, and the agent navigates from the agency to house B, passing house A on the way, then the starting position and the target position are different positions outside the target physical space.

[0076] In an optional embodiment, the second electronic terminal may also periodically send a data request to the server to request an update of the current navigation status, thereby avoiding navigation errors due to excessive navigation time.

[0077] The technical solutions provided by the embodiments of the present application utilize an electronic terminal to move in a target physical space, utilize a camera to collect image data within a current field of view during the movement, and render the image data within the current field of view collected by the camera on a graphical user interface based on WebGL to obtain a real-life space picture within the current field of view of the camera; then, a first target object to be marked in the real-life space picture is selected, and based on a parameter relationship between the first target object and the corresponding viewing angle position of the camera in the real-life picture space, the target position coordinates of the first target object in the world coordinate system are determined; after determining the marking information of the first target object, a corresponding relationship between the target position coordinates and the marking information is established, so as to display the first target object and its marking information in a three-dimensional real-life space corresponding to the target physical space, thereby solving the problem that a user cannot flexibly obtain relevant object information when roaming in a three-dimensional real-life space, thereby improving the user experience when roaming in a three-dimensional real-life space, especially in an online house viewing scenario, the user can flexibly obtain detailed information about each house roamed to, thereby improving the user experience of online house viewing.

[0078] In the process of constructing the 3D real scene space corresponding to the target physical space, in addition to providing an information annotation method based on WebGL, a real scene navigation method based on WebGL is also provided. Figure 2 Description of the illustrated embodiment.

[0079] Figure 2 The flowchart of the WebGL-based real-scene navigation method provided by an embodiment of the present application is shown. The WebGL-based information annotation method provided by the embodiment of the present application is applied to an electronic terminal, which is at least equipped with a gyroscope and a camera and can provide a first graphical user interface. Figure 2 As shown, the real-scene navigation method based on WebGL provided in the embodiment of the present application includes:

[0080] 201. In response to a navigation trigger operation, image data within a current field of view of the camera is rendered into a graphical user interface based on WebGL to obtain a real-scene navigation screen; wherein the current location of the electronic terminal is the starting location of the navigation;

[0081] 202. Determine a navigation target location, and obtain navigation path data from the starting location to the target location, where the navigation path data includes forward direction data and travel node data;

[0082] 203. Determine the marked position in the real-view navigation picture according to the position coordinates corresponding to the traveling direction data and the rotation angle information currently collected by the gyroscope;

[0083] 204. A forward indication icon is set at the marked position in the real-scene navigation picture based on WebGL, and the node position corresponding to the travel node data in the real-scene navigation picture is marked to guide the user to carry the electronic terminal from the starting position to the target position.

[0084] The technical solution provided by the embodiment of the present application is that the electronic terminal uses the camera to collect image data within the current field of view during movement, and renders the data within the current field of view collected by the camera on the graphical user interface based on WebGL to obtain a real-life navigation picture within the current field of view of the camera; then the navigation target position is determined, and the forward direction data and travel node data from the starting position to the target position are obtained; according to the position coordinates corresponding to the travel direction data and the rotation angle information currently collected by the gyroscope, the marked position in the real-life navigation picture is determined; finally, a forward indication icon is set at the marked position in the real-life navigation picture based on WebGL, and the node position corresponding to the travel node data in the real-life navigation picture is marked to guide the user to carry the electronic terminal from the starting position to the target position, thereby solving the problem that the user cannot flexibly obtain information related to his or her location and cannot accurately reach the destination during space roaming, thereby improving the user's real-life navigation experience.

[0085] It should be noted here that the detailed implementation of the above steps can be found in the description of the corresponding steps in the above embodiments, which will not be repeated here.

[0086] Figure 3 FIG. 1 shows a schematic diagram of the structure of an electronic terminal provided by an embodiment of the present application. Figure 3 As shown, the electronic terminal includes: a memory 30a and a processor 30b; wherein the memory 30a is used to store a computer program; the processor 30b is coupled to the memory 30a, and is used to execute the computer program to perform the following operations:

[0087] During the movement, the camera is used to collect first image data within the current field of view, and the first image data is rendered on the first graphical user interface based on WebGL to obtain a real-space picture within the current field of view, wherein the real-space picture includes a viewing angle position mapped from the camera to the real-space picture;

[0088] Selecting a first target object to be marked in the real-scene spatial picture, and calculating a first relative position relationship between the first target object and the viewing angle position, wherein the first target object is at least one of a spatial object appearing in the real-scene spatial picture, an object in the spatial object, and a position point in the spatial object;

[0089] Determine the target position coordinates of the first target object in the world coordinate system based on the first rotation angle information and the first relative position relationship; the first rotation angle information is the rotation angle information collected by the gyroscope when the camera collects the first image data;

[0090] Determine the annotation information of the first target object, and establish a corresponding relationship between the target position coordinates and the annotation information, so as to display the first target object and its annotation information in a three-dimensional real scene space corresponding to the target physical space.

[0091] Furthermore, when the processor 30b is used to select the first target object to be marked in the real-scene space picture, it is specifically used to: determine the selected target position on the real-scene space picture in response to a selection operation on the real-scene space picture, and determine the first target object based on the target position; or, input the real-scene space picture or the first image data into a target detection model running on the first electronic terminal side for target detection to obtain the first target object.

[0092] Furthermore, when the processor 30b is used to determine the first target object based on the target position, it is specifically used in at least one of the following ways: using the target position as the first target object; using the spatial object to which the target position belongs as the first target object; using the object in the real-life space picture that is within the specified range of the target position as the first target object.

[0093] Furthermore, when the processor 30b is used to input the real-scene space picture or the first image data into the target detection model running on the first electronic terminal side for target detection to obtain the first target object, it is specifically used to: input the real-scene space picture or the first image data into the target detection model, use the target detection frame to perform target detection on the real-scene space picture or the first image data, and give the probability that an object exists in the target detection frame; when the probability is greater than a set threshold, determine that there is an object in the target detection frame, and use the object in the target detection frame as the first target object.

[0094] Furthermore, when the processor 30b is used to calculate the first relative position relationship between the first target object and the viewing position, it is specifically used to: construct at least one spherical coordinate system with the viewing position as the origin, and different spherical coordinate systems have different radii to determine the target spherical coordinate system where the first target object is located; in the target spherical coordinate system, establish a position vector from the viewing position to the first target object, and calculate the zenith angle, azimuth angle and modulus corresponding to the position vector according to the viewing position and the position coordinates of the first target object in the screen coordinate system to represent the first relative position relationship.

[0095] Further, when the processor 30b is used to determine the target position coordinates of the first target object in the world coordinate system based on the first rotation angle information and the first relative position relationship, it is specifically used to: calculate the rotation vector of the camera as the first rotation vector of the viewing position according to the position coordinates of the first electronic terminal when the camera collects the first image data and the first rotation angle information; determine the target position coordinates of the first target object in the world coordinate system according to the first rotation vector of the viewing position and the first relative position relationship.

[0096] Furthermore, the processor 30b is also used to construct a three-dimensional real space corresponding to the target physical space according to the real space images, the first target objects and the correspondence between the target position coordinates of the first target objects and their annotation information obtained during the movement.

[0097] Furthermore, after constructing the three-dimensional real scene space corresponding to the target physical space, the processor 30b is also used to: provide a second graphical user interface through a second electronic terminal, and display the three-dimensional real scene space on the second graphical user interface, wherein the three-dimensional real scene space includes multiple roaming points; determine, according to the target roaming point currently roamed to, the first target object in the first real scene space picture corresponding to the target roaming point as the second target object; and display the first real scene space picture and the second target object and its annotation information on the second graphical user interface according to the correspondence between the target position coordinates and the annotation information of the second target object.

[0098] Further, when the processor 30b is used to display the first real-scene space picture and the second target object and its annotation information on the second graphical user interface based on the correspondence between the target position coordinates of the second target object and the annotation information, it is specifically used to: determine the second relative position relationship between the second target object and the target roaming point based on the target position coordinates of the second target object, in combination with the current position coordinates of the second electronic terminal and the second rotation angle information currently collected by the gyroscope of the second electronic terminal; and display the first real-scene space picture and the second target object and its annotation information on the second graphical user interface based on the second relative position relationship.

[0099] Furthermore, the annotation information of the second target object is an AR interaction module, and the processor 30b is also used to: in response to a triggering operation on the AR interaction module, display an AR interaction interface, and the AR interaction interface includes detailed information and / or other related information of the second target object.

[0100] Furthermore, the processor 30b is also used to: in response to a navigation trigger operation, render the second image data within the current field of view of the camera to the first graphical user interface based on WebGL to obtain a real-life navigation screen; wherein the current location of the first electronic terminal is the starting location of the navigation; determine the target location of the navigation, and obtain navigation path data from the starting location to the target location, the navigation path data including forward direction data and travel node data; determine the marked position in the real-life navigation screen according to the position coordinates corresponding to the travel direction data and the third rotation angle information currently collected by the gyroscope; set a forward indication icon at the marked position in the real-life navigation screen based on the WebGL, and mark the node position corresponding to the travel node data in the real-life navigation screen to guide the user to carry the first electronic terminal from the starting position to the target position.

[0101] Furthermore, when the processor 30b is used to obtain navigation path data from the starting position to the target position, it is specifically used to: send the second image data and the target position to the server, so that the server can determine the current position of the first electronic terminal as the starting position based on the second image data and plan the navigation path data from the starting position to the target position; and receive the navigation path data returned by the server.

[0102] Further, the starting position and the target position are different positions within the target physical space. Accordingly, when the processor 30b is used to collect the first image data within the current field of view using the camera, it is specifically used to: when the first electronic terminal moves to the target position, collect the first image data within the current field of view using the camera; or, the starting position is a position outside the target physical space, the target position is a position where the target physical space is located, and the real-scene navigation screen is used to guide the user from a position outside the target physical space to the target physical space; or, the starting position and the target position are different positions outside the target physical space, and the real-scene navigation screen is used to guide the user from a position outside the target physical space to another position outside the target physical space.

[0103] Furthermore, if Figure 3 As shown, the electronic device also includes: a communication component 3c, a display 3d, a power component 3e, an audio component 3f and other components. Figure 3 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 3 The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone or an IOT device.

[0104] It should be noted here that: the electronic device provided in the above embodiment can implement the technical solution described in the above WebGL-based information annotation method embodiment, and the specific implementation principles of the above modules or units can be found in the corresponding contents of the above method embodiments, which will not be repeated here.

[0105] Figure 4 FIG. 1 shows a schematic diagram of the structure of an electronic terminal provided by an embodiment of the present application. Figure 4 As shown, the electronic terminal includes: a memory 40a and a processor 40b; wherein the memory 40a is used to store a computer program; the processor 40b is coupled to the memory 40a and is used to execute the computer program to perform the following operations:

[0106] In response to the navigation trigger operation, the image data within the current field of view of the camera is rendered into the graphical user interface based on WebGL to obtain a real-scene navigation screen; wherein the current location of the electronic terminal is the starting location of the navigation;

[0107] Determine a navigation target location, and obtain navigation path data from the starting location to the target location, wherein the navigation path data includes forward direction data and travel node data;

[0108] Determine the marked position in the real-view navigation picture according to the position coordinates corresponding to the traveling direction data and the rotation angle information currently collected by the gyroscope;

[0109] Based on the WebGL, a forward indication icon is set at the marked position in the real-scene navigation picture, and the node position corresponding to the travel node data in the real-scene navigation picture is marked to guide the user to carry the electronic terminal from the starting position to the target position.

[0110] Furthermore, if Figure 4 As shown, the electronic device also includes: a communication component 40c, a display 40d, a power component 40e, an audio component 40f and other components. Figure 4 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 4 The electronic device of this embodiment can be implemented as a terminal device such as a desktop computer, a laptop computer, a smart phone or an IOT device.

[0111] It should be noted here that: the electronic device provided in the above embodiment can implement the technical solution described in the above WebGL-based real-scene navigation method embodiment. The specific implementation principles of the above modules or units can be found in the corresponding contents of the above method embodiments, which will not be repeated here.

[0112] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the following operations:

[0113] During the movement, the camera is used to collect first image data within the current field of view, and the first image data is rendered on the first graphical user interface based on WebGL to obtain a real-space picture within the current field of view, wherein the real-space picture includes a viewing angle position mapped from the camera to the real-space picture;

[0114] Selecting a first target object to be marked in the real-scene spatial picture, and calculating a first relative position relationship between the first target object and the viewing angle position, wherein the first target object is at least one of a spatial object appearing in the real-scene spatial picture, an object in the spatial object, and a position point in the spatial object;

[0115] Determine the target position coordinates of the first target object in the world coordinate system based on the first rotation angle information and the first relative position relationship; the first rotation angle information is the rotation angle information collected by the gyroscope when the camera collects the first image data;

[0116] Determine the annotation information of the first target object, and establish a corresponding relationship between the target position coordinates and the annotation information, so as to display the first target object and its annotation information in a three-dimensional real scene space corresponding to the target physical space.

[0117] Furthermore, when the processor is used to select the first target object to be marked in the real-scene space picture, it is specifically used to: determine the selected target position on the real-scene space picture in response to a selection operation on the real-scene space picture, and determine the first target object based on the target position; or, input the real-scene space picture or the first image data into a target detection model running on the first electronic terminal side to perform target detection to obtain the first target object.

[0118] Furthermore, when the processor is used to determine the first target object based on the target position, it is specifically used in at least one of the following ways: using the target position as the first target object; using the spatial object to which the target position belongs as the first target object; using the object in the real-life space picture that is within the specified range of the target position as the first target object.

[0119] Furthermore, when the processor is used to input the real-scene space picture or the first image data into a target detection model running on the first electronic terminal side for target detection to obtain the first target object, it is specifically used to: input the real-scene space picture or the first image data into the target detection model, use the target detection frame to perform target detection on the real-scene space picture or the first image data, and give the probability that an object exists in the target detection frame; when the probability is greater than a set threshold, determine that there is an object in the target detection frame, and use the object in the target detection frame as the first target object.

[0120] Furthermore, when the processor is used to calculate the first relative position relationship between the first target object and the viewing position, it is specifically used to: construct at least one spherical coordinate system with the viewing position as the origin, and different spherical coordinate systems have different radii to determine the target spherical coordinate system where the first target object is located; in the target spherical coordinate system, establish a position vector from the viewing position to the first target object, and calculate the zenith angle, azimuth angle and modulus corresponding to the position vector based on the viewing position and the position coordinates of the first target object in the screen coordinate system to represent the first relative position relationship.

[0121] Furthermore, when the processor is used to determine the target position coordinates of the first target object in the world coordinate system based on the first rotation angle information and the first relative position relationship, it is specifically used to: calculate the rotation vector of the camera as the first rotation vector of the viewing position according to the position coordinates of the first electronic terminal when the camera collects the first image data and the first rotation angle information; determine the target position coordinates of the first target object in the world coordinate system according to the first rotation vector of the viewing position and the first relative position relationship.

[0122] Furthermore, the processor is also used to construct a three-dimensional real scene space corresponding to the target physical space according to the real scene space images, the first target objects and the correspondence between the target position coordinates of the first target objects and their annotation information obtained during the movement.

[0123] Furthermore, after constructing the three-dimensional real scene space corresponding to the target physical space, the processor is also used to: provide a second graphical user interface through a second electronic terminal, and display the three-dimensional real scene space on the second graphical user interface, wherein the three-dimensional real scene space includes multiple roaming points; determine, according to the target roaming point currently roamed to, the first target object in the first real scene space picture corresponding to the target roaming point as the second target object; and display the first real scene space picture and the second target object and its annotation information on the second graphical user interface according to the correspondence between the target position coordinates and the annotation information of the second target object.

[0124] Furthermore, when the processor is used to display the first real-scene space picture and the second target object and its annotation information on the second graphical user interface based on the correspondence between the target position coordinates of the second target object and the annotation information, it is specifically used to: determine the second relative position relationship between the second target object and the target roaming point based on the target position coordinates of the second target object, in combination with the current position coordinates of the second electronic terminal and the second rotation angle information currently collected by the gyroscope of the second electronic terminal; and display the first real-scene space picture and the second target object and its annotation information on the second graphical user interface based on the second relative position relationship.

[0125] Furthermore, the annotation information of the second target object is an AR interaction module, and the processor is also used to: in response to a triggering operation on the AR interaction module, display an AR interaction interface, and the AR interaction interface includes detailed information and / or other related information of the second target object.

[0126] Furthermore, the processor is also used to: in response to a navigation trigger operation, render the second image data within the current field of view of the camera to the first graphical user interface based on WebGL to obtain a real-life navigation screen; wherein the current location of the first electronic terminal is the starting location of the navigation; determine the target location of the navigation, and obtain navigation path data from the starting location to the target location, the navigation path data including forward direction data and travel node data; determine the marked position in the real-life navigation screen according to the position coordinates corresponding to the travel direction data and the third rotation angle information currently collected by the gyroscope; set a forward indication icon at the marked position in the real-life navigation screen based on the WebGL, and mark the node position corresponding to the travel node data in the real-life navigation screen to guide the user to carry the first electronic terminal from the starting position to the target position.

[0127] Furthermore, when the processor is used to obtain navigation path data from the starting position to the target position, it is specifically used to: send the second image data and the target position to the server, so that the server can determine the current position of the first electronic terminal as the starting position based on the second image data and plan the navigation path data from the starting position to the target position; and receive the navigation path data returned by the server.

[0128] Further, the starting position and the target position are different positions within the target physical space. Accordingly, when the processor is used to use the camera to collect the first image data within the current field of view, it is specifically used to: when the first electronic terminal moves to the target position, use the camera to collect the first image data within the current field of view; or, the starting position is a position outside the target physical space, the target position is a position where the target physical space is located, and the real-scene navigation screen is used to guide the user from a position outside the target physical space to the target physical space; or, the starting position and the target position are different positions outside the target physical space, and the real-scene navigation screen is used to guide the user from a position outside the target physical space to another position outside the target physical space.

[0129] It should be noted here that: the electronic device provided in the above embodiment can implement the technical solution described in the above WebGL-based information annotation method embodiment, and the specific implementation principles of the above modules or units can be found in the corresponding contents of the above method embodiments, which will not be repeated here.

[0130] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the following operations:

[0131] In response to the navigation trigger operation, the image data within the current field of view of the camera is rendered into the graphical user interface based on WebGL to obtain a real-scene navigation screen; wherein the current location of the electronic terminal is the starting location of the navigation;

[0132] Determine a navigation target location, and obtain navigation path data from the starting location to the target location, wherein the navigation path data includes forward direction data and travel node data;

[0133] Determine the marked position in the real-view navigation picture according to the position coordinates corresponding to the traveling direction data and the rotation angle information currently collected by the gyroscope;

[0134] Based on the WebGL, a forward indication icon is set at the marked position in the real-scene navigation picture, and the node position corresponding to the travel node data in the real-scene navigation picture is marked to guide the user to carry the electronic terminal from the starting position to the target position.

[0135] It should be noted here that: the electronic device provided in the above embodiment can implement the technical solution described in the above WebGL-based real-scene navigation method embodiment. The specific implementation principles of the above modules or units can be found in the corresponding contents of the above method embodiments, which will not be repeated here.

[0136] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0137] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present application can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0138] The present application is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable coordinate determination device to produce a machine, so that the instructions executed by the processor of the computer or other programmable coordinate determination device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0139] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable coordinate determination device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0140] These computer program instructions may also be loaded onto a computer or other programmable coordinate determination device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in a flowchart and / or one or more boxes in a block diagram.

[0141] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0142] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0143] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A WebGL-based information annotation method provides a first graphical user interface through a first electronic terminal equipped with a gyroscope and a camera, wherein the first electronic terminal is located in a target physical space and is movable, characterized in that: The method comprises: During the movement, the camera is used to collect first image data within the current field of view, and the first image data is rendered on the first graphical user interface based on WebGL to obtain a real-space picture within the current field of view, wherein the real-space picture includes a viewing angle position mapped from the camera to the real-space picture; Selecting a first target object to be marked in the real-scene spatial picture, and calculating a first relative position relationship between the first target object and the viewing angle position, wherein the first target object is at least one of a spatial object appearing in the real-scene spatial picture, an object in the spatial object, and a position point in the spatial object; Calculate the rotation vector of the camera as the first rotation vector of the viewing angle position according to the position coordinates and the first rotation angle information of the first electronic terminal when the camera collects the first image data; determine the target position coordinates of the first target object in the world coordinate system according to the first rotation vector of the viewing angle position and the first relative position relationship; the first rotation angle information is the rotation angle information collected by the gyroscope when the camera collects the first image data; Determine the annotation information of the first target object, and establish a corresponding relationship between the target position coordinates and the annotation information, so as to display the first target object and its annotation information in a three-dimensional real scene space corresponding to the target physical space.

2. The method according to claim 1, characterized in that Selecting a first target object to be marked in the real scene space picture includes: In response to a selection operation on the real scene space screen, determining a target position selected on the real scene space screen, and determining the first target object according to the target position; or, The real scene space picture or the first image data is input into a target detection model running on the first electronic terminal side to perform target detection to obtain the first target object.

3. The method according to claim 2, characterized in that Determining the first target object according to the target position includes at least one of the following methods: Taking the target position as the first target object; Taking the spatial object to which the target position belongs as the first target object; An object in the real-space image that is within the specified range of the target position is used as the first target object.

4. The method according to claim 2, characterized in that: Inputting the real scene space picture or the first image data into a target detection model running on the first electronic terminal side to perform target detection to obtain the first target object includes: Inputting the real scene space picture or the first image data into the target detection model, performing target detection on the real scene space picture or the first image data using a target detection frame, and providing a probability that an object exists in the target detection frame; When the probability is greater than a set threshold, it is determined that there is an object in the target detection frame, and the object in the target detection frame is used as the first target object.

5. The method according to claim 1, characterized in that Calculating a first relative position relationship between the first target object and the viewing angle position includes: Taking the viewing angle position as the origin, constructing at least one spherical coordinate system, where different spherical coordinate systems have different radii, and determining a target spherical coordinate system where the first target object is located; In the target spherical coordinate system, a position vector from the viewing position to the first target object is established, and based on the viewing position and the position coordinates of the first target object in the screen coordinate system, the zenith angle, azimuth angle and modulus corresponding to the position vector are calculated to represent the first relative position relationship.

6. The method according to claim 1, characterized in that Also includes: According to the real scene space images, the first target objects and the correspondence between the target position coordinates of the first target objects and their annotation information obtained during the movement, a three-dimensional real scene space corresponding to the target physical space is constructed.

7. The method according to claim 6, characterized in that After constructing the three-dimensional real scene space corresponding to the target physical space, the method further includes: Providing a second graphical user interface through a second electronic terminal, displaying the three-dimensional real scene space on the second graphical user interface, wherein the three-dimensional real scene space includes a plurality of roaming points; According to the target roaming point currently roamed, determining a first target object in the first real scene space picture corresponding to the target roaming point as the second target object; The first real space picture and the second target object and its annotation information are displayed on the second graphical user interface according to the correspondence between the target position coordinates and the annotation information of the second target object.

8. The method according to claim 7, characterized in that Displaying the first real space image and the second target object and its annotation information on the second graphical user interface according to the correspondence between the target position coordinates and the annotation information of the second target object includes: Determine a second relative position relationship between the second target object and the target roaming point according to the target position coordinates of the second target object, in combination with the current position coordinates of the second electronic terminal and the second rotation angle information currently collected by the gyroscope of the second electronic terminal; The first real space image and the second target object and its annotation information are displayed on the second graphical user interface according to the second relative position relationship.

9. The method according to claim 8, characterized in that The annotation information of the second target object is an AR interaction module, and the method further includes: In response to a triggering operation on the AR interaction module, an AR interaction interface is displayed, where the AR interaction interface includes detailed information of the second target object and / or other relevant information.

10. The method according to any one of claims 1 to 9, characterized in that: Also includes: In response to the navigation trigger operation, the second image data within the current field of view of the camera is rendered into the first graphical user interface based on WebGL to obtain a real-scene navigation screen; wherein the current location of the first electronic terminal is the starting location of the navigation; Determine a navigation target location, and obtain navigation path data from the starting location to the target location, wherein the navigation path data includes travel direction data and travel node data; Determine the marked position in the real-view navigation picture according to the position coordinates corresponding to the traveling direction data and the third rotation angle information currently collected by the gyroscope; Based on the WebGL, a forward indication icon is set at the marked position in the real-scene navigation picture, and the node position corresponding to the travel node data in the real-scene navigation picture is marked to guide the user to carry the first electronic terminal from the starting position to the target position.

11. The method according to claim 10, characterized in that Acquiring navigation path data from the starting position to the target position, including: The second image data and the target location are sent to a server, so that the server can determine the current location of the first electronic terminal as the starting location based on the second image data and plan navigation path data from the starting location to the target location; and receive the navigation path data returned by the server.

12. The method according to claim 10, characterized in that The starting position and the target position are different positions in the target physical space, and accordingly, using the camera to collect first image data within the current field of view includes: when the first electronic terminal moves to the target position, using the camera to collect first image data within the current field of view; or, The starting position is a position outside the target physical space, the target position is a position where the target physical space is located, and the real-scene navigation screen is used to guide the user from the position outside the target physical space to the target physical space; or, The starting position and the target position are different positions outside the target physical space, and the real-scene navigation screen is used to guide the user from one position outside the target physical space to another position outside the target physical space.

13. A real-scene navigation method based on WebGL, providing a graphical user interface through an electronic terminal equipped with a gyroscope and a camera, characterized in that: The method comprises: In response to the navigation trigger operation, the image data within the current field of view of the camera is rendered into the graphical user interface based on WebGL to obtain a real-scene navigation screen; wherein the current location of the electronic terminal is the starting location of the navigation; Determine a navigation target location, and obtain navigation path data from the starting location to the target location, wherein the navigation path data includes travel direction data and travel node data; Determine the marked position in the real-view navigation picture according to the position coordinates corresponding to the traveling direction data and the rotation angle information currently collected by the gyroscope; Based on the WebGL, a forward indication icon is set at the marked position in the real-scene navigation picture, and the node position corresponding to the travel node data in the real-scene navigation picture is marked to guide the user to carry the electronic terminal from the starting position to the target position.

14. An electronic terminal, characterized in that: include: A memory and a processor; wherein the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to perform the steps in the method according to any one of claims 1 to 12 or 13.

15. An electronic terminal, characterized in that: include: A memory and a processor; wherein the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program to perform the steps in the method according to any one of claims 1 to 12 or 13.

16. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the steps of the method according to any one of claims 1 to 12 or 13.

Images