An ar information visualization marking method and device based on lbs and image recognition

Abstract

This specification provides an AR information visualization annotation method and apparatus based on LBS and image recognition, comprising: responding to a user sending AR message information via a mobile phone, obtaining the latitude and longitude of the current AR message sending location through location-based services (LBS), and determining the latitude and longitude coordinates; establishing a three-dimensional coordinate system with the camera position as the origin, adjusting the camera to the direction of the AR message sending location, and determining the horizontal and vertical rotation angles between the current camera direction and the starting direction; obtaining the horizontal and vertical rotation angles, determining the endpoint coordinates of the message location using trigonometric functions, and generating AR message information at the endpoint coordinates. By combining virtual and real scenes, the accuracy of information annotation location is improved, and the operation is simple and recording is fast.

Description

Technical Field

[0001] This document relates to the field of data visualization technology, and in particular to an AR information visualization annotation method and device based on LBS and image recognition. Background Technology

[0002] With economic development, my country's construction standards have continuously improved, forming a complete building system. Correspondingly, the workload of building operation and maintenance has increased. When carrying out building operation and maintenance, operation and maintenance personnel can use information technology to monitor and display various aspects of the operation and maintenance process. For example, when equipment is replaced or repaired during operation and maintenance, maintenance and debugging records will be generated, including: time, location, personnel, problems solved, and problems to be solved. Electronic information archives for operation and maintenance are established to record the operation and maintenance process, making operation and maintenance information visible.

[0003] In recent years, building operation and maintenance data visualization has developed rapidly. However, the current methods for recording and visualizing operation and maintenance information are very limited. Most of the information can only be recorded in computer and mobile phone software interfaces. For example, if the screws of a cabinet in a certain room on a certain floor are loose, it can be recorded by software. However, when there are a large number of cabinets and the amount of information is too large, it will be difficult for staff to quickly find the cabinet with the problem.

[0004] Currently, commonly used methods for establishing electronic information archives mainly include: Radio Frequency Identification (RFID) technology and information management systems. However, the above two methods have the following drawbacks: RFID requires the addition of RFID modules, incurring secondary equipment costs and causing interference with special equipment; the operation and maintenance records of information management systems cannot be visually displayed in buildings; the connection between data information and real-world objects is limited, and the information's directional explanation of physical space may not be sufficient; the number of operators is complex, and the method of recording operation and maintenance information is not simple and direct enough.

[0005] Therefore, there is a need for a simple, fast, and information visualization annotation method that does not require additional devices. Summary of the Invention

[0006] This specification provides one or more embodiments of an AR information visualization annotation method based on LBS and image recognition, including:

[0007] S1. In response to a user sending an AR message via mobile phone, obtain the latitude and longitude coordinates of the current AR message sending location through location-based services (LBS) and determine the latitude and longitude coordinates.

[0008] S2. Establish a three-dimensional coordinate system with the camera position as the origin, rotate the camera to the direction of sending AR message information, and determine the horizontal and vertical rotation angles between the current camera direction and the starting direction;

[0009] S3. Obtain the horizontal rotation angle and the vertical rotation angle, determine the endpoint coordinates of the message location using trigonometric functions, and generate AR message information at the endpoint coordinates.

[0010] Furthermore, the method also includes:

[0011] After determining the horizontal and vertical rotation angles of the angle between the current camera direction and the starting position direction, the system uses the latitude and longitude coordinates of the AR message sending location to determine whether a prefabricated model exists within a certain range of the AR message sending location. If it exists, the system uses the prefabricated model to correct the camera direction, thereby obtaining the corrected horizontal and vertical rotation angles of the angle between the camera direction and the starting position direction.

[0012] Furthermore, a three-dimensional coordinate system is established with the camera position as the origin. The camera is rotated to the direction of sending AR message information. The horizontal and vertical rotation angles between the current camera direction and the starting direction are determined, specifically including:

[0013] Establish a three-dimensional coordinate system with the camera as the origin (0, 0, 0), set a unit vector N(0, 0, 1) along the Z-axis of the coordinate system, and set the starting direction of the camera as the positive direction along the Z-axis;

[0014] By analyzing the parameter changes of the phone's gyroscope, the horizontal and vertical rotation angles between the current camera orientation and the initial orientation are obtained.

[0015] Furthermore, the specific method for determining the endpoint coordinates of the message location using trigonometric functions is as follows:

[0016] Obtain the distance coefficient from the camera to the location where the AR message is sent; the distance coefficient is a user-defined setting.

[0017] The coordinates of the final location of the message are calculated using trigonometric functions based on the horizontal and vertical rotation angles and the distance coefficient L.

[0018] This specification provides one or more embodiments of an AR information visualization and annotation device based on LBS and image recognition, including:

[0019] Location determination module: In response to users sending AR message information via mobile phone, it obtains the latitude and longitude of the current AR message sending location through location-based services (LBS) and determines the latitude and longitude coordinates.

[0020] Angle determination module: used to establish a three-dimensional coordinate system with the camera position as the origin, adjust the camera to the direction of the AR message sending position, and determine the horizontal and vertical rotation angles between the current camera direction and the starting direction;

[0021] Message generation module: used to obtain the horizontal and vertical rotation angles, determine the endpoint coordinates of the message position through trigonometric functions, and generate AR message information at the endpoint coordinates.

[0022] Furthermore, the angle correction module is specifically used for:

[0023] After determining the horizontal and vertical rotation angles of the angle between the current camera direction and the starting position direction, the system uses the latitude and longitude coordinates of the AR message sending location to determine whether a prefabricated model exists within a certain range of the AR message sending location. If it exists, the system uses the prefabricated model to correct the camera direction, thereby obtaining the corrected horizontal and vertical rotation angles of the angle between the camera direction and the starting position direction.

[0024] Furthermore, the angle determination module is specifically used for:

[0025] Establish a three-dimensional coordinate system with the camera as the origin (0, 0, 0), set a unit vector N(0, 0, 1) along the Z-axis of the coordinate system, and set the starting direction of the camera as the positive direction along the Z-axis;

[0026] By analyzing the parameter changes of the phone's gyroscope, the horizontal and vertical rotation angles between the current camera orientation and the initial orientation are obtained.

[0027] Furthermore, the message generation module is specifically used for:

[0028] Obtain the distance coefficient from the camera to the location where the AR message is sent; the distance coefficient is a user-defined setting.

[0029] The coordinates of the final location of the message are calculated using trigonometric functions based on the horizontal and vertical rotation angles and the distance coefficient L.

[0030] This specification provides one or more embodiments of an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the above-described AR information visualization and annotation method based on LBS and image recognition.

[0031] This specification provides one or more embodiments of a storage medium storing an information transmission implementation program, which, when executed by a processor, implements the steps of the above-described AR information visualization annotation method based on LBS and image recognition.

[0032] Compared with traditional systems for recording information, this invention can better correspond to the mapping relationship between real space and information, enabling faster real-time recording during operation and maintenance. The correction algorithm based on prefab comparison improves the accuracy of information labeling positions. It is simple to operate, requires no additional hardware equipment, and has wider applicability.

[0033] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A flowchart illustrating an AR information visualization and annotation method based on LBS and image recognition, provided for one or more embodiments of this specification;

[0036] Figure 2 A flowchart illustrating a specific AR information visualization and annotation method based on LBS and image recognition, provided for one or more embodiments of this specification;

[0037] Figure 3 A schematic diagram illustrating the rotation of a camera in an AR information visualization and annotation method based on LBS and image recognition, provided for one or more embodiments of this specification.

[0038] Figure 4 A schematic diagram of three-dimensional spatial coordinate transformation for an AR information visualization annotation method based on LBS and image recognition provided in one or more embodiments of this specification;

[0039] Figure 5 A schematic diagram illustrating the three-dimensional spatial information message correction process of an AR information visualization annotation method based on LBS and image recognition, provided for one or more embodiments of this specification;

[0040] Figure 6 A schematic diagram of an AR information visualization and annotation device based on LBS and image recognition, provided for one or more embodiments of this specification;

[0041] Figure 7This is a schematic diagram of the structure of an electronic device provided for one or more embodiments of this specification. Detailed Implementation

[0042] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0043] Method Implementation Examples

[0044] According to embodiments of the present invention, an AR information visualization and annotation method based on LBS and image recognition is provided. Figure 1 A flowchart illustrating an AR information visualization and annotation method based on LBS and image recognition, provided for one or more embodiments of this specification. Figure 2 A detailed flowchart of an AR information visualization annotation method based on LBS and image recognition, provided for one or more embodiments of this specification, is shown below. Figure 1 and Figure 2 As shown, the AR information visualization annotation method based on LBS and image recognition according to an embodiment of the present invention specifically includes:

[0045] S1. In response to a user sending an AR message via mobile phone, obtain the latitude and longitude coordinates of the current AR message sending location through location-based services (LBS) and determine the latitude and longitude coordinates.

[0046] Specifically, when a user uses the AR message function on their mobile phone, in response to the user composing and sending AR message information via their mobile phone, the system will activate the LBS service to obtain the latitude and longitude coordinates (gpsN, gpsE) of the current location when sending the message information, and determine the location.

[0047] S2. Establish a three-dimensional coordinate system with the camera position as the origin, rotate the camera to the direction of sending AR message information, and determine the horizontal and vertical rotation angles between the current camera direction and the starting direction.

[0048] Specifically, in three-dimensional space, a coordinate system is established with the camera as the origin (0, 0, 0), and a unit vector is set along the Z-axis. like Figure 3As shown, the positive Z-axis direction is set as the starting direction of the camera. Users can customize an adjustable distance coefficient L according to their personal habits, which is the distance from the camera to the location where the message is generated.

[0049] The system uses the phone's built-in gyroscope to obtain the current camera position and unit vector. The included angles include the horizontal rotation angle (rotation.x) and the vertical rotation angle (rotation.y).

[0050] When sending a message indoors or when the user's mobile phone has poor sensitivity, GPS positioning and the phone's gyroscope may be inaccurate. Therefore, this embodiment includes auxiliary corrections to the current positioning and angle, such as... Figure 5 As shown, the specific method is as follows: It determines whether a prefabricated model exists near the current latitude and longitude coordinates (gpsN, gpsE). The prefabricated model is a pre-modeled file already stored in the system database. If a prefabricated model exists near the current latitude and longitude coordinates, it further determines whether the comparison direction of this prefabricated model is in the direction of the current camera. If it exists, this prefabricated model is called and compared with the image captured by the camera for correction. This method reduces the calling and repeated comparison of prefabricated models, thus reducing the consumption of system resources.

[0051] S3. Obtain the horizontal rotation angle and the vertical rotation angle, determine the endpoint coordinates of the message location using trigonometric functions, and generate AR message information at the endpoint coordinates.

[0052] Specifically, when the camera's orientation is not along the positive z-axis, i.e., it rotates at a certain angle, the vector after rotation is obtained using trigonometric functions. The coordinates of the message's endpoint after multiplying by the distance coefficient L are as follows: Figure 4 As shown, the specific method is as follows:

[0053] Obtain the distance coefficient L from the camera to the location where the AR message is sent;

[0054] The coordinates of the message's endpoint are calculated using trigonometric functions based on the horizontal and vertical rotation angles and the distance coefficient L.

[0055] x=L*cosα*sinβ

[0056] y=L*sinα

[0057] z = L * cosα * cosβ

[0058] Where α = rotation.x; β = rotation.y; the endpoint coordinates of the AR message location are (x, y, z), and a message text is generated at the endpoint coordinates of the AR message location. The orientation of the text is the opposite of the direction the camera is facing.

[0059] The beneficial effects of this invention are as follows:

[0060] Compared with traditional systems for recording information, this invention can better correspond to the mapping relationship between real space and information, enabling faster real-time recording during operation and maintenance. The correction algorithm based on prefab comparison improves the accuracy of information labeling positions. It is simple to operate, requires no additional hardware equipment, and has wider applicability.

[0061] System Implementation Examples

[0062] According to embodiments of the present invention, an AR information visualization and annotation device based on LBS and image recognition is provided. Figure 6 A schematic diagram of an AR information visualization and annotation device based on LBS and image recognition, provided for one or more embodiments of this specification, is shown below. Figure 6 As shown, the AR information visualization and annotation device based on LBS and image recognition according to an embodiment of the present invention specifically includes:

[0063] Location determination module 60: In response to a user sending an AR message via mobile phone, it obtains the latitude and longitude coordinates of the current AR message sending location through location-based services (LBS) and determines the latitude and longitude coordinates.

[0064] Angle determination module 62: used to establish a three-dimensional coordinate system with the camera position as the origin, adjust the camera to the direction of sending AR message information, and determine the horizontal and vertical rotation angles of the angle between the current camera direction and the starting direction.

[0065] The angle determination module is specifically used for:

[0066] Establish a three-dimensional coordinate system with the camera as the origin (0, 0, 0), set a unit vector N(0, 0, 1) along the Z-axis of the coordinate system, and set the starting direction of the camera as the positive direction along the Z-axis;

[0067] By analyzing the parameter changes of the phone's gyroscope, the horizontal and vertical rotation angles between the current camera orientation and the initial orientation are obtained.

[0068] The device also includes an angle correction module, which is specifically used for:

[0069] After determining the horizontal and vertical rotation angles of the angle between the current camera direction and the starting position direction, the system uses the latitude and longitude coordinates of the AR message sending location to determine whether a prefabricated model exists within a certain range of the AR message sending location. If it exists, the system uses the prefabricated model to correct the camera direction, thereby obtaining the corrected horizontal and vertical rotation angles of the angle between the camera direction and the starting position direction.

[0070] Message generation module 64: used to obtain the horizontal rotation angle and the vertical rotation angle, determine the endpoint coordinates of the message position through trigonometric functions, and generate AR message information at the endpoint coordinates.

[0071] The message generation module 64 is specifically used for:

[0072] Obtain the distance coefficient from the camera to the location where the AR message is sent; the distance coefficient is a user-defined setting.

[0073] The coordinates of the final location of the message are calculated using trigonometric functions based on the horizontal and vertical rotation angles and the distance coefficient L.

[0074] The embodiments of the present invention are system embodiments corresponding to the above method embodiments. The specific operation of each module can be understood by referring to the description of the method embodiments, and will not be repeated here.

[0075] Device Example 1

[0076] This invention provides an electronic device, such as... Figure 7 As shown, it includes: a memory 70, a processor 72, and a computer program stored in the memory 70 and executable on the processor 72. When the computer program is executed by the processor 72, it performs the following method steps:

[0077] S1. In response to a user sending an AR message via mobile phone, obtain the latitude and longitude coordinates of the current AR message sending location through location-based services (LBS) and determine the latitude and longitude coordinates.

[0078] S2. Establish a three-dimensional coordinate system with the camera position as the origin, rotate the camera to the direction of sending AR message information, and determine the horizontal and vertical rotation angles between the current camera direction and the starting direction;

[0079] S3. Obtain the horizontal rotation angle and the vertical rotation angle, determine the endpoint coordinates of the message location using trigonometric functions, and generate AR message information at the endpoint coordinates.

[0080] Device Example 2

[0081] This invention provides a storage medium storing an information transmission implementation program. When executed by a processor 72, the program performs the following method steps:

[0082] S1. In response to a user sending an AR message via mobile phone, obtain the latitude and longitude coordinates of the current AR message sending location through location-based services (LBS) and determine the latitude and longitude coordinates.

[0083] S2. Establish a three-dimensional coordinate system with the camera position as the origin, rotate the camera to the direction of sending AR message information, and determine the horizontal and vertical rotation angles between the current camera direction and the starting direction;

[0084] S3. Obtain the horizontal rotation angle and the vertical rotation angle, determine the endpoint coordinates of the message location using trigonometric functions, and generate AR message information at the endpoint coordinates.

[0085] The readable storage medium described in this embodiment includes, but is not limited to, ROM, RAM, disk, or optical disk.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for AR information visualization and annotation based on LBS and image recognition, characterized in that, include: S1. In response to a user sending an AR message via mobile phone, obtain the latitude and longitude coordinates of the current AR message sending location through location-based services (LBS) and determine the latitude and longitude coordinates. S2. Establish a three-dimensional coordinate system with the camera position as the origin, rotate the camera to the direction of sending AR message information, and determine the horizontal and vertical rotation angles between the current camera direction and the starting direction; After determining the horizontal and vertical rotation angles of the angle between the current camera direction and the starting position direction, the system uses the latitude and longitude coordinates of the AR message sending location to determine whether a prefabricated model exists within a certain range of the AR message sending location. If it exists, the camera direction is corrected using the prefabricated model to obtain the corrected horizontal and vertical rotation angles of the angle between the camera direction and the starting position direction. The prefabricated model is a pre-modeled file. S3. Obtain the horizontal and vertical rotation angles, determine the endpoint coordinates of the message location using trigonometric functions, and generate AR message information at the endpoint coordinates.

2. The method according to claim 1, characterized in that, The method described in step S2 specifically includes: Establish a three-dimensional coordinate system with the camera as the origin (0, 0, 0), set a unit vector N (0, 0, 1) along the Z-axis of the coordinate system, and set the starting direction of the camera as the positive direction of the Z-axis; By analyzing the changes in the parameters of the phone's gyroscope, the horizontal and vertical rotation angles between the current camera orientation and the initial orientation can be obtained.

3. The method according to claim 1, characterized in that, The specific method for determining the endpoint coordinates of the message location using trigonometric functions is as follows: Obtain the distance coefficient from the camera to the location where the AR message is sent; the distance coefficient is a user-defined setting. The coordinates of the message's endpoint are calculated using trigonometric functions based on the horizontal and vertical rotation angles and the distance coefficient L.

4. An AR information visualization and annotation device based on LBS and image recognition, characterized in that, include: Location determination module: In response to users sending AR message information via mobile phone, it obtains the latitude and longitude coordinates of the current AR message sending location through location-based services (LBS) and determines the latitude and longitude coordinates. Angle determination module: used to establish a three-dimensional coordinate system with the camera position as the origin, adjust the camera to the direction of the AR message sending position, and determine the horizontal and vertical rotation angles between the current camera direction and the starting direction; After determining the horizontal and vertical rotation angles of the angle between the current camera direction and the starting position direction, the system uses the latitude and longitude coordinates of the AR message sending location to determine whether a prefabricated model exists within a certain range of the AR message sending location. If it exists, the camera direction is corrected using the prefabricated model to obtain the corrected horizontal and vertical rotation angles of the angle between the camera direction and the starting position direction. The prefabricated model is a pre-modeled file. Message generation module: used to obtain the horizontal and vertical rotation angles, determine the endpoint coordinates of the message position through trigonometric functions, and generate AR message information at the endpoint coordinates.

5. The apparatus according to claim 4, characterized in that, The angle determination module is specifically used for: Establish a three-dimensional coordinate system with the camera as the origin (0, 0, 0), set a unit vector N (0, 0, 1) along the Z-axis of the coordinate system, and set the starting direction of the camera as the positive direction of the Z-axis; By analyzing the changes in the parameters of the phone's gyroscope, the horizontal and vertical rotation angles between the current camera orientation and the initial orientation can be obtained.

6. The apparatus according to claim 4, characterized in that, The message generation module is specifically used for: Obtain the distance coefficient from the camera to the location where the AR message is sent; the distance coefficient is a user-defined setting. The coordinates of the message's endpoint are calculated using trigonometric functions based on the horizontal and vertical rotation angles and the distance coefficient L.

7. An electronic device, characterized in that, include: processor; as well as, A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the LBS-based and image recognition-based AR information visualization annotation method as described in any one of claims 1 to 3.

8. A storage medium, characterized in that, Used to store computer-executable instructions, which, when executed, implement the steps of the AR information visualization and annotation method based on LBS and image recognition as described in any one of claims 1 to 3.

Images