An MR / AR / VR annotation and search control method, a mobile terminal, and a readable storage medium

Abstract

An MR / AR / VR annotation and search control method, a mobile terminal, and a readable storage medium obtain an image within the field of view through a sensing module of smart glasses, identify the image through an algorithm, identify the object contours in the image, obtain user behavior operations through the sensing module, complete the selection of the annotated object according to the user operations. At the same time, since the information obtained by a single smart glass is limited, in order to increase the search range of the annotated object, the smart glass can be connected to the network to complete the search in the network domain. This method and device can not only improve people's execution efficiency and life efficiency, but also provide guarantees for life.

Description

Technical Field

[0001] The present application relates to the field of augmented reality technology, and in particular to an MR / AR / VR annotation and search control method, a mobile terminal, and a readable storage medium. Background Art

[0002] Currently, VR glasses based on the Metaverse are experiencing a surge in popularity, with an increasing number of companies and enterprises announcing their participation in the development of related products. As a mainstream product within the Metaverse, VR glasses focus on immersion and full virtuality. However, since these products create a fully virtual world platform, they can easily lead users to become addicted to the virtual world and escape from the real world, potentially negatively impacting per capita productivity and production value.

[0003] At the same time, in real life, the actual reproduction cost of ideas or designs is often high and the construction cycle is long. Due to the lack of interactivity and fun in the real world, more and more people choose to surf the Internet. The high action and search costs in the real world lead more and more people to shop and order takeout online. The above phenomena are all caused by the fact that the real world and the virtual world have not yet been interconnected.

[0004] Therefore, the system and method of integrating the virtual world and the real world on the same platform can not only prevent users from being immersed in the virtual world, but also reduce the implementation cost of the real world and improve the fun, interactivity and pertinence of the real world. This technology, system and application can truly enable people to enjoy the convenience of virtual life while experiencing real life. Summary of the Invention

[0005] A method for controlling an MR / AR / VR annotation scene includes the following steps:

[0006] S1. Switch or select the augmented reality interface of the marked scene through the interface on the smart glasses;

[0007] S2. The smart glasses' perception module acquires an image within the field of view, identifies the image using an algorithm, identifies the outlines of objects in the image, and uses the perception module to detect user actions and selects labeled objects based on the user actions.

[0008] S3. After labeling is complete, the smart glasses system senses the labeled object through the perception module, automatically follows the path of the labeled object, and stores the final position coordinates of the labeled object before it disappears from the perception module's perception area. Simultaneously, the perception module detects in real time whether the labeled object reappears in the perception area. If so, the system automatically continues to follow the path of the labeled object and updates the final position coordinates of the labeled object before it disappears from the perception module's perception area. This cycle repeats.

[0009] S4. Obtain the instruction to search for a certain marked object through the input device, retrieve the final position coordinates of the marked object stored in the background, form a guiding path based on the final position coordinates and the real-time position coordinates of the smart glasses at this time, generate a guiding sign through the guiding path, and perform visual imaging on the guiding sign through the imaging device.

[0010] The sensing of the marked object by the sensing module includes the following steps:

[0011] S5. When marking the object, extract the feature points of the marked object through the sensing module and complete the storage of the corresponding data;

[0012] S6. When the sensing module senses a corresponding object, extract its feature points and compare them with the feature points of the marked object stored. If they match, the system determines that the marked object has been sensed.

[0013] The formation of the guiding path and the generation of the guiding sign through the guiding path include the following steps:

[0014] S7. Store the position coordinates obtained by the positioning module before the marked object disappears within the sensing area of the sensing module and the image data obtained by the sensing module;

[0015] S8. When the input device obtains the instruction to search for the marked object, form a guiding path through the final position coordinates obtained by the positioning module of the marked object and the real-time position coordinates of the smart glasses at this time, and guide the user to the final position coordinates obtained by the positioning module through the imaged guiding sign;

[0016] S9. Obtain the image within the user's field of view at this time through the sensing module, compare the real-time image with the image data of the marked object obtained and stored by the sensing module. If they match, the system gives a visual prompt. If the marked object is recognized in the image at this time, a visual prompt for the marked object is given. If they do not match, generate a guiding identifier for the direction through the imaging device to guide the user to approach the position of the marked object until they match.

[0017] An MR / AR / VR search scenario control method includes the following steps:

[0018] S10. When the input device of a single smart glasses obtains the instruction to search for a certain marked object, upload the data information of the marked object to the server through the communication module, and the server distributes the data information of the marked object to other single smart glasses within a certain area range;

[0019] S11. Other single smart glasses within the range of this area will compare the data information of the labeled object received with the image information within the field of view obtained by their own perception module. If a matching object is found, the smart glasses that find the matching object will upload the coordinate data of their own positioning module and the image data of their own perception module to the server through their own communication module;

[0020] S12. After receiving the data, the server will send the data back to the smart glasses that issued the instruction to find the labeled object. The smart glasses with the labeled object instruction will form a guiding path based on the returned position coordinates and their own real-time position coordinates at this time, generate a guiding sign through the guiding path, and perform visual imaging on the guiding sign through the imaging device.

[0021] A mobile terminal includes a processor, a memory, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, the steps of the control method are implemented.

[0022] A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the control method are implemented. Description of the Drawings

[0023] Figure 1 It is the hardware logic block diagram of the smart glasses system of this application;

[0024] Figure 2 It is the internal logic diagram of Embodiment 1 of this application;

[0025] Figure 3 It is the external presentation diagram of Embodiment 1 of this application;

[0026] Figure 4 It is the interface presentation diagram of Embodiment 2 of this application;

[0027] Figure 5 It is the external framework diagram of Embodiment 2 of this application;

[0028] Figure 6 It is the internal logic diagram of Embodiment 2 of this application;

[0029] Figure 7 It is the internal logic diagram of Embodiment 3 of this application;

[0030] Figure 8 It is the internal logic diagram of Embodiment 4 of this application;

[0031] Figure 9 It is the internal logic diagram of Embodiment 6 of this application;

[0032] Figure 10 It is the external framework diagram of Embodiment 6 of this application. Detailed implementation manners

[0033] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, but should not be construed as limiting the present application.

[0034] As Figure 1 shown above, a hardware of an augmented reality-based smart glasses system includes a number of VR / AR / MR smart glasses access devices, a server, and a number of multi-layer Internet regions based on the server. Among them, the VR / AR / MR smart glasses access devices are connected to the server through wireless communication. The server runs the number of multi-layer Internet regions. The Internet regions can be regarded as slices of a certain virtual world. The number of slices can be stacked and combined in a specific number to form new slices. Users can select a certain slice through the VR / AR / MR smart glasses access devices for projection. When the VR / AR / MR smart glasses access devices obtain data from the Internet regions, that is, perform information retrieval and information interaction functions: the server will screen and classify the real-time data uploaded by the VR / AR / MR smart glasses access devices according to the target information data corresponding to the selected slice, retain specific data, and the VR / AR / MR smart glasses access devices complete the association and interaction of specific information data with this slice on the server. This slice on the server transmits the multi-dimensional data corresponding to the information data to the VR / AR / MR smart glasses access devices for projection according to the confirmation of the specific information data. The VR / AR / MR smart glasses access devices include a number of different types of sensors and a number of data input devices. When the VR / AR / MR smart glasses access devices upload data to the Internet regions, that is, perform information publishing and information marking functions: the sensors capture the current environmental information and human action information, and the data is input through the data input devices. The obtained data is uploaded to the server. This slice on the server classifies and screens the data, stores the screened data, and can be retrieved, interacted with, and projected as specific information data under this slice of the Internet region to the VR / AR / MR smart glasses access devices that establish specific information data association and interaction with the server.

[0035] As Figure 1As shown below, the hardware of an augmented reality-based smart glasses system includes several smart glasses access devices, a smartphone, a server, and several multi-layer Internet regions based on the server. Among them, the smart glasses access device establishes a data channel with the smartphone through Bluetooth connection. The smartphone is connected to the server through wireless communication. The server runs the several multi-layer Internet regions. The Internet regions can be regarded as slices of a certain virtual world. The several slices can be superimposed and combined in a specific number to form new slices. The user can select a certain slice through the smartphone to display the slice information data on the APP interface. The smart glasses include a front-end sensing module. When the smartphone obtains data from the Internet region, that is, performs information retrieval and information interaction functions: the video information data is obtained in real time through the sensing module, and the video information data is transmitted to the APP of the smartphone through Bluetooth connection. A data connection between the APP client and the Internet region is established through wireless communication. The server will screen and classify the real-time video information data uploaded by the APP client according to the target information data corresponding to the selected slice, retain specific data, and the smartphone and the slice on the server complete the association and interaction of specific video information data. The slice on the server transmits the multi-dimensional data corresponding to the information data to the smartphone according to the confirmation of the specific information data and presents it in the APP client. When the smartphone uploads data to the Internet region, that is, performs information publishing and information marking functions: data input is performed through the smartphone, and the obtained data is uploaded to the server. The slice on the server classifies and screens the data, stores the screened data, and can be retrieved, interacted with, and presented as specific information data under the slice of the Internet region on the APP interface of the smartphone.

[0036] Embodiment 1, such as Figure 2As shown in the figure, an augmented reality-based smart glasses system software includes multiple scenarios, which are divided into different functional scenarios. The functional scenarios include, but are not limited to, a message scene, a creation scene, an interaction scene, a biological scene, a shopping scene, a retrieval scene, a push scene, a co-branded scene, a design scene, a marking scene, a friend-making scene, a navigation scene, and a live broadcast scene. Each functional scenario is visually presented on the smart glasses by means of switching or overlaying. Each of the functional scenarios can be further divided into different thematic scenarios, which include, but are not limited to, scenarios corresponding to different characters, games, movies, and animations. The thematic scenarios are the above-mentioned slices, and several slices can be overlaid and combined in a specific number to form a new slice, that is, the thematic scenarios can be visually presented on the smart glasses alone or in combination. If there are N thematic scenarios, the upper limit of the number of thematic scenarios that the user can switch is: (2 N - 1).

[0037] As Figure 3 shown, the hardware of the augmented reality-based smart glasses system in Embodiment 1 includes a smart glasses body 101 and an interactive control device 102. The smart glasses body 101 is connected to the interactive control device 102 to establish a data connection. The user can complete the switching of virtual scenarios on the smart glasses body 101 through the interactive control device 102. The virtual imaging on the smart glasses body 101 includes scene labels: Scene One 201, Scene Two 202, Scene Three 203, Scene Four 204..., to achieve visual switching between scenes. Further, the scene labels can have sub-labels to expand the sub-scenes under a certain scene, and the switching and drilling down of scenes are realized through the interactive control device 102.

[0038] Further, in the data background, each scenario corresponds to a usage rate, at least one feature standard, and a set of AR parameters. The sorting order of Scene One, Scene Two, Scene Three, Scene Four... is from high to low according to the corresponding usage rate of each scenario. Among them, the calculation formula of the usage rate can be: In the formula, t(i) is the usage duration of the i-th scenario, is the total usage duration of the user using the smart glasses.

[0039] Further, in order to gradually adapt to the user's habits for data update services, the specific method is as follows:

[0040] S52. Establish a user database to store the user's characteristic data, such as the usage duration of each scenario, or the usage rate of each scenario, or the usage frequency of each scenario;

[0041] S53. Update the user's feature data at regular intervals and sort the scenarios from high to low according to the feature data.

[0042] Furthermore, to provide more personalized customization services that better match user needs, the specific method is as follows:

[0043] S54. Classify the feature data of all users in the background;

[0044] S55. Build user portraits based on the classified user feature data. Each type of user portrait corresponds to an interval of a certain user feature data and also corresponds to a certain scenario sorting;

[0045] S56. Classify the new user on the user portrait according to the new user's operations and execute the scenario sorting corresponding to this type of user portrait.

[0046] Furthermore, autonomous switching and selection of scenarios can be achieved. The specific method is as follows:

[0047] S57. Analyze and detect the surrounding environment data to obtain at least one main feature of the surrounding environment;

[0048] S58. Compare the feature criteria of each scenario with the main feature. If the match is completed, the scenario will be automatically switched to the scenario corresponding to the feature criteria that match the main feature.

[0049] In the autonomous switching and selection of the scenarios described above, the feature criteria can be set to one-third or more of the proportion of the human figure in the full image, or other proportions, such as one-fourth, one-fifth, etc., which are set by the manufacturer themselves.

[0050] For example, the scene modes commonly used by college student users mostly focus on audio-visual modes. Among the usage rates of all scene modes, the usage rate of the audio-visual mode can be 45%, the usage rate of chatting and socializing can be 35%, and the usage rate of other modes can be 20%. In addition, for the device that applies the automatic selection method of the scene panel according to the first embodiment of the present invention, when leaving the factory, according to the habits of ordinary people (or the main consumer group of this AR / VR / MR device) using various scene modes, the usage rates, feature standards, and model parameters of various scene modes can be set and stored according to the number of times and probabilities. When using the AR / VR / MR device, the main features of the surrounding environment are captured and analyzed by the imaging device, and the main features of the 3D image data are compared with the stored feature standards of various scenes in order of the usage rate from high to low to obtain the scene mode that matches the main features. In other words, since the users of the AR / VR / MR device have the habit of live broadcast scenes - single-person indoor live broadcast, by determining the comparison order through the usage rate of the scene mode, the number of comparisons can be reduced, and thus the speed of automatically selecting the scene mode can be accelerated. For example: The commonly used scene modes of ordinary people are mainly audio-visual modes, followed by chatting and socializing modes, and then office modes and other modes. Therefore, when obtaining the main features of the environment, currently compare with the feature standards of the audio-visual mode. If the main features of the environment do not match the feature standards of the audio-visual mode, then compare with the feature standards of the chatting and socializing mode. If the main features of the environment do not match the feature standards of the chatting and socializing mode, then sequentially compare with the office mode and other modes. When obtaining the scene mode that is the same as the main features of the environment, replace the current VR parameters with the VR parameters of the scene mode. Therefore, by using the usage rates of ordinary people using various scenes for comparison, the search and comparison time can be reduced, allowing users to immediately use the VR parameters that best match the scene to improve the user experience.

[0051] Embodiment 2, as Figure 4 shown, in the message function of an augmented reality-based smart glasses system, the visual scene superposition state can superimpose the visual imaging of voice data and text data when passing through the lens of the smart glasses in the real scene. The hardware of the smart glasses system for realizing the message function is as Figure 5As shown in the figure, it includes the smart glasses body 301, the imaging device 401, the sensing module 402, the voice input device 403, the text input device 404, the positioning module 405, the communication module 406, and the server 407. Among them, the imaging device 401, the sensing module 402, the voice input device 403, the text input device 404, the positioning module 405, and the communication module 406 are respectively connected to the smart glasses body 301 to establish a data connection, and the communication module 406 and the server 407 establish a data connection by means of remote communication. Among them, the sensing module 402 can be a camera or a lidar.

[0052] As Figure 6 shown, the method for controlling the augmented reality-based smart glasses system with a message function in Embodiment 2 is as follows:

[0053] S1. Switch or select through the interface on the smart glasses to enter the augmented reality interface of the message scenario;

[0054] S2. Obtain real-time GPS information and image information through the positioning module and the sensing module on the smart glasses;

[0055] S3. Upload the GPS information and image information to the server through the communication module, and match them with the GPS information and image information attached to the voice data and text data in the historical data stored in the server;

[0056] S4. After successful matching, the server will transmit the corresponding matched voice data and text data in the historical data back, receive the data through the communication module of the smart glasses, and present the data through the imaging device of the smart glasses.

[0057] Furthermore, in order to reduce the number of matches and improve the matching speed during the matching process of historical data and real-time data, preprocess the historical data, and the method is as follows:

[0058] S5. Divide the park according to one or more of the GPS information and image information attached to the voice data and text data in the historical data, and determine the range of one or more of the GPS information and image information corresponding to the park;

[0059] S6. Classify the voice data and text data in the historical data according to the defined range to complete the data division with the park as the main body.

[0060] Among them, the division of the park described in S5 can specifically be:

[0061] S51. Make a preliminary division of the park according to the GPS information, and Park 1, Park 2, Park 3, and Park 4 respectively correspond to a GPS range;

[0062] S52. Redivide the park according to the image information obtained by the perception module, identify and extract the image feature quantities or markers, and complete the annotation for each park feature quantity or marker, Park 1 - Marking Point 1, Park 1 - Marking Point 2, Park 1 - Marking Point 3, Park 2 - Marking Point 1, Park 2 - Marking Point 2, Park 2 - Marking Point 3, Park 3 - Marking Point 1, Park 3 - Marking Point 2, Park 3 - Marking Point 3.

[0063] The method for matching the pre - processed data is as follows:

[0064] S7. Match the voice data and text data in the server historical data with the GPS information and image information of each block of the park according to the GPS information and image information attached to them;

[0065] S8. If the matching is completed, migrate the voice data or text data to the matching park block;

[0066] S9. Match the real - time GPS information and real - time image information on the smart glasses with the GPS information and image information of each block of the park;

[0067] S10. If the matching is completed, virtual - image the voice data and text data in the park block onto the smart glasses.

[0068] Furthermore, in order to complete the update of historical data, capture real - time data, and the method is as follows:

[0069] S11. Obtain voice data and text data through the voice input device and text input device of the smart glasses, including the GPS information and image information attached to the voice data and text data, and obtain real - time information through the positioning module and perception module;

[0070] S12. Upload the voice data, text data, and their attached GPS information and image information to the historical database of the server.

[0071] Embodiment Three, as Figure 7 shown, the visual scene superposition state of an augmented - reality - based smart glasses system under the retrieval function can superimpose the visual imaging of relevant retrieval data when passing through the lens of the smart glasses in the real scene. The hardware of the smart glasses system for realizing the retrieval function is the same as that in Embodiment Two. Furthermore, in order to enable Embodiment Three to have the function of information screening as in Embodiment Six, a rear - facing camera can be set on the smart glasses.

[0072] The method for controlling the augmented - reality - based smart glasses system with a retrieval function in Embodiment Three is as follows:

[0073] S42. Enter the augmented reality interface of the retrieval scenario by switching or selecting on the interface of the smart glasses;

[0074] S43. Obtain the image within the field of view through the sensing module of the smart glasses, identify the image through an algorithm, identify the feature points in the image and complete the locking of the feature object, and upload the feature object data to the server through the communication module;

[0075] S44. Transmit the information associated with the feature object stored in the network database and the built-in database in the server back to the smart glasses through the communication module, and perform imaging of the associated information through the imaging device of the smart glasses.

[0076] Among them, the associated information may include multi-dimensional data such as text data, voice data, and image data uploaded by other users for the feature object, and multi-dimensional data such as text data, voice data, and image data uploaded by other ports for the feature object.

[0077] Taking the real-scene imaging of the store street view as an example, the multi-dimensional data such as text data, voice data, and image data uploaded by other users for the feature object include historical evaluation information released by users after consuming in the store, or real-time social networking information released by users when staying in the store; the multi-dimensional data such as text data, voice data, and image data uploaded by other ports for the feature object include information uploaded by merchants about the services provided in the store, the main products, or information uploaded by merchants about the activities held in the store, the promotions claimed, or three-dimensional image information of the brand mascot and brand spokesperson uploaded by the merchants.

[0078] Taking the real-scene imaging of a book or movie poster as an example, the multi-dimensional data such as text data, voice data, and image data uploaded by other users for the feature object include book reviews or movie reviews released by users after reading or watching, or real-time social networking information released by users after reading or watching; the multi-dimensional data such as text data, voice data, and image data uploaded by other ports for the feature object include reading activities or movie-watching activities uploaded by organizers, peripheral products of books or movies uploaded by merchants, or creative experiences uploaded by authors.

[0079] Furthermore, when performing imaging of the associated information through the imaging device of the smart glasses in S44, the imaging position is the same as or offset in a certain specific direction from the position where the real scene of the feature object is imaged through the lens.

[0080] Furthermore, in order to achieve precise positioning of the feature object, in S43, the user's location can be obtained through the positioning module, and the locking of the feature object can be completed in combination with the image information.

[0081] Furthermore, in order to prevent the visual display of information explosion, information shielding is achieved through folding or filtering.

[0082] The folding method may specifically be as follows: the associated information is folded and unfolded by user operation, and when folded, a prompt mark indicating the existence of the associated information may be made on the lens, and when unfolded, the associated information is imaged.

[0083] The screening method may specifically be screening the associated information during imaging:

[0084] S441. The eye information is captured by the rear camera to obtain the optical focus of the eye at the current moment, and the focus is projected onto the image obtained by the perception module to determine whether the focus is within the range of the identified feature object;

[0085] S442. If the focus is within the range of a certain identified feature object and the focus stay time exceeds the set threshold, the associated information of the feature object is displayed through the imaging device, otherwise the associated information of the feature object is not displayed.

[0086] Furthermore, in order to create a database for information retrieval, a retrieval model library is created based on the data actively acquired by the system and the data uploaded by the user. The retrieval model library includes the following types: one-dimensional database, two-dimensional database, and three-dimensional database.

[0087] The data in a 3D database can be based on all existing, historical, virtual, and materialistic data in human society and nature, or it can be user-created, imaginary concepts or models. A 2D database can be derived from data mapping or slicing a 3D database, or it can be generated and uploaded by the user. A 1D database can be derived from data mapping or slicing a 2D database, or it can be generated and uploaded by the user.

[0088] The data mapping or slicing may specifically be, taking three-dimensional data as an example:

[0089] Solution 1: Take any cross section of the 3D data and extract the corresponding data information;

[0090] Solution 2: Convert three-dimensional data into two-dimensional data information through a certain function.

[0091] Among them, in the retrieval model library, the uploaded feature object data is retrieved, and the retrieval results are spatially arranged and displayed according to the matching degree, such as structural matching degree, appearance matching degree, principle matching degree, information matching degree, etc.

[0092] The display of the retrieval results can be a combined display of the retrieval results in a one-dimensional database, a two-dimensional database, and a three-dimensional database; it can also be a display by freely selecting one or more of the search libraries, namely the one-dimensional database, the two-dimensional database, and the three-dimensional database, by the user.

[0093] Furthermore, the retrieval function can not only retrieve in combination with a network database, but also retrieve in combination with a local database. The local database can be a digital twin world that completes the mapping of the local real world. Taking the digital local database of a library as an example: when a user enters the library area, the user can retrieve in the library's local database. After confirming the retrieval target, the system will generate a three-dimensional indication sign to guide the user to find the location of the target. After identifying the target, the user can obtain corresponding information through the annotations made by other users on the target.

[0094] Embodiment Four, as Figure 8 shown, in the visualization scene superposition state of an augmented reality-based smart glasses system under the push function, relevant push data visualization imaging can be superimposed when passing through the lens of the smart glasses in the real scene. The hardware of the smart glasses system for realizing the push function is the same as that of Embodiment Two except that it does not require a positioning module.

[0095] The method for controlling an augmented reality-based smart glasses system with a push function in Embodiment Four is as follows:

[0096] S52. Switch or select through the interface on the smart glasses to enter the augmented reality interface of the push scene;

[0097] S53. Obtain the images within the field of view through the perception module of the smart glasses, complete the recognition of the video content for content positioning, and upload the specific positioning content information data to the server according to the system settings. The server will send the built-in advertisements, information, and content peripherals back to the smart glasses;

[0098] S54. The smart glasses visually present the pushed-back advertisements, information, and content peripherals through the imaging module.

[0099] Furthermore, in order to increase the interest and interactivity of the video content, a comment section can be added to complete the discussion among users of the smart glasses regarding the video content.

[0100] Embodiment Five, in the visualization scene superposition state of an augmented reality-based smart glasses system under the design function, relevant design element data visualization imaging can be superimposed when passing through the lens of the smart glasses in the real scene. The hardware of the smart glasses system for realizing the design function is the same as that of Embodiment Four except that it does not require a positioning module.

[0101] The method for controlling an augmented reality-based smart glasses system with a design function in Embodiment Five is as follows:

[0102] S55. Enter the augmented reality interface of the design scenario by switching or selecting through the interface on the smart glasses;

[0103] S56. Virtually model the design elements on the basic framework through the port. When the modeling is completed, save the structural data of the design elements and the position relationship data of the design elements relative to the basic framework and upload them to the cloud. Then upload the design element data to the smart glasses through the cloud server;

[0104] S57. Obtain the actual image within the field of view through the sensing module, identify the basic framework of the actual image through the system, allow the user to select the design elements through the motion capture device, and complete the visual presentation of the design elements through the imaging device.

[0105] The basic framework described in S56 and S57 can be the human skeleton or contour, or the border and contour of the clothes.

[0106] Among them, there are the following two solutions for the visual presentation of the design elements completed through the imaging device described in S57:

[0107] Solution 1: Confirm and lock the position of the design element completed by imaging on the identified basic framework of the actual image through the position relationship data of the design element relative to the basic framework;

[0108] Solution 2: Manually move the design element to the specified position through the motion capture device.

[0109] Embodiment 6, as Figure 9 shown, in the visualization scene superposition state of a smart glasses system based on augmented reality under the annotation function, relevant annotation data can be visually imaged and superimposed when passing through the lens of the smart glasses in the real scene. The hardware of the smart glasses system for realizing the annotation function is as Figure 10 shown, including the smart glasses body five 1001, the input device five 1002, the imaging device five 1004, the sensing module five 1003, the positioning module five 1005, the communication module five 1006, and the server five 1007. Among them, the input device five 1002, the imaging device five 1004, the sensing module five 1003, the positioning module five 1005, and the communication module five 1006 are respectively connected to the smart glasses body five 1001 to establish a data connection, and the communication module five 1006 and the server five 1007 establish a data connection by means of remote communication. Among them, the sensing module five 1003 can be a camera or a lidar; the input device five 1002 can be voice input or text input.

[0110] The method for controlling the smart glasses system based on augmented reality with the annotation function in Embodiment 6 is as follows:

[0111] S45. Enter the augmented reality interface of the annotation scene by switching or selecting on the interface of the smart glasses;

[0112] S46. Obtain the image within the field of view through the sensing module of the smart glasses, identify the image through an algorithm, identify the object contour in the image, obtain the user's behavioral operations through the sensing module, and complete the selection of the annotation object according to the user's operations;

[0113] S47. After completing the annotation, the smart glasses system automatically follows the path of the annotation object through the sensing module and stores the final position coordinates of the annotation object before it disappears within the sensing area of the sensing module. At the same time, the sensing module continuously senses whether the annotation object reappears within the sensing area in real time. If it appears, it continues to automatically follow the path of the annotation object and updates the final position coordinates of the annotation object before it disappears within the sensing area of the sensing module, and so on;

[0114] S48. Obtain the instruction to find a certain annotation object through the input device, retrieve the final position coordinates of the annotation object stored in the background, form a guiding path based on the final position coordinates and the real-time position coordinates of the smart glasses at this time, generate a guiding sign through the guiding path, and perform visual imaging on the guiding sign through the imaging device.

[0115] Among them, the specific method for the sensing module to sense the annotation object can be:

[0116] S471. Extract the feature points of the annotation object through the sensing module during the annotation of the object and complete the storage of the corresponding data;

[0117] S472. When the sensing module senses the corresponding object, extract its feature points and compare them with the feature points of the stored annotation object. If they match, the system determines that the annotation object is sensed.

[0118] Among them, the matching in S472 can specifically be that the similarity of the comparison exceeds the set threshold.

[0119] Among them, the position coordinates in S47 can be obtained through the positioning module; further, in order to make the position coordinates more accurate, the positioning module and the sensing module are used together to complete the acquisition of the position coordinates and the generation of the guiding path. Specifically, it can be:

[0120] S481. Store the position coordinates obtained by the positioning module and the image data obtained by the sensing module before the annotation object disappears within the sensing area of the sensing module;

[0121] S482. When the input device obtains an instruction to search for the labeled object, a guiding path is formed by the final position coordinates obtained through the positioning module of the labeled object and the real-time position coordinates of the smart glasses at this time. The user is guided to the final position coordinates obtained by the positioning module through the guiding signs in the imaging.

[0122] S483. The image within the user's field of view at this time is obtained through the sensing module, and the real-time image is compared with the image data of the labeled object obtained and stored by the sensing module. If they match, the system gives a visual prompt. If the labeled object is recognized in the image at this time, a visual prompt for the labeled object is given. If they do not match, a guiding sign indicating the direction is generated by the imaging device to guide the user to approach the position of the labeled object until they match.

[0123] Among them, the system's visual prompt described in S483 can specifically be that the imaging device presents a flashing view; among them, the visual prompt for the labeled object described in S483 can specifically be that after the labeled object is recognized, the imaging device presents the contour line of the labeled object; among them, the matching described in S483 can specifically be that the similarity of the comparison exceeds the set threshold.

[0124] Among them, the method of generating a guiding coordinate in the direction by the imaging device to guide the user to approach the position of the labeled object until they match when they do not match described in S483 can specifically be:

[0125] S484. A guiding sign in any direction is generated by the imaging device, the real-time image obtained by the sensing module is compared with the image data of the labeled object obtained and stored by the sensing module, and the similarity index of the image data is calculated. If the growth rate of the similarity index exceeds the set threshold within a certain time, it is confirmed that this direction is the correct direction, otherwise it is the wrong direction;

[0126] S485. If the initial direction is the correct direction, taking the position where the smart glasses are facing when the slope of the similarity index is the largest as the reference line, a guiding sign along the initial direction is generated, and the change of the similarity index when moving along the guiding sign is calculated. If the similarity index increases, continue to generate a guiding sign along the initial direction until the labeled object is recognized; if the similarity index decreases, a guiding sign in the opposite direction of the initial direction is generated, and the change of the similarity index when moving along the guiding sign is calculated. If the similarity index increases, continue to generate a guiding sign in the opposite direction of the initial direction until the labeled object is recognized;

[0127] S486. If the initial direction is the wrong direction, taking the initial position as the reference line, a guiding sign in the opposite direction of the initial direction is generated, and the change of the similarity index when moving along the guiding sign is calculated. If the similarity index increases, continue to generate a guiding sign in the opposite direction of the initial direction until the labeled object is recognized.

[0128] Due to the limited information obtained by a single smart glasses, further, in order to increase the search range of the labeled object, the smart glasses are connected to the network to complete the network search. The specific method can be as follows:

[0129] S49. When a single smart glasses input device obtains an instruction to find a certain labeled object, the data information of the labeled object is uploaded to the server through the communication module, and the server distributes the data information of the labeled object to other single smart glasses within a certain area range;

[0130] S50. Other single smart glasses within the area range compare the received data information of the labeled object with the image information within the field of view obtained by their own perception module. If a matching object is found, the smart glasses that find the matching object upload the coordinate data of their own positioning module and the image data of their own perception module to the server through their own communication module;

[0131] S51. After receiving the data, the server transmits the data back to the smart glasses that issued the instruction to find the labeled object. The smart glasses with the labeled object instruction form a guiding path based on the transmitted position coordinates and its own real-time position coordinates at this time, generate a guiding mark through the guiding path, and perform visual imaging on the guiding mark through the imaging device.

[0132] Among them, the server accessed by the smart glasses system can be a centrally deployed server or an edge-deployed distributed server, and the number of servers is not limited. If the server is a distributed server, it can be arranged at various positions. The smart glasses can access the distributed server through various spatial sensing methods such as GPS sensing, network sensing, and radar sensing. The distributed server can be arranged in public spaces such as buses, stores, schools, hospitals, public institutions, and enterprises.

[0133] Embodiment 7, a distributed server smart glasses system based on augmented reality includes a number of distributed servers arranged at different positions and a number of AR / MR / VR smart glasses that can access the network. A control method for controlling the distributed server smart glasses system is as follows:

[0134] S59. The AR / MR / VR smart glasses access the distributed server arranged in the spatial area through the network or GPS or radar or image;

[0135] S60. The accessed distributed server transmits the stored two / three-dimensional image / video data, audio data, and text data to the accessed AR / MR / VR smart glasses through data communication;

[0136] The S61 AR / MR / VR smart glasses visually present the received two / three-dimensional image / video data, audio data, and text data through an imaging device.

[0137] Among them, the AR / MR / VR smart glasses access the distributed server arranged in this spatial area through the network. Specifically, the AR / MR / VR smart glasses complete the access through the wireless local area network accessing the distributed server.

[0138] Among them, the AR / MR / VR smart glasses access the distributed server arranged in this spatial area through GPS. Specifically:

[0139] S591. The distributed server uploads the GPS information of its own block to the cloud.

[0140] S592. The cloud compares the GPS information uploaded by the AR / MR / VR smart glasses in real time with the GPS information of the block uploaded by the distributed server.

[0141] S593. If the matching is completed during the comparison, the cloud connects to the AR / MR / VR smart glasses to access the corresponding matching distributed server.

[0142] Among them, the device for obtaining human operations through the above-mentioned embodiments can be an image sensor, a radar sensor, a touch sensor, a button sensor, a voice sensor, etc., which are devices that can obtain human behaviors.

[0143] Among them, in the above-mentioned embodiments, entering a certain scene is completed through manual selection, or it can also automatically enter the scene by identifying whether there is a scene set in this area. Further, if multiple scenes are identified, the user's habits are obtained through an algorithm to complete the scene selection and automatically enter the scene.

[0144] Among them, the smart glasses protected by the present invention can be single-function smart glasses with only a single function / single scene, or multi-function smart glasses with multiple functions / multiple scenes. The multi-function smart glasses can be a combination of two or more of single functions / single scenes, including combinations of hardware and functions.

[0145] In the description of this specification, the descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of this application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0146] Although the embodiments of this application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on this application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for controlling an MR / AR / VR annotation scene, characterized in that It includes the following steps: S1. Enter the augmented reality interface of the annotation scene by switching or selecting on the interface of the smart glasses; S2. Obtain the images within the field of view through the sensing module of the smart glasses, identify the images through algorithms, identify the object contours in the images, obtain the user behavior operations through the sensing module, and complete the selection of the annotated object according to the user operations; S3. After the annotation is completed, the smart glasses system senses the annotated object through the sensing module, automatically follows the path of the annotated object, and stores the final position coordinates of the annotated object before it disappears within the sensing area of the sensing module. At the same time, the sensing module senses in real time whether the annotated object reappears within the sensing area. If it appears, it continues to automatically follow the path of the annotated object and updates the final position coordinates of the annotated object before it disappears within the sensing area of the sensing module, and so on; S4. Obtain the instruction to find a certain annotated object through the input device, retrieve the final position coordinates of the annotated object stored in the background, form a guiding path based on the final position coordinates and the real-time position coordinates of the smart glasses at this time, generate a guiding sign through the guiding path, and perform visual imaging on the guiding sign through the imaging device; The sensing of the annotated object by the sensing module includes the following steps: S5. Extract the feature points of the annotated object through the sensing module during annotation and complete the storage of the corresponding data; S6. When the sensing module senses the corresponding object, extract its feature points and compare them with the feature points of the stored annotated object. If they match, the system determines that it has sensed the annotated object.

2. The control method according to claim 1, characterized in that The formation of the guiding path and the generation of the guiding sign through the guiding path include the following steps: S7. Store the position coordinates obtained by the positioning module and the image data obtained by the sensing module before the annotated object disappears within the sensing area of the sensing module; S8. When the input device obtains the instruction to find the annotated object, form a guiding path based on the final position coordinates obtained by the positioning module of the annotated object and the real-time position coordinates of the smart glasses at this time, and guide the user to the final position coordinates obtained by the positioning module through the imaging guiding sign; S9. Obtain the image within the user's field of view at this time through the sensing module, compare the real-time image with the stored image data of the annotated object obtained by the sensing module. If they match, the system gives a visual prompt. If the annotated object is recognized in the image at this time, a visual prompt of the annotated object is given. If they do not match, a guiding sign indicating the direction is generated through the imaging device to guide the user to approach the position of the annotated object until they match.

3. A mobile terminal, characterized in that, It includes a processor, a memory, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the control method according to any one of claims 1 to 2.

4. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, it implements the steps of the control method according to any one of claims 1 to 2.

Images