Reading method and device based on AR equipment, and AR equipment

Abstract

The invention relates to the technical field of AR, and provides a reading method and device based on AR equipment, and the AR equipment. The method comprises the following steps: when the AR equipment is in a reading mode, acquiring reading data of the AR equipment at the current moment, and extracting target information from the reading data; and starting a search function of the AR equipment to search associated information associated with the target information, and displaying the associated information on the AR equipment. In this way, AR equipment data display is achieved, more detailed and specific description can be provided for displayed content in real time, and the problem that the experience of a user for reading by using AR equipment is reduced due to the fact that existing AR equipment cannot conduct secondary analysis on displayed information in a reading scene is solved.

Description

Technical Field

[0001] This disclosure relates to the field of AR technology, and in particular to a reading method and apparatus based on an AR device, and an AR device. Background Technology

[0002] Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It can be applied to various fields to achieve the combination and interaction of virtual and real world scenes, especially in wearable devices.

[0003] Current AR devices are mostly used to display data. AR devices can capture images of the scene in front of the user, then perform image recognition on the objects in the scene image to obtain information about the objects and display it to the user. However, the collected information often contains multiple elements, and the current display only extracts information from the environment and projects it onto the AR device, without providing any additional description or supplement to the displayed information.

[0004] When users need to view certain information further, they need to use devices other than AR devices, which reduces the user experience of displaying data using AR devices, especially in scenarios where they are reading documents. Summary of the Invention

[0005] In view of this, the purpose of this disclosure is to provide a reading method and apparatus based on AR devices, as well as an AR device, to solve the problem that existing AR devices cannot perform secondary analysis and viewing of displayed information in reading scenarios.

[0006] In a first aspect, embodiments of this disclosure provide a reading method based on an AR device, including:

[0007] When the AR device is in reading mode, the reading data of the AR device at the current moment is acquired, and the target information is extracted from the reading data;

[0008] The AR device's search function is activated to find related information associated with the target information and displayed on the AR device.

[0009] In one embodiment of the first aspect of this application, the step of acquiring the reading data of the AR glasses at the current moment when the AR device is in reading mode includes:

[0010] When the AR device is in reading mode, it identifies whether there is a reading object within the field of view of the AR device;

[0011] If there is a reading material, the camera on the AR device is used to scan the reading material located in front of the AR device, extract the content of the reading material, and obtain reading data;

[0012] If there is no reading material, the displayed content is extracted from the display area of ​​the AR device to obtain reading data.

[0013] In one embodiment of the first aspect of this application, the step of activating the search function of the AR device to search for related information associated with the target information and displaying it on the AR device includes:

[0014] Determine the search method corresponding to the target information;

[0015] Call the search engine corresponding to the search method to search for related information that matches the target information;

[0016] The associated information is displayed on the display area of ​​the lens of the AR device.

[0017] In one embodiment of the first aspect of this application, the step of invoking a search engine corresponding to the search method to search for related information matching the target information includes:

[0018] If the target information is an image, extract the image features of the image in the image, and use the image features as an index to retrieve the associated information of the image from the corresponding search engine;

[0019] If the target information is a cited reference, extract the text features from the cited reference, use the text features as an index, and retrieve the related information of the cited reference from the corresponding search engine.

[0020] In one embodiment of the first aspect of this application, the step of extracting text features from the cited references when the target information is a cited reference, and using the text features as an index to retrieve the related information of the cited references from the corresponding search engine, includes:

[0021] If the target information is a cited reference, extract the text features from the cited reference and determine the content type of the cited reference based on the text features;

[0022] If the content type is a link, the link is accessed using the corresponding search engine, and the page content corresponding to the link is loaded to obtain the association information of the link;

[0023] If the content type is a file name, the text feature is used as an index to retrieve the associated information of the file name from the corresponding search engine.

[0024] In one embodiment of the first aspect of this application, after the step of activating the search function of the AR device to search for related information associated with the target information and displaying it on the AR device, the method further includes:

[0025] After detecting that the AR device has completed reading the reading material corresponding to the reading data, all reading data of the reading material is acquired;

[0026] The system uses a pre-set AI model to analyze all reading data, obtains a summary text, and generates reading records which are stored in the AR device.

[0027] In one embodiment of the first aspect of this application, after the step of activating the search function of the AR device to search for related information associated with the target information and displaying it on the AR device, the method further includes:

[0028] The camera in the AR device is used to collect the user's notes on reading data within the AR device's field of view, as well as the content at the location of the notes;

[0029] The notes are bound to their corresponding content to generate a note record, which is then stored in the AR device.

[0030] In one embodiment of the first aspect of this application, before the step of acquiring the reading data of the AR device at the current moment and extracting target information from the reading data, the method further includes:

[0031] Check if the AR device contains reading records and / or note records for the currently read data;

[0032] If reading records and / or note records exist, the most recent reading record and / or note record will be retrieved and displayed.

[0033] Secondly, embodiments of this disclosure provide a reading device based on an AR device, comprising:

[0034] The recognition module is used to acquire the reading data of the AR device at the current moment when the AR device is in reading mode, and extract target information from the reading data;

[0035] The search module is used to activate the search function of the AR device to search for related information that is associated with the target information;

[0036] The display module is used to display the associated information on the AR device.

[0037] In one embodiment of the second aspect of this application, the identification module is specifically used for:

[0038] When the AR device is in reading mode, it identifies whether there is a reading object within the field of view of the AR device;

[0039] If there is a reading material, the camera on the AR device is used to scan the reading material located in front of the AR device, extract the content of the reading material, and obtain reading data;

[0040] If there is no reading material, the displayed content is extracted from the display area of ​​the AR device to obtain reading data.

[0041] In one embodiment of the second aspect of this application, the search module includes:

[0042] The judgment unit is used to determine the search method corresponding to the target information;

[0043] The search unit is used to invoke the search engine corresponding to the search method to query related information that matches the target information.

[0044] In one embodiment of the second aspect of this application, the search unit is specifically used for:

[0045] If the target information is an image, extract the image features of the image in the image, and use the image features as an index to retrieve the associated information of the image from the corresponding search engine;

[0046] If the target information is a cited reference, extract the text features from the cited reference, use the text features as an index, and retrieve the related information of the cited reference from the corresponding search engine.

[0047] In one embodiment of the second aspect of this application, the search unit is specifically used for:

[0048] If the target information is a cited reference, extract the text features from the cited reference and determine the content type of the cited reference based on the text features;

[0049] If the content type is a link, the link is accessed using the corresponding search engine, and the page content corresponding to the link is loaded to obtain the association information of the link;

[0050] If the content type is a file name, the text feature is used as an index to retrieve the associated information of the file name from the corresponding search engine.

[0051] In one embodiment of the second aspect of this application, the AR-based reading device further includes: a summarizing module, which is used for:

[0052] After detecting that the AR device has completed reading the reading material corresponding to the reading data, all reading data of the reading material is acquired;

[0053] The system uses a pre-set AI model to analyze all reading data, obtains a summary text, and generates reading records which are stored in the AR device.

[0054] In one embodiment of the second aspect of this application, the AR-based reading device further includes: a note-taking module, which is used for:

[0055] The camera in the AR device is used to collect the user's notes on reading data within the AR device's field of view, as well as the content at the location of the notes;

[0056] The notes are bound to their corresponding content to generate a note record, which is then stored in the AR device.

[0057] In one embodiment of the second aspect of this application, the search module is further configured to:

[0058] Check if the AR device contains reading records and / or note records for the currently read data;

[0059] The display module is also used to extract and display the most recent reading record and / or note record when a query shows that reading records and / or note records exist.

[0060] Thirdly, this disclosure provides an AR device, including a processor and a memory. The memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the reading method based on the AR device provided above.

[0061] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing machine-executable instructions. When the machine-executable instructions are invoked and executed by a processor, the machine-executable instructions cause the processor to implement the reading method based on the AR device described above.

[0062] The embodiments disclosed herein bring the following beneficial effects:

[0063] The aforementioned reading method, device, AR device, and storage medium based on AR devices acquire the current reading data of the AR device when it is in reading mode, and extract target information from the reading data. The AR device's search function is then activated to retrieve related information that is associated with the target information and displayed on the AR device. This method extracts target information from the AR device's reading data and uses the AR device's search function to retrieve corresponding related information. This allows users to not only view the content when reading with an AR device, but also to perform real-time analysis of the displayed content, providing users with more detailed and specific descriptions. This addresses the problem that existing AR devices cannot perform secondary analysis of displayed information in reading scenarios, thus reducing the user's reading experience when using AR devices.

[0064] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objects and other advantages of this disclosure are realized and obtained through the structures particularly pointed out in the description, claims and drawings.

[0065] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0067] Figure 1 This is a schematic diagram of the first embodiment of the device-based reading method provided in the disclosed embodiments of the present invention;

[0068] Figure 2 A schematic diagram of a second embodiment of the reading method based on an AR device provided in the present invention;

[0069] Figure 3 This is a schematic diagram showing the illustrations in a book provided for an embodiment of the present invention;

[0070] Figure 4 This is a schematic diagram illustrating the display of the content of the searched image provided in an embodiment of the present invention;

[0071] Figure 5 A schematic diagram of a third embodiment of the reading method based on an AR device provided in the present invention;

[0072] Figure 6 A schematic diagram of an AR device provided in an embodiment of the present invention;

[0073] Figure 7 This is another schematic diagram of an AR device provided in an embodiment of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0075] Existing AR devices have display and communication functions, especially their ability to communicate via the Internet. Therefore, users can currently use existing AR devices to view documents, which can be downloaded directly from the Internet or received from other devices. However, whether downloaded or received, it only displays the data and does not provide further analysis of the displayed data. Especially when reading books, if users want to further understand certain content, they need to exit the AR device and use other devices to search, which greatly reduces the user's reading experience.

[0076] To address the aforementioned issues, this disclosure provides a reading method, AR glasses, and storage medium based on an AR device, which can be applied to scenarios where AR devices display various data, such as reading physical books and e-books.

[0077] In one embodiment of this disclosure, the AR-based reading method can run on either the AR device or a server. When the AR-based reading method runs on an AR device, the method can be implemented and executed based on a controller within the AR device. The controller includes a communication module and a memory, which stores a corresponding application program for controlling communication, recognition, data searching, etc.

[0078] For ease of understanding, the specific process of the embodiments of this disclosure is described below. Please refer to [link / reference]. Figure 1 This method can be applied to the aforementioned AR devices. One embodiment of the reading method based on an AR device in this disclosure includes the following steps:

[0079] Step 101: When the AR device is in reading mode, acquire the reading data of the AR device at the current moment, and extract the target information from the reading data.

[0080] Specifically, when the AR device is in reading mode, the content within the field of view of the AR device is acquired to obtain reading data; information that meets preset recognition rules is extracted from the reading data to obtain target information.

[0081] The reading mode should be understood as a mode for using AR devices to view or display information. Of course, it can also be a state of browsing any information. The reading mode can be triggered by specific conditions or it can be continuously executed when the AR device is turned on.

[0082] Specifically, the AR device can be wearable devices equipped with AR technology, such as AR glasses or AR helmets. The AR glasses can be ordinary glasses with added AR functionality, or they can be wearable devices with AR capabilities. These AR glasses include a frame and a camera, with the camera positioned on one end of each temple near the lens. When a user is reading, they can activate the AR function and use the camera to capture data from the book in front of them, thus obtaining reading data.

[0083] In practical applications, the camera on AR glasses captures an image in front of the user or scans objects within the user's field of vision, determining whether the captured image or scanned object is a physical book. If so, the content of the image or object is extracted to obtain reading data. For example... Figure 3 As shown, if an object with an outline similar to or the same as that of a physical book is identified within the field of view, then the object is determined to be a book. The book is then scanned using scanning technology to extract its content and obtain reading data.

[0084] Of course, the reading data can also be e-books. When AR glasses are used as a data display screen, they can establish a communication connection with other devices by turning on the AR function, and receive data from other devices or grab data from the display screen.

[0085] When a user needs to use the AR device to view documents, the device switches to reading mode in response to the user's touch operation on the AR device's control interface or buttons. In this mode, the device identifies the type of object within its field of view in real time. This object can be understood as an animal (such as a person, rabbit, etc.), a physical object (such as a table, tree, etc.), words, books, etc. After acquiring the object, key data is extracted from it using scanning technology. This key data is actually text, images, etc. Reading data is generated based on this key data and then parsed to obtain target information. This target information includes images and cited references, which include URLs and file names. The file names include book titles / author titles, and may even be citation identifiers.

[0086] Furthermore, the target information can also be extracted through recognition rules, that is, based on preset recognition rules for unknown features, feature extraction technology is used to extract feature information that meets the recognition rules from the reading data to obtain the target information. Specifically, the recognition rules are determined based on the composition rules of the information to be searched by the user. The information to be searched should be understood as information that needs further interpretation, including information that cannot be converted into text using image-to-text conversion technology, images of websites, book titles, and book titles. Of course, it can also be user-defined, such as personal names, landmark names, etc.

[0087] In this embodiment, after the AR device is started, it monitors its own working mode in real time through a self-test program. The working modes include: 1) viewing mode, which mainly refers to the mode used by the user when viewing the screen content, including full-screen display mode and screen follow mode; 2) interaction mode, which mainly refers to the mode in which the user needs to interact with the AR device through voice, trackball, gyroscope, etc.; 3) usage mode, which mainly refers to some interactions between the user and the AR glasses, including focus mode, sports mode, navigation mode, reading mode and custom mode.

[0088] If the working mode is usage mode, the type of usage mode will continue to be detected. If it is reading mode, the environmental data within the field of view of the AR device will be collected through the camera on the AR device. This environmental data can be understood as an image, that is, the scene within the field of view is captured by the camera, and then the content in the scene is identified to obtain reading data.

[0089] Of course, you can also directly capture the image displayed on the lens of the AR device, then identify the content in the image to obtain reading data.

[0090] As an example, and not a limitation, it involves using artificial intelligence recognition algorithms to extract information from captured images, such as OCT recognition technology and AI algorithms (such as ChatGPT). The artificial intelligence algorithms are then used to convert the images into an editable document, which includes text, images, and links (i.e., URLs). The images can be book-like pictures or portraits.

[0091] In this embodiment, an artificial intelligence recognition algorithm is used to convert the image into a document. Specifically, the text and non-text areas in the image are identified. For the text areas, a text recognition algorithm is used to extract the text and generate text. For the non-text areas, a contour extraction algorithm is used to extract the contour features and generate patterns. The text and patterns are then formatted according to the display format to obtain an editable document, which is the reading data.

[0092] Furthermore, the extraction of target information from reading data can be achieved through a model. This model is constructed using semantic and image recognition algorithms. Specifically, an initial model is built using semantic and image recognition algorithms, and then training samples are used to identify the initial model to obtain the final model that can extract the target information.

[0093] The training process includes: first, collecting samples and labeling them with website, book title, book name, image, etc.; then, inputting the labeled samples into the initial model for learning and training; for example, the model extracts the labeled information according to the label classification, and then performs semantic parsing to obtain the corresponding semantic summary or information combination rules, thereby obtaining a model that can extract the target information.

[0094] Specifically, the trained model is used to extract target information from reading data. This involves using text and image recognition models to parse and transform the content of the document, resulting in text and images. For the images, a book image recognition model is used to analyze them, filtering out book-related and non-book-related images. For the text, it is also necessary to identify whether there are links, book titles, or other words or phrases without actual semantic meaning, i.e., to identify hyperlinks or URLs, thereby obtaining the target information.

[0095] In practical applications, in addition to the image recognition methods mentioned above, the presence of books can also be identified from the text based on the book's markings and expressions.

[0096] Step 102: Activate the search function of the AR device to search for related information that is associated with the target information and display it on the AR device.

[0097] After identifying and extracting the target information, enable the internet access function of the AR device, such as turning on the signal switch of the 2 / 3 / 4 / 5G communication module in the AR device.

[0098] Optionally, after recognizing the target information, the AR device generates a switch command, powers on the communication module based on the switch command, constructs retrieval information from the extracted target information, and then calls the search engine on the AR device based on the retrieval information to search for corresponding related information from the Internet and displays it on the content currently being read by the AR device.

[0099] In practical applications, AR devices are equipped with multiple search software programs, such as browsers, various types of databases, such as the National Library of China, and various scientific and technological journals. After identifying the target information, the network is activated, the various search software programs are run, and then one of them is selected as the search engine based on the type of target information to retrieve the corresponding information and display it.

[0100] Of course, AR devices also have a small database, such as a cache, which stores data that has been viewed recently. When the AR device is in reading mode and reading an article, the recognized target information is retrieved from the cache to match the corresponding cache record. If it exists, it is retrieved and displayed.

[0101] Furthermore, if a matching cached record is found, the access IP address in the cached record can be extracted and accessed through a browser to extract the associated information corresponding to the target information, and then converted into this text to obtain the associated text information.

[0102] In this embodiment, when displaying associated text information on an AR device, the associated text information is either replaced with the corresponding target information, or it is marked and displayed in the edge area of ​​the target information through annotations, indexes, or other means for the user to view.

[0103] The aforementioned reading method based on AR devices acquires the current reading data of the AR device when it is in reading mode, extracts target information from the reading data, and then activates the AR device's search function to find related information that is associated with the target information and displays it on the AR device. This addresses the problem that existing AR devices cannot perform secondary analysis of the displayed information in reading scenarios, thus reducing the user's reading experience when using AR devices.

[0104] See Figure 2-4 The reading method based on AR devices provided by this invention will be described in detail below, taking into account specific scenarios. The method includes the following steps:

[0105] Step 201: When the AR device is in reading mode, identify whether there is a reading object within the field of view of the AR device.

[0106] By collecting the user's touch operations on the AR device, it is identified whether the touch operation is a working mode selection operation. If so, the number of times the touch operation triggers the working mode selection is counted to determine the corresponding working mode.

[0107] As an example, and not a limitation, the AR glasses are equipped with a viewing mode, an interaction mode, and a usage mode, where the usage mode corresponds to the reading mode, also known as a book reading mode. After determining that the touch operation is a selection operation for the working mode, the number of touches or touch logic after triggering the working mode operation is identified. It is determined whether the number of touches meets the threshold set for the reading mode or whether the touch logic is the same as the sliding trajectory set for the reading mode. If the reading mode is determined based on the number of touches or the touch logic, the reading data to be read is obtained through the AR device. Specifically, firstly, the camera is turned on to capture a video of the environment in front, and it is identified whether the content in the video is a book-like reading material. If it exists, step 202 is executed; otherwise, step 203 is executed to collect reading data. Of course, the reading material can also be an electronic product carrying reading data.

[0108] In one embodiment, after entering the selection of the working mode, the touch trajectory of the touch operation is acquired. The touch trajectory may be a series of multiple clicks, long presses, etc. The logic based on the touch trajectory is matched with the touch logic corresponding to different working modes to determine the corresponding working mode.

[0109] When the working mode is reading mode, the camera on the glasses is turned on to monitor and collect information in the environment in front in real time. Specifically, it collects images within the field of view according to the movement path of the AR device controlled by the user, and uses AI algorithms to perform preliminary recognition of the images. Based on the results of the preliminary recognition, it is determined whether there is a book.

[0110] In practical applications, AI algorithms are used to extract the outline of a book from images captured by a camera and calculate the proportion of the book outline occupying the field of view. If the proportion is greater than a preset value, it is determined that a book exists and that the book is the target to be read.

[0111] Step 202: If there is a reading material, use the camera on the AR device to scan the reading material located in front of the AR device, extract the content of the reading material, and obtain reading data.

[0112] In this embodiment, if the reading material is a book, and a book is detected within the AR device's field of view, a scanning function is activated to scan the image within the field of view and extract the content located within the book's area to obtain reading data. This content includes text and / or images. The scanning here can be understood as extracting the page content of the book within the field of view. The book can be understood as a physical book or the display screen of an electronic device that displays text or other content. Besides identifying the outline of an object, the detection of the presence of a book within the field of view can also be achieved by directly using text scanning technology to identify areas with text within the field of view. If text is present, it is determined whether the length of the text within that area meets a preset threshold. If it does, the presence of a book within the field of view is confirmed, and the content within that area is extracted to obtain reading data.

[0113] In practical applications, high-definition images of the book are captured using a camera on an AR device. These images are then input into a feature extraction model to extract image features, which include both text and images. Specifically, the feature extraction model can be built using text extraction algorithms. When an image is input into this model, text information is extracted. Content that can be converted to text is converted into text; for content that cannot be converted to text, the outline of the corresponding area is extracted to generate graphic features. Reading data is then generated based on these text and graphic features.

[0114] Step 203: If there is no reading material, extract the displayed content from the display area of ​​the AR device to obtain reading data.

[0115] In this embodiment, when the camera detects that no book is present, it checks whether the AR device has a communication connection with other terminal devices or whether there is a network access operation. If so, it receives the information to be displayed through the communication connection and obtains reading data.

[0116] In another embodiment, upon detecting a communication connection or access operation, the page data displayed on the AR device's screen at the current moment is acquired through screen recording, and the content in the page data is extracted to generate reading data. This content includes text and / or images, and the page data is projection data received by the AR device from another terminal or a page downloaded by the AR device from the internet. Figure 3 As shown, a book is present in the image captured by the camera of the AR device, and the current page of the book contains text and an image of a person. Using existing object and behavior recognition schemes and image-to-text technology, the text in the image is extracted, and the outline features of the person are scanned and displayed together on the AR device.

[0117] Step 204: Extract target information from the reading data.

[0118] Specifically, information that meets preset recognition rules is extracted from reading data to obtain target information. For example, image recognition and semantic recognition technologies are used to extract graphic features and / or text information from reading data to generate target information.

[0119] In this embodiment, the target information includes images and cited references. The cited references include URLs (such as links) and file names, specifically book names / work titles. Specifically, the reading data is analyzed and extracted for features, and the extracted features are classified as text and images. If the extracted features are text, the extracted text information is combined with semantic recognition to extract features without semantic meaning, thus obtaining the target information. For example, adjacent and continuous combinations of text in the extracted text information are performed, and the combined string is compared and analyzed with the combination rules of semantically insensitive words and phrases such as URLs / book names / work titles to obtain the analysis results. Based on the analysis results, words and phrases such as URLs / book names / work titles are extracted from the reading data.

[0120] If the extracted features are images, then contour extraction technology is used to extract the corresponding portrait features to obtain the target information of the image.

[0121] Furthermore, it includes identifying whether a portrait feature represents a book or a non-book image, and classifying the image into two categories based on the identification results. Specifically, it calculates the contour similarity between the portrait feature and an image of a book, and determines whether the feature belongs to the book category based on the contour similarity.

[0122] Similarly, the principle for extracting target information from images directly captured from the display screen of an AR device is the same as that for extracting from a camera, and will not be repeated here.

[0123] Step 205: Determine the search method corresponding to the target information.

[0124] Step 206: Call the search engine corresponding to the search method to search for related information that matches the target information.

[0125] In this embodiment, the target information includes both images and text. Here, text should be understood as URLs, book titles, author titles, etc., while images are categorized into book images and other images. Of course, books can also be represented through text. Specifically, by determining the type of the target information and based on that type, a corresponding search method is determined; the search engine corresponding to the search method is invoked to query data that matches the content of the target information.

[0126] In one embodiment, the information type includes two types: images and cited references, specifically:

[0127] If the target information is an image, extract the image features of the image in the image, and use the image features as an index to retrieve the associated information of the image from the corresponding search engine, such as from an Internet image database;

[0128] If the target information is a cited reference, extract the text features from the cited reference, use the text features as an index, and use the corresponding search engine to retrieve the related information of the cited reference, such as by searching in a browser / Internet library.

[0129] Furthermore, since images can be categorized into book images and non-book images, the portrait features of the images are extracted. These portrait features are then used as indexes to retrieve the associated information of the images from the corresponding search engines, including:

[0130] Identify whether the content in the image is a book image;

[0131] If it is a book image, the title information in the image is extracted, and the title information is used as an index to retrieve the related information of the book image from the corresponding search engine, such as the complete text and / or synopsis of the book image.

[0132] If it is not a book image, then the image features of the portrait in the image are extracted, and the image features are used as an index to retrieve the associated information of the portrait from the corresponding search engine. The associated information can be the descriptive data of the portrait, such as the information retrieved from an Internet image database.

[0133] For example, if the image captured by the camera is a picture of a book, and the outline of the book image is a regular rectangle with title information within the outline, the image captured by the camera is analyzed based on this image structure to determine whether it is a picture of a book. If it is, text recognition technology is used to extract the text in the image to obtain the book's title information. Based on the title information, a book search engine is called to detect relevant information about the book from the corresponding database.

[0134] In another embodiment, if the target information is a cited reference, the text features in the cited reference are extracted, and the content type of the cited reference is determined based on the text features;

[0135] If the content type is a link, the link is accessed using the corresponding search engine, and the page content corresponding to the link is loaded to obtain the link's association information, such as by accessing it using a browser.

[0136] If the content type is a file name, the text feature is used as an index to retrieve the associated information of the file name, such as the complete file and / or a brief description, from the corresponding search engine (such as an Internet literature database). The file name is a book name, an author's name, etc.

[0137] As an example, and not a limitation, the link can be a hyperlink or a specific URL. When the non-text information is a hyperlink, its characteristics include the word text and the background link redirection relationship. The browser is invoked to query related information using the word text as the first index, and then the background link redirection relationship corresponding to the related information is matched to filter out matching terms. The page of the term is then accessed, and the data in the page is extracted to obtain the data that matches the non-text information.

[0138] In this embodiment, when querying related information of target information, the network communication function of the AR glasses must first be enabled, and then the search engine in the AR device is triggered.

[0139] As an example and not a limitation, the network communication function refers to the module that enables AR devices to access the Internet wirelessly, such as a 2 / 3 / 4 / 5G communication module, which is powered by control commands to enable network operation.

[0140] In one embodiment, the network communication function can also be enabled by switching reading modes, or by real-time monitoring and recognizing the appearance of target information within the AR device's field of view.

[0141] Triggering a search engine in an AR device can be understood as configuring network settings for various applications within the device. For example, when reading a book using an AR device, if the user has pre-downloaded the complete book content, the AR device can identify non-textual information in real time during reading to control the reading app's network settings. Specifically, this might involve granting the application database interface connection permission or granting the application permission to call the search engine page, etc.

[0142] Step 207: Display the associated information on the display area of ​​the AR device's lens.

[0143] In this embodiment, there may be multiple related information retrieved based on the target information. To address this, the method further includes extracting keywords from each related information, inputting the keywords and features of the target information into an AI model for semantic calculation and matching, obtaining matching results, and then selecting at least one related information as the target information from the retrieved related information based on the matching results.

[0144] Alternatively, one could calculate the matching degree of relevant information retrieved from various search engines, rank the matching degrees of the results from different search engines, and select the top N results as relevant information for the target information. From the retrieved relevant information, those that meet the preset matching degree are filtered out and converted into text for display on the AR device.

[0145] When displaying related information on AR devices, the related information is either converted into text information and replaced with the corresponding target information, or it is marked and displayed in the edge area of ​​the target information through annotations, indexes, etc., for users to view.

[0146] like Figure 4 As shown, by extracting the facial features of the person in the image, accessing the Baidu search engine through a browser, the corresponding person image is searched based on the facial features. Further information such as the person's introduction is then searched based on the person image. Finally, the information is summarized into text and displayed in the area next to the person. In the image, annotations are displayed below the image. In the image, "X" represents the related information found, "*" represents the text in the reading data, and the person image is the target information.

[0147] In another embodiment, after the step of activating the search function of the AR device to search for related information associated with the target information and displaying it on the AR device, the method further includes: after detecting that the AR device has completed reading the reading material corresponding to the reading data, acquiring all reading data of the reading material; analyzing all reading data using a preset AI model to obtain a summary text, and generating a reading record stored in the AR device.

[0148] In another embodiment, after the step of activating the search function of the AR device to search for related information associated with the target information and displaying it on the AR device, the method further includes: using the camera in the AR device to collect notes on reading data within the field of view of the AR device and the content at the location of the notes; binding the notes with the corresponding content to generate a note record and storing it in the AR device.

[0149] In another embodiment, before the step of obtaining the reading data of the AR device at the current moment and extracting target information from the reading data, the method further includes: querying whether there are reading records and / or note records of the currently read reading data in the AR device; if there are reading records and / or note records, then extracting the most recent reading record and / or note record for display.

[0150] The aforementioned reading method based on AR devices extracts images and cited literature in real time and uses the AR device's search function to retrieve corresponding related information. This enables AR devices to not only display content during reading but also perform real-time analysis of the displayed content, providing users with more detailed and specific descriptions. This addresses the problem that existing AR devices cannot perform secondary analysis of displayed information in reading scenarios, thus reducing the user's reading experience when using AR devices.

[0151] See Figure 5 This is the third embodiment of the reading method based on AR devices provided by the present invention. In this embodiment, the AR device is AR glasses. The AR glasses include a camera, a frame, lenses, and a control circuit. The lenses are disposed on the frame, the camera is disposed on one end of the two temples of the frame near the lenses, the control circuit is installed inside the two temples and connected to the camera, and an image coupling area is also provided on the lenses for displaying data.

[0152] The control circuitry for the AR glasses includes:

[0153] AR glasses recognition module: By turning on the camera, it uses existing object and behavior recognition schemes, as well as image-to-text technology, to recognize the collected image data. Specifically, it includes the following recognition steps: 1) Recognizing images, URLs, titles / book names, etc. in the book within the user's current field of view; 2) Recognizing the current page of the book, converting it into text using image-to-text technology, and saving it; 3) Recognizing the user's action of writing notes in the book; 4) Recognizing the user's completion of this reading session.

[0154] AR glasses search module: For content such as images, URLs, and book titles, a browser will be opened and a search engine will be used to search for it. The search results will then be displayed through the AR glasses display module.

[0155] AR Glasses Summary Module: Starting from the AR glasses recognition module and continuing until it finishes, the text recognized from each page of the book is fed into an AI large model (such as ChatGPT, Wenxin Yiyan, Xinghuo large model, etc.) to summarize the text content in 50-100 words. This summary is then displayed through the AR glasses display module. This summary is also saved, and when the user starts reading again, it will display the summary of the most recently read content.

[0156] AR glasses note-taking module: Some users like to take handwritten notes while reading to enhance their understanding. The system automatically saves a screenshot of the area around the user's handwritten notes (currently, only screenshots of the area around the handwritten notes can be saved; underlines and other notes don't need to be saved). When the user starts reading again, a screenshot of their most recent notes will be displayed. Up to three screenshots will be displayed in the user's field of vision initially; additional screenshots can be manually scrolled through using the glasses' temples or a ring.

[0157] AR Glasses Display Module: Within the user's current field of view, the AR glasses display search results from the AR glasses search module, summary content from the AR glasses summary module, and notes from the AR glasses notes module.

[0158] Based on the structure of the AR glasses provided above, a reading method based on an AR device is proposed, which includes the following steps:

[0159] Step 501: The user has put on the AR glasses.

[0160] The AR glasses are equipped with sensors to identify whether they are being worn by the user. For example, a contact sensor is installed on the temple to detect whether there is skin contact with the temple. If so, it is determined that the user is wearing the AR glasses and step 502 is executed.

[0161] Step 502: The user turns on the AR glasses to read books.

[0162] After detecting that the AR glasses are being worn, the system continues to detect the user's touch operations on the AR glasses to activate the reading mode (i.e., the reading mode mentioned above). At the same time as the reading mode is activated, the camera is powered on.

[0163] Step 503: The AR glasses automatically activate the AR glasses recognition module.

[0164] Step 504: The user begins reading the book.

[0165] The camera collects information in front of the AR glasses in real time and outputs it to the AR glasses' recognition module. The AR glasses' recognition module uses AI recognition algorithms to extract text and non-text information from the collected information.

[0166] To recognize when a user starts reading, the system checks whether the position in the image data captured by the camera in multiple frames before and after the reading mode is triggered is consistent. In other words, it determines whether the AR glasses are in a static state for a certain period of time. If so, it confirms that the user is reading.

[0167] Furthermore, it can also detect and determine when a user has started reading by identifying whether there are page-turning operations in the video captured by the camera.

[0168] After the user starts reading, the information displayed in the AR glasses and the user's actions within the field of vision are detected in real time. Based on the detection results, the system jumps to step 505, step 509, step 511 or step 513.

[0169] Step 505: The AR glasses recognition module recognizes content such as images, URLs, and book titles.

[0170] Step 506: The AR glasses pass the recognized content to the AR glasses search module.

[0171] When the AR glasses recognition module recognizes an image, if the image type is an image, it identifies whether the content of the image is a book image. If it is a book image, it extracts the title information from the image and uses the title information as an index to retrieve the complete text and synopsis corresponding to the book image from an internet library. If it is not a book image, it extracts the image features of the image and uses the image features as an index to retrieve the description data of the image from an internet image database.

[0172] If the non-text information is a link, the link is used as the access address to call the browser in the AR glasses to access it and extract the content displayed in the browser.

[0173] Step 507: The AR glasses search module passes the search results to the AR glasses display module.

[0174] Step 508: The AR glasses display module shows the search results.

[0175] Specifically, it displays the results of searching for images, URLs, and titles / book names from search engines, such as image descriptions, corresponding web pages for websites, and text or indirect information for books.

[0176] Step 509: The AR glasses recognition module identifies the current page of the book.

[0177] Step 510: Convert the image to text using image-to-text technology and save it to the AR glasses summary module.

[0178] Specifically, the system detects user touch operations or AR glasses mode switching operations within the AR glasses' field of view; when the reading mode is detected to be exiting, it extracts all data displayed by the lenses in the AR glasses at the current moment; it calls an AI model to analyze all the data, obtains a summary text, and generates a reading record which is stored in the AR glasses.

[0179] Step 511: The AR glasses recognition module recognizes the user's handwritten notes in the book.

[0180] Step 512: The AR glasses note module saves a screenshot of the area near the user's handwritten notes.

[0181] Specifically, the system detects whether there is a user's touch operation within the AR glasses' field of view; if so, it collects the user's annotation information on the content displayed on the lenses of the AR glasses at the current moment; the annotation information is bound to the content displayed at the current moment to generate a note record, which is then stored in the AR glasses.

[0182] Step 513: The AR glasses recognition module recognizes that the user has finished reading the book.

[0183] Step 514: The AR glasses summary module summarizes the text content.

[0184] Step 515: The AR glasses display module shows a summary of the text content.

[0185] Step 516: The user will start reading the book next time.

[0186] Step 517: The AR glasses display module shows a summary of the most recent reading and screenshots of the notes you took during your most recent reading.

[0187] After saving all records and summaries, each time the book is viewed through AR glasses, the reading record and notes can be retrieved to check if there are any records related to the content currently displayed by the AR glasses; if a record exists, the most recent record is retrieved and displayed.

[0188] In this embodiment, by activating the reading mode of the AR glasses while the user is reading, when an image appears in the book, the system automatically connects to the internet and displays related search results through the AR glasses; when a website address appears in the book, it automatically opens in a browser and is displayed to the user; when a specific book appears in the book, the system automatically connects to the internet and displays related search results through the AR glasses; when the user finishes reading the book or starts reading again, a summary of the most recent reading is displayed; and when the user starts reading again, the user's notes from the most recent reading are displayed. This approach not only enables the AR glasses to display data while reading but also allows for real-time analysis of the displayed content, providing the user with more detailed and specific descriptions. This addresses the problem that existing AR glasses cannot perform secondary analysis of displayed information in reading scenarios, thus reducing the user's reading experience.

[0189] For the corresponding method embodiments described above, see [link to relevant documentation]. Figure 6 The diagram shows a reading device based on an AR device, the device comprising:

[0190] The identification module 610 is used to acquire the reading data of the AR device at the current moment when the AR device is in reading mode, and extract target information from the reading data;

[0191] The search module 620 is used to activate the search function of the AR device to search for related information that is associated with the target information;

[0192] Display module 630 is used to display the associated information on the AR device.

[0193] Optionally, the identification module 610 is specifically used for:

[0194] When the AR device is in reading mode, it identifies whether there is a reading object within the field of view of the AR device;

[0195] If there is a reading material, the camera on the AR device is used to scan the reading material located in front of the AR device, extract the content of the reading material, and obtain reading data;

[0196] If there is no reading material, the displayed content is extracted from the display area of ​​the AR device to obtain reading data.

[0197] Optionally, the search module 620 includes:

[0198] The judgment unit 621 is used to determine the search method corresponding to the target information;

[0199] Search unit 622 is used to invoke a search engine corresponding to the search method to query related information that matches the target information.

[0200] Optionally, the search unit 622 is specifically used for:

[0201] If the target information is an image, extract the image features of the image in the image, and use the image features as an index to retrieve the associated information of the image from the corresponding search engine;

[0202] If the target information is a cited reference, extract the text features from the cited reference, use the text features as an index, and retrieve the related information of the cited reference from the corresponding search engine.

[0203] Optionally, the search unit 622 is specifically used for:

[0204] If the target information is a cited reference, extract the text features from the cited reference and determine the content type of the cited reference based on the text features;

[0205] If the content type is a link, access the link using the corresponding search engine, load the page content corresponding to the link, and obtain the association information of the link;

[0206] If the content type is a file name, the text feature is used as an index to retrieve the associated information of the file name from the corresponding search engine.

[0207] Optionally, the AR-based reading device further includes a summary module 640, which is used for:

[0208] After detecting that the AR device is reading the reading material corresponding to the reading data, all reading data of the reading material is acquired;

[0209] The system uses a pre-set AI model to analyze all reading data, obtains a summary text, and generates reading records which are stored in the AR device.

[0210] Optionally, the AR-based reading device further includes: a note-taking module 650, which is used for:

[0211] The camera in the AR device is used to collect the user's notes on reading data within the AR device's field of view, as well as the content at the location of the notes;

[0212] The notes are bound to their corresponding content to generate a note record, which is then stored in the AR device.

[0213] Optionally, the search module 620 is further configured to:

[0214] Check if the AR device contains reading records and / or note records for the currently read data;

[0215] The display module 630 is further configured to, when querying for existing reading records and / or note records, extract and display the most recent reading record and / or note record.

[0216] This embodiment also provides an AR device, including a processor and a memory. The memory stores machine-executable instructions that can be executed by the processor. The processor executes the machine-executable instructions to implement the reading method based on the AR device provided in the above embodiment. The AR device can be a server or a wearable device.

[0217] See Figure 7 As shown, the AR device includes a processor 700 and a memory 701. The memory 701 stores machine-executable instructions that can be executed by the processor 700. The processor 700 executes the machine-executable instructions to implement the reading method based on the AR device provided in the above embodiments.

[0218] Furthermore, Figure 7 The AR device shown also includes a bus 702 and a communication interface 703. The processor 700, the communication interface 703, and the memory 701 are connected via the bus 702.

[0219] The memory 701 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 703 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network. The bus 702 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 7 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0220] The processor 700 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 700 or by instructions in software form. The processor 700 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 701. The processor 700 reads the information in memory 701 and, in conjunction with its hardware, completes the AR-based reading method provided in the above embodiments.

[0221] By extracting the target information from the reading data of the AR device at the current moment and querying the corresponding related information on the Internet, the AR device can display data while reading. It can also perform real-time analysis of the displayed content and provide users with more detailed and specific descriptions. This solves the problem that existing AR devices cannot perform secondary analysis of the displayed information in reading scenarios, which reduces the user's reading experience when using AR devices.

[0222] This embodiment also provides a machine-readable storage medium storing machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the reading method based on the AR device provided in the above embodiment.

[0223] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0224] Furthermore, in the description of the embodiments of this disclosure, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure based on the specific circumstances.

[0225] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0226] In the description of this disclosure, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0227] Finally, it should be noted that the above embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A reading method based on an AR device, characterized in that, The AR-based reading method includes: When the AR device is in reading mode, the reading data of the AR device at the current moment is acquired, and the target information is extracted from the reading data; The AR device's search function is activated to find related information associated with the target information and displayed on the AR device.

2. The reading method based on an AR device according to claim 1, characterized in that, The step of acquiring the reading data of the AR glasses at the current moment when the AR device is in reading mode includes: When the AR device is in reading mode, it identifies whether there is a reading object within the field of view of the AR device; If there is a reading material, the camera on the AR device is used to scan the reading material located in front of the AR device, extract the content of the reading material, and obtain reading data; If there is no reading material, the displayed content is extracted from the display area of ​​the AR device to obtain reading data.

3. The reading method based on an AR device according to claim 1 or 2, characterized in that, The step of activating the search function of the AR device to search for related information that is associated with the target information and displaying it on the AR device includes: Determine the search method corresponding to the target information; Call the search engine corresponding to the search method to search for related information that matches the target information; The associated information is displayed on the display area of ​​the lens of the AR device.

4. The reading method based on an AR device according to claim 3, characterized in that, The step of calling the search engine corresponding to the search method to search for related information matching the target information includes: If the target information is an image, extract the image features of the image in the image, and use the image features as an index to retrieve the associated information of the image from the corresponding search engine; If the target information is a cited reference, extract the text features from the cited reference, use the text features as an index, and retrieve the related information of the cited reference from the corresponding search engine.

5. The reading method based on an AR device according to claim 4, characterized in that, The step of extracting text features from the cited literature when the target information is a reference, and using the text features as an index to retrieve the related information of the cited literature from the corresponding search engine, includes: If the target information is a cited reference, extract the text features from the cited reference and determine the content type of the cited reference based on the text features; If the content type is a link, the link is accessed using the corresponding search engine, and the page content corresponding to the link is loaded to obtain the association information of the link; If the content type is a file name, the text feature is used as an index to retrieve the associated information of the file name from the corresponding search engine.

6. The reading method based on an AR device according to claim 2, characterized in that, After the step of activating the search function of the AR device to search for related information that is associated with the target information and displaying it on the AR device, the method further includes: After detecting that the AR device has completed reading the reading material corresponding to the reading data, all reading data of the reading material is acquired; The system uses a pre-set AI model to analyze all reading data, obtains a summary text, and generates reading records which are stored in the AR device.

7. The reading method based on an AR device according to claim 2, characterized in that, After the step of activating the search function of the AR device to search for related information that is associated with the target information and displaying it on the AR device, the method further includes: The camera in the AR device is used to collect the user's notes on reading data within the AR device's field of view, as well as the content at the location of the notes; The notes are bound to their corresponding content to generate a note record, which is then stored in the AR device.

8. The reading method based on an AR device according to claim 6 or 7, characterized in that, Before the step of acquiring the reading data of the AR device at the current moment and extracting target information from the reading data, the method further includes: Check if the AR device contains reading records and / or note records for the currently read data; If reading records and / or note records exist, the most recent reading record and / or note record will be retrieved and displayed.

9. A reading device based on an AR device, characterized in that, The AR-based reading device includes: The recognition module is used to acquire the reading data of the AR device at the current moment when the AR device is in reading mode, and extract target information from the reading data; The search module is used to activate the search function of the AR device to search for related information that is associated with the target information; The display module is used to display the associated information on the AR device.

10. An AR device, characterized in that, It includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the reading method based on an AR device according to any one of claims 1-8.

Images