Writed document-based interaction method, system and device, medium and program product

Abstract

The invention provides an interaction method, system and device based on written documents, a medium and a program product. The method comprises the steps of obtaining data capable of generating interaction instruction information according to a written document page; generating interaction instruction information according to the data capable of generating the interaction instruction information; acquiring electronic mapping document data corresponding to the written document page; determining an interaction processing action according to interaction instruction processing action information in the interaction instruction information; according to interaction instruction position information in the interaction instruction information, extracting interaction content data related to the interaction instruction position information from electronic mapping document data corresponding to the written document page; and executing an interaction processing action on the extracted interaction content data.

Description

Technical Field

[0001] This disclosure relates to the field of information processing, and more particularly to interactive methods, systems, electronic devices, non-transitory computer-readable storage media, and computer program products based on written documents. Background Technology

[0002] Currently, numerous digital education and office tools exist that can digitize and intelligently process electronic documents. However, due to the negative impact of screen backlighting on eyesight, primary and secondary schools in my country still primarily use written materials for learning, and adults also frequently use written materials for study and work. How to better utilize written materials to achieve electronic data interaction, providing digital and intelligent support for such written learning or office scenarios, and improving the efficiency of teacher-student interaction and team collaboration are problems that need to be addressed. Summary of the Invention

[0003] In view of this, the present disclosure provides an interactive method, system, electronic device, non-transitory computer-readable storage medium, and computer program product based on written documents.

[0004] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part by practice of this disclosure.

[0005] According to one aspect of this disclosure, an interaction method based on a written document is proposed, characterized in that the method includes: acquiring data capable of generating interactive instruction information from a written document page; generating interactive instruction information from the data capable of generating interactive instruction information; acquiring electronic mapping document data corresponding to the written document page; determining an interactive processing action based on interactive instruction processing action information in the interactive instruction information; extracting interactive content data related to the interactive instruction location information from the electronic mapping document data corresponding to the written document page based on the interactive instruction location information in the interactive instruction information; and performing an interactive processing action on the extracted interactive content data.

[0006] Here, the electronic mapping document data includes interactive content data corresponding to the written document page, and can obtain the location information of the interactive content data on the written document page or the electronic mapping document page. The interactive instruction information includes interactive instruction processing action information and / or interactive instruction location information and / or information used to obtain the electronic mapping document data corresponding to the written document page. The electronic mapping document data includes one or more electronic mapping document pages. The interactive instruction location information refers to one or more location information on the written document page or the electronic mapping document page related to the interactive instruction information. The interactive content data includes at least one of the following: document content, interactive preset data, interactive process data, and interactive result data. The document content refers to the content visible on the written document page or the corresponding document content data stored in the electronic mapping document data. The electronic mapping document page refers to the electronic document page stored or that can be generated in the electronic mapping document data corresponding to the written document page.

[0007] Here, a written document can be a document presented on paper or other media that is not electronically stored or displayed. Its content includes text, graphics, images, etc., on the written document. Alternatively, a written document page can be a paper-like page presented using electronic ink or other methods. It can also include documents displayed on various objects (including documents displayed on other display devices). Its essential characteristics are: first, it exists in the user's field of vision as a non-editable document image; second, it has matching electronically mapped document data. A written document includes one or more written document pages, and each written document page includes written document content. The written document content consists of one or more document content elements, which are the units that constitute the document content. In one embodiment, the display unit can be smart glasses, and the written document page may not be a real paper document, but rather an image displayed on the smart glasses.

[0008] In the method disclosed in this invention, the key to interaction lies in the electronically mapped document data corresponding to the written document. The written document is only used to visualize the document content and generate interactive instructions; it does not provide interactive content data for participation. The provision and storage of interactive content data are achieved through the electronically mapped document data. The electronically mapped document data includes one or more electronically mapped document pages. The electronically mapped document pages and their corresponding written document pages have the same document content and document content distribution. That is, selecting one or more document content elements on the written document page will yield the same document content elements within the corresponding selected area on the electronically mapped document page. Without affecting the document content and document content distribution, the font, background color, background image, etc., of the electronically mapped document pages can differ from those of the written document pages, and multimedia data can also be added.

[0009] Interactive content data in eMapped Document Data can include not only document content data, but also pre-defined interactive data, interactive process data, and interactive result data. Pre-defined interactive data refers to data that is stored in advance in the eMapped Document Data and may be used in subsequent interactive processes. For example, when the document content is an exam paper, the answers can be used as pre-defined interactive data. When the answers need to be retrieved, the stored pre-defined interactive data can be directly retrieved. Interactive process data is data generated during the interactive process. For example, user information and process data of interactive actions can be recorded. Interactive result data refers to the result data generated by the interactive process. For example, when querying the Chinese translation of a word in an English document, the query result can be stored as interactive result data in the eMapped Document Data. Interactive process data and interactive result data can also become new pre-defined interactive data.

[0010] Interactive processing actions typically include editing document content (e.g., adding, deleting, modifying, and querying document content; the original document content can be saved), processing actions related to specific document content (processing actions associated with specific document content but not changing the document content, such as looking up the English word "white" in a document, reading a passage of document content aloud, playing a video related to an image in a document; interactive process data or interactive result data can be associated with and saved with specific document content or document content location information), actions of communicating with others or services, such as conversations with others about document content, communication with artificial intelligence services, and other processing actions related to document content. In one embodiment of the present invention, interactive processing actions include communicating with others, and interactive result data includes data from communicating with others. A region within or outside the written document can be set as a communication region. Screenshots or recognized text acquired within this region are transmitted to the communication software, and the sent content or received feedback can be stored or displayed as interactive result data.

[0011] Electronic mapping document data can typically include images or formatted text, and can also include any data storing interactive content data and its location information (such as arrays, lists, tuples, etc.). It can also include interactive content data and its location information calculation rules and parameters. In one embodiment, the electronic mapping document data stores interactive instruction location information corresponding to one or more interactive processing actions. Based on the interactive instruction location information in the interactive instruction information, interactive content data related to the interactive instruction location information is extracted from the electronic mapping document data corresponding to the written document page. The method also includes: obtaining the interactive instruction location information from the electronic mapping document data or preset data based on the interactive instruction information. For example, a projection area is set on the written document page, and the preset data stores the coordinate information of the projection area as the interactive instruction location information for the "projection" interactive processing action. Assuming the projection area is set to the top of the written document page, when the "projection" action is executed, the coordinate information of the projection area at the top of the written document page is obtained from the electronic mapping document data, and the information to be projected is projected onto the corresponding area of ​​the written document page. In one embodiment, a question-answering area is provided on the written document page. The coordinate information of the question-answering area is stored in the electronic mapping document data as the location information of the interaction instruction for the "get answer" interactive processing action. When the "get answer" action is executed, the coordinate information of the question-answering area is obtained from the electronic mapping document data, and the information is obtained from the corresponding area on the written document page as the answer.

[0012] In one embodiment, the data capable of generating interactive instruction information includes data capable of generating interactive instruction location information. In one embodiment, the data capable of generating interactive instruction information includes written document elements or combinations of written document elements. By retrieving written document elements or combinations of written document elements, the corresponding location information or interactive content data can be obtained. For example, it is possible to request the counting or highlighting of the word "white" in the document content. In one embodiment, the interactive instruction location information may include the location information of the document content to be processed, the associated location information of the interactive result data, and the display location information of the interactive result data. For example, if there is a word "white" on a written document page, and an interactive instruction to "add the English translation 'white' to the word 'white'" is to be generated, the position coordinates of the word "white" in the written document are the location information of the document content to be processed. 'white' is an interactive result data that needs to be associated with the word "white". The associated location information of the interactive result data is the position coordinates of the word "white" in the written document. It is also possible to request that "white" be displayed above the word "white", and the desired display position of "white" is the display location information of the interactive result data.

[0013] In one embodiment, the steps of the method can be combined and adjusted according to the actual situation. For example, one can first "obtain data that can generate interactive instruction information" and "determine interactive content data", and then determine "obtain data that can generate interactive instruction information" and "determine interactive processing actions".

[0014] In this way, interactive processing of electronically mapped document data can be achieved using written documents. On the user's end, the written document page acts like a display, firstly showing the document content to the user, and secondly providing a coordinate system for the written document page. Combined with other input information, the written document page becomes a visual document data editor and a digital interactive platform. On the data processing end, the written document page is only responsible for providing interactive instructions; all data to be processed is obtained from the electronically mapped document data. While enabling interaction using written documents, it also improves the accuracy of data acquisition, enriches the content of interactive processing, and avoids the impact of handwriting or damage to the document content on the interactive effect, thus maintaining the independence of using written documents.

[0015] This disclosure contains the following:

[0016] In some embodiments, the data capable of generating interactive instruction information includes an image with a written document page, the image having one or more visual identifiers within it, the visual identifiers being either within or outside the written document page in the image. Generating interactive instruction information based on the data capable of generating interactive instruction information specifically includes: identifying one or more visual identifiers within the image with the written document page; obtaining a coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the written document page, or obtaining a coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the electronically mapped document page; and determining interactive instruction information based on the visual identifiers and the obtained coordinate correspondence.

[0017] Thus, on the operating end, the location information of the visual identifier on the written document page can be obtained from the image with the written document page, and further interactive instruction processing action information and interactive instruction location information can be generated to obtain the interactive processing operation instructions.

[0018] In some embodiments, the data capable of generating interactive instruction information includes one or more first coordinate system coordinate information. The first coordinate system coordinate information is the coordinate information in the indicated first coordinate system, which intersects with, is parallel to, or coincides with the written document page. Generating interactive instruction information based on the data capable of generating interactive instruction information specifically includes: obtaining the coordinate correspondence between the first coordinate system and the written document page coordinate system, or obtaining the coordinate correspondence between the first coordinate system and the electronic mapping document page coordinate system; and determining the interactive instruction information based on the first coordinate system coordinate information and the obtained coordinate correspondence.

[0019] Here, the written document page coordinate system is a two-dimensional coordinate system within the written document page, and the electronically mapped document page coordinate system is a two-dimensional coordinate system within the electronically mapped document page. The first coordinate system intersects, is parallel to, or coincides with the written document page coordinate system. The first coordinate system coordinate information of any point on the written document page can be obtained through the first coordinate system. This first coordinate system coordinate information is used to retrieve interactive content data that meets the location information conditions from the electronically mapped document data. Whether the desired coordinate correspondence is obtained between the first coordinate system and the written document page coordinate system, or between the first coordinate system and the electronically mapped document page coordinate system, depends on whether retrieving the interactive content data from the electronically mapped document data requires location information from the written document page or location information within the electronically mapped document page. In one embodiment, the written document page is fixed at a fixed position within the first coordinate system, and a fixed coordinate correspondence exists between the first coordinate system and the written document page coordinate system. In another embodiment, the position of the written document page within the first coordinate system can be automatically measured, thereby allowing the coordinate correspondence between the first coordinate system and the written document page coordinate system to be obtained at any time. In one embodiment, the position of the indicated location in the coordinate system of the written document page is determined based on the coordinate information of the first coordinate system and the coordinate correspondence between the first coordinate system and the coordinate system of the written document page. This position is then combined with other data capable of generating interactive instruction information to determine the interactive instruction information. In another embodiment, the electronically mapped document data can be directly manipulated using the display and the coordinate information of the first coordinate system. In one embodiment, the coordinate information of the first coordinate system is the coordinate information of an image captured by an imaging device.

[0020] Thus, on the operating end, the coordinate information in the first coordinate system can be converted into coordinates in the coordinate system of the written document page or the electronically mapped document page, further generating interactive command information. This allows the use of touchscreens, mice, or other pointing devices, or spatial positioning devices such as wireless positioning, as input devices for operating commands on the written document page. Similar mouse or touchscreen operation effects can be directly achieved on the written document page, enriching the selection of supporting devices and further improving the convenience of operation.

[0021] In some embodiments, based on the interaction instruction location information in the interaction instruction information, interactive content data related to the interaction instruction location information is extracted from the electronically mapped document data corresponding to the written document page. Specifically, this includes: determining the interactive content area range based on the interaction instruction location information in the interaction instruction information; obtaining interactive content data from the electronically mapped document data, and obtaining the location information of the interactive content data on the written document page or the electronically mapped document page; and selecting interactive content data whose location information meets certain conditions as interactive content data related to the interaction instruction location information based on the location information of the interactive content data on the written document page or the electronically mapped document page, and the interactive content area range, wherein the condition refers to the location information being within the interactive content area range and / or having an intersection with the interactive content area range.

[0022] Here, the electronic mapping document data typically stores interactive content data, the location information of the interactive content data on the written document page or the electronic mapping document page, and the mapping relationship between the two; or it stores interactive content data and rules for calculating the location information of the interactive content data on the written document page or the electronic mapping document page (e.g., rich text or other electronic documents that can generate electronic mapping document pages corresponding to the written document pages). In one embodiment, the location information of the interactive content data on the written document page or the electronic mapping document page is relative location information on the page, for example, the horizontal coordinate = distance from the coordinate point to the left side of the page / page width, and the vertical coordinate = distance from the coordinate point to the top of the page / page length. In one embodiment, the interactive content data is divided into different types, and interactive content data whose location information meets the conditions is selected as interactive content data related to the location information of the interactive instruction. It also includes selecting the corresponding interactive content data according to the interactive processing action. For example, if the interactive processing action is "select to view the notes within the area", then only interactive content data within the interactive content area and of the type of notes is selected.

[0023] Thus, on the data processing end, interactive content data can be retrieved from the electronically mapped document data based on the location information corresponding to the interactive command information and the interactive content data, enabling operations on the electronically mapped document data using the written document page. This design brings the following benefits: first, it enables operations on the document content data in the electronically mapped document data, improving data accuracy; second, it allows interaction with other data not visible in the written document, greatly enriching the interactive content; and third, because the written document page does not need to provide interactive content data, the design of the operating end is not limited by the imaging device, allowing for greater flexibility.

[0024] In some embodiments, interactive content data includes document content elements or combinations of document content elements, and electronically mapped document data includes document content elements or combinations of document content elements. The location information of the document content elements or combinations of document content elements on the written document page or the electronically mapped document page can be obtained. A document content element is a unit constituting the content of the written document, and a combination of document content elements is a combination of multiple document content elements. Based on the location information of the interactive content data on the written document page or the electronically mapped document page, and the range of the interactive content area, interactive content data whose location information meets the conditions is selected as the interactive content data related to the location information of the interactive instruction. Specifically, this includes: determining document content elements or combinations of document content elements whose location information meets the conditions based on the range of the interactive content area and the location information of the document content elements or combinations of document content elements on the written document page or the electronically mapped document page; and arranging and combining the document content elements or combinations of document content elements whose location information meets the conditions based on the location information of the document content elements or combinations of document content elements on the written document page or the electronically mapped document page to generate interactive content data.

[0025] Here, document content elements can be characters, words, phrases, sentences, paragraphs that constitute an article, or images or parts of images, or graphics, shapes, formulas, or tables in the document, or multimedia information such as videos. In one embodiment, characters, words, phrases, and paragraphs in document content elements can be saved as characters or as images. In one embodiment, document content elements or combinations of document content elements can be deleted, modified, or added during interaction. In one embodiment, multi-line interactive content data can be arranged into multiple lines according to their original positional relationship, or concatenated into a single line based on positional information. In one embodiment, document content elements or combinations of document content elements are saved not only in relation to positional information but also in relation to their historical interaction process data or interaction result data. In one embodiment, a WYSIWYG electronic document page can be generated on the display based on the document content elements and their corresponding positional information, allowing for visual modification using the display. In one embodiment, positional information includes the center point position of the document element and parameters describing the size of the document element (e.g., the font size of the characters).

[0026] Thus, on the data processing side, a structured document content storage method is provided. This storage method supports three data processing methods: document content can be processed through written document operations using the methods disclosed in this invention; data modification operations can be performed directly on document content elements and their corresponding location information; and a WYSIWYG electronic document page can be generated on the display for visual modification. At the same time, this structured data structure also has good scalability.

[0027] In some embodiments, the eMapped Document Data includes an image of an eMapped Document Page corresponding to a written document page, or an electronic document capable of generating an eMapped Document Page corresponding to a written document page. Based on the interaction instruction location information in the interaction instruction information, interactive content data related to the interaction instruction location information is extracted from the eMapped Document Data corresponding to the written document page. Specifically, this includes: determining the corresponding interactive content area range in the eMapped Document Page based on the interaction instruction location information; extracting a portion of the interactive content area range from the eMapped Document Page based on the corresponding interactive content area range; and extracting the document content within the interactive content area range as interactive content data based on the portion extracted from the eMapped Document Page.

[0028] Here, the electronically mapped document data includes, or can generate, an image of an electronically mapped document page corresponding to the written document page. The image of the electronically mapped document page corresponding to the written document page can be completely identical to the written document page, or it can differ without affecting the accuracy of the interactive content data extraction. For example, the image of the electronically mapped document page may lack background colors, special effects, or other decorative elements, or it may not use difficult-to-recognize fonts. In one embodiment, the position range occupied by any document content element in the electronically mapped document page is equal to or contained within its position range in the written document page. That is, after selecting any document content element in the written document page, the selected interactive content area is mapped onto the electronically mapped document page. The mapped interactive content area contains only the same document content element. For example, for the same character, the font size on the electronically mapped document page can be smaller than the font size on the written document page. The extracted interactive content data can be a screenshot of the interactive content area, characters extracted from a screenshot, or images extracted from a screenshot. In one embodiment, determining the corresponding interactive content area range in the electronically mapped document page based on the interactive instruction location information in the interactive instruction information includes first determining the corresponding interactive content area range in the written document page based on the interactive instruction location information in the interactive instruction information, and then determining the corresponding interactive content area range in the electronically mapped document page based on the coordinate correspondence between the coordinate system of the written document page and the coordinate system of the electronically mapped document page.

[0029] In one embodiment, the coordinate correspondence between the written document page and the electronically mapped document page is determined by obtaining the coordinates of specific points on the written document page. Further, the specific points on the written document page can be one or more document content elements. In another embodiment, the image of the written document page is corrected based on the coordinate correspondence between the written document page and the electronically mapped document page. In yet another embodiment, the electronically mapped document data includes an image of the electronically mapped document page corresponding to the written document page. The image of the electronically mapped document page is completely identical to the written document page. After determining the interactive content area, the image is cropped according to the document content area, and then the document content within the interactive content area is obtained through image recognition. In yet another embodiment, the electronically mapped document data includes an electronic document capable of generating an electronically mapped document page corresponding to the written document page, such as a Word document, PDF document, or PPT document. After determining the interactive content area, characters, text, images, videos, and other data within the document content area are extracted as the document content within the interactive content area according to the electronic document's format generation rules. In yet another embodiment, the interaction process can change the document content of the electronically mapped document page. After the document content is changed, a new written document page can be printed based on the electronically mapped document page. In one embodiment, the image file can be printed as a written document, or the image file can be directly displayed as a written document on a monitor, and the electronically mapped document data can be manipulated according to the method disclosed in this invention.

[0030] Thus, on the data processing side, common image or text files can be used for interaction, expanding the interactive materials. Furthermore, from the perspective of specific implementation methods, the electronically mapped document page does not need to be completely identical to the printed document page. Therefore, the printed document page can use more aesthetically pleasing fonts and background colors, while the electronically mapped document page can use simple and easily recognizable fonts, thereby improving the accuracy of document content recognition while ensuring the aesthetics of the printed document page.

[0031] In some embodiments, determining the interaction instruction information based on visual identifiers and the coordinate correspondence between the image with the written document page and the written document page specifically includes: in response to one or more visual identifiers including at least one visual identifier of type interaction instruction location information identifier, determining the position of the interaction instruction location information identifier in the image with the written document page; and determining the interaction instruction location information in the interaction instruction information based on the position of the location information identifier in the image with the written document page and the coordinate correspondence between the image with the written document page and the written document page.

[0032] In one embodiment, a highlighter is used to mark the interactive content area on the written document page as an indicator of the interactive command location information. Furthermore, different colored highlighters can correspond to different interactive actions. In one embodiment, a pen of a certain color is used to circle the interactive content area on the written document page as an indicator of the interactive command location information. In another embodiment, a specific shape (by placing an object of a specific shape or drawing a specific shape on the page) is used to mark feature points of the interactive content area on the written document page as an indicator of the interactive command location information. In yet another embodiment, notes are written, and a specific shape is used to circle the notes and point to the note addition location on the written document page. The specific shape serves as the interactive command location information indicator, and the coordinates of the note addition location on the written document page serve as the interactive command location information.

[0033] Thus, on the operating side, visual identifiers in images containing written document pages can be used to determine the location information of interactive instructions in interactive instruction information.

[0034] In some embodiments, obtaining the electronically mapped document data corresponding to a written document page specifically includes one or more of the following steps: determining the electronically mapped document data corresponding to the written document page by comparing the similarity between the whole or a part of the image of the written document page and the whole or a part of the electronically mapped document data; determining the electronically mapped document data corresponding to the written document page in response to one or more visual identifiers including at least one visual identifier of the type of written document identification identifier, and in response to the written document identification identifier being consistent with the identification identifier stored in the electronically mapped document data; determining the information recorded in the written document identification identifier in response to one or more visual identifiers including at least one visual identifier of the type of written document identification identifier, and in response to the recorded information being consistent with the information in the electronically mapped document data; and determining the electronically mapped document data corresponding to the written document page in response to input information, and in response to the input information being consistent with the information in the electronically mapped document data.

[0035] In one embodiment, the written document page carries a written document identification identifier or data used to identify the electronically mapped document.

[0036] Thus, on the operating side, it is possible to automatically or manually obtain the electronic mapping document data corresponding to the pages of the written document using various methods.

[0037] In some embodiments, performing interactive processing actions on the extracted interactive content data further includes: obtaining the relative positional relationship between the projection or display position of the interactive processing result and the written document page or the image of the written document page, based on the interactive instruction position information in the interactive instruction information or a preset rule; and projecting or displaying the processing result of the interactive processing action onto a specified position inside or outside the written document page, or onto a specified position inside or outside the written document page in the user's field of vision, based on the relative positional relationship between the projection or display position of the interactive processing result and the written document page or the image of the written document page.

[0038] In one embodiment, the interaction instruction location information includes the projection or display location. In one embodiment, the processing result of a specific interaction action according to preset rules is projected or displayed at a specific location on the written document page. In one embodiment, the projection or display location is determined based on the relative position of the user's eyes and the written document page or an image of the written document page (e.g., virtual reality VR glasses), ensuring that the interaction processing result is located within or outside the written document page in the user's field of vision. In one embodiment, the relative position of the display screen and the written document page is detected; if the display screen moves, the content displayed on the display screen changes accordingly, ensuring that the processing result is continuously displayed within or outside the written document page. In one embodiment, the projection or display processing result is detected; if the projection or display processing result is not in the specified location, it is adjusted to the specified location based on the deviation. In one embodiment, the display device may be a transparent display screen covering the written document.

[0039] Thus, at the output end, the interactive result data can be projected or displayed at a specified location on the written document page. Based on the interactive method for the position coordinates of a written document page disclosed in this invention, the display of document content, the input of interactive operation commands, and the output of interactive results are realized.

[0040] In some embodiments, performing interactive processing actions on the extracted interactive content data further includes: obtaining the location information of the written document page or electronically mapped document page corresponding to the interactive result data of the interactive processing action according to the interactive instruction location information in the interactive instruction information or a pre-set interactive result data positioning rule; and associating and saving the interactive result data of the interactive processing action with the obtained location information or the document content at the location corresponding to the location information.

[0041] In one embodiment, a pre-defined processing result location rule is established, which includes several interactive processing actions and their associated location information. Different processing actions are associated with different locations. In one embodiment, visual identifiers are used to generate interactive instruction information. Based on the interactive instruction location information in the interactive instruction information, the location information that the processing result of the interactive processing action needs to be associated with is determined. In another embodiment, visual identifiers are printed on the written document page. Based on the visual identifiers, the location that the processing result of a specific interactive processing action needs to be associated with can be determined. For example, a summary of the written document can be associated with the location information at the top of the written document page. In another embodiment, the interactive content data and the corresponding location information on the written document page include the associated stored interactive result data and the location information of the interactive result data. The stored interactive result data can be used as interactive content data and can be called according to the interactive instruction location information. For example, after querying the explanation of a word in a document, the explanation of the word can be associated with the location information of the word and saved. The next time this location is queried, the explanation of the word can be directly retrieved from the saved results. Furthermore, the explanation of the word can also be associated with the word itself and saved. By entering the word, the location and explanation of the word in the document can be retrieved.

[0042] In this way, on the data side, the interaction result data can be associated with and saved on the location information of the written document page or the electronic mapping document page through interactive processing actions, or the interaction result data can be associated with and saved on the document content. The interaction content data can be continuously expanded, and the newly generated interaction content data can also be operated on using the written document page.

[0043] In some embodiments, performing interactive processing actions on the extracted interactive content data further includes: projecting or displaying the interactive content data at a specified location, or reading the interactive content data aloud; and after confirming the interactive content data, performing interactive processing actions on the extracted interactive content data.

[0044] In this way, by confirming the interactive content data before performing the interactive processing action, the accuracy of data processing can be improved, and the precision requirements for the step of obtaining interactive content data can be reduced.

[0045] In some embodiments, performing the interactive processing action on the extracted interactive content data further includes: processing action information or preset rules according to interactive instructions, calling, sharing or copying eligible interactive content data from other electronic mapping document data locally or via a network; and performing interactive processing actions on the called, shared or copied interactive content data.

[0046] Here, eMapped document data can be accessed, shared, or copied, and can be accessed according to criteria such as identifying the pages of a written document, identifying the pages of an eMapped document, time information, identifying eMapped document data, user information, group information, interactive instruction processing actions, and interactive content data classification information.

[0047] In one embodiment, multiple users are grouped together. A user can access the processing results data of other users in their group or share their own processing results data with other users in the group. In another embodiment, if a user has used multiple eMapped Document data sets, all interactive content data from all eMapped Document data used by the user can be provided as a database to the AI ​​model. The AI ​​model then combines this database information to provide interactive processing results data. In one embodiment, eMapped Document data includes not only content data and its corresponding location information, but also information such as information identifying the physical document page, information identifying the eMapped Document page, time information, information identifying the eMapped Document data, user information, group information, interactive instruction processing actions, and interactive content data classification information.

[0048] In this way, on the data side, it is possible to call, share and copy electronically mapped document data. It can not only support one-to-one correspondence between written documents and electronically mapped documents, but also enable collaborative interaction among multiple people, as well as data interaction between multiple electronically mapped document pages or electronically mapped document data.

[0049] In some embodiments, the interactive instruction processing action information includes at least one of the following: processing action information, information about the invoked program or service, parameter information, address information of the invoked program or service, name information of the invoked model, user information, and password information.

[0050] In one embodiment, an artificial intelligence model is invoked to summarize the document content and associate it with the bottom of the written document page. The interactive instruction processing action information includes processing action information, address information of the service invoked, name information of the invoked model, user information, and password information. In one embodiment, a region on the written document page is designated as a communication area. Content written within the communication area is sent to a designated person's email address or instant messaging account, and the returned information is displayed in a designated location. The interactive instruction processing action information includes processing action information, parameter information (e.g., the designated person's account), address information of the service invoked, information of the invoked program, user information, and password information. In one embodiment, different regions of the written document page can be used to activate different programs or services, achieving a desktop-like effect. In one embodiment, parameter information can be characters or screenshots; the parameter information can be obtained by taking a screenshot of the area where the parameter information is input or by performing text recognition on the screenshot. In one embodiment, parameter information is provided in the form of a barcode or QR code.

[0051] In this way, on the operating side, it is possible to call other programs or services.

[0052] In some embodiments, the parameter information includes information added by the user on the written document page. The interaction processing action is determined based on the interaction instruction processing action information in the interaction instruction information. Specifically, this includes: removing the document content from the image of the acquired written document page based on the document content in the e-mapping data, and then obtaining the information added by the user on the written document page from the image as the parameter information.

[0053] In one embodiment, the written document is an exam paper containing a question-and-answer section. When extracting the answer content, the document content (i.e., the question section) can be removed from the image, and the remaining content can be used as the answer content.

[0054] In this way, the information added by the user on the written document page can be obtained more accurately, avoiding the interference of the added information with the original document content, which would affect the extraction of information.

[0055] In some embodiments, electronically mapped document data can be generated by printing out written document pages based on electronically mapped document data, or by photographing written document pages.

[0056] In one embodiment, the document content can be changed during the interaction process, and a new written document can be printed. In one embodiment, the electronically mapped document data includes document content elements and their position information on the written document page. A printed document can be generated based on the document content elements and their position information on the written document page. In another embodiment, the electronically mapped document data includes document content elements and their position information on the written document page. While ensuring the center position of each document content element remains unchanged and sufficient space is maintained between elements, the font size or font can be changed to generate a more aesthetically pleasing printed document. In one embodiment, when photographing the written document page to generate electronically mapped document data, a thresholding method is used to remove background colors and other parts that affect recognition. In yet another embodiment, when photographing the written document page to generate electronically mapped document data, each character or image in the image and its position are identified and stored in the electronically mapped document data as document content elements and their position information on the written document.

[0057] This enables the conversion between electronically mapped document data and printed documents. Firstly, it allows existing electronic or printed documents to quickly interact using the methods disclosed herein. Secondly, it enables the updating and optimization of electronically mapped document data or printed documents.

[0058] In some embodiments, the interaction instruction location information in the interaction instruction information includes the coordinates of one side of the quadrilateral interactive content area and the direction of the opposite side of the quadrilateral interactive content area. Determining the interactive content area based on the interaction instruction location information in the interaction instruction information further includes: obtaining the coordinates of one side of the quadrilateral interactive content area and obtaining the direction of the opposite side of the quadrilateral interactive content area; scanning from one side of the quadrilateral interactive content area towards the opposite side, and defining the area from the first encounter with a pixel row of two or more colors to the end of a pixel row of only one color as the interactive content area; or, based on the position information of each character in the document content, selecting the area containing the row of characters closest to one side of the quadrilateral interactive content area in the direction from one side to the opposite side as the interactive content area.

[0059] In one embodiment, the document content element to be selected is a line of text above the underline. One side of the quadrilateral interactive content area is the underline of this line of text, and the opposite side of the quadrilateral interactive content area is oriented upwards from the underline. The line of text above the underline can be obtained as interactive content data according to the method disclosed in this invention.

[0060] In this way, the complete quadrilateral interactive content area can be obtained by using one side of the area that displays or indicates the extent of the interactive content.

[0061] Unless the order of the steps in this application is explicitly stated or there is a clear logical relationship between them, the order of the steps can be adjusted according to actual needs, and is not limited to the order recorded in the text.

[0062] This disclosure also provides an interactive system, characterized in that the system includes: an interactive instruction information acquisition unit configured to acquire data capable of generating interactive instruction information based on a written document page; and a control unit configured to: generate interactive instruction information based on the data capable of generating interactive instruction information; acquire electronic mapping document data corresponding to the written document page; determine an interactive processing action based on interactive instruction processing action information in the interactive instruction information; extract interactive content data related to the interactive instruction location information from the electronic mapping document data corresponding to the written document page based on the interactive instruction location information in the interactive instruction information; and perform the interactive processing action on the extracted interactive content data.

[0063] In one embodiment, the interactive instruction information acquisition unit can acquire at least the coordinate information of a point on the written document page. In another embodiment, the interactive instruction information acquisition unit further includes other input devices such as buttons.

[0064] Thus, this system can acquire the user's actions on the written document page through the interactive instruction information acquisition unit, and accordingly use the control unit to interactively process the electronically mapped document data.

[0065] In some embodiments, the interactive instruction information acquisition unit includes an imaging device configured to acquire an image with a written document page, wherein one or more visual identifiers are present on the image with the written document page; the interactive system further includes a recognition unit configured to recognize one or more visual identifiers on the image with the written document page; the control unit generates interactive instruction information based on data capable of generating interactive instruction information by: acquiring a coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the written document page, or acquiring a coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the electronically mapped document page; and determining the interactive instruction information based on the visual identifiers and the acquired coordinate correspondence.

[0066] In one embodiment, the visual identifiers can be placed inside or outside the written document page, or they can be written, drawn, or printed on the written document page, and the visual identifiers can cover the content of the written document. In one embodiment, visual identifiers are printed on multiple specific points on the written document page, and the coordinates of the specific points on the written document page are known. After recognizing the visual identifiers, the coordinates of the specific points on the image containing the written document page can be obtained based on the coordinates of the visual identifiers. The coordinate correspondence can be obtained based on the coordinates of the specific points on the written document page and the coordinates of the specific points on the image containing the written document page. In one embodiment, a mobile phone, digital camera, or webcam can be used as the imaging device.

[0067] Thus, on the user side, an imaging device can be used to obtain an image of a written document page. Combined with visual symbols or other input information, data that can generate interactive command information can be obtained. Directly marking visual symbols on the written document page to input operation commands aligns with children's learning habits and is conducive to its promotion among children. At the same time, using widely available mobile phones and digital cameras as interactive command information acquisition units can reduce usage costs.

[0068] In some embodiments, the interactive instruction information acquisition unit includes a coordinate acquisition device configured to acquire coordinate information of the indicated position in a first coordinate system. The coordinate acquisition device has a first coordinate system, which may be a two-dimensional or three-dimensional coordinate system, and the first coordinate system intersects, is parallel to, or coincides with the written document page. The control unit generates interactive instruction information based on data capable of generating interactive instruction information through the following steps: acquiring the coordinate correspondence between the first coordinate system and the coordinate system of the written document page, or acquiring the coordinate correspondence between the first coordinate system and the coordinate system of the electronically mapped document page; determining the interactive instruction information based on the coordinate information of the first coordinate system and the acquired coordinate correspondence.

[0069] In one embodiment, the interactive instruction information acquisition unit further includes other input devices. In one embodiment, the coordinate acquisition device may be a mouse, touchscreen, trackball, touchpad, pointing stick, keyboard, or other device capable of indicating coordinates; it may also be a wireless positioning, laser positioning, 3D scanning, or other positioning and measurement device. In one embodiment, the coordinate acquisition device includes a mouse and a transparent display screen, which covers the written document page. The coordinate system of the transparent display screen is parallel to the coordinate system of the written document page. The coordinates in the first coordinate system are obtained by pointing to a position using the mouse, thereby generating interactive instructions. Furthermore, interactive instructions can be generated using mouse buttons and menus to process action information. In one embodiment, multiple specific points are on the written document page, and the coordinates of these specific points on the written document page are known. After recognizing a visual identifier, indicating the location of a specific point allows the acquisition of its coordinates in the first coordinate system. The coordinate correspondence can be obtained based on the coordinates of the specific point on the written document page and its coordinates in the first coordinate system. In one embodiment, the interactive instruction information is determined based on the movement trajectory of the indicated position. In one embodiment, the coordinate acquisition device is a transparent touchscreen, which is used to cover the written document page. The first coordinate system is the touchscreen's coordinate system. Furthermore, the written document page has multiple specific points; clicking on these points allows the acquisition of the coordinate correspondence between the first coordinate system and the written document page's coordinate system. The touchscreen also has a keyboard input area for inputting the corresponding electronically mapped document page code, thus enabling the complete execution of the interactive method disclosed in this invention without using an imaging device. In one embodiment, the written document page is placed at a fixed position in the first coordinate system, and the coordinate correspondence can be pre-set based on this fixed position. In another embodiment, based on the indicated first coordinate system coordinate information and the coordinate correspondence, the coordinates of the indicated position in the written document page's coordinate system are determined. Then, based on the coordinates of the indicated position in the written document page's coordinate system, the interactive instruction location information or interactive instruction processing action information is determined.

[0070] Thus, on the operating end, coordinate acquisition devices such as mice and touch screens can be used to input operating commands, making it easier for users to understand and accept them.

[0071] In some embodiments, the interactive system further includes a cover over the written document page, wherein the cover can form one or more visual identifiers on the cover by changing local color or transparency, and the interactive system further includes an imaging unit configured to acquire an image of the written document page as electronically mapped document data when the cover or the visual identifiers formed on the cover do not obscure the written document.

[0072] Here, because the interactive content data is obtained from the eMapped Document data, the cover can be transparent, semi-transparent, or opaque. In one embodiment, the cover is a transparent plastic sheet, which can be colored using a highlighter with a transparent effect to form visual identifiers, such as using a yellow highlighter to mark (the marked words can be seen through the plastic sheet) the words to be translated. Alternatively, other non-transparent pigments can be used to change the color to form visual identifiers, such as using yellow pigment to cover (the marked words cannot be seen through the plastic sheet) the words to be translated. In one embodiment, the cover can be a liquid crystal panel, which can form visual identifiers by changing the transparency of the liquid crystal, such as by applying pressure or current to the liquid crystal to change its transparency. In one embodiment, the cover is a touch liquid crystal display panel, which can change the content displayed on the liquid crystal at the touch location to form visual identifiers, such as displaying yellow at the touched location, and further, displaying selected document content elements. In one embodiment, because the written document will not be smudged during use, the imaging unit can acquire an image of the written document page as eMapped Document data before or after the cover or the visual identifier formed on the cover obscures the written document.

[0073] In this way, visual identifiers can be created using coverings. This method allows markings to be made on the coverings of written document pages without smudged the pages. Furthermore, it enables the generation of electronically mapped document data by capturing images of the document without any covering or visual identifiers obstructing the view, resulting in accurate and clear document content. Different sizes of coverings can also be provided, offering users options at varying costs.

[0074] In some embodiments, the interactive system is characterized by further including a display unit configured to project or display content to be displayed at a specified location, and the control unit configured to: obtain the relative positional relationship between the projection or display position of the interactive processing result and the written document page or the image of the written document page according to the interactive instruction position information in the interactive instruction information or a preset rule; and project or display the processing result of the interactive processing action at a specified location inside or outside the written document page, or at a specified location inside or outside the written document page in the user's field of vision, according to the relative positional relationship between the projection or display position of the interactive processing result and the written document page or the image of the written document page.

[0075] Here, the display unit can typically be a projector, a display screen, or smart glasses. In one embodiment, the display unit can be a transparent display screen that overlays the written document page, and the control unit is configured to: acquire the relative positional relationship between the display screen and the written document page (e.g., measure the relative positional relationship between the display screen and the written document page using an imaging device or a positioning measurement device), and display the interaction processing result at a designated location within or outside the written document page based on the relative positional relationship between the transparent display screen and the written document page. In another embodiment, the display unit is smart glasses, which are configured to project or display the interaction processing result at a designated location within or outside the written document page in the user's field of vision, based on the relative positional relationship between the projection or display position of the interaction processing result and the written document page or an image of the written document page.

[0076] In this way, the interactive processing results can be displayed in a designated location using the display unit. The document content is presented in written form, while the interactive results are displayed on the display unit, which can greatly reduce the time users spend looking at the display device. At the same time, transparent displays, projectors, and smart glasses can reduce the harmful blue light from the backlight of ordinary displays, allowing users to conveniently obtain information feedback while protecting their eyesight.

[0077] An electronic device is provided, including a processor and a memory, the memory being coupled to the processor and storing computer instructions therein for performing methods according to embodiments of the present disclosure when executed by the processor.

[0078] In some embodiments, the electronic device further includes a transparent display screen and / or a transparent touch screen.

[0079] Thus, the methods according to embodiments of this disclosure can be performed using electronic devices.

[0080] A non-transitory computer-readable storage medium is provided, having stored computer instructions thereon that, when executed by a processor, perform a method according to embodiments of the present disclosure.

[0081] A computer program product is provided, having computer instructions that, when executed by a processor, perform a method according to embodiments of the present disclosure.

[0082] In addition, the technical solution disclosed herein brings many other advantages, which will be described in detail in the specific embodiments.

[0083] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0084] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of the present invention and are not intended to limit the present invention.

[0085] Figure 1 A schematic diagram illustrating data processing of a written document-based interactive method according to an embodiment of the present disclosure is shown.

[0086] Figure 2 A flowchart of a written document-based interactive method according to an embodiment of the present disclosure is shown.

[0087] Figure 3 A system block diagram of a written document-based interaction method, system, electronic device, and computer-readable medium according to embodiments of the present disclosure is shown.

[0088] Figure 4 This is a schematic diagram comparing a written document page with electronically mapped document data according to an embodiment of the present disclosure.

[0089] Figure 5 This is a comparative diagram of a written document page and an electronically mapped document page according to an embodiment of the present disclosure.

[0090] Figure 6 A schematic diagram is shown of an interactive system 300 according to an embodiment of the present disclosure acquiring interactive content data using an imaging device and visual identifiers.

[0091] Figure 7 This is a schematic diagram of an overlay that forms a visual identifier by changing the transparency of a liquid crystal layer according to an embodiment of the present disclosure.

[0092] Figure 8A A schematic diagram of a document page having multiple visual identifiers according to an embodiment of the present disclosure is shown.

[0093] Figure 8B A schematic diagram is shown illustrating a visual identifier of type selection according to an embodiment of the present disclosure.

[0094] Figure 8C A schematic diagram showing an embodiment of the present disclosure includes a region location identifier and a processing action identifier.

[0095] Figure 8D A schematic diagram illustrating an example of a region location identifier according to an embodiment of the present disclosure.

[0096] Figure 9A An example of an Ebbinghaus forgetting curve according to an embodiment of the present disclosure is shown.

[0097] Figure 9BAn example of Ebbinghaus forgetting curves for users of different age groups according to embodiments of the present disclosure is shown.

[0098] Figure 10A1 and Figure 10A2 A schematic diagram is shown illustrating the review cycle for high-importance document content based on a memory curve, according to an embodiment of the present disclosure.

[0099] Figure 10B A schematic diagram illustrating the review cycles of older adults and younger adults based on memory curves according to embodiments of the present disclosure is shown.

[0100] Figure 11 This is a schematic diagram of an electronic device with a transparent touchscreen according to an embodiment of the present disclosure.

[0101] Figure 12A This is a schematic diagram of an electronic device with a transparent touch display screen according to an embodiment of the present disclosure.

[0102] Figure 12B This is a physical image of an electronic device with a transparent touch display screen according to an embodiment of the present disclosure. Detailed Implementation

[0103] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0104] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0105] Various aspects of this disclosure can utilize written document pages and corresponding electronically mapped document data to jointly realize written document-based interaction. The electronically mapped document data contains or can display the mapping relationship between interactive content data and location information. The function of the written document page is to provide a written document page coordinate system with location information and generate interactive instruction information accordingly, as well as provide visually accessible document content for the user. The function of the electronically mapped document data is to provide interactive digital interactive content data based on location information and record interactive result data when necessary.

[0106] One aspect of this disclosure provides a written document interaction method, characterized in that the method includes: acquiring data capable of generating interactive instruction information based on a written document page; the interactive instruction information includes interactive instruction processing action information and / or interactive instruction location information and / or information for acquiring electronically mapped document data corresponding to the written document page; the interactive instruction location information refers to one or more location information on the written document page or electronically mapped document page related to the interactive instruction information; the electronically mapped document data includes interactive content data corresponding to the written document page; and the location information of the interactive content data on the written document page or electronically mapped document page can be acquired; the interactive content data includes at least one of document content, interactive preset data, interactive process data, and interactive result data; and the document content... Content refers to the visible content on a written document page or the corresponding document content data stored in electronically mapped document data. An electronically mapped document page refers to an electronic document page stored or that can be generated in the electronically mapped document data corresponding to a written document page. The electronically mapped document data may include one or more electronically mapped document pages. The process involves: generating interactive instruction information based on data that can generate interactive instruction information; obtaining the electronically mapped document data corresponding to the written document page; determining interactive processing actions based on the interactive instruction processing action information in the interactive instruction information; extracting interactive content data related to the interactive instruction location information from the electronically mapped document data corresponding to the written document page based on the interactive instruction location information in the interactive instruction information; and performing interactive processing actions on the extracted interactive content data.

[0107] In this way, interactive instruction information for electronically mapped document data can be obtained based on the user's operation on the written document page, interactive content data to be processed can be obtained from the electronically mapped document data, and the processing of interactive content data can be performed according to the interactive instruction information. Thus, the written document page is used to operate on the electronically mapped document data, turning the written document page into a visual editor and digital interactive platform for electronically mapped document data.

[0108] A written document interaction system is provided, characterized by comprising: an interaction instruction information acquisition unit configured to acquire data capable of generating interaction instruction information based on a written document page; and a control unit configured to: generate interaction instruction information based on the data capable of generating interaction instruction information; acquire electronically mapped document data corresponding to the written document page; determine an interaction processing action based on interaction instruction processing action information in the interaction instruction information; extract interaction content data related to the interaction instruction location information from the electronically mapped document data corresponding to the written document page based on the interaction instruction location information in the interaction instruction information; and perform the interaction processing action on the extracted interaction content data.

[0109] An electronic device is provided, including a processor and a memory, the memory being coupled to the processor and storing computer instructions therein for performing methods according to embodiments of the present disclosure when executed by the processor.

[0110] A non-transitory computer-readable storage medium is provided, having stored computer instructions thereon that, when executed by a processor, perform a method according to embodiments of the present disclosure.

[0111] A computer program product is provided, having computer instructions that, when executed by a processor, perform a method according to embodiments of the present disclosure.

[0112] In summary, it can provide convenient digital and intelligent support for users who use written materials for learning and office work, improve the efficiency of teacher-student interaction and team collaboration, enable users to manually create interactive written documents, and enable manual operation to achieve interaction, as well as help users review written materials, thereby protecting users' eyesight, and providing digital and intelligent applications for written materials.

[0113] This application will be further illustrated by the following embodiments.

[0114] Figure 1 A schematic diagram illustrating data processing of a written document-based interactive method according to an embodiment of the present disclosure is shown.

[0115] A common method for processing electronic document data is traditional interactive methods that rely on display devices, such as... Figure 1 As shown, the embodiments of this disclosure provide a novel written document-based interaction method as a useful supplement, which can process electronic map document data using both traditional interaction methods and written document-based interaction methods.

[0116] Figure 2 A flowchart of a written document-based interactive method according to an embodiment of the present disclosure is shown.

[0117] like Figure 2As shown, the interactive method based on written documents according to an embodiment of the present invention specifically includes steps S210-S260:

[0118] Step S210: Obtain data that can generate interactive instruction information from the written document page.

[0119] In step S210, in one embodiment, the data capable of generating interactive instruction information includes an image with a written document page, and the image with the written document page contains one or more visual identifiers. The visual identifiers may be inside or outside the written document page in the image. Generating interactive instruction information based on the data capable of generating interactive instruction information specifically includes: identifying one or more visual identifiers within the image with the written document page; obtaining the coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the written document page, or obtaining the coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the electronically mapped document page; and determining interactive instruction information based on the visual identifiers and the obtained coordinate correspondence.

[0120] In one embodiment, determining the interaction instruction information based on visual identifiers and the coordinate correspondence between the image with the written document page and the written document page specifically includes: in response to one or more visual identifiers including at least one visual identifier of type interaction instruction location information identifier, determining the position of the interaction instruction location information identifier in the image with the written document page; and determining the interaction instruction location information in the interaction instruction information based on the position of the location information identifier in the image with the written document page and the coordinate correspondence between the image with the written document page and the written document page.

[0121] In one embodiment, the data capable of generating interactive instruction information includes one or more first coordinate system coordinate information. The first coordinate system coordinate information is the coordinate information in the indicated first coordinate system, which intersects with, is parallel to, or coincides with the written document page. Generating interactive instruction information based on the data specifically includes: obtaining the coordinate correspondence between the first coordinate system and the written document page coordinate system, or obtaining the coordinate correspondence between the first coordinate system and the electronically mapped document page coordinate system; and determining the interactive instruction information based on the first coordinate system coordinate information and the obtained coordinate correspondence. In one embodiment, obtaining the coordinate correspondence between the first coordinate system and the electronically mapped document page coordinate system includes first obtaining the coordinate correspondence between the first coordinate system and the written document page coordinate system, and then obtaining the coordinate correspondence between the written document page coordinate system and the electronically mapped document page coordinate system, thereby obtaining the coordinate correspondence between the first coordinate system and the electronically mapped document page coordinate system.

[0122] Step S220: Generate interactive instruction information based on the data that can generate interactive instruction information.

[0123] Step S230: Obtain the electronic mapping document data corresponding to the written document page.

[0124] In step S230, in one embodiment, obtaining the electronically mapped document data corresponding to the written document page specifically includes one or more of the following steps: determining the electronically mapped document data corresponding to the written document page by comparing the similarity between the whole or part of the image of the written document page and the whole or part of the electronically mapped document data; determining the electronically mapped document data corresponding to the written document page in response to one or more visual identifiers including at least one visual identifier of the type being a written document identification identifier, and in response to the written document identification identifier being consistent with the identification identifier stored in the electronically mapped document data; determining the information recorded in the written document identification identifier in response to one or more visual identifiers including at least one visual identifier of the type being a written document identification identifier, and in response to the recorded information being consistent with the information in the electronically mapped document data; and determining the electronically mapped document data corresponding to the written document page in response to input information, and in response to the input information being consistent with the information in the electronically mapped document data.

[0125] Step S240: Determine the interaction processing action based on the interaction instruction processing action information in the interaction instruction information.

[0126] In step S240, in one embodiment, the electronic mapping document data stores interactive instruction location information corresponding to one or more interactive processing actions. Determining the interactive processing action based on the interactive instruction processing action information in the interactive instruction information further includes: obtaining the interactive instruction location information from the electronic mapping document data or preset data based on the interactive instruction processing action information in the interactive instruction information.

[0127] Step S250: Based on the interaction instruction location information in the interaction instruction information, extract the interaction content data related to the interaction instruction location information from the electronic mapping document data corresponding to the written document page.

[0128] In step S250, in one embodiment, based on the interaction instruction location information in the interaction instruction information, interactive content data related to the interaction instruction location information is extracted from the electronic mapping document data corresponding to the written document page. Specifically, this includes: determining the interactive content area range based on the interaction instruction location information in the interaction instruction information; obtaining interactive content data from the electronic mapping document data, and obtaining the location information of the interactive content data on the written document page or the electronic mapping document page; and selecting interactive content data whose location information meets the conditions as interactive content data related to the interaction instruction location information based on the location information of the interactive content data on the written document page or the electronic mapping document page and the interactive content area range, wherein the conditions refer to the location information being within the interactive content area range and / or having an intersection with the interactive content area range.

[0129] In one embodiment, the interactive content data includes document content elements or combinations of document content elements, and the electronically mapped document data includes document content elements or combinations of document content elements. The location information of the document content elements or combinations of document content elements on the written document page or the electronically mapped document page can be obtained. A document content element is a unit constituting the content of the written document, and a combination of document content elements is a combination of multiple document content elements. Based on the location information of the interactive content data on the written document page or the electronically mapped document page, and the range of the interactive content area, interactive content data whose location information meets the conditions is selected as the interactive content data related to the location information of the interactive instruction. Specifically, this includes: determining document content elements or combinations of document content elements whose location information meets the conditions based on the range of the interactive content area and the location information of the document content elements or combinations of document content elements on the written document page or the electronically mapped document page; and arranging and combining the document content elements or combinations of document content elements whose location information meets the conditions based on the location information of the document content elements or combinations of document content elements on the written document page or the electronically mapped document page to generate interactive content data.

[0130] In one embodiment, the eMapped Document Data includes an image of an eMapped Document Page corresponding to a written document page, or an electronic document capable of generating an eMapped Document Page corresponding to a written document page. Based on the interaction instruction location information in the interaction instruction information, interactive content data related to the interaction instruction location information is extracted from the eMapped Document Data corresponding to the written document page. Specifically, this includes: determining the corresponding interactive content area range in the eMapped Document Page based on the interaction instruction location information; extracting a portion of the interactive content area range from the eMapped Document Page based on the corresponding interactive content area range; and extracting the document content within the interactive content area range as interactive content data based on the portion extracted from the eMapped Document Page.

[0131] In one embodiment, the interaction instruction location information in the interaction instruction information includes the coordinates of one side of the quadrilateral interactive content area and the direction of the opposite side of the quadrilateral interactive content area. Determining the interactive content area based on the interaction instruction location information in the interaction instruction information further includes: obtaining the coordinates of one side of the quadrilateral interactive content area and obtaining the direction of the opposite side of the quadrilateral interactive content area; scanning from one side of the quadrilateral interactive content area towards the opposite side, and defining the area from the first encounter with a pixel row of two or more colors to the end of a pixel row of only one color as the interactive content area; or, based on the position information of each character in the document content, selecting the area containing the row of characters closest to one side of the quadrilateral interactive content area in the direction from one side to the opposite side as the interactive content area.

[0132] Step S260: Perform interactive processing actions on the extracted interactive content data.

[0133] In step S260, in one embodiment, performing interactive processing actions on the extracted interactive content data further includes: obtaining the relative positional relationship between the projection or display position of the interactive processing result and the written document page or the image of the written document page, based on the interactive instruction position information or preset rules in the interactive instruction information; and projecting or displaying the processing result of the interactive processing action onto a specified position inside or outside the written document page, or onto a specified position inside or outside the written document page in the user's field of vision, based on the relative positional relationship between the projection or display position of the interactive processing result and the written document page or the image of the written document page.

[0134] In one embodiment, performing interactive processing actions on the extracted interactive content data further includes: obtaining the location information of the written document page or electronically mapped document page corresponding to the interactive result data of the interactive processing action based on the interactive instruction location information in the interactive instruction information or a pre-set interactive result data positioning rule; and associating and saving the interactive result data of the interactive processing action with the obtained location information or the document content at the location corresponding to the location information.

[0135] In one embodiment, performing interactive processing actions on the extracted interactive content data further includes: projecting or displaying the interactive content data at a specified location, or reading the interactive content data aloud; and after confirming the interactive content data, performing interactive processing actions on the extracted interactive content data.

[0136] In one embodiment, performing the interactive processing action on the extracted interactive content data further includes: processing action information or preset rules according to interactive instructions, calling, sharing or copying eligible interactive content data from other electronic mapping document data locally or via a network; and performing interactive processing actions on the called, shared or copied interactive content data.

[0137] In one embodiment, the interactive instruction processing action information includes at least one of the following: processing action information, information about the called program or service, parameter information, address information of the called program or service, name information of the called model, user information, and password information.

[0138] In one embodiment, the parameter information includes information added by the user on the written document page. The interaction processing action is determined based on the interaction instruction processing action information in the interaction instruction information. Specifically, this includes: removing the document content from the image of the acquired written document page based on the document content in the electronic mapping data, and then obtaining the information added by the user on the written document page from the image as the parameter information.

[0139] In one embodiment, a written document page can be generated by printing from the electronically mapped document data, or an electronically mapped document data can be generated by photographing the written document page.

[0140] In some embodiments, the method is characterized by including acquiring an image of a written document page, wherein one or more visual identifiers are present on the written document page; identifying one or more visual identifiers on the image of the written document page; determining location information of an interactive content area in the image of the written document page based on the one or more visual identifiers; extracting document content within the interactive content area related to the location information from e-mapping document data corresponding to the written document page, based on the location information of the interactive content area in the image of the written document page, wherein the extracted document content does not contain one or more visual identifiers, and the document content includes at least one of characters, images, or videos; determining an interactive processing action based on the one or more visual identifiers; and performing the interactive processing action on the extracted document content.

[0141] In some embodiments, the processing action identifier includes at least one of the following: processing action information, program invocation information, service invocation address information, model invocation name information, user information, and password information. The action corresponding to the processing action identifier includes at least one of the following: saving, screenshotting, translating, calculating, reading aloud, retrieving relevant information, editing or revising an electronic mapping document, inserting information, sharing or interacting with others on an interactive written document page or electronic mapping document, invoking a service, invoking a program, invoking a model, and invoking an intelligent agent. Thus, various interactive processing actions can be implemented through the processing action identifier. Here, the program, service, model, or intelligent agent can be a program, service, model, or intelligent agent that implements functions such as saving, screenshotting, translating, calculating, reading aloud, and retrieving relevant information, or it can be a program, service, model, or intelligent agent that implements any other function. The scope of processing actions is not limited here.

[0142] In some embodiments, determining a processing action based on one or more visual identifiers includes: in response to one or more visual identifiers including at least one visual identifier indicating a processing action identifier, determining the processing action corresponding to the processing action identifier.

[0143] Note that the same icon can indicate both a processing action icon and other icons. For example, a yellow box can indicate both translation (processing action icon) and an interactive content area (area positioning icon).

[0144] Thus, by using visual identifiers to determine the desired processing action and performing that action on the document content, interactive access to the document content becomes possible through visual identifiers on the written document. Various interactive processing actions can be achieved through these action identifiers. For example, the content of the original state of the written document page can be processed; edits and revisions (such as adding, deleting, or modifying characters, inserting annotations, etc.) can be recorded in the electronically mapped document while preserving the original content; interaction with others can be achieved based on the interactive written document page or electronically mapped document, enabling network collaboration; and other services and programs can be invoked.

[0145] In some embodiments, the method further includes acquiring an image of an eMap document page and storing the image of the eMap document page in a first memory.

[0146] In this way, it is possible to obtain images of the electronic mapping document pages used for comparison.

[0147] In some embodiments, the method may include obtaining the coordinate correspondence between an image of a written document page and an electronically mapped document page; obtaining the mapping position information of the interactive content area on the electronically mapped document page based on the position information and coordinate correspondence of the interactive content area in the image of the written document page; and extracting document content from the electronically mapped document data as the document content within the interactive content area based on the mapping position information of the interactive content area on the electronically mapped document page.

[0148] In this way, the coordinate correspondence between the image of the written document page and the electronically mapped document page can be obtained, and the document content in the interactive content area can be extracted.

[0149] Those skilled in the art will understand that all or part of the steps of the above embodiments are implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, it performs the functions defined by the methods provided in this disclosure. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk.

[0150] Furthermore, it should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0151] Figure 3 A system block diagram of a written document-based interaction method, system, electronic device, and computer-readable medium according to embodiments of the present disclosure is shown.

[0152] like Figure 3 As shown, the interactive system 300 based on written documents includes an interactive instruction information acquisition unit 310 and a control unit 330.

[0153] The interactive instruction information acquisition unit 310 is configured to acquire data that can generate interactive instruction information based on the written document page.

[0154] The control unit 330 is configured to: generate interactive instruction information based on data capable of generating interactive instruction information; acquire electronic mapping document data corresponding to the written document page; determine interactive processing actions based on interactive instruction processing action information in the interactive instruction information; extract interactive content data related to the interactive instruction location information from the electronic mapping document data corresponding to the written document page based on the interactive instruction location information in the interactive instruction information; and perform interactive processing actions on the extracted interactive content data.

[0155] In some embodiments, the interactive instruction information acquisition unit 310 includes an imaging device 312 configured to acquire an image with a written document page, wherein one or more visual identifiers are present on the image with the written document page; the interactive system 300 further includes a recognition unit 320 configured to recognize one or more visual identifiers on the image with the written document page; the control unit 330 generates interactive instruction information based on data capable of generating interactive instruction information by: acquiring a coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the written document page, or acquiring a coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the electronically mapped document page; and determining the interactive instruction information based on the visual identifiers and the acquired coordinate correspondence.

[0156] In some embodiments, the interactive instruction information acquisition unit 310 includes a coordinate acquisition device 314, configured to acquire coordinate information of the indicated position in a first coordinate system. The coordinate acquisition device has a first coordinate system, which is a two-dimensional or three-dimensional coordinate system, and the first coordinate system intersects, is parallel to, or coincides with the written document page. The control unit 330 generates interactive instruction information based on data capable of generating interactive instruction information through the following steps: acquiring the coordinate correspondence between the first coordinate system and the written document page coordinate system, or acquiring the coordinate correspondence between the first coordinate system and the electronic mapping document page coordinate system; and determining the interactive instruction information based on the first coordinate system coordinate information and the acquired coordinate correspondence.

[0157] In some embodiments, the interactive system further includes a cover over the pages of a written document, wherein the cover can form one or more visual identifiers on the cover by changing local color or transparency, and the interactive system further includes an imaging device configured to acquire an image of the pages of the written document as electronically mapped document data when the cover or the visual identifiers formed on the cover do not obscure the written document.

[0158] In some embodiments, a display unit 340 is further included, configured to project or display the content to be displayed at a designated location. The control unit 330 is configured to: obtain the relative positional relationship between the projection or display position of the interaction processing result and the written document page or an image of the written document page, based on the interaction instruction position information or preset rules in the interaction instruction information; and, based on the relative positional relationship between the projection or display position of the interaction processing result and the written document page or an image of the written document page, project or display the processing result of the interaction processing action at a designated location within or outside the written document page, or project or display it at a designated location within or outside the written document page in the user's field of vision. The display unit is typically a projector, a display screen, smart glasses, or other similar device.

[0159] Here, the specified location can be determined based on visual identifiers or coordinate information from the first coordinate system. For example, a visual identifier can be designated as the projection positioning identifier, and a certain area around this identifier can be used as the projection area. The specified information can also be determined based on the location information of the interactive content area, or it can be considered as determined based on visual identifiers.

[0160] The specified location can also be determined based on the information stored in the eMap document. For example, the eMap document data stores the result of a certain processing action and associates it with location information. During projection, the processing result can be projected onto the area corresponding to the associated location information.

[0161] In this way, by displaying document content in written form and projecting or displaying feedback information on a display device, the use of electronic screens can be reduced, the negative effects of blue light from display screens on eyesight can be avoided, and the interactive learning effect of using paper media can be achieved.

[0162] The imaging device 312 can convert optical signals, electromagnetic signals, etc., into processable image data. It can be a camera, scanner, infrared imager, spectral imager, etc. The user can hold the imaging device 312 or fix the imaging device 312 in a certain place (such as a desk lamp on a desk) to take pictures (images) of the pages of a written document (placed on the desktop) to obtain an image of the written document pages.

[0163] The coordinate acquisition device 314 may include pointing devices such as a mouse, trackball, or touchscreen, or coordinate measuring devices capable of measuring two-dimensional or three-dimensional coordinates. Users can hold the coordinate acquisition device 314, fix it in a location (e.g., at a desk lamp), or cover it over a written document (e.g., covering a written document with a transparent touchscreen) to acquire the coordinates of the indicated location based on user actions. In addition to coordinate information, other input information such as key presses and clicks can also be acquired.

[0164] Imaging device 312 and coordinate acquisition device 314 are optional components of interactive command information acquisition unit 310, not mandatory. They can be present simultaneously, or only one of them can be present, or neither can be included. Interactive command information acquisition unit 310 may also include other input devices such as a keyboard capable of inputting coordinate information and other information.

[0165] In some embodiments, the control unit 330 is configured to determine the electronically mapped document data corresponding to a written document page by one or more of the following steps: determining the electronically mapped document page corresponding to the written document page by comparing the similarity between the whole or a part of an image of the written document page and the whole or a part of an electronically mapped document page stored in a memory; determining the electronically mapped document page corresponding to the written document page in response to one or more visual identifiers including at least one visual identifier of the type being a written document identification identifier, and in response to the written document identification identifier being consistent with the identification identifier of an electronically mapped document page stored in a first memory; determining the information recorded in the written document identification identifier in response to one or more visual identifiers including at least one visual identifier of the type being a written document identification identifier, and determining the electronically mapped document page corresponding to the written document page in response to the recorded information being consistent with the information of an electronically mapped document page stored in the first memory; and determining the electronically mapped document data corresponding to the written document page in response to input information, and in response to the input information being consistent with the information of the electronically mapped document data.

[0166] In this way, the corresponding e-mapped document page can be determined through various methods. This information can include identification identifiers, as well as other information such as the specific URL link derived from the identification identifier, the book title, the e-mapped document page code, and so on.

[0167] In some embodiments, the control unit 330 is configured to determine a processing action based on one or more visual identifiers by: in response to one or more visual identifiers including at least one visual identifier indicating a processing action identifier, determining the processing action corresponding to the processing action identifier.

[0168] In this way, by using visual identifiers to determine what processing action needs to be performed and then performing that processing action on the document content, it becomes possible to interact with the document content through visual identifiers on the written document.

[0169] In some embodiments, the processing action information includes at least one of the following: processing action information, parameter information, calling program information, address information of the calling service, name information of the calling model, user information, and password information.

[0170] Here, processing actions include processing document content, such as saving, screenshotting, translating, calculating, reading aloud, retrieving relevant information, editing or revising electronic mapping documents, inserting information such as annotations into documents, and communicating with other people or services, such as sharing or interacting with others, calling services, calling programs, calling models, and calling agents, at least one of these.

[0171] In this way, a wide variety of interactive actions can be achieved based on written documents through interactive instructions.

[0172] In some embodiments, the written document interaction system further includes a second memory configured to store electronically mapped document data, which includes at least one of the following: an image of an electronically mapped document page; an electronic document or multimedia file that has or can generate an electronically mapped document page; data storing the document content of a written document page and its correspondence with location information; identification marks of the written document page and the electronically mapped document page; historical document content that has been or is currently marked by visual identifiers or processed by processing actions; the location of the historical document content; processing actions on the historical document content; the executor of the processing actions on the historical document content; the type of the past or present processing actions; and the processing result of the past or present processing actions. The executor is stored in association with the historical document content and the processing actions on the historical document content.

[0173] Here, the processing results can be associated with the location of visual identifiers on the written document page, creating a sticker-like storage method. This can be achieved by associating the visual identifiers with their corresponding locations in the electronically mapped document and saving the processing results. Recording the type of processing action allows for later categorized retrieval of the processing results. In this way, various types of information can be stored for future retrieval and review.

[0174] In some embodiments, performing interactive processing actions on the extracted document content includes saving the processing result of the interactive processing action in the e-mapped document data and associating it with a specified location on the e-mapped document page. The control unit 330 is further configured to: obtain the location information of the processing result of the interactive processing action based on the location information of the interactive content area or the location information of one or more visual identifiers in the image of the written document page or the location rules of the processing result of the e-mapped document data; and associate the processing result of the interactive processing action with the location information of the processing result of the interactive processing action.

[0175] In this way, the processing results can be associated with and saved in relation to the location of the visual identifiers on the written document page.

[0176] In some embodiments, when multiple historical document contents exist on an image of a written document page: the control unit 330 is further configured to: in response to one or more visual identifiers including at least one visual identifier indicating a selection identifier, determine the processing result of an interactive action of a certain type stored in the e-mapped document data based on the selection identifier, or determine the historical document contents or the processing result of a historical interactive action in an interactive content area related to the location information of the selection identifier stored in the e-mapped document data based on the location information of the selection identifier; and obtain at least one of the determined historical document contents, the processing action on the determined historical document contents, the executor information of the processing action on the determined historical document contents, the processing result of the processing action on the determined historical document contents, and the processing result of the historical interactive action.

[0177] Here, selection markers can be used to select previously marked or processed document content or processing results (with processing results and location information already associated and stored), obtain historical information about the document content, and enable various applications. One can select content or results processed by a specific type of action; for example, projecting all translated content and results. Another can select content or results associated with a specific location; for example, projecting all document content or processing results associated with the location of the selection marker. Yet another can select content or results managed by a specific type of action at a specific location; for example, projecting the translation results associated with the location of the selection marker.

[0178] In this way, selection markers can be used to select previously marked or processed document content or processing results, obtain historical information of the document content, and enable a variety of applications.

[0179] In some embodiments, eMapped document data can be shared among users: Control unit 330 is further configured to: in response to one or more visual identifiers or instructions, acquire data in eMapped document data belonging to other users corresponding to the same written document page, or share data in eMapped document data with other users, or allow other users to acquire data in eMapped document data.

[0180] In this way, electronically mapped document data can be shared among users to achieve collaboration between groups or teaching between teachers and students.

[0181] In some embodiments, the second memory stores multiple e-mapping document data, and the control unit 330 is further configured to: retrieve the multiple e-mapping document data stored in the second memory, and generate feedback information based on the retrieval results. Thus, enhanced retrieval generation can be achieved using the e-mapping document data.

[0182] In some embodiments, the written document interaction system further includes a display screen 340 and / or a speaker, and the control unit 330 is further configured to: formulate a review plan based on information stored in a second memory, wherein the review plan includes a portion or all of the historical document content and / or processing actions related to the historical document content that are periodically or non-periodically or in stages based on the memory curve summarized for the executor and presented or notified on the display screen and / or speaker for review.

[0183] In this way, after students identify and process the document content, they can review the document content regularly, thereby improving learning efficiency and effectiveness.

[0184] In some embodiments, the control unit 330 is further configured to: in response to one or more visual identifiers including at least one visual identifier of type as a region positioning identifier and / or a processing action identifier and / or a user identification identifier, assign importance levels to historical document content marked by visual identifiers or processed by processing actions according to different visual features of the region positioning identifiers and / or processing action identifiers, wherein different visual features correspond to different importance levels; for historical document content assigned different importance levels, for different executors represented by user identification identifiers, summarize in stages with different periods or different memory curves or different understanding curves or combinations thereof, and present on a display screen and / or speaker a portion or all of the historical document content and / or processing actions of the historical document content associated with the executor for review.

[0185] In this way, review materials of different importance can be distinguished according to their importance level, and review plans can be formulated for review materials of different importance and / or different implementers (users), which can ensure that review time is allocated reasonably and that more important materials are reviewed.

[0186] In some embodiments, the written document interaction system further includes a timer configured to acquire time information, wherein the importance level of the historical document content is N, where N is a positive integer, and the larger the value of N, the higher the importance; the control unit is further configured to set the period t for reviewing the historical document content via the timer. i , where t i It is the number of hours between the i-th review and the (i-1)-th review, where i is a positive integer. Where c, d, and e are constants, and k is the basic review interval (in hours).

[0187] In this way, a study plan can be developed using a formula that specifies the review intervals based on the importance level.

[0188] In some embodiments, one or more visual identifiers include region positioning identifiers and processing action identifiers, and the control unit 330 is configured to: combine both the region positioning identifier and the processing action identifier to perform a processing action determined based on the processing action identifier on the document content extracted based on the region positioning identifier; or perform a uniform processing action determined based on the processing action identifier on all document content on the image of the written document page.

[0189] In this way, processing actions can be combined with specific document content or unified page processing actions to more flexibly customize the corresponding processing actions for the document content.

[0190] In some embodiments, one or more visual identifiers include at least one type of visual identifier as a region positioning identifier. The region positioning identifier includes a rectangular outline or filled area of ​​a rectangular region, or four corners or two corners, or the top and bottom sides or left and right sides or any side of a rectangular region to define the document content covered by the rectangular region; or includes the center and radius of a circular region, or a circular outline or filled area of ​​a circular region to define the document content covered by the circular region; or an outline or filled area of ​​any shape to define the document content covered by an area of ​​any shape.

[0191] In this way, the areas of document content can be determined based on visual identifiers.

[0192] In some embodiments, when the region positioning identifier includes the bottom edge of a rectangular region, the control unit 330 is configured to extract document content by scanning to one side of the bottom edge according to the length of the bottom edge, and finding the document content closest to the bottom edge as the document content.

[0193] In this way, the document content can be identified by a visual identifier with only one side.

[0194] In some embodiments, the document content closest to the bottom edge includes: scanning pixel by pixel towards one side of the bottom edge, and using the content of the region from the first row where two or more colors of pixels are encountered to the end of the row where only one color of pixels is encountered as the document content; or obtaining the characters in the region where the closest character to the bottom edge is located by optical character recognition (OCR) based on the position of the bottom edge as the document content.

[0195] In this way, we can obtain the visual identifier of the most recent document content with only one edge.

[0196] In some embodiments, the electronically mapped document data includes the position information of each character, image, or video on the written document page or the position information in the electronically mapped document page, or rules for calculating the position information. The control unit 330 is configured to: determine the characters, images, or videos in the interactive content area based on the position information of the interactive content area on the written document page or the mapped position information on the electronically mapped document page, and the position information on the written document page or the position information in the electronically mapped document page, or the rules for calculating the position information; and arrange the characters, images, or videos in the interactive content area based on the position information of each character, image, or video in the interactive content area on the written document page or the position information in the electronically mapped document page to generate document content in the interactive content area.

[0197] In this way, document content can be generated within the interactive content area.

[0198] The systems or units described herein can be implemented by software, hardware, or a combination of both. Hardware implementations may include, for example, integrated circuit chips, while software implementations may include computer program code, firmware, etc. The names of the systems or units do not limit the scope of the system or unit.

[0199] In the embodiments of this disclosure, a written document can be a document presented on paper or other media that is not electronically stored or displayed. Its content includes text, graphics, images, etc., on the written document. Alternatively, a written document page can be a paper-like page presented using electronic ink or other methods. It can also include documents displayed on various objects (including documents displayed on other display devices). Its essential characteristics are: first, it exists in the user's field of vision as a non-editable document image; second, it has matching electronically mapped document data. A written document includes one or more written document pages, and each written document page includes written document content. The written document content is composed of one or more document content elements, which are the units constituting the written document content.

[0200] A written document page with one or more visual identifiers can be obtained by adding one or more visual identifiers to the page, such as by highlighting the page with a highlighter.

[0201] The electronically mapped document data includes, or can reflect, interactive content data and its corresponding location information within the written document page. The interactive content data includes at least one of the following: written document content, preset interactive data, interactive process data, and interactive result data. Written document content refers to the content visible on the written document page or the content data corresponding to the visible content stored in the electronically mapped document data. The electronically mapped document data includes one or more electronically mapped document pages. These electronically mapped document pages and their corresponding written document pages have the same document content and distribution; that is, selecting one or more document content elements on the written document page will yield the same document content elements within the corresponding selected area on the electronically mapped document page. Without affecting the document content and distribution, the font, background color, background image, etc., of the electronically mapped document page can differ from those of the written document page, and multimedia data can also be added. In one embodiment, the document content corresponding to an image on the written document page in the electronically mapped document page can be a video related to the image. The interactive content data in the electronically mapped document data can include not only document content data but also interactive process data and interactive result data, such as translations and annotations of document content, or communication information with other users. Electronic mapping document data can typically be image data, rich text data, or data containing interactive content and its location information within the written document page. Electronic mapping document data can exist in only one form or in multiple forms. In one embodiment, the written document content exists as an image in the electronic mapping document data, while the interactive process data and interactive result data exist in the form of a data table.

[0202] In some embodiments, the electronically mapped document page is an original image, electronic document, or multimedia file of a written document page, or an image, electronic document, or multimedia file of a written document page that has been digitally modified, or an image, electronic document, or multimedia file of a written document page with at least one of the following elements removed: background color, background texture, or italic text that affects the accuracy of text recognition.

[0203] In this way, a suitable electronic mapping document page can be obtained.

[0204] Figure 4 This is a schematic diagram comparing a written document page with electronically mapped document data according to an embodiment of the present disclosure.

[0205] like Figure 4As shown, the content within the white rectangular box is the written document page, and the grey rectangular box and grey table are electronic mapping document data. The grey rectangular box is the electronic mapping document data in image form, and the grey table is the electronic mapping document data in data table form. Within the white rectangular box and the grey rectangular box, the text "The sun along the mountain bows; The Yellow River seawards flows. You can enjoy a great sight; By climbing to a greater height." is the content of the written document. The rounded rectangular area with a dark background is the display area. The display area can be directly set on the written document page according to coordinates, or marked on the written document page using visual identifiers (including directly printing the visual identifiers on the written document page). After reading or identifying the display area, the coordinates and the displayed content of the display area can be recorded in the electronic mapping document data. The information to be displayed (such as messages for communicating with others) can be projected or displayed within this area. The grey table is an example of using a data table as the electronic mapping document data. The data table includes "location information", "interaction content data", and "type". The data in the table corresponds to the written document page within the white rectangular box. The characters "白" and "日" are both content of the written document. The location information fills in the four corner coordinates of the rectangular area occupied by each character. Due to space limitations, the data of other content of the written document is not shown. "white" is the English translation of the character "白", which is the interaction result data obtained when previously translating the character "白" into English. The location information fills in the four corner coordinates of the rectangular area occupied by the character "白". In the future, when requesting a translation for the character "白", "white" can be directly provided as feedback. The last row is the data of the display area. The location information fills in the four corner coordinates of the display area, and the interaction content data is the currently displayed content "empty", indicating that there is no message to be displayed.

[0206] In one embodiment, the image of the electronic mapping document page can be an image of the written document page or an electronic document page, or an image or electronic document page that is consistent with the document content and the distribution of the document content of the written document page but has other differences. For example, in the image of the electronic mapping document page, there may be no decorative elements such as background color or special effects, or fonts that are difficult to recognize may not be used. In another embodiment, the position range occupied by any document content element within the electronic mapping document page is equal to or included in the position range it occupies within the written document page. That is, after selecting any document content element within the electronic mapping document page and mapping the selected interaction content area range to the electronic mapping document page, the mapped interaction content area range only contains the selected document content element.

[0207] Figure 5 It is a comparison schematic diagram of the written document page and the electronic mapping document page according to an embodiment of the present disclosure.

[0208] Such as Figure 5As shown, page one of the e-mapped document is completely identical to the printed document page. Page two of the e-mapped document uses a font that is easier to read than the printed document page. Compared to the printed document page, page three of the e-mapped document has the same center point distribution for each character, but it uses a smaller font that is easier to read.

[0209] The written document content and its distribution in the electronically mapped document data are consistent with the corresponding written document pages, but other elements that do not affect the accurate acquisition of document content may be inconsistent.

[0210] Figure 6 A schematic diagram is shown illustrating how an interactive system 300, according to an embodiment of the present disclosure, acquires interactive content data using an imaging device and visual identifiers. The following is in conjunction with... Figure 3 and Figure 6 The operation of the written document interaction system 300 according to embodiments of the present disclosure will be described.

[0211] In step S610, one or more visual identifiers can be selected to mark the interactive content data to be processed on the written document page according to the desired interactive processing action.

[0212] In some embodiments, one or more visual identifiers may include at least one of hand-drawn identifiers, stamped identifiers, pasted identifiers, placed physical identifiers, and identifiers printed using a micro-inkjet printer or handheld printer. Furthermore, one or more visual identifiers may be formed by altering the background color through applying, placing, or pasting at least one of translucent or transparent substances, by using different colors or line types to delineate closed lines or specific graphics, or by marking shape feature points of the processed area. Thus, a wide variety of visual identifiers can be created.

[0213] In a common and specific application, visual identity can include markings that users manually make on a written document page using tools such as highlighters. For example, a user might manually highlight a sentence or paragraph to indicate that they want it translated into English. Another example is that a user might manually underline or circle a sentence or paragraph with a red pen or other colored pen, or make marks at the four corners of a paragraph to indicate that they believe the content needs to be read aloud. Users can also create logos by stamping, affixing stickers, or even printing them using a micro-inkjet printer or handheld printer. Many other examples of visual identity applications exist and should be included in this disclosure, but will not be described in detail here.

[0214] like Figure 6 As shown, the visual identity is a gray oval logo overlaid on the letters "1, XX".

[0215] Different or different types of visual identifiers, or different elements of the same visual identifier, can represent different meanings. For example, a paragraph highlighted with a highlighter can be a region marker, used to mark an interactive content area, while a stamp-like visual identifier can indicate an interactive action to be performed on that area, such as translating document content (e.g., sentences or paragraphs composed of Chinese characters). However, different elements of the same visual identifier can also represent different meanings. For example, ... Figure 2 As shown, a gray oval icon can indicate that the area covered by the gray oval is an interactive content area, where the document content "1, XX" is the document content to be processed. The oval itself can also represent reading the document content aloud, i.e., playing the pronunciation of "1, XX". Alternatively, in another example, a gray square icon can indicate that the area covered by the gray square is an interactive content area, where the document content "1, XX" is the document content to be processed. The square itself can also represent translating the document content, i.e., translating "1, XX" into English or other languages. Examples are not exhaustive. The following will further describe different types of visual icons in detail with specific embodiments.

[0216] In step S620, the imaging device (312) can be configured to acquire an image with a written document page, wherein one or more visual identifiers are present on the written document page.

[0217] An image with a written document page typically includes the written document page and content beyond it. When the imaging device (312) is used simultaneously with the coordinate acquisition device 314, the image with the written document page may also lack one or more visual identifiers. Only the specific method of interaction using the imaging device and visual identifiers will be described here.

[0218] In step S630, the recognition unit 320 can be configured to recognize one or more visual identifiers on an image containing a written document page. This step can, of course, be performed at other times, and its order of execution with other steps is not restricted. The recognition unit 320 can employ image recognition algorithms, including traditional recognition methods based on image features and recognition methods based on deep learning. Traditional recognition methods include image recognition using edge detection (including Sobel, Canny, and Prewitt operators), feature extraction (including corner detection, texture analysis, and shape descriptors), and feature matching (nearest neighbor matching, Euclidean distance matching, and K-Nearest Neighbors (KNN) matching). Deep learning methods can use visual identifier samples to train an artificial intelligence model, thereby enabling the model to recognize visual identifiers.

[0219] Assuming in Figure 6 In the example, the recognition unit 320 identifies the visual identifier as a gray elliptical identifier located at a certain position. Here, a certain position can be represented by a coordinate (or a set of coordinates) on the written document page. The coordinate system can be an XY coordinate system formed with the bottom left vertex of the written document page as the origin (0,0), the shorter side of the written document page as the x-axis, and the longer side of the written document page as the y-axis. In one embodiment, the coordinate information is relative position information on the page. For example, the x-axis information = distance from the left side of the page or number of pixels / page width, and the y-axis information = distance from the top of the page or number of pixels / page length. In this way, the specific coordinates of the written document page or the e-mapping document page can be obtained by inputting the page width or page length.

[0220] In step S640, the control unit 330 can be configured to acquire electronically mapped document data corresponding to the written document page.

[0221] In step S650, the control unit 330 can be configured to: acquire the coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the written document page, or acquire the coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the electronically mapped document page. The control unit 330 can be configured to: determine interactive instruction information based on the visual identifier and the acquired coordinate correspondence. The control unit 330 can be configured to: determine the interactive content area range based on the interactive instruction location information in the interactive instruction information. The control unit 330 can be configured to: acquire interactive content data and the corresponding location information of the interactive content data on the written document page from the electronically mapped document data. The control unit 330 can be configured to: extract a portion of the interactive content area from the electronically mapped document page based on the corresponding interactive content area range on the electronically mapped document page.

[0222] Because images featuring pages of written documents often also include other content besides the written document page, the coordinate system of the image featuring the written document page usually does not coincide with the coordinate system of the written document page. For example... Figure 6As shown, the image with the written document page is a rectangle with a light blue background. The coordinate system of the image with the written document page can be an XY coordinate system formed by using the bottom left vertex of the light blue rectangle as the origin (0,0), the shorter side of the light blue rectangle as the x-axis, and the longer side of the light blue rectangle as the y-axis. The written document page is a white rectangle with text. The coordinate system of the written document page can be an XY coordinate system formed by using the bottom left vertex of the written document page as the origin (0,0), the shorter side of the written document page as the x-axis, and the longer side of the written document page as the y-axis. The coordinate system of the written document page can be on the actual written document (in terms of length or scale) or on the image of the written document page (in terms of pixels or scale). Figure 6 For example, the origin (0,0) of the coordinate system of the written document page might have coordinates (10,200) in the coordinate system of the image containing the written document page. The coordinate correspondence between the coordinate system of the image containing the written document page and the coordinate system of the written document page refers to the correspondence of the coordinates of the same point in the two coordinate systems.

[0223] In some embodiments, an electronically mapped document page refers to an electronic document page whose content and distribution are consistent with a written document page. The content of a specific area within the document can be obtained based on location information through optical character recognition (OCR) or other methods such as reading file data. The electronic document can be in the form of an image, a text file format such as Word or TXT capable of generating a page, or other data formats. It may also include content that facilitates digital interaction; for example, an electronic document can be a multimedia file. Figure 6 As shown, the printed document page contains the format "1, XX 2, XX 3, XX". The electronic mapping document page also contains the format "1, XX 2, XX 3, XX" in the same location.

[0224] like Figure 6 As shown, when the eMap document page is an image, the "1, XXX" in the ellipse are portions of the interactive content area extracted from the eMap document page based on the corresponding interactive content area.

[0225] In step S660, the control unit 330 can be configured to determine the interaction processing action based on the interaction instruction processing action information in the interaction instruction information.

[0226] The control unit 330 can be configured to extract interactive content data related to the interactive instruction location information from the electronic mapping document data corresponding to the written document page, based on the interactive instruction location information in the interactive instruction information.

[0227] The control unit 330 can be configured to extract document content within the interactive content area as interactive content data, based on a portion of the interactive content area captured from the e-mapping document page.

[0228] The control unit 330 can be configured to perform interactive processing actions on the extracted interactive content data.

[0229] In one embodiment, the extracted interactive content data can be a screenshot, or it can be characters, images, tables, or other content extracted from an image using technologies such as OCR and image recognition.

[0230] For example, such as Figure 6 As shown above, the gray oval icon indicates that the area covered by the gray oval icon is the interactive content area. The document content "1, XX" in the interactive content area is the document content to be processed. The oval itself can also indicate that the document content is read aloud, that is, the pronunciation of "1, XX" is played. Therefore, the user can hear the pronunciation of "1, XX", thereby realizing the interaction between the user and the written document.

[0231] In this way, the document content to be processed can be obtained based on visual identifiers and the e-mapped document page, and the processing of the document content can be performed based on the visual identifiers. This allows for manipulation of e-mapped document data using visual identifiers and the written document page, achieving an effect similar to using a mouse and monitor. Even if the visual identifiers may cover or obscure the document content, the original document content to be processed can still be processed and interacted with using the e-mapped document page, corresponding to the visual identifiers. Based on the position information of the interactive content area within the image of the written document page, the document content within the interactive content area related to the position information is extracted from the e-mapped document data corresponding to the written document page.

[0232] The electronically mapped document data includes the document content in the written document page, and can extract the document content that is consistent with the interactive content area in the written document page based on the position information of any interactive content area in the written document page. The extracted document content does not contain one or more visual identifiers, and the document content includes at least one of characters, images, or videos.

[0233] Electronic mapping document data can be images or text files, or any data format or file that can extract document content based on the location information of interactive content areas on a written document page, such as data, databases, and datasets.

[0234] In some embodiments, the eMapped Document Data may include an image of an eMapped Document Page corresponding to a written document page, or an electronic document capable of generating an eMapped Document Page corresponding to a written document page.

[0235] The control unit 330 can be configured to: determine the corresponding interactive content area range in the electronic mapping document page based on the interactive instruction location information in the interactive instruction information; extract a portion of the interactive content area range from the electronic mapping document page based on the corresponding interactive content area range in the electronic mapping document page; and extract the document content in the interactive content area range as interactive content data based on the portion extracted from the electronic mapping document page.

[0236] Two types of e-mapped documents (images of e-mapped document pages and text documents that can generate e-mapped document pages) can extract document content within the interactive content area by obtaining the coordinate correspondence between the coordinate system of the written document page and the coordinate system of the e-mapped document page, and by cropping a portion of the interactive content area on the e-mapped document page.

[0237] In this way, a large number of existing image files or rich text files (such as Word, WPS, and other electronic files) can be fully utilized to print out written document pages and interact with them using the written document-based interactive method disclosed in this publication.

[0238] In some embodiments, electronically mapped document data can be any data format or file, such as data, database, or dataset, that can extract document content based on the location information of the interactive content area on the written document page.

[0239] Therefore, the control unit 330 can extract interactive content data related to the interactive instruction location information from the electronically mapped document data corresponding to the written document page by the following steps: determining the interactive content area range based on the interactive instruction location information in the interactive instruction information; obtaining interactive content data from the electronically mapped document data, and obtaining the location information of the interactive content data on the written document page or the electronically mapped document page; selecting interactive content data whose location information meets the condition as interactive content data related to the interactive instruction location information based on the location information of the interactive content data on the written document page or the electronically mapped document page, and the interactive content area range, wherein the condition refers to the location information being within the interactive content area range and / or having an intersection with the interactive content area range.

[0240] like Figure 6As shown, although the image of the e-mapped document page has the same document content and content distribution as the printed document page, their coordinate systems may be different. For example, the coordinate system of the e-mapped document page image may be in pixels, while the coordinate system of the printed document page on the actual printed document may be in length. Even the coordinate system of the printed document page in the printed document image may differ due to the difference in the number of pixels between the printed document image and the e-mapped document page image. For example, both may have the bottom left corner of the printed document page as the origin (0,0), but the coordinates of the bottom right corner of the printed document page in the printed document page coordinate system may be (0,1000) or (0,1500). If the scale is used as the parameter of the coordinate system, the page width and page length of both the printed document image and the e-mapped document page image need to be provided to describe the coordinate correspondence.

[0241] Coordinate correspondence refers to the relationship between the coordinates of a point on a written document page in the coordinate system of the written document page and the coordinates of a point on an e-mapped document page. For example, the coordinates (10, 10) on the written document page correspond to the coordinates (20, 20) on the e-mapped document page, because the coordinates of the same text may differ depending on whether the image of the written document page or the e-mapped document page was captured or scanned. However, by using the coordinate correspondence between the image of the written document page and the e-mapped document page, the corresponding coordinates of one or more visual identifiers (if the visual identifiers include area positioning identifiers used to locate the area where the document content to be processed is located) on the written document page (or the image of the written document page) can be obtained, thereby extracting the document content in the location-related interactive content area on the e-mapped document page. Here, the location can be represented by a single point or a set of multiple points. For example, for a piece of text covered by a highlighter, its coordinates can be the starting point of the highlighter in the text or a set of coordinates of the entire coverage area of ​​the highlighter.

[0242] In other words, in some embodiments, the control unit 330 may be configured to determine the location information of the interactive content area range in the image of the written document page based on one or more visual identifiers, and extract the document content within the location-related interactive content area from the electronically mapped document data corresponding to the written document page based on the location information of the interactive content area range in the written document page: in response to one or more visual identifiers including at least one visual identifier of type region positioning identifier, determining the location of the region positioning identifier in the written document page and the location-related interactive content area; determining the electronically mapped document page corresponding to the written document page; and obtaining the document content corresponding to the interactive content area in the electronically mapped document page based on the location of the region positioning identifier in the written document page, the location-related interactive content area, and the corresponding electronically mapped document page.

[0243] Here, the area positioning identifier can be an identifier used to locate the area (interactive content area range) where the document content to be processed is located, such as an identifier drawn with a highlighter.

[0244] In some embodiments, the region positioning identifier includes a rectangular outline or filled area of ​​a rectangular region, or four corners or two corners, or the top and bottom sides, or the left and right sides, or any side of a rectangular region to define the document content covered by the rectangular region; or it includes the center and radius of a circular region, or the circular outline or filled area of ​​a circular region to define the document content covered by the circular region; or it includes an outline or filled area of ​​any shape to define the document content covered by an area of ​​any shape. Here, the circle includes ellipses and perfect circles. Of course, the shape of the region positioning identifier is not limited to these. Thus, the interactive content area can be determined based on the visual identifier, and then the document content within the interactive content area can be determined.

[0245] In some embodiments, when the region positioning identifier includes the bottom edge of a rectangular region, the control unit 330 can be configured to extract document content by scanning along the length of the bottom edge to find the document content closest to the bottom edge. For example, if the region positioning identifier is a red underline, the document content can be obtained by scanning along the length of the red underline to find the document content closest to the red underline. Thus, document content can be determined using a visual identifier with only one edge.

[0246] In some embodiments, obtaining the document content closest to the bottom edge includes: the interaction instruction location information in the interaction instruction information includes the coordinates of one side of the quadrilateral interactive content area and the direction of the opposite side of the quadrilateral interactive content area; determining the interactive content area based on the interaction instruction location information in the interaction instruction information further includes: obtaining the coordinates of one side of the quadrilateral interactive content area and obtaining the direction of the opposite side of the quadrilateral interactive content area; scanning from one side of the quadrilateral interactive content area towards the opposite side, the area from the first encounter with a row of pixels of two or more colors to the end of a row of pixels of only one color is taken as the interactive content area; or, based on the position information of each character in the document content, selecting the area containing the row of characters closest to one side of the quadrilateral interactive content area in the direction from one side to the opposite side as the interactive content area. For example, scanning pixel by pixel upwards from the red underline, from the first encounter with a row of black and white pixels (e.g., black text printing and a white background) to the end of a row of white pixels, these areas can be considered as the areas where the text is located, and therefore are the document content to be marked by the red underline. In this way, the document content that a visual identifier with only one edge is expected to indicate can be obtained.

[0247] In step S650, the control unit 330 obtains interactive content data and the corresponding position information of the interactive content data on the written document page from the electronically mapped document data. This avoids visual markers on the current written document page obscuring or damaging the document content, thus preventing the correct document content from being extracted. For example, a red pen underline may connect to or intersect with the document content (text), causing subsequent extraction of document content with red pen underlines to fail to perform correct OCR recognition or generate other errors.

[0248] In summary, based on the type and location of the visual identifiers, the document content identified by the visual identifiers can be obtained by mapping them to electronic document data. This avoids the situation where visual identifiers on the current printed document page obscure or damage the document content, preventing the extraction of the correct document content.

[0249] In addition to obtaining the correct document content based on electronically mapped document pages, other data formats can also be used, such as directly establishing a mapping relationship between location information and document content, so that the document content can be obtained directly by inputting the location information.

[0250] In some embodiments, one or more visual identifiers include document positioning identifiers. The control unit 330 can obtain the coordinate correspondence between the image of the written document page and the electronically mapped document page by establishing the coordinate correspondence between the image of the written document page and the electronically mapped document page based on the document positioning identifiers. In one embodiment, the document positioning identifiers can be printed at specific locations on the written document page, such as the four corners of the page.

[0251] Thus, by utilizing the positioning markers in written documents, it is beneficial to compare the position of the document image with the original document image or electronic document to obtain the coordinate correspondence between the image of the written document page and the electronically mapped document page. This provides convenient digital and intelligent support for users who use written materials for learning and work, and allows for the manual creation of interactive written documents and manual operation to achieve interaction, thereby protecting users' eyesight and providing digital and intelligent applications of written materials.

[0252] In some embodiments, the imaging unit 312 may be configured to acquire an image of an eMapped document page; wherein the interactive system further includes a first memory configured to store the image of the eMapped document page. For example, the imaging unit 312 may be used to photograph, scan, or otherwise acquire an image of the eMapped document page. Other forms of eMapped document pages (e.g., rich text format) may also be acquired and stored.

[0253] In this way, it becomes possible to acquire and store electronically mapped document pages.

[0254] In some embodiments, the control unit 330 may be configured to determine the electronically mapped document page corresponding to the written document page by one or more of the following steps: determining the electronically mapped document page corresponding to the written document page by comparing the similarity between the whole or part of the image of the written document page and the whole or part of the electronically mapped document page stored in the memory.

[0255] In some embodiments, the control unit 330 may be configured to determine the electronically mapped document page corresponding to the written document page by one or more of the following steps: in response to one or more visual identifiers including at least one visual identifier of the type of a written document identification identifier, and in response to the written document identification identifier being consistent with the identification identifier of the electronically mapped document page stored in the electronically mapped document data, the control unit 330 may determine the electronically mapped document page corresponding to the written document page.

[0256] For example, the identification mark for a written document can be a barcode, QR code, string, etc., and the identification mark for an electronically mapped document page can also be a barcode, QR code, string, etc. The electronically mapped document page corresponding to the written document page is determined by comparing whether the two are consistent.

[0257] In some embodiments, the control unit 330 may be configured to determine the electronically mapped document page corresponding to the written document page by one or more of the following steps: in response to one or more visual identifiers including at least one visual identifier of the type of written document identification identifier, determining the information recorded in the written document identification identifier, and in response to the recorded information being consistent with the information of the electronically mapped document page stored in the electronically mapped document data, determining the electronically mapped document page corresponding to the written document page.

[0258] For example, written document identification identifiers can be barcodes, QR codes, strings, etc., which can record information such as serial number, book title and page number, and URL links. Information about the electronically mapped document page can be obtained using the written document identification identifier on the written document page, and then the corresponding electronically mapped document page can be retrieved based on this information. For instance, if the barcode on the written document page indicates page 22 of the book title "Elementary School Chinese, Book 1", then the electronically mapped document page of page 22 of "Elementary School Chinese, Book 1" (which has been photographed, scanned, saved, etc.) can be retrieved based on the book title "Elementary School Chinese, Book 1, Book 22".

[0259] In this way, the corresponding electronic mapping document page can be determined through various methods.

[0260] In some embodiments, the written document interaction system may further include a cover over the written document page, wherein the cover can form one or more visual identifiers on the cover by changing local color or transparency, wherein the imaging unit is configured to acquire an image of the written document page as an electronically mapped document page when the cover or the visual identifiers formed on the cover do not obscure the written document.

[0261] Here, the covering material can be transparent, translucent, or opaque, such as film, plastic, resin, paper, or LCD panel. Thus, when using this method, markings can be made on the covering material of the written document page, and location information and interactive processing actions can be determined based on visual identifiers. Interactive content data can then be obtained from the electronically mapped document data. In one embodiment, the covering material is a transparent touch screen, on which visual identifiers can be displayed by touch. This avoids contaminating the written document page while allowing for convenient interactive processing using annotations, resulting in accurate and clear document content.

[0262] Figure 7This is a schematic diagram of an overlay that forms a visual identifier by changing the transparency of a liquid crystal layer according to an embodiment of the present disclosure.

[0263] like Figure 7 As shown in the figure, the covering includes at least a liquid crystal layer, a conductive layer, and a control circuit, and may also include a transparent protective film. The liquid crystal layer can change its transparency to form a visual identifier. For example, the liquid crystal layer is initially transparent, and after pressure is applied, the pressed part becomes opaque to form a visual identifier. The control circuit can restore the transparency of the liquid crystal layer.

[0264] Interactive actions are handled in this way. By using this cover, visual markers can be easily and quickly created or cleared by pressing the cover or control circuitry, thus improving the user experience.

[0265] In some embodiments, the control unit 330 may also be configured to: after removing the covering, project or display one or more visual identifiers onto the current position of the written document page using a display device, based on the position of one or more visual identifiers on the image of the written document page, so that one or more visual identifiers are presented on the current written document page.

[0266] In this way, after removing the covering and the visual signs on the covering, the effect of the visual signs can be restored using a projector, reducing the use of electronic screens and achieving the effect of interactive learning using paper media.

[0267] In some embodiments, the control unit 330 may be configured to determine a processing action based on one or more visual identifiers by: in response to one or more visual identifiers including at least one visual identifier indicating a processing action identifier, determining the processing action corresponding to the processing action identifier.

[0268] Figure 8A A schematic diagram of a document page with multiple visual identifiers according to an embodiment of the present disclosure is shown. The visual identifiers can be applied to text or images, and different visual identifiers can correspond to different processing actions.

[0269] like Figure 8A As shown, visual identifiers include gray boxes, dashed boxes, and rounded-corner boxes for text, and elliptical boxes for images. For example, a gray box could correspond to reading aloud, a dashed box to retrieving relevant information, a rounded-corner box to translation, and an elliptical box to screenshots, etc. No restrictions are imposed here.

[0270] Figure 8B A schematic diagram is shown illustrating a visual identifier of type selection according to an embodiment of the present disclosure.

[0271] like Figure 8BAs shown, by selecting an identifier, the historical document content in the interaction content area related to the position in the image of the written document page can be determined. For example, for "The sun along the mountain bows", the processing actions for "The sun along the mountain bows", the executor information of the processing actions for the determined historical document content, the processing results of the processing actions for the determined historical document content, etc. can be obtained. For these pieces of information, many applications can be implemented. For example, if "The sun along the mountain bows" has been read aloud, then in response to the selection identifier for "The sun along the mountain bows", it can be read aloud again, or the executor information of those who have processed "The sun along the mountain bows" can be collected to collect user learning information, etc. In some examples, the executor information can also be used for group learning and review collaboration, etc.

[0272] In this way, the selection identifier can be used to select the document content that has been marked or processed before, obtain the historical information of this document content, and implement many applications.

[0273] In some embodiments, one or more visual identifiers include area positioning identifiers and processing action identifiers. The control unit 330 can be configured to: combine both the area positioning identifier and the processing action identifier to perform the processing action determined based on the processing action identifier on the document content extracted based on the area positioning identifier; or perform a unified processing action determined based on the processing action identifier on all the document content on the image of the written document page.

[0274] Figure 8C A schematic diagram including area positioning identifiers and processing action identifiers according to an embodiment of the present disclosure is shown.

[0275] Take Figure 8C as an example. Suppose the area positioning identifier includes two gray boxes covering "The sun along the mountain bows" and "To see a thousand miles further", indicating that the interaction content area is the text areas of "The sun along the mountain bows" and "To see a thousand miles further". Suppose the processing action identifier is the QR code picture below, indicating that the processing action is to perform translation. Therefore, combining both the area positioning identifier and the processing action identifier to perform the processing action determined based on the processing action identifier on the document content extracted based on the area positioning identifier means translating "The sun along the mountain bows" and "To see a thousand miles further". Or, performing a unified processing action determined based on the processing action identifier on all the document content on the image of the written document page, that is, translating "The sun along the mountain bows", "The Yellow River flows into the sea", "To see a thousand miles further", and "Ascend another storey to see a thousand miles more".

[0276] In this way, the processing action can be more flexibly customized for the corresponding processing action of the document content by combining it with a certain document content or a unified processing action for the page.

[0277] Figure 8DSchematic diagram showing an example of a regional positioning identifier according to an embodiment of the present disclosure.

[0278] As Figure 8D shown, marks are made at the four corners of the document paragraph as regional positioning identifiers to indicate that the text part of "The sun along the mountain bows" is the interactive content area range, and "The sun along the mountain bows" therein is the document content within the interactive content area range.

[0279] Note that various combinations can be made for the types or forms of the visual identifiers mentioned herein to achieve rich interactive activities.

[0280] In some embodiments, one or more visual identifiers include document positioning identifiers. Among them, the document positioning identifier corresponds to a specific position on the electronic mapped document page. The coordinate correspondence between the image of the written document page and the electronic mapped document page is obtained through the following steps: using the document positioning identifier to indicate the position on the written document page corresponding to the specific position on the electronic mapped document page, and establishing the coordinate correspondence between the image of the written document page and the electronic mapped document page according to multiple specific positions on the electronic mapped document page and the corresponding multiple document positioning identifiers.

[0281] In this way, by using the written document positioning identifier, it is beneficial to compare the positions of the document image with the original document image or the electronic document, so as to obtain the coordinate correspondence between the image of the written document page and the electronic mapped document page.

[0282] In some embodiments, the interactive system 300 may further include a display screen and / or a speaker (not shown in the figure). The control unit 330 may also be configured to: based on the information stored in the second memory, formulate a review plan, where the review plan includes periodically or aperiodically or stage-by-stage based on the memory curve summarizing and presenting or notifying on the display screen and / or the speaker a part or all of the historical document content associated with the executor and / or the processing actions on the historical document content for review.

[0283] In this way, after the student identifier and the processing of the document content, the student can regularly review the document content, improving the learning efficiency and effect. Here, the display screen may include a light-emitting diode LED display screen or an electronic ink display screen, etc. However, for protecting eyesight, the electronic ink display screen can be preferably used.

[0284] In some embodiments, the control unit 330 may also be configured to: in response to one or more visual identifiers including at least one visual identifier of type being a region positioning identifier and / or a processing action identifier and / or a user identification identifier, assign importance levels to historical document content marked by visual identifiers or processed by processing actions according to different visual features of the region positioning identifier and / or the processing action identifier, wherein different visual features correspond to different importance levels; for historical document content assigned different importance levels, for different executors represented by user identification identifiers, summarize in stages with different periods or different memory curves or different understanding curves or combinations thereof, and present on the display screen and / or speaker a portion or all of the historical document content and / or processing actions of the historical document content associated with the executor for review.

[0285] Here, importance levels can be divided from 1 to 10, where 1 represents low importance and 10 represents high importance. Conversely, 10 can represent low importance and 1 high importance. The importance level can be determined by the user instructing the control unit 330, or it can be determined automatically by the control unit 330. When the control unit 330 determines the importance automatically, it can collect data such as the number of times the same document content is marked with visual identifiers or processed by actions, or the types of visual identifiers used to mark the document content, and determine the importance level based on this information. For example, content marked with a yellow highlighter has lower importance, while content marked with a red highlighter has higher importance; similarly, content marked with an underline has lower importance, while content marked with an underlined circle has higher importance, and so on.

[0286] For example, historical document content marked with visual identifiers or processed by actions (such as a certain English word) can be assigned a high importance level, while historical document content marked with visual identifiers or processed by actions (such as another English word) can be assigned a low importance level. Thus, the low importance level document content can be summarized in stages using a longer period or a less frequent memory curve or a less frequent comprehension curve, or a combination thereof, and presented on the screen and / or speakers for fewer reviews, while the high importance level document content can be summarized in stages using a shorter period or a more frequent memory curve or a more frequent comprehension curve, or a combination thereof, and presented on the screen and / or speakers for more reviews, so as to better remember the more important review content.

[0287] Here, the memory curve is, for example, the Ebbinghaus forgetting curve. Figure 9A An example of an Ebbinghaus forgetting curve according to an embodiment of the present disclosure is shown.

[0288] like Figure 9AAs shown, forgetting begins immediately after learning, and the process of forgetting is not uniform. Specifically, the initial rate of forgetting is rapid, gradually slowing down over time. Therefore, a review plan can be developed by conducting multiple reviews within a short period after initially learning new knowledge, and then gradually extending the review intervals. For example, the first review could be conducted within 24 hours of learning new content, followed by subsequent reviews on the seventh, thirtieth, and ninetieth days. In this way, the review cycle can be determined based on the Ebbinghaus forgetting curve.

[0289] The comprehension curve can be the opposite of the Ebbinghaus forgetting curve or other curves. That is, you may not understand the document content the first time you encounter it, meaning your comprehension level is low. However, with repeated review over time, your comprehension level can increase. Thus, the comprehension curve can be used to determine review cycles, etc.

[0290] Figure 10A1 and Figure 10A2 A schematic diagram is shown illustrating the review cycle for high-importance document content based on a memory curve, according to an embodiment of the present disclosure.

[0291] like Figure 10A1 The study shows that for high-importance document content, it's possible to review it based on the forgetting curve: first when memory drops to 80%, second when memory drops to 60%, third when memory drops to 40%, fourth when memory drops to 30%, and fifth when memory drops to 20%. In other words, there are a total of five reviews, a relatively short review cycle, and a high review frequency.

[0292] like Figure 10A2 The data shows that for low-importance document content, it is possible to review it a third time based on the forgetting curve: once when memory drops to 60%, a second time when memory drops to 30%, and a third time when memory drops to 20%. In other words, there are a total of three reviews, resulting in a longer review cycle and a lower review frequency.

[0293] In addition, the above review cycles are all determined based on the forgetting curve, which conforms to the physiological memory characteristics of humans. Therefore, this is a relatively reasonable review plan that can ensure that the student's review frequency matches the importance of the review content and the physiological memory characteristics of humans.

[0294] When considering different performers (users), it's also possible to differentiate between different user cycles, memory curves, comprehension curves, or combinations thereof. For example, review content can be presented using different cycles, memory curves, comprehension curves, or combinations thereof based on user characteristics such as age group, gender, and user preferences (e.g., user-defined preferences). Visual identifiers can also include user identification markers, such as a yellow highlighter or a dog sticker to indicate visual identification performed by user A, or a red highlighter or a cat sticker to indicate visual identification performed by user B. Examples are not exhaustive.

[0295] Figure 9B An example of Ebbinghaus forgetting curves for users of different age groups according to embodiments of the present disclosure is shown.

[0296] For example, such as Figure 9B As shown, young people forget more slowly, while older people forget more quickly. However, the memory curves of different users are not limited to this. In this way, it is possible to distinguish different periods, different memory curves, different comprehension curves, or combinations thereof among different users.

[0297] Figure 10B A schematic diagram illustrating the review cycles of older adults and younger adults based on memory curves according to embodiments of the present disclosure is shown.

[0298] like Figure 10B The top section shows that for young users, the review can be based on their memory curve: first review when memory drops to 80%, second review when memory drops to 60%, and third review when memory drops to 55%. In other words, there are a total of three review sessions, a relatively long review cycle, and a low review frequency.

[0299] like Figure 10B The lower part shows that for elderly users, the review can be based on their memory curve: first review when memory drops to 80%, second review when memory drops to 60%, third review when memory drops to 50%, fourth review when memory drops to 40%, fifth review when memory drops to 35%, and sixth review when memory drops to 30%. In other words, there are a total of 6 review sessions, a short review cycle, and a high review frequency.

[0300] In this way, review materials of different importance can be distinguished according to their importance level, and review plans can be formulated for review materials of different importance and / or different implementers (users), which can ensure that review time is allocated reasonably and that more important materials are reviewed.

[0301] The specific examples above are merely illustrative and not limiting. Furthermore, those skilled in the art can formulate appropriate review plans based on the importance of the review materials, the characteristics of the implementer (user), the cycle setting, the memory curve or comprehension curve, or other various memory patterns and other considerations.

[0302] In some embodiments, the written document interaction system further includes a timer configured to acquire time information, wherein the importance level of the historical document content is N, where N is a positive integer, and the larger the value of N, the higher the importance, and the control unit 130 is further configured to set the period t for reviewing the historical document content via the timer. i , where t i It is the number of hours between the i-th review and the (i-1)-th review, where i is a positive integer. Where c, d, and e are constants, and k is the basic review interval (in hours). Assume the first review time here is the time when the historical document content is first processed, i.e., the time when the user first sees the document content. Where t... i =k·r (i-1) , Here, r is a growth factor representing the rate of increase of the review interval related to N. The closer r is to 1, the slower the review interval increases. Ideally, r should be within the range of 0. <r<1.5。

[0303] For example, the value of c determines the upper limit of the growth factor r related to N (when N is close to 0). By adjusting c, the maximum speed of the growth factor r related to N can be controlled. d is used to adjust the degree of influence of N on the growth factor r; a smaller d value will make changes in N have a greater impact on the growth factor r. e determines the rate at which the number of reviews i affects the growth of the review interval. A larger e value will make the review interval grow faster with the increase of the number of reviews.

[0304] In one example, c = 10, d = 1, and e = 2 can be chosen. Of course, these values ​​can be adjusted according to needs. This allows for flexibility when N is large. It will approach 0, growth factor It will be a number close to 1 but less than 1. This means that the review interval will slowly increase with the number of reviews. When N is small, It will be a large number, growth factor It will remain within a reasonable range. e=2 makes the rate of increase in the review interval relatively stable, neither too fast nor too slow.

[0305] For example, if N = 1 and k = 24 hours, then t2 (the number of hours between the first and second review sessions) is: Therefore, t2 is approximately 53.664 hours. t3 (the number of hours between the first and third review sessions) is... That is, t3 is 120 hours.

[0306] For example, if N = 10 and k = 24 hours, then t2 (the number of hours between the first and second review sessions) is: Therefore, t2 is approximately 7.42 hours. t3 (i.e., the number of hours between the first and third review sessions) is... It takes approximately 21.82 hours.

[0307] Other numerical examples can be calculated from the above formula, and will not be described one by one here.

[0308] Here, a study plan can be developed using a formula that specifies the review intervals based on importance levels.

[0309] In summary, it can provide convenient digital and intelligent support for users who use written materials for learning and work, improve the efficiency of teacher-student interaction and team collaboration, enable users to manually create interactive written documents, and allow manual operation to achieve interaction, as well as help users review written materials, thereby protecting users' eyesight, and providing digital and intelligent applications for written materials.

[0310] Embodiments of this disclosure may also include an electronic device, which may include a processor and a memory, the memory being coupled to the processor and storing computer instructions therein for performing the steps of various methods of at least one embodiment of this disclosure when executed by the processor.

[0311] In one embodiment, the electronic device may also include a touchscreen and / or a display screen.

[0312] Figure 11 This is a schematic diagram of an electronic device with a transparent touchscreen according to an embodiment of the present disclosure.

[0313] like Figure 11 As shown, in one embodiment, an electronic device with a transparent touchscreen covers a written document page. The memory stores software and electronically mapped document data. The user selects document content on the written document page via the touchscreen and determines an interactive action. Upon receiving information from the touchscreen, the processor retrieves the interactive content data from the memory and executes the interactive action. Determining the electronically mapped document page and obtaining the coordinate correspondence can be done by inputting information via the touchscreen (such as entering a page number or clicking the four corners of the written document page to obtain the coordinate correspondence), or by adding other input modules such as an imaging module or a barcode scanning module to the electronic device. The touchscreen can be designated to a specific area for inputting characters or interactive commands. The area or position of the display screen and the touchscreen can be the same or different.

[0314] Figure 12A It is a schematic diagram of an electronic device with a transparent touch display screen according to an embodiment of the present disclosure.

[0315] As Figure 12A shown, in one embodiment, the electronic device with a transparent touch display screen is covered on a written document page. Software and electronic mapping document data are stored in a memory. By selecting document content on the written document page through the touch display screen and determining an interaction processing action, after the processor receives the information of the touch screen, it obtains interaction content data from the memory and executes the interaction processing action, and the feedback information can be displayed on the transparent touch display screen. A virtual keyboard area can also be set on the touch screen for inputting instructions or characters.

[0316] Figure 12B It is a physical diagram of an electronic device with a transparent touch display screen according to an embodiment of the present disclosure.

[0317] As Figure 12B shown, in one embodiment, the electronic device with a transparent touch display screen is covered on a written document page. By circling the three characters "see clearly to" on the touch screen, interaction processing can be performed on these three characters.

[0318] Embodiments of the present disclosure may also include a non - transitory computer - readable storage medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the various methods described above can be executed.

[0319] Embodiments of the present disclosure may also include a computer program product, where the computer program product has computer instructions, and when the computer instructions are executed by a processor, the methods, steps, and operations given herein can be performed. For example, such a computer program product can be a computer software package, computer code instructions, a computer - readable tangible medium having computer instructions stored (and / or encoded) thereon, and the instructions can be executed by a processor to perform the operations described herein.

[0320] Through the description of the above - mentioned embodiments, those skilled in the art can easily understand that the exemplary embodiments described here can be implemented by software or by a combination of software and necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non - volatile storage medium (which can be a CD - ROM, USB flash drive, mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, server, or network device, etc.) to execute the above - mentioned methods according to the embodiments of the present disclosure.

[0321] The software product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0322] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0323] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0324] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0325] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0326] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. An interactive method based on written documents, characterized in that, include: Obtain data from the written document page that can generate interactive instruction information; Generate interactive instruction information based on the data capable of generating interactive instruction information; Obtain the electronically mapped document data corresponding to the written document page; The interaction processing action is determined based on the interaction instruction processing action information in the interaction instruction information. Based on the interaction instruction location information in the interaction instruction information, the interaction content data related to the interaction instruction location information is extracted from the electronic mapping document data corresponding to the written document page; The interactive processing action is performed on the extracted interactive content data.

2. The interaction method according to claim 1, wherein The electronically mapped document data includes interactive content data corresponding to the written document page, and it is possible to obtain the location information of the interactive content data on the written document page or the electronically mapped document page. The interactive instruction information includes interactive instruction processing action information and / or interactive instruction location information and / or information for obtaining the electronically mapped document data corresponding to the written document page. The eMapping document data includes one or more eMapping document pages; The interactive instruction location information refers to one or more location information on a written document page or an electronically mapped document page related to the interactive instruction information. The interactive content data includes at least one of the following: document content, preset interactive data, interactive process data, and interactive result data. The document content refers to the content visible on the pages of a written document or the corresponding document content data stored in electronically mapped document data. The electronically mapped document page refers to an electronic document page that is stored or can be generated in the electronically mapped document data corresponding to the written document page.

3. The interaction method according to claim 1, characterized in that, The data capable of generating interactive instruction information includes an image with a written document page, wherein the image with the written document page contains one or more visual identifiers, which are either inside or outside the written document page in the image. The step of generating interactive instruction information based on data capable of generating interactive instruction information specifically includes: Identify one or more visual identifiers within the image containing the written document page; Obtain the coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the written document page, or obtain the coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the electronically mapped document page; The interaction instruction information is determined based on the visual identifier and the obtained coordinate correspondence.

4. The interaction method according to claim 1, characterized in that, The data capable of generating interactive command information includes one or more coordinate information in a first coordinate system. This first coordinate system is the coordinate information in the indicated first coordinate system, which may intersect, be parallel to, or coincide with the written document page. The step of generating interactive instruction information based on data capable of generating interactive instruction information specifically includes: Obtain the coordinate correspondence between the first coordinate system and the written document page coordinate system, or obtain the coordinate correspondence between the first coordinate system and the electronically mapped document page coordinate system; The interaction instruction information is determined based on the coordinate information of the first coordinate system and the obtained coordinate correspondence.

5. The interaction method according to claim 1, characterized in that, The step of extracting interactive content data related to the interactive instruction location information from the electronic mapping document data corresponding to the written document page, based on the interactive instruction location information in the interactive instruction information, specifically includes: The range of the interactive content area is determined based on the location information of the interactive instructions in the interactive instruction information. Obtain interactive content data from the eMapped document data, and obtain the location information of the interactive content data on the written document page or the eMapped document page; Based on the location information of the interactive content data on the written document page or the electronically mapped document page, and the range of the interactive content area, interactive content data whose location information meets the conditions is selected as the interactive content data related to the location information of the interactive instruction. The conditions refer to the fact that the location information intersects with the range of the interactive content area and / or the range of the interactive content area.

6. The interaction method according to claim 5, characterized in that, The interactive content data includes document content elements or combinations of document content elements, and the e-mapped document data includes document content elements or combinations of document content elements. It is also capable of acquiring the positional information of the document content elements or combinations of document content elements within the written document page or the e-mapped document page. The document content elements are units that constitute the content of the written document, and the document content element combinations are combinations of multiple document content elements. The step of selecting interactive content data whose location information meets the criteria as the interactive content data related to the location information of the interactive instruction, based on the location information of the interactive content data on the written document page or the electronic mapping document page and the range of the interactive content area, specifically includes: Based on the range of the interactive content area and the location information of the document content element or combination of document content elements in the written document page or the electronically mapped document page, determine the document content element or combination of document content elements whose location information meets the conditions. Based on the location information of the document content elements or combinations of document content elements in the written document page or the electronically mapped document page, the document content elements or combinations of document content elements that meet the location information conditions are arranged and combined to generate the interactive content data related to the location information of the interactive instruction.

7. The interaction method according to claim 1, characterized in that, The electronically mapped document data contains an image of an electronically mapped document page corresponding to the written document page, or contains an electronic document capable of generating an electronically mapped document page corresponding to the written document page. The step of extracting interactive content data related to the interactive instruction location information from the electronic mapping document data corresponding to the written document page, based on the interactive instruction location information in the interactive instruction information, specifically includes: The range of the corresponding interactive content area in the electronic mapping document page is determined based on the interactive instruction location information in the interactive instruction information. Extract a portion of the interactive content area from the electronic mapping document page based on the corresponding interactive content area range in the electronic mapping document page; Based on a portion of the interactive content area captured from the electronically mapped document page, the document content within the interactive content area is extracted as the interactive content data related to the location information of the interactive instruction.

8. The interaction method according to claim 3, characterized in that, The step of determining the interaction instruction information based on the visual identifier and the obtained coordinate correspondence specifically includes: In response to the fact that one or more visual identifiers include at least one visual identifier of type interactive instruction location information identifier, the position of the interactive instruction location information identifier in the image with the written document page is determined; Based on the location of the location information identifier in the image with the written document page and the obtained coordinate correspondence, the interaction instruction location information in the interaction instruction information is determined.

9. The interaction method according to claim 1, characterized in that, Obtaining the electronically mapped document data corresponding to the written document page specifically includes one or more of the following steps: The electronic mapping document data corresponding to the written document page is determined by comparing the similarity between the whole or part of the image of the written document page and the whole or part of the electronic mapping document data. In response to the fact that at least one of the one or more visual identifiers is of the type of a written document identification identifier, and in response to the fact that the written document identification identifier is consistent with the identification identifier stored in the eMapped Document Data, the eMapped Document Data corresponding to the written document page is determined; In response to the fact that at least one of the visual identifiers is of the type of a written document identification identifier, the information recorded in the written document identification identifier is determined, and in response to the fact that the recorded information is consistent with the information of the electronically mapped document data, the electronically mapped document data corresponding to the written document page is determined; The electronic mapping document data corresponding to the written document page is determined in response to the input information and in response to the input information being consistent with the information in the electronic mapping document data.

10. The interaction method according to claim 1, characterized in that, The step of performing the interactive processing action on the extracted interactive content data further includes: Based on the interaction instruction location information or preset rules in the interaction instruction information, obtain the relative positional relationship between the projection or display position of the interaction processing result and the written document page or the image of the written document page; Based on the relative position of the projection or display position of the interaction processing result with the written document page or the image of the written document page, the processing result of the interaction processing action is projected or displayed at a specified position inside or outside the written document page, or projected or displayed at a specified position inside or outside the written document page in the user's field of vision.

11. The interaction method according to claim 1, characterized in that, The step of performing the interactive processing action on the extracted interactive content data further includes: The location information of the written document page or electronic mapping document page corresponding to the interaction result data of the interaction processing action is obtained based on the interaction instruction location information in the interaction instruction information or the pre-set interaction result data location rules. The interaction result data of the interaction processing action is associated with the obtained location information or the document content at the location of the location information and saved.

12. The interaction method according to claim 1, characterized in that, The step of performing the interactive processing action on the extracted interactive content data further includes: Project or display the interactive content data at a designated location, or read the interactive content data aloud; After confirming the interactive content data, the interactive processing action is performed on the extracted interactive content data.

13. The interaction method according to claim 1, characterized in that, The step of performing the interactive processing action on the extracted interactive content data further includes: Based on interactive instructions, process action information or preset rules to call, share or copy interactive content data that meets the conditions from other electronic mapping document data, either locally or via the network. Perform the interactive processing action on the interactive content data that is invoked, shared, or copied.

14. The interaction method according to claim 1, characterized in that, The interactive instruction processing action information includes at least one of the following: processing action information, parameter information, information of the called program or service, address information of the called program or service, name information of the called model, user information, and password information.

15. The interaction method according to claim 1, characterized in that, Also includes: The electronically mapped document data is used to print the written document page or to photograph the written document page to generate electronically mapped document data.

16. The interaction method according to claim 5, characterized in that, The interaction command location information in the interaction command information includes the coordinates of one side of the quadrilateral interaction content area and the direction of the opposite side of the quadrilateral interaction content area. Determining the interaction content area based on the interaction command location information specifically includes: Obtain the coordinates of one side of the quadrilateral interactive content area, and obtain the direction of the opposite side of the quadrilateral interactive content area; The interactive content area is defined as the region from the first encounter with a row of pixels of two or more colors to the end of a row of pixels of only one color, by scanning from one side of the quadrilateral interactive content area towards the opposite side. Alternatively, based on the position information of each character in the document content, the region containing the row of characters closest to one side of the quadrilateral interactive content area is selected as the interactive content area.

17. The interaction method according to claim 14, characterized in that, The parameter information includes information added by the user on the written document page. The step of determining the interaction processing action based on the interaction instruction processing action information in the interaction instruction information specifically includes: Based on the document content in the electronic mapping data, the document content is removed from the image of the acquired written document page, and then the information added by the user on the written document page is obtained from the image as the parameter information.

18. An interactive system based on written documents, characterized in that, include The interactive instruction information acquisition unit is configured to acquire data that can generate interactive instruction information based on the written document page; The control unit is configured as follows: Generate interactive instruction information based on the data capable of generating interactive instruction information; Obtain the electronically mapped document data corresponding to the written document page; The interaction processing action is determined based on the interaction instruction processing action information in the interaction instruction information. Based on the interaction instruction location information in the interaction instruction information, the interaction content data related to the interaction instruction location information is extracted from the electronic mapping document data corresponding to the written document page; The interactive processing action is performed on the extracted interactive content data.

19. The interactive system according to claim 18, characterized in that, The interactive instruction information acquisition unit includes an imaging device configured to acquire an image with a written document page, wherein the image with the written document page has one or more visual identifiers. It also includes, The recognition unit is configured to recognize one or more visual identifiers on the image containing the written document page; The control unit generates interactive command information based on the data capable of generating interactive command information through the following steps: Obtain the coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the written document page, or obtain the coordinate correspondence between the coordinate system of the image with the written document page and the coordinate system of the electronically mapped document page; The interaction instruction information is determined based on the visual identifier and the obtained coordinate correspondence.

20. The interactive system according to claim 18, characterized in that, The interactive instruction information acquisition unit includes a coordinate acquisition device configured to acquire coordinate information of the indicated position in a first coordinate system. The coordinate acquisition device includes the first coordinate system, which intersects with, is parallel to, or coincides with the written document page. The control unit generates interactive command information based on the data capable of generating interactive command information through the following steps: Obtain the coordinate correspondence between the first coordinate system and the written document page coordinate system, or obtain the coordinate correspondence between the first coordinate system and the electronically mapped document page coordinate system; The interaction instruction information is determined based on the coordinate information of the first coordinate system and the obtained coordinate correspondence.

21. The interactive system according to claim 18, characterized in that, The interactive system also includes an overlay covering the written document page, wherein the overlay can form one or more visual identifiers on the overlay by changing local colors or transparency. The imaging device is configured to acquire an image of a page of a written document as the electronically mapped document data when the document is not obscured by a covering or a visual identifier formed on the covering.

22. The written document interaction system according to claim 18, characterized in that, It also includes a display unit, configured to project or display the content to be displayed at a specified location. The control unit is configured to: Based on the interaction instruction location information or preset rules in the interaction instruction information, obtain the relative positional relationship between the projection or display position of the interaction processing result and the written document page or the image of the written document page; Based on the relative position of the projection or display position of the interaction processing result with the written document page or the image of the written document page, the processing result of the interaction processing action is projected or displayed at a specified position inside or outside the written document page, or projected or displayed at a specified position inside or outside the written document page in the user's field of vision.

23. An electronic device comprising a processor and a memory, the memory being coupled to the processor and storing computer instructions therein for performing the method according to claims 1 to 17 when executed by the processor.

24. A non-transitory computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, perform the method according to claims 1 to 17.

25. A computer program product having computer instructions that, when executed by a processor, perform the method according to claims 1 to 17.

Images