Written Document Processing Method, Apparatus, Electronic Device and Computer Readable Medium

Abstract

The present disclosure relates to a method, apparatus, electronic device, and computer-readable medium for processing a written document. The method includes: obtaining a graphic and text control identifier within a preset area; determining a target interaction instruction and a target interaction object area based on the graphic and text control identifier; obtaining a document image of the written document within the target interaction object area; and processing the document image based on the target interaction instruction. The method, apparatus, electronic device, and computer-readable medium for processing a written document according to the present disclosure can implement the multimedia interaction function of the user's written document without relying on a display screen, and the user can also call computer multimedia resources for interaction.

Description

Technical Field

[0001] The present disclosure relates to the field of computer information processing, and in particular, to a written document processing method, device, electronic device and computer-readable medium. Background Art

[0002] Online homework methods such as multimedia, online classes, and remote office are becoming more and more widely used. Since traditional online homework methods are extremely dependent on display screens, they have a greater impact on the user's eyesight, especially children's eye health. Although products such as point reading pens and smart pens that use dot matrix code technology can be operated on paper documents to obtain multimedia interactive information, they need to be used with specially printed paper books, which requires high printing costs.

[0003] Therefore, a new method, device, electronic device and computer-readable medium for processing written documents are needed.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0005] In view of this, the present disclosure provides a written document processing method, device, electronic device and computer-readable medium, which can realize the multimedia interactive function of the user's written document without relying on a display screen, and the user can also call computer multimedia resources for interaction.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a method for processing written documents is proposed, the method comprising: obtaining a graphic control mark within a preset area; determining a target interaction instruction and a target interaction object area based on the graphic control mark; obtaining a document image of a written document within the target interaction object area; and processing the document image based on the target interaction instruction.

[0008] In an exemplary embodiment of the present disclosure, it also includes: generating interactive instruction mapping rules based on one or more graphic control identifiers and their corresponding interactive instructions; and / or generating auxiliary control instruction mapping rules based on one or more graphic control identifiers and their corresponding auxiliary control instructions.

[0009] In an exemplary embodiment of the present disclosure, it further includes: obtaining a graphic control identifier within a preset area; extracting a target auxiliary control instruction from an auxiliary control instruction mapping rule based on the graphic control identifier; and controlling a processing process or a multimedia device participating in the processing process based on the target auxiliary control instruction.

[0010] In an exemplary embodiment of the present disclosure, generating a target auxiliary control instruction based on the graphic control identifier and the auxiliary control instruction mapping rule further includes: when the graphic control identifier includes a parameter to be input, inputting parameter information according to a preset manner, where the parameter includes at least one of voice data, text data, image data, and gesture data.

[0011] In an exemplary embodiment of the present disclosure, obtaining a graphic control identifier within a preset area includes: obtaining an object image of an identification object placed within the preset area, where the preset area is within the imaging range of an imaging device; and performing image recognition on the object image of the identification object to generate the graphic control identifier.

[0012] In an exemplary embodiment of the present disclosure, performing image recognition on the object image of the identification object to generate the graphic control identifier includes: performing image recognition on the object image of the identification object to generate a recognition result; and removing invalid identification objects from the recognition result to generate the graphic control identifier.

[0013] In an exemplary embodiment of the present disclosure, determining a target interaction instruction and a target interaction object area based on the graphic control identifier includes: extracting a target interaction instruction from an interaction instruction mapping rule according to the graphic control identifier; and determining the target interaction object area based on the target interaction instruction.

[0014] In an exemplary embodiment of the present disclosure, processing the document image based on the target interaction instruction includes: performing image recognition on the document image based on the target interaction instruction to generate the document information; and performing data processing on the document information.

[0015] In an exemplary embodiment of the present disclosure, performing image recognition on the document image based on the target interaction instruction to generate the document information includes: determining a target area based on the target interaction instruction; and performing image recognition on the image in the target area of the document image to generate the document information.

[0016] In an exemplary embodiment of the present disclosure, processing the document image based on the target interaction instruction further includes: when the graphic control identifier requires input of a parameter, inputting parameter information according to a preset manner, where the parameter includes at least one of voice data, text data, image data, and gesture data.

[0017] In an exemplary embodiment of the present disclosure, it further includes: generating multimedia information based on the processing result; performing a multimedia operation based on the multimedia information.

[0018] In an exemplary embodiment of the present disclosure, performing a multimedia operation based on the multimedia information includes: determining a display area based on the graphic and text control identifier; projecting the processing result onto the display area.

[0019] In an exemplary embodiment of the present disclosure, it further includes: correcting the document image and / or the projected image projected on the display area.

[0020] In an exemplary embodiment of the present disclosure, correcting the document image and / or the projected image projected on the display area includes: obtaining a first relative position of a known point on the written document corresponding to the document image; obtaining a second relative position of the known point in the document image; determining a distortion correction mapping between the written document and the document image based on the first relative position and the second relative position; correcting the document image based on the distortion correction mapping.

[0021] In an exemplary embodiment of the present disclosure, correcting the document image and / or the projected image projected on the display area further includes: superimposing the projected image on the corrected document image to generate a superimposed image; determining a deviation adjustment mapping of the projected image according to the superimposed image; correcting the projected image based on the deviation adjustment mapping.

[0022] According to one aspect of the present disclosure, a method for processing a written document is provided, the method including: obtaining a graphic and text control identifier from a graphic and text control identifier area; obtaining a document image of the written document from an interaction object area; determining a target interaction instruction based on the graphic and text control identifier; processing the document image based on the target interaction instruction.

[0023] In an exemplary embodiment of the present disclosure, it further includes: generating at least one interaction instruction mapping rule based on one or more graphic and text control identifiers, at least one graphic and text control identifier area, and its corresponding interaction instruction; and / or generating at least one auxiliary control instruction mapping rule based on one or more graphic and text control identifiers, at least one graphic and text control auxiliary area, and its corresponding auxiliary control instruction.

[0024] In an exemplary embodiment of the present disclosure, it further includes: obtaining a graphic control identifier from a graphic control auxiliary area; extracting a target auxiliary control instruction from the at least one auxiliary control instruction mapping rule based on the graphic control identifier and the number of its corresponding graphic control auxiliary area; and controlling a processing process or a multimedia device participating in the processing process based on the target auxiliary control instruction.

[0025] In an exemplary embodiment of the present disclosure, extracting a target auxiliary control instruction from the at least one auxiliary control instruction mapping rule based on the graphic control identifier and the number of its corresponding graphic control auxiliary area further includes: when the graphic control identifier includes a parameter to be input, inputting parameter information according to a preset method, where the parameter includes at least one of voice data, text data, image data, and gesture data.

[0026] In an exemplary embodiment of the present disclosure, it further includes: adjusting the area range of the graphic control identifier area and / or the interaction object area and / or the graphic control auxiliary area; where the graphic control identifier area, the interaction object area, and the graphic control auxiliary area are within the imaging range of the imaging device.

[0027] In an exemplary embodiment of the present disclosure, adjusting the area range of the graphic control identifier area and / or the interaction object area and / or the graphic control auxiliary area includes: obtaining an object image of an identification object placed within the imaging range for marking the area range; performing image recognition on the object image of the identification object to generate a graphic control identifier for marking the area range; determining a target area range according to the graphic control identifier for marking the area range; and adjusting the area range of the interaction object area and / or the graphic control identifier area and / or the graphic control auxiliary area based on the target area range.

[0028] In an exemplary embodiment of the present disclosure, obtaining a graphic control identifier from a graphic control identifier area includes: determining the number of graphic control identifier areas by system settings; and obtaining a graphic control identifier from all graphic control identifier areas.

[0029] In an exemplary embodiment of the present disclosure, obtaining a graphic control identifier from all graphic control identifier areas includes: obtaining an object image of an identification object placed in all graphic control identifier areas; and performing image recognition on the object image of the identification object to generate the graphic control identifier.

[0030] In an exemplary embodiment of the present disclosure, determining a target interaction instruction based on the graphic control identifier includes: determining a target interaction instruction based on the graphic control identifier and the number of its corresponding graphic control identifier area.

[0031] In an exemplary embodiment of the present disclosure, determining a target interaction instruction based on the graphic control identifier and the number of the corresponding graphic control identifier area includes: determining a target interaction instruction mapping rule from the at least one interaction instruction mapping rule based on the number of the graphic control identifier area; and extracting the target interaction instruction from the target interaction instruction mapping rule according to the graphic control identifier.

[0032] In an exemplary embodiment of the present disclosure, processing the document image based on the target interaction instruction includes: performing image recognition on the document image based on the target interaction instruction to generate the document information; and performing data processing on the document information.

[0033] In an exemplary embodiment of the present disclosure, performing image recognition on the document image based on the target interaction instruction to generate the document information includes: determining a target area based on the target interaction instruction; and performing image recognition on the image in the target area of the document image to generate the document information.

[0034] In an exemplary embodiment of the present disclosure, it further includes: generating multimedia information based on the processing result; and performing multimedia operations based on the multimedia information.

[0035] In an exemplary embodiment of the present disclosure, performing multimedia operations based on the multimedia information includes: determining a display area based on the graphic control identifier; and projecting the processing result onto the display area.

[0036] In an exemplary embodiment of the present disclosure, it further includes: correcting the document image and / or the projected image projected on the display area.

[0037] In an exemplary embodiment of the present disclosure, correcting the document image and / or the projected image projected on the display area includes: obtaining a first relative position of a known point on the written document corresponding to the document image; obtaining a second relative position of the known point in the document image; determining a distortion correction mapping between the written document and the document image based on the first relative position and the second relative position; and correcting the document image based on the distortion correction mapping.

[0038] In an exemplary embodiment of the present disclosure, correcting the document image and / or the projected image projected on the display area further includes: superimposing the projected image on the corrected document image to generate a superimposed image; determining a deviation adjustment mapping of the projected image according to the superimposed image; and correcting the projected image based on the deviation adjustment mapping.

[0039] According to one aspect of the present disclosure, a written document processing system is provided, which includes: an identification object for displaying graphic and text control identifiers; an imaging device for acquiring the graphic and text control identifiers within a preset area, and / or acquiring a document image of a written document within a target interaction object area; a processing device for determining a target interaction instruction and the target interaction object area based on the graphic and text control identifiers; and processing the document image based on the target interaction instruction; and / or determining a target auxiliary control instruction based on the graphic and text control identifiers; controlling the processing process or a multimedia device participating in the processing process based on the target auxiliary control instruction.

[0040] In an exemplary embodiment of the present disclosure, it further includes: a multimedia device for generating multimedia information based on a processing result; and performing a multimedia operation based on the multimedia information.

[0041] According to one aspect of the present disclosure, a written document processing system is provided, which includes: an identification object for displaying graphic and text control identifiers; an imaging device for acquiring the graphic and text control identifiers from a graphic and text control identifier area, and / or acquiring the graphic and text control identifiers from a graphic and text control auxiliary area, and / or acquiring a document image of a written document from an interaction object area; a processing device for determining a target interaction instruction based on the graphic and text control identifiers; and processing the document image based on the target interaction instruction; and / or determining a target auxiliary control instruction based on the graphic and text control identifiers, and controlling the processing process or a multimedia device participating in the processing process based on the target auxiliary control instruction.

[0042] In an exemplary embodiment of the present disclosure, it further includes: a multimedia device for generating multimedia information based on a processing result; and performing a multimedia operation based on the multimedia information.

[0043] According to one aspect of the present disclosure, an electronic device is provided, which includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described above.

[0044] According to one aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method as described above is implemented.

[0045] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features, and advantages of the present disclosure will become more apparent by describing its exemplary embodiments in detail with reference to the accompanying drawings. The following described drawings are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0047] Figure 1 is a system block diagram of a written document processing method and apparatus shown according to an exemplary embodiment.

[0048] Figure 2 is a flowchart of a written document processing method shown according to an exemplary embodiment.

[0049] Figure 3 is a schematic diagram of a written document processing method shown according to an exemplary embodiment.

[0050] Figure 4 is a flowchart of a written document processing method shown according to another exemplary embodiment.

[0051] Figure 5 is a schematic diagram of a written document processing method shown according to an exemplary embodiment.

[0052] Figure 6 is a flowchart of a written document processing method shown according to another exemplary embodiment.

[0053] Figure 7 is a flowchart of a written document processing method shown according to another exemplary embodiment.

[0054] Figure 8 is a block diagram of a written document processing apparatus shown according to an exemplary embodiment.

[0055] Figure 9 is a block diagram of a written document processing apparatus shown according to another exemplary embodiment.

[0056] Figure 10 is a block diagram of an electronic device shown according to an exemplary embodiment.

[0057] Figure 11 is a block diagram of a computer-readable medium shown according to an exemplary embodiment. Detailed implementation manners

[0058] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. Like reference numerals in the figures denote like or similar parts, and thus their repetitive description will be omitted.

[0059] In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be used. In other cases, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present disclosure.

[0060] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0061] The flowcharts shown in the drawings are merely illustrative and not necessarily inclusive of all content and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined, so the actual execution order may change according to the actual situation.

[0062] It should be understood that although terms such as first, second, and third may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Thus, the first component discussed below can be referred to as the second component without departing from the teachings of the concept of the present disclosure. As used herein, the term "and / or" includes any one and all combinations of one or more of the associated listed items.

[0063] Those skilled in the art can understand that the drawings are only schematic diagrams of the example embodiments, and the modules or processes in the drawings are not necessarily essential for implementing the present disclosure, and thus cannot be used to limit the protection scope of the present disclosure.

[0064] Figure 1 is a system block diagram of a method, apparatus, electronic device, and computer-readable medium for processing written documents shown according to an exemplary embodiment.

[0065] As Figure 1As shown, the system architecture 10 may include an identification object 101, an imaging device 102, a network 103, and a processing device 104. The system architecture 10 may also include several multimedia devices 105. The network 103 is used to provide a medium for the communication link between the imaging device 102 and the processing device 104; the network 103 is also used to provide a medium for the communication link between the multimedia device 105 and the processing device 104. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0066] The identification object 101 is used to display graphic control identification. The identification object 101 can be one-dimensional and / or two-dimensional and / or three-dimensional. One or several of points, lines, shading, background color, text, symbols, numbers, two-dimensional codes, barcodes, pictures, graphics, charts, two-dimensional or three-dimensional identification objects, or other graphic information that can be recognized by computer vision can be selected to form the identification object 101;

[0067] Furthermore, one and / or more identification objects 101 can be combined according to certain rules in time and / or space to form a new identification object 101. When combining the identification objects 101, only the original identification objects 101 can be used, or identification objects 101 for representing the combination relationship between the identification objects 101 can be added to the original identification objects 101.

[0068] Any one identification object 101 being in different positions, and / or the combination of one and / or more identification objects 101, and / or one and / or more identification objects 101 being in different positions and / or relative positions, and / or the relative position between one or more identification objects 101 and the imaging device 102, and / or one and / or more identification objects 101 appearing and / or disappearing in different orders, and / or the different movement trajectories and / or speeds and / or accelerations and / or angular velocities and / or angular accelerations and / or relative position changes and / or relative speeds and / or relative accelerations and / or relative angular velocities and / or relative angular accelerations and / or vibration frequencies and / or vibration amplitudes of one or more identification objects 101, and / or one or more identification objects 101 changing color and / or deforming, can correspond to different interaction instructions or auxiliary control instructions.

[0069] The graphic control identification can be displayed in a preset area by one or several of writing, drawing, printing, pasting, displaying, placing, inserting, or other ways that can display the identification object 101, and be set at an appropriate position on or around the written document.

[0070] The mapping rule between the identification object 101 and the interaction instruction or the auxiliary control instruction, or its storage address, the program for executing the task of identifying the graphic control identification or its storage address, and the program for converting the interaction instruction or its storage address can be stored in the memory of the client, server or cloud, or can be stored in a visually recognizable information storage manner (such as a QR code) on the graphic control identification. When stored in a visually recognizable information storage manner (such as a QR code) on the graphic control identification, the content stored in the visually recognizable information storage manner on the graphic control identification can be obtained by identifying the graphic control identification.

[0071] The imaging device 102 is used to obtain the graphic control identification within a preset area and / or obtain the document image of the written document within the target interaction object area; the imaging device 102 interacts with the processing device 104 through the network 103 to receive or send messages, etc. The imaging device 102 can be an intelligent camera, camera, monitoring device, etc. with a camera function.

[0072] The processing device 104 is used to determine the target interaction instruction and the target interaction object area based on the graphic control identification; and process the document image based on the target interaction instruction; and / or determine the target auxiliary control instruction based on the graphic control identification; control the processing process or the multimedia device participating in the processing process based on the target auxiliary control instruction. Various data transmission applications can be installed on the processing device 104, such as image recognition applications, education platform applications, multimedia applications, etc. The processing device 104 can be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.

[0073] It is worth mentioning that the processing device 104 can be a physical device, a remote server, or a virtual device in the cloud. The present disclosure is not limited thereto.

[0074] The multimedia device 105 is used to generate multimedia information based on the processing result; and perform multimedia operations based on the multimedia information. The multimedia device 105 can include various types of settings, such as intelligent lights, intelligent speakers, multimedia players, projectors, etc.

[0075] In a specific usage scenario, a written document and an identification object 101 are placed on a plane, which can be a desk or other platform, and this platform serves as a preset area. The imaging device 102 can, for example, acquire graphic control identifiers within the preset area; the processing device 104 can, for example, determine a target interaction instruction and a target interaction object area based on the graphic control identifiers; the imaging device 102 can, for example, acquire a document image of the written document within the target interaction object area; the processing device 104 can, for example, process the document image based on the target interaction instruction.

[0076] Before the above processing, the processing device 104 can also, for example, generate an interaction instruction mapping rule based on one or more graphic control identifiers and their corresponding interaction instructions; and / or generate an auxiliary control instruction mapping rule based on one or more graphic control identifiers and their corresponding auxiliary control instructions.

[0077] During the above processing, for example, the identification object 101 is placed on the desktop, the imaging device 102 can, for example, acquire graphic control identifiers within the preset area; the processing device 104 can, for example, extract a target auxiliary control instruction from the auxiliary control instruction mapping rule based on the graphic control identifiers; the processing device 104 can, for example, control the processing process or the multimedia device 105 participating in the processing process based on the target auxiliary control instruction.

[0078] Regarding the technical content of this usage scenario, it will be described in detail in Figure 2 , Figure 3 the corresponding embodiments.

[0079] In another specific usage scenario, a written document and an identification object 101 are placed on a plane, which can be a desk or other platform, and this platform serves as a preset area. The imaging device 102 can, for example, acquire graphic control identifiers from the graphic control identifier area; the imaging device 102 can, for example, determine a target interaction instruction based on the graphic control identifiers; the imaging device 102 can, for example, acquire a document image of the written document from the interaction object area; the processing device 104 can, for example, process the document image based on the target interaction instruction.

[0080] Before the above processing, the processing device 104 can also, for example, generate at least one interaction instruction mapping rule based on one or more graphic control identifiers, at least one graphic control identifier area and their corresponding interaction instructions; and / or generate at least one auxiliary control instruction mapping rule based on one or more graphic control identifiers, at least one graphic control auxiliary area and their corresponding auxiliary control instructions.

[0081] In the above processing, for example, it can be identified that the object 101 is placed on the desktop, and the imaging device 102 can obtain the graphic control identification from, for example, the graphic control auxiliary area; the processing device 104 can extract the target auxiliary control instruction from the at least one auxiliary control instruction mapping rule based on the graphic control identification and the number of its corresponding graphic control auxiliary area; and control the processing process or the multimedia device 105 participating in the processing process based on the target auxiliary control instruction.

[0082] The technical content regarding this usage scenario will be in Figure 4 , Figure 5 the corresponding embodiments will be described in detail.

[0083] Figure 2 is a flowchart of a written document processing method shown according to an exemplary embodiment. The written document processing method 20 is a written document processing method for the scenario as Figure 3 described. In the Figure 3 corresponding schematic diagram, the written document is placed on a flat desk, and the user has previously placed the identification object (graphic control identification) to be processed at any position on the desk. The written document processing method 20 at least includes steps S202 to S208.

[0084] As Figure 2 shown, in S202, the graphic control identification within the preset area is obtained. For example, the object image of the identification object placed within the preset area can be obtained, and the preset area is within the imaging range of the imaging device; wherein, performing image recognition on the object image of the identification object to generate the graphic control identification includes: performing image recognition on the object image of the identification object to generate a recognition result; and removing the invalid identification objects in the recognition result to generate the graphic control identification.

[0085] In one embodiment, before obtaining the graphic control identification within the preset area, it further includes: generating an interaction instruction mapping rule based on one or more graphic control identifications and their corresponding interaction instructions; and / or generating an auxiliary control instruction mapping rule based on one or more graphic control identifications and their corresponding auxiliary control instructions.

[0086] Among them, the interaction instruction mapping rule and the auxiliary control instruction mapping rule can specifically include; setting one or more graphic control identifications; setting the recognition method or recognition program of the graphic control identification, when recognizing the graphic control identification, it can be recognized according to all or part of the constituent elements of the graphic control identification; setting the instruction mapping rule for converting the graphic control identification into the corresponding operation instruction; it can also include: setting the method for supplementing or modifying the instruction mapping rule of the graphic control identification, and supplementing or modifying the operation instruction corresponding to a certain graphic control identification by recognizing voice information or graphic information.

[0087] The specific content of the interaction instructions and the auxiliary control instructions will be described in detail later.

[0088] In S204, the target interaction instruction and the target interaction object area are determined based on the graphic control identifier. For example, the target interaction instruction can be extracted from the interaction instruction mapping rule according to the graphic control identifier; the target interaction object area is determined based on the target interaction instruction.

[0089] Among them, the interaction object area refers to a specific area in the written document image, and the image within the specific area or the visually recognizable elements within this area will participate in the specific interaction operation. The interaction object area can be the entire image captured by the imaging device or one or several local areas thereof. When calibrating the interaction object area, it can be described according to its fixed position in the image captured by the imaging device, or according to its relative position to one or several graphic control identifiers in the image captured by the imaging device.

[0090] Furthermore, the interaction object area refers to a range that the imaging device can capture, and the written document image within this range can be interacted with. The interaction object area can be one or multiple. The interaction object area can be the entire written document or a part thereof. Multiple interaction object areas can be within the same written document or in multiple written documents respectively. The interaction object area can be the entire image that the imaging device can obtain or a part thereof.

[0091] In S206, the document image of the written document within the target interaction object area is obtained. Among them, the obtained written document image can be the written document, or can also be a dynamic or static image of the operation process on the written document.

[0092] In S208, the document image is processed based on the target interaction instruction. It can include: performing image recognition on the document image based on the target interaction instruction to generate the document information; performing data processing on the document information.

[0093] Among them, performing image recognition on the document image based on the target interaction instruction to generate the document information includes: determining the target area based on the target interaction instruction; performing image recognition on the image in the target area of the document image to generate the document information.

[0094] Among them, processing the document image based on the target interaction instruction can also be: when the graphic control identifier requires input of parameters, inputting parameter information according to a preset method, and the parameters include at least one of voice data, text data, image data, and gesture data.

[0095] According to the written document processing method of the present disclosure, a graphic control identifier within a preset area is obtained; a target interaction instruction and a target interaction object area are determined based on the graphic control identifier; a document image of a written document within the target interaction object area is obtained; the manner of processing the document image based on the target interaction instruction can implement the multimedia interaction function of the user's written document without relying on a display screen, and the user can also call computer multimedia resources for interaction.

[0096] It should be clearly understood that the present disclosure describes how to form and use specific examples, but the principles of the present disclosure are not limited to any details of these examples. On the contrary, based on the teachings of the content disclosed in the present disclosure, these principles can be applied to many other embodiments.

[0097] Figure 4 is a flowchart of a written document processing method shown according to an exemplary embodiment. The written document processing method 40 is a written document processing method for the scenario as Figure 5 described. In Figure 5 the corresponding schematic diagram, the written document is placed on a flat desk, and the user has previously placed one or more identification objects (graphic control identifiers) to be processed within a preset area on the desk. The written document processing method 40 at least includes steps S402 to S408.

[0098] As Figure 4 shown, in S402, a graphic control identifier is obtained from the graphic control identifier area. It includes: determining the number of graphic control identifier areas by system settings; obtaining the graphic control identifier from all graphic control identifier areas.

[0099] One or several graphic control identifier areas are set within the shooting range of the imaging device. The graphic control identifier area can be all of the images captured by the imaging device or one or several of the areas. After setting the graphic control identifier area, within the graphic control identifier area, the graphic control identifier is identified according to the corresponding mapping rule. Within different graphic control identifier areas, the graphic control identifiers that can be identified can be the same or different. Within different graphic control identifier areas, the interaction instructions corresponding to the same graphic control identifier can be the same or different.

[0100] Among them, obtaining the graphic control identifier from all graphic control identifier areas includes: obtaining an object image of the identification object placed in all graphic control identifier areas; performing image recognition on the object image of the identification object to generate the graphic control identifier.

[0101] In one embodiment, before obtaining the graphic control identifier from the graphic control identifier area, it further includes: generating at least one interaction instruction mapping rule based on one or more graphic control identifiers, at least one graphic control identifier area, and its corresponding interaction instruction; and / or generating at least one auxiliary control instruction mapping rule based on one or more graphic control identifiers, at least one graphic control auxiliary area, and its corresponding auxiliary control instruction.

[0102] Among them, the interaction instruction mapping rule and the auxiliary control instruction mapping rule may specifically include: setting one or more graphic control identifiers; setting an identification method or program for the graphic control identifier. When identifying the graphic control identifier, it can be identified according to all or part of the constituent elements of the graphic control identifier; setting an instruction mapping rule for converting the graphic control identifier into a corresponding operation instruction; it may further include: setting a method for supplementing or modifying the instruction mapping rule of the graphic control identifier, and supplementing or modifying the operation instruction corresponding to a certain graphic control identifier by identifying voice information or graphic information. When supplementing or modifying the existing graphic data according to the information of the graphic control identifier, one or more graphic control auxiliary areas can be set within the shooting range of the imaging device, and graphic information can be added or changed in the graphic control auxiliary area.

[0103] In S404, a document image of the written document is obtained from the interaction object area.

[0104] In S406, a target interaction instruction is determined based on the graphic control identifier. It includes: determining a target interaction instruction based on the graphic control identifier and the number of its corresponding graphic control identifier area.

[0105] Among them, determining a target interaction instruction based on the graphic control identifier and the number of its corresponding graphic control identifier area includes: determining a target interaction instruction mapping rule from the at least one interaction instruction mapping rule based on the number of the graphic control identifier area; extracting a target interaction instruction from the target interaction instruction mapping rule according to the graphic control identifier.

[0106] In S408, the document image is processed based on the target interaction instruction. It includes: performing image recognition on the document image based on the target interaction instruction to generate the document information; performing data processing on the document information.

[0107] Among them, performing image recognition on the document image based on the target interaction instruction to generate the document information includes: determining a target area based on the target interaction instruction; performing image recognition on the image in the target area of the document image to generate the document information.

[0108] Among them, processing the document image based on the target interaction instruction may further include: when the graphic control identifier requires input of parameters, inputting parameter information according to a preset method, where the parameters include at least one of voice data, text data, image data, and gesture data.

[0109] In one embodiment, it further includes: adjusting the area range of the graphic control identifier area and / or the interaction object area and / or the graphic control assistance area; where the graphic control identifier area, the interaction object area, and the graphic control assistance area are within the imaging range of the imaging device.

[0110] Among them, adjusting the area range of the graphic control identifier area and / or the interaction object area and / or the graphic control assistance area includes: obtaining an object image of an identification object placed within the imaging range for marking the area range; performing image recognition on the object image to generate a graphic control identifier for marking the area range; determining a target area range according to the graphic control identifier for marking the area range; and adjusting the area range of the interaction object area and / or the graphic control identifier area and / or the graphic control assistance area based on the target area range.

[0111] Among them, the graphic control identifiers for performing interaction operations that a computer or other intelligent device can achieve on all or part of a written document image, and the corresponding interaction instructions may include:

[0112] Graphic control identifiers that mark the interaction scope of a written document and the corresponding interaction instructions; graphic control identifiers that mark one or more interaction object areas within the interaction scope of a written document and the corresponding interaction instructions; graphic control identifiers that select one or more interaction object areas within the interaction scope of a written document for a certain interaction operation and the corresponding interaction instructions; graphic control identifiers that mark the recognition scope of a graphic control identifier and the corresponding interaction instructions; graphic control identifiers that are used to combine multiple graphic control identifiers into a new graphic control identifier and the corresponding interaction instructions; graphic control identifiers that explain the recognition method of one or more graphic control identifiers or the program mapping rules of the corresponding interaction instructions and the corresponding interaction instructions; graphic control identifiers of the program that recognizes the graphic control identifiers of the written document interaction operation and the corresponding interaction instructions; graphic control identifiers of the program that converts the recognized graphic control identifiers into interaction instructions and the corresponding interaction instructions; graphic control identifiers of the program that executes the interaction instructions and the corresponding interaction instructions; graphic control identifiers that provide relevant parameters or data of the interaction instructions for the interaction operation, and the parameters or data required for the interaction operation can come from one or more data sources; graphic control identifiers that provide the addresses of the execution program and / or data sources of the interaction operation and the corresponding interaction instructions; graphic control identifiers that set the operation permissions of the execution program and / or data sources of the interaction operation and the corresponding interaction instructions; graphic control identifiers that annotate or explain the written document being interacted with or the operation process on the written document and the corresponding interaction instructions; graphic control identifiers that explain or assign values to the attributes of the identified object, and the attributes of the identified object include the size information of the writing medium, and / or all or part of the visually recognizable elements printed or printed on the writing medium, and / or the position of the visually recognizable elements on the medium, and / or the relative position between the visually recognizable elements; graphic control identifiers that precisely locate one or more points on or around the written document and the corresponding interaction instructions, and the graphic control identifiers that precisely locate one or more points on or around the written document are those set to more precisely calibrate the relative positions between multiple points inside or around the written document, the relative positions between one or more points inside or around the written document and the imaging device, and the relative positions between one or more points inside or around the written document and the projection device; graphic control identifiers that correct or distort the acquired written document image and the corresponding interaction instructions; graphic control identifiers that mark the projection feedback display area and the corresponding interaction instructions; graphic control identifiers that correct, distort, move, reverse, or rotate the projection feedback image and the corresponding interaction instructions;Graphic control identifiers for controlling devices participating in interactive operations or devices affecting the interactive surrounding environment, and corresponding interactive instructions; graphic control identifiers for adjusting volume, lighting, or projection brightness, and corresponding interactive instructions.

[0113] The foregoing graphic control identifiers and their corresponding interactive instructions are only for more convenient description of interactive operations. In actual situations, according to the complexity of interactive operations, the foregoing various types of interactive operations can be independent of each other, can be arbitrarily combined, can all exist, or there can be only any one or several types of interactive operations. Between the interactive operations and the graphic control identifiers, there can be a one-to-one and / or one-to-many and / or many-to-many and / or many-to-one correspondence.

[0114] Figure 6 It is a flowchart of a written document processing method shown according to another exemplary embodiment. Figure 6 A detailed description of the specific processing of the graphic control identifier shown in the flow.

[0115] As Figure 6 shown, in S602, obtain the graphic control identifier within the preset area. It is worth mentioning that in the Figure 4 usage scenario, the imaging device obtains the graphic control identifier from the graphic control auxiliary area.

[0116] In S604, based on the graphic control identifier, extract the target auxiliary control instruction in the auxiliary control instruction mapping rule. It is worth mentioning that in the Figure 4 usage scenario, the processing device extracts the target auxiliary control instruction in the at least one auxiliary control instruction mapping rule based on the graphic control identifier and the number of its corresponding graphic control auxiliary area.

[0117] In S606, based on the target auxiliary control instruction, control the processing process or the multimedia device participating in the processing process.

[0118] Among them, when the graphic control identifier requires input parameters, parameter information can be input according to a preset method, and the parameters include at least one of voice data, text data, image data, and gesture data.

[0119] The recognized interactive instruction can be determined according to the preset graphic control identifier and the auxiliary control instruction. During the recognition process of the graphic control identifier, it can be judged whether the graphic control identifier needs to supplement or change the interactive instruction corresponding to a certain graphic control identifier by recognizing voice information or graphic information. If necessary, collect and recognize the voice information or graphic information to supplement or modify the interactive instruction.

[0120] It is also possible to determine whether there is an interaction object area for this graphic control identifier or its corresponding interaction. If there is an interaction object area, obtain an image of the interaction object area or elements within the interaction object area that can be visually recognized as the interaction object of the operation instruction and / or instruction set.

[0121] Figure 7 It is a flowchart of a written document processing method shown according to another exemplary embodiment. Figure 7 The subsequent processing steps of document processing are described in the shown process.

[0122] Such as Figure 7 As shown, in S702, based on the processing result, multimedia information is generated.

[0123] In S704, multimedia operations are performed based on the multimedia information.

[0124] In S706, when the multimedia operation is a projection operation, a display area is determined based on the graphic control identifier.

[0125] In S708, the document image and / or the projection image projected in the display area is corrected.

[0126] In one embodiment, the first relative position of known points on the written document corresponding to the document image can be obtained; the second relative position of the known points in the document image can be obtained; a distortion correction mapping between the written document and the document image is determined based on the first relative position and the second relative position; and the document image is corrected based on the distortion correction mapping. Specifically, for example, the relative positions between multiple points inside or around the written document, and / or the relative positions between one or more points inside or around the written document and the imaging device can be obtained; the images of one or more points inside or around the written document are obtained; according to the above images and relative positions, a distortion correction mapping of one or more points inside or around the written document is obtained. Distortion correction of the image is performed so that the images of one or more points inside or around the written document match the relative positions between multiple points inside or around the written document, and / or the relative positions between one or more points inside or around the written document and the imaging device.

[0127] In one embodiment, the subsequent processing steps of document processing further include: superimposing the projected image on the corrected document image to generate a superimposed image; determining a deviation adjustment mapping of the projected image according to the superimposed image; and correcting the projected image based on the deviation adjustment mapping. Obtain a feedback image or a calibration image and the expected relative positions of one or more points inside or around the written document with known precise positioning on the superimposed image. Specifically, for example, project a feedback image or a calibration image, and obtain a superimposed image of the projection of the feedback image or the calibration image and the document image; determine the deviation adjustment mapping of the feedback image or the calibration image according to the actual relative positions of the projection of the feedback image or the calibration image and the image of one or more points inside or around the written document with known precise positioning on the superimposed image; and adjust the feedback image according to the deviation adjustment mapping so that the projection of the feedback image and one or more points inside or around the written document with known precise positioning are in the expected relative positions on the superimposed image.

[0128] Furthermore, with reference to the positioning results of graphic control identifiers that precisely locate one or more points on or around the written document, use auxiliary lines or auxiliary graphics on the written document display medium for positioning, specifically including: obtaining the precise positioning results of one or more points on or around the written document; obtaining the auxiliary lines or auxiliary graphics on the written document display medium; and obtaining the relative positions of one or more points on the auxiliary lines or auxiliary graphics on the written document display medium and one or more points with known precise positioning results on or around the written document.

[0129] In S710, project the processing result in the display area.

[0130] Those skilled in the art can understand that all or part of the steps of implementing the above embodiments are realized as a computer program executed by a CPU. When the computer program is executed by the CPU, it executes the above functions defined by the above method provided by the present disclosure. The program can be stored in a computer-readable storage medium, which can be a read-only memory, a magnetic disk, an optical disk, etc.

[0131] In addition, it should be noted that the above drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, rather than for limiting purposes. It is easy to understand that the processes shown in the above drawings do not indicate or limit the chronological order of these processes. Additionally, it is also easy to understand that these processes can be executed synchronously or asynchronously in, for example, multiple modules.

[0132] The following is an embodiment of the device of the present disclosure, which can be used to execute the embodiment of the method of the present disclosure. For details not disclosed in the embodiment of the device of the present disclosure, please refer to the embodiment of the method of the present disclosure.

[0133] Figure 8 is a block diagram of a written document processing apparatus shown according to an exemplary embodiment. As Figure 8 shown, the written document processing apparatus 80 can be applied to Figure 2 the scenario described above, including: an identification object 802, an imaging device 804, a processing device 806, and the written document processing apparatus 80 may further include: a multimedia device 808.

[0134] The identification object 802 is used to display a graphic control identifier;

[0135] The imaging device 804 is used to obtain the graphic control identifier within a preset area, and / or obtain a document image of a written document within a target interaction object area;

[0136] The processing device 806 is used to determine a target interaction instruction and the target interaction object area based on the graphic control identifier; and process the document image based on the target interaction instruction; and / or determine a target auxiliary control instruction based on the graphic control identifier; control the processing process or a multimedia device participating in the processing process based on the target auxiliary control instruction;

[0137] The multimedia device 808 is used to generate multimedia information based on a processing result; and perform a multimedia operation based on the multimedia information.

[0138] Figure 9 is a block diagram of a written document processing apparatus shown according to another exemplary embodiment. As Figure 9 shown, the written document processing apparatus 90 can be applied to Figure 4 the scenario described above, including: an identification object 902, an imaging device 904, a processing device 906, and the written document processing apparatus 90 may further include: a multimedia device 908.

[0139] The identification object 902 is used to display a graphic control identifier;

[0140] The imaging device 904 is used to obtain a graphic control identifier from a graphic control identifier area, and / or obtain a graphic control identifier from a graphic control auxiliary area, and / or obtain a document image of a written document from an interaction object area;

[0141] The processing device 906 is used to determine a target interaction instruction based on the graphic control identifier and the number of its corresponding graphic control identifier area; and process the document image based on the target interaction instruction; and / or determine a target auxiliary control instruction based on the graphic control identifier and the number of its corresponding graphic control auxiliary area, and control the processing process or a multimedia device participating in the processing process based on the target auxiliary control instruction;

[0142] The multimedia device 908 is used to generate multimedia information based on the processing result and perform multimedia operations based on the multimedia information.

[0143] The written document processing device according to the present disclosure obtains graphic and text control identifiers within a preset area, determines a target interaction instruction and a target interaction object area based on the graphic and text control identifiers, obtains a document image of a written document within the target interaction object area, and is capable of implementing a multimedia interaction function of a user's written document without relying on a display screen. The user can also call computer multimedia resources for interaction based on the manner of processing the document image according to the target interaction instruction.

[0144] Figure 10 It is a block diagram of an electronic device shown according to an exemplary embodiment.

[0145] Next, refer to Figure 10 to describe the electronic device 1000 according to this embodiment of the present disclosure. Figure 10 The illustrated electronic device 1000 is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present disclosure.

[0146] As Figure 10 shown, the electronic device 1000 is presented in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one storage unit 1020, a bus 1030 connecting different system components (including the storage unit 1020 and the processing unit 1010), a display unit 1040, etc.

[0147] Among them, the storage unit stores program code, and the program code can be executed by the processing unit 1010, so that the processing unit 1010 executes the steps according to various exemplary embodiments of the present disclosure described in this specification. For example, the processing unit 1010 can execute steps such as Figure 2 , Figure 4 , Figure 6 , Figure 7 shown.

[0148] The storage unit 1020 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 10201 and / or a cache storage unit 10202, and may further include a read-only storage unit (ROM) 10203.

[0149] The storage unit 1020 may further include a program / utility 10204 having a set (at least one) of program modules 10205. Such program modules 10205 include, but are not limited to: an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0150] The bus 1030 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus structures.

[0151] The electronic device 1000 may also communicate with one or more external devices 1000' (such as a keyboard, a pointing device, a Bluetooth device, etc.), devices that enable a user to interact with the electronic device 1000, and / or any device (such as a router, a modem, etc.) with which the electronic device 1000 can communicate with one or more other computing devices. Such communication may be through the input / output (I / O) interface 1050. Also, the electronic device 1000 may communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 1060. The network adapter 1060 may communicate with other modules of the electronic device 1000 through the bus 1030. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0152] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by a combination of software and necessary hardware. Therefore, as Figure 11 shown, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiments of the present disclosure.

[0153] The software product may employ any combination of one or more readable media. The readable media can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0154] The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, in which the readable program code is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable storage medium may also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0155] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, executed as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0156] The above computer-readable medium carries one or more programs, which, when executed by the device, cause the computer-readable medium to implement the following functions: obtaining graphic control identifiers within a preset area; determining a target interaction instruction and a target interaction object area based on the graphic control identifiers; obtaining a document image of a written document within the target interaction object area; and processing the document image based on the target interaction instruction. The computer-readable medium can also implement the following functions: obtaining graphic control identifiers from a graphic control identifier area; obtaining a document image of a written document from an interaction object area; determining a target interaction instruction based on the graphic control identifiers; and processing the document image based on the target interaction instruction.

[0157] Those skilled in the art can understand that the above-mentioned modules can be distributed in the device according to the description of the embodiments, or can be correspondingly changed and distributed in one or more devices that are different from the embodiments. The modules of the above embodiments can be combined into one module, or can be further split into multiple sub-modules.

[0158] Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described here can be implemented by software, or can be implemented by the way of software combined with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on the network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0159] The above specifically shows and describes the exemplary embodiments of the present disclosure. It should be understood that the present disclosure is not limited to the detailed structures, setting manners or implementation methods described here; on the contrary, the present disclosure intends to cover various modifications and equivalent settings included in the spirit and scope of the appended claims.

Claims

1. A method for processing written documents, characterized in that, including: obtaining an object image of an identification object for marking a region range placed within an imaging range, where a graphic control identification region, an interaction object region, and a graphic control assistance region are within the imaging range; performing image recognition on the object image to generate a graphic control identification for marking the region range; determining a target region range according to the graphic control identification for marking the region range; adjusting the region range of the interaction object region and / or the graphic control identification region and / or the graphic control assistance region based on the target region range; obtaining the graphic control identification within the graphic control identification region; determining a target interaction instruction and a target interaction object region in an interaction instruction mapping rule based on the graphic control identification within the graphic control identification region, specifically, determining the target interaction instruction based on the graphic control identification and the number of its corresponding graphic control identification region, and determining the target interaction object region based on the target interaction instruction; obtaining a document image of a written document within the target interaction object region; processing the document image based on the target interaction instruction; obtaining the graphic control identification within the graphic control assistance region; extracting a target assistance control instruction from an assistance control instruction mapping rule based on the graphic control identification within the graphic control assistance region, specifically, extracting the target assistance control instruction from at least one of the assistance control instruction mapping rules based on the graphic control identification and the number of its corresponding graphic control assistance region; controlling the processing process or a multimedia device participating in the processing process based on the target assistance control instruction.

2. The method according to claim 1, wherein further including: generating at least one interaction instruction mapping rule based on one or more graphic control identifications, at least one graphic control identification region, and its corresponding interaction instruction; and / or generating at least one assistance control instruction mapping rule based on one or more graphic control identifications, at least one graphic control assistance region, and its corresponding assistance control instruction.

3. The method according to claim 1, characterized in that Obtaining the graphic control identification within the graphic control identification region includes: obtaining an object image of an identification object placed within the graphic control identification region, where the graphic control identification region is within the imaging range of the imaging device; performing image recognition on the object image of the identification object to generate the graphic control identification.

4. The method according to claim 1, characterized in that Determining the target interaction instruction and the target interaction object region in the interaction instruction mapping rule based on the graphic control identification within the graphic control identification region includes: extracting the target interaction instruction from the interaction instruction mapping rule according to the graphic control identification; determining the target interaction object region based on the target interaction instruction.

5. The method according to claim 1, characterized in that, Processing the document image based on the target interaction instruction includes: performing image recognition on the document image based on the target interaction instruction to generate document information; performing data processing on the document information.

6. The method according to claim 1, wherein Processing the document image based on the target interaction instruction further includes: when the graphic control identification requires input of parameters, inputting parameter information according to a preset method, where the parameters include at least one of voice data, text data, image data, and gesture data.

7. A written document processing system, characterized in that, including: An identification object for displaying graphic and text control identifiers; An imaging device for obtaining graphic and text control identifiers from a graphic and text control identifier area, and / or from a graphic and text control assistance area, and / or for obtaining a document image of a written document from an interaction object area; A processing device for obtaining an object image of the identification object placed within an imaging range for marking a range of an area, where the imaging range has a graphic and text control identifier area, an interaction object area, and a graphic and text control assistance area; Performing image recognition on the object image to generate a graphic and text control identifier for marking a range of an area; Determining a target area range based on the graphic and text control identifier for marking a range of an area; Adjusting the area range of the interaction object area and / or the graphic and text control identifier area and / or the graphic and text control assistance area based on the target area range; Obtaining a graphic and text control identifier within the graphic and text control identifier area; Determining a target interaction instruction and a target interaction object area in an interaction instruction mapping rule based on the graphic and text control identifier within the graphic and text control identifier area. Specifically, determining the target interaction instruction based on the graphic and text control identifier and the number of its corresponding graphic and text control identifier area, and determining the target interaction object area based on the target interaction instruction; obtaining a document image of the written document within the target interaction object area; Processing the document image based on the target interaction instruction; Obtaining a graphic and text control identifier within the graphic and text control assistance area; Extracting a target auxiliary control instruction in an auxiliary control instruction mapping rule based on the graphic and text control identifier within the graphic and text control assistance area. Specifically, extracting the target auxiliary control instruction in at least one of the auxiliary control instruction mapping rules based on the graphic and text control identifier and the number of its corresponding graphic and text control assistance area; controlling a processing process or a multimedia device participating in the processing process based on the target auxiliary control instruction.

8. The system according to claim 7, wherein Further comprising: A multimedia device for generating multimedia information based on a processing result; And performing multimedia operations based on the multimedia information.

9. An electronic device, characterized in that, Comprising: One or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, enabling the one or more processors to implement the method according to any one of claims 1 - 6.

10. A computer-readable medium having a computer program stored thereon, characterized in that, The program, when executed by a processor, implements the method according to any one of claims 1 - 6.

Images