Reading auxiliary system and method for associating book object with interactive content

Abstract

The invention discloses a reading assistance system and a reading assistance method which recognize a shooting object arranged in front through an interaction object with a reflecting surface and output interaction content combined with the shooting object. The disclosed reading assistance system includes a reading assistance device that outputs interaction content including an object to be photographed reflected by a reflection object having a reflection surface formed thereon. The reading assistance device detects a reflection object and a photographing object by performing image analysis on an active area image including the photographing object and based on an image analysis result, detects interaction content associated with at least one of the active area image and the reflection object, and outputs interaction content including the photographing object. Therefore, the output interaction content is overlapped with the activity area image.

Description

Technical Field

[0001] The present invention relates to a reading assistance system and method, and more particularly, to a reading assistance system and method for providing additional content related to a physical book to assist a user in reading. Background Art

[0002] Generally, reading refers to the act of acquiring knowledge by reading books that include text, images, etc. In recent years, as people get used to electronic devices such as smartphones and tablets, the frequency of acquiring knowledge by surfing the Internet, watching online videos, etc. has gradually increased compared to using physical books that only consist of text and images (i.e., still images).

[0003] In the publishing industry, physical books are being replaced by multimedia books consisting of text, images, voice, sound effects, videos, etc. However, existing multimedia books provide the required information passively and in a one-dimensional manner, and only provide audio, images, etc. related to the content of the book.

[0004] In response to this, the publishing industry is researching various reading assistance technologies to guide active reading and participation in reading.

[0005] In the above-mentioned background technology, the matters recorded are only used to understand the background of the present invention and may include non-prior art matters known to ordinary technicians in the technical field to which the present invention belongs.

[0006] Prior art literature

[0007] Patent Literature

[0008] Korean Patent No. 10-1587471 (Title: Book system and reading enhancement method using wearable computer to enhance reading). Summary of the invention

[0009] Technical issues

[0010] In view of the above matters, an object of the present invention is to provide a reading assistance system and method that recognizes a photographed object arranged in front by forming an interactive object with a reflective surface and outputs interactive content synthesized with the above photographed object.

[0011] Technical Solution

[0012] In order to achieve the above-mentioned purpose, a reading assistance system of an embodiment of the present invention includes: a reflective object, which is formed with a reflective surface and is configured in an active area; and a reading assistance device, which is configured on a physical surface with an active area, receives a photographed object reflected by the reflective surface of the reflective object, and outputs interactive content including the photographed object, and the reading assistance device includes: an image processing engine, which receives an active area image including the active area, the reflective object and the photographed object reflected by the reflective surface of the reflective object, performs image analysis on the active area image and detects the reflective object and the photographed object based on the image analysis result; a processor, which is connected to the image processing engine for communication, and is used to detect the active area image and interactive content associated with at least one of the reflective objects and output an output signal including the interactive content; and a display, which is connected to the image processing engine and the processor for communication, and is used to output the active area image, and outputs the interactive content including the photographed object detected by the image processing engine in response to the output signal, so that the output interactive content overlaps with the active area image.

[0013] The reflective object includes: a main body, formed with a reflective surface; an upper shell, which is in a frame shape and has a hole that exposes the reflective surface of the main body, and is arranged on the upper surface of the main body; and a lower shell, which is arranged on the lower surface of the main body and is combined with the upper shell. The reflective surface can make the photographed object arranged on the front side of the reading auxiliary device reflected to the reading auxiliary device.

[0014] The processor outputs a synthesis request signal including interactive content, and the image processing engine can detect the interactive content from the synthesis request signal and generate synthetic content that synthesizes the photographed object detected based on the image analysis result and breaks the interactive content.

[0015] The processor may set the synthesized content generated by the image processing engine as the interactive content.

[0016] The processor detects interactive content in a set synthesis area, and the image processing engine can synthesize the photographed object detected based on the image analysis result into the synthesis area to generate the synthesized content.

[0017] The image processing engine detects a first image including a reflective object based on an image analysis result of the active area image, and can detect a partial area of ​​the reflective object as a photographing object based on the image analysis result of the first image.

[0018] The reading auxiliary device may further include a camera for capturing an image of the active area including the active area, the reflective object, and the captured object reflected by the reflective surface of the reflective object, and transmitting the image of the active area to the image processing engine and the display.

[0019] In order to achieve the above-mentioned purpose, the reading assistance method of an embodiment of the present invention includes the following steps: outputting an active area image including an active area and a reflective object configured in the active area through a display; performing analysis on the active area image through an image processing engine and detecting the reflective object and the photographed object reflected by the reflective surface of the reflective object based on the image analysis result; detecting interactive content associated with at least one of the active area image and the reflective object through a processor; outputting an output signal containing the interactive content through the processor; and outputting the interactive content containing the photographed object through the display in response to the output signal, so that the output interactive content overlaps with the active area image.

[0020] The reading assistance method of the embodiment of the present invention may also include the following steps: before the outputting step of the output signal, outputting a synthesis request signal including the interactive content detected in the detecting step of the interactive content through the processor; and responding to the synthesis request signal through the image processing engine to detect the interactive content from the synthesis request signal and generate synthetic content that synthesizes the photographed object detected based on the image analysis result into the interactive content.

[0021] The reading assistance method of the embodiment of the present invention also includes the following steps: before the outputting step of the output signal, the synthesized content generated in the synthesized content generating step is set as interactive content by a processor; in the interactive content detecting step, the interactive content of the set synthesized area is detected; in the synthesized content generating step, the synthesized content can be generated by synthesizing the photographed object detected based on the image analysis result into the synthesized area.

[0022] The step of detecting the photographing object may include the following steps: detecting a first image including a reflective object based on an image analysis result of the active area image; and detecting a partial area of ​​the reflective object as the photographing object based on the image analysis result of the first image.

[0023] The reading assistance method of the embodiment of the present invention may further include the following step: photographing, by a camera, an image of the activity area including the activity area, the reflective object, and the photographed object reflected by the reflective surface of the reflective object.

[0024] Effects of the Invention

[0025] The reading assistance system and reading assistance method of the present invention have the following effects, namely, by forming an interactive object with a reflective surface to identify the photographed object arranged in front and output interactive content that has been synthesized with the above-mentioned photographed object, and can provide various activities such as analysis, reaction activities, shooting, synthesis, etc. through user face recognition. Therefore, compared with the existing technology that simply provides related images, it can provide an environment that allows users to concentrate more on reading.

[0026] Furthermore, as the interactive object with a reflective surface recognizes the photographed object arranged in front and outputs the interactive content synthesized with the photographed object, the reading assistance system and the reading assistance method of the present invention can guide the user who uses the interactive object to actively participate in reading.

[0027] Furthermore, as the interactive object formed with a reflective surface recognizes the photographed object arranged in front and outputs the interactive content synthesized with the photographed object, the reading assistance system and the reading assistance method of the present invention can guide active reading compared to the existing technology that simply provides relevant images.

[0028] The above-mentioned technical effects of the present invention are not limited to the effects mentioned above, and ordinary technicians in the technical field to which the present invention belongs can clearly understand other technical effects not mentioned through the following records. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A diagram for explaining a reading assistance system according to an embodiment of the present invention.

[0030] Figure 2 For illustration Figure 1 Image of a reading aid.

[0031] Figure 3 and Figure 4 For illustrative purposes Figure 1 Diagram of the bracket required for the reading aid.

[0032] Figure 5 and Figure 6 For illustrative purposes Figure 1 Diagram of the connector required for the reading aid.

[0033] Figure 7 For illustration Figure 1 Diagram of the structure of a reading assistance device.

[0034] Figures 8 to 15 For illustration Figure 1 Illustration of the working of a reading aid.

[0035] Fig.16 For illustration Figure 1 Diagram of the structure of a service server.

[0036] Fig.17 and Fig.18 For illustration Figure 1 Diagram of the working of the service server.

[0037] Fig.19 For illustration Figure 1 Diagram of the structure of the database.

[0038] Fig. 20 For illustration Fig.19 Diagram of the book identifier repository.

[0039] Fig.21 For illustration Fig.19 Diagram of the book information repository.

[0040] Fig. 22 For illustration Fig.19 Diagram of the reading assistance information repository.

[0041] Figure 23 to Figure 26 The flowchart is used to illustrate the reading assistance method of associating a book object with interactive content according to an embodiment of the present invention.

[0042] Description of reference numerals:

[0043] 100: Reading aids

[0044] 110: Applications

[0045] 120: Processor

[0046] 130: Camera

[0047] 140: Display

[0048] 150: Image processing engine

[0049] 160: First communication module

[0050] 170: Repository

[0051] 180: Memory

[0052] 200: Service server

[0053] 210: Second communication module

[0054] 220: Book Recognition Engine

[0055] 230: Book information detection engine

[0056] 240: Reading auxiliary information detection engine

[0057] 250: Control Engine

[0058] 300: Database

[0059] 310: Query Processing Engine

[0060] 320: Book Identifier Repository

[0061] 330: Book Information Repository

[0062] 340: Reading Aid Information Repository

[0063] 400: Bracket

[0064] 410: Ontology

[0065] 412: Fixtures

[0066] 420: Bracket

[0067] 430: Combined axis

[0068] 500: Connector

[0069] 520: Shell

[0070] 540: Reflector

[0071] 600: Reflection Object

[0072] 620: Ontology

[0073] 640: Upper housing

[0074] 642: Hole

[0075] 660: Lower housing

[0076] AS: Activity Area

[0077] ASV: Active Region Video

[0078] IO: Interaction Objects

[0079] IOI: Interactive Object Image

[0080] IC: Interactive Content

[0081] BO: Book Object

[0082] BC: Book Content

[0083] HL: Horizontal straight line

[0084] VL: Vertical Line

[0085] CSG: Cover Shoot Guide

[0086] SA: Shootable Area

[0087] FS: Subject

[0088] RS: Reflective surface

[0089] CC: Composite Content

[0090] CA: Composition Area DETAILED DESCRIPTION

[0091] Hereinafter, the preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, so that a person skilled in the art of the present invention can easily implement the technical idea of ​​the present invention. First, in the process of assigning reference numerals to the structural elements of each drawing, it should be noted that even if the same structural elements are shown in different drawings, the same reference numerals are assigned as much as possible. In addition, in the process of describing the present invention, when it is determined that the specific description of the known structure or function may confuse the main purpose of the present invention, its detailed description will be omitted.

[0092] It should be understood by those skilled in the art that the structures disclosed in this specification and the various blocks, devices or operations illustrated in the related descriptions can be implemented by electronic hardware, computer software or a combination thereof. Such blocks, devices or operations can be implemented or executed by a processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC) or an application-specific standard product (ASSP), a field-programmable gate array (FPGA) or other programmable logic device, individual gate or transistor logic, individual hardware structural elements or any combination designed to produce the structures disclosed in this specification. The processor can be implemented by a combination of computing devices, for example, a combination of a digital signal processor and a microprocessor, a plurality of microprocessors, more than one microprocessor combined with a digital signal processor core, or any other structure. Computer software or programs may be stored in a random access memory (RAM), a read-only memory (ROM), a non-volatile random access memory such as a flash random access memory, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a read-only compact disk (CD-ROM), or any other form of storage medium known in the technical field to which the present invention belongs, and may include machine-readable instructions executed by a logic element array such as a processor or a digital signal processor. The exemplified storage medium may be combined with a processor, and the processor may read information from the storage medium and record the information in the storage medium. Alternatively, the storage medium may be formed as one with the processor. The processor and the storage medium may be located in a dedicated integrated circuit. The dedicated integrated circuit may be located in a user terminal. Alternatively, the processor and the storage medium may exist separately as structural elements in a user terminal.

[0093] Reference Figure 1The reading assistance system of the embodiment of the present invention includes a reading assistance device 100, a service server 200 and a database 300. Figure 1 For the convenience of explaining the embodiment of the present invention, the service server 200 and the database 300 are shown separately, but the present invention is not limited thereto. The service server 200 and the database 300 may be formed by one server, or the service server 200 and the database 300 may be implemented as modules in the reading assistance device 100 .

[0094] The reading assistance device 100 may be a terminal with a photographing function. As an example, the reading assistance device 100 may be a smart phone, a tablet computer, a laptop computer, etc. equipped with a camera 130 capable of photographing images. The reading assistance device 100 may be connected to the service server 200 and the database 300 via a network, and may send and receive various information for providing reading assistance services.

[0095] Reference Figure 2 The reading assistance device 100 captures an activity surface (AS) for configuring a book object (BO) through the camera 130. The activity surface AS refers to a partial area on a physical surface for configuring the reading assistance device.

[0096] The reading assistance device 100 can output an active area image ASV including book contents (BC) through the display 140. The active area AS is a part of the bottom surface area where the reading assistance device and the book object BO are configured, including the area where the book object BO is configured. The active area image ASV refers to an image of the active area AS captured by the camera 130. The book object BO refers to a physical book, and the book contents BC refers to the content displayed on the display 140, including texts, images, etc. included in each page of the image object BO.

[0097] If an interactive object (IO) is recognized in the active area image ASV, the reading auxiliary device 100 outputs interactive contents (IC) corresponding to the recognized interactive object IO through the display 140 .

[0098] When the user holds the reading assistance device 100, the reading assistance device 100 may slide or move. In this case, since the reading assistance device 100 only captures a portion of the active area AS, the display 140 may output a text or image of the book object BO that omits a portion of the book object BO, or the active area image ASV may become blurred and unrecognizable.

[0099] In this regard, refer to Figure 3 and Figure 4 , the reading auxiliary device 100 can photograph the activity area AS when hung on the bracket 400 .

[0100] Usually, the reading auxiliary device 100 is equipped with a camera 130 with a standard viewing angle (40 to 50 degrees). In order for the camera 130 with a standard viewing angle to capture all the active areas AS, the camera 130 should be spaced from the active areas AS by a set distance or more.

[0101] To this end, the bracket 400 is composed of a body 410 and a bracket 420, and the body 410 includes a fixing device 412 for supporting the lower part of the reading auxiliary device 100. The fixing device 412 is arranged at a position separated from the active area AS by a set height H or more, and as the bracket 400 separates the camera 130 of the reading auxiliary device 100 from the active area AS by a set distance or more, the camera 130 can capture all the active area AS.

[0102] The reading auxiliary device 100 includes a front surface on which the display 140 is arranged, a back surface facing the front surface, and four side surfaces. The lower portion of the reading auxiliary device 100 refers to a side surface of the four side surfaces of the reading auxiliary device 100 that faces the active area AS when the reading auxiliary device 100 is hung on the bracket 400. The set height H varies with the structure of the reading auxiliary device 100. Therefore, the embodiment of the present invention does not limit the value of the set height H.

[0103] Furthermore, in order to enable the camera 130 of the reading auxiliary device 100 to capture all the active areas AS, the body 410 is combined with the bracket 420 to be able to rotate around the coupling axis 430. The reading auxiliary device 100 is hung on the front surface of the body 410, and the bracket 420 can rotate toward the back of the body 410 around the coupling axis 430. In this case, the angle θ (included angle) between the body 410 and the bracket 420 should be within the set angle range so as to ensure the viewing angle of capturing all the active areas AS.

[0104] The angle refers to the angle between the body 410 and the bracket 420 when the bracket 420 is rotated and arranged on the back of the body 410. The set angle may vary with the structure of the reading auxiliary device 100, the body 410 and the bracket 420, and therefore, the embodiment of the present invention does not limit the value of the set angle.

[0105] The reading auxiliary device 100 captures the active area AS through the camera 130 (i.e., the front camera) in order to allow the user to view the active area image ASV. In general, since the camera 130 of the reading auxiliary device 100 can capture the front direction, it cannot capture the active area AS.

[0106] In this regard, the reading auxiliary device 100 captures the active area AS in a state where the adaptor 500 is mounted. The adaptor 500 includes a housing 520 and a reflector 540 disposed inside the housing 520 .

[0107] The housing 520 may be made of an elastic material, and when combined with the reading auxiliary device 100, it is used to prevent the reading auxiliary device 100 from being damaged and maintain the fastening force. The housing 520 is installed on the upper part of the reading auxiliary device 100, overlapping with the camera 130 arranged on the front surface of the reading auxiliary device 100. The upper part of the reading auxiliary device 100 refers to one of the four side surfaces of the reading auxiliary device 100, which is opposite to the side surface arranged toward the active area AS when the reading auxiliary device 100 is hung on the bracket 400.

[0108] The reflector 540 is disposed inside the housing 520 to reflect the active area AS toward the camera 130 of the reading assistance device 100. Inside the housing 520, the reflector 540 forms an acute angle with the upper surface of the reading assistance device 100 and is disposed facing the camera 130 of the reading assistance device 100.

[0109] Reference Figure 5 and Figure 6 The connector 500 is inserted and coupled to the area where the camera 130 of the reading auxiliary device 100 is configured. In this case, the connector 500 prevents the camera 130 from photographing other areas through the housing 520 and reflects the active area AS to the camera 130 through the reflector 540.

[0110] On the other hand, the reading auxiliary device 100 may be formed with a guide device for guiding the connector 500 to be installed at an accurate position. The guide device is configured on the reading auxiliary device 100 and may be formed in one or a combination of two or more of various forms such as protrusions and grooves. In this case, the guide device may be formed as a whole with the housing 520 of the reading auxiliary device 100. The housing 520 of the connector 500 may be formed with protrusions, grooves, etc., and when combined with the reading auxiliary device 100, the housing 520 corresponds to the guide device in the area overlapping with the guide device.

[0111] When hung on the bracket 400, the reading assistance device 100 can photograph the active area AS where the book object BO is arranged through the connector 500. The reading assistance device 100 outputs interactive content related to the book content BC using the active area image ASV generated based on the photographing result. In this case, the reading assistance device 100 outputs the interactive content IC to overlap (cover) the active area image ASV.

[0112] Among them, the interactive content IC is the content that outputs additional images to the real-time image, such as augmented reality (AR) content, mixed reality (MR) content, extended reality (XR) content, etc.

[0113] Reference Figure 7 The reading auxiliary device 100 includes an application 110, a processor 120, a camera 130, a display 140, an image processing engine 150, a first communication module 160, a storage 170, and a memory 180. The application 110, the processor 120, the camera 130, the display 140, the image processing engine 150, the first communication module 160, the storage 170, and the memory 180 transmit and receive signals and data via a bus.

[0114] In order to use the reading assistance service, the user drives the application 110 provided in the reading assistance device 100. That is, after driving the reading assistance device 100, the user can execute the application 110 supporting the reading assistance service through operations such as touching, clicking, and double-clicking.

[0115] When the user activates the application 110, a camera drive request signal may be output to the processor 120. The processor 120 outputs a shooting start signal to the camera 130 in response to the camera drive request signal. The camera 130 starts shooting the active area AS in response to the shooting start signal of the processor 120, and transmits an acknowledgment character (Ack) signal to the application 110. The camera 130 generates an active area image ASV of the active area AS, and transmits it to the display 140 in real time. The display 140 outputs the active area image ASV received from the camera 130 in real time.

[0116] In order to shoot the cover of the image object BO as the reading assistance service object, the application 110 transmits a guide output signal to the processor 120 in response to the confirmation character signal of the camera 130. The processor 120 transmits a cover shooting guide output signal to the display 140 in response to the confirmation character signal of the camera 130 and the cover guide output signal of the application 110. The display 140 outputs a cover shooting guide CSG in response to the guide output signal to overlap (cover) the active area image ASV.

[0117] Reference Figure 8 The display 140 outputs a cover shooting guide CSG to overlap (cover) the active area image ASV. The cover shooting guide CSG includes a horizontal straight line HL and a vertical straight line VL connecting the center point of the screen, and a cover identification area.

[0118] Reference Fig. 9 If the user configures the book object BO so that the book object BO is located in the cover recognition area output by the display 140 , the camera 130 captures the area where the book object BO is configured, and the display 140 outputs the active area image ASV captured by the camera 130 .

[0119] The image processing engine 150 performs image recognition on an area corresponding to the cover recognition area of ​​the cover shooting guide CSG in the entire area of ​​the active area image ASV. The image processing engine 150 generates a cover image based on the image recognition result, and the memory 180 temporarily stores the cover image generated by the image processing engine 150.

[0120] The processor 120 generates a book recognition request signal including the cover image stored in the memory 180 and transmits it to the first communication module 160. The first communication module 160 transmits the received book recognition request signal to the service server 200 through the network.

[0121] The first communication module 160 receives a book recognition response corresponding to the book recognition request signal. The first communication module 160 receives the book recognition response from the service server 200, and the book recognition response includes book information corresponding to the book object BO. In this case, the first communication module 160 receives the book information including the book name and the interactive object information from the service server 200. As an example, when performing book reading assistance, the interactive object information includes the interactive object image IOI of the interactive object IO.

[0122] When the cover image is not the cover of the book object BO or is not the book object BO registered in the service server 200, the first communication module 160 may also receive a book identification response including a book identification error message. The first communication module 160 transmits the book identification response received from the service server 200 to the processor 120.

[0123] When the book recognition response includes a book recognition error message, the processor 120 transmits a guide output signal to the display 140. In response to the guide output signal, the display 140 outputs a cover shooting guide CSG to overlap (cover) the active area image ASV. In this case, the display 140 may also output a replacement guide, a position movement guide, etc. of the book object BO together with the cover shooting guide CSG.

[0124] When the book recognition response includes the book information, the processor 120 transmits a book information output signal to the display 140. In response to the book information output signal, the display 140 outputs the service start confirmation object of the book object BO together with the book information to overlap (cover) the active area image ASV.

[0125] If the user selects a service start confirmation object among the service start confirmation objects displayed by the reading assistance device 100 , the application 110 transmits the reading assistance service start confirmation object to the processor 120 .

[0126] The processor 120 detects book information based on the book identification response in response to the reading assistance service start confirmation object. The processor 120 generates a reading assistance information signal including the detected book information and transmits it to the first communication module 160. The first communication module 160 transmits the received reading assistance information signal to the service server 200 through the network.

[0127] The first communication module 160 receives the reading assistance information corresponding to the reading assistance information signal. In this case, the first communication module 160 receives the reading assistance information and transmits it to the processor 120. The reading assistance information includes a page image for identifying a page of the book object BO, an interactive object image IOI associated with the page of the book object BO, an additional image matching the page of the book object BO or the interactive object image IOI, image deformation information, etc.

[0128] The processor 120 stores the reading assistance information received through the first communication module 160 in the storage 170. In this case, the storage 170 maintains the storage state of the reading assistance information corresponding to the book object BO until the reading assistance of the book object BO is completed.

[0129] On the other hand, the reading auxiliary information includes images, etc., so it takes a specified time to receive (download) it from the service server 200. In this regard, the processor 120 transmits the interactive object information output information included in the book information to the display 140 for the preparation request of the interactive object IO.

[0130] like Fig.10 As shown, the display 140 outputs a signal outputting an interactive object preparation request screen including the interactive object information included in the book information in response to the interactive object information. In this case, the display 140 detects the interactive object image IOI based on the book information included in the book recognition response, and outputs the interactive object preparation request screen in which the detected interactive object image IOI is arranged according to the specified conditions. Thus, the display 140 outputs the interactive object preparation request screen until the reception of the reading auxiliary information is completed.

[0131] If the reading auxiliary information is received, the processor 120 detects the interactive content IC based on the active area image ASV and the page image. That is, the processor 120 detects the page of the book object BO by comparing the active area image ASV captured by the camera 130 with the page image of the reading auxiliary information. The processor 120 transmits a page detection signal including the detected page to the storage 170.

[0132] The storage 170 detects a page image corresponding to the page included in the page detection signal from the pre-stored reading auxiliary information. The storage 170 detects the reading auxiliary information including the detected page image. The page image is not only an image of the photographed page, but also information included in the page, such as a specific object image, a page number, and words included in the page.

[0133] The storage 170 transmits the detected reading assistance information to the processor 120 , and the processor 120 detects the additional image included in the received reading assistance information as the interactive content IC. The processor 120 transmits the interactive content output signal including the detected interactive content IC to the display 140 .

[0134] The display 140 outputs the interactive content IC to overlap (cover) the active area image ASV in response to the interactive content output signal of the processor 120. In this case, the display 140 can output the interactive content IC to the entire screen, or output the interactive content IC to a portion of the screen where the active area image ASV is output.

[0135] Reference Fig.11 The reading auxiliary device 100 is configured to capture the active area AS at the lower end through the connector 500 formed with the reflector 540 . Therefore, since the imageable area SA is limited by the active area AS, the front object FS cannot be captured.

[0136] Therefore, in order to photograph the front photographing subject FS using the reading auxiliary device 100 , it is necessary to photograph the photographing subject FS after the connector 500 is separated.

[0137] When the connector 500 is disconnected, the reading assistance device 100 is likely to lose recognition of the book object BO, which causes inconvenience to the user because the book object BO recognition step needs to be performed again in order to continue using the service of the reading assistance device 100.

[0138] In this regard, the reading assistance system according to the embodiment of the present invention includes a reflective object 600 having a reflective function and used to photograph the front photographing object FS when the connector 500 is not separated from the reading assistance device 100 .

[0139] Reference Fig.12The reflective object 600 has a reflective surface RS and is disposed in the active area AS. The reflective object 600 reflects the front photographing object FS to the reading auxiliary device 100 through the reflective surface RS. The reading auxiliary device 100 photographs the photographing object FS reflected by the reflective object 600 and outputs interactive content IC including the photographing object FS.

[0140] Reference Fig.13 , the reflective object 600 includes a body 620, an upper shell 640 and a lower shell.

[0141] The body 620 is a base material having a reflective surface RS, and can be made of a material capable of exhibiting a mirror effect, such as a mirror, plastic, etc. In this case, as an example, the reflective surface RS is an upper surface of the body 620, so that the photographic object FS disposed in front of the reading auxiliary device 100 is reflected to the connector 500, and the reflector 540 of the connector 500 reflects the photographic object FS reflected by the reflective surface RS to the camera 130 of the reading auxiliary device 100.

[0142] The upper housing 640 is in a frame shape and is disposed on the upper part (upper surface) of the body 620. The upper housing 640 is in a frame shape and has a hole RS formed therein for exposing the reflective surface RS of the body 620. The lower housing 660 is disposed on the lower part (lower surface) of the body 620. The upper housing 640 and the lower housing 660 are combined to accommodate the body 620.

[0143] The reading assistance device 100 outputs the interactive content IC including the photographed object FS reflected by the reflective object 600. As an example, the reading assistance device 100 synthesizes the photographed object FS into one of the objects included in the interactive content IC and outputs the result.

[0144] When the user places the reflective object 600 in the active area AS, the camera 130 captures the active area AS and generates an active area image ASV including the reflective object 600 .

[0145] The camera 130 transmits the active area image ASV to the display 140 and the image processing engine 150 .

[0146] The display 140 outputs the active area image ASV received from the camera 130 .

[0147] The image processing engine 150 performs image analysis on the active area image ASV and detects the reflective object 600 and the photographic object FS based on the image analysis result.

[0148] The image processing engine 150 detects the first image including the reflective object 600 based on the image analysis result of the active area image ASV. That is, the image processing engine 150 detects the area where the reflective object 600 is arranged in the active area image ASV and detects it as the first image. The image processing engine 150 performs image analysis on the first image and detects a partial area of ​​the reflective object 600 as the photographing object FS based on the image analysis result. In this case, as an example, the image processing engine 150 detects the positioning area of ​​the reflective surface RS in the entire area of ​​the reflective object 600 as the photographing object FS.

[0149] The image processing engine 150 transmits an interactive content IC detection request including the detected reflective object 600 to the processor 120 .

[0150] The processor 120 detects the reflective object 600 from the interactive content IC detection request in response to the interactive content IC detection request. The processor 120 transmits the additional image detection request including the detected reflective object 600 to the storage repository 170.

[0151] The repository 170 detects the reading assistance information associated with the reflective object 600 of the additional image detection request and detects the additional image of the reading assistance information. The repository 170 generates a detection response including the detected additional image and transmits it to the processor 120.

[0152] The processor 120 detects the additional image from the detection response of the storage 170 , and sets the detected additional image as the interactive content IC. The processor 120 generates a synthesis request signal including the interactive content IC and transmits the signal to the image processing engine 150 .

[0153] In response to the synthesis request signal, the image processing engine 150 generates synthesized content CC by synthesizing the subject FS into the interactive content IC. That is, the image processing engine 150 detects the interactive content IC from the synthesis request signal, and synthesizes the subject FS detected based on the image analysis result into the interactive content IC to generate the synthesized content CC.

[0154] As an example, refer to Fig.14 The interactive content IC includes a girl object wearing a Korean traditional dress. When the user's face is detected as the shooting object FS, the image processing engine 150 synthesizes the user's face as the shooting object FS into the face area of ​​the girl object to generate a synthesized content CC.

[0155] The image processing engine 150 transmits the synthesized content CC to the processor 120 , and the processor sets the synthesized content CC as the interactive content IC. The processor 120 transmits an output signal including the interactive content IC to the display 140 .

[0156] In response to the output signal, the display 140 outputs the interactive content IC including the photographic subject FS detected by the image processing engine 150. The display 140 outputs the interactive content IC to overlap the active area image ASV.

[0157] On the other hand, the processor 120 may also detect the interactive content IC in which the composite area CA has been set. A partial area including the interactive content IC may be set as an object of the composite area CA.

[0158] The image processing engine 150 detects the interactive content IC from the synthesis request signal of the processor 120 , and synthesizes the photographic subject FS detected based on the image analysis result into the synthesis area CA to generate the synthesized content CC.

[0159] As an example, refer to Fig.15 The interactive content IC includes a girl object whose face area is set as a synthesis area CA. When the user's face is detected as the shooting subject FS, the image processing engine 150 synthesizes the user's face as the shooting subject FS into the synthesis area CA of the girl object to generate the synthesized content CC.

[0160] The service server 200 is a server that provides reading assistance services. The service server 200 detects book information, interactive object information, interactive content IC, etc. from the database 300 based on a signal from the reading assistance device 100, and transmits the detected information to the reading assistance device 100.

[0161] Reference Fig.16 The service server 200 includes: a second communication module 210 for communicating with the reading auxiliary device 100; a book recognition engine 220 for detecting a book identifier from a database 300 in response to a book identifier detection request from the reading auxiliary device 100; a book information detection engine 230 for detecting book information from the database 300 based on the book identifier; a reading auxiliary information detection engine 240 for detecting reading auxiliary information based on the book recognition information; and a control engine 250 for controlling the second communication module 210, the book recognition engine 220, the book information detection engine 230, and the reading auxiliary information detection engine 240 in order to provide a reading auxiliary service. The control engine 250, the book recognition engine 220, the book information detection engine 230, and the reading auxiliary information detection engine 240 may also be constituted and operated by one processor 120.

[0162] Reference Fig.17 , the second communication module 210 receives a book recognition request signal from the reading auxiliary device 100. The second communication module 210 receives the book recognition request signal including the cover image of the book object BO and transmits it to the control engine 250.

[0163] The control engine 250 detects the cover image based on the received book recognition request signal. The control engine 250 generates a book identifier detection signal including the detected cover image and transmits it to the book recognition engine 220.

[0164] The book recognition engine 220 detects the book identifier from the cover image in response to the book identifier detection signal. That is, the book recognition engine 220 detects the cover image based on the book identifier detection signal received from the control engine 250. The book recognition engine 220 detects the object image included in the cover image by performing image recognition on the detected cover image. The book recognition engine 220 can also detect the object image composed of text such as the book name, publisher, and book group name included in the cover image through OCR text recognition.

[0165] The book recognition engine 220 generates query data including the detected object image and transmits it to the database 300. As an example, the object image includes one or more of the book name, cover structure image, publisher, and book group name included in the cover image.

[0166] The book recognition engine 220 receives the query response from the database 300 as a response to the query data. In this case, the book recognition engine 220 receives the query response, including book information such as a book identifier. As an example, the book identifier refers to a unique identifier assigned to each book object BO provided by the reading auxiliary service.

[0167] The book recognition engine 220 detects a book identifier based on the query response received from the database 300, and transmits the detected book identifier to the control engine 250. The control engine 250 generates and transmits to the book information detection engine 230 a book information detection request message including the received book identifier.

[0168] The book information detection engine 230 detects book information from the database 300 in response to the book information detection request signal of the control engine 250. The book information detection engine 230 detects the book identifier based on the book information detection request signal. The book information detection engine 230 generates query data including the detected book identifier and transmits it to the database 300.

[0169] Among them, Fig.21 In the example, although the book information detection engine 230 generates query data based on the book identifier detected by the image recognition of the book recognition engine 220, it is not limited to this, and the cover image can also be used as query data to detect book information. In this case, the book recognition engine 220 is not a component of the service server 200.

[0170] The book information detection engine 230 receives the query response from the database 300 as a response to the query data and transmits it to the control engine 250. In this case, the book information detection engine 230 receives the query response including image information, including the book name and interactive object information. As an example, when using the book reading assistance, the interactive object information includes the interactive object image IOI of the interactive object IO.

[0171] The control engine 250 detects book information based on the query response received from the book information detection engine 230. In this case, the control engine 250 detects book information including the book name and interactive object information based on the query response. The control engine 250 generates a book identification response including the detected book information, and transmits a book identification response signal including the book identification response to the second communication module 210.

[0172] The second communication module 210 detects the book recognition response based on the book recognition response signal. The second communication module 210 transmits the book recognition response to the reading auxiliary device 100 that transmitted the book recognition request signal.

[0173] On the other hand, when the cover image is not the cover of the book object BO or the book object BO registered in the service server 200, a query response of the query data cannot be generated, or the book information detection engine 230 may receive a query response without a detection result. In this case, the control engine 250 generates a book recognition response including a book recognition error message, and the second communication module 210 transmits the book recognition response including the book recognition error message to the reading auxiliary device 100 that transmits the book recognition request signal.

[0174] Reference Fig.18 , the second communication module 210 receives the reading assistance information signal from the reading assistance device 100 . The second communication module 210 receives the reading assistance information signal including the book information and transmits it to the control engine 250 .

[0175] The control engine 250 detects book information based on the received reading auxiliary information signal. The control engine 250 generates a reading auxiliary information detection signal including the detected book information and transmits the signal to the reading auxiliary information detection engine 240.

[0176] The reading auxiliary information detection engine 240 detects interactive content information from the database 300 in response to the reading auxiliary information detection signal of the control engine 250. The reading auxiliary information detection engine 240 detects book information based on the reading auxiliary information detection signal, and detects a book identifier based on the detected book information. The reading auxiliary information detection engine 240 generates query data including the detected book identifier and transmits it to the database 300. The reading auxiliary information detection engine 240 receives a query response from the database 300 as a response to the query data and transmits it to the control engine 250.

[0177] In this case, the reading auxiliary information detection engine 240 receives a query response including reading auxiliary information, wherein the reading auxiliary information includes a page image for identifying a page of a book object BO, an interactive object image IOI associated with a page of the book object BO, an additional image matching the page of the book object BO or the interactive object image IOI, etc.

[0178] The control engine 250 detects the reading assistance information based on the query response received from the reading assistance information detection engine 240. The control engine 250 generates and transmits to the second communication module 210 a reading assistance information transmission signal including the detected reading assistance information.

[0179] The second communication module 210 detects the reading auxiliary information based on the reading auxiliary information transmission signal. The second communication module 210 transmits the reading auxiliary information to the reading auxiliary information device 100 that transmits the reading auxiliary information signal.

[0180] The database 300 is composed of a server storing various information for providing book supplementary services. The database 300 stores book information, interactive object information, interactive content IC, etc., and can detect information corresponding to the signal of the service server 200 and transmit it to the service server 200.

[0181] Reference Fig.19 The database 300 includes: a query processing engine 310, which generates a query response including information detected from the repository 170 based on the query data received from the service server 200 and transmits it to the service server 200; a book identifier repository 320, which is used to store book identifiers associated with book objects BO; a book information repository 330, which is used to store book information associated with book identifiers; and a reading auxiliary information repository 340, which is used to store reading auxiliary information associated with book identifiers.

[0182] The query processing engine 310 detects a book identifier from the book identifier repository 320 based on the query data, generates a query response including the detected book identifier, and transmits it to the service server 200 .

[0183] The query processing engine 310 receives query data including an object image including a cover image. In this case, the query processing engine 310 receives query data including an object image including one or more of a book name, a cover structure image, a publisher, and a book group name.

[0184] The query processing engine 310 detects an object image from the query data. The query processing engine 310 detects a book identifier from the book identifier repository 320 based on the object image. The query processing engine 310 detects a book identifier associated with the detected object image. The query processing engine 310 generates a query response including the detected book identifier and transmits it to the service server 200.

[0185] As an example, refer to Fig. 20 The book identifier repository 320 stores the book identifiers assigned to each book object BO that can implement reading auxiliary services. The book identifier repository 320 stores the book identifiers by associating the book name (subject image), book group name (group image), publisher (publisher image), etc. with the book identifiers.

[0186] Among them, Fig.21 In the above description, although the book identifier storage 320 is shown and described as storing the book name, book group name, publisher, etc. in the form of an image, the present invention is not limited thereto. The book identifier storage 320 may also be stored in the form of a text.

[0187] When the query processing engine 310 receives query data, if the query data includes an object image corresponding to "Subject Image #1" as a book name, "a0001" associated with the same book name may be detected as a book identifier.

[0188] When the query processing engine 310 receives query data, if the query data includes an object image corresponding to "Group Image #1" as a book group name and "Publisher B Image" as a publisher, "a0002" associated with the same book group name and publisher can be detected as a book identifier.

[0189] The query processing engine 310 may also receive query data including a cover image. The query processing engine 310 detects a book identifier associated with the cover image of the query data from the book identifier repository 320. The query processing engine 310 generates a query response including the detected book identifier and transmits it to the service server 200.

[0190] The query processing engine 310 detects book information from the book information repository 330 based on the query data, generates a query response including the detected book information, and transmits the query response to the service server 200 .

[0191] The query processing engine 310 receives query data including a book identifier from the service server 200 (i.e., the book information detection engine 230). The query processing engine 310 detects book information associated with the book identifier of the query data from the book information repository 330. In this case, the query processing engine 310 receives query data including a cover image from the service server 200, and may also detect book information associated with the cover image from the book information repository 330. As an example, the book information includes a book name and interactive object information (i.e., interactive object image 101).

[0192] Reference Fig.21 The book information repository 330 associates and stores the book identifier, the book name, and the interactive object image IOI.

[0193] When the query processing engine 310 receives query data, if the query data includes a book identifier "a0002", the book name "Subject #2" and interactive object images IOI#1, IOI#2 and IOI#5 associated with the book identifier "a0002" are detected.

[0194] The query processing engine 310 generates book information including the detected “topic #2”, interactive object images 101#1, 101#2, and 101#5, and transmits a query response including the generated book information to the service server 200.

[0195] The query processing engine 310 detects reading assistance information from the reading assistance information repository 340 based on the query data, generates a query response including the detected reading assistance information, and transmits the query response to the service server 200 .

[0196] The query processing engine 310 receives query data including a book identifier from the service server 200 (ie, the book auxiliary information detection engine 240).

[0197] The query processing engine 310 detects a book identifier from the query data. The query processing engine 310 detects reading auxiliary information from the reading auxiliary information repository 340 based on the book identifier.

[0198] Reference Fig. 22 The reading auxiliary information repository 340 associates the page image, the interactive object image IOI, and the additional image with the book identifier for storage.

[0199] The page image is a standard image for identifying a page of the book object BO, is associated with the book identifier and the page, and is stored in the book auxiliary information storage repository 340 .

[0200] The interactive object image IOI is a standard image used to identify the interactive object IO located on the book object BO, is associated with the book identifier and the page, and is stored in the reading auxiliary information repository 340.

[0201] The additional image is an augmented reality content, mixed reality content, extended reality content, etc. associated with the interactive object 10, and is associated with the book identifier, page, and interactive object image 101 and stored in the reading auxiliary information repository 340. In this case, the additional image can be an image of a page that is not associated with the interactive object 10 but only associated with the book object BO, and is associated with the book identifier and page and stored in the reading auxiliary information repository 340. The additional image is actual image data, or can be a web page address storing actual image data.

[0202] When receiving query data including the book identifier "a0001", the query processing engine 310 detects all reading auxiliary information, including at least one of the page, page image, interactive object image IOI and additional image associated with the book identifier "a0001". The query processing engine 310 transmits the detected reading auxiliary information to the service server 200.

[0203] On the other hand, when there is no detection result corresponding to the query data of the service server 200 , the query processing engine 310 transmits a query response including “NULL” to the service server 200 .

[0204] Hereinafter, a reading assistance method in which a book object BO is associated with interactive content IC according to an embodiment of the present invention will be described with reference to the accompanying drawings. Figure 23 to Figure 25 It is a flow chart for illustrating a reading assistance method in which a book object BO is associated with interactive content IC according to an embodiment of the present invention.

[0205] First, the reading assistance device 100 is in a network connection state. In order to use the reading assistance service, the user drives the application 110 set in the reading assistance device 100. That is, after driving the reading assistance device 100, the user can execute the application 110 supporting the reading assistance service by touching, clicking, double-clicking, etc.

[0206] When the user activates the application 110 , the application 110 outputs a camera driving request signal to the processor 120 (step S111 ), and the processor 120 outputs a shooting start signal to the camera 130 and the display 140 in response to the camera driving request signal (step S112 ).

[0207] The camera 130 responds to the shooting start signal to start shooting the activity area AS (step S113 ). In this case, if the activity area AS is shot normally, the camera 130 outputs a confirmation character signal of the shooting start signal to the application 110 and the processor 120 .

[0208] The display 140 responds to the shooting start signal to output the active area image ASV shot by the camera 130 in real time (step S114 ).

[0209] The application 110 receiving the confirmation character signal from the camera 130 outputs guidance output information for cover photography of the book object BO to the processor 120 (step S115 ).

[0210] After receiving the confirmation character signal from the camera 130 and the guide output signal from the application 110 , the processor 120 outputs the cover shooting guide output signal to the display 140 (step S116 ).

[0211] The display 140 receiving the cover guide output signal of the processor 120 outputs the cover shooting guide CSG in response to the cover guide output signal (step S117). In this case, the display 140 outputs the cover shooting guide CSG to overlap (cover) the active area image ASV being output.

[0212] The user places a book object BO in the active area AS to capture the cover of the book object BO. If the book object BO is placed in the active area AS, the display 140 outputs an active area image ASV capturing the active area AS where the book object BO is placed. In this case, the active area image ASV output by the display 140 includes the cover of the book object BO.

[0213] If the active area image ASV including the cover of the book object BO is output (step S118), the image processing engine 150 generates a cover image by performing image recognition on the active area image ASV (step S119). That is, the image processing engine 150 performs image recognition on the area corresponding to the cover recognition area of ​​the cover shooting guide CSG in the entire area of ​​the active area image ASV. The image processing engine 150 generates a cover image based on the image recognition result and temporarily stores it in the memory 180.

[0214] The processor 120 generates a book recognition request signal including a cover image (step S120). That is, the processor 120 detects a cover image stored in the memory 180 and generates a book recognition request signal including the detected cover image. The processor 120 transmits the book recognition request signal to the first communication module 160 via the bus.

[0215] The first communication module 160 transmits the book identification request signal generated by the processor 120 to the service server 200 through the network, and receives a book identification response based on the response to the book identification request signal (step S121). In this case, the first communication module 160 can receive the book identification response, including one of the book information and the book identification error message.

[0216] The book information includes the book name and interactive object information. When the book reading assistance is performed, the interactive object information may include the interactive object image IOI of the interactive object IO.

[0217] The book recognition error message means that the cover image included in the book recognition request information is not the cover of the book object BO or the book object BO registered in the service server 200 .

[0218] When the book recognition response received by the first communication module 160 includes a book recognition error message (step S122, No), the reading auxiliary device 100 performs the above steps S117 to S121 again. In this case, the display 140 can also output the replacement guidance, position movement guidance, etc. of the book object BO together with the cover shooting guidance CSG.

[0219] When the book identification response received by the first communication module 160 includes book information (step S122, yes), the processor 120 transmits a book information output signal to the display 140 (step S123). That is, the processor 120 detects book information including the book name and interactive object information from the book identification response, and generates a book information output signal including the detected book information. The processor 120 transmits the generated book information output signal to the display 140.

[0220] The display 140 outputs the book information and the service start confirmation object of the book object BO in response to the book information output signal of the processor 120 (step S124). In this case, the display 140 may also output the book information and the service start confirmation object to overlap (cover) the active area image ASV.

[0221] When the user selects the service start confirmation object (step S125 , Yes), the application 110 transmits a reading assistance service start signal to the processor 120 (step S126 ).

[0222] The processor 120 detects book information from the book recognition response in response to the book reading assistance service start signal, and generates a book reading assistance information signal including the detected book information (step S127 ). The processor 120 transmits the generated book reading assistance information signal to the first communication module 160 .

[0223] The first communication module 160 transmits the reading assistance information signal generated by the processor 120 to the service server 200 through the network, and receives the reading assistance information based on the response to the reading assistance information signal (step S128). In this case, as an example, the reading assistance information includes a page image for identifying a page of the book object BO, an interactive object image IOI associated with the page of the book object BO, an additional image matching the page of the book object BO or the interactive object image IOI, etc. The first communication module 160 transmits the received reading assistance information to the processor 120.

[0224] The processor 120 stores the reading assistance information received through the first communication module 160 in the storage 170 (step S129). In this case, the storage 170 maintains the storage state of the reading assistance information corresponding to the book object BO until the reading assistance of the book object BO is completed.

[0225] The reading auxiliary information includes images, etc., so it takes a specified time to receive (download) it from the service server 200. In response, the processor 120 transmits the interactive object information output information included in the book information to the display 140, indicating the preparation state of the interactive object IO (step S130).

[0226] The display 140 outputs an interactive object preparation request screen including the interactive object information included in the book information in response to the interactive object information output information (step S131). In this case, the display 140 detects the interactive object image IOI based on the book information included in the book recognition response, and outputs the interactive object preparation request screen in which the detected interactive object image IOI is arranged according to the specified conditions. Thus, the display 140 outputs the interactive object preparation request screen until the reception of the reading auxiliary information is completed.

[0227] If the reading assistance information is received (step S132, yes), the processor 120 detects the page of the book object BO by comparing the book content BC of the active area image ASV captured by the camera 130 with the page image of the reading assistance information (step S133). The processor 120 transmits a page detection signal including the page to the storage 170.

[0228] The storage 170 detects a page image corresponding to the page included in the page detection signal from the pre-stored reading auxiliary information (step S134), and detects the reading auxiliary information including the detected page image (step S135). That is, the storage 170 detects the reading auxiliary information including the page image detected by step S134 from the pre-stored reading auxiliary information. The page image is not only an image of the photographed page, but also information included in the page, such as a specific object image, a page number, and a word included in the page. The storage 170 transmits the detected reading auxiliary information to the processor 120.

[0229] The processor 120 detects the additional image included in the received reading auxiliary information as the interactive content IC (step S136 ), and transmits an interactive content output signal including the detected interactive content IC to the display 140 (step S137 ).

[0230] The display 140 overlaps (covers) the interactive content IC outputting the interactive content output signal on the active area image ASV including the book content BC (step S138 ).

[0231] In the reading assistance method of the embodiment of the present invention, the output interactive content includes the photographed object reflected by the reflective object arranged in the active area.

[0232] Reference Fig.26 , the display 140 outputs an active area image ASV including the active area AS and the reflective object 600 arranged in the active area AS (step S205). That is, when the user arranges the reflective object 600 in the active area AS, the camera 130 photographs the active area AS and generates an active area image ASV including the reflective object 600. The camera 130 transmits the active area image ASV to the display 140 and the image processing engine 150. The display outputs the active area image ASV received from the camera 130.

[0233] The image processing engine 150 analyzes the active area image ASV, and detects the reflective object and the photographic object FS reflected by the reflective surface RS of the reflective object 600 based on the image analysis result (step S210). The image processing engine 150 detects the first image including the reflective object 600 based on the image analysis result of the active area image ASV. That is, the image processing engine 150 detects the area where the reflective object 600 is configured in the active area image ASV and detects it as the first image. The image processing engine 150 may perform image analysis on the first image and detect a partial area of ​​the reflective object 600 as the photographic object FS based on the image analysis result. In this case, as an example, the image processing engine 150 detects the reflective surface RS positioning area in the entire area of ​​the reflective object 600 as the photographic object FS.

[0234] The image processing engine 150 transmits an interactive content IC detection request including the detected reflective object 600 to the processor 120 (step S215), and the processor 120 detects the interactive content IC associated with at least one of the active area image ASV and the reflective object 600 (step S220). The processor 120 detects the reflective object 600 from the interactive content IC detection request in response to the interactive content IC detection request. The processor 120 transmits an additional image detection request including the detected reflective object 600 to the repository 170. The repository 170 detects the reading assistance information associated with the reflective object 600 of the additional image detection request and detects the additional image of the reading assistance information. The repository 170 generates a detection response including the detected additional image and transmits it to the processor 120. The processor 120 detects the additional image from the detection response of the repository 170 and sets the detected additional image as the interactive content IC.

[0235] In this case, the processor 120 may also detect the interactive content IC for which the composite area CA has been set. A partial area including the interactive content IC may be set as an object of the composite area CA.

[0236] The processor 120 generates a synthesis request signal including the interactive content IC and transmits the signal to the image processing engine 150 (step S225 ).

[0237] The image processing engine 150 generates a composite content CC that composites the photographed object FS into the interactive content IC in response to the composite request signal (step S230). That is, the image processing engine 150 detects the interactive content IC from the composite request signal. The image processing engine 150 composites the photographed object FS detected based on the image analysis result into the interactive content IC to generate the composite content CC. In this case, when the composite area CA is defined in the interactive content IC, the image processing engine 150 composites the photographed object FS detected based on the image analysis result into the composite area CA of the interactive content IC to generate the composite content CC.

[0238] The image processing engine 150 transmits the synthesized content CC to the processor 120 (step S235 ), and the processor 120 sets the synthesized content as the interactive content IC (step S240 ).

[0239] The processor 120 transmits an output signal including the interactive content IC to the display 140 (step S245), and the display 140 outputs the interactive content IC including the photographing object FS detected by the image processing engine 150 in response to the output signal (step S250). The display 140 outputs the interactive content IC to overlap the active area image ASV.

[0240] Although the embodiments are described above by limiting the embodiments and the accompanying drawings, a person skilled in the art of the present invention may make various modifications and variations based on the above. For example, even if the described techniques are performed in a different order from the described methods and / or the described systems, structures, devices, circuits and other structural elements are combined or combined in a different implementation from the described methods, or even if they are replaced or substituted by other structural elements or equivalent technical solutions, appropriate results can be achieved.

[0241] Therefore, other implementation modes, other examples and contents equivalent to the protection scope of the invention also fall within the protection scope of the present invention.

Claims

1. A reading assistance system, characterized in that: include: a reflective object, formed with a reflective surface, disposed in the active area; and The reading auxiliary device is arranged on the physical surface having the above-mentioned active area, receives the photographed object reflected by the reflective surface of the above-mentioned reflective object, and outputs the interactive content including the above-mentioned photographed object. The above-mentioned reading auxiliary device includes: an image processing engine, receiving an image of the activity area including the activity area, the reflective object, and the photographed object reflected by the reflective surface of the reflective object, performing image analysis on the activity area image, and detecting the reflective object and the photographed object based on the image analysis result; a processor connected to the image processing engine for communication, configured to detect the active area image and interactive content associated with at least one of the reflective objects and output an output signal including the interactive content; and The display is connected to the image processing engine and the processor for communication, and is used to output the active area image, and output interactive content including the photographed object detected by the image processing engine in response to the output signal, so that the output interactive content overlaps with the active area image.

2. The reading assistance system according to claim 1, characterized in that: The above reflection objects include: A main body, formed with the above-mentioned reflecting surface; An upper housing, in a frame shape, having a hole for exposing the reflective surface of the body, and disposed on the upper surface of the body; and The lower shell is arranged on the lower surface of the main body and is combined with the upper shell. The reflection surface reflects the photographed object disposed on the front side of the reading assistance device toward the reading assistance device.

3. The reading assistance system according to claim 1, characterized in that: The processor outputs a composite request signal including the interactive content. The image processing engine detects interactive content from the synthesis request signal, and generates synthesized content by synthesizing the photographic subject detected based on the image analysis result into the interactive content.

4. The reading assistance system according to claim 3, characterized in that: The processor sets the synthesized content generated by the image processing engine as the interactive content.

5. The reading assistance system according to claim 3, characterized in that: The processor detects the interactive content of the set synthesis area, The image processing engine synthesizes the photographed object detected based on the image analysis result into the synthesis area to generate the synthesized content.

6. The reading assistance system according to claim 1, characterized in that: The image processing engine detects a first image including the reflective object based on an image analysis result of the active area image, and detects a partial area of ​​the reflective object as the photographed object based on the image analysis result of the first image.

7. The reading assistance system according to claim 1, characterized in that: The reading auxiliary device further comprises a camera for capturing an image of the activity area including the activity area, the reflective object and the captured object reflected by the reflective surface of the reflective object, and transmitting the image of the activity area to the image processing engine and the display.

8. A reading assistance method, characterized in that: The steps include: Outputting, through a display, an image of the active area including the active area and a reflective object disposed in the active area; Analyzing the activity area image by an image processing engine and detecting the reflective object and the photographed object reflected by the reflective surface of the reflective object based on the image analysis result; Detecting, by a processor, interactive content associated with at least one of the active area image and the reflective object; Outputting an output signal including the interactive content through the processor; and The display is used to output the interactive content including the photographed object in response to the output signal, so that the output interactive content overlaps with the active area image.

9. The reading assistance method according to claim 8, characterized in that: The following steps are also included: Before the outputting step of the output signal, outputting, by the processor, a synthetic request signal including the interactive content detected in the interactive content detecting step; and The image processing engine detects the interactive content from the synthesis request signal in response to the synthesis request signal and generates synthesized content in which the photographed object detected based on the image analysis result is synthesized into the interactive content.

10. The reading assistance method according to claim 9, characterized in that: The method further includes the following step: before the outputting step of the output signal, setting the synthesized content generated in the synthesized content generating step as the interactive content by the processor.

11. The reading assistance method according to claim 9, characterized in that: In the above interactive content detection step, the interactive content of the set synthesis area is detected. In the step of generating the composite content, the composite content is generated by compositing the photographic subject detected based on the image analysis result into the composite area.

12. The reading assistance method according to claim 8, characterized in that: The above-mentioned step of detecting the photographed object comprises the following steps: Detecting a first image including the reflective object based on an image analysis result of the active area image; and Based on the image analysis result of the first image, a partial area of ​​the reflective object is detected as the photographed object.

13. The reading assistance method according to claim 8, characterized in that: The method further includes the following steps: using a camera to shoot an image of the activity area including the activity area, the reflective object, and the photographed object reflected by the reflective surface of the reflective object.

Images